JP5823270B2 - Image recognition apparatus and method - Google Patents

Description

  The present invention relates to an image recognition apparatus and method for digitalized images, and more particularly to not only automatically recognizing what each object is but also a plurality of objects included in an image. The present invention relates to an image recognition apparatus and method capable of automatically recognizing a place existing in a computer.

  In recent years, digital cameras have become widespread, and a huge number of digital images are taken and accumulated every day. In order to use images as information resources, it is desirable to convert them into text information that can be easily processed on a computer in advance. For this purpose, the image contents are converted into text information by image recognition without human intervention. Informatization is effective.

  One conventional approach is an image recognition method for estimating an object included in an image. For example, Torralba et al. Have proposed an image recognition method for estimating the name of an object included in a newly input image based on similar cases in a large-scale image data set that has been constructed in advance. This method is disclosed in Non-Patent Document 1.

  The conventional image recognition method uses about 80 million large-scale data sets constructed by collecting a large number of images labeled with names of objects included in the images in advance. That is, an image similar to a newly input image is searched from a large-scale data set, and the name of an object included in the input image is estimated by voting from a label assigned to the similar image group.

  In the related art, expecting the size of the data set, and the larger the data set, the more accurately the names of objects included in the newly input image are displayed in the similar images in the data set. Since many images to which the label shown in (2) is given are included, estimation by voting becomes possible. In addition, the larger the data set, the higher the probability that a similar image exists in the data set regardless of the name of the input object.

  The related art disclosed in Non-Patent Document 1 will be introduced below. FIG. 15 is a diagram showing functional blocks of an image recognition unit that performs image recognition according to the related art and auxiliary explanations (1) to (5). The image recognition unit 12 includes an image size normalization unit 81, an image feature amount conversion unit 82, a data set storage unit 83, a comparison unit 84, and a vote determination unit 85. Each unit is as follows. Function.

  The image size normalization unit 81 performs scaling to cope with the input of images of various sizes as shown in (1), and always converts them into images of a constant size as shown in (2). To do. FIG. 15 shows an example in which a normalized image having a height of 32 pixels and a width of 32 pixels is created as described in Non-Patent Document 1.

  The image feature amount conversion unit 82 extracts feature amounts from the normalized image as shown in (3), and converts them into image feature vectors. In Non-Patent Document 1, a color image expressed by three components of 32 pixels in height, 32 in width, and RGB is considered as a 32x32x3 = 3072 dimensional image vector, and zero average and norm normalization (unit norm) The 3072-dimensional feature vector is obtained by applying the signal normalization.

  As shown in (4), the data set accumulation unit 83 performs image size normalization unit 81 and image feature amount for each of the images labeled with the names of objects included in images collected in large quantities in advance. The feature vector converted by the conversion unit 82 and the set of labels attached to the image are accumulated on a large scale. In the example of Non-Patent Document 1, about 80 million large-scale image data sets relating to about 80,000 types of objects (the types are specified by the same number of about 80,000 words) are stored.

  The comparison unit 84, with respect to the input image, the feature vector converted by the image size normalization unit 81 and the image feature amount conversion unit 82, and each of all the feature vectors registered in the data set storage unit 83 Are compared, and the respective similarities are calculated. The degree of similarity indicates how similar the input image and the registered image corresponding to the feature vector are. Further, the comparison unit 84 lists a predetermined number M of search cases specified in order from registered images having a high degree of similarity as similar images. (4) shows an example where M = 24.

  As shown in (5), the voting determination unit 85 performs voting based on the labels assigned to the top M similar images. In other words, voting is to measure the appearance frequency of each label appearing on a similar image. The voting determination unit 85 further uses a label whose appearance frequency exceeds the set threshold as an estimation result of the name of the object included in the input image. (5) shows, as an example, the result of the correct estimation of “car” as included in the input image.

Antonio Torralba, Rob Fergus and William T. Freeman, "80 million tiny images: a large dataset for non-parametric object and scene recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 30, Issue 11, No. 11, November 2008, pp. 1958-1970.

  However, as problems of the prior art described in Non-Patent Document 1, (Problem 1) There is no special consideration for images containing multiple objects, and it is not supported, (Problem 2) Where objects exist in images There are three reasons: it cannot be recognized until now, and (Problem 3) there is room for improvement in estimation accuracy. Specifically, it is as follows.

  (Problem 1) As described above, the conventional image recognition technique described in Non-Patent Document 1 uses the entire input image as an estimation target and compares it with a data set. Even if a plurality of objects are included in the input image, it is not compared with the data set for each object, but the entire input image is used as an estimation target. Therefore, it cannot be said that it can always properly cope with recognition of an image including a plurality of objects. For example, with respect to an input image including “car” and “person” as shown in the example of FIG. 15, there was no special consideration for estimating each of these two objects with high accuracy.

  (Problem 2) The conventional image recognition technology described in Non-Patent Document 1 can estimate the name of an object included in the image, but recognizes where the object is located in the image. It wasn't. In order to utilize images as information resources and to improve video information processing, it is useful not only to estimate the names of objects contained in images, but also to recognize existing locations. It was not possible with technology.

  (Problem 3) In the conventional image recognition technology described in Non-Patent Document 1, the quality of the estimation accuracy is greatly influenced by the threshold set by the voting determination unit 85. For example, in the example of FIG. 15, when the threshold value is lowered, not only “automobile” included in the input image but also a plurality of estimation candidates such as “people” and “trees” are excessively listed. In some cases, the estimation result was strictly judged as “not applicable”. Therefore, there is room for improvement and stabilization of estimation accuracy.

  Based on the above points, the present invention solves the above-mentioned problems 1 and 2, and for only a plurality of objects (or targets) included in an image, it only recognizes what each object is automatically. The first object of the present invention is to provide an image recognition apparatus and method capable of automatically recognizing a position existing in an image.

  Further, the present invention solves the third problem in addition to the first object, and recognizes with high accuracy what each of the plurality of objects (or objects) included in the image is, A second object is to provide an image recognition apparatus and method capable of recognizing a position existing in an image.

  In order to achieve the above object, an image recognition apparatus according to the present invention includes a partial region extraction unit that cuts out a plurality of partial regions from an input image and sets each as a target image, and an image recognition unit that recognizes what the target image represents. The image recognition unit extracts a first image feature quantity from the target image and converts it into an image feature vector, and assigns a label to each of a plurality of given images. A data set storage unit that stores each image in association with an image feature vector converted by the image feature amount conversion unit, an image feature vector of the target image, and an image feature vector stored in the data set storage unit A comparison unit that compares each of the target image and obtains a similarity between each of the plurality of accumulated images, and the similarity is given to a predetermined number of images at a higher level. Label An image recognition unit including a voting determination unit that obtains a label whose appearance frequency satisfies a predetermined criterion as a recognition result indicating what the target image represents, and a plurality of partial regions extracted by the partial region extraction unit Recognizing what each of the plurality of objects included in the input image is based on the recognition result by the image recognition unit in each, and recognizing the position of each of the plurality of objects in the input image One feature.

  In the present invention, the partial region extraction unit may occupy a rectangular region having a side parallel to the vertical or horizontal axis of the input image as the partial region. And a second feature that includes a candidate selection unit that specifies and selects a plurality of rectangles according to size, and a rectangle extraction unit that cuts out the plurality of selected rectangular regions from the input image to form the partial regions. To do.

  In the present invention, the candidate selection unit may include a random number generation unit that specifies a predetermined number of relative positions and sizes occupied by the rectangular area in the input image using random numbers, or in a predetermined image A statistical information measuring unit that measures statistical information related to the relative position and size of the target in advance, a probability distribution storage unit that stores a prior probability distribution related to a predetermined combination of the measured position and size, and A third feature is that it includes a candidate extraction unit that specifies a predetermined number of the relative positions and sizes in descending order of prior probabilities.

  In the present invention, the partial region extraction unit further selects only a part of the plurality of rectangular regions selected by the candidate selection unit based on the overlapping relationship between the regions. A fourth feature is that a candidate selection unit is included, and the rectangle extraction unit cuts out only the selected region.

  Further, according to the present invention, the candidate selecting unit covers the input image into regions of arbitrary shapes based on similarity of color and / or texture characteristics, and covers the divided regions without excess or deficiency. An object reliability calculation unit that calculates a reliability regarding whether a plurality of rectangular regions selected by the candidate selection unit captures the target using the degree of the image as an index, and the calculated high reliability And a final candidate determination unit that selects only the part based on the small overlap of the plurality of rectangular regions selected by the candidate selection unit.

  According to the present invention, the region dividing unit includes a small region dividing unit that excessively divides the input image into a plurality of small regions having an arbitrary shape based on similarity of color and / or texture characteristics, and the plurality of excessively divided plural images. A target area selection unit that outputs an internal area that belongs to the small area and an external area that does not belong to the small area in the input image for each of the small areas; An area for calculating the area similarity of a pair of the small area expansion areas, and a small area expansion section that integrates an area similar in color and / or texture characteristics with the internal area and outputs it as a small area expansion area Among the pairs of small areas, the similarity calculation unit determines that all the small area pairs in which the area similarity of the corresponding small area expansion area pairs satisfies a predetermined standard belong to the same object, and integrates them. A small region integration unit that obtains a result of dividing into regions of any shape as small regions integrated as regions corresponding to each object by determining that small region pairs that do not satisfy the predetermined criteria belong to different objects The sixth feature is to include Note that an object is used as a word indicating an area as a background as well as an area occupied by an object included in an image.

  Further, according to the present invention, the partial region extraction unit includes a saliency extraction unit that extracts a saliency corresponding to a degree of attention for each pixel from the input image, and a side parallel to the vertical or horizontal axis of the input image. A rectangle that is a rectangular area and has a predetermined number of areas whose saliency satisfies a predetermined standard as the partial area, and is selected and specified by a relative position and size occupied by the area in the input image A seventh feature includes a determination unit and a rectangular extraction unit that cuts out the selected plurality of rectangular regions from the input image to form the partial regions.

  In the present invention, each of the labels satisfying the predetermined criterion in the voting determination unit is set as a candidate, a second image feature amount previously associated with the candidate, and a second image feature extracted from the target image. A re-determination recognizing unit that recognizes what the target image represents based on a comparison with a quantity, and the image recognizing unit recognizes what each of the plurality of targets included in the input image is. It is an eighth feature that the recognition result of the redetermination recognition unit is adopted instead of the recognition result.

  Further, according to the present invention, the image feature amount conversion unit converts an image feature vector having pixel values as elements as the first image feature amount from the target image, and the redetermination recognition unit includes the target image. A local feature amount extraction unit that extracts a local feature amount as the second image feature amount, and extracts local feature amounts from a plurality of given images, and performs clustering to calculate a representative vector of each cluster. A clustering processing unit, a code book storage unit that stores the calculated representative vector as a code book, and a local feature extracted from the target image is quantized into a representative vector with reference to the stored code book In addition, the vector quantization unit for measuring the appearance frequency of each representative vector to obtain an image feature vector, and a plurality of given images each assigned a label. An image feature vector storage unit that stores an image feature vector in which the vector quantization unit is applied to each image in association with each other, and a labeled image feature vector stored in the image feature vector storage unit as learning data As a machine learning unit that outputs a discriminator for each label, a discriminator storage unit that stores a discriminator for each label, and a label obtained as a recognition result in the voting determination unit The classifier corresponding to each is read from the classifier storage unit, and the image feature vector obtained by the vector quantization unit is input to the target image to obtain a label satisfying a predetermined criterion. A ninth feature is that it includes a discriminating unit that recognizes what the target image represents as a label.

  Further, according to the present invention, the identification unit further measures a co-occurrence relationship between labels from a plurality of given images each configured as an image including a plurality of targets and provided with a label corresponding to each target. A co-occurrence relationship measuring unit, a prior probability distribution accumulating unit for accumulating prior probabilities relating to co-occurrence of labels based on the measured co-occurrence relationship, each of the target images, and the voting determination unit in each of the target images A total reliability calculation unit that calculates a total reliability by multiplying the prior probability by a combination with the score output by the discriminator for each of the labels obtained as a recognition result in A tenth feature is that a label for each of the target images corresponding to the combination having the maximum overall reliability is a recognition result of what each of the plurality of targets included in the input image is.

  Further, according to the present invention, the co-occurrence relationship measurement unit is configured as an image each including a plurality of objects, and a label corresponding to each object and information on a rectangular area occupied by each object in the image are given. An eleventh feature is that a co-occurrence relationship between labels is measured as a co-occurrence relationship of area occupancy ratios of rectangular regions corresponding to the label from a plurality of images.

  Further, the present invention is such that at least one of the given plurality of images in the data set storage unit and the given plurality of images in the image feature vector storage unit is configured as an image including a single target. The twelfth feature.

  Further, in order to achieve the above object, the image recognition method of the present invention extracts a plurality of partial areas from the input image and extracts each of them as a target image, and an image for recognizing what the target image represents. A recognition step, wherein the image recognition step extracts a first image feature amount from the target image and converts it into an image feature vector, and assigns a label to each of a plurality of given images. And a data set storage step for storing each image in association with an image feature vector converted by the image feature amount conversion unit, an image feature vector of the target image, and an image stored in the data set storage unit A comparison step of comparing each of the feature vectors to obtain a similarity between the target image and each of the accumulated plurality of images, and the similarity is higher An image recognition step including a voting determination unit that obtains, as a recognition result of what the target image represents, a label whose appearance frequency satisfies a predetermined criterion among the given labels for a predetermined number of images, Recognizing what each of the plurality of targets included in the input image is based on the recognition result of the image recognition step in each of the partial regions cut out by the partial region extraction step, and for each of the plurality of targets A thirteenth feature is that the position in the input image is also recognized.

  Further, according to the present invention, each of the labels satisfying the predetermined criterion in the voting determination step is used as a candidate, a second image feature amount previously associated with the candidate, and a second image feature extracted from the target image. A re-determination recognition step for recognizing what the target image represents based on a comparison with a quantity, and the image recognition step for recognizing what each of the plurality of targets included in the input image is. It is a fourteenth feature that the recognition result of the redetermination recognition step is adopted instead of the recognition result.

  According to the first or thirteenth feature, a plurality of objects included in an image can be recognized up to where they exist in addition to what each object is. Therefore, the first purpose is achieved.

  According to the second feature, by extracting a rectangular partial region, for example, the x coordinate of the upper left vertex of the rectangle, the y coordinate of the upper left vertex of the rectangle without handling complicated information in the representation of the region shape Further, simplification is possible so that the rectangle can be uniquely defined with only four parameters such as the width of the rectangle and the height of the rectangle, and the processing capacity and the storage capacity for holding the region shape can be reduced.

  According to the third feature, since the partial area is determined based on the random number without performing complicated processing for determining the partial area, the partial area can be extracted in a shorter time, or instead. Compared to the case where partial areas are determined in order from a rectangle with a high probability of existence of an object calculated from existing cases, and the partial areas generated by random numbers are used as they are, the extraction of partial areas is emphasized with a greater emphasis on accuracy. It becomes possible.

  According to any of the fourth to sixth features, the signal in the image is analyzed, and the partial region is selected based on the probability that the object exists, so compared to the case where the selection is not applied, It is possible to extract candidate partial areas with higher accuracy.

  According to the seventh feature, since the partial area is obtained based on the saliency by analyzing the signal in the image, the partial area where the target may exist is obtained.

  According to the eighth, ninth, or fourteenth feature, first, according to the first or tenth feature, candidates for a large number of places and various kinds of object names (what the object is) are overexposed. Then, the final name of the object contained in the image is determined by determining in detail using a higher-level image feature quantity which is one of the N types of objects (targets) listed as candidates. In addition to improving the accuracy of estimation, it is possible to specify with high accuracy the location where the object truly exists from among the many object candidates listed in the partial region extraction unit. Therefore, the second goal is achieved.

  According to the tenth feature, since the recognition result is obtained by the total reliability based on the prior probability indicating whether the co-occurrence relationship in which the objects exist in the image at the same time is appropriate, a more accurate result can be obtained. .

  According to the eleventh feature, the co-occurrence relationship in which the objects exist simultaneously in the image is described by the area occupancy ratio of the objects, and the prior occurrence relating to the co-occurrence relationship including the size relationship between the objects. Since the recognition result is obtained based on the total reliability based on the probability, a more accurate result can be obtained.

  According to the twelfth feature, consistency between feature vectors to be compared at the time of executing the recognition process can be taken, and the estimation accuracy of the name of the object included in the image can be improved.

It is a figure which shows the functional block of the image recognition apparatus which concerns on embodiment of this invention, and its description. It is a figure which shows the functional block of the partial region extraction part which concerns on one Example, and its description. It is a figure which shows the functional block of the candidate selection part which concerns on one Example, and its description. It is a figure which shows the functional block of the candidate selection part which concerns on one Example, and its description. It is a figure which shows the functional block of the partial region extraction part which concerns on one Example, and its description. It is a figure which shows the functional block of a candidate selection part, and its description. It is a functional block diagram of an area dividing unit. It is a figure explaining the outline | summary of the process by an area | region similarity calculation part and a small area integration part. It is a figure explaining the production | generation result of the small area expansion area | region produced | generated by the small area integration part etc. from the small area | region which an area division part outputs, and the grouping of the small area based on the result. It is a figure for demonstrating the process of an object reliability measurement part. It is a figure which shows the functional block of a redetermination recognition part, and its description. It is a figure explaining calculation of the image signal value as a local feature-value. It is explanatory drawing about the calculation of the image signal value performed by the local feature-value extraction part about the various acquired images collected. It is explanatory drawing about the clustering performed by a clustering process part, the vector quantization performed by a vector quantization part, and histogram formation. It is a figure which concerns on a prior art and is a figure which shows the functional block of the image recognition part used by this invention, and the description of the process of the said image recognition part. It is a figure which shows the functional block of the partial region extraction part which concerns on one Example, and its description. It is a functional block diagram of the identification part which concerns on one Example. It is a functional block diagram of the image recognition apparatus which concerns on one Example.

  FIG. 1 shows functional blocks of the image recognition apparatus according to the embodiment of the present invention and a description of the processing. The image recognition device 1 includes a partial region extraction unit 11, an image recognition unit 12, and a redetermination recognition unit 13, and identifies what each of the plurality of objects included in the input image is, and The position of each of the plurality of objects in the input image is also identified. In the first embodiment, the redetermination recognition unit 13 is omitted. In the second embodiment, the redetermination recognition unit 13 is used without being omitted, and it is possible to identify what each object is with higher accuracy than in the first embodiment.

  The partial area extraction unit 11 extracts a partial area from the input image. In the example of the input image shown in (1), two types of targets, “car” and “person”, are included, but the partial region extraction unit 11 is configured to extract the partial region that truly includes the target without missing it. A large number of areas are extracted slightly excessively, and extracted as candidate areas that may include the target as shown in (2). In the example of the input image of (1), the boundary of the candidate area extracted in (2) is also drawn on the input image as a white line.

  The image recognition unit 12 has the same individual functional blocks as those described with reference to FIG. The present invention relates to the image recognition unit 12, the partial region extraction unit 11 (first and second embodiments) as the former configuration, and the redetermination recognition unit 13 (second embodiment) as the subsequent configuration. ) With the feature. The image recognition unit 12 performs processing similar to that described with reference to FIG. 15 as the target image to be processed for each of the partial regions as the target candidate regions extracted from the input image by the partial region extraction unit 11 as the preceding stage. I do.

  The image recognition unit 12 receives each partial region extracted by the partial region extraction unit 11 as a target image to be input, and estimates what is included in each of them as shown in (3). That is, as described with reference to FIG. 15, each input target image is converted into an image feature vector by the processing of the image size normalization unit 81 and the image feature amount conversion unit 82, and the image feature vector An image similar to the target image is searched from the data set storage unit 83 by the comparison unit 84, and the voting determination unit 85 votes based on the labels attached to the searched similar image group, and measures the appearance frequency of each label. To do.

  Examples of labels for each target image are as shown in (3). In this example, by setting the threshold value in the voting determination unit 85 to be low, a large number of estimation results as to what is the target are output slightly excessively as a candidate list. That is, when a partial area that truly includes “car” is input as the target image, not only the correct “car” as an estimation result, but also “person”, “dog”, “building”, “bike”, Several names are listed as candidates, including incorrect names.

  In the second embodiment, it is particularly necessary to set the threshold value to be lower and output the candidate list in an excessive manner as in the above example. At this time, a threshold value is commonly set in each target image, and the number of names listed in the candidate list in each target image is changed while changing the threshold value so that the number of names satisfies a predetermined standard. The threshold value may be automatically set. In the first embodiment, the highest name in the candidate list may be used as the final recognition result.

  The redetermination recognition unit 13 receives the target images extracted by the partial region extraction unit 11 and obtained from the candidate list by the image recognition unit 12 one by one together with the candidate list corresponding to the target image, and the target image is Which of the N types of names listed in the candidate list is represented is re-determined in detail using an image feature amount higher than the image feature amount used by the image recognition unit 12. Note that the value of N is determined by the setting of the image recognition unit 12, and generally differs depending on the target image.

  By this redetermination, in the candidate list consisting of N types of names as shown in (3), in the first embodiment, names other than the target actually represented in the target image are listed as the first, and the actual target Even in the case where the name is listed at the lower level, the actual name can be obtained more reliably as the final authorization result.

  In addition, if the target image itself is excessively extracted as a result, the candidate list including N types of names is determined by the re-determination if the image does not capture any target. If it does not correspond to any of the above, it will be judged more reliably. That is, as shown in (4), the target image whose candidate list is “Kirin”, “Church”,... Does not properly capture any target. In the first embodiment, depending on the threshold value setting, Even if it is such a target image, it may be determined as “giraffe” or the like, whereas in the second embodiment, it is determined more reliably as “not applicable”.

  The redetermination recognition unit 13 itself recognizes with high accuracy whether or not the image represents a certain target designated by the user or the like. It does not automatically recognize what it represents. It is not realistic from the viewpoint of the amount of calculation to apply the redetermination recognition unit 13 to any number of objects. In the second embodiment of the present invention, in particular, by providing the partial region extraction unit 11 and the image recognition unit 12 in the preceding stage of the redetermination recognition unit 13, what the image represents is estimated with a certain degree of accuracy. As a result, the re-determination recognition unit 13 can perform final estimation with high accuracy with a realistic amount of calculation.

  In both the first and second embodiments, the final recognition result is given as the name of the target image determined to capture some target in the input image and the position of the target image in the input image. In the example shown in (5), “car” and its position, “person” and its position are obtained as recognition results from the input image of (1).

  Hereinafter, specific configurations of the partial area extraction unit 11, the image recognition unit 12, the redetermination recognition unit 13, and the three functional blocks will be described. First, three examples of the partial region extraction unit 11 will be described as [Example A], [Example B], and [Example C].

[Example A]
FIG. 2 shows a functional block of the partial area extraction unit 11 in the embodiment A and a description (1) of the processing. In Example A, the partial region extraction unit 11 includes a candidate selection unit 21 and a rectangle extraction unit 22.

  The candidate selection unit 21 selects candidates for regions that may be occupied by objects included in the input image as in (1). Here, it is assumed that the shape of the extracted partial region is a rectangle, and the rectangle is arranged with its sides parallel to each other without being inclined with respect to the rectangle of the entire input image. A rectangle can be uniquely defined with only four parameters: the x coordinate of the upper left vertex of the rectangle, the y coordinate of the upper left vertex of the rectangle, the width of the rectangle, and the height of the rectangle, without handling complex information in the area shape representation. . As shown in (2), a plurality of rectangular parameters of the selected candidate area are output as a set.

  All four rectangle parameters use values normalized to a value of 0 to 1 by the width and height of the original image. That is, a rectangular width of 1 is equal to the width of the original image, and similarly a rectangular height of 1 is equal to the height of the original image. If the x coordinate of the upper left vertex of the rectangle is 0, the rectangle is located at the leftmost edge in the original image. Similarly, if the y coordinate of the upper left vertex of the rectangle is 0, the rectangle is located at the uppermost edge in the original image. It means to do. If the x coordinate of the upper left vertex of the rectangle is 1, the rectangle is positioned at the rightmost end in the original image. Similarly, if the y coordinate of the upper left vertex of the rectangle is 1, the rectangle is positioned at the lowest end in the original image. It means to do.

  The parameter that defines the rectangle may employ the x coordinate of the center of the rectangle instead of the x coordinate of the upper left vertex, and the y coordinate of the center of the rectangle instead of the y coordinate of the upper left vertex. By making a rectangle composed of sides parallel to the sides of the input image, more generally, the rectangle can be defined by only a total of four parameters for the vertical and horizontal positions and vertical and horizontal sizes. In addition, by setting the position and size relative to the input image, the rectangle defining parameter can be commonly used for input images of an arbitrary size.

  As shown in (3), the rectangle extraction unit 22 cuts out a partial area as a corresponding rectangle for each of the rectangle parameters output by the candidate selection unit 21 from the input image. As described with reference to FIG. 1, the partial area is a candidate area where a target can exist, and each of the partial areas is passed to the image recognition unit 12 as a target image.

  Functional blocks according to two embodiments a and b of the candidate selection unit 21 and their descriptions are shown in FIGS. 3 and 4, respectively. In the embodiment a of FIG. 3, the candidate selection unit 21 includes a random number generation unit 24. The random number generator 24 determines a predetermined number of sets of four rectangular parameters based on random numbers as in (1). For example, when using four rectangle parameters specified by the x coordinate of the upper left vertex of the standardized rectangle, the y coordinate of the upper left vertex of the rectangle, the width of the rectangle, and the height of the rectangle, a range of 0 to 1 Generate random numbers to determine the value of each parameter.

  In the embodiment b of FIG. 4, the candidate selection unit 21 includes a statistical information measurement unit 25, a probability distribution storage unit 26, and a candidate extraction unit 27, and measures rectangular parameters with a high appearance probability based on existing cases in advance. The rectangle parameters are determined in descending order of appearance probability. For the determination method, the following method of Non-Patent Document 2 can be used.

  [Non-Patent Document 2] Rahtu E., Kannala J., Blaschko M. B., "Learning a Category Independent Object Detection Cascade," Proceedings of International Conference on Computer Vision (ICCV 2011).

  As shown in (1), the statistical information measuring unit 25 measures the prior probability distribution regarding the place where the object exists from the existing cases with respect to the four parameters of the rectangle. For this reason, for a wide variety of objects, a large number of images including them are collected, and the rectangular parameters of the positions where the objects are included are visually associated with each other. In Non-Patent Document 2, the x-coordinate RX of the center point of the rectangle, the y-coordinate RY of the center point of the rectangle, the width RW of the rectangle, and the height RH of the rectangle are adopted as the four parameters. The value is measured with.

  Further, the statistical information measurement unit 25 sets a class interval in which the range of 0 to 1 is P equally divided for each of the measured rectangular parameters, and obtains a four-dimensional histogram. The first class has parameter values 0 to 1 / P, the second class has 1 / P to 2 / P, ..., the qth class has (q-1) / P to q / In the P,..., P th class, the rectangles whose parameter values are in the range of (P-1) / P to 1 are counted. P is set in advance as a fixed value. Here, as a numerical example that can cover various rectangles, for example, if P = 80, the space of the prior probability distribution becomes a huge space that reaches the fourth power of 80, and the storage capacity in terms of mounting It is difficult to handle.

  Therefore, the probability distribution accumulating unit 26 accumulates the prior probability distribution related to the place where the target exists in order to cope with the above-mentioned difficulty. Specifically, as shown in (2), three two-dimensional histograms P (RW, RH), P (RY | RH), and P (RX | RW) measured by the statistical information measurement unit 25 are stored. To do. In (2), those visualized are schematically shown as (21), (22) and (23), respectively. Shading corresponds to the magnitude of the probability.In (21), the probability is large near the origin (0,0), and in (22) and (23), the probability is near the point (1/2, 1). Although it is large, this indicates that there is a high probability that a target rectangle is formed in almost the entire input image.

The candidate extraction unit 27 selects the candidate areas in the descending order of prior probabilities using a combination of four rectangular parameters. At this time, the three two-dimensional histograms P (RW, RH), P (RY | RH), P (RX | RW) stored in advance in the probability distribution storage unit 26 are read out, and four rectangles are obtained by their product. The parameter prior probability distribution P (RX, RY, RW, RH) is approximately calculated as follows:
P (RX, RY, RW, RH) = P (RY | RH) P (RX | RW) P (RW, RH)

  Then, NR combinations of rectangular parameters are selected in descending order of P (RX, RY, RW, RH). A fixed value is set in advance for NR. In Non-Patent Document 2, NR = 10,000 is adopted.

[Example B]
FIG. 5 shows functional blocks of the partial area extracting unit 11 according to the embodiment B and the description thereof. As a difference from the embodiment A shown in FIG. 2, the partial region extraction unit 11 according to the embodiment B includes a candidate selection unit 23 as an additional configuration. Candidate selection unit 23, as shown in (2), from the set of a large number of rectangular parameters selected by candidate selection unit 21, the one representing a highly reliable rectangle in which the target exists is selected and the selected The parameter set is passed to the rectangle extraction unit 22. For example, about 10 ⁵ sets of parameters are selected by the candidate selection unit 21, but the number is narrowed down to about 10 ² to 10 ³ by the selection by the candidate selection unit 23.

  FIG. 6 shows a functional block of the candidate selection unit 23 and a description thereof. The candidate selecting unit 23 includes an area dividing unit 28, an object reliability measuring unit 29, and a final candidate determining unit 210. The candidate selection unit 23 indicates that each candidate area selected as a rectangular parameter by the above-described candidate selection unit 21 shown in (3) may include a target such as an object, as in (1). Using this signal, the reliability is calculated, and only candidate areas with high reliability are selected as shown in (4).

  The area dividing unit 28 divides the input image into similar small areas such as colors and textures as shown in (2). The area dividing unit 28 includes Patent Document 1 (prior application by the present inventors: image area dividing apparatus, image area dividing method and image area dividing program) or Patent Document 2 (prior application by the present inventors: image area dividing). The apparatus described in the apparatus, the image area dividing method, and the image area dividing program can be used.

[Patent Document 1] Japanese Unexamined Patent Application Publication No. 2011-150605 “Image Region Dividing Device, Image Region Dividing Method, and Image Region Dividing Program”
[Patent Document 2] Japanese Patent Application No. 2010-232914 “Image Region Dividing Device, Image Region Dividing Method, and Image Region Dividing Program”

  FIG. 7 shows functional blocks of the area dividing unit 28. The area dividing unit 28 subdivides the input image into a plurality of small areas based on color characteristics, and for each of the plurality of excessively divided small areas, A target area selection unit 2 that outputs an internal area that belongs to an area and an external area that does not belong to the small area, and an area similar in color characteristic to the corresponding internal area among the external areas Among the small region pairs, the small region expansion unit 30 that integrates and outputs as a small region expansion region, the region similarity calculation unit 6 that calculates the region similarity of the pair of the small region expansion regions, and the small region pair correspond to each other. It is determined that all the small region pairs in which the region similarity of the small region extended region pair satisfies the predetermined standard belong to the same object and are integrated, and the small region pairs that do not satisfy the predetermined standard are determined to belong to different objects. about Thus a small region integrating unit 7 to obtain each object, separates and extracts each object included in the input image.

  The area dividing unit 28 realized by using the image area dividing apparatus described in Patent Document 1 is roughly divided as described above, and is divided into a small area dividing unit 10, a target area selecting unit 2, and a small area expanding unit 30. Extraction of objects included in the image is realized by the five functional blocks of the region similarity calculation unit 6 and the small region integration unit 7. The extracted object is extracted as an arbitrary shape corresponding to the object, not as a rectangle, and is used as a region in the present invention. The object includes not only a region occupied by a target such as an object included in the image but also a region serving as a background. The function outline of each part is as follows.

  The small area dividing unit 10 divides the image into much smaller areas than the number of objects actually included in the image based on the similarity of the properties of the image signal such as color and luminance. The details are disclosed in Patent Document 1 and the like.

  The small area expansion unit 30 statistically models the luminance and color distribution in the small area of interest, and identifies pixels that follow the same luminance and color distribution model as the original small area elsewhere in the image Then, an extended area is obtained by combining with the original small area. By combining with the target area selection unit, one extended area is obtained for each of the small areas obtained by the excessive division. The details are disclosed in Patent Document 1 and the like.

  The region similarity calculation unit 6 measures and digitizes the similarity between the extended regions as a ratio including matching pixels. Here, the region similarity S (R1, R2) between the small region expansion regions is calculated from the following equation (Equation 1).

  In Equation 1, the numerator E1∩E2 is the number of pixels belonging to both the small region expansion regions E1 and E2 corresponding to the small region pair R1 and R2, and the denominator E1∪E2 is either the small region expansion region region E1 or E2. Indicates the number of pixels to which it belongs. Also, 0 ≦ S (E1, E2) ≦ 1, S (E1, E2) = 1 if the regions E1 and E2 are completely matched, and S (E1, E2) if there is no matching pixel = 0. A larger region similarity indicates that regions (including shape and position) are more similar.

  The small region integration unit 7 determines that the two expansion regions are similar to each other when the similarity between the expansion regions calculated by the region similarity calculation unit 6 exceeds a preset threshold. It is determined that the original small areas of the extended area belong to the same object, and are integrated into the same group.

  In the integration process as described above in the image region dividing device described in Patent Document 1, each object included in an image has a characteristic color distribution. This is based on the characteristic that the color distribution does not change significantly. Therefore, when there are many overlapping areas in the extended areas output based on the color distribution of each of the two small areas selected for a certain color distribution (including cases where the degree of coincidence of position, shape, etc. is large) The selected subregions have the same color distribution and the two selected subregions (one selected subregion pair) are likely to be part of the same object To do.

  FIG. 8 shows an outline of processing by the region similarity calculation unit 6 and the small region integration unit 7. That is, from the result (a) of being excessively divided by the small region dividing unit 1, each of (b) small region R1 and (d) small region R2 selected by the target region selecting unit 2 is converted by the small region expanding unit 30. In this example, (c) the expansion area E1 and (e) the expansion area E2 are obtained by expansion. Since the similarity is calculated as 0.92 from the ratio of the matching pixels in the extension region E1 and the extension region E2 by the region similarity calculation unit 6, and exceeds a predetermined threshold value 0.90, the small region integration unit 7 performs the original small region 1 And integrate the small area 2.

  FIG. 9 shows an example in which the small region integration unit 7 obtains region similarity for all the small region pairs in the input image and divides the small regions into groups corresponding to the respective objects as shown in FIG. As a result of dividing the input image into small areas by the small area dividing unit 1, 17 small areas R1 to R17 are generated and identified by numbers. Corresponding small region expansion regions E1 to E17 are obtained for each small region by the processing of the target region selection unit 2 and the small region expansion unit 30. As a result of determining the area similarity of all pairs of small area expansion areas and determining whether it is above or below a predetermined threshold, 3 groups of small area expansion areas ((E14, E15, E16, E17), (E5, E6, E7, E8, E9, E10, E11, E12, E13), (E1, E2, E3, E4)] are obtained. The three groups are grouped so that all the small region extended region pairs in each group have a region similarity higher than a predetermined value, and all pairs outside each group have a predetermined value or less. Therefore, from these 3 groups, the corresponding 3 groups of the original small area (Group 1 (R14, R15, R16, R17), Group 2 (R5, R6, R7, R8, R9, R10, R11, R12, R13), Group 3 (R1, R2, R3, R4)]. Each group is considered part of each object.

  As described above, the image region dividing device described in Patent Document 1 divides an image into an excessive number of small regions, and then, based on the similarity between the extended regions for each small region, Integration determination is performed on a unit basis, and an object included in the image is extracted. The area dividing unit 28 of the present invention is configured using the image dividing apparatus.

  Note that only the small region dividing unit 10 in the configuration of FIG. 7 may be used as the region dividing unit 28 of the present invention, and the excessively divided small region may be passed to the object reliability measuring unit 29 as it is. This is because, as will be described later, according to the object reliability measurement unit 29, it is possible to construct a rectangle that appropriately captures a target even from an excessively divided small region.

  The object reliability measurement unit 29 quantifies the object-likeness of each candidate area (degree of whether an object or some target is captured) as the reliability based on the area division result. The approach described in Non-Patent Document 3 can be used for this method.

  [Non-Patent Document 3] B. Alexe, T. Deselaers, and V. Ferrari., "What is an object ?," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR2010), June 2010.

  FIG. 10 is a diagram for explaining the processing of the object reliability measurement unit 29. As shown in (2), the image is divided into a plurality of regions s by the region dividing unit 28, and the region division result RD is configured as a plurality of regions s. In the example of (2), each side of the body frame, each window, each tire, and the like are obtained as the region s from the region corresponding to the vehicle as the object.

  In addition, as shown in (1), the candidate selection unit 21 has obtained as candidates some rectangles w in which an object may exist in the input image. The object reliability measurement unit 29 calculates an object-likeness reliability SS (w) from the following equation (Equation 2) for each rectangle w that is a candidate for a region where the object exists.

  The reliability SS (w) of object-likeness takes a value of 0 to 1, and the larger value means that the rectangle w is more likely to exist. Here, | s∩w | indicates the area inside the rectangle w for the region s, | s \ w | indicates the area outside the rectangle w for the region s, and | w | indicates the area of the rectangle w. In either case of digital images, the number of pixels can be measured instead of the area. The second term on the right side indicates that the sum is obtained for all the regions s included in the region division result RD.

  For example, for the rectangles w1, w2, and w3 shown in (3), each region s corresponding to each part of the car object in (Expression 2) contributes to the calculation of the reliability SS (w). And according to (Equation 2), the reliability of the rectangle that covers the area with as much size as possible is as high as possible, so when comparing between the rectangles w1, w2 and w3, the reliability of the rectangle w2 Is calculated high. Note that (4) shows | s∩w1 | and | s \ w1 | in (Expression 2) as an example in the relationship between the region corresponding to the tire in the region s and the rectangle w1.

  The final candidate determination unit 210 uses the following algorithm to select a candidate selection unit using the reliability SS (w) of the object-likeness of each rectangle w that is a candidate area measured by the object reliability measurement unit 29 and each rectangle parameter. It is finally determined whether or not the initial candidate area generated in 21 can be adopted as a target area where an object or the like exists.

[algorithm]
Step 1. The rectangles that are candidate regions are arranged in descending order of reliability SS (w), and the order i in which the repetitive determination algorithm in step 2 and subsequent steps is applied.
Step 2. If the i-th rectangle overlaps the rectangle adopted so far with a certain ratio RO or less, the i-th rectangle is adopted as the target area, otherwise it is not adopted.
Step 3. If the number of hires reaches the number of final candidates NF specified by the parameter, the process ends.
Step 4. If there remains a rectangle to be adopted / not adopted, i is incremented and the value returns to 2. If it does not remain, RO is reduced, and the determination in step 2 is restarted from the candidate i that has the highest reliability SS (w) and is not adopted.

  The algorithm starts with the number of adoptions set to an initial value of zero and the counter i set to an initial value of 1, and the final candidate number NF of the end determination in step 3 is set smaller than the predetermined number selected by the candidate selection unit 21. To do. In order to reduce RO in Step 4, a predetermined method is determined such as multiplying by a positive constant less than 1.

[Example C]
FIG. 16 is a functional block diagram of the partial area extracting unit 11 according to the embodiment C. The partial area extraction unit 11 includes a saliency extraction unit 210, a rectangle determination unit 230, and a rectangle extraction unit 22. The rectangular extraction unit 22 is the same as in the embodiments A and B.

  The saliency extraction unit 210 calculates the saliency as the degree that a person is particularly gazing in the image from each pixel of the input image according to the method disclosed in Non-Patent Document 5. For example, in the case of an image in which a red flower growing in grass village is photographed, it is considered that a human gazes at a region corresponding to the red flower, and a high saliency is calculated for the region.

  [Non-Patent Document 5] Ming-Ming Cheng, Guo-Xin Zhang, Niloy J. Mitra, Xiaolei Huang, Shi-Min Hu. Global Contrast based Salient Region Detection. IEEE CVPR, p. 409-416, Colorado Springs, Colorado, USA, June 21-23, 2011.

  Since the saliency is defined for each pixel of the input image, if the input image is configured with a size of vertical IH pixel × horizontal IW, the pixel saliency extraction unit 210 has the same size of vertical IH row × horizontal IW column. A saliency map composed of arrays is calculated. Here, the saliency of the place corresponding to each pixel of the input image is stored in the saliency map. The saliency is obtained as a real number normalized within the range of 0 to 1, and the greater the value, the stronger the degree of attention. Here, the saliency of the element in the Py row and Px column in the saliency map is denoted as SAL (Px, Py).

  The rectangle determination unit 230 determines a predetermined number CN of rectangular regions as candidate regions in which the target exists from the saliency map obtained by the saliency extraction unit 210, and uses the same method as the rectangular parameter in the candidate selection unit 21 Then, the determined rectangular area is transferred to the rectangle extracting unit 22. The procedure for determining is as follows. When starting the procedure, the fixed number CN as an extraction target in the rectangular extraction unit 22 and the threshold values TV, TS, and TE are determined in advance manually. All threshold values are set as real numbers in the range of 0 to 1.

(Procedure 0)
The center of gravity (Ox, Oy) of the saliency in the saliency map is obtained as in the following equation (Equation 3), and the procedure proceeds to step 1.

(step 1)
A rectangle including the center of gravity and having a maximum area where the saliency of all the included elements exceeds the threshold TV is specified in the saliency map. That is, the saliency of all the elements included in the specified rectangle exceeds the threshold value TV. In addition, any point on the outer periphery (the outer periphery having a width of one pixel) surrounding the specific rectangle is an element equal to or less than the threshold TV. The threshold TV can be set to 0.8 in advance, for example, but it may be set to 1.0 which is the maximum settable value. If there is at least one element that exceeds the threshold TV, go to step 2, otherwise go to step 3.

(Step 2)
For the rectangle specified in step 1, if the percentage of the specified rectangle in the input image exceeds the threshold value TS, this specified rectangle is determined as one of the CN candidate areas, and at the same time, the determined candidate After the saliency of all the elements in the saliency map included in the area is replaced with 0, the procedure proceeds to step 3. If not, the candidate area is not fixed and the process proceeds to step 3. For example, TS is set to 0.05, and a very small specific rectangle having an area ratio lower than this is prevented from being extracted as a candidate region.

  Note that the orientation of the specific rectangle is not arbitrary, and is such that each side is parallel to the vertical and horizontal sides of the input image. Also, by replacing the saliency with 0, it is possible to sequentially obtain a plurality of specific rectangles as candidate regions while avoiding mutual overlap.

(Step 3)
Next, the value of the threshold TV is updated. _After TV _update = α _before TV _update , α is set as a real number in the range of 0 to 1. Therefore, the threshold TV decreases each time it is updated. For example, α = 0.99. Thereafter, returning to the procedure 0, the determination of the next candidate area is repeated, but when the determined candidate areas reach the target number CN, the determination process is terminated. Alternatively, although the number of CNs has not been reached, as a result of updating the threshold value TV, the determination process is also terminated when the value of TV falls below the threshold value TE. In this case, less than CN rectangular areas are determined, but it is possible to prevent a specific rectangle having a too low degree of saliency being included as a candidate area. In this sense, the threshold value TE is set to TE = 0.20, for example.

  In determining the rectangular candidate area, the saliency extraction unit 210 applies a Gaussian filter to the saliency map in advance, and smoothes the saliency that is considered to be extremely different from the surrounding noise. May be.

  Note that the saliency extraction unit 210 and the rectangle determination unit 230 in Example C generally correspond to the candidate selection unit 21 and candidate selection unit 23 in Example B shown in FIG. The reliability SS (w) in Example B has substantially the same significance. As a difference, the example B is preferable in that the accuracy can be improved by learning the prior probability regarding the existence of the object by the configuration of FIG.

  The three examples [Example A], [Example B], and [Example C] of the partial region extraction unit 11 have been described above. Next, the redetermination recognition unit 13 (see FIG. 1) will be described.

  FIG. 11 is a diagram illustrating a functional block of the redetermination recognition unit 13 and a description thereof. The re-determination recognition unit 13 includes a local feature amount extraction unit 31, a clustering processing unit 32, a code book storage unit 33, a vector quantization unit 34, an image feature vector storage unit 35, a machine learning unit 36, a classifier storage unit 37, and a discrimination Part 38 is included.

  The local feature amount extraction unit 31 extracts a plurality of points with large signal changes such as edges and irregularities from the image as key points, and calculates local feature amounts (image signal values) from colors, shapes, patterns, etc. near each key point. ) Is output. FIG. 12 is a diagram for explaining calculation of an image signal value as the local feature amount. In the example of the dog image shown in (1) of FIG. 12, as shown in (2), a plurality of local feature amounts extracted with the eyes, nose, contour, etc. as key points are output.

  The calculation of the local feature amount performed here can be realized using a known technique. For example, SIFT (Scale-Invariant Feature Transform, Lowe, D. “Distinctive Image Features from scale-invariant keypoints,” International journal of Computer Vision, Vol. 60, No. .2, pp. 91-110, 2004) can be used. In this case, as shown in (3) of FIG. 12, the local feature amount based on the SIFT feature amount is output as a 128-dimensional vector.

  By using the above-mentioned SIFT when calculating local feature values, the same key points are extracted and the same feature vector is extracted even if the images have different appearances, such as rotation and size, if they are the same subject and the same content. Extracted. In SIFT, each process of (A) key point detection and (B) feature vector extraction is performed in the following flow.

[Process flow of SIFT]
(A) Keypoint detection
(a) Detection of key point candidate points
(b) Localizing key points
(B) Feature vector extraction
(c) Calculation of orientation
(d) Feature extraction

  In the detection of key point candidate points in (a), a plurality of points with large signal changes such as edges and irregularities are detected from the image by DoG (Difference-of-Gaussian) processing as key point candidate points. Change the scale of the Gaussian function in several stages, create multiple smoothed images convoluted with the Gaussian function and the input image, and key points are the points that become extreme values in the difference image (DoG image) of those smoothed images Detect as candidate points.

  In the localization of key points in (b), the key points that can be stably extracted from the key point candidate points detected in (a) are narrowed down. That is, a point having a small contrast and a point having a large main curvature are deleted from the key point candidate points as points affected by noise and points not suitable for stable extraction.

  In the orientation calculation of (c), in order to extract the same feature vector even if the image is rotated at the same key point, the direction characterizing each key point is determined from the gradient of each point in the smoothed image. calculate. Specifically, a histogram relating to the gradient direction and gradient intensity is measured from a rectangular area around the key point. First, classification is performed by a class quantized to 36 with respect to the gradient direction. Next, the gradient strength is added to the classified class, and the direction of the class showing the mode in the histogram is calculated as the orientation.

  In the feature quantity extraction in (d), the feature vector extraction target area at each key point is normalized based on the orientation obtained in (c), and the feature vector extraction target around the key point extracted by normalization is extracted. A feature vector is calculated from the region and set as a local feature amount.

  The local feature quantity extracted by the local feature quantity extraction unit 31 is not limited to the SIFT feature quantity, but other types of local feature quantities such as SURF (Speeded Up Robust Features) and HOG (Histograms of Oriented Gradients) are adopted. May be. SURF is detailed in the following non-patent document 6, and HOG is detailed in the following non-patent document 7.

  [Non-Patent Document 6] Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool, "SURF: Speeded-Up Robust Features, In Ninth European Conference on Computer Vision, 2006

  [Non-Patent Document 7] Navneet, Dalal and Bill Triggs., "Histograms of Oriented Gradients for Human Detection," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. II, pages 886-893, June 2005.

  FIG. 13 is an explanatory diagram of image signal value calculation performed by the local feature quantity extraction unit 31 for various collected images. FIG. 14 is an explanatory diagram of clustering performed by the clustering processing unit 32, vector quantization performed by the vector quantization unit 34, and histogram formation. Hereinafter, description will be made with reference to FIGS. 13 and 14 as appropriate.

  As shown in FIG. 13, the clustering processing unit 32 uses the local feature amount extracted by the local feature amount extraction unit 31 from a variety of different collected images, as shown in FIG. By plotting on the space (each x point is a local feature), the distribution characteristics in the space of the local feature set are measured, the space is clustered with local features close to each other, and each cluster Centroid (representative vector, xx points in FIG. 14A) is output as a codebook for the vector quantization unit 34 in the subsequent stage.

  As a specific means for realizing clustering, k-means, which is a known technique, can be used. Clustering by k-means is performed by the following procedures (1) to (4). In this case, the number of divided clusters k can be arbitrarily set, and the generated image feature amount is k-dimensional.

[k-means procedure]
(1) The data is divided into k clusters, which is an arbitrary number.
(2) Calculate the center of gravity for each cluster.
(3) For all data, find a cluster that minimizes the distance from the center of gravity, and assign each data to the smallest cluster.
(4) Terminate if there is no change from the previous cluster. If there is a change, return to (2).

  The codebook storage unit (codebook storage unit) 33 stores the centroid (representative vector) calculated by the clustering processing unit 32 as a codebook for vector quantization.

  The vector quantization unit 34 measures the appearance frequency of each representative vector and outputs it as an image feature vector simultaneously with quantizing the local feature into the nearest representative vector.

  Specifically, first, the local feature amount newly measured by the local feature amount extraction unit 31 based on the codebook stored in the codebook storage unit 33 is calculated as shown in FIG. Quantize to (representative vector).

  For example, when an image such as one in FIG. 13 is newly input, a local feature amount is extracted by the local feature amount extraction unit 31 to calculate a local feature amount, and the local feature amount is calculated by the vector quantization unit 34. In FIG. 14, the quantization is performed as shown in FIG.

  At this time, by using local feature values extracted from various collected images, the distribution of patterns of local feature values that can appear as natural images is obtained, and representative local feature values are obtained from the centroid of each cluster. It will be extracted as (representative vector).

  Next, the frequency distribution of the local feature quantity quantized to each centroid is measured, and a k-dimensional histogram is output as an image feature vector as shown in FIG.

  In this way, the image feature vector obtained by the vector quantization unit 34 is allowed to be a slight variation in the image feature vector due to the influence of noise, is resistant to the influence of noise, and is clearly defined by the type of image. It is possible to convert the difference into an image feature vector in which the difference is clearly identified.

  The image feature vector storage unit 35 stores image feature vectors relating to various types of objects generated in advance by the vector quantization unit 34 in association with the names of the objects.

  The machine learning unit 36 extracts, by machine learning, differences that are greatly different from the image feature amounts of both the object of interest and the other objects, and a criterion for identifying the object of interest. Is generated.

  In the present invention, it is assumed that SVM (Support Vector Machine), which is a known technique described in Non-Patent Document 4, is used as a specific means for realizing machine learning. SVM is a method for determining which of two given classes a given data belongs to, and was proposed in 1995 by Corinna Cortes and V. Vapnik of AT & T. In SVM, criteria are extracted in the form of a discriminator by learning from sample data collected in advance.

  [Non-Patent Document 4] Corinna Cortes, Vladimir Vapnik, "Support-Vector Networks", Machine Learning, 20, pp.273-297 (1995)

  An explanation will be given by taking “face” as an example. In order to classify whether a given image is a “face image” or not, a large number of images other than the face image as well as the face image are collected and used as sample images for learning. In this case, the face image is a positive example sample, and the image other than the face is a negative example sample. This image gives prior knowledge such as “is a face” or “not a face”, and generates a criterion for classification based on this prior knowledge.

  Actually, each sample image is converted into an image feature vector, and the boundary (hyperplane) that classifies the positive example sample (face) and negative example sample (non-face) from the distribution state in the feature space is obtained and classified. As a criterion for This criterion is output as a discriminator. The boundary (hyperplane), which is a criterion for classification, is used when classifying a newly given image, and the classifier 38 described later uses a classifier corresponding to “face” to give a new classifier. Whether the image is a face image is determined depending on which side of the boundary (hyperplane) the image feature vector extracted from the extracted image is included.

  Image feature vectors are read from the image feature vector storage unit 35 in which image feature vectors of various types of objects are stored in advance, and one classifier is created for each object. For example, when creating a discriminator for “automobile”, an image feature vector labeled “automobile” is read as a positive sample, and an image feature vector labeled other than that is read as a negative sample. Read as. Since the image feature vectors that can be the target of the negative sample are enormous, the classifier is created by applying machine learning by SVM after thinning out to the same number as the positive sample.

  The discriminator accumulating unit 37 accumulates discriminators relating to various types of objects generated in advance by the machine learning unit 36 in association with names of objects (or targets) identified by the discriminator.

  The discriminator 38 reads the corresponding discriminator from the discriminator accumulator 37 for candidates of N types of object names (or what the target is) estimated by the image recognition unit 12, and newly input images Is one of N types of objects.

  Here, a boundary (hyperplane) serving as a determination criterion for identifying the corresponding object (target) is given to the classifier of each read object (each target). Whether or not the object is a corresponding object is determined depending on which side of the boundary (hyperplane) the image feature vector extracted from the newly input image is included. In SVM, as a scale indicating whether the new image feature vector is on the positive example side or the negative example side, the reliability according to the distance from the boundary (hyperplane) is output. The reliability of 0 is considered to be on the boundary, and the reliability is positive if it is located on the positive example side, and the reliability is negative if it is located on the negative example side.

  Therefore, an object (target) showing the maximum reliability among the reliability output from each of the N types of discriminators is specified for a newly input image. If the reliability exceeds a specified threshold value, it is determined that the input image is the object, and if the reliability falls below the threshold value, it is determined that there is no corresponding to N types of objects.

  FIG. 17 shows functional blocks according to another embodiment (referred to as embodiment d) of the identification unit 38 for the above-described one embodiment (referred to as embodiment c). The identification unit 38 includes a co-occurrence relationship measurement unit 381, a prior probability distribution accumulation unit 382, and an overall reliability calculation unit 383. Each unit performs an additional process on the reliability of the N objects of each target image obtained by the identification unit 38 according to Example c to obtain the overall reliability. The identification unit 38 obtains a final identification result for the input image (the input image to the partial region extraction unit 11 in FIG. 1) based on the total reliability obtained by each unit.

  In Example d, rather than determining independently from the reliability of the rectangle candidates of each object region as in Example c, more accurate is considered by considering the co-occurrence between rectangle candidates of a plurality of object regions. The objects included in the image input to are simultaneously determined.

Co-occurrence means, for example, that “car” and “person” appear simultaneously in the example of FIG. The co-occurrence relationship measuring unit 381 measures the frequency at which objects appear simultaneously from a number of sample images in advance as the co-occurrence frequency. Here, C is the total number of types of objects handled by the device of the present invention. It is within the image type of object c (c = 1, 2, ..., C) is to _{f c = 0 f c = 1} , to be included if included. Also, let f = (f ₁ , f ₂ ,..., F _C ) be a vector in which the appearance states in the image are arranged. If the appearance state vector in the sample image i is f _i and the total number of sample images is NS, the co-occurrence relation measuring unit 381 measures the following (formula 4 and formula 5) in advance.

  Here, ν means an average of appearance state vectors, and Σ means a covariance matrix of appearance state vectors. At this time, the prior probability p (f) calculated from the appearance state vector f of the new input image can be calculated from the following equation (Equation 6). The calculation is performed based on the co-occurrence frequency measured by the co-occurrence relation measuring unit 381, and the prior probability distribution accumulating unit 382 accumulates the calculated prior probabilities.

  Note that the total number C of object types handled by the apparatus of the present invention is the total number of label types registered in advance in the data set storage unit 83 in the image recognition unit 12 in FIG. Also, the kind of object c (c = 1, 2,..., C) in the image is the object c (or more generally the target c) specified by each registered label c.

Alternatively, as another example of the co-occurrence relationship, the co-occurrence relationship may be described by an area occupancy ratio that the object c occupies as a rectangular region in the sample image. That is, the existence ratio r _c (Σr _c = 1) corresponding to the area of the rectangular area occupied by the kind of object c (c = 1, 2,..., C) in the sample image is measured, and the existence ratios are arranged. The vector r = (r ₁ , r ₂ ,..., R _C ), and using the vector r in which the existence ratios are arranged instead of the appearance state vector f, corresponding to (Expression 4) to (Expression 6) The prior probability p (r) can be obtained by the following procedures (Equation 7) to (Equation 9).

  The total reliability calculation unit 383 indicates the total reliability for the combination of each candidate region (each target image) and each label with the prior probability p (f) or p (r) stored in the prior probability distribution storage unit 382. ) And the candidate area of each object are multiplied by the reliability of each candidate object, and the combination with the maximum total reliability is the final identification result.

  For example, in the example of FIG. 1, label examples for three candidate regions (target images) “car” (reliability 0.9 obtained independently according to Example c), “person” (independent reliability 0.7), and “not applicable” The total reliability of (independent reliability of 1.0) is 0.9 × 0.7 (× 1.0) × 0.95 = 0.5985, assuming that the prior probability is 0.95 because it is a co-occurrence relationship in nature. On the other hand, the overall reliability of the label example “automobile” (independent reliability 0.9), “not applicable” (independent reliability 1.0) and “Kirin” (independent reliability 0.2) co-occurs in nature. Assuming that the prior probability was 0.1 because of the difficult relationship, 0.9 (× 1.0) × 0.2 × 0.1 = 0.018, which is a very small value.

  Further, when using the co-occurrence relationship based on the area occupancy rate, the total reliability considering whether the size relationship between the objects is appropriate is calculated in the same manner.

  It should be noted that the combinations for which the total reliability is calculated by the total reliability calculation unit 383 are all combinations of each candidate area and each label obtained by adding “not applicable”. In the case of “not applicable”, the reliability is set to a standardized maximum value 1.0 as shown in the above calculation example. For example, when the three candidate areas (target image) and the label (3) shown in (2) of FIG. 1 are given, “automobile” to “bike” and “not applicable” for the first candidate area ”5 + 1 = 6 label,“ person ”to“ electric pole ”and“ not applicable ”4 + 1 = 5 label for the second candidate area, and“ giraffe ”to“ The total reliability is obtained for all combinations 6 × 5 × 6 = 180 with “5 months” and “not applicable” 5 + 1 = 6 labels, and the combination with the maximum value is the final recognition result.

  In addition, since “not applicable” is meaningless for all candidate areas, it may be excluded. For example, in the above example, 180-1 = 179 patterns may be evaluated. Alternatively, if the sample is appropriately set, the prior probability that all candidate areas are “not applicable” is 0, and therefore, the total reliability may be evaluated as 0.

  In addition, the given plurality of sample images read by the co-occurrence relation measuring unit 381 each include a plurality of targets in advance, and are provided with a label for specifying each of the plurality of targets. And Further, in the case of the embodiment in which the co-occurrence relationship is described by the ratio of the occupied area, in addition to the label, a rectangular area similar to that extracted by the partial area extracting unit 11 of the present invention for each of a plurality of objects Information is given.

  As described above, in Example d, instead of determining independently from the reliability of the rectangle candidates of each object region as in Example c, consideration was given to co-occurrence between rectangle candidates of a plurality of object regions. By calculating the total reliability and comprehensively determining from the calculated total reliability, it becomes possible to simultaneously determine objects included in the input image with higher accuracy.

[Registered images]
The description of the main part of the present invention is as described above. Finally, the feature vector stored in the data set storage unit 83 in the image recognition unit 12 (see FIG. 1) and the image feature vector storage in the redetermination recognition unit 13 With regard to the image feature vectors stored in the unit 35, notes for increasing the recognition accuracy of the present invention will be described. The registration flow for the accumulation is as shown by the dotted arrows in FIGS.

  That is, the feature vector stored in the data set storage unit 83 (see FIG. 11) and the image feature vector stored in the image feature vector storage unit 35 are generated not by the original image itself but by the partial region extraction unit 11 (see FIG. 1). ) Is used. This is because when the recognition process is executed, the partial area image extracted by the partial area extraction unit 11 is processed by the image recognition unit 12 and the re-determination recognition unit 13. The purpose is to improve the estimation accuracy and identification accuracy of the names of the objects included in the image by eliminating the gap and taking consistency between the feature vectors to be compared.

  Alternatively, as an effect similar to that performed through the partial area extraction unit 11, the original image may be limited to an image including only a single object in advance, for example, only a car, not both a car and a person. In addition, the sample image read by the co-occurrence relation measuring unit 381 is not limited to a single object, and needs to include a plurality of objects, contrary to the registered image.

[About the third embodiment of the image recognition apparatus 1]
FIG. 18 shows a functional block diagram of the image recognition apparatus 1 according to the third embodiment. The image recognition unit 12 is excluded as a difference from the configuration of the first or second embodiment shown in FIG. 1, and the image recognition device 1 includes only the partial region extraction unit 11 and the redetermination recognition unit 13. Also, in FIG. 18, unlike FIG. 1, the arrows between the functional blocks show only the flow at the time of recognition, and the flow at the time of registration and CB (code book) creation is omitted.

  In the third embodiment, as shown in (1), a predetermined specific list is prepared in the candidate list that should be given by the image recognition unit 12 to the target image as each candidate area. Then, processing in the redetermination recognition unit 13 is performed. That is, the redetermination recognition unit 13 performs processing using the same candidate list for all candidate regions. By preparing in advance a candidate list in which only typical ones that are supposed to appear in the input image are listed and moderately suppressing the number of candidates, the amount of calculation required for the search is within a realistic range. In addition, it is possible to provide the image recognition device 1 while ensuring a predetermined accuracy.

[Application of the present invention]
The image recognition apparatus 1 of the present invention can automatically recognize the names and locations of a plurality of objects included in an image and convert them into text information. As a result, the digital camera image uploaded by the user on the image cloud server can be tagged and applied to image search using keywords.

  DESCRIPTION OF SYMBOLS 1 ... Image recognition apparatus, 11 ... Partial area extraction part, 12 ... Image recognition part, 13 ... Redetermination recognition part, 21 ... Candidate selection part, 22 ... Rectangle extraction part, 23 ... Candidate selection part, 82 ... Image feature-value conversion , 83 ... Data set storage unit, 84 ... Comparison unit, 85 ... Vote determination unit, 31 ... Local feature amount extraction unit, 32 ... Clustering processing unit, 33 ... Codebook storage unit, 34 ... Vector quantization unit, 35 ... Image feature vector storage unit 36 ... Machine learning unit 37 ... Discriminator storage unit 38 ... Discrimination unit

Claims (11)

A partial area extraction unit that cuts out a plurality of partial areas from the input image and sets each as a target image;
An image recognition unit for recognizing what the target image represents,
The image recognition unit
An image feature amount conversion unit that extracts a first image feature amount from the target image and converts it into an image feature vector;
A data set storage unit that assigns a label to each of a plurality of given images, and stores the images in association with the image feature vectors converted by the image feature amount conversion unit;
The image feature vector of the target image is compared with each of the image feature vectors stored in the data set storage unit, and the similarity between the target image and each of the plurality of stored stored images is obtained. A comparison unit;
Of the labels the similarity is the applied for a predetermined number of images of the upper includes a voting decision unit which occurrence frequency is determined as the recognition result of or represents what the target image a label that meets the predetermined criteria, the ,
Recognizing what each of the plurality of targets included in the input image is based on the recognition result by the image recognition unit in each of the partial regions cut out by the partial region extraction unit, and for each of the plurality of targets Recognizing the position in the input image ,
Based on a comparison between a second image feature amount previously associated with the candidate and a second image feature amount extracted from the target image, with each of the labels satisfying the predetermined criterion in the voting determination unit as a candidate And a re-determination recognition unit that recognizes what the target image represents,
Recognizing what each of the plurality of objects included in the input image is, the recognition result of the re-determination recognition unit is adopted instead of the recognition result of the image recognition unit,
The image feature amount conversion unit converts the pixel value from the target image into an image feature vector having each element as a first image feature amount,
The redetermination recognition unit includes a local feature amount extraction unit that extracts a local feature amount as the second image feature amount from the target image;
A clustering processing unit that extracts local feature amounts from a plurality of given images and performs clustering to calculate a representative vector of each cluster;
A codebook storage unit for storing the calculated representative vector as a codebook;
A vector quantization unit that quantizes the local feature amount extracted from the target image into a representative vector with reference to the accumulated codebook, and measures an appearance frequency of each representative vector to obtain an image feature vector; ,
An image feature vector storage unit that stores a plurality of given images each assigned a label and an image feature vector obtained by applying the vector quantization unit to each image in association with each other;
Applying machine learning using the labeled image feature vector stored in the image feature vector storage unit as learning data, and outputting a discriminator for each label;
A discriminator accumulating unit for accumulating a discriminator for each of the labels;
The discriminator corresponding to each of the labels obtained as the recognition results in the voting judgment unit is read from the discriminator storage unit, and the image feature vector obtained by the vector quantization unit is input to the target image. Thus, a label satisfying a predetermined standard is obtained, and the obtained label is used as a recognition result indicating what the target image represents. If a label satisfying the predetermined standard cannot be obtained, what the target image represents An image recognition apparatus comprising: an identification unit that determines that the recognition result is not applicable .   The partial area extraction unit specifies a plurality of rectangular areas having sides parallel to the vertical or horizontal axis of the input image as the partial areas according to the relative position and size of the input image in the input image. 2. The image recognition according to claim 1, further comprising: a candidate selection unit that is selected and a rectangular extraction unit that cuts out the plurality of selected rectangular regions from the input image and forms the partial regions. apparatus.   The candidate selection unit includes a random number generation unit that specifies a predetermined number of relative positions and sizes that the rectangular area occupies in the input image using random numbers, or a target in which a target exists in advance in a predetermined image Statistical information measuring unit for measuring statistical information on a specific position and size, a probability distribution storing unit for storing a prior probability distribution for a predetermined combination of the measured position and size, and 3. The image recognition apparatus according to claim 2, further comprising a candidate extraction unit that specifies a predetermined number of relative positions and sizes.   The partial region extraction unit further includes a candidate selection unit that selects only a part of the plurality of rectangular shapes selected by the candidate selection unit based on an overlapping relationship between the regions, 4. The image recognition apparatus according to claim 2, wherein the rectangular extraction unit cuts out only the selected area. The candidate selection unit
A region dividing unit that divides the input image into regions of arbitrary shapes based on similarity of color and / or texture features;
An object reliability calculation unit that calculates the reliability of whether or not the plurality of rectangular regions selected by the candidate selection unit capture the target using the degree of covering the divided regions without excess or deficiency as an index When,
A final candidate determining unit that selects only the part based on the calculated high reliability and a small overlap between the plurality of rectangular regions selected by the candidate selecting unit. The image recognition device according to claim 4, wherein: The region dividing unit is
A small region dividing unit that excessively divides the input image into a plurality of small regions of arbitrary shapes based on similarity of color and / or texture characteristics;
A target area selecting unit that outputs an internal area belonging to the small area and an external area not belonging to the small area in the input image in association with each of the plurality of subdivided small areas;
A small area expansion unit that integrates an area similar in color and / or texture characteristics with the corresponding internal area in the external area and outputs the area as a small area expansion area;
An area similarity calculation unit for calculating the area similarity of a pair of the small area expansion areas;
Among the small region pairs, the small region pairs in which the region similarity of the corresponding small region extended region pair satisfies a predetermined criterion are determined to belong to the same object and are integrated, and the small region does not satisfy the predetermined criterion And a small region integration unit that obtains a result of division into the region of the arbitrary shape as a small region integrated as a region corresponding to each object by determining that the pairs belong to different objects. 6. The image recognition device according to claim 5. The partial region extraction unit,
A saliency extraction unit that extracts a saliency corresponding to the degree of attention for each pixel from the input image;
A predetermined number of areas that have a rectangular shape consisting of sides parallel to the vertical or horizontal axis of the input image and the saliency satisfies a predetermined standard are determined as the partial areas, and the area occupies in the input image A rectangle determining unit that is selected and specified by a relative position and size;
2. The image recognition apparatus according to claim 1, further comprising: a rectangular extraction unit that cuts out the plurality of selected rectangular areas from the input image and forms the partial areas. The identification unit further includes:
A co-occurrence relation measuring unit that measures a co-occurrence relation between labels from a plurality of given images each configured as an image including a plurality of objects and provided with a label corresponding to each object;
A prior probability distribution accumulating unit that accumulates prior probabilities related to the co-occurrence of labels based on the measured co-occurrence relationship;
Total confidence obtained by multiplying the combination of each of the target images and the score output by the discriminator for each of the labels obtained as a recognition result in the voting determination unit in each of the target images by the prior probability. An overall reliability calculation unit for calculating the degree,
Claims 1, wherein the overall reliability and recognition result if each of the plurality of target included labels to the input image is what for each of the target image corresponding to a combination of maximum 7 The image recognition apparatus in any one of. The co-occurrence relationship measuring unit is configured as an image including a plurality of targets, and a label corresponding to each target and a given plurality of images to which information on a rectangular area occupied by each target in the image is given. The image recognition apparatus according to claim 8 , wherein a co-occurrence relationship between each other is measured as a co-occurrence relationship of area occupancy ratios of rectangular regions corresponding to the labels. At least one of the given plurality of images in the data set storage unit and the given plurality of images in the image feature vector storage unit is configured as an image including a single target. The image recognition apparatus according to claim 1 . A partial region extraction step of cutting out a plurality of partial regions from the input image and making each a target image;
An image recognition step for recognizing what the target image represents,
The image recognition step includes:
An image feature amount converting step of extracting a first image feature amount from the target image and converting it into an image feature vector;
A data set accumulation step of assigning a label to each of a plurality of given images, and accumulating the images in association with the image feature vectors converted in the image feature amount conversion step ;
The image feature vector of the target image is compared with each of the image feature vectors stored in the data set storage step, and the similarity between the target image and each of the stored plurality of images is determined. The desired comparison step,
Includes among label the similarity is the applied for a predetermined number of images of the upper, and voting decision step of frequency is determined as one of the recognition result represents what the target image a label that meets the predetermined criteria, the ,
Recognizing what each of the plurality of targets included in the input image is based on the recognition result of the image recognition step in each of the partial regions cut out by the partial region extraction step, and for each of the plurality of targets Recognizing the position in the input image ,
Based on a comparison between a second image feature amount previously associated with the candidate and a second image feature amount extracted from the target image, with each of the labels satisfying the predetermined criterion in the voting determination step as a candidate Further comprising a redetermination recognition step for recognizing what the target image represents,
In the recognition result of what each of the plurality of objects included in the input image is, the recognition result of the redetermination recognition step is adopted instead of the recognition result of the image recognition step,
In the image feature amount conversion step, the target image is converted into an image feature vector having a pixel value as each element as the first image feature amount,
The re-determination recognition step includes a local feature amount extraction step of extracting a local feature amount as the second image feature amount from the target image;
A clustering process step of extracting local feature amounts from a plurality of given images and performing clustering to calculate a representative vector of each cluster;
A codebook storage step of storing the calculated representative vector as a codebook;
A vector quantization step of quantizing a local feature extracted from the target image into a representative vector with reference to the accumulated codebook, and determining an image feature vector by measuring an appearance frequency of each representative vector; ,
An image feature vector storing step of storing a plurality of given images each assigned a label and an image feature vector obtained by applying the vector quantization step to each of the images in association with each other;
Applying machine learning using the image feature vector with the label stored in the image feature vector storing step as learning data, and outputting a discriminator for each label; and
A discriminator accumulating step for accumulating a discriminator for each label;
The classifier corresponding to each of the labels obtained as the recognition result in the voting determination step is read from the result of the classifier accumulation step, and the image feature vector obtained by the vector quantization step is obtained for the target image. By inputting, a label satisfying a predetermined standard is obtained, and the obtained label is used as a recognition result indicating what the target image represents. If a label satisfying the predetermined standard is not obtained, what is the target image An image recognition method comprising: an identification step that makes a recognition result of whether or not to represent no match .