JP4967045B2 - Background discriminating apparatus, method and program - Google Patents

Description

  The present invention relates to image processing using a computer.

  2. Description of the Related Art In recent years, a so-called similar image search service has been proposed in which an image similar to an image designated by a user is searched with a computer and the result is displayed on an Internet website or the like. Similar image search uses information (referred to as “features”) representing features such as colors and shapes contained in images, but focused on objects such as objects (also referred to as “objects”) in the images. When a similar image search is performed, there is a problem that the feature amount of the background area decreases the search accuracy.

  For this reason, a technique has been proposed in which the influence of the background is suppressed and the search accuracy is improved by determining and removing the background region from the image. In this type of technology, the feature amounts for each part in the image are clustered to extract ranges having similar feature amounts as independent regions. In addition, as an example of a method for determining whether or not a part or region is a background based on the feature amount, in Patent Document 1, a part of a specific color that becomes a background such as sky blue or green grass is determined as a background. (Paragraph 0067), an example is shown in which for each region into which an image is divided, the background is determined based on feature quantities such as lines and shading patterns (paragraph 0068).

JP-A-9-138470

  However, when clustering is used as described above, depending on the type of object or background in the image and how it is captured, multiple independent regions are extracted, and it is often difficult to recognize the main object from them. there were.

  For example, in Patent Document 1, the background is determined based on the feature amount such as the color shape of a part or region, but the color or shape of the object is not necessarily different from the background. On the other hand, other objects and backgrounds located around the object are not necessarily reflected in a specific color or texture. Furthermore, although it is some kind of object (object) other than the sky or plants, it is often only positioned as a background in relation to the main object intended by the image.

  As described above, in the case of an image in which a plurality of areas such as objects are extracted by clustering, it is difficult to appropriately perform processing such as extraction by appropriately distinguishing the main object and the background.

  In view of the above problems, an object of the present invention is to accurately discriminate between a main object and a background.

Based on the above object, one aspect (1) of the present invention is a background discriminating apparatus that discriminates a background in an image, and extracts feature amounts for each block corresponding to all pixels of the original image to be discriminated. A feature amount extracting means, a dividing means for dividing the image into a plurality of regions by clustering the blocks based on the extracted feature values of the block, and a probability of being a background for each block of the image is set And an average value of the probability set for each block at a position corresponding to the divided area, and whether each area is a background based on the average value. For each of the divided regions, the representative feature amount of the region is calculated based on the feature amount of each block included in the region, and corresponds to all the pixels of the extracted original image For each block Determining whether the feature amount is most similar to the representative feature quantity of any region, most similar regions on the basis of whether it is determined that the background, and the pixel determination means for that pixel to determine the background, the It is characterized by providing.

Another aspect ( 4 ) of the present invention is a background determination method in which the above aspect is captured by a method category, in which the computer determines the background in the image, and the computer determines the original image to be determined. A feature amount extracting step for extracting feature amounts for each block corresponding to all pixels, and a computer dividing the image into a plurality of regions by clustering the blocks based on the extracted feature amounts of the blocks A setting step in which the computer sets a probability of being a background for each block of the image, and the computer sets an average value of the probabilities set for each of the blocks at positions corresponding to the divided areas calculated for each of the areas, a determining step of each region is determined whether or not the background on the basis of this average value, before the computer was divided For each region, a representative feature amount of the region is calculated based on the feature amount of each block included in the region, and the feature amount for each block corresponding to all the pixels of the extracted original image And a pixel determining step of determining whether the pixel is the background based on whether the most similar region is determined as the background .

Another aspect ( 5 ) of the present invention is a background determination program for capturing the above aspect in the category of a computer program, and for causing a computer to determine the background in an image. The feature amount is extracted for each block corresponding to all the pixels of the image, and the computer is clustered based on the extracted feature amount of the block, thereby dividing the image into a plurality of regions, and the computer. The probability of being a background is set for each block of the image, and the computer is made to calculate the average value of the probability set for each block at the position corresponding to the divided area for each area. each region based on the values allowed is judged whether or not the background, in a computer, divided the each region included in the region Based on the feature value of each block, the representative feature value of the area is calculated, and the feature value of each block corresponding to all the pixels of the extracted original image is most similar to the representative feature value of which area. And determining whether the pixel is the background based on whether the most similar region is determined to be the background .

In this way, the probability that the background is higher as the distance from the center of the image is higher, and the probability that the background is set for each part of the image is set. The average value of probability is calculated. Since this average value represents the degree to which each area in the image is the background, the main object and the background can be identified regardless of the color or shape of the object or background by determining whether it is the background based on this average value. It can be determined with high accuracy. In addition, although feature amounts are extracted from all pixels, the feature amounts to be clustered are thinned out or the processing load is reduced as a block unit of a plurality of pixels. On the other hand, a representative feature amount is calculated for each region of the clustering result, and whether each pixel is a background is determined based on the similarity between the representative feature amount and the feature amount for each pixel. This makes it possible to achieve both processing efficiency by suppressing the clustering load and discrimination accuracy including pixels near the region boundary.

  According to another aspect (2) of the present invention, in any one of the above aspects, there is provided learning data storage means for storing an image for each genre as learning data, and the setting means is provided for each block of the image. A background probability distribution, which is data associated with a probability that is a background, is generated in advance for each genre using the learning data stored in the learning data storage means and stored in a predetermined probability distribution storage means. The probability of being a background is set for each block of the original image based on the background probability distribution stored in the probability distribution storage means.

  Thus, the discrimination accuracy can be further improved by generating a background probability distribution for each image genre in advance based on the learning data and using it for the discrimination of the background.

  According to another aspect (3) of the present invention, in any one of the above aspects, the setting unit represents the distribution of the probability as the background for each genre using a coefficient related to a two-dimensional normal distribution. And

  In this way, the background probability distribution is expressed using a two-dimensional normal distribution (Gaussian distribution), and by setting coefficients such as an average value and deviation for each genre, the background probability distribution for each genre can be easily and little data. It becomes possible to memorize in quantity.

  It should be noted that a category (method for the apparatus, program for the method, etc.) different from each of the above-described modes and more specific modes described below are also included in the present invention. For different categories, “means” shall be appropriately read as “step”.

  According to the present invention, the main object and the background can be distinguished with high accuracy.

The functional block diagram which shows the structure of embodiment of this invention. The figure which illustrates the information (data) used by embodiment of this invention. The flowchart which shows the process sequence in embodiment of this invention. The conceptual diagram which shows the background probability distribution in embodiment of this invention. The conceptual diagram showing an example of the feature-value extraction in embodiment of this invention. The conceptual diagram showing an example of the feature-value extraction in embodiment of this invention. The figure which shows an example of the original image in embodiment of this invention. The conceptual diagram which shows the area | region in embodiment of this invention. The conceptual diagram which shows the relationship between the area | region and background probability distribution in embodiment of this invention. The conceptual diagram which shows the average value of the background probability for every area | region in embodiment of this invention with whitishness. The figure which shows the image which removed the background in embodiment of this invention.

  Next, modes for carrying out the present invention (referred to as “embodiments”) will be described with reference to the drawings. It should be noted that assumptions common to those already described in the background art and problems are omitted as appropriate.

[1. Constitution〕
As shown in FIG. 1 (configuration diagram), the present embodiment relates to a background discriminating apparatus 1 (hereinafter referred to as “this apparatus 1” or “this apparatus”) that discriminates a background in an image. The device 1 also serves as a server having a function of providing a similar image search service to the terminal T via the communication network N, but may be a device that does not serve as such a function and only determines the background.

  This apparatus 1 has at least an arithmetic control unit 6 such as a CPU, a storage device 7 such as an external storage device (HDD or the like) and a main memory, and a communication network N (Internet, mobile phone network, Communication means 8 (LAN adapter etc.). The terminal T is an information processing device such as a personal computer (PC), a smartphone, or a mobile phone terminal device used by the user, and the number thereof is not limited.

  In the apparatus 1, a predetermined computer program (not shown) stored (installed) in advance in the storage device 7 controls the arithmetic control unit 6, so that elements (11, 12, 13, etc.) shown in FIG. Etc.). Among these elements, the information storage means can be realized in the storage device 7 in any format such as various databases (also referred to as “DB”), variables such as files and arrays, various stacks and registers, and system setting values.

  Among such storage means, the original image storage means 11 is an original image that is an image whose background is to be discriminated (for example, a large number of images that accumulate information such as feature amounts as candidates for similar image search). Is a means for storing. Further, the learning data storage means 12 is a means for storing an image for each genre as learning data used for setting the probability of being the background for each block in the image. Each means other than the storage means is a processing means for realizing and executing the following information processing functions and operations.

[2. Action)
The processing procedure for determining the background in the apparatus 1 as described above is shown in the flowchart of FIG.
[2-1. (Generation of background probability distribution)
Here, the setting means 13 sets the probability of being the background (hereinafter referred to as “background probability”) for each block of the image. The setting of the background probability in the present embodiment is the background discrimination (FIG. 3). ) Prior to the determination of the background probability distribution in the entire image (referred to as “background probability distribution”) for each image genre (eg, person, product, landscape, etc.) And applying a background probability distribution to set a background probability for each block of each original image. Here, an example of the background probability distribution is shown in the conceptual diagram of FIG. In this example, as indicated by the dashed concentric circles, the darker portion has a higher probability of being the main object and the background probability is lower, and conversely the surrounding whitish color portion has a higher background probability.

  In order to obtain such a background probability distribution, it may be obtained by some appropriate calculation formula or manual setting for each coordinate in the image on the assumption that the edge of the image is higher in the background, In the present embodiment, for each image genre, a large number of sample images obtained by manually determining the background and extracting only the background portion are prepared as learning data in the learning data storage unit 12 and then statistically processed. To calculate the background probability of each pixel.

  That is, the setting unit 13 generates a background probability distribution, which is data in which the probability of the background is associated with each block of the image, in advance for each genre using the learning data stored in the learning data storage unit 12. And stored in the predetermined probability distribution storage means 14. As statistical processing for obtaining the background probability distribution, for each pixel, block, coordinate, and the like on the image, the ratio of the background in the genre learning data may be aggregated.

[2-2. Example of background probability distribution)
As described above, the expression format of the background probability distribution generated and stored by the setting means 13 is arbitrary. As an example, the background probability distribution for each genre is expressed using a coefficient related to the two-dimensional normal distribution. can do. That is, first, the background probability is basically lower at the center of the image and higher at the edge, so it is opposite to the probability of being the main object (referred to as “target probability”). There is a so-called complement relationship that becomes 1.

  Therefore, in terms of the target probability which is a complement, the target probability is higher as the center of the image. Take a dimensional normal distribution. In this case, assuming that the horizontal direction of the image is the x-axis and the vertical direction is the y-axis, the vicinity of the center of the image corresponds to the average value that is the peak of the mountain shape of the two-dimensional normal distribution.

と表すことができる。 For example, x, dispersing sigma _{x 2} of ^y, and sigma _y ^2, the correlation coefficient is sigma _xy, if the exponent of the density function of the two-dimensional normal distribution represented as "-c ^2/2", the density function Represents a similar ellipse, and all points (x, y) on the ellipse are expressed by a density function of a two-dimensional normal distribution.

It can be expressed as.

のように表される。なお、σ_ｘ ^２，σ_ｙ ^２や、指数部に含まれる平均μ_ｘ，μ_ｙは、識別結果より適当な値を決めてもよいし、平均は画像の中心座標としてもよい。 However, when used for the background probability, it can be assumed that there is no correlation between x and y, so if the correlation coefficient is 0, the background probability distribution is

It is expressed as It should be noted that σ _x ² , σ _y ² and the average μ _x , μ _y included in the exponent part may be determined from appropriate values based on the identification result, or the average may be the center coordinates of the image.

  When using the two-dimensional normal distribution as described above and representing the background probability distribution in association with the complement relationship with the target probability, as the coefficient for each genre, for example, the center coordinate that is the center of the probability (mountain) is the x-axis. The background probability distribution pattern according to the genre can be represented and stored by setting it to any position in the image using the average value in the direction or y-axis direction, or by setting the probability concentration by variance, standard deviation, etc. (For example, FIG. 2).

[2-3. Feature extraction)
A processing procedure (FIG. 3) for discriminating and extracting the background on the premise that the background probability distribution is generated and stored as described above will be described. First, the feature quantity extraction unit 15 divides an original image (hereinafter also referred to as “target image” or simply “image”) as a background discrimination target from the original image storage unit 11 into blocks. A feature amount is extracted for each block by calculation or the like (step S11). The feature amount extracted here may be a general one such as a color histogram or a texture feature amount.

  Further, as a block on which the feature amount is extracted, individual pixels may be used as they are. However, as illustrated in FIG. 5, for example, the feature amount C1 is extracted from the block B1 centered on the pixel G1, and the neighboring pixels are extracted. If the feature amount is extracted at each position while shifting the range of the block like a window having a plurality of pixels as a unit, such as extracting the feature amount C2 from the block B2 centering on the pixel G2 The amount of information can be enhanced, and improvement in discrimination accuracy can be expected.

  In order to prevent an excessive load of clustering to be performed thereafter, when extracting feature amounts corresponding to all pixels as in the example of FIG. 5, pixels to be extracted are thinned out, or As illustrated in FIG. 6, the feature amounts C11 and C12 may be extracted in units of blocks B11 and B12 in which a predetermined number of pixels are collected. With these measures, it is possible to reduce the feature amount to be clustered and suppress the processing load, and to improve the processing speed.

[2-4. Clustering based on features)
Subsequently, the dividing unit 16 clusters the blocks based on the feature amounts of the blocks extracted as described above, and generates a region (also referred to as a “partial region”) for each block group belonging to the same cluster. Thus, the image is divided into a plurality of regions (step S12). This clustering of feature quantities is performed in the feature quantity space, and each clustered feature quantity is arranged as a region in the original image. An example is shown below.

  For example, it is assumed that as a result of extracting and clustering feature amounts based on the original image as shown in FIG. 7, the region is divided into regions R1 to R5 as shown in FIG. In this example, the region R2 is a remaining region excluding the regions R1, R3, R4, and R5. Note that, even in the feature amount space, even in the same cluster, there is a possibility that it becomes an independent area on the image depending on the position or group of the corresponding block. In the description of FIG. 7 and subsequent figures, each region is indicated by reference numerals R1 to R5. However, in terms of implementation, identification information such as ID, label, and number for identifying each region and its order are, for example, clustering immediately after clustering. What is in the order may be changed as appropriate, for example, by changing the order in the position in the image.

[2-5. (Calculation of average value of probability and discrimination of background)
Furthermore, the determination means 17 calculates the average value of the probability set for each block at the position corresponding to the divided area for each area. In this process, the determination unit 17 first selects a background probability distribution corresponding to the same genre as the target image from the background probability distributions stored in the probability distribution storage unit 14 (step S13). Based on the background probability distribution, that is, the discriminating means 17 enlarges or reduces the size of the background probability distribution according to the target image (step S14), and applies it to the target image to include each partial region. The background probability corresponding to each block of the image is obtained, and the average value is calculated (step S15).

  For example, the background probability distribution shown in FIG. 4 and the partial areas shown in FIG. 8 are overlapped with the same size as shown in FIG. If the average value is obtained, for example, as illustrated in FIG. 10, the average value of the background probabilities for each partial region is obtained. In FIG. 10, it is assumed that the whiter the background probability is higher.

  Then, the discriminating means 17 discriminates whether or not each area is the background based on the average value of the background probabilities calculated for each partial area as described above, and extracts the background area (step S16). If the background region thus determined and extracted is removed from the original image, a similar image search image that accurately represents the features of the object can be obtained.

  As a criterion for determining whether or not the region is the background, for example, a threshold value is determined in advance, and the average value of the background probability of the region exceeds the threshold value (in the example of FIG. 10, when it is close to white) It can be considered that the area is determined as the background. In addition, for example, according to an individual specific image, when the gap of the average value of the background probability between the regions or the ratio of the difference is more than a predetermined value, it can be used as a delimiter as to whether it is the background, The threshold value of the average value may be obtained in accordance with individual original images so that a predetermined proportion of the area becomes the background.

  For example, when the regions R1, R2, and R3 other than the regions R4 and R5 shown in FIG. 10 are determined as the background based on the original image in FIG. The image portion is removed, and an image as shown in FIG. 11 in which only two buildings corresponding to the regions R4 and R5 remain is taken as the processing result. The image obtained as a result of such processing is stored in the result storage unit 18 by the determination unit 17, for example, and a similar image search unit 19 such as a search engine that has received a similar image search request from the terminal T performs comparison between the images. Used as a target for search processing.

[3. effect〕
In the present embodiment, as described above, the probability of being a background is set for each part of the image, such as the possibility of the background being higher as the distance from the center of the image increases (for example, FIG. 4 and FIG. 9). By calculating the average value of the probabilities distributed in the region and determining whether it is the background based on the average (for example, FIG. 8), the color or shape of the object or background is obtained. Regardless of the above, the main object and the background can be accurately distinguished. Thereby, the accuracy of the similar image search can be improved by accurately expressing the feature of the object in the image.

  In particular, in this embodiment, the discrimination accuracy can be further improved by generating a background probability distribution for each image genre in advance based on the learning data and using it for discrimination of the background.

  In the present embodiment, the background probability distribution is expressed using a two-dimensional normal distribution (Gaussian distribution), and a coefficient such as an average value or a deviation is set for each genre (for example, FIG. 2), so that the background for each genre is set. The probability distribution can be easily stored with a small amount of data.

[4. Second Embodiment]
An example in which both the accuracy and efficiency of the process are achieved by further improving the background discrimination method with respect to the above embodiment (hereinafter referred to as “first embodiment”) will be described as a second embodiment. That is, when the feature amount is reduced by thinning out the target pixel from which the feature amount is extracted in the example of FIG. 5 or by extracting the feature amount in block units as illustrated in FIG. If a process for determining whether or not the background is present is further added, it is possible to improve the determination accuracy particularly in the vicinity of the boundary between regions while maintaining the processing efficiency.

  In this case, the feature quantity extraction unit 15 extracts the feature quantity for each block corresponding to all pixels from the original image that is the background discrimination target, while the pixel judgment unit 20 performs the process for each divided area. Based on the feature amount of each block included in the region, the representative feature amount of the region is calculated. The representative feature amount of each region can be obtained, for example, by averaging feature amount vectors.

  Then, the pixel determining unit 20 determines which feature amount for each block extracted from the original image is most similar to the representative feature amount in which region, and based on whether the most similar region is determined as the background. It is determined whether the pixel is the background. That is, if the most similar region is determined to be the background region, the pixel determining unit 20 determines that the pixel is the background, and otherwise determines that it is not the background. As a result, both processing accuracy and efficiency can be achieved.

  As described above, in the second embodiment, feature quantities are extracted from all pixels, but the feature quantities to be clustered are thinned out or the processing load is reduced in units of blocks of a plurality of pixels. On the other hand, a representative feature amount is calculated for each region of the clustering result, and whether each pixel is a background is determined based on the similarity between the representative feature amount and the feature amount for each pixel. This makes it possible to achieve both processing efficiency by suppressing the clustering load and discrimination accuracy including pixels near the region boundary.

  In addition to the similarity between the representative feature amount of the region and the feature amount of the pixel as described above, as a criterion for which region the pixel is similar to, the average value of the background probability of each region and the original image It is also possible to use the similarity between the feature amount of each pixel. That is, when the feature amount of each pixel of the original image is similar to the average value of the background probabilities of the area, it is determined that the pixel is the background.

[5. Other embodiments]
In addition, said each embodiment is only an illustration, and this invention includes what is illustrated below and other embodiment other than that. For example, regarding the setting of the background probability for each block, it is not essential to prepare a background probability distribution for each genre in advance, and the background probability for each block constituting each original image based on a certain fixed background probability criterion. May be calculated and set.

  In addition, each element such as means may be realized by other information processing mechanisms such as an electronic circuit based on a wired logic or the like without being limited to an arithmetic control unit of a computer. Also, each configuration diagram, data and image, screen and other diagrams, flowchart diagrams, and the like are merely examples, and the presence / absence of each element, its order, specific contents, and the like can be changed as appropriate. For example, the apparatus of the present invention may be realized by using a plurality of apparatuses such as servers, and a configuration in which each storage unit is realized by a separate and independent server apparatus or system is also common. Depending on the function, the configuration can be flexibly changed, for example, by calling an external platform or the like with an API (application program interface) or network computing (so-called cloud or the like).

DESCRIPTION OF SYMBOLS 1 Background discrimination device 6 Operation control part 7 Storage device 8 Communication means 11 Original image storage means 12 Learning data storage means 13 Setting means 14 Probability distribution storage means 15 Feature quantity extraction means 16 Dividing means 17 Discriminating means 18 Result storage means 19 Similar Image search means B1, B2, B11, B12 Blocks C1, C2, C11, C12 Features G, G1, G2 Pixel N Communication network R1-R5 Region T Terminal

Claims (5)

A background discriminating device for discriminating a background in an image,
Feature amount extraction means for extracting feature amounts for each block corresponding to all pixels of the original image to be determined;
Dividing means for dividing the image into a plurality of regions by clustering the blocks based on the extracted feature quantities of the blocks;
Setting means for setting the probability of being a background for each block of the image;
Discrimination that calculates an average value of the probability set for each block at a position corresponding to the divided region for each region, and determines whether each region is a background based on the average value Means,
For each divided region, a representative feature amount of the region is calculated based on the feature amount of each block included in the region, and the feature amount for each block corresponding to all pixels of the extracted original image is Pixel determining means for determining which region is most similar to the representative feature amount, and determining whether the pixel is the background based on whether the most similar region is determined as the background;
A background discrimination apparatus comprising: Having learning data storage means for storing images for each genre as learning data;
The setting means includes
A background probability distribution which is data in which a probability of a background is associated with each block of an image is generated in advance for each genre using the learning data stored in the learning data storage means, and a predetermined probability distribution Memorize it in the memory means,
The background determination apparatus according to claim 1, wherein a probability of being a background is set for each block of the original image based on the background probability distribution stored in the probability distribution storage unit.   The background determination apparatus according to claim 1, wherein the setting unit represents the probability distribution as the background using a coefficient related to a two-dimensional normal distribution. A background determination method in which a computer determines a background in an image,
A feature amount extraction step in which the computer extracts a feature amount for each block corresponding to all pixels of the original image to be determined; and
A dividing step in which the computer divides the image into a plurality of regions by clustering the blocks based on the extracted feature quantities of the blocks;
A setting step in which the computer sets a probability of being a background for each block of the image;
The computer calculates an average value of the probabilities set for each of the blocks at positions corresponding to the divided areas, and determines whether each area is a background based on the average value. A determination step to determine;
For each of the divided regions, the computer calculates a representative feature amount of the region based on the feature amount of each block included in the region, and the block for each block corresponding to all pixels of the extracted original image A pixel determination step of determining whether the feature amount is most similar to the representative feature amount in which region, and determining whether the pixel is the background based on whether the most similar region is determined as the background;
A background discrimination method comprising: A background determination program for causing a computer to determine the background in an image,
Let the computer extract the feature value for each block corresponding to all pixels of the original image to be determined,
By causing the computer to cluster the blocks based on the extracted feature quantities of the blocks, the image is divided into a plurality of regions,
Let the computer set the probability that it is the background for each block of the image,
Let the computer calculate the average value of the probability set for each block at the position corresponding to the divided area, and determine whether each area is a background based on the average value. to discrimination,
For each of the divided regions, the computer calculates a representative feature amount of the region based on the feature amount of each block included in the region, and the block for each block corresponding to all the pixels of the extracted original image A background discrimination characterized by determining which region has the most similar feature to the representative feature and determining whether the pixel is the background based on whether the most similar region is determined to be the background program.

Images