JP7077624B2 - Content classification device, content classification method, and program - Google Patents

Description

The present invention relates to a content classification device, a content classification method, and a program for quantifying the atmosphere and taste perceived by a person for contents such as images and sounds.

In general, the atmosphere that humans perceive for an object to be viewed and the sense of taste (taste) tend to show a high correlation with the characteristics of the object to be viewed and the situation in which a person visually recognizes the object.
For example, a statistic (average value or dispersion value) showing a statistical relationship between the above-mentioned sensation and a visual feature amount (image feature amount in the case of an image, image feature amount in the case of a video) as a characteristic of the viewing object itself. Etc.) are known to have a strong correlation (for example, Non-Patent Document 1).
Further, it is known that the sensation and a statistic with respect to an auditory feature amount (acoustic feature amount for acoustics, audio feature amount for audio) as a characteristic of the object to be viewed have a strong correlation. (For example, Non-Patent Document 2).
Further, as an image feature amount, a method of acquiring a correlation between images by a statistic showing a statistical relationship with a GIST image feature amount, which is one of the image expression methods, is known (for example, Non-Patent Document 3). ).

Also, in daily life, a person may search for another object that suits or is similar to the atmosphere, based on the atmosphere that the person feels about the object. For example, if you are looking for new pants to match your favorite shirt, you may want to get pants that have a similar feel to that shirt.
In such a case, it is difficult to automatically extract images with similar atmosphere and taste by referring to images that combine various shirts and pants using a web page (Web Page) on the Internet. be. The reason is that in order to extract an image similar to a favorite shirt, it is sufficient to determine whether or not the image on the web page has characteristics similar to the image characteristics of the shirt, but the image features are similar. This is because doing so does not always have the same atmosphere and elements that make people feel. In other words, whether or not the atmosphere and taste are the same depends on the subjectivity or sensation of the individual, so it is difficult to objectively quantify it, and it is difficult to replace it with a specific physical quantity or feature quantity. Is. Therefore, the actual situation is that a combination of clothes (coordination) having a desired atmosphere is proposed based on the knowledge and experience of an expert.

Japanese Unexamined Patent Publication No. 2005-250562

JAVIER PORTILLA and EERO P. SIMONCELLI, “A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients.” International Journal of Computer Vision, Vol.40, issue-1, pp.49-71, 2000. Josh H. McDermott and Eero P. Simoncelli, “Sound Texture perception via statistics of the auditory peripheral: Evidence from sound synthesis.” Neuron, Vol.71, issue-5, pp.926-940 2011, September 8. AUDE OLIVA, ANTONIO TORRALBA, “Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope,” International Journal of Computer Vision, Vol.42, issue-3, pp.145-175, 2001.

The present invention has been made in view of such a situation, and an object of the present invention is to quantify the feelings such as atmosphere and taste that human beings have for the content to be viewed, thereby classifying the image group to be classified. It is an object of the present invention to provide a content classification device, a content classification method, and a program capable of classifying whether or not each of the included images has the same atmosphere and taste as the viewing target.

In order to solve the above-mentioned problems, the content classification device according to one aspect of the present invention obtains the same atmosphere and taste as the original content which is the image to be classified from the target content group which is the image to be classified. A content classification device that classifies the target content to be possessed , and the combination of two different regions that can be taken in each region determined when the characteristics, scale, and classification according to the characteristics in the image are specified. The relationship amount indicating the relationship between the two areas is defined as the content atmosphere amount in which the atmosphere and taste perceived by a person for the image are quantified, and the content atmosphere amount is defined for each of the original content and the target content. Based on the distance calculation unit for calculating the distance between the content atmosphere amount calculation unit to be calculated, the coordinate value of the content atmosphere amount of the original content, and the coordinate value of the content atmosphere amount of the target content. It is characterized by including a content classification unit for classifying the target content.

Further, the content classification method according to one aspect of the present invention classifies the target content having the same atmosphere and taste as the original content which is the image used as the classification standard from the target content group which is the image to be classified. Regarding the combination of two different regions that can be taken in each region determined when the content atmosphere amount calculation unit specifies the characteristics, scale, and classification according to the characteristics in the classification method. The relationship amount indicating the relationship between the two areas is defined as the content atmosphere amount in which the atmosphere and taste perceived by a person for the image are quantified, and the content atmosphere amount is defined for each of the original content and the target content. Between the content atmosphere amount calculation process to be calculated and the coordinate value of the content atmosphere amount of the original content calculated by the content atmosphere amount calculation unit and the coordinate value of the content atmosphere amount of the target content by the distance calculation unit. The content classification unit includes a distance calculation step of calculating the distance in the above, and a content classification step of classifying the target content based on the distance.

Further, the program according to one aspect of the present invention is a program for making a computer function as the content classification device.

As described above, according to the present invention, it is possible to determine whether the content to be classified has the same atmosphere and taste as the original content by quantifying and showing the feeling such as the atmosphere and taste that a person perceives. Can be classified.

It is a block diagram which shows the structural example of the content classification apparatus 1 of 1st Embodiment of this invention. It is a figure which shows the configuration example of the content feature amount table in the content feature amount database 16. It is a figure which shows the configuration example of the content statistic table in the content statistic model database 17. It is a flowchart which shows the operation example of the process performed by the content classification apparatus 1 of the 1st Embodiment of this invention. It is a figure explaining the effect of the processing performed by the content classification apparatus 1 of the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the content classification apparatus 1A of the 2nd Embodiment of this invention. It is a flowchart which shows the operation example of the process performed by the content classification apparatus 1A of the 2nd Embodiment of this invention. It is a flowchart which shows the operation example of the process which generates the estimated content feature amount performed by the content feature amount generation unit 110A by the 2nd Embodiment of this invention. It is a flowchart which shows the operation example of the process which generates the estimated content statistic performed by the content statistic generation unit 111A by the 2nd Embodiment of this invention. It is a block diagram which shows the structural example of the content generation apparatus 1B according to the 3rd Embodiment of this invention. It is a flowchart which shows the operation example of the process A1 which generates a deep learning image model. It is a flowchart which shows the operation example of the process A2 which generates a deep learning image model. It is a flowchart which shows the operation example of the process performed by the content generation apparatus 1B according to the 3rd Embodiment of this invention.

Hereinafter, the content classification device, the content classification method, and the program of the embodiment will be described with reference to the drawings.

<First Embodiment>
First, the first embodiment will be described.
FIG. 1 is a block diagram showing a configuration example of the content classification device 1 according to the first embodiment of the present invention. The content classification device 1 in FIG. 1 includes a content selection unit 10, a content atmosphere amount calculation unit 11, a coordinate distance calculation unit 12, a content classification unit 13, a content output unit 14, a content database 15, a content feature amount database 16, and a content statistic. Each of the model database 17 and the classification content storage unit 18 is provided. Further, the content atmosphere amount calculation unit 11 includes a content feature amount calculation unit 110 and a content statistic amount calculation unit 111. Hereinafter, in the present embodiment, the content will be described as a still image, but the configuration may be applied to other contents such as moving images, videos, sounds, and sounds.

The content selection unit 10 inputs an original image having an atmosphere and taste that the user likes as image data from an external device, a content database 15, or an input means (scanner or the like) (not shown). The content selection unit 10 outputs the input image data of the original image to the content atmosphere amount calculation unit 11.
Further, the content selection unit 10 selects, for example, an image group (classification target image group) to be classified from the viewpoint of having the atmosphere and taste of the original image. For example, the content selection unit 10 selects a classification target image group from the content database 15. Alternatively, the content selection unit 10 may select an image group input from an external device, a scanner, or the like (not shown) as a classification target image group. The content selection unit 10 outputs the image data of the selected image group to be classified to the content atmosphere amount calculation unit 11.

The content atmosphere amount calculation unit 11 includes a content feature amount calculation unit 110 and a content statistic calculation unit 111.
The content atmosphere amount calculation unit 11 has physical characteristics (for example, density characteristics, gradation characteristics, color characteristics, spatial frequency characteristics, resolution characteristics, etc.) of each image (classification target image) included in the original image and the classification target image group. Of these, the statistical feature amount (statistics) calculated by statistically calculating the degree of features that tend to be perceived by humans, especially in terms of atmosphere and taste (feature amount). By calculating the relationship between one image and another by comparing them, the content atmosphere amount, which is an index showing how similar the atmospheres are to each other, is calculated.

The content feature amount calculation unit 110 particularly among the physical characteristics (for example, density characteristics, gradation characteristics, color characteristics, spatial frequency characteristics, resolution characteristics, etc.) of the original image and the classification target image included in the classification target image group. Based on the feature amount (perceived feature amount) that tends to be perceived by humans based on the concept of atmosphere and taste, the content feature amount of each of the original image and the image to be classified is calculated.

The content statistic calculation unit 111 is obtained by applying a statistical operation to the perceived characteristics of a predetermined physical characteristic of an image in a predetermined region from the content feature amount calculated by the content feature amount calculation unit 110. Based on the obtained statistic (perceptual statistic), the content statistic of each of the original image and the classification target image is calculated.

The content atmosphere amount calculation unit 11 calculates a content relationship amount that quantitatively indicates the relationship between the content statistic in a certain area and the content statistic in another area from the content statistic. For example, if the distance between the content relational quantity of the content statistic in different regions of one image and the content relational quantity of the content statistic in different regions of another image is close, a predetermined image in one image. The atmosphere of the region and the predetermined region in another image will be similar to each other.

This will be explained in more detail.
The content feature amount calculation unit 110 calculates the content feature amount of the original image and the classification target image. Here, the content feature amount is a feature that tends to be perceived by humans in terms of atmosphere and taste, among physical characteristics (for example, density characteristics, gradation characteristics, color characteristics, spatial frequency characteristics, resolution characteristics, etc.). It is a quantity (perceptual feature quantity). The content feature amount (perceptual feature amount) is, for example, luminance, chromaticity, contrast, gradient, edge, optical flow, etc., which indicate the visual perceptual feature of the image.

For example, the content feature amount calculation unit 110 tends to perceive the contrast or gradient in an image as an atmosphere and taste with respect to the density (shade) characteristic which is a physical characteristic in the image. The intensity is calculated as a content feature amount (perceptual feature amount). Further, since the content feature amount calculation unit 110 tends to easily perceive the brightness and chromaticity of the image with respect to the color characteristic which is the physical characteristic of the image, for example, the brightness and color of the pixel value in the image. The degree is calculated as a content feature amount (perceptual feature amount).

The content feature amount calculation unit 110 extracts the content feature amount (perceived feature amount) from the content feature amount table of the content feature amount database 16. Further, the content feature amount calculation unit 110 reads out the calculation formula used for each calculation of the selected content feature amount from the content feature amount table of the content feature amount database 16. Then, the content feature amount calculation unit 110 calculates the content feature amount from each image of the original image and the classification target image by the read calculation formula.

The content statistic calculation unit 111 calculates a content statistic (perceptual statistic) from the content feature amount calculated by the content feature amount calculation unit 110. The content statistic is a statistic (perceptual statistic) obtained by applying a statistical operation to a perceptual feature in a predetermined region of a predetermined physical characteristic of an image, and is a statistic (perceptual statistic) of a predetermined region in the image. It is a quantity that is an indicator of the nature of perceptual features. The content statistic (perceptual statistic) is, for example, the distribution of each visual perceptual feature amount by direction and resolution, the envelope of the distribution, the histogram shape characteristic, or the mean, variance, histogram, power spectrum, and the like.

The content statistic calculation unit 111, for example, determines the degree of correlation between the shade intensity in each of the two regions with respect to the combination of two different regions that can be taken in each region from the shade intensity in each region where the image is divided. Is calculated as a content statistic (perceptual statistic). The content statistic calculation unit 111 calculates, for example, the degree of correlation between the pixel values (brightness and chromaticity) in each region of the image as the content statistic (perceptual statistic).

The content statistic calculation unit 111 extracts the content statistic (perceptual statistic) from the content statistic table of the content statistic model database 17. Further, the content statistic calculation unit 111 reads out the calculation formula used for each calculation of the extracted content statistic from the content statistic model table of the content statistic model database 17. Then, the content statistic calculation unit 111 calculates the content statistic from the content feature amount obtained by the content feature amount calculation unit 110 by the read calculation formula.

The content atmosphere amount calculation unit 11 is a combination of two regions that can be taken by each of the content statistic of a predetermined region in a predetermined characteristic for an image and the content statistic of a predetermined region at a predetermined scale (resolution level) for the image. For each of all the combinations of, the relationship between the two regions is calculated. Here, the scale is the resolution of the image. Further, increasing or decreasing the resolution of an image with respect to the image is called scaling. For example, when increasing the resolution of an image to 2 (= 2 ¹ ) times the current resolution, scale up to level 1 based on 2 and increase it to 4 (= 2 ² ) times. Is a level 2 scale-up ... On the contrary, when the resolution of the image is reduced to one half of the current resolution, it is called level 1 scale down based on 2, and when it is reduced to one quarter, it is called level 2 scale down. A scale (resolution level) is a representation of the degree of fineness of pixels in an image based on the original resolution of the image.
The content atmosphere amount calculation unit 11 represents a vector having each of the calculated relationships as an element as the content atmosphere amount.
The content atmosphere amount is an index showing the relationship between two regions, and is, for example, a simple correlation, a shift correlation, a cross correlation, or the like.

The coordinate distance calculation unit 12 has a coordinate value (original content coordinate value) of the content atmosphere amount (original content atmosphere amount) of the original image and a content atmosphere amount (target) of the classification target image in the vector space expressed as the content atmosphere amount. Calculate the coordinate distance from the coordinate value (target content coordinate value) of the content atmosphere amount).

Here, for the above distance, for example, a simple distance comparison method (Euclidean distance, cosine distance, Hamming distance, Manhattan distance, etc.) is used. In addition, although it is said that the closer the distance is, the higher the similarity is, the mutual information amount, entropy, mutual correlation, etc. of the pixel values in the area occupying the entire image or a part of the image can be determined between the original image, the initial image, and the image to be classified. It may be configured to be adopted as a measure of similarity with each. That is, a method of comparing the distances of the content feature distributions (entropy, mutual information amount, cross correlation, etc.) may be used.

Here, the mutual information amount indicates the degree of mutual dependence between the two images (the original image and the classification target image), and for example, the pixel value distributions of the original image and the classification target image are completely irrelevant. When it is independent, it becomes "0", and when each pixel value distribution becomes equal, it becomes maximum. Pixel values are used as image features and pixel value distributions are used as content statistics. At this time, if the image feature is selected as an RGB value, that is, a pixel value of each of the color components R, G, and B, it is represented by the three-dimensional feature (r, g, b). The pixel value distribution between the original image and the classification target image represents the distribution of (r, g, b) of all the pixels. That is, if the probability that a pixel value appears in an image is considered, the pixel value corresponds to a random variable, which can be read as an information problem, and can be considered as a probability distribution.

In addition, entropy is shown as a measure of ambiguity of "what kind of value the pixel value in an image actually takes". For example, an image having a large entropy has a large variation in pixel values in this image, while an image having a small entropy has a small variation in pixel values in this image (a monochromatic image becomes smaller). Assuming that the entropy is H, the degree of dependence between the original image (X) and the classification target image (Y) in the mutual information amount is based on knowing the entropy H (X) of the image X and the ambiguity of the image Y. The distribution distance of the entropy difference H (X; Y) is expressed as the difference (H (X) −H (X | Y) = H (X; Y)) of the image X from the entropy H (X | Y). To.

In addition, the image feature amount can be read as a probability distribution, and the distance as a probability distribution is a distribution distance represented by KL (Calback Lipler distance) divergence or a more generalized I (generalized KL) divergence distance. Become. These distribution distances may be replaced with the above distances to determine the degree of similarity. In addition, general statistical classification methods (K-means method, Ward method, etc., hierarchical clustering method, etc.) or machine learning methods (Support Vector Machine method, Random Forest method, Boosting method, Neural Network method, Deep Neural method, etc.) The similarity of the content statistics between the original image and the image to be classified may be replaced with the above distance by the Network method or the like).

The content classification unit 13 classifies the image to be classified based on the coordinate distance from the viewpoint of having the same atmosphere and taste as the original image. For example, the content classification unit 13 sorts the coordinate distances in ascending order, and classifies the images as having the same atmosphere as the original image in order from the classification target image having the smallest coordinate distance. In this case, the content classification unit 13 compares the coordinate distance with the preset threshold value, and when the coordinate distance is equal to or less than this threshold value, the original image and the classification target image have the same atmosphere and taste as the user. That is, the image to be classified may be classified as having the same atmosphere and taste as the original image. On the other hand, when the coordinate distance exceeds this threshold value, the content classification unit 13 does not make the user feel the same atmosphere and taste as the original image and the classification target image, that is, the classification target image has the same atmosphere and the same atmosphere as the original image. Classify as having no taste. Here, the threshold value is such that the classification target image has the same atmosphere and taste as the original image by comparing the plurality of original images with the classification target image for the original image by, for example, a plurality of humans. It is set to a value that can be determined to be present.

The content output unit 14 outputs an image classified by the content classification unit 13 as an image having the same atmosphere and taste as the original image (for example, it is displayed on a display screen of a display means (not shown)). Further, the content output unit 14 imparts classification image identification information to an image classified by the content classification unit 13 as an image having the same atmosphere and taste as the original image, and stores the classification content as a set with the classification image identification information. Write to and memorize part 18.

In the content database 15, a content group table showing image group identification information, an image group name, and image data is written and stored.
In the content feature amount database 16, a content feature amount table showing each of the feature amount identification information, the feature amount name, and the feature amount calculation formula (model) is written and stored.
In the content statistic model database 17, a content statistic table showing each of the statistic identification information, the statistic name, the feature amount identification information, and the statistic calculation formula (model) is written and stored.
The classification content storage unit 18 is a storage medium such as a hard disk, and the classification image identification information of the classified image to be classified and the image data associated with the coordinate distance are written and stored.

FIG. 2 is a diagram showing a configuration example of a content feature amount table in the content feature amount database 16. The content feature amount table of FIG. 2 has each column of the feature amount name and the feature amount calculation formula (model) corresponding to the feature amount identification information for each record. The feature amount identification information is identification information for identifying each of the content feature amounts. The feature amount name indicates each name of the content feature amount, and is, for example, the shading, luminance, and chromaticity of the image, and their contrast, gradient, edge, optical flow, and the like. Further, as the image feature amount, not only the above-mentioned type but also other features proposed in the image recognition field may be used. The feature amount calculation formula (model) is a calculation formula (or model) used to obtain the feature amount from the image (pixel value of each pixel).

FIG. 3 is a diagram showing a configuration example of a content statistic table in the content statistic model database 17. The content statistic table of FIG. 3 has columns for a statistic name and a statistic calculation formula (model) for each record, corresponding to the statistic identification information. The statistic identification information is identification information that identifies each of the content statistics. The statistic name indicates each name of the content statistic, for example, mean, variance, histogram, skewness, sharpness, maximum value, minimum value, median value, mode, bias, density, spectrum, energy spectrum, etc. Includes probability distribution, regression analysis, principal component analysis, independent component analysis, clustering, etc. The statistic calculation formula (model) is an calculation formula (or model) used to obtain the statistic from each content feature amount of the feature amount identification information shown in the same record.

Here, each explanation of the above-mentioned content feature amount and the above-mentioned content statistic is supplemented. The content feature amount is a low-order image feature amount (for example, brightness, chromaticity, contrast, gradient, etc.) obtained from information on a pixel-by-pixel basis or other information in the vicinity of the pixel.
Content statistics are statistical operations (eg, averaging, variance) on a large number of pixels in a given image area (a relatively large area of the image or a large area covering the entire image) with respect to the physical properties of the given image. , Maximum value, minimum value, median value, most frequent value, histogram, bias, density, spectrum, energy spectrum, probability distribution, regression analysis, principal component analysis, independent component analysis, clustering, etc.

Further, the content atmosphere amount is, for example, a correlation vector between the same content statistics obtained for different physical characteristics for a predetermined image region, or an image different from each other for a predetermined physical characteristic. Represents a correlation vector between the same content statistics found in a region, or a correlation vector between the same content statistics found for different physical properties in different image regions, eg, different resolutions in the same direction segment. Fixed-direction spatial frequency correlation vector by resolution, which is the correlation between the spatial frequencies obtained for, or fixed-resolution spatial frequency by direction, which is the correlation between the spatial frequencies obtained for different direction divisions at the same resolution. Includes correlation vectors, etc.

Further, the mutual correlation is used to express the similarity between images, for example, the mutual correlation between the numerical value of the color component R and the numerical value of the color component G in the RGB (Red, Green, Blue) value in the image, and the color component R. The mutual correlation between the numerical value and the numerical value of the color component B and the mutual correlation between the numerical value of the color component G and the numerical value of the color component B are the color component R and the color component G, the color component R and the color component B, and the color in the image. It can be used as a content statistic related to each of the component G and the color component B. Here, each of the RGB values is the content feature amount of the image.

The physical characteristics of the above image represent what kind of physical characteristics each image is viewed (that is, whether it is evaluated by the physical characteristics), and the density characteristics, gradation characteristics, color characteristics, spatial frequency characteristics, and resolution characteristics. Etc. are included. Further, the predetermined area of the image indicates a predetermined division area of the image, and the frequency division (classification of the image according to the spatial frequency already described), the direction classification (classification of the information of the image according to the direction already described), and the predetermined region of the image. It includes color classification (classification by color component of pixels already described) and the like.
Here, the frequency classification is supplemented. When the spatial frequency is calculated from the image using a general Fourier transform, the spatial frequency characteristic indicating the correspondence between the frequency and the intensity in the entire image is calculated. Even if the spatial frequency characteristics calculated in this way are divided for each frequency division, the relationship between the position information of the image and the spatial frequency characteristics cannot be obtained. Therefore, in the present embodiment, the spatial frequency space is calculated from the image by using a frequency conversion method that retains the position information such as wavelet transform. Then, a statistic for each spatial frequency component (for example, a statistic such as a correlation) is calculated based on the spatial frequency characteristics of each fractionated image fractionated based on the position of the image. This makes it possible to recognize whether or not the separated images have a strong correlation from the viewpoint of spatial frequency characteristics.

Here, the correlation vector between the image (characteristic image) I _α1 ^β1 obtained by the physical characteristic α1 and the scale (resolution level) β1 and the characteristic image I _α2 ^β2 obtained by the physical characteristic α2 and the scale β2 is obtained. An example of calculating as a content statistic will be described.

First, the characteristic image I _α1 ^β1 and the characteristic image I _α2 ^β2 are each divided into N pieces. However, N is an arbitrary natural number. The characteristic image I _α1 ^β1 and the characteristic image I _α2 ^β2 are, for example, N pieces by being separated by a boundary line extending radially from the center of the image at every angle (360 ° / N) as a direction division. It is divided into. Here, when the characteristic image is an intensity image showing the intensity characteristic of shading in the image, the intensity image is divided. When the characteristic image is a chromaticity image showing the chromaticity characteristic in the image, each component of the pixel value (R, G, B), each component of (RG, YB), or (L). The chromaticity image showing each component of *, a *, b *) is divided. When the characteristic image is an image showing spatial frequency characteristics, the original image of the characteristic image is divided, and for each divided image, the spatial frequency in the horizontal direction of the image and the space in the vertical direction of the image are obtained. A spatial frequency plane showing the frequency is generated. Alternatively, each of the characteristic images may be divided into N pieces in a square shape on the image plane as an arrangement division. In this case, the characteristic image is equally divided into n1 images in the horizontal direction and n2 images in the vertical direction, for example. However, each of n1 and n2 is a natural number satisfying the relationship of n1 × n2 = N.

Next, for each division in which the characteristic image I _α1 ^β1 is fractionated into N pieces, a correlation value between each division in which the characteristic image I _α2 ^β2 is divided into N pieces is calculated. The correlation value is represented by, for example, the following equation (1). Here, VPstat1 is the correlation value, α1 is the characteristic of the characteristic image I1, β1 is the scale of the characteristic image I1, α2 is the characteristic of the characteristic image I2, β2 is the scale of the characteristic image I2, and N is the number of divisions of the characteristic images I1 and I2. , S1 is one division of the characteristic image I1, S2 is one division of the characteristic image I2, ν _k ¹ is each pixel value in the division S1, ν _k ² is each pixel value in the division S2, and w is the characteristic image. The number of pixels in the width (horizontal) direction and h indicate the number of pixels in the height (vertical) direction of the characteristic image.

The correlation value shown in the equation (1) is calculated for each of the N divisions of the characteristic image I _α1 ^β1 and all the combinations with the N divisions of the characteristic image I _α2 ^β2 . The correlation value is calculated for each characteristic and each scale. That is, assuming that the number of characteristics is N _α and the number of scales is N _β , ^two correlation values (N _α × N _β × N) are calculated, and in the end, the correlation values as the content statistic (perceptual statistic) are as follows. It becomes a (N _α × N _β × N) ^two -dimensional vector as shown in the equation (2) of.

Alternatively, the relationship between each division in which the characteristic image I _α1 ^β1 is fractionated into N pieces and the division in which the characteristic image I _α2 ^β2 is divided into N pieces is shown in the following equation (3). The shift correlation value may be calculated. Here, VPstat3 is a shift correlation value, α1 is a characteristic of the characteristic image I1, β1 is the scale of the characteristic image I1, α2 is the characteristic of the characteristic image I2, β2 is the scale of the characteristic image I2, and N is the classification of the characteristic images I1 and I2. Number, S1 is one division of the characteristic image I1, S2 is one division of the characteristic image I2, m is the number of shifts, ν _k ¹ is each pixel value in the division S1, and ν _k ² is each pixel value in the division S2. , W indicate the number of pixels in the width (horizontal) direction of the characteristic image, and h indicates the number of pixels in the height (vertical) direction of the characteristic image.

For the correlation vector as the content atmosphere amount based on the shift correlation shown in the equation (3), the correlation values corresponding to the values (from 0 to N-1) that the shift number m can take, that is, N correlation values are calculated. To. After all, the shift correlation vector as the content statistic (perceptual statistic) becomes a vector of (N _α × N _β × N) ² × N dimension.

Further, one element of the "intensity" as the content feature amount (perceived feature amount) in each category obtained by fractionating the characteristic image I _α1 ^β1 into N pieces is represented by the following equation (4) (N _α ×). N _β × N) It becomes a dimensional vector. Here, VPstat4 is energy, α1 is the characteristic of the characteristic image I1, β1 is the scale of the characteristic image I1, N is the number of divisions of the characteristic image I1, S1 is one division of the characteristic image I1, and ν _k ¹ is in the division S1. Each pixel value, w indicates the number of pixels in the width (horizontal) direction of the characteristic image, and h indicates the number of pixels in the height (vertical) direction of the characteristic image.

"Energy" as a content statistic (perceptual statistic) corresponding to the energy of the formula (4) has one element represented by the following formula (5) (N _α × N _β × N) ^two -dimensional. It becomes a vector. Here, VPstat5 is energy correlation, α1 is the characteristic of the characteristic image I1, β1 is the scale of the characteristic image I1, α2 is the characteristic of the characteristic image I2, β2 is the scale of the characteristic image I2, and N is the number of divisions of the characteristic images I1 and I2. , S1 is one division of the characteristic image I1, S2 is one division of the characteristic image I2, ν _k ¹ is each pixel value in the division S1, ν _k ² is each pixel value in the division S2, and w is the characteristic image. The number of pixels in the width (horizontal) direction and h indicate the number of pixels in the height (vertical) direction of the characteristic image.

FIG. 4 is a flowchart showing an operation example of a process (hereinafter, simply referred to as a classification process) for classifying images having the same atmosphere and taste as the original image from the classification target image group performed by the content classification device 1 according to the present embodiment. .. In the flowchart shown below, the calculation formula (model) for obtaining the content feature amount and the content feature amount is already written and stored in the content feature amount database 16, and the content statistic and the content statistic are obtained. The description will be made on the premise that the arithmetic expression (model) is written and stored in the content statistic model database 17.

Step S10:
The content selection unit 10 outputs the original image, which is input from the external device by the user's operation and serves as a reference when classifying the images, to the content feature amount calculation unit 110.

Step S11:
The content selection unit 10 extracts a classification target image group from the content database 15. Then, the content selection unit 10 outputs one image of the extracted target selection image group as a classification target image to the content feature amount calculation unit 110. The content selection unit 10 may select an image group input by a user's operation via an external memory or a communication network as a classification target image group.

Step S12:
The content feature amount calculation unit 110 selects the type of image feature amount used for the classification process. The content feature amount calculation unit 110 searches, for example, the feature amount identification information corresponding to the type of the image feature amount designated by the user's operation in the content feature amount table of the content feature amount database 16. Then, the content feature amount calculation unit 110 reads out the feature amount calculation formula (model) for obtaining the content feature amount corresponding to the feature amount identification information from the content feature amount table.

Step S13:
The content statistic calculation unit 111 selects the type of image statistic used for the classification process. The content statistic calculation unit 111 selects, for example, a statistic name stored in the content statistic model table of the content statistic model database 17 corresponding to the type of image statistic specified by the user's operation. The content statistic calculation unit 111 reads the feature amount identification information corresponding to the selected statistic name from the content statistic model table of the content statistic model database 17. The type of content statistic specified by the user here may be one or a plurality.

Step S14:
The content feature amount calculation unit 110 uses a feature amount calculation formula (model) for obtaining the read content feature amount, and calculates the content feature amount from each of the image data of the original image and the image data of the image to be classified. .. Then, the content feature amount calculation unit 110 outputs the calculated content feature amounts of the original image and the classification target image to the content statistic calculation unit 111.

Step S15:
The content statistic calculation unit 111 calculates the content statistic of each of the original image and the classification target image by using the content feature amounts of the original image and the classification target image supplied from the content feature amount calculation unit 110.

Step S16:
The content atmosphere amount calculation unit 11 has a relationship (for example, for example) between the content statistics in the two regions for all combinations of the combinations of the two regions that can be taken for each region for each of the original image and the classification target image. Correlation value) is calculated, and a vector having each of the calculated relational content statistics as an element is constructed as the content atmosphere quantity.

Step S17:
The coordinate distance calculation unit 12 is a distance between the coordinate value of the content atmosphere amount of the original image and the coordinate value of the content atmosphere amount of the classification target image in the atmosphere amount space centered on the content atmosphere amount set by the user. Calculate (find) a certain coordinate distance. For example, when the content atmosphere amount is singular, the distance between the coordinate value of the content statistic of the original image in the one-dimensional atmosphere amount space and the coordinate value of the content atmosphere amount of the classification target image is obtained. On the other hand, when the content statistic is a vector having a plurality of dimensions (n ≧ 2), the coordinate values indicated by the vector of the content atmosphere of the original image in the n-dimensional atmosphere space and the coordinates indicated by the vector of the content atmosphere of the image to be classified. Find the coordinate distance between the values. Then, the coordinate distance calculation unit 12 outputs the coordinate distance between the coordinates of the content atmosphere amount of the obtained original image and the classification target image to the content classification unit 13.

Step S18:
The content classification unit 13 determines whether or not the coordinate distance has been calculated for all the images included in the classification target image group. At this time, if the coordinate distance is calculated for all the images included in the classification target image group, the coordinate distance determination unit 106 proceeds to step S18. On the other hand, if the coordinate distance is not calculated for all the images included in the classification target image group, the coordinate distance determination unit 106 proceeds to step S19.

Step S19:
The content classification unit 13 classifies the images included in the classification target image group based on the coordinate distance. The content classification unit 13 sorts all the images included in the classification target image group in ascending order from the smallest to the largest coordinate distance corresponding to each image. The content classification unit 13 classifies the image having the smallest coordinate distance as an image having the same atmosphere and taste as the original image. The content classification unit 13 classifies the image as an image that does not have the same atmosphere and taste as the original image as the coordinate distance increases.
Further, the content output unit 14 writes and stores the result of classifying the classification target image in the classification content storage unit 18 in a predetermined image format.

Step S20:
The content selection unit 10 outputs an image that has not yet been classified among the images included in the target selection image group to the content feature amount calculation unit 110 as a classification target image. Then, the content selection unit 10 advances the process to step S14. As a result, the content feature amount, the content statistic amount, and the content atmosphere amount of the classification target image are obtained, and the coordinate distance from the original image is determined again.

In the present embodiment, when the classification target image is classified, the coordinate distance between the coordinate values of each content statistic of the original image and the classification target image is sorted, and the classification target image has the same atmosphere and atmosphere as the original image. They are classified in descending order of degree of taste (smallest coordinate distance).
However, the classification target image whose coordinate distance is equal to or less than a predetermined threshold value may be classified as an image having the same atmosphere and taste as the original image. Further, a predetermined number (for example, 10) of images may be classified in ascending order of coordinate distance as images having the same atmosphere and taste as the original image.

FIG. 5 is a diagram showing a processing result of the classification process in the flowchart shown in FIG. 5 (a) is a group of images to be classified (images G-1 to G-23, ...), FIG. 5 (b) is an original image G-30 used as a classification reference, and FIG. 5 (c) is FIG. 5 (a). ), The top 10 images classified as having the same atmosphere and taste as the original image G-30 in the classification target image group, respectively, are shown. FIG. 5D shows the original image G-50 as the classification standard, and FIG. 5E shows the same atmosphere and taste as the original image G-50 in the classification target image group shown in FIG. 5A. Then, the images G-60 to G-69, which are the top 10 images classified, are shown. As shown in FIG. 5, the content classification device 1 of the embodiment classifies images having the same atmosphere and taste as the original image G-30 (or G-50) from the images included in the classification target image group. It can be confirmed that it was done.

According to the first embodiment, an image feature amount (content feature amount) and an image statistic (content statistic amount) corresponding to the human sensation such as atmosphere and taste with respect to the original image (original content) to be viewed. ) Is quantified as an image statistic (content statistic) that reflects the processing process model of the neural mechanism of the human visual (perceptual) system, and is shown as a coordinate value in the statistic space. By determining the coordinate distance between the coordinate values with (), it is possible to more accurately reproduce the sensations such as the atmosphere and taste that humans have (perceptual processing of the human brain) compared to the past, and it is an object to be viewed. It is possible to classify other contents (target contents) having the same feeling as the original image of the above from the unspecified image group (classification target image group).

As the elucidation of human brain function progresses, many other visual perceptual features are being discovered, and they may be included.

<Second embodiment>
Hereinafter, the second embodiment of the present invention will be described with reference to the drawings.
FIG. 6 is a block diagram showing a configuration example of the content classification device 1A according to the second embodiment of the present invention. In the content classification device 1A in FIG. 6, the content feature amount calculation unit 110A calculates the content feature amount using the content feature amount model generated by machine learning, and the content statistic calculation unit 111A is generated by machine learning. It differs from the first embodiment in that the content statistic is calculated using the content statistic model. Further, in the present embodiment, the content classification device 1A includes a trained model database 19. In addition, in this embodiment, the same reference numerals are given to the same configurations as those in FIG. 1 according to the first embodiment.
Hereinafter, in the present embodiment, the content will be described as a still image, but as in the first embodiment, the content may be configured to be applied to other content such as moving images, video, sound, and audio.

Hereinafter, only the configuration and operation different from the content classification device 1 of the first embodiment will be described for the content classification device 1A according to the second embodiment.
The content feature amount calculation unit 110A generates a content feature amount model that outputs the content feature amount by machine learning. That is, the user forms an input layer for an estimation model such as a neural network and inputs the image data of the teacher image (the amount of information of each of all the pixels). Further, for the above estimation model, the user forms an output layer and outputs the image data of the teacher image. The neural network as this estimation model is composed of, for example, three layers, an input layer, an intermediate layer, and an output layer. Here, the intermediate layer is composed of the number of inputs to the input layer and the number of inputs and the number of outputs smaller than the number of outputs from the output layer.

As a result, the content feature amount calculation unit 110A learns an estimation model (autoencoder: self-encoder) in which the image data of each of a plurality of different teacher data is input and the image data of each teacher image is output. conduct. That is, when the image data of the teacher image is input, the content feature amount calculation unit 110A generates an estimation model in which an image similar to the input teacher image is output by machine learning. Then, the content feature amount calculation unit 110A extracts each of the input layer and the intermediate layer (the layer in front of the output layer) constituting the estimation model as the estimated content feature amount model. The content feature amount calculation unit 110A uses the output of the estimated content feature amount model as the estimated content feature amount (estimated image feature amount).
Further, the content feature amount calculation unit 110A stores the estimation model generated by machine learning in the trained model database 19.

The content statistic calculation unit 111A generates a content statistic model that outputs the content statistic by machine learning. That is, the user forms an input layer for an estimation model such as a neural network and inputs the content features of the teacher image. The content feature amount here is, for example, the content feature amount shown in the first embodiment, or the estimated content feature output by the content feature amount calculation unit 110A shown in the second embodiment from the estimated content feature amount model. The quantity. Further, for the above estimation model, the user forms an output layer and outputs the image data of the teacher image. The neural network as this estimation model is composed of, for example, three layers, an input layer, an intermediate layer, and an output layer. Here, the intermediate layer is composed of the number of inputs to the input layer and the number of inputs and the number of outputs smaller than the number of outputs from the output layer.

As a result, the content statistic calculation unit 111A receives the data of the content feature amount of each of the plurality of different teacher images as input, and trains the estimation model in which the content feature amount of each teacher image is output. That is, when the content feature amount of the teacher image is input, the content statistic calculation unit 111A outputs an estimation model (autoencoder: self-encoding) similar to the content feature amount of the input teacher image. (Vessel) is generated by machine learning. Then, the content feature amount calculation unit 110A extracts each of the input layer and the intermediate layer (the layer in front of the output layer) constituting the estimation model as the estimation content statistic model. The content statistic calculation unit 111A uses the output of the estimated content statistic model as the estimated content statistic (estimated image statistic).
Further, the content statistic calculation unit 111A stores the estimation model generated by machine learning in the trained model database 19.

FIG. 7 is a flowchart showing an operation example of the classification process performed by the content classification device 1A according to the present embodiment. In the flowchart shown below, the estimated model that has already been generated by the content feature amount calculation unit 110A and that outputs the same image as the input teacher image when the image data of the teacher image is input is the trained model. The description will be made on the premise that it is written and stored in the database 19 and that the content statistic and the calculation formula (model) for obtaining the content statistic are written and stored in the content statistic model database 17. ..
Since the processes shown in steps S10 to S11 and S16 to S19 in FIG. 7 are the same as the processes shown by the same reference numerals in FIG. 4, the description thereof will be omitted.

Step S12A:
The content feature amount calculation unit 110A selects an estimation model for extracting an image feature amount model from the trained model database 19. The content feature amount calculation unit 110A may select the estimation model according to, for example, the amount of data of the original image and the classification target image.

Step S13A:
The content statistic calculation unit 111A selects an estimation model for extracting an image statistic model from the trained model database 19. The content statistic calculation unit 111A may select the estimation model according to, for example, the number of images to be classified or the category.

Step S14A:
The content feature amount calculation unit 110A calculates the content feature amount in each image of the original image or the classification target image by inputting the image data of the original image or the classification target image into the selected estimation model.

Step S15A:
The content statistic calculation unit 111A calculates the image statistic between the original image and the classification target image by inputting the content feature amount of the original image and the classification target image into the selected estimation model.

FIG. 8 is a flowchart showing an operation example of a process for calculating an estimated content feature amount performed by the content feature amount calculation unit 110A in the content classification device 1A according to the present embodiment.

Step S21:
The content feature amount calculation unit 110A temporarily writes and stores each of the plurality of teacher images input by the user in a storage unit (not shown). In the image data of these teacher images, the number of pixels in each of the vertical and horizontal directions is set to be the same.

Step S22:
The content feature amount calculation unit 110A sets the configuration of an input layer for inputting information of each pixel of the image data of the teacher image and an output layer for outputting the pixel of the image data of the estimated estimated image. Then, the content feature amount calculation unit 110A sets an intermediate layer having a smaller number of data inputs and outputs as compared with the image data supplied from the input layer and the image data output from the output layer. The content feature amount calculation unit 110A constitutes an estimation model of a neural network composed of the input layer, the intermediate layer, and the output layer.

Step S23:
The content feature amount calculation unit 110A inputs pixel information in the image data of the teacher image to the input layer of the estimation model, and outputs pixel information similar to the image data of the teacher image input from the output layer. As you can see, machine learning of the middle layer is performed. The content feature amount calculation unit 110A performs the above machine learning process using each image data of all the teacher images, and the pixel information of each output image data is the image data of the input teacher image. If the information in the pixel is similar to the information in a predetermined range, the estimation model at that time is used as the original image estimation model.

Step S24:
The content feature amount calculation unit 110A removes the output layer in the original image estimation model obtained by learning, and extracts it as an estimated content feature amount model (estimated image feature amount model) including an input layer and an intermediate layer.

Step S25:
The content feature amount calculation unit 110A writes (registers) the estimated content feature amount model, which is the output of the intermediate layer, in the trained model database 19 as a model for outputting the content feature amount.

FIG. 9 is a flowchart showing an operation example of a process for calculating an estimated content statistic performed by the content statistic calculation unit 111A in the content classification device 1A according to the present embodiment.

Step S30:
The content statistic calculation unit 111A temporarily writes and stores each of the plurality of teacher images input by the user in a storage unit (not shown). In the image data of these teacher images, the number of pixels in each of the vertical and horizontal directions is set to be the same.

Step S31:
The user inputs a combination of content features (image features) used when calculating the estimated content statistic model to the content statistic calculation unit 111A via the input means of the content classification device 1A.

Step S32:
The content statistic calculation unit 111A causes the content feature amount calculation unit 110A to calculate the content feature amount in the selected combination from each of the teacher images.
The content feature amount calculation unit 110 obtains each of the types of content feature amounts in the above combination in each of the teacher images.

Step S33:
The content statistic calculation unit 111A sets the configuration of an input layer for inputting the content feature amount data of the teacher image and an output layer for outputting the estimated content feature amount data. Then, the content statistic calculation unit 111A sets an intermediate layer having a smaller number of inputs and outputs of data as compared with the image data supplied from the input layer and the image data output from the output layer. The content statistic calculation unit 111A constitutes an estimation model of a neural network composed of the input layer, the intermediate layer, and the output layer.

Step S34:
The content statistic calculation unit 111A inputs the data of the feature amount of the teacher image to the input layer of the estimation model, and outputs the same data as the data of the feature amount of the teacher image input from the output layer. In addition, machine learning of the middle layer is performed. The content statistic calculation unit 111A performs the above machine learning process using each combination of content feature amounts of all teacher images, and the data of each output content feature amount is the input teacher image. When the data of the content feature amount is similar to the data in a predetermined range, the estimation model at that time is used as the original image estimation model.

Step S35:
The content statistic calculation unit 111A removes the output layer in the original image estimation model obtained by learning, and extracts it as an estimated content statistic model (estimated image statistic model) including an input layer and an intermediate layer.

Step S36:
The content statistic calculation unit 111A writes the estimated content statistic, which is the output of the intermediate layer, as the content statistic, and writes the estimated content statistic model to the trained model database 19 as the model for outputting the content statistic. Remember (register).

According to the second embodiment, the image feature amount (content feature amount) and the image statistic (content statistic) corresponding to the human sensation such as the atmosphere and taste of the original image (original content) to be viewed. Based on, it is quantified as an image atmosphere amount (content atmosphere amount) that reflects the processing process model of the neural mechanism of the human visual (perceptual) system, and is shown as a coordinate value in the atmosphere amount space, and the original image and the classification target image (target). By determining the coordinate distance between the coordinate values of the content), it is possible to more accurately reproduce the sensations such as the atmosphere and taste that humans have (perceptual processing of the human brain), and appreciate it. It is possible to classify other contents (target contents) having the same feeling as the original image of the target from the unspecified image group (classification target image group).
Further, in the present embodiment, since the intermediate layer of the autoencoder (three-layer neural network) for inputting the image data of the teacher image is used as the content feature amount, the content feature amount corresponding to the feature extraction of the neural mechanism in the brain. It becomes possible to select and classify other contents (target contents) having a feeling closer to the original image to be viewed as compared with the first embodiment.

<Third embodiment>
FIG. 10 is a block diagram showing a configuration example of the content classification device 1B according to the third embodiment of the present invention. In the content classification device 1B in FIG. 10, the content atmosphere amount calculation unit 11B does not use the content feature amount calculation unit 110 (110A) and the content statistic calculation unit 111 (111A), and the content classification device 1B is generated by deep learning. It differs from the above embodiment in that the content statistic is calculated from the image data using the image model. Further, the content classification device 1B includes a deep learning model database 20. In the third embodiment, the same reference numerals are given to the same configurations as those in FIG. 1 according to the first embodiment.

Hereinafter, in the present embodiment, the content will be described as a still image, but as in the first embodiment, the content may be configured to be applied to other content such as moving images, video, sound, and audio.
Hereinafter, only the configuration and operation different from the content classification device 1 of the first embodiment will be described for the content classification device 1B according to the third embodiment.

The content atmosphere amount calculation unit 11B generates a deep learning image model by deep learning, and stores the generated deep learning image model in the deep learning completed model database 20.
Further, the content atmosphere amount calculation unit 11B calculates an image statistic for each image from each image data of the original image and the classification target image by using the deep learning image model stored in the deep learning model database 20. do.
The deep-learned model database 20 stores the deep-learned image model generated by the deep-learning as a content statistic model for calculating the content statistic.

There are two types of processing A1 and processing A2 below as processing for generating the deep learning image model generated by the content atmosphere amount calculation unit 11B.
FIG. 11 is a flowchart showing an operation example of the process A1 for generating the deep learning image model. In the case of this process A1, for the image of interest, a plurality of images for perceiving that the atmosphere is similar and images for perceiving that the atmosphere is not similar are prepared as a learning image.
Step S40:
The content atmosphere amount calculation unit 11B temporarily writes and stores each of the plurality of learning images input by the user in a storage unit (not shown). In the image data of these learning images, the number of pixels in each of the vertical and horizontal directions is set to be the same.

Step S41:
The content atmosphere amount calculation unit 11B normalizes the deep neural network model in which the intermediate layer (pooling layer and convolution layer) has a multi-layer structure with an input layer for inputting information of each pixel of the image data of the learning image. Set the output layer, which is a fully connected layer. This output layer is configured to output the numerical value of the decimal point between "1" and "0".

Step S42:
When the content atmosphere amount calculation unit 11B inputs an image that perceives that the atmosphere is similar to the image of interest to the input layer of the deep neural network model, the output layer indicates that the image is similar to "1". If you input an image that you perceive that the atmosphere is not similar to the image you like, the output layer will output a value close to "0", which indicates that it is similar. The weight parameter of each layer of each network is optimized. That is, the content atmosphere amount calculation unit 11B performs machine learning of classification on the deep neural network model, and generates a deep learning image model as a learning result.

At this time, the content atmosphere amount calculation unit 11B perceives that the trained deep neural network model does not have a similar atmosphere to an image that is different from the learning image and has a similar atmosphere to the favorite image. An image is input and a learning test (cross validation) is performed on the trained deep neural network model.
Then, when the content atmosphere amount calculation unit 11B inputs an image perceived as having a similar atmosphere into the deep neural network model, the numerical value output by the output layer becomes equal to or higher than the preset first threshold value, and the atmosphere is similar. When an image that is perceived as not being input is input to the deep neural network model, and the numerical value output by the output layer is equal to or less than the preset second threshold value, this deep neural network model is used as a deep learning image model for the image of interest. do. On the other hand, in the above learning test, the content atmosphere amount calculation unit 11B indicates that the numerical value output by the output layer of the deep neural network model is less than the preset first threshold value or the atmosphere for the images perceived to have similar atmospheres. When the numerical value output from the output layer of the deep neural network model is equal to or higher than the preset second threshold value for the images perceived as dissimilar, the deep neural network model is relearned.

Step S43:
From the generated deep learning image model, the content atmosphere amount calculation unit 11B obtains each of the output parameters of the pooling layer and the convolution layer in the intermediate layer of the multi-layer structure, the type of activation function, the output parameters, and the like, as deep learning statistics. (Or deep learning features) Extract as each.

Step S44:
The content atmosphere amount calculation unit 11B writes and stores the generated deep learning image model and the extracted deep learning statistic (deep learning feature amount) in the deep learning completed model database 20 (registration process).
The above processing A1 is performed for each image of interest, and a deep learning image model corresponding to each is generated.

FIG. 12 is a flowchart showing an operation example of the process A2 that generates a deep learning image model. In the case of this process A2, transfer learning of the deep learning image model generated in the process A1 is performed, and another deep learning image model (deep learning image model corresponding to another favorite image) is generated.
Step S50:
The content atmosphere amount calculation unit 11B temporarily writes and stores each of the plurality of learning images input by the user in a storage unit (not shown). In the image data of these learning images, the number of pixels in each of the vertical and horizontal directions is set to be the same.

Step S51:
The content atmosphere amount calculation unit 11B includes an input layer for inputting information of each pixel of the image data of the learning image to which the intermediate layer (pooling layer and convolution layer) is connected to the deep neural network model having a multi-layer structure. Set the output layer, which is the fully connected layer to be normalized. This output layer is configured to output the numerical value of the decimal point between "1" and "0".

Step S52:
When a user generates a deep learning image model corresponding to a favorite image by a predetermined input means (not shown) for the content classification device 1B, a new deep learning image model is created by using a deep neural network model. It controls whether to generate or to generate a new deep learning image model by transfer learning using a deep learning image model of another favorite image. For example, when a user generates a deep learning image model for an image that he / she likes to prepare a large amount of learning images, he / she controls to learn and generate a deep neural network model by machine learning. On the other hand, when the user generates a deep learning image model for a favorite image for which a large amount of learning images cannot be prepared, the user transfers and learns a deep learning image model corresponding to another favorite image already obtained by learning. , Controls to generate a deep learning image model.

At this time, when the user selects a process for newly generating a deep learning image model of the image of interest from the deep neural network model, the content atmosphere amount calculation unit 11B advances the process to step S55. On the other hand, when the content atmosphere amount calculation unit 11B selects a process of performing transfer learning on a deep learning image model of another favorite image and generating a deep learning image model of the favorite image, the process is stepped. Proceed to S53.

Step S53:
The content atmosphere amount calculation unit 11B generates a predetermined deep learning model from the deep learning image models already stored in the deep learning model database 20. For example, a deep learning image model of another favorite image that the user perceives as dissimilar to the favorite image of the target for which the user generates a deep learning image model is specified, and the content atmosphere amount calculation unit 11B uses this deep learning image. Select the model as a deep learning image model for transfer learning. Here, the content atmosphere amount calculation unit 11B reads out the deep learning image model used for the transfer learning from the deep learning completed model database 20.

Step S54:
Since the content atmosphere amount calculation unit 11B is used for transfer learning, the content atmosphere amount calculation unit 11B extracts from the read deep learning image model from the input layer to the intermediate layer (adapted layer) designated by the user or designated in advance as the transfer learning model.
Then, the content atmosphere amount calculation unit 11B extracts the intermediate layer after the adaptation layer from the deep neural network model, connects it to the adaptation layer of the transfer learning model, and connects the output layer to perform transfer learning. Generate a transition deep learning image model to be used.

Step S55:
The content atmosphere amount calculation unit 11B selects and reads a predetermined deep neural network model from the deep neural network model stored in the deep learned model database 20.

Step S56:
The content atmosphere amount calculation unit 11B outputs when an image that perceives that the atmosphere is similar to the image of interest is input to the input layer of the training target model (the transition deep learning image model or the deep neural network model). If you input a value close to "1" that indicates that the layer is similar, or if you input an image that you perceive that the atmosphere is not similar to the image you like, the output layer will indicate that it is not similar. The weight parameter of each layer of each network is optimized so that a value close to "0" is output. That is, the content atmosphere amount calculation unit 11B performs machine learning of classification on the learning target model, and generates a deep learning image model as a learning result.

At this time, the content atmosphere amount calculation unit 11B perceives that the created learning target model has a similar atmosphere to an image that is different from the learning image, and an image that perceives that the atmosphere is not similar. Is input, and a learning test is performed on the trained target model. Then, when the content atmosphere amount calculation unit 11B inputs an image perceived to have a similar atmosphere into the learning target model, the numerical value output by the output layer becomes equal to or higher than a preset first threshold value, and the atmosphere is not similar. When the value output by the output layer is equal to or less than the preset second threshold value when the image to be perceived is input to the training target model, this training target model is used as a deep training image model for the desired image. On the other hand, in the above learning test, the content atmosphere amount calculation unit 11B indicates that the numerical value output by the output layer of the learning target model is less than the preset first threshold value for the image perceived to have a similar atmosphere, or the atmosphere is similar. When the numerical value output from the output layer of the learning target model is equal to or higher than the preset second threshold value for the image perceived as not being used, the learning target model is retrained.

Step S57:
From the generated deep learning image model, the content atmosphere amount calculation unit 11B obtains each of the output parameters of the pooling layer and the convolution layer in the intermediate layer of the multi-layer structure, the type of activation function, the output parameters, and the like, as deep learning statistics. (Or deep learning features) Extract as each.

Step S58:
The content atmosphere amount calculation unit 11B writes and stores the generated deep learning image model and the extracted deep learning statistic (deep learning feature amount) in the deep learning statistic model table of the deep learning completed model database 20. (registration process).
The above processing A2 is performed for each image of interest, and a deep learning image model corresponding to each is generated.

FIG. 13 is a flowchart showing an operation example of a process for classifying a classification target image having the same atmosphere and taste as the original image, which is performed by the content classification device 1B according to the present embodiment. In the flowchart shown below, it is assumed that the deep learning image model has already been written and stored in the deep learning model database 20 as an arithmetic expression for obtaining the content statistics (or content features). ..
Since the processing shown in each of steps S64 to S68 in FIG. 13 is the same as the processing shown in steps S16 to S20 in FIG. 4, the description thereof will be omitted.

Step S60:
The content selection unit 10 inputs an original image (a favorite image) as a reference for classification from an external device and outputs it to the content atmosphere amount calculation unit 11B by the user's operation.

Step S61:
The content selection unit 10 extracts a classification target image group from the content database 15. Then, the content selection unit 10 outputs an image of one of the extracted target selection image groups as a classification target image to the content atmosphere amount calculation unit 11B. The content selection unit 10 may select an image group input by a user's operation via an external memory or a communication network as a classification target image group.

Step S62:
The content atmosphere amount calculation unit 11B selects a deep learning image model corresponding to the desired image, which is the original image, from the deep learning image models stored in the deep learning model database 20.

Step S63:
The content atmosphere amount calculation unit 11B calculates the deep learning statistic (that is, the content statistic) of each of the original image and the classification target image by the selected deep learning image model.
That is, the content atmosphere amount calculation unit 11B inputs the pixel value of each pixel in the original image to the input layer of the deep learning image model. Then, the content atmosphere amount calculation unit 11B deep-learns the output parameters of the pooling layer and the convolution layer in the intermediate layer of the multi-layer structure of the deep learning image model, the type of the activation function, the output parameters, and the like. Extract as a statistic.

Similarly, the content atmosphere amount calculation unit 11B inputs the pixel value of each pixel in the classification target image to the input layer of the deep learning image model. Then, the content atmosphere amount calculation unit 11B sets each of the output parameters of the pooling layer and the convolution layer in the intermediate layer of the multi-layered structure of the deep learning image model, the type of the activation function, the output parameters, and the like into the deep layer of the image to be classified. Extract as a learning statistic.

According to the third embodiment, the processing process of the neural mechanism of the human visual (perceptual) system is deepened by making the original image (original content) to be viewed correspond to the sensation such as the atmosphere and taste that the human has. The neural network model is approximated as a deep learning image model generated by deep learning, and the content atmosphere amount that reflects the processing process model of the neural mechanism of the human visual (perceptual) system is quantified from the output, and in the content atmosphere amount space. By showing it as a coordinate value and determining the coordinate distance between the coordinate value between the original image and the classification target image (target content), the human senses such as atmosphere and taste (perceptual processing of the human brain) have been conventionally performed. It can be reproduced more accurately by comparison, and it becomes possible to generate other contents (target contents) having the same feeling as the original image to be viewed.
Further, in the present embodiment, a deep learning image model is generated by using a deep neural network model in which image data of a training image is input and the result of classification is obtained, and the intermediate layer of the deep learning model is used as content statistics. Since it is a quantitative model, it is possible to obtain content statistics corresponding to the extraction of the characteristics of the neural mechanism in the brain, and the original image to be viewed has a feeling closer to that of the first embodiment. It becomes possible to classify the contents (target contents) of.

A program for realizing all or part of the functions of the content classification device 1 (1A, 1B) in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is recorded in the computer system. The process may be performed by loading it into the computer and executing it. The term "computer system" as used herein includes hardware such as an OS and peripheral devices.
Further, the "computer system" shall also include a WWW system provided with a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, it shall include those that hold the program for a certain period of time.

Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned function in combination with a program already recorded in the computer system.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention.

1, 1A, 1B ... Content classification device 10 ... Content selection unit 11, 11A, 11B ... Content atmosphere amount calculation unit 110, 110A ... Content feature amount calculation unit 111, 111A ... Content statistics calculation unit 12 ... Coordinate distance calculation unit 13 … Content classification unit 14… Content output unit 15… Content database 16… Content feature amount database 17… Content statistic model database 18… Classification content storage unit 19… Trained model database 20… Deeply trained model database

Claims (13)

A content classification device that classifies target content that has the same atmosphere and taste as the original content, which is the image used as the classification standard, from the target content group that is the image to be classified.
For the combination of two regions that are different from each other that can be taken in each region determined when the characteristics, scale, and division according to the characteristics are specified in the image, the relationship quantity that indicates the relationship between the two regions is shown. , A content atmosphere amount calculation unit that calculates the content atmosphere amount for each of the original content and the target content by using the content atmosphere amount that quantifies the atmosphere and taste perceived by a person for the image .
A distance calculation unit that calculates the distance between the coordinate value of the content atmosphere amount of the original content and the coordinate value of the content atmosphere amount of the target content.
A content classification device including a content classification unit for classifying the target content based on the distance. The content atmosphere amount calculation unit has a plurality of characteristics that are different from each other as the characteristics, and two different characteristics that can be taken in each region determined when the classification according to the characteristics, the scale, and the plurality of characteristics that are different from each other is specified. The relationship amount is calculated for the combination of regions, and the relationship amount corresponding to each combination of a plurality of characteristics different from each other is calculated as the content atmosphere amount.
The content classification device according to claim 1. The content classification according to claim 2, wherein the content atmosphere amount calculation unit calculates at least one of simple correlation, shift correlation, and cross-correlation as a relationship between the two regions. Device. A content feature amount storage unit that stores a plurality of types of arithmetic expressions corresponding to each content feature amount of the original content and the target content, and a content feature amount storage unit.
Further, it is provided with a content statistic storage unit in which a plurality of types of arithmetic expressions corresponding to the content statistic, which is the statistic of the content feature amount, are stored.
The content atmosphere amount calculation unit
A content feature amount calculation unit that obtains the content feature amount from each of the original content and the target content by an arithmetic expression corresponding to the content feature amount.
Using the content feature amount calculated by the content feature amount calculation unit, the content statistic is obtained for each of the original content and the target content by an arithmetic expression corresponding to the content statistic. ,
Have,
The content atmosphere amount calculation unit
A claim characterized in that the content atmosphere amount for each of the original content and the target content is calculated by an arithmetic formula corresponding to the relationship by using the content statistic calculated by the content statistic calculation unit. Item 2 or the content classification device according to claim 3. The content feature amount is an image feature amount obtained from the pixel information of the image, and is characterized by being an image feature amount including at least the shading, brightness, chromaticity, contrast, gradient, edge, and optical flow of the image. The content classification device according to claim 4. The content statistic is a statistic obtained by applying a certain statistical operation to the content feature amount obtained in a predetermined region of a predetermined physical characteristic in an image, and the physical characteristic is a statistic. The region includes at least spatial frequency divisions, direction divisions, and color divisions, including at least density characteristics, gradation characteristics, color characteristics, spatial frequency characteristics, and resolution characteristics, and the statistical operation is at least average, variance, and histogram. , The content classification apparatus according to claim 5, wherein the content classification apparatus includes a power spectrum. The content classification device according to claim 5 or 6, wherein the distance is any one of a Euclidean distance, a cosine distance, a Hamming distance, a Manhattan distance, and a Calvac-Ripler distance. When the distance becomes equal to or less than a preset threshold value, the content classification unit classifies the target content corresponding to the distance as an image having the same atmosphere and taste as the original content. The content classification device according to any one of claims 1 to 7. Claim 1 to claim 1, wherein the content classification unit classifies the target content corresponding to the distance into images having the same atmosphere and taste as the original content in ascending order of the distance. The content classification device according to any one of 7. The content feature amount calculation unit obtains the content feature amount as the output of the intermediate layer of the autoencoder in which a plurality of learning images are trained as inputs and outputs.
The content classification device according to claim 4, wherein the content statistic calculation unit obtains the content statistic as an output of an intermediate layer of an autoencoder trained to use the content feature amount as an input and an output. The content atmosphere amount calculation unit uses the pixel values of the pixels of a plurality of learning images to classify a neural network having a multi-stage intermediate layer by deep learning, or by a deep learning image model of another deep learning image model. The content classification device according to claim 1, wherein the content atmosphere amount is obtained by a deep learning image model classified by transfer learning using an intermediate layer. It is a content classification method for classifying target content having the same atmosphere and taste as the original content which is the image to be classified from the target content group which is the image to be classified.
The relationship between the two areas regarding the combination of two different areas that can be taken in each area determined when the content atmosphere amount calculation unit specifies the characteristics, scale, and the classification according to the characteristics in the image. The relationship amount indicating the sex is defined as the content atmosphere amount in which the atmosphere and taste perceived by a person for the image are quantified, and the content atmosphere amount calculation for calculating the content atmosphere amount for each of the original content and the target content. Process and
A distance calculation step in which the distance calculation unit calculates the distance between the coordinate value of the previous content atmosphere amount of the original content calculated by the content atmosphere amount calculation unit and the coordinate value of the content atmosphere amount of the target content. When,
A content classification method, characterized in that the content classification unit has a content classification step of classifying the target content based on the distance. Computer,
The program for functioning as the content classification device according to any one of claims 1 to 11.

Images