JP2017168057A - Device, system, and method for sorting images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device, system, and program, capable of accurately sorting various images collected from the Internet in any classification.SOLUTION: An image sorting device 1 is provided with a plurality of image determination units 10-14 with machine learning capability. The image determination unit 10-14 are arrange in a tree structure. Each image determination unit 10-14 is configured to determine classification of target images based on learning data that is built using image data sets, each belonging to a different classification.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image classification device, an image classification system, and an image classification method for classifying images such as still images and moving images. More specifically, the present invention relates to a technique for classifying various types of images collected from the Internet or the like based on the characteristics of the images themselves.

  In recent years, techniques for analyzing and classifying images posted on the Internet have been developed (see, for example, Patent Documents 1 and 2). For example, Patent Literature 1 describes a method of defining a base shape included in an image and classifying the image based on whether or not a base shape exists in the determination target image. Patent Document 2 describes a method of classifying an image corresponding to a concept of a specific keyword using explanatory text information and image information around the image.

  On the other hand, a method of classifying images using machine learning (machine learning) has also been proposed (see Patent Document 3). The image classification device described in Patent Document 3 calculates a feature vector including, as an element, a gradient moment that is a product of a gradient of pixel values and coordinates for an image to be determined, and obtains it by machine learning using a known image. Image classification is performed based on the similarity to the obtained results.

  Conventionally, a technique has also been proposed in which learning data for object recognition is automatically created from a plurality of image data collected by a web search based on keywords without human intervention (see Patent Document 4). In the learning data creation device described in Patent Literature 4, the area image of the collected image data is classified into predetermined clusters based on the feature amount, and a cluster with a small area image distribution is identified from the distribution state of each cluster. Thus, the area image belonging to the cluster is deleted from the image data and used as learning data.

JP-A-11-345340 JP 2007-317133 A JP-A-2015-11552 JP 2012-22419 A

  There are a wide variety of images such as still images, moving images, and computer graphics (CG) on the Internet, and there are also images with a plurality of elements. However, with the conventional image classification method described above, it is difficult to accurately classify all these images. Furthermore, when a conventional image classification method is used, an extremely high technical capability and a great deal of experience and knowledge are required to improve classification accuracy and realize high-speed processing.

  Accordingly, an object of the present invention is to provide an image classification device, an image classification system, and an image classification method that can accurately classify various images collected from the Internet into arbitrary classifications.

An image classification device according to the present invention has a plurality of image determination units having a machine learning function, and the plurality of image determination units are arranged in a tree structure, and classify target images based on different learning data. Determine.
In the image classification device of the present invention, the learning data may be constructed from sample images having different sections for each of the image determination units.
Further, the lower image determination unit of the tree structure may determine a lower concept category than the upper image determination unit of the tree structure.
On the other hand, as the machine learning, for example, deep learning can be applied.
In addition, a convolutional neural network may be mounted on the image determination unit.

  An image classification system according to the present invention includes the above-described image classification device.

An image classification method according to the present invention is an image classification method for classifying a target image using a machine learning method, wherein the first learning data is constructed based on two or more types of image data belonging to the first category. Are compared with the target image to determine which type of image in the first category the target image is, and a second subordinate concept than the first category. The second learning data constructed on the basis of two or more types of image data belonging to the category is compared with the target image, and the type of the image in the second category is the target image A second determination step for determining the target image and the third learning data constructed based on two or more types of image data belonging to a third category of a lower concept than the first category, and the target image And the target image is any type of the third section A third determination step of determining whether the image, and based on the determination result of the first determination step, the second determination step or said third determination step.
In addition to the above-described steps, the image classification method of the present invention includes fourth learning data constructed based on two or more types of image data belonging to a fourth category that is a lower concept than the third category, A fourth determination step may be performed in which the target image is compared with the target image to determine which type of image in the fourth section.
In the determination step, the determination may be performed based on a matching rate with the learning data calculated using a feature amount extracted from the target image.
On the other hand, as the machine learning, for example, deep learning can be applied.
The determination step can also be executed by a convolutional neural network.
The target image may be an image collected from the Internet.

  According to the present invention, since a plurality of image determination units having a machine learning function are arranged in a tree shape and determination is performed based on different learning data for each determination unit, various images collected from the Internet can be used for the purpose. It is possible to classify with high accuracy in any category set accordingly.

It is a conceptual diagram which shows the structural example of the image classification apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the method of classifying an image using the image classification apparatus 1 shown in FIG. It is a conceptual diagram which shows the structural example of the image classification system which concerns on the 2nd Embodiment of this invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments described below.

(First embodiment)
First, an image classification apparatus according to the first embodiment of the present invention will be described. FIG. 1 is a conceptual diagram illustrating a configuration example of an image classification apparatus according to the present embodiment. As shown in FIG. 1, the image classification device 1 according to the present embodiment analyzes image data such as still images and moving images, classifies subjects and shooting scenes into arbitrary categories, and has a plurality of machine learning functions. Are determined in a tree structure (tree shape).

[Image determination unit]
The image determination units 10 to 14 determine image classifications based on determination criteria (image characteristics) acquired using a machine learning technique. Each of the image determination units 10 to 14 learns using sample image data having different sections, and has different determination criteria. Here, examples of the image classification include, for example, image types such as still images, moving images, live-action images, and artificial images, and genres of subjects such as people, landscapes, and artificial objects, but are not limited thereto. It can be arbitrarily set according to the purpose.

  The machine learning method used for the image determination units 10 to 14 is not particularly limited, and can be implemented using a known algorithm, but from the viewpoint of classification accuracy, deep learning is used. It is preferable. Deep learning is a machine learning method using a high-level neural network, and image identification can be performed based on learning data. Among deep learning, a convolutional neural network (CNN) is particularly suitable as the determination units 10 to 14 because it can identify images with high accuracy.

  The image determination units 10 to 14 are arranged to determine the classification of the higher concept on the upper side, that is, the side on which the image data to be determined is input, and to determine the classification of the lower concept as the lower level is reached. Has been placed. Specifically, the image determination unit 10 arranged at the uppermost level (first level) determines an image type such as a real image or an artificial image. In this case, the image determination unit 10 may be trained using a sample image previously tagged as “actual image” and a sample image tagged as “artificial image”.

  The image determination units 11 and 12 branching from the image determination unit 10 determine images according to the lower concept classification than the image type. For example, when the image determination unit 11 determines an “artificial image” and the image determination unit 12 determines a “real image”, the image determination unit 11 performs animation or computer graphics (CG: computer). graphics) or the like, and determination results A1 and A2 are obtained. On the other hand, the image determination unit 12 that determines “real images” determines the genre of a subject such as a landscape or a person.

  In the image determination units 13 and 14 branched from the image determination unit 12 described above, an image in which the subject is determined to be “landscape” or “person” is determined based on a lower concept. For example, the image determination unit 13 determines whether the subject is streetscape, natural, or other for an image determined to be “landscape”, and obtains determination results B11 to B13. On the other hand, the image determination unit 14 determines whether or not the image determined as “person” is self-portrait, and obtains determination results B21 and B22.

  Similar to the image determination unit 10, the image determination units 11 to 14 may be learned using the sample images corresponding to the sections to be determined in advance. Further, the arrangement of the image determination units 10 to 14 is not limited to the configuration illustrated in FIG. 1, and the number of branches and the hierarchy from each image determination unit can be appropriately selected according to the purpose. Furthermore, two or more groups of image determination units of a tree structure may be arranged in parallel to improve the processing speed. In this case, one image determination unit group and another image determination unit group may perform the same processing or different processing.

[Others]
In addition to the above-described image determination unit, the image classification device 1 according to the present embodiment may include an object detection unit that detects an object included in an image, an object classification unit that classifies the detected object, and the like. .

[Operation]
Next, an operation of the image classification apparatus according to the embodiment of the present invention, that is, a method for classifying images using the image classification apparatus according to the present embodiment will be described. FIG. 2 is a flowchart showing a method for classifying images using the image classification apparatus 1 shown in FIG. Images classified by the image classification device of the present embodiment can be collected from various social media connected to the Internet, for example, and the type, size, format, etc. are not limited.

[Step S1]
When classifying an image with the image classification device 1 of the present embodiment, first, the image type is determined. Specifically, the determination target image data is input to the first-stage image determination unit 10 in which the learning data of the “artificial image” and the “real image” is constructed by the sample image, and the “artificial image” and the “real image” are input. Is determined.

  In the image determination unit 10, for example, with respect to an image to be determined, a feature amount is extracted from the whole instead of a specific part, and compared with learning data, the probability of being an “artificial image” and a “real image” ( The matching rate with each learning data) is calculated. Then, the image determination unit 10 determines whether the image is an “artificial image” or a “real image” depending on whether or not the calculated matching rate is equal to or greater than a preset reference value.

  Note that if both of the coincidence rates of the “artificial image” and the “real image” are less than the reference value, it is assumed that the coincidence rate is higher and the next determination is performed. If the match rate is the same value, the next determination may be performed assuming that the predetermined priority is higher.

  Subsequently, based on the determination result, determination is performed by the second-stage image determination units 11 and 12. Specifically, when the image determination unit 10 determines “artificial image”, the image data to be determined is input to the image determination unit 11 to determine the creation method. On the other hand, if the image determination unit 10 determines “real image”, the image data to be determined is input to the image determination unit 12 to determine the genre of the subject.

[Step S11]
When the image determination unit 10 determines “artificial image”, the second-stage image determination unit 11 in which learning data of “animation” and “other artificial images” is constructed by the sample image is used as the determination target image. Enter the data. The image data input to the image determination unit 11 is the same as that input to the image determination unit 10.

  The image determination unit 11 extracts a feature amount from the entire determination target image, compares it with learning data, and calculates the probability of “animation” and “other artificial images” (matching rate with each learning data). ) Is calculated. Then, the image determination unit 11 determines whether the image is “animation” or “other artificial image” depending on whether or not the calculated matching rate is equal to or higher than a preset reference value. Output the result.

  If the matching rates of “animation” and “other artificial images” are both lower than the reference value, it is assumed that the matching rate is higher, and the result is output. If the match rate becomes the same value, the result may be output assuming that the predetermined priority is higher.

[Step S21]
On the other hand, when the image determination unit 10 determines “real image”, the second-stage image determination unit 12 in which learning data of “landscape image” and “person image” is constructed by the sample image is determined. Input image data. The image data input to the image determination unit 12 is the same as that input to the image determination unit 10.

  The image determination unit 12 extracts a feature amount from the entire determination target image, compares it with learning data, and calculates the probability of “landscape image” and “person image” (matching rate with each learning data). Is calculated. Then, the image determination unit 12 determines whether the image is a “landscape image” or a “person image” depending on whether the calculated matching rate is equal to or higher than a preset reference value.

  If the matching rates of the “landscape image” and the “person image” are both less than the reference value, it is assumed that the matching rate is higher and the next determination is performed. If the match rate becomes the same value, the next determination may be performed on the assumption that the predetermined priority is higher.

  Subsequently, based on the determination result, determination is performed by the third-stage image determination units 13 and 14. Specifically, when the image determination unit 12 determines “landscape image”, the image data to be determined is input to the image determination unit 13 to determine the shooting scene. On the other hand, if the image determination unit 12 determines “person image”, the image data to be determined is input to the image determination unit 14 to determine the shooting method.

[Step S22]
When the image determination unit 12 determines “landscape image”, the third-stage image in which three types of learning data of “cityscape image”, “natural image”, and “other landscape image” are constructed by the sample image. The determination target image data is input to the determination unit 13. The image data input to the image determination unit 13 is the same as that input to the image determination unit 10.

  The image determination unit 13 extracts a feature amount from the entire determination target image and compares it with learning data to determine the probability of being a “cityscape image”, “natural image”, and “other landscape image” (respectively The coincidence rate with the learning data is calculated. Then, the image determination unit 13 determines whether the image is a “cityscape image”, “natural image”, or “other landscape image” depending on whether the calculated matching rate is equal to or higher than a preset reference value. Determine whether.

  Note that if the match rate of “cityscape image”, “natural image”, and “other landscape image” is less than the reference value, it is assumed that the match rate is the highest and the result is output. . If the match rate is the same for two types of “cityscape image”, “natural image”, and “other landscape image”, the value is equal to or higher than a preset reference value or less than the reference value, but the remaining If it is higher than one type, it is assumed that the predetermined priority is higher of the two types, and the result may be output.

[Step S23]
On the other hand, if the image determination unit 12 determines “person image”, the third-stage image determination unit 14 in which learning data of “self-portrait image” and “other person image” is constructed by the sample image is displayed. Input image data to be judged. The image data input to the image determination unit 14 is the same as that input to the image determination unit 10.

  The image determination unit 14 extracts a feature amount from the entire determination target image, compares it with learning data, and calculates the probability of “self-portrait image” and “other person image” (with each learning data (Coincidence rate) is calculated. Then, the image determination unit 14 determines whether the image is a “self-portrait image” or an “other person image” depending on whether or not the calculated matching rate is equal to or higher than a preset reference value. .

  Note that if both of the coincidence rates of the “self-portrait image” and the “other person image” are less than the reference value, it is assumed that the coincidence rate is higher, and the result is output. If the match rate becomes the same value, it is assumed that the predetermined priority is higher and the result may be output.

  If machine learning techniques such as deep learning are used, it is possible to compare and determine multiple categories in a single operation, but in that case, the learning time and the time required for the determination process will be long and sufficient. Accurate classification accuracy cannot be obtained. On the other hand, in the image classification apparatus 1 according to the present embodiment, since one image data is determined a plurality of times with different criteria (learning data), each image determination unit can reduce the time required for learning, Processing time is also shortened.

  In addition, since this image classification method is determined for the entire image, it is more accurate than the conventional method for determining by extracting a part from image data and the conventional method for determining using surrounding text data. Various images can be classified. Note that the image classification apparatus according to the present embodiment can classify not only still images but also moving images, and can also be applied to classification of data groups in which still images and moving images are mixed. When classifying moving images, for example, before performing the determination by the first-stage image determination unit, it is possible to perform the determination by performing processing similar to that for a still image by disassembling into a plurality of frames.

  As described in detail above, the image classification device of the present embodiment arranges a plurality of image determination units having a machine learning function in a tree shape, and performs determination based on different learning data for each determination unit. It is possible to classify various images collected from the Internet with high accuracy according to arbitrary categories set according to the purpose. The image classification apparatus according to the present embodiment is configured to first determine the classification of the upper concept and determine the classification of the lower concept as it goes downstream, so that a large amount of image data is processed at high speed. be able to.

  The above-mentioned image classification technology can be used for various purposes such as illegal video detection, news video detection, inappropriate content removal, specific brand analysis, and regional trend analysis, as well as image classification collected from the Internet. It is possible to apply to.

(Second Embodiment)
Next, an image classification system according to the second embodiment of the present invention will be described. FIG. 3 is a conceptual diagram showing a configuration example of the image classification system of the present embodiment. As shown in FIG. 3, the image classification system according to the present embodiment collects images from the Internet 3 and classifies the images into arbitrary sections, and includes one or more image classification apparatuses 1. .

  The image classification apparatus 1 includes a plurality of image determination units having a machine learning function arranged in a tree shape, and performs determination based on different learning data for each determination unit. For example, the first embodiment described above. The image classification device 1 can be used. In the image classification system according to the present embodiment, a plurality of information processing apparatuses are connected to each other, each image processing apparatus is provided with a plurality of image determination units having different learning data, and is configured in a tree shape across the plurality of information processing apparatuses You can also take In this case, a plurality of information processing apparatuses connected to each other function as one image classification apparatus 1.

  In the image classification system of the present embodiment, a plurality of image processing apparatuses 1 in which a plurality of image determination units having a machine learning function are arranged in a tree shape may be connected in parallel. As described above, when a plurality of image classification apparatuses 1 and information processing apparatuses are provided, at least one image classification apparatus 1 or information processing apparatus may be connected to the Internet 3.

  Furthermore, in the image classification system of the present embodiment, an image collection unit that collects image data posted on the social media 4 via the Internet 3 is provided in the image classification device 1 or an information processing device provided separately. It may be. The image collection unit collects specific or unspecified still images and moving images, and stores them in, for example, the image classification device 1 or a storage device provided separately. In addition, the image classification system according to the present embodiment can also allow the images classified by the image classification device 1 to be viewed on the user terminals 2a and 2b via the Internet 3.

  In the image classification system according to the present embodiment, a plurality of image determination units having a machine learning function are arranged in a tree shape, and determination is performed based on different learning data for each determination unit. The image can be classified with high accuracy in any category set according to the purpose.

  The configuration and effects other than those described above in the image classification system of this embodiment are the same as those of the first embodiment described above.

DESCRIPTION OF SYMBOLS 1 Image classification device 2a, 2b User terminal 3 Internet 4 Social media 10-14 Image determination part

Claims (12)

Having a plurality of image determination units having a machine learning function;
The plurality of image determination units are arranged in a tree structure, and are image classification devices that determine the classification of target images based on different learning data.   The image classification device according to claim 1, wherein the learning data is constructed by sample images having different sections for each of the image determination units.   The image classification apparatus according to claim 1, wherein the lower image determination unit of the tree structure determines a lower concept classification than the upper image determination unit of the tree structure.   The image classification device according to claim 1, wherein the machine learning is deep learning.   The image classification apparatus according to claim 1, wherein a convolutional neural network is implemented as the image determination unit.   An image classification system comprising the image classification device according to claim 1. An image classification method for classifying target images using a machine learning method,
The first learning data constructed based on two or more types of image data belonging to the first section is compared with the target image, and the target image is any type of image in the first section. A first determination step of determining whether there is,
The target image is compared with the second learning data constructed based on two or more types of image data belonging to the second category of the lower concept than the first category, and the target image is A second determination step of determining which type of image of the two sections;
The target image is compared with the target image by comparing third learning data constructed based on two or more types of image data belonging to a third category of a lower concept than the first category. A third determination step of determining which type of image in the three sections;
Have
An image classification method for performing the second determination step or the third determination step based on a determination result of the first determination step.   Furthermore, the target image is compared by comparing the target image with fourth learning data constructed based on two or more types of image data belonging to a fourth category of a lower concept than the third category. The image classification method according to claim 7, further comprising a fourth determination step of determining which type of image in the fourth category.   The image classification method according to claim 7 or 8, wherein the determination step performs determination based on a matching rate with the learning data calculated using a feature amount extracted from the target image.   The image classification method according to claim 7, wherein the machine learning is deep learning.   The image classification method according to claim 7, wherein the determination step is executed by a convolutional neural network.   The image classification method according to claim 7, wherein the target image is collected from the Internet.

Images