JP2011100459A - Learning type image classification device and method, and recording medium imaging device that stores processing program for the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a learning type image classification device and method, for reducing manual workload in clipping a target area in learning, or designating the target area, and to provide a recording medium for storing a processing program for the same. <P>SOLUTION: The learning type image classification device clips an area from a learning image according to an area clipping mode indicating how to clip a partial area of the learning image, adds a teacher signal, creates a characteristic vector from a characteristic amount of the clipped area, learns a classification parameter for classifying the character vector, determines a classification category of an area of an input image by use of the classification parameter and a characteristic amount extracted from the input image, and adds a category name thereof to the input image. The area clipping mode is a semi-automatic area clipping mode to clip an area obtained by integrating areas selected by a person from a plurality of areas of the automatically divided learning image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a learning type image classification apparatus and method and a recording medium on which the processing program is recorded, and in particular, automatically classifies images using supervised learning and automatically assigns keywords for image search. In particular, the present invention relates to a learning type image classification apparatus and method for efficiently acquiring teacher data and a recording medium on which the processing program is recorded.

The work of manually sorting and organizing innumerable images in a database, a computer, or on a network, and adding search keywords manually requires a great deal of labor.
Therefore, in order to reduce these work loads, an automatic image classification device and an automatic keyword assignment device for image search have been proposed.

  An example of an apparatus for automatically classifying images is disclosed in JP-A-11-328422 (Patent Document 1), and an apparatus for automatically assigning keywords for image search is disclosed in JP-A-8-249349. (Patent Document 2), JP-A-10-49542 (Patent Document 3), JP-A-10-55366 (Patent Document 4), and the like.

First, Japanese Patent Application Laid-Open No. 11-328422 proposes an apparatus that automatically classifies a natural image and an artificial image, for example, using feature amounts sampled and extracted from the image.
This device may work well for images that can be classified by features that are uniformly distributed throughout the image, but is not sufficient to classify objects that are present locally in the image.

That is, in order to classify objects, it is necessary to devise a method for cutting out only the object area.
JP-A-8-249349, JP-A-10-49542, and JP-A-10-55366 introduce such a device.
In Japanese Patent Application Laid-Open No. 10-49542, first, a region is divided using a change in color or luminance in an image, a feature of each divided region is extracted, a phase relationship between the regions is analyzed, and a dictionary is used. It has been proposed to automatically assign a search keyword to an image by referring to.

  In Japanese Patent Application Laid-Open Nos. 8-249349 and 10-55366, a region is divided by using a change in color or luminance in an image, and the divided regions are integrated to perform a concept of higher layers. We have proposed a device that automatically transitions the image to an image using a more advanced concept as a keyword.

These devices aim to automatically assign keywords based on the characteristics of the objects in the image, and in the process, a knowledge dictionary that the blue sky is located at the top of the image and has a blue color is created. We are using.
Accordingly, there remains a problem of how to set the value of the parameter for classification when it is desired to perform classification that is not in the knowledge dictionary.

What can be dealt with under such circumstances is a classification device having a supervised learning function, and an example of this is JP-A-11-34450 (Patent Document 5).
Hereinafter, the conventional example according to Japanese Patent Application Laid-Open No. 11-34450 will be described in detail for a while.

FIG. 6 is a block diagram showing an example of a classification apparatus having such a conventional supervised learning function.
In FIG. 6, reference numeral 1001 is, for example, an image input / display unit that inputs and displays a wafer image, and reference numeral 1002 is a defect image selection unit that monitors the displayed image and selects a defect image. Reference numeral 1003 is a comparison operation section with a reference image that calculates a difference from the reference image, and reference numeral 1004 is a defect area that performs threshold processing on the calculation result of the comparison operation section 1003 with the reference image and extracts a defect area. An extraction unit, reference numeral 1005 is a teacher signal (category name) assigning unit in which an operator assigns a category name which is a teacher signal to the extracted defect area in the learning step.

Reference numeral 1006 denotes a feature extraction unit that extracts the feature amount of the extracted defect area, and reference numeral 1007 learns classification parameters so that learning data with a teacher signal is classified well in the learning step. This is a classification parameter learning unit.
Further, reference numeral 1008 is a category determination unit that determines which category the defect area belongs to in the classification step, and reference numeral 1009 is a category name addition unit that assigns the determined category name to the defect image.

Next, the operation will be described.
FIGS. 7A and 7B are flowcharts for explaining the operation of a conventional classification apparatus having a supervised learning function.
The processing of this classification device is divided into a learning process and a classification process.
First, the flow of the learning process will be described.

First, in the image input / display unit 1001, for example, a wafer image is input and displayed (step ST1001).
Next, in the defect image selection unit 1002, the operator selects a defect image for teacher data (step ST1002).
Subsequently, the comparison operation unit 1003 with the reference image calculates a difference between the selected defect image and the reference image (step ST1003).

This calculation result is subjected to threshold processing in the defect area extraction unit 1004 to extract a defect area (step ST1004).
In the teacher signal (category name) assigning unit 1005, the operator assigns a category name that is a teacher signal to the defective area (step ST1005).

Next, the feature extraction unit 1006 extracts a feature amount from the extracted defect area (step ST1006).
Subsequently, the classification parameter learning unit 1007 learns the classification parameter so that the learning data with the teacher signal is well classified (step ST1007).

Thus, the learning process is completed.
Next, the classification process will be described.
Since the rough flow of the classification process is almost the same as that of the learning process, only different processing parts will be described.
First, in the learning process, a defect image for teacher data is selected, but in the classification process, all classification target images are targeted (step ST1009).

Of course, it is not necessary to give a teacher signal.
Instead, the category is determined (step ST1013), and a category name is assigned (step ST1014), thereby completing the classification process.
In the above, the example of the classification apparatus provided with the supervised learning function by Unexamined-Japanese-Patent No. 11-34450 was demonstrated.

JP 11-328422 A JP-A-8-249349 Japanese Patent Laid-Open No. 10-49542 Japanese Patent Laid-Open No. 10-55366 JP-A-11-34450

  By the way, in the above-mentioned Japanese Patent Application Laid-Open No. 11-34450, since it is a proposal of a defect classification apparatus in defect inspection of a semiconductor wafer or the like, a defect position is determined from a difference between a defective image and a reference image which is a non-defective product. However, when classifying an object region of an image such as a general photograph without a reference image, it is extremely difficult to detect only the region to be classified.

However, since the automatic image search keyword assigning device and the automatic image classification device are configured as described above, it is a big problem to detect the classification target region.
One method of detecting the classification target area is a method of sorting and integrating each area divided into the knowledge dictionary, but if you want to classify that is not in the knowledge dictionary, The problem of how to set the value of the classification parameter remains.

A learning type classification apparatus can solve this problem, and can also accumulate what has been learned as knowledge.
In the configuration of a classification apparatus having a supervised learning function according to Japanese Patent Application Laid-Open No. 11-34450, it is convenient when examining a variation from a reference image. It is extremely difficult to cut out.

For this reason, it is conceivable to manually cut out the region to be classified during learning, but this is not preferable because it requires a great deal of labor.
The present invention has been made to deal with these situations, and its purpose is to automatically assign a keyword for image search so as to cut out a classification target area at the time of learning or to specify a classification target area. It is an object of the present invention to provide a learning type image classification apparatus and method capable of reducing the human work burden and a recording medium recording the processing program.

  According to the present invention, in order to solve the above-described problem, the above learning learning is performed according to a region extraction mode that indicates how to extract a partial region of the learning image from the learning image input in the learning step. Area extraction executing means for extracting an area from an image, teacher signal providing means for applying a teacher signal to the extracted area, and feature quantity extracting means for extracting a feature quantity from the extracted area A classification parameter determining means for generating a feature vector using the extracted feature quantity and learning and determining a classification parameter for classifying the generated feature vector according to the teacher signal; Classification category determining means for determining a classification category of the region of the input image using the feature amount extracted from the input image and the classification parameter, and the determination Category name assigning means for assigning the category name of the classified category to the input image or the area of the input image, and the area cutout mode automatically uses the change in color or luminance value to The learning image is divided into a plurality of regions, and a region obtained as a result of integrating regions selected by a person from the plurality of divided regions is a semi-automatic region cutout mode for cutting out from the learning image. A feature learning type image classification device is provided.

  Further, according to the present invention, in order to solve the above problem, according to the region extraction mode indicating how to extract a partial region of the learning image from the learning image input in the learning step, A procedure for extracting a region from the learning image, a procedure for giving a teacher signal to the region extracted from the learning image, a procedure for extracting a feature value from the extracted region, and the extraction A feature vector is created using the obtained feature quantity, a procedure for learning and determining a classification parameter for classifying the created feature vector according to the teacher signal, and a feature extracted from the input image input in the classification step The procedure for determining the classification category of the region of the input image using the quantity and the classification parameter, and the category name of the determined classification category are input image or input The region cropping mode automatically divides the learning image into a plurality of regions using a change in color or luminance value, and the divided regions are divided into a plurality of divided regions. A learning type image classification method is provided, which is a semi-automatic region cutout mode in which a region obtained as a result of integrating regions selected by a person from the region is cut out from the learning image.

  Further, according to the present invention, in order to solve the above problem, according to the region extraction mode indicating how to extract a partial region of the learning image from the learning image input in the learning step, A procedure for extracting a region from the learning image, a procedure for giving a teacher signal to the region extracted from the learning image, a procedure for extracting a feature value from the extracted region, and the extraction A feature vector is created using the obtained feature quantity, a procedure for learning and determining a classification parameter for classifying the created feature vector according to the teacher signal, and a feature extracted from the input image input in the classification step The procedure for determining the classification category of the region of the input image using the quantity and the classification parameter, and the category name of the determined classification category are input image or input In the region extraction mode, the learning image is automatically divided into a plurality of regions using a change in color or luminance value, and the divided image is divided into a plurality of regions. Provided is a recording medium on which a learning type image classification program is recorded, which is a semi-automatic region segmentation mode in which a region obtained by integrating regions selected by a person from a plurality of regions is segmented from the learning image. Is done.

  Therefore, according to the present invention, it is possible to automatically assign a keyword for image search, and it is possible to reduce a manual work load such as extraction of a classification target area or instruction of a classification target area at the time of learning. It is possible to provide a learning type image classification apparatus and method and a recording medium on which the processing program is recorded.

FIG. 1 is a block diagram showing a configuration of a learning type image classification apparatus according to the first embodiment of the present invention. FIGS. 2A and 2B are flowcharts for explaining the operation of the learning type classification apparatus according to the first embodiment of the present invention. (A), (b), and (c) of FIG. 3 are for explaining the three modes of the semi-automatic area extraction mode, the learning use area extraction mode, and the interactive area extraction mode as types of the area extraction mode of FIG. FIG. FIG. 4 is a block diagram showing the configuration of the learning use area cutout mode in the area cutout mode in the learning type image classification apparatus according to the second embodiment of the present invention. FIG. 5 is a block diagram showing the configuration of the learning use area cutout mode in the area cutout mode in the learning-type image classification device according to the third embodiment of the present invention. FIG. 6 is a block diagram showing an example of a conventional classification apparatus having a supervised learning function. FIGS. 7A and 7B are flowcharts for explaining the operation of a conventional classification apparatus having a supervised learning function.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram showing a configuration of a first embodiment including a learning type image classification apparatus, a learning type image classification method, and a recording medium storing a learning type image classification program according to the present invention.

  In FIG. 1, reference numeral 101 is an image input / display unit, reference numeral 102 is a classification category content receiving unit that the system accepts what category classification is to be performed, and reference numeral 103 is an area corresponding to a category in the image A mode or region selection mode selection unit for the user or system to select in which mode is selected, reference numerals 103 and 104 are learning image selection units used when performing category learning, and reference numeral 105 is region extraction. This is a region cutout execution unit that cuts out a region corresponding to a category from an image in any mode selected by the mode selection unit 103.

  Reference numeral 106 is a teacher signal (category name) assigning unit that gives a category name as a classification category label as a set to an image or a segmented area, and reference numeral 107 is a feature that quantitatively represents the characteristics of the segmented area. A feature quantity extraction unit for extracting quantities, reference numeral 108 is a classification parameter learning unit that adjusts parameters in the system so that the extracted feature quantity data set can be classified based on a teacher signal.

  Reference numeral 109 is a classification target image input unit that inputs an image to be classified after learning, reference numeral 110 is an area dividing unit that divides an image input after learning into several areas, and reference numeral 111 is learning. A category determining unit that determines a category of an image input later, and a reference numeral 112 is a category name adding unit that assigns the determined category name to an image or a region.

Reference numeral 113 denotes a recording medium on which a learning type image classification program executed by each unit is recorded.
Next, the operation will be described.
FIGS. 2A and 2B illustrate the operation of the first embodiment including a learning type image classification device, a learning type image classification method, and a recording medium in which a learning type image classification program is recorded according to the present invention. It is a flowchart shown for this purpose.

First, the learning process will be described.
First, the image input / display unit 101 inputs an image to this apparatus or system and displays it on the screen (step ST101).
Next, the user decides how to classify the image while viewing the group of images displayed on the screen, and conveys the contents to the system by the classification category content receiving unit 102 (step ST102). .

Specifically, assuming that the system needs some sort of classification needs, for example, it is possible to classify human face images separately from other images, or to treat individual-level face images separately. Let the user choose from among them.
Next, in the region cut-out mode selection unit 103, a person or system selects in what mode the region corresponding to the category is cut out from the image (step ST103).

Examples of the area cutout mode include a semi-automatic area cutout mode, a learning use area cutout mode, and an interactive area cutout mode.
Hereinafter, for a while, these three modes will be described with reference to (a), (b), and (c) of FIG.
First, the semi-automatic area cutout mode shown in FIG.

First, in the area dividing unit A1, the system automatically divides an image into areas using changes in color and luminance values.
Subsequently, in the area designating part A2, several areas divided by the person are clicked.
As a result, in the area integration unit A3, the system consolidates these areas and adopts the integrated area as the area cutout result.

Next, the learning use area cutout mode shown in FIG.
This is a one-touch type region cutout mode that does not make the user aware of region cutout, and there are some restrictions in using this mode.
In other words, this is a method of reducing the area extraction of the category to be classified from now on to the area detection of the already learned category.

  In order to do this, first, in the inter-category region relationship defining unit B1, the relationship between the learned category and the region of the category that is about to be learned is given as knowledge to, for example, a dictionary, and the relationship between the regions matches. In such a case, the extraction of the area of the category that is about to be learned can be reduced to the extraction of the area of the learned category.

  For example, if learning that distinguishes a person's face image from other images has been performed in advance, when performing classification that distinguishes images at the individual level, the region segmentation execution unit B2 based on the learned category performs A face image is cut out using learned parameters, and a category name of an individual name is assigned to the cut out face area.

If this mode can be used, the user only has to collect learning images and give a teacher signal, and does not have to be aware of the troublesome region extraction processing.
Third, the interactive area cutout mode shown in FIG.
This is because, for example, the face segmentation result described in the learning use region segmentation mode is displayed on the region segmentation result display unit C3, and the region detection rate in the segmentation result is determined by the user in the region segmentation result adoption determination unit C4. This is a method of confirming with eyes and selecting images used for learning.

  Returning to FIG. 2, after selecting the region cutout mode, the user selects a learning image in the learning image selection unit 104 (step ST <b> 104), and the region cutout execution unit 105 selects the image for each mode. The classification category area is cut out from (step ST105).

To this, the teacher signal (category name) assigning unit 106 assigns classification category names such as face and Tanaka to the image or the cut-out area as a set (step ST106).
Next, the feature quantity extraction unit 107 extracts features such as color, shape, texture, etc. from the designated area (step ST107).

Subsequently, the classification parameter learning unit 108 creates a feature vector from the extracted feature quantity, and learns and determines the classification parameter so that the feature vector can be classified according to the teacher signal (step ST108).
At this time, processing such as weighting the feature amount in accordance with the degree effective for classification (for example, JP-A-9-101970) may be used.

As a result of classification parameter learning, for example, in a Kohonen-type neural network, the neuron element corresponds to a representative vector in vector quantization, and the representative vector is labeled with a classification category name. It shall be.
Now, learning is completed as described above, and this time, the process proceeds to the classification process.

First, an image group is input or displayed on the image input / display unit 101 (step ST109).
Next, the classification target image input unit 109 inputs an image to be classified (step ST110), and then the region dividing unit 110 divides the region using changes in color and luminance values (step ST111).

Further, the feature extraction unit 107 extracts a feature amount such as a color, a shape, and a texture for each divided region (step ST112).
Next, the classification parameter learning unit 108 first generates a feature vector from a set of feature amounts obtained in each region.

Next, the feature vector of each region is classified using the learned classification parameter, and the category determination unit 111 determines which category the region belongs to (step ST113).
Finally, the category name assigning unit 112 assigns the determined category name to the area or image (step ST114).

The processing flow of the learning process and the subsequent classification process has been described above. It is assumed that the learning type image classification program executed by each unit is recorded in the recording medium 113.
(Second Embodiment)
FIG. 4 is a block diagram showing the configuration of the learning use area cutout mode among the area cutout modes in the learning type image classification apparatus according to the second embodiment.

The configuration of the second embodiment is generally similar to the configuration of the first embodiment.
The main difference from the first embodiment is that the system determines the reliability of area segmentation by the learned category in addition to the inter-category area relationship defining unit B201 and the area segmentation execution unit B202 by the learned category. This is a cutout reliability determination unit B203.

For example, the reliability is digitized and teacher data of an image having a high numerical value is used.
Alternatively, if the numerical value representing the reliability of face detection in the image group selected by the user is low, the mode shifts to the semi-automatic segmenting mode of the category corresponding area in the image.
As a result, it is possible to avoid a situation in which the accuracy of area extraction deteriorates because the work load of area extraction is reduced.
(Third embodiment)
FIG. 5 is a block diagram showing a configuration of the learning use area cutout mode in the area cutout mode in the learning-type image classification device according to the third embodiment.

The configuration of the third embodiment is generally similar to the configuration of the first embodiment.
The main difference from the first embodiment is that, in addition to the inter-category region relationship defining unit B301 and the region segmentation execution unit B302 based on the learned category, the number of regions for counting the number of regions as a result of region segmentation based on the learned category The point is that a counting unit B303 and a learning image candidate selection unit B304 that uses an image whose number of areas is counted as one as a learning image candidate are provided.

  For example, when classifying a person's face level, learning for classifying a person's face image from other landscape images or the like is performed in advance, and the cutout of this face area is used to In the case where a plurality of faces are shown in one image, it is necessary to have the user specify which face region is to be classified.

In the third embodiment, only when the number of human face regions is 1, it is determined as a candidate for a learning image, and it is not necessary for the user to specify which region it is.
As described above, according to each embodiment of the present invention, since the region segmentation mode selection unit is provided, it is possible to greatly reduce the burden of the target region segmentation processing performed manually in the image classification learning process. There is an effect that can be.

  In addition, since the area segmentation mode selection unit is provided, it is possible to select an area segmentation mode that reduces the burden of segmentation for the user and has good segmentation performance. This has the effect of reducing the burden of region segmentation during learning.

Further, since the image classification suitability determination unit for the learned category is provided, it is possible to remove an image with poor suitability from the learning data of the new category classification.
This has the effect of improving the classification accuracy.
In addition, since the area number counting unit and the teacher image candidate selection unit are provided, by selecting only the image having the number of areas of 1 as a candidate for the teacher image, a person can select the target area from a plurality of areas. Thus, there is an effect that it is possible to omit the work of instructing selection and to avoid the problem of waiting until the selection instruction.

  In each of the embodiments described above, the viewpoint is based on the region extraction in the “learning step”, but the present invention is not necessarily limited to only the region extraction in the “learning step”. It is suitable for the learning process, but is not limited to this.

  DESCRIPTION OF SYMBOLS 101 ... Image input / display part, 102 ... Classification category content reception part, 103 ... Selection part of area extraction mode to select, 103, 104 ... Image selection part for learning, 105 ... Area extraction execution part, 106 ... Teacher signal (category) Name) assigning unit 107... Feature quantity extracting unit 108... Classification parameter learning unit 109 109 classification target image input unit 110 .. region segmentation unit 111. Category determining unit 112 .. category name assigning unit 113. Medium, B201, B301 ... inter-category region relationship defining unit, B202, B302 ... region segmentation execution unit based on learned category, B203 ... reliability determination unit, B303 ... region number counting unit, B304 ... learning image candidate selection unit.

Claims (4)

An area extraction execution means for extracting an area from the learning image according to an area extraction mode indicating how to extract a partial area of the learning image from the learning image input in the learning step;
Teacher signal giving means for giving a teacher signal to the clipped region;
A feature amount extraction means for extracting a feature amount from the extracted region;
A classification parameter determining means for generating a feature vector using the extracted feature quantity, learning and determining a classification parameter for classifying the generated feature vector according to the teacher signal, and an input input in the classification step Classification category determining means for determining a classification category of the region of the input image using the feature amount extracted from the image and the classification parameter;
Category name giving means for giving the category name of the determined classification category to the input image or the area of the input image;
Have
The region extraction mode is a result of automatically dividing the learning image into a plurality of regions using a change in color or luminance value, and integrating regions selected by a person from the plurality of divided regions. A learning-type image classification device, which is a semi-automatic region cut-out mode for cutting out the obtained region from the learning image. A procedure for cutting out the region from the learning image according to a region cutting mode indicating how to cut out a partial region of the learning image from the learning image input in the learning step;
A procedure for giving a teacher signal to a region cut out from the learning image,
A procedure for extracting a feature value from the clipped region;
A procedure for creating a feature vector using the extracted feature quantity, learning and determining a classification parameter for classifying the created feature vector according to the teacher signal,
A procedure for determining a classification category of a region of the input image using the feature amount extracted from the input image input in the classification step and the classification parameter;
A procedure for assigning the category name of the determined classification category to the input image or the region of the input image;
Have
The region extraction mode is a result of automatically dividing the learning image into a plurality of regions using a change in color or luminance value, and integrating regions selected by a person from the plurality of divided regions. A learning type image classification method, wherein the obtained region is a semi-automatic region segmentation mode for segmenting from the learning image. A procedure for cutting out the region from the learning image according to a region cutting mode indicating how to cut out a partial region of the learning image from the learning image input in the learning step;
A procedure for giving a teacher signal to a region cut out from the learning image,
A procedure for extracting a feature value from the clipped region;
A procedure for creating a feature vector using the extracted feature quantity, learning and determining a classification parameter for classifying the created feature vector according to the teacher signal,
A procedure for determining a classification category of a region of the input image using the feature amount extracted from the input image input in the classification step and the classification parameter;
A procedure for assigning the category name of the determined classification category to the input image or the region of the input image;
To the computer,
The region extraction mode is a result of automatically dividing the learning image into a plurality of regions using a change in color or luminance value, and integrating regions selected by a person from the plurality of divided regions. A recording medium on which a learning type image classification program is recorded, which is a semi-automatic region cutout mode for cutting out the obtained region from the learning image. In the semi-automatic area cutout mode according to claim 1,
The region cutout execution unit is configured to specify a region to be classified among a plurality of divided regions, and cut out a region obtained by integrating the designated regions from the learning image.

Images