JP7350208B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, a learning device, a focus control device, an exposure control device, an image processing method, a learning method, and a program.

In recent years, research on segmenting images into regions has also been widely conducted. For example, a person area, a car area, a road area, a building area, a sky area, or the like can be extracted from an image. This is called semantic segmentation, and the segmentation results can be applied to image correction or scene interpretation depending on the type of subject.

As a semantic region division method, there is a method in which an image is divided into several regions in advance and each divided region is classified into classes. For example, an image can be divided into a plurality of rectangular blocks and each block can be classified into classes. As a method for classifying images, classification using deep learning has been widely studied, as described in Non-Patent Document 1. Alternatively, an image may be divided into irregularly shaped small regions (superpixels) using the method described in Non-Patent Document 2, and the regions may be classified into classes using the feature amount of the region and the context feature amount around the region. can. For class classification, an estimator trained using learning images can be used.

In recent years, region segmentation using deep learning has also been studied. Non-Patent Document 3 uses the intermediate layer output of a CNN (Convolutional Neural Network) as a feature quantity, and integrates the class determination results for each pixel based on a plurality of intermediate layer features. With this method, class determination can be performed directly for each pixel without using the results of subregion division.

A. Krizhevsky et al. "ImageNet Classification with Deep Convolutional Neural Networks", Proc. Advances in Neural Information Processing Systems 25 (NIPS 2012). R. Achanta et al. "SLIC Superpixels", EPFL Technical Report 149300, 2010. J. Long et al. "Fully Convolutional Networks for Semantic Segmentation", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015.

According to the conventional method, each small area on an image could be classified into classes based on the type of subject. For example, based on the feature amount of each region, it was possible to determine whether the region represents the sky or the foreground (other than the sky). On the other hand, it has been difficult to make appropriate determinations in areas where different types of subjects coexist. For example, when determining multiple areas where the sky can be seen between the branches of many trees, it may be determined that all areas are in the foreground because the textures are similar, or all areas are determined to be empty. There was a high possibility that it would be determined that

An object of the present invention is to classify each region of an image into classes so as to improve the accuracy of processing using classification results.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention has the following configuration. That is,
an acquisition means for acquiring an input image;
extraction means for extracting features from the input image;
an estimating means for estimating information corresponding to the area of a region belonging to a specific class in the input image when the feature extracted by the extracting means is input;
The estimation means is characterized in that the estimation is performed using parameters learned using learning images .

According to the present invention, each region of an image can be classified into classes so that the accuracy of processing using classification results can be improved.

FIG. 1 is a block diagram illustrating a configuration example of a device according to each embodiment. 5 is a flowchart illustrating processing according to each embodiment. FIG. 3 is a diagram illustrating learning images and class labels. FIG. 6 is a diagram illustrating class labels and mixed states when two classes are identified. FIG. 6 is a diagram illustrating class labels and mixed states when identifying three classes. FIG. 3 is a diagram illustrating a mixed state when three classes are identified. An explanatory diagram of mapping of a mixed state. The figure which shows the example of a structure of the learning part or estimation part based on each embodiment. FIG. 3 is a diagram illustrating class label detailing in a mixed area. 1 is a diagram illustrating an example of the configuration of a computer that can implement each embodiment; FIG.

According to an embodiment of the present invention, it is possible to estimate how a plurality of classes are mixed (hereinafter referred to as a mixed state) in a predetermined region that is a classification unit on an input image. Below, an image within a region to be estimated may be referred to as a target image. More specifically, according to one embodiment of the present invention, a mixed state of regions having mutually different attributes in a target image is determined. The area of each attribute is an area occupied by subjects belonging to the same class. That is, one of the regions with this attribute is a region of a subject belonging to a specific class, and another one of the regions with this attribute is a region of a subject belonging to a class different from the specific class.

According to one embodiment, for example, for an area where the sky is visible between many tree branches (foreground), the mixing state of the foreground part and the sky part (for example, area ratio, edge area, or placement pattern) is determined. It can be estimated. By using not only the class information of each region (for example, information indicating whether it is a foreground region or a sky region) that can be obtained with conventional methods, but also information indicating such a mixed state, it is possible to later apply information to the image. The accuracy of the processing performed can be improved. Specific examples will be described in detail in each embodiment.

Embodiments of the present invention will be described below based on the drawings. However, the scope of the present invention is not limited to the following embodiments. In the following embodiments, each processing unit shown in FIG. 1 and the like may be realized by a computer or by dedicated hardware.

FIG. 10 is a diagram showing the basic configuration of a computer that can implement each embodiment. In FIG. 10, a processor 101 is, for example, a CPU, and controls the operation of the entire computer. The memory 102 is, for example, a RAM, and temporarily stores programs, data, and the like. The computer-readable storage medium 103 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. for a long period of time. In this embodiment, a program stored in the storage medium 103 that implements the functions of each part is read into the memory 102. The functions of each part are realized by the processor 101 operating according to the program on the memory 102.

In FIG. 10, an input interface 104 is an interface for acquiring information from an external device. Further, the output interface 105 is an interface for outputting information to an external device. A bus 106 connects the above-mentioned units and enables data exchange.

[Embodiment 1]
The basic configurations of an image processing device and a learning device according to the first embodiment will be explained along FIGS. 1(A) and 1(B).

First, the outline of the device configuration of the learning device will be explained according to FIG. 1(A). In this embodiment, the learning device generates an estimator that is used when an image processing device (described later) performs a process of recognizing a mixed state, from a training image prepared in advance. Details of the learning process will be described later. The learning data storage unit 5100 stores learning data prepared in advance. The learning data includes learning images and teacher information. The data acquisition unit 2100 acquires learning images and teacher information from the learning data storage unit 5100. The learning unit 2200 uses the feature extracting unit 610 to extract the feature amount of the identification image used for learning the estimator, which is located in a predetermined region of the learning image. Further, the learning unit 2200 performs learning of an estimator that estimates a mixed state from the feature amount using a combination of the feature amount of the identification image and the teacher information. For example, the learning unit 2200 can perform learning of an estimator that outputs information indicating a mixed state when a feature quantity is input. Here, the teacher information is information indicating a mixing state between regions of mutually different attributes in the identification image. The estimator obtained through learning is stored in the estimator storage unit 5200. Specifically, the estimator storage unit 5200 can store estimator parameters determined through learning.

Next, the outline of the device configuration of the image processing device will be explained according to FIG. 1(B). In this embodiment, the image processing device performs a process of estimating a mixing state in an unknown input image. The details of the processing will be described later. The image acquisition unit 1100 acquires an input image. The estimation unit 1200 uses the feature extraction unit 610 to extract a feature amount from a target image that is a mixed state identification target in a predetermined region of the input image. The feature extraction unit 610 used by the estimation unit 1200 can operate in the same manner as the feature extraction unit 610 used by the learning unit 2200. Furthermore, the estimation unit 1200 estimates a mixed state of regions having mutually different attributes in the target image based on the feature amount. For example, the estimation unit 1200 loads the estimator 620 that has been trained in advance from the estimator storage unit 5200, and inputs feature amounts to the estimator to obtain a mixture between regions of different attributes in the target image. Information indicating the state is output to the output unit 1300. The estimator 620 may be obtained through learning by the learning unit 2200. The output unit 1300 outputs the estimation result by the estimation unit 1200.

The data acquisition unit 2100 and the learning unit 2200 of the learning device may be realized on the same computer, may be configured as independent modules, or may be implemented as a program running on the computer. The learning data storage unit 5100 and estimator storage unit 5200 of the learning device can be realized using storage inside or outside the computer.

The image acquisition unit 1100 and the estimation unit 1200 of the image processing device may be realized on the same computer, may be configured as independent modules, or may be implemented as a program running on the computer. . Further, these may be implemented as a circuit or a program inside a photographing device such as a camera.

The image processing device may be realized on the same computer as the learning device, or may be realized on a separate computer. The estimator storage unit 5200 included in the learning device and the image processing device may be the same storage or different storages. When using a different storage, the estimator stored in the estimator storage unit 5200 by the learning device can be copied or moved to the estimator storage unit 5200 included in the image processing device.

The processing according to this embodiment will be described in detail below. First, the processing during learning performed by the learning device will be described according to the flow shown in FIG. 2(A). In S2100, the data acquisition unit 2100 acquires the learning image and the teacher information of the mixed state from the learning data storage unit 5100 as learning data.

The learning data storage unit 5100 stores in advance a plurality of learning images and mixed state teacher information. A training image refers to an image used for learning the estimator. The learning image may be, for example, image data taken with a digital camera or the like. The format of the image data is not particularly limited, and may be, for example, JPEG, PNG, or BMP. In the following, the number of prepared learning images is N, and the n-th learning image is expressed as I _n (n=1...N).

The mixed state teacher information indicates the mixed state in a predetermined region of the learning image. This teacher information is prepared in advance and can be created, for example, by a person while looking at the learning images. In this embodiment, a plurality of regions serving as identification units are set in the learning image, and teacher information is prepared for each region. Hereinafter, an image of a predetermined area in a learning image, which serves as one identification unit, will be referred to as an identification image.

The method of setting the area is not particularly limited. For example, multiple regions can be set in the input image according to a predetermined region setting pattern. As a specific example, a learning image can be divided into a plurality of rectangular areas of a predetermined size (for example, 16×16 pixels), and each rectangular area can be treated as an identification unit. Further, the small area obtained by the method described in Non-Patent Document 2 can be treated as an identification unit area. On the other hand, a rectangular area of a predetermined size may be set only in a part of the learning image. Note that the learning data storage unit 5100 may store identification images of a predetermined size as learning data.

The mixed state indicated by the teacher information will be explained below. Objects on images can be classified into multiple classes. FIG. 3 shows an example of such class classification. FIG. 3A shows an example of a learning image 500. The learning image 500 includes the sky, people, and plants, and each can be classified into different classes. That is, as shown in FIG. 3B, pixels included in area 541 are assigned a class label of "empty," pixels included in area 542 are assigned a class label of "person," and pixels included in area 543 are assigned a class label of "person." can each be given a class label of ``plant''.

There are various definitions of classes and class labels, and the method of class classification is not particularly limited. In the example of FIG. 3, class classification was performed according to the type of subject. Examples of other class labels include skin or hair regions, animals such as dogs or cats, and man-made objects such as cars or buildings. A class label indicating a specific object, such as part A or part B used in a factory, can also be used. On the other hand, each pixel may be classified into a main subject area and a background area. Further, classification may be performed according to differences in surface properties such as glossy surfaces or matte surfaces, or differences in materials such as metal surfaces or plastic surfaces. In the following, it is assumed that there are M types of classes in total.

The mixed state of classes is a mixed state between regions of mutually different attributes in the target image. The area of each attribute is an area occupied by subjects belonging to the same specific class. One of the regions with mutually different attributes is a region of a subject belonging to a specific class, and the other is a region of a subject belonging to a class different from the specific class. Below, an attribute area occupied by a subject belonging to a certain class may be simply referred to as an area belonging to that class. Further, the class of the subject reflected in each pixel may be referred to as the attribute or class of the pixel below.

There are various possible definitions of mixed state. In this embodiment, the mixed state is expressed numerically as follows. In one embodiment, the information indicating the mixed state is information determined depending on the distribution of attribute regions in the target image. For example, the information indicating the mixed state is information indicating the ratio of each attribute area in the target image. As a specific example, the information indicating the mixed state may be the area ratio of regions belonging to each class in the target image. A case where there are two classes, "empty" and "non-empty", will be described with reference to the example of FIG. 4. FIG. 4 shows a learning image 510 and its class label 520. The class label 520 represents pixels whose class is "empty" in white, and pixels whose class is "non-empty" in black. FIG. 4 shows an enlarged view (526) of an area 525 on the class label 520 corresponding to the identification image 515 in the learning image 510. The enlarged view (526) shows a non-empty area 511 and an empty area 522. At this time, the mixed state of the identification image 515 can be expressed by the area ratio r between the sky area and the non-sky area in the corresponding area 525. For example, in a rectangular area of 16×16 pixels, if the number of empty area pixels is 192 pixels and the number of non-empty area pixels is 64 pixels, r=192/256=0.75.

Although the above example describes the area ratio of two classes, it is also possible to represent the area ratio of three or more classes. FIG. 5 shows a learning image 530, an identification image 535 in the learning image 530, and a class label 536 for the identification image 535. In this example, each pixel of the learning image 530 is classified into three classes: "sky", "plant", and "artificial object". The mixed state in this case can be determined according to the area ratio of the plant area 531, the sky area 532, and the artificial object area 533. As an example, in this case, points 545 indicating the area ratio can be plotted on the simplex 540 (triangle in the case of FIG. 5) in the coordinate space shown in FIG. 6(A) according to the area ratio of each class. can. This point 545 can be uniquely expressed using the internal division ratios t ₁ and t ₂ that internally divide the two sides of the simplex, so the area ratio at this time is expressed as r = (t ₁ , t ₂ ). It can be expressed as a vector. This is the same in general M dimensions, so the area ratio when the number of classes is M is uniquely expressed by an M-1 dimension vector r = (t ₁ , t ₂ ,...t _M-1 ). be able to. Note that the area ratio in the case of two classes as described above is the same as in the case of setting M=2 in this generalized format.

Furthermore, the area ratio of the M class expressed as above may be handled by being mapped to a lower-order space. For example, the area ratio in the identification image can be plotted in an M-dimensional space and mapped to a lower-order space using SOM (Self-Organizing Map) or LLE (Locally Linear Embedding). FIG. 6(B) shows an example in which the space of the above three classes of mixture ratios is quantized using a one-dimensional SOM 550. 551 represents the start node of the SOM 550, and 552 represents the end node of the SOM 550. FIG. 6(C) is obtained by mapping these onto a one-dimensional space. When the position of the start node is set to 0, the position of the end node is set to 1, and the nodes are arranged evenly, a point 545 indicating the area ratio can be expressed by a scalar value representing the position on the map. For example, point 545 indicating the area ratio shown in FIG. 6(B) can be approximated as point 546 on FIG. 6(C), and the location of point 546 on the map (in this example r=0.37 ) can be used to express the area ratio. The number of dimensions after mapping is not limited to one dimension as shown in FIG. r can be expressed as an R-dimensional vector. For example, when the number of classes M=5 and the area ratio space is quantized using a two-dimensional SOM, r can be expressed as a two-dimensional vector.

Furthermore, the area ratio of the M class may be expressed as a composite vector of a plurality of basis vectors. For example, class area ratios obtained from various identification images can be decomposed into a plurality of basis vectors using principal component analysis, sparse coding, etc., and this can be approximated by a small number of vectors with a large degree of contribution. In this case, the area ratio in the area ratio space can be expressed as a composite vector of these basis vectors, and the area ratio can be expressed using the weighting coefficient for each basis vector at that time.

As another example, the information indicating the mixed state may be information regarding a boundary between regions of mutually different attributes in the target image, for example, information representing the ratio of pixels representing this boundary in the target image. As an example, edge detection is performed on a binary image indicating the class of each pixel (e.g. sky area or non-sky area), the number of obtained edge pixels is counted, and the number of pixels in a predetermined area is compared with the number of edge pixels. The ratio e can be used to represent the mixed state. FIG. 4 shows an edge detection result 527 for a class label 526, and a detected edge pixel 523 is represented. When the count result of edge pixels in a 16×16 pixel rectangular area is 64, the edge pixel ratio can be expressed as e=64/256=0.25.

As yet another example, the information indicating the mixed state may be information indicating the arrangement of attribute regions in the target image. For example, a mixed state can be expressed according to the arrangement pattern of pixels of each class within a predetermined area. When the number of classes is M and the number of pixels in a predetermined region is K, the class of each pixel in the predetermined region can be represented by an M×K-dimensional binary vector. For example, if two classes "empty" and "non-empty" are defined and the size of the predetermined area is 16 x 16 pixels, the class label arrangement pattern within the predetermined area is set to a 2 x 16 x 16 = 512-dimensional It can be expressed as a value vector. By plotting various binary vectors obtained from the identification image in this way on a vector space and quantizing them using SOM, LLE, etc., it is possible to express the class label arrangement pattern in a predetermined region as a vector p. can. Alternatively, a method may be used in which various binary vectors obtained from identification images are expressed as basis vectors using principal component analysis, sparse coding, or the like.

FIG. 7 shows a map 900 obtained by mapping the class label arrangement pattern in a predetermined area using a two-dimensional SOM. In the map 900, each rectangle is a node of the SOM and represents a quantized class label arrangement pattern. Due to the characteristics of SOM, similar patterns are placed close to each other on the map. The class label arrangement pattern in the identification image can be represented by position coordinates p on the map based on the proximity of each node to the pattern. For example, in the example of the two-dimensional SOM shown in FIG. 7, the position on the map can be represented by a two-dimensional vector p=(p ₁ , p ₂ ).

Thus, a mixed state can be expressed in various ways. A mixed state may be expressed using any one of these expressions. For example, if the mixed state is expressed only by the area ratio, it can be defined as C=r, if it is expressed only by the edge pixel ratio, it can be expressed as C=e, and if it is expressed only by the class label arrangement pattern, it can be defined as C=r. It is sufficient to define C=p. Further, a mixed state may be expressed by combining a plurality of expressions. For example, a mixed state may be defined as a combination of area ratio and edge pixel rate C = (r, e), or a combination of area ratio, edge pixel rate, and class label arrangement pattern C = (r, e, p). In the present invention, the method of expressing the mixed state is not particularly limited.

As mentioned above, the mixed state C can be expressed as a vector represented by one or more numerical values. That is, it can be said that the information representing the mixed state obtained in one embodiment is a feature amount representing the mixed state in a predetermined area. Let L be the number of dimensions of the vector representing the mixed state C. In the following, the mixed state vector in a predetermined area i on the image I _n is expressed as C _ni , and the l (l=1, ..., L)th element of the mixed state vector C _ni is expressed as c (n, i, l). Expressed as Note that the mixed state may indicate not only how the pixels of each class are mixed in the predetermined region, but also that the predetermined region is composed of pixels of one specific class.

In this embodiment, it is assumed that a class label is given to each pixel of each learning image as shown in FIG. 3(B). Then, based on this class label, for each identified image obtained from the learning images, the mixture state C expressed as a scalar value or a vector value as described above is calculated in advance as teacher information, and is stored in the learning data storage unit 5100. It is assumed that this is stored in advance. However, the data acquisition unit 2100 acquires information indicating the attributes of each pixel of the identification image, and generates information indicating the mixed state using the information indicating the attributes of each pixel, thereby acquiring the teacher information. good. For example, the data acquisition unit 2100 can calculate the mixture state C of each identified image as described above based on the class label of each pixel of the learning image stored in the learning data storage unit 5100. Furthermore, it is not essential that each pixel of each learning image be given a class label as shown in FIG. 3(B). For example, the mixture state of this identification image input by the worker while looking at the identification image obtained from the learning image, or the mixture of this identification image automatically calculated based on information (edge information, etc.) input by the worker. The state may be stored in the learning data storage unit 5100 in advance.

In step S2200, the learning unit 2200 acquires the identification image and the teacher information of the mixture state from the data acquisition unit 2100, and performs learning of the estimator for estimating the mixture state. Below, a case will be described in which a CNN (Convolutional Neural Network) is used as an estimator. A conventionally known configuration can be used as the CNN configuration. Typically, CNNs perform recognition tasks by gradually combining local features of the input signal by repeating convolution and pooling layers to obtain features that are robust to deformations and misalignments. It is a neural network.

An example of estimation processing using CNN will be described with reference to FIG. 8(A). The learning unit 2200 uses the feature extraction unit 610 to extract the feature amount of the identification image used for learning the estimator. Further, the learning unit 2200 performs learning of an estimator that outputs information indicating a mixed state when a feature amount is input, using a combination of the feature amount of the identification image and the teacher information. FIG. 8A shows an example of a CNN that the learning unit 2200 can use for processing. FIG. 8A shows a portion corresponding to the processing performed by the feature extraction unit 610, which corresponds to the convolution layer of the CNN that performs feature extraction. Further, FIG. 8A shows a portion corresponding to an estimator 620 that performs learning, and this corresponds to a fully connected layer of a CNN that performs pattern estimation.

The convolution layer has an input layer 611 that receives convolution calculation results at each position of the identification image 630, which is a partial image of the learning image, as a signal. The signal from the input layer 611 is sent to the final layer 615 via a plurality of intermediate layers 612 and 613, in which a convolution layer and a pooling layer are arranged, and convolution operations and signal selection by pooling are repeated. The output signal from the final layer 615 of the feature extractor 610 is sent to an estimator 620. In the following, the output signal of the feature extraction unit 610 is assumed to be X. In the fully connected layer, the elements of each layer are fully connected to the previous and succeeding layers, and the signal input from the feature extraction unit 610 is sent to the output layer 640 through a sum-of-products operation using weighting coefficients. The output layer 640 has the same number of output elements as the number of dimensions L of the mixed state vector C.

When learning the estimator, the learning unit 2200 uses the value of the output signal obtained in the output layer 640 as the teacher information when inputting the identification image obtained from the predetermined region i of the learning image _In to the CNN. Compare with. Here, the feature amount obtained by inputting a predetermined region i of the learning image I _n to the feature extraction unit 610 is X ⁿ _i , and the l-th element in the output layer 640 obtained as a result of inputting this to the estimator 620 Let the output signal of y _l (X n i ) be y l (X ⁿ _i ). Further, the teacher signal at the l-th output element in the output layer 640 is represented by the l-th element c(n, i, l) of the mixed state C _ni . In this case, the error between the output signal and the teacher information is calculated as follows.

CNN learning can be performed by sequentially backpropagating the errors obtained in this way from the output layer to the input layer using the error backpropagation method. For example, the weighting coefficients of each layer in the CNN can be updated using stochastic gradient descent or the like. As the initial value of the CNN weighting coefficient, a random value may be used, or a weighting coefficient obtained by learning regarding a certain task may be used. For example, an image classification task uses a training image in which a class label is assigned to each image, but a region segmentation task uses a training image in which a class label is assigned to each pixel. The burden on humans to prepare is heavy. On the other hand, training images for image classification tasks are publicly available and can be easily obtained. For example, the ILSVRC (ImageNet Large-scale Visual Recognition Challenge) has released 1.2 million learning images for image classification tasks. Therefore, even if CNN is trained for such an image classification task and the weighting coefficients obtained by this learning are used as initial values, learning for a mixed state estimation task as in this embodiment is performed. good.

Although an estimator using CNN has been described here, the configuration of the estimator is not particularly limited. FIG. 8B shows another example of a configuration that can be used by the estimation unit 1200 for processing. FIG. 8B shows a portion corresponding to the processing performed by the feature extraction unit 610 and a portion corresponding to the processing performed by the estimator 650. The estimator 650 provides a regression value for one feature obtained by connecting the output signals of each layer in the feature extractor 610. Examples of methods used by the estimator 650 include SVR (Support Vector Regression) and logistic regression, but are not particularly limited. Then, the regression function used by this estimator 650 can be trained using the learning images. For example, the regression function can be trained to minimize the error function based on the error between the output signal and the teacher information. Alternatively, the CNN may be trained in advance using the configuration shown in FIG. 8A, and then only the estimator 650 may be trained using feature amounts based on the output signals of each layer of the CNN. Here, if the estimator 650 is configured with a fully connected multilayer neural network, the learning of the CNN and the estimator 650 can be performed simultaneously using the error backpropagation method, similar to the configuration of FIG. 8(A). .

Further, the feature extraction unit 610 can extract the feature amount using another feature extraction method such as HOG or SIFT. Additionally, the estimator can perform mixed state estimation using a discriminant function such as SVR, logistic regression, or a multilayer neural network. Thus, in one embodiment, any combination of feature extraction techniques and estimation techniques may be used. Even in such a case, the estimator can be trained according to the conventional method. The estimator parameters obtained through learning in step S2200 are stored in estimator storage unit 5200.

A method for identifying a mixed state of an input image using the estimator trained in this manner will be described with reference to the flowchart in FIG. 2(B). In S1100, the image acquisition unit 1100 acquires an input image to be classified as a mixed state. The image acquisition unit 1100 can also acquire undeveloped image data obtained from an imaging device.

In the following, an image whose mixed state is to be estimated in a predetermined region of the input image will be referred to as a target image. The image acquisition unit 1100 can set a plurality of regions in the input image according to a predetermined region setting pattern. A partial image of the input image included in each of the set areas becomes a target image. The target image is a partial image of a predetermined size according to the identification unit, and the setting method thereof is not particularly limited. For example, as in the case of learning, the input image can be divided into a plurality of rectangular regions of a predetermined size (for example, 16×16 pixels), and determination can be made for a plurality of target images in each rectangular region. On the other hand, the determination may be made on a target image located in a partial area of the input image.

In step S1200, the estimation unit 1200 uses the feature extraction unit 610 to extract a feature amount from the target image in a predetermined region of the input image obtained in S1100. Furthermore, the estimating unit 1200 reads the learned estimator 620 from the estimator storage unit 5200 and inputs the feature amount to the estimator 620, thereby obtaining information indicating the mixing state between regions with different attributes in the target image. generate. In this way, the estimation unit 1200 estimates the mixed state of the target image among the input images acquired in step S1100. FIG. 8(A) shows an example of a CNN that the estimation unit 1200 can use for processing. FIG. 8A shows a part corresponding to the processing performed by the feature extraction unit 610, in which signals in a predetermined area in the input image are sequentially propagated to each layer, and the feature amount X _i of the target image is Extracted. Further, FIG. 8A shows a part corresponding to the estimator 620, and here, from the obtained feature amount X _i , an output signal at the output element 621 assigned to each element of the mixed state vector is generated. The value of the output signal of each element l becomes the value of each element y _l (X _i ) of the mixed state vector.

In step S1300, output unit 1300 outputs the estimation result obtained in step S1200. The processing performed by the output unit 1300 depends on how the identification result is used and is not particularly limited. An example of processing using information indicating a mixed state is given below.

For example, image processing for each region of the input image can be changed depending on the mixing state in that region. In this case, the output unit 1300 can output the mixed state of each region to the image correction application.

Furthermore, as another example, it is also possible to perform focus control of the camera according to the mixing state. For example, a focus control device for an imaging device including a plurality of distance measurement points can include an acquisition section and a control section. The acquisition unit acquires information indicating an area ratio of a region of a specific attribute to the region corresponding to each of the plurality of ranging points in the image obtained by the imaging device. Then, the control unit weights the plurality of distance measurement points according to the area ratio and performs focus control of the imaging device. More specifically, when performing multi-point distance measurement AF, it is possible to increase the weight of a distance measurement point that has more subject components to be focused. For example, when performing focus control that places emphasis on the foreground, you can increase the weight of the focusing point that has more foreground components, and when performing focus control that places emphasis on a specific subject, the weight of the focusing point that has more foreground components can be increased. The weight of the distance measurement point can be increased. Such a focus control device may acquire information indicating a mixed state from the above information processing device, may have each of the above configurations included in the above information processing device, or may have the above configurations included in the above information processing device. Information indicating a mixed state generated by a method different from the method may be obtained.

As yet another example, camera exposure control may be performed depending on the mixing state. For example, an exposure control device for an imaging device can include an acquisition section, a calculation section, a selection section, and a control section. The acquisition unit can acquire information indicating an area ratio of a region of a specific attribute to the image obtained by the imaging device and each region of the image. The calculation unit can calculate the area ratio of a region of a specific attribute to the entire image. The selection unit can select an exposure control algorithm according to the calculated area ratio. The control unit can control the exposure of the imaging device using the selected exposure control algorithm. More specifically, when performing different exposure controls depending on the area of the sky in the field of view, the area of the sky can be calculated based on the mixed state. In this case, as in the conventional technology, the area of the sky is reduced by determining that most of the area where the sky and branches are mixed is in the foreground or determining that most of the area is the sky. It can be expected that the possibility of the value differing greatly from the actual value can be reduced.

Although the case where a still image is used as a learning image and an input image has been described here, a moving image can also be used as a learning image and an input image. In this case, the definition of mixed state is extended in the time direction. For example, when a 16×16 pixel predetermined area and 5 frames are used as identification units, a mixed state can be defined for 16×16×5 voxels. For example, by extending the above example of representing a mixed state using an area ratio, it is possible to represent a mixed state using a volume ratio.

In this embodiment, the input image (and learning image) was divided into multiple regions each including multiple pixels, and the mixed state within this region was estimated. According to such processing, the number of times of estimation processing is reduced compared to the case where the class is estimated for each of all pixels, so it can be expected that the processing speed will be increased. On the other hand, it is also possible to estimate the mixing state for each pixel of the input image. That is, one pixel may contain a plurality of subjects belonging to different classes, and it is also possible to estimate the mixed state of subjects of each class in the subject area corresponding to this one pixel.

In this embodiment, the information indicating the mixed state is obtained as a scalar value or a vector composed of a plurality of scalar values. On the other hand, the information indicating the mixed state may be information selected from three or more values. For example, information indicating a mixed state of classes "empty" and "non-empty" in a predetermined area may include a value indicating that the predetermined area is composed of "empty" and a value indicating that the predetermined area is composed of "non-empty". or a value indicating that a predetermined region is a mixture of "empty" and "non-empty." Information indicating such a mixed state can also be used in the processing example described above and in the fourth and fifth embodiments described later.

[Embodiment 2]
The first embodiment has been described on the assumption that a class label is set for each pixel of a learning image. However, setting a class label for each pixel takes time. In the second embodiment, a method for reducing the user's work of inputting class labels for learning images will be described. In this embodiment, the data acquisition unit 2100 automatically calculates a class label for each pixel based on the class label input for each region of the learning image.

The basic configuration of the learning device in this embodiment will be described below with reference to FIG. 1(C). The configuration of the image processing apparatus in this embodiment is the same as that in Embodiment 1, and the description thereof will be omitted. In this embodiment, the learning data storage unit 5100 stores, in addition to the identification image, an area of the first attribute, an area of the second attribute, and an area of the first attribute and the second attribute in the identification image. Contains information indicating a mixed area where . For example, the learning data storage unit 5100 stores learning data including a learning image and a class label assigned to each region on the learning image. Here, an area in which a plurality of classes coexist is given a class label indicating that it is a mixed area.

The data acquisition unit 2100 reads learning data from the learning data storage unit 5100. That is, in addition to the identification image, the data acquisition unit 2100 detects a region of the first attribute, a region of the second attribute, and a region of the first attribute and the second attribute in the identification image. Obtain information indicating the mixed area in which the

The detailing unit 2300 determines the attribute of each pixel in the mixed area based on the pixel value of the pixel included in the first attribute area and the pixel value of the pixel included in the second attribute area. For example, the detailing unit 2300 calculates teacher information indicating a mixed state for an area that is given a class label indicating that it is a mixed area. Details will be described later. The learning unit 2200 performs learning of the estimator similarly to the first embodiment using the learning image and the teacher information of the mixed state.

The flow of processing performed by the learning device in this embodiment will be described with reference to FIG. 2(C). In step S2100, the data acquisition unit 2100 reads the learning image and class label data as learning data from the learning data storage unit 5100. A plurality of learning images and class label data for each are prepared in advance in the learning data storage unit 5100.

Here, class label data in this embodiment will be explained. FIG. 9(A) shows a learning image 500, and FIG. 9(B) shows class label data 400 regarding the learning image 500. In this example, the learning image 500 is composed of a sky region 410, a non-sky region 420, and a mixed region 430, and the pixels in each region have classes of "sky", "non-sky", and "mixed", respectively. It is attached as a label. In this way, the learning image 500 has a single class region and a region where multiple classes coexist.

These class labels can be input in advance by a human using a tool or the like. For example, an operator can determine sky and non-sky regions of the training images. At this time, in areas where the branches of the tree in the foreground are fine and complicated, accurately separating the sky area from the non-sky area requires a large workload on the operator. Therefore, the operator can give a class label of "mixed" to an area where a plurality of classes coexist in this way.

Here, a region in which "empty" and "non-empty" are mixed has been described, but as described in the first embodiment, the class definition is not limited to such a region. Furthermore, when there are three or more classes, it is possible to define as many types of mixed areas as there are combinations of classes. For example, as shown in Figure 5, if three classes are defined: "Sky", "Plant", and "Artificial Object", then "Sky and Plant Mixed Area" and "Sky and Artificial Mixed Area" are defined. Four types of mixed area classes can be defined: , "mixed area of plants and artificial objects", and "mixed area of sky, plants, and artificial objects". In the following, an example will be explained in which two classes, "empty" and "non-empty", are defined.

In step S2300, the detailing unit 2300 refines the class label regarding the mixed area. Specifically, the detailing unit 2300 sets a class label for each pixel in the mixed area. Here, the detailing unit 2300 determines the attribute of each pixel in the mixed area based on the pixel value of the pixel included in the area of the first attribute and the pixel value of the pixel included in the area of the second attribute. do. For example, the detailing unit 2300 can determine the class label of the mixed area with reference to the color information of each class. As a specific example, the detailing unit 2300 extracts the RGB values of each pixel for each of the sky region and the non-sky region in the learning image _In , and plots them in the RGB color space. Sky areas and non-sky areas other than the mixed area are shown in the training data. The detailing unit 2300 then estimates a Gaussian mixture distribution for each of the sky region and the non-sky region. Then, for each pixel in the mixed region, the likelihood of the sky region can be determined based on its RGB values and the Gaussian mixture distribution of the sky region, and the likelihood of the sky region can be determined for each pixel of the mixed region based on its RGB values and the Gaussian mixture distribution of the non-sky region. It is possible to estimate the likelihood in a region. The detailing unit 2300 can then assign a class label of "empty" or "non-empty" with a higher likelihood to the pixel. In this way, the detailing unit 2300 can refine the class labels in the mixed area. FIG. 9C shows the class label data 450 detailed in this manner, in which detailed empty areas 460 and non-empty areas 470 are represented.

Based on the class label data detailed in this manner, the detailing unit 2300 calculates teacher information representing a mixed state for the identification area serving as the identification unit. The definition and calculation method of the identification area and the teacher information indicating the mixed state are as described in detail in Embodiment 1, so detailed explanation will be omitted here. Note that it is not essential that the detailing unit 2300 refines the class label. For example, it is possible to estimate the mixing state in the mixed area based on the RGB value distribution of pixels in the mixed area in the identification area and the mixed Gaussian distribution of the sky area and non-sky area, and based on this, Teacher information representing the mixed state in the identification area may be calculated.

As a modification, a mixed state may be set for an area in which a plurality of classes coexist in the learning data. For example, the operator can input information indicating the area ratio of a class, such as "the ratio of non-empty areas is 30%" for a specific area. In this case, the detailing unit 2300 can calculate the teacher information representing the mixed state for the identification area serving as the identification unit without estimating the class label for each pixel. On the other hand, the detailing unit 2300 can also estimate the class label of each pixel of the input image by referring to the mixed state. In this case, similar to Embodiment 5 described later, the similarity between information representing a mixture state that can be calculated from learning data and information representing a mixture state calculated based on the estimated attributes of each pixel is determined. Estimation can be performed using an evaluation value that increases as the value increases.

[Embodiment 3]
In the first and second embodiments, the description has been made on the assumption that the identification area serving as the identification unit is set in advance as a rectangular area or a small area. On the other hand, the size and cutting method of the identification area can be changed based on various photographic information. For example, in areas with strong blur, detailed texture information is lost, so by performing estimation over a wider identification area, it is possible to improve the accuracy of estimating the mixed state.

The photographing information includes information unique to the imaging device and information unique to the photographed image. Information specific to the imaging device includes the size of the sensor or the diameter of the permissible circle of confusion, the brightness or focal length of the optical system, and the like. Information unique to the photographed image includes aperture value, focusing distance, Bv value, RAW image, exposure time, gain (ISO sensitivity), white balance coefficient, distance information, location information such as GPS, and time information such as date and time. , etc. In addition, the information unique to the photographed image includes the gravity sensor value, acceleration, geomagnetic direction, temperature, humidity, atmospheric pressure, altitude, etc. at the time of photographing. There are also imaging systems that can obtain information from infrared light and ultraviolet light in addition to visible light. The obtained imaging information differs depending on the specifications of the imaging device. The photographing information may be information attached in association with the input image when the input image was photographed, information indicating the state of the imaging device at the time of photographing the input image, or information measured by the imaging device at the time of photographing the input image. Further, the photographing information may be information representing the characteristics of the input image detected by the imaging device when the input image was photographed. Further, the photographing information is information different from the data of the input image itself.

The basic configuration of the learning device according to the third embodiment will be explained along FIG. 1(D). The learning data storage unit 5100 stores learning data in advance. In this embodiment, the learning data includes learning images, photographic information corresponding to each learning image, and mixed state teacher information assigned to regions of various sizes on the learning images. The data acquisition unit 2100 reads learning images, shooting information, and teacher information from the learning data storage unit 5100. The learning unit 2200 performs learning of an estimator for estimating the mixed state using the learning image and the teacher information of the mixed state, and stores the obtained estimator in the estimator storage unit 5200. Here, the learning unit 2200 generates a first estimator by learning using identification images in a predetermined region set according to a first region setting pattern, and generates a first estimator in a predetermined region set according to a second region setting pattern. A second estimator is generated by learning using the identified images in . The evaluation unit 2400 uses the confirmation data read from the confirmation data storage unit 5400 to evaluate the estimation accuracy of each estimator obtained through learning. The evaluation unit 2400 then generates a region setter based on the imaging information and the estimation accuracy, and stores it in the setter storage unit 5300.

Next, the outline of the device configuration of the image processing device will be explained along FIG. 1(E). The image acquisition unit 1100 acquires an input image and photographing information. The area setting unit 1400 selects an area setting pattern to be used for setting a target image from among a plurality of area setting patterns according to the imaging information. In this embodiment, the area setting unit 1400 reads the area setting device from the setting device storage unit 5300 and sets the area to be the identification unit according to the imaging information. The estimating unit 1200 reads the estimator from the estimator storage unit 5200, and uses the estimator to estimate the mixed state of the target image in a predetermined area set according to the set identification unit.

A detailed explanation of the processing in this embodiment will be described below. First, processing during learning will be described with reference to the flowchart in FIG. 2D. In step S2100, the data acquisition unit 2100 reads the learning image, the shooting information, and the teacher information of the mixed state from the learning data storage unit 5100 as learning data.

In step S2200, the learning unit 2200 uses the learning image acquired by the data acquisition unit 2100 and the teacher information of the mixture state to train an estimator for estimating the mixture state. As described above, in this embodiment, identification units are set according to each of a plurality of types of area setting patterns. That is, various areas are prepared as identification units. For example, it is possible to prepare a plurality of patterns of identification units having different sizes, such as 3×3, 9×9, and 15×15 rectangular areas. As described in the first embodiment, the identification unit is not limited to a rectangular area. For example, as described in the first embodiment, a plurality of parameters used when setting small regions by region division can be prepared as a plurality of region setting patterns.

Due to differences in area setting patterns, the teacher information of the mixed state may change even at the same position on the image. FIG. 3C shows rectangular areas 551, 552, and 553 of various sizes located at the same position in the learning image. In the smallest rectangular region 551, the empty:non-empty area ratio is r=1. On the other hand, since the rectangular areas 552 and 553 each include a non-empty area, the area ratios are r=0.9 and r=0.8, respectively.

The learning unit 2200 performs learning of an estimator corresponding to each area setting pattern. That is, the learning unit 2200 performs learning of the estimator corresponding to the region of interest setting pattern based on the identification region set according to the region of interest setting pattern and the teacher information given for this identification region. As a result, the learning unit 2200 generates an estimator corresponding to each of the plurality of area setting patterns. For example, if the index of a region setting pattern is q and the total number of region setting patterns is Q, Q types of estimators yq can be obtained by learning. Learning of the estimator can be performed in the same manner as in the first embodiment. As an example, each estimator yq may perform mixed state estimation according to a regression function fq(X) (q=1,...,Q). The estimator obtained through learning is stored in the estimator storage unit 5200.

In step S2300, the evaluation unit 2400 evaluates the identification accuracy of the estimator obtained in step S2200 together with the imaging information, and generates a region setter. For example, the evaluation unit 2400 can evaluate the identification accuracy of each estimator using a verification image with which teacher information and photographic information are associated. Then, the evaluation unit 2400 generates information indicating an estimator corresponding to specific imaging information so that good identification accuracy can be obtained when determining an identification image associated with predetermined imaging information. I can do it.

Among the photographic information, there is information obtained for each pixel of the learning image. Furthermore, new photographic information can be obtained by combining photographic information. For example, if the distance Z(p) from the lens surface to the subject at the pixel position p and the focal length f of the optical system are obtained as photographic information, the image magnification S(p) can be calculated.

In addition, when the F value of the optical system, focal length f, focusing distance Z _f at the time of shooting, and distance Z (p) to the subject at pixel position p are obtained as shooting information, the amount of blur B at each pixel position (p) can be obtained.

Furthermore, if the values r(p), g(p), and b(p) at each pixel position p of the RAW image, the exposure time T, the gain G, and the aperture amount F are obtained as shooting information, then the pixel position p It is possible to obtain the absolute value of the incident light amount BV(p) at .

Hereinafter, a case will be described in which a region setter is generated using the amount of blur B(p) at pixel position p as photographic information. However, the photographing information to be used is not limited to this, and other photographing information such as the image magnification S(p) or the amount of incident light BV(p) may be used. Further, a plurality of photographic information may be combined, for example, the amount of blur B(p) and the amount of incident light BV(p) may be used in combination.

First, the evaluation unit 2400 divides the amount of blur B into a plurality of bins and generates a table regarding the area setting pattern q. In this example, the blur amount B is divided into four bins: less than 2, 2 or more and less than 3, 3 or more and less than 4, and 4 or more. Furthermore, three types of area setting patterns q are used: 3×3, 9×9, and 15×15, and a 3×4 table is obtained.

Next, evaluation section 2400 reads confirmation data from confirmation data storage section 5400. The confirmation data, like the learning data, includes a plurality of confirmation images, class label data for each confirmation image, and photographing information. Here, the total number of confirmation images is expressed as _Nv , and the v-th confirmation image is expressed as Iv ( _v =1,..., _Nv ).

The evaluation unit 2400 extracts the feature amount in the region i that is the identification unit in the confirmation image according to each of the region setting patterns q, and inputs it to the corresponding estimator. In this way, it is possible to obtain the estimated value y _q (X ^ν _i ) of the mixing state of the region i in the confirmation image Iν when the region setting pattern q is used. At this time, the square error for the mixed state teacher information c _q (ν, i) can be expressed as follows.

ここでδ_Ｂ（ν，ｉ）は、確認画像Ｉνの領域ｉの中心位置におけるボケ量がビンＢの範囲内であるときに１、そうでないときに０を返すものとする。 Further, the mean square error MSE (B, q) in the bin (B, q) for the combination of the amount of blur B and the area setting pattern q is expressed as follows.

Here, δ _B (ν, i) returns 1 when the amount of blur at the center position of area i of confirmation image Iν is within the range of bin B, and returns 0 otherwise.

The reliability T(B, q) regarding the bin (B, q) can be defined as a value obtained by subtracting the root mean square error from 1.

In this way, the evaluation unit 2400 can obtain a table of reliability T(B, q) for each bin (B, q). An example of the table thus obtained is shown below. The evaluation unit 2400 stores the table obtained in this way in the setter storage unit 5300 as a region setter.

In this embodiment, the obtained table is stored in the setter storage unit 5300 as an area setter. On the other hand, the evaluation unit 2400 uses the value of the reliability T(B, q) as training information and calculates a regression function g _q (B) that outputs the reliability T for the amount of blur B as a regression value for each area setting pattern q. It is also possible to generate a region using the same method and use it as a region setter.

The process of estimating the mixture state of an input image using the mixture state estimator and region setter obtained in this way will be explained using the flowchart of FIG. 2(E). In step S1100, the image acquisition unit 1100 acquires image data and photographing information obtained by the imaging device.

In step S1400, the area setting unit 1400 reads the area setting device from the setting device storage unit 5300, and determines the area setting pattern to be used according to the imaging information. For example, the area setting unit 1400 determines, for each area i of the input image I, an area setting pattern q _win that maximizes the reliability T obtained from the amount of blur B(i) obtained as the shooting information, according to the following formula. You can choose. Note that the amount of blur B(i) refers to the amount of blur at the center position of area i of input image I. Although the specific processing is not particularly limited, for example, if the reliability is higher if the input image I is divided into multiple regions according to one region setting pattern and one region is subdivided according to another region setting pattern. This subdivision can be done in As another example, regions with similar amounts of blur can be connected, and each connected region can be divided into regions using a region setting pattern according to the amount of blur.

In step S1200, the estimation unit 1200 reads the estimator from the estimator storage unit 5200 and estimates the mixing state at each position of the input image. Specifically, the estimation unit 1200 estimates the mixing state at this position p by extracting the feature amount of the image of a predetermined region set at each position p and inputting the extracted feature amount to the estimator. can do. Here, the predetermined area for each position p is set according to the area setting pattern q _win determined in step S1400. As described above, in this embodiment, estimators corresponding to each of a plurality of area setting patterns are generated. Therefore, the estimator 1200 can use an estimator selected from a plurality of estimators according to the region setting pattern determined in step S1400. For example, y _qwin is selected as the estimator at position p, and the estimated value of the mixing state of the predetermined region at position p is obtained as y _qwin (X _i ). FIG. 3(D) shows an example in which the region setting method is changed depending on the position on the image, and each rectangle represents one region input to the estimator.

The processing related to step S1300 is the same as that in the first embodiment, so a description thereof will be omitted. As in this embodiment, by changing the method of setting the area that is the identification unit for estimating the mixed state using photographic information, it is possible to estimate the mixed state with less error.

[Embodiment 4]
In Embodiments 1 to 3, the mixed state in a predetermined region serving as an identification unit was estimated. In the fourth embodiment, a method will be described in which a detailed region segmentation result is obtained by subdividing the region using the obtained mixed state estimation result. The basic configurations of the learning device and the image processing device are the same as those in Embodiment 1, and the description thereof will be omitted.

The processing during learning will be described below according to the flowchart of FIG. 2(A). In step S2100, the data acquisition unit 2100 reads the learning image, teacher information of the mixed state, and learning data from the learning data storage unit 5100.

In step S2200, the learning unit 2200 performs the same process as in the third embodiment. That is, areas of various sizes are prepared as identification units. For example, a plurality of patterns of rectangular areas of different sizes such as 1×1, 3×3, 9×9, and 15×15 can be prepared according to each of the plurality of area setting patterns. Then, the learning unit 2200 can perform learning of the estimator corresponding to each region size using the teacher information of the mixed state obtained for each region size, similarly to the third embodiment. That is, if the region size index is q and the total number of region sizes is Q, Q types of estimators yq (q=1, . . . , Q) can be obtained by learning. As an example, each estimator yq may perform mixed state estimation according to a regression function fq(X). The estimator yq obtained through learning is written into the estimator storage section 5200.

Next, the process at the time of determination will be explained according to the flowchart of FIG. 2(F). In step S1100, the image acquisition unit 1100 acquires an input image. In step S1200, the estimator 1200 estimates the mixing state in a predetermined region in the input image using an estimator. Here, the estimation unit 1200 performs region setting using the first region setting pattern among the plurality of region setting patterns. That is, the estimation unit 1200 determines the mixing state of the first target image having the size according to the first area setting pattern. In this embodiment, the largest size among the Q types of area sizes is used as the identification unit. In the above example, 15×15 pixels are selected as the identification unit, and an estimator corresponding to 15×15 pixels is used as the estimator.

Then, the estimating unit 1200 determines whether to re-determine the mixing state of the first portion according to information indicating the estimated mixing state of the first target image in the first portion of the input image. For example, the estimating unit 1200 determines whether or not to re-determine the mixed state of a predetermined region in which the mixed state has been estimated. For example, the estimation unit 1200 employs this class estimation result for a region where the class purity is equal to or higher than the threshold value.

On the other hand, the estimation unit 1200 re-determines the mixed state of a region where the class purity is lower than the threshold value. In response to the determination to perform re-determination, the estimating unit 1200 outputs information indicating the mixed state of the second target image in the first portion and having a size according to the second area setting pattern. Here, the second target image is smaller than the first target image. That is, the estimating unit 1200 re-divides the region in accordance with smaller identification units for the region where the class purity is lower than the threshold, and estimates the mixed state of each of the re-divided regions using the estimator again. For example, the estimation unit 1200 can perform re-division using a region size that is one step smaller. As described above, in this embodiment, estimators corresponding to each of a plurality of area setting patterns are generated. Therefore, the estimation unit 1200 can use an estimator selected from a plurality of estimators according to the region setting pattern used for re-segmentation.

Here, class purity indicates the ratio of pixels to which the same class label is assigned within a region. For example, when the value of the area ratio r shown in Embodiment 1 is 0.8 or more or 0.2 or less, it can be defined that the class purity is high. When using the map shown in FIG. 7, class purity can also be defined as high when p1≧0.9 and p2≦0.8.

In this way, detailed region segmentation can be performed by subdividing regions with low class purity and re-estimating the mixing state. If the region cannot be subdivided or the class purity of all regions exceeds the threshold, the process can proceed to step S1300. The processing in step S1300 is the same as that in the first embodiment, so a description thereof will be omitted. The detailed region division results obtained in this way can be used for high image quality processing such as tone mapping or white balance adjustment for each region.

[Embodiment 5]
In the fourth embodiment, detailed region division results are calculated by subdividing the identification unit, but the method of region division is not limited to this method. In the fifth embodiment, a method will be described in which detailed region segmentation results are obtained by performing class determination on a pixel-by-pixel basis using the mixed state estimation results for each region.

The basic configuration of the image processing device according to this embodiment is shown in FIG. 1(F). The functions of the image acquisition unit 1100 and the estimation unit 1200 are the same as those in the first embodiment, so the description thereof will be omitted. The determination unit 1500 determines the attribute of each pixel of the target image. The determination unit 1500 determines the attribute of each pixel based on the evaluation value, and the evaluation indicated by this evaluation value is determined based on the information indicating the mixed state calculated based on the attribute of each pixel and the information obtained by the estimation unit 1200. The greater the similarity between the information indicating the mixed state and the information indicating the mixed state, the higher the similarity. In this embodiment, the determination unit 1500 estimates the class label of each pixel of the input image based on the mixture state estimation result and image information. The output unit 1300 outputs information indicating the class label estimated for each pixel of the input image.

Details of the determination process according to this embodiment will be explained with reference to FIG. 2(G). The processing in steps S1100 and S1200 is the same as that in the first embodiment, so the description thereof will be omitted. In step S1500, the determination unit 1500 estimates the class for each pixel of the input image using the mixed state of each region estimated in step S1200. For example, the class of each pixel can be estimated so that the mixed state obtained according to the estimated class of each pixel is close to the mixed state estimated in step S1200. The class of each pixel can be estimated using the color information of each pixel, for example, so that the colors of pixels belonging to the same class are similar, or the colors of pixels belonging to different classes are dissimilar. can.

As an example of a method for estimating the class of each pixel of an input image, a case will be described below in which repetitive processing such as CRF (Conditional Random Field) is used. CRF sequentially transitions the state of each node to a stable state for a graph consisting of multiple nodes, taking into account the pairwise potential based on the similarity between paired nodes and the unary potential of each node. This is the way to go. When using CRF to discriminate pixels of an image, a CRF model can be used in which each node corresponds to each pixel of the image.

ここで、右辺第一項のψはｕｎａｒｙ ｐｏｔｅｎｔｉａｌ、右辺第二項のφはｐａｉｒｗｉｓｅ ｐｏｔｅｎｔｉａｌを表す。θ_ψ及びθ_φはそれぞれのパラメータであり、後述する学習処理において算出される。ε_ｉは、画素ｉの近傍画素の集合である。ｇ_ｉｊは画素ｉと画素ｊとの相互関係を表す関数であり、Ｚは正規化項である。判定部１５００は、このモデル式に従って各画素のクラスラベルを更新していくことによって、画像全体のポテンシャルが高い状態へと判定結果を収束させていく。 The conditional probability of class label ci of pixel i on input image I can be expressed by the following formula.

Here, ψ in the first term on the right side represents a unary potential, and φ in the second term on the right side represents a pairwise potential. θ _ψ and θ _φ are respective parameters, and are calculated in a learning process described later. ε _i is a set of neighboring pixels of pixel i. g _ij is a function representing the mutual relationship between pixel i and pixel j, and Z is a normalization term. The determination unit 1500 updates the class label of each pixel according to this model formula, thereby converging the determination results to a state where the potential of the entire image is high.

ここで、ｘ_ｉ及びｘ_ｊはそれぞれ画素ｉ及び画素ｊの色情報であり、ＲＧＢ値を持つ３次元ベクトルで表わされる。βはユーザの定義するハイパーパラメータであって、β＝１などと設定することができる。このように、ｐａｉｒｗｉｓｅ ｐｏｔｅｎｔｉａｌは、時刻ｔにおいて違うクラスに属する画素の色が類似する場合に評価が低くなるように設定することができる。 The pairwise potential can be expressed by the following formula.

Here, x _i and x _j are color information of pixel i and pixel j, respectively, and are represented by three-dimensional vectors having RGB values. β is a hyperparameter defined by the user, and can be set as β=1. In this way, the pairwise potential can be set so that the evaluation becomes low when the colors of pixels belonging to different classes at time t are similar.

ここでｙ_ｃ（Ｘ_ｉ）は、画素位置ｉにおけるクラスｃに関する混合状態推定値である。ｙ_ｃ（Ｘ_ｉ）は、画素位置ｉが含まれる所定領域について推定部１２００が推定した混合状態に基づいて算出することができ、例えば所定領域内における、クラスｃの面積比、エッジ画素率、又はクラスラベル配置パターン等でありうる。このように、ｕｎａｒｙ ｐｏｔｅｎｔｉａｌは、時刻ｔにおいて各画素の属性に基づいて計算される混合状態と、推定部１２００により得られた混合状態と、の類似度が大きいほど評価が高くなる。 The unary potential can be expressed as below.

Here y _c (X _i ) is the mixed state estimate for class c at pixel location i. y _c (X _i ) can be calculated based on the mixture state estimated by the estimation unit 1200 for a predetermined region including pixel position i, and for example, the area ratio of class c, edge pixel ratio, Alternatively, it may be a class label arrangement pattern, etc. In this way, the evaluation of the unary potential becomes higher as the degree of similarity between the mixed state calculated based on the attributes of each pixel at time t and the mixed state obtained by the estimation unit 1200 is greater.

L ⁱ _c (t) is the mixed state of class c in a predetermined area including pixel i at time t when the class label of each pixel changes according to the CRF. L ⁱ _c (t) is the same type of information as the mixed state estimated by the estimation unit 1200, and can be calculated by the estimation unit 1200 with reference to the class estimated for each pixel in the predetermined area at time t. A specific example will be given below according to the example of the mixed state described in Embodiment 1. For example, at time t during the transition, the area ratio r(t) of class c can be determined by counting the pixels to which class label c is assigned within a predetermined region including pixel i. Further, by extracting and counting edge pixels according to the arrangement of class labels within a predetermined region, the edge pixel ratio e(t) can be determined. Further, by determining which of the maps shown in FIG. 7 the class label arrangement within the predetermined area is closest to, the class label arrangement pattern p(t) can be determined. As described in the first embodiment, L ⁱ _c (t) can also be expressed by a combination of these mixed states at time t.

In this way, the degree of similarity between the mixed state in a predetermined region at time t determined based on the class label arrangement at the pixel level at time t during transition and the mixed state estimated value for the predetermined region is calculated as unary It can be expressed as potential. Specifically, the unary potential can be expressed such that the greater the similarity between the mixed state within the predetermined region at time t and the estimated mixed state value for the predetermined region, the higher the evaluation.

The learning process in this embodiment will be explained with reference to FIG. 2(A). In step S2100, the data acquisition unit 2100 acquires a learning image and teacher data. In step S2200, the learning unit 2200 performs estimator learning similarly to the first embodiment. Further, the learning unit 2200 determines the values of parameters (for example, θ _ψ and θ _φ described above) used when estimating the class of each pixel of the input image. The learning unit 2200 can perform this process using a plurality of learning images and class label data indicating the class of each pixel in the learning images. As the class label data, for example, data created as shown in FIG. 9C according to the second embodiment can be used. In this embodiment, the learning unit 2200 can calculate the values of θ _ψ and θ _φ so that the potential for all learning images is maximized. That is, the values of θ _ψ and θ _φ that maximize the following expressions can be found by the gradient method or the like.

The learning unit 2200 stores the obtained parameters together with the estimator in the estimator storage unit 5200. In this embodiment, the values of θ _ψ and θ _φ are stored in the estimator storage unit 5200 and used by the determination unit 1500 as described above. The class label data for each pixel obtained in this way can be used when performing image quality enhancement processing for each area, as in the fourth embodiment.

The method of performing pixel-by-pixel class determination using the mixed state estimation result is not limited to the above method. For example, as in Embodiment 2, pixel-by-pixel class determination is performed based on the Gaussian mixture distribution of each class obtained using a region where the class has been determined and the similarity of the mixture state described above. You can also do that.

The processing according to this embodiment can be performed using any of the area ratio, edge pixel ratio, and class label arrangement pattern as the mixed state, and the usable mixed states are not limited to these. Further, by using a mixed state expressed by combining a plurality of expressions, it is possible to improve the determination accuracy. For example, by using the edge pixel ratio in addition to the area ratio, we can distinguish between cases where the contour is simple, such as the boundary between a building and the sky, and cases where the contour is complex, such as the boundary between a branch and the sky. becomes possible.

In the present embodiment, information indicating the mixed state is obtained through processing by the estimation unit 1200. However, the determination unit 1500 can also acquire information indicating a mixed state obtained by a different method and determine the attribute of each pixel using a similar method.

(Other examples)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1100: Image acquisition unit, 1200: Estimation unit, 1300: Output unit, 1400: Area setting unit, 1500: Determination unit, 2100: Data acquisition unit, 2200: Learning unit, 2300: Detailing unit

Claims (14)

an acquisition means for acquiring an input image;
extraction means for extracting features from the input image;
an estimating means for estimating information corresponding to the area of a region belonging to a specific class in the input image when the feature extracted by the extracting means is input;
The image processing apparatus is characterized in that the estimation means performs estimation using parameters learned using learning images. The image processing apparatus according to claim 1, wherein the estimating means outputs information specifying a ratio of regions belonging to the specific class. The image processing apparatus according to claim 1, further comprising a control unit that controls imaging by the imaging unit based on the estimation result by the estimation unit. 4. The image processing apparatus according to claim 3, wherein the control means controls the imaging by the imaging means with respect to exposure, focus, or white balance based on the estimation result by the estimation means. 3. The ratio estimated by the estimation means is a ratio of (1) the number of pixels of the region belonging to the specific class to (2) the total number of pixels of the input image. The image processing device described. The image processing apparatus according to claim 1, wherein the input image is a partial image of the entire image captured by an imaging means. 7. The image processing apparatus according to claim 6, further comprising a control unit that integrates the results estimated by the estimation unit for the partial image and controls imaging by the imaging unit. The estimation means has parameters learned based on (a) features extracted from the learning image, and (b) teacher information representing information corresponding to an area of a region belonging to a specific class in the learning image. ,
The image processing apparatus according to claim 1, wherein the teacher information represents information corresponding to a total area of a plurality of areas belonging to the specific class in the learning image. acquisition means for acquiring divided images obtained by dividing the input image;
Extracting means for extracting features from the divided images;
an estimating means for estimating information corresponding to an area of a region belonging to a specific class in the divided image when the feature extracted by the extracting means is input;
The image processing apparatus is characterized in that the estimation means performs estimation using parameters learned using learning images. The image processing apparatus according to claim 1, wherein the estimating means outputs information specifying a ratio of regions belonging to the specific class. The image processing apparatus according to claim 9, further comprising a control means for controlling imaging by the imaging means based on the estimation result by the estimation means. an acquisition step of acquiring an input image;
an extraction step of extracting features from the input image;
an estimation step of estimating information corresponding to the area of a region belonging to a specific class in the input image when the feature extracted in the extraction step is input;
An image processing method, wherein in the estimation step, estimation is performed using parameters learned using a learning image. an acquisition step of acquiring divided images obtained by dividing the input image;
an extraction step of extracting features from the divided images;
an estimation step of estimating information corresponding to an area of a region belonging to a specific class in the divided image when the feature extracted in the extraction step is input;
An image processing method, wherein in the estimation step, estimation is performed using parameters learned using a learning image. A program for causing a computer to operate as the image processing apparatus according to claim 1.

Images