JP7300027B2 - Image processing device, image processing method, learning device, learning method, and program - Google Patents

Description

The present invention relates to an image processing device, a learning device, a focus control device, an exposure control device, an image processing method, a learning method, and a program.

In recent years, research on image segmentation has also been widely conducted. For example, a person's area, an automobile area, a road area, a building area, a sky area, or the like can be cut out from an image. This is called semantic segmentation, and the result of segmentation can be applied to image correction or scene interpretation corresponding to the type of subject.

As a semantic segmentation method, there is a method of segmenting an image into several regions in advance and classifying each segmented region. For example, an image can be divided into a plurality of rectangular blocks and each block can be classified. As a method for classifying images, classification using deep learning, as described in Non-Patent Document 1, has been widely studied. Alternatively, for example, the method described in Non-Patent Document 2 can be used to divide an image into small irregular-shaped regions (superpixels), and classify the regions using the feature values of the regions and the context feature values around the regions. can. For class classification, an estimator trained using training images can be used.

In recent years, region segmentation using deep learning has also been studied. Non-Patent Document 3 uses an intermediate layer output of a CNN (Convolutional Neural Network) as a feature amount, and integrates class determination results for each pixel based on a plurality of intermediate layer features. With this method, class determination can be performed directly for each pixel without using the result of segmentation into small regions.

A. Krizhevsky et al. "ImageNet Classification with Deep Convolutional Neural Networks", Proc. Advances in Neural Information Processing Systems 25 (NIPS 2012). R. Achanta et al. "SLIC Superpixels", EPFL Technical Report 149300, 2010. J. Long et al. "Fully Convolutional Networks for Semantic Segmentation", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015.

According to the conventional method, each small area on the image can be classified into classes based on the type of subject. For example, based on the feature amount of each area, it was possible to determine whether the area represents the sky or the foreground (other than the sky). On the other hand, it has been difficult to make an appropriate determination for an area where different types of subjects coexist. For example, if multiple regions where the sky can be seen through the gaps between many tree branches are judged, all regions will be judged to be foreground, or all regions will be judged to be empty because the textures are similar. There was a high possibility that it would be determined that

An object of the present invention is to classify each region of an image so as to improve the accuracy of processing using the classification results.

In order to achieve the object of the present invention, for example, the image processing apparatus of the present invention has the following configuration. i.e.
an acquisition means for acquiring an input image;
extracting means for extracting features from the input image;
output means for outputting information corresponding to the area of a region belonging to a specific class in the input image from the estimator to which the features extracted from the input image are input;
The estimator has parameters learned using training images.

According to the present invention, each region of an image can be classified into classes so as to improve the accuracy of processing using classification results.

FIG. 2 is a block diagram showing a configuration example of an apparatus according to each embodiment; 4 is a flowchart for explaining processing according to each embodiment; FIG. 4 is a diagram for explaining learning images and class labels; FIG. 4 is a diagram for explaining class labels and a mixed state at the time of 2-class identification; FIG. 10 is a diagram for explaining class labels and a mixed state at the time of 3-class identification; The figure explaining the mixed state at the time of 3 class identification. Explanatory diagram of mixed state mapping. The figure which shows the structural example of the learning part or estimation part which concerns on each embodiment. FIG. 10 is a diagram for explaining class label refinement in a mixed area; The figure which shows the structural example of the computer which can implement|achieve each embodiment.

According to one embodiment of the present invention, it is possible to estimate how a plurality of classes are mixed (hereinafter referred to as a mixed state) in a predetermined area serving as a discrimination unit on an input image. Hereinafter, the image within the estimation target area may be referred to as the target image. More specifically, according to one embodiment of the present invention, the mixed state of regions having mutually different attributes in the target image is determined. Each attribute area is an area occupied by subjects belonging to the same class. That is, one of the regions with this attribute is the region of the subject belonging to a specific class, and the other one of the regions of this attribute is the region of the subject belonging to a class different from the specific class.

According to one embodiment, for example, for an area where the sky can be seen between many tree branches (foreground), the mixture state of the foreground part and the sky part (for example, area ratio, edge area, arrangement pattern, etc.) is calculated. can be estimated. By using not only the class information of each region (for example, information indicating whether it is a foreground region or a sky region) as obtained by the conventional method, but also information indicating such a mixture state, it is possible to later apply to the image It is possible to improve the accuracy of the processing performed by A specific example will be described in detail in each embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings. However, the scope of the present invention is not limited to the following embodiments. In the following embodiments, each processing unit shown in FIG. 1 and the like may be implemented by a computer or may be implemented by dedicated hardware.

FIG. 10 is a diagram showing the basic configuration of a computer that can implement each embodiment. In FIG. 10, a processor 101 is, for example, a CPU and controls the operation of the entire computer. The memory 102 is, for example, a RAM, and temporarily stores programs, data, and the like. The computer-readable storage medium 103 is, for example, a hard disk or CD-ROM, and stores programs and data for a long period of time. In this embodiment, a program that implements the function of each unit stored in the storage medium 103 is read out to the memory 102 . The processor 101 operates in accordance with the programs on the memory 102 to implement the functions of each unit.

In FIG. 10, an input interface 104 is an interface for acquiring information from an external device. An output interface 105 is an interface for outputting information to an external device. A bus 106 connects the above units and enables data exchange.

[Embodiment 1]
The basic configurations of the image processing device and the learning device according to the first embodiment will be described with reference to FIGS.

First, according to FIG. 1A, the outline of the device configuration of the learning device will be described. In the present embodiment, the learning device generates an estimator, which is used when an image processing device, which will be described later, performs processing for recognizing a mixed state, from learning images prepared in advance. Details of the learning process will be described later. The learning data storage unit 5100 stores learning data prepared in advance. The learning data includes learning images and teacher information. The data acquisition unit 2100 acquires learning images and teacher information from the learning data storage unit 5100 . The learning unit 2200 uses the feature extraction unit 610 to extract the feature amount of the identification image used for the learning of the estimator in a predetermined region of the learning image. In addition, the learning unit 2200 performs learning of an estimator that estimates a mixed state from a feature amount using a combination of the feature amount of the identification image and teacher information. For example, the learning unit 2200 can learn an estimator that outputs information indicating a mixed state when a feature amount is input. Here, the teacher information is information indicating a mixed state between regions with different attributes in the identification image. The estimator obtained by learning is stored in estimator storage section 5200 . Specifically, the estimator storage unit 5200 can store estimator parameters determined by learning.

Next, according to FIG. 1B, an overview of the configuration of the image processing apparatus will be described. In this embodiment, the image processing apparatus performs processing for estimating a mixed state in an unknown input image. Details of the processing contents will be described later. The image acquisition unit 1100 acquires an input image. The estimating unit 1200 uses the feature extracting unit 610 to extract a feature amount from a target image, which is a mixed state identification target, located in a predetermined region of the input image. The feature extraction unit 610 used by the estimation unit 1200 can operate in the same manner as the feature extraction unit 610 used by the learning unit 2200 . In addition, the estimation unit 1200 estimates the mixed state of regions having mutually different attributes in the target image based on the feature amount. For example, the estimating unit 1200 reads the pre-trained estimator 620 from the estimator storage unit 5200, and inputs the feature amount to the estimator. Information indicating the state is output to the output unit 1300 . The estimator 620 can be obtained by learning by the learning unit 2200 . Output section 1300 outputs the result of estimation by estimation section 1200 .

The data acquisition unit 2100 and the learning unit 2200 of the learning device may be implemented on the same computer, configured as independent modules, or implemented as programs running on a computer. The learning data storage unit 5100 and the estimator storage unit 5200 of the learning device can be realized using storage inside or outside the computer.

The image acquiring unit 1100 and the estimating unit 1200 of the image processing apparatus may be implemented on the same computer, configured as independent modules, or implemented as programs running on a computer. . Also, these may be implemented as a circuit or a program inside an imaging device such as a camera.

The image processing device may be implemented on the same computer as the learning device, or may be implemented on a separate computer. The estimator storage unit 5200 included in the learning device and the image processing device may be the same storage or different storages. When using a different storage, the estimator stored in the estimator storage unit 5200 by the learning device can be copied or moved to the estimator storage unit 5200 included in the image processing device.

The processing according to this embodiment will be described in detail below. First, processing during learning performed by the learning device will be described according to the flow of FIG. 2(A). In S2100, the data acquisition unit 2100 acquires the learning image and the mixed state teacher information from the learning data storage unit 5100 as learning data.

In the learning data storage unit 5100, a plurality of learning images and teacher information of a mixed state are stored in advance. A training image refers to an image used for training the estimator. The learning image can be, for example, image data captured by a digital camera or the like. The format of the image data is not particularly limited, and may be JPEG, PNG, BMP, or the like. In the following description, the number of prepared learning images is N, and the n-th learning image is represented as I _n (n=1 . . . N).

The mixed state teacher information indicates the mixed state in a predetermined region of the learning image. This teacher information is prepared in advance, and can be created by a person, for example, while viewing the learning image. In the present embodiment, a plurality of regions that are identification units are set in the learning image, and teacher information is prepared for each region. Hereinafter, an image of a predetermined region in a learning image, which serves as one identification unit, will be referred to as an identification image.

A method of setting the area is not particularly limited. For example, multiple regions can be set in the input image according to a predetermined region setting pattern. As a specific example, it is possible to divide the learning image into a plurality of rectangular regions of a predetermined size (for example, 16×16 pixels), and treat each rectangular region as an identification unit. In addition, the small area obtained by the method described in Non-Patent Document 2 can be treated as an identification unit area. On the other hand, a rectangular area of a predetermined size may be set only in part of the learning image. Note that the learning data storage unit 5100 may store an identification image of a predetermined size as learning data.

The mixed state indicated by the teacher information will be described below. A subject on an image can be classified into a plurality of classes. FIG. 3 shows an example of such class classification. FIG. 3A shows an example of a learning image 500. FIG. The learning image 500 includes the sky, people, and plants, which can be classified into different classes. That is, as shown in FIG. 3B, the pixels included in the region 541 are assigned the class label of “sky”, the pixels included in the region 542 are assigned the class label of “person”, and the pixels included in the region 543 are assigned the class label of “person”. can each be given a class label of "Plant".

There are various definitions of classes and class labels, and the method of classifying is not particularly limited. In the example of FIG. 3, class classification is performed according to the type of subject. Examples of other class labels include skin or hair regions, animals such as dogs or cats, and man-made objects such as automobiles or buildings. A class label can also be used to indicate a particular object, such as Part A or Part B used in the factory. On the other hand, each pixel may be classified into a main subject area and a background area. Classification may also be performed according to the difference in surface properties such as a glossy surface or a matte surface, or the difference in materials such as a metal surface or a plastic surface. In the following, it is assumed that there are M types of classes in total.

A mixed state of classes is a mixed state between regions of mutually different attributes in the target image. Each attribute area is an area occupied by subjects belonging to the same specific class. One of the regions with attributes different from each other is a region of a subject belonging to a specific class, and the other is a region of a subject belonging to a class different from the specific class. Hereinafter, an attribute area occupied by a subject belonging to a certain class may simply be referred to as an area belonging to that class. Also, the class of the subject captured in each pixel may be referred to as the attribute or class of the pixel below.

There are many possible definitions of mixed state. In this embodiment, the mixed state is expressed numerically as follows. In one embodiment, the information indicating the mixed state is information determined depending on the distribution of attribute regions in the target image. For example, the information indicating the mixed state is information representing the ratio of each attribute area in the target image. As a specific example, the information indicating the mixed state can be the area ratio of the regions belonging to each class in the target image. A case where there are two classes, "empty" and "non-empty", will be described with reference to the example in FIG. FIG. 4 shows a learning image 510 and its class label 520 . The class label 520 represents pixels with class "empty" in white and pixels with class "non-empty" in black. FIG. 4 shows an enlarged view (526) in which a region 525 on the class label 520 corresponding to the identification image 515 in the learning image 510 is enlarged. The enlarged view (526) shows the non-empty area 511 and the empty area 522. FIG. At this time, the mixed state of the identification image 515 can be represented by the area ratio r between the sky area and the non-sky area in the corresponding area 525 . For example, in a rectangular area of 16×16 pixels, if the sky area pixels are 192 pixels and the non-sky area pixels are 64 pixels, then r=192/256=0.75.

Although the above example describes area ratios of two classes, area ratios of three or more classes can also be represented. FIG. 5 shows a training image 530, an identification image 535 in the training image 530, and a class label 536 for the identification image 535. FIG. In this example, each pixel of the learning image 530 is classified into three classes: "sky", "plant", and "artifact". The mixed state in this case can be determined according to the area ratio of the plant area 531 , the sky area 532 , and the artifact area 533 . As an example, a point 545 indicating the area ratio in this case can be plotted on a simplex 540 (a triangle in the case of FIG. 5) in the coordinate space shown in FIG. 6(A) according to the area ratio of each class. can. Since this point 545 can be uniquely expressed using the internal division ratios t ₁ and t ₂ that internally divide the two sides of the simple substance, the area ratio at this time is r=(t ₁ , t ₂ ). It can be represented by a vector. Since this is the same for general M dimensions, the area ratio when the number of classes is M is uniquely represented by an M−1 dimensional vector r=(t ₁ , t ₂ , . . . t _M−1 ) be able to. Note that the area ratio in the case of two classes as described above is the same value as when M=2 in this generalized form.

Also, the M-class area ratio expressed as above may be mapped to a lower-order space. For example, the area ratio in the identification image can be plotted in an M-dimensional space and mapped in a low-dimensional space using SOM (Self-Organizing Map) or LLE (Locally Linear Embedding). FIG. 6B shows an example of quantizing the three-class mixing ratio space with a one-dimensional SOM550. 551 represents the start node of SOM 550 and 552 represents the end node of SOM 550 . FIG. 6(C) is obtained by mapping these to a one-dimensional space. If the position of the start node is 0 and the position of the end node is 1, and the nodes are arranged evenly, a point 545 indicating the area ratio can be represented by a scalar value representing the position on the map. For example, the area ratio point 545 shown in FIG. 6B can be approximated as point 546 on FIG. 6C, and the map location of point 546 (r=0.37 ) can be used to represent the area ratio. The number of dimensions after mapping is not limited to one dimension as shown in FIG. r can be represented by an R-dimensional vector. For example, when the number of classes M=5 and the area ratio space is quantized by a two-dimensional SOM, r can be represented by a two-dimensional vector.

Also, the M-class area ratio may be expressed as a composite vector of a plurality of basis vectors. For example, class area ratios obtained from various identification images can be decomposed into multiple basis vectors using principal component analysis, sparse coding, or the like, and approximated by a small number of vectors with large contribution. In this case, the area ratio in the area ratio space can be expressed as a composite vector of these basis vectors, and the area ratio can be expressed using a weighting factor for each basis vector at that time.

As another example, the information indicating the mixed state may be information relating to the boundary between regions of different attributes in the target image, for example, information representing the ratio of pixels representing this boundary in the target image. As an example, edge detection is performed on a binary image that indicates the class of each pixel (for example, sky region or non-sky region), the number of obtained edge pixels is counted, and the number of pixels in a predetermined region and the number of edge pixels are calculated. The ratio e can be used to represent the mixing state. FIG. 4 shows an edge detection result 527 for class label 526, representing detected edge pixels 523. FIG. If the count result of edge pixels in a rectangular area of 16×16 pixels is 64, the edge pixel ratio can be expressed as e=64/256=0.25.

As yet another example, the information indicating the mixed state may be information representing the arrangement of attribute regions in the target image. For example, the mixed state can be represented according to the arrangement pattern of pixels of each class within a predetermined area. If the number of classes is M and the number of pixels in the predetermined area is K, the class of each pixel in the predetermined area can be represented by an M×K binary vector. For example, if two classes of "empty" and "non-empty" are defined, and the size of the predetermined area is 16×16 pixels, the class label arrangement pattern in the predetermined area is a 2×16×16=512-dimensional two-dimensional It can be represented as a value vector. By plotting various binary vectors obtained from the identification image in this way on a vector space and quantizing them using SOM, LLE, etc., it is possible to express the class label arrangement pattern in a predetermined region as a vector p. can. It is also possible to use a method of expressing various binary vectors obtained from an identification image with basis vectors using principal component analysis, sparse coding, or the like.

FIG. 7 shows a map 900 obtained by mapping class label arrangement patterns in a predetermined area using a two-dimensional SOM. In map 900, each square is a node of the SOM and each represents a quantized class label placement pattern. Due to the characteristics of the SOM, similar patterns are placed close to each other on the map. The class label arrangement pattern in the identification image can be represented by position coordinates p on the map based on the closeness of each node to the pattern. For example, in the example of the two-dimensional SOM in FIG. 7, the position on the map can be represented by a two-dimensional vector p=(p ₁ , p ₂ ).

Thus, the mixed state can be represented in various ways. Mixed states may be represented using any one of these representations. For example, if the mixed state is expressed only by the area ratio, it can be defined as C=r, if it is expressed only by the edge pixel rate, it can be expressed as C=e, and if it is expressed only by the class label arrangement pattern, It suffices to define C=p. Also, the mixed state may be expressed by combining multiple expressions. For example, the mixed state may be defined as a combination of area ratio and edge pixel ratio C=(r, e), or a combination of area ratio, edge pixel ratio, and class label arrangement pattern C=(r, e, p). In the present invention, the method of expressing the mixed state is not particularly limited.

As noted above, the mixture state C can be represented as a vector of one or more numbers. That is, it can be said that the information representing the mixed state obtained in one embodiment is a feature quantity representing the mixed state in the predetermined region. Let L be the number of dimensions of the vector representing the mixed state C. In the following, the mixture state vector in a predetermined region i on the image _In is represented as _Cni , and the l (l=1, ..., L) element of the mixture state vector _Cni is c(n, i, l) is represented as Note that the mixed state may indicate not only how the pixels of each class are mixed in the given area, but also whether the given area is composed of pixels of one specific class.

In this embodiment, it is assumed that each pixel of each learning image is given a class label as shown in FIG. 3(B). Then, based on this class label, for each identification image obtained from the learning images, the mixed state C represented as a scalar value or vector value as described above is calculated in advance as teacher information, and stored in the learning data storage unit 5100. It is assumed that it is stored in advance. However, the data acquisition unit 2100 acquires the information indicating the attribute of each pixel of the identification image, and generates the information indicating the mixed state using the information indicating the attribute of each pixel, thereby obtaining the teacher information. good. For example, the data acquisition unit 2100 can calculate the mixed state C of each identification image as described above based on the class label of each pixel of the learning image stored in the learning data storage unit 5100 . Furthermore, it is not essential that each pixel of each learning image is given a class label as shown in FIG. 3(B). For example, a mixed state of the identification images input by the worker while viewing the identification images obtained from the learning images, or a mixture of the identification images automatically calculated based on the information (edge information, etc.) input by the worker. The state may be stored in the learning data storage unit 5100 in advance.

In step S2200, the learning unit 2200 acquires the identification image and the teacher information of the mixture state from the data acquisition unit 2100, and learns an estimator for estimating the mixture state. A case of using a CNN (Convolutional Neural Network) as an estimator will be described below. A conventionally known configuration can be used as the configuration of the CNN. Typically, CNNs perform recognition tasks by progressively summarizing local features of the input signal by repeating convolutional and pooling layers to obtain features that are robust against deformation and misalignment. It is a neural network.

An example of estimation processing using CNN will be described with reference to FIG. The learning unit 2200 uses the feature extraction unit 610 to extract the feature amount of the identification image used for learning the estimator. Also, the learning unit 2200 learns an estimator that outputs information indicating a mixed state when a feature amount is input, using a combination of the feature amount of the identification image and the teacher information. FIG. 8A shows an example of a CNN that can be used for processing by the learning unit 2200. FIG. FIG. 8A shows a portion corresponding to the processing performed by the feature extraction unit 610, which corresponds to the convolution layer of CNN that performs feature extraction. Also, FIG. 8A shows a portion corresponding to the estimator 620 that performs learning, which corresponds to the fully connected layer of CNN that performs pattern estimation.

The convolution layer has an input layer 611 that receives as a signal the convolution operation result at each position of the identification image 630, which is a partial image of the learning image. A signal from the input layer 611 is sent to the final layer 615 through a plurality of intermediate layers 612 and 613 in which a convolution layer and a pooling layer are arranged, and the convolution operation and signal selection by pooling are repeated. The output signal from final layer 615 of feature extractor 610 is sent to estimator 620 . The output signal of the feature extraction unit 610 is assumed to be X below. In the fully connected layer, the elements of each layer are fully connected to the layers before and after, and the signal input from the feature extraction unit 610 is sent to the output layer 640 through sum-of-products operation using weighting coefficients. The output layer 640 has the same number of output elements as the number of dimensions L of the mixture state vector C. FIG.

When learning the estimator, the learning unit 2200 inputs the identification image obtained from the predetermined region i of the learning image _In to the CNN, and converts the value of the output signal obtained in the output layer 640 into teacher information. Compare with Here, X ⁿ _i is the feature amount obtained by inputting the predetermined region i of the learning image I _n to the feature extraction unit 610, and the l-th element in the output layer 640 obtained as a result of inputting this to the estimator 620 Let y _l (X ⁿ _i ) be the output signal of . Also, the teacher signal at the l-th output element in the output layer 640 is represented by the l-th element c(n, i, l) of the mixed state C _ni . In this case, the error between the output signal and teacher information is calculated as follows.

CNN can be trained by sequentially back-propagating the errors obtained in this way from the output layer to the input layer using the error back-propagation method. For example, a stochastic gradient descent method or the like can be used to update the weight coefficients of each layer in the CNN. A random value can be used as the initial value of the weighting factor of the CNN, or a weighting factor obtained by learning about some task may be used. For example, in the image classification task, training images with class labels assigned to each image are used, but in the segmentation task, training images with class labels assigned to each pixel are used. The burden for humans to prepare is large. On the other hand, training images for image classification tasks are publicly available and can be easily obtained. For example, in ILSVRC (ImageNet Large-scale Visual Recognition Challenge), 1.2 million learning images for image classification tasks are published. Therefore, even if the CNN is trained for such an image classification task, and the weighting coefficients obtained by this learning are used as initial values, learning for the mixed state estimation task as in the present embodiment is performed. good.

Although the estimator using CNN has been described here, the configuration of the estimator is not particularly limited. FIG. 8B shows another example of a configuration that estimation section 1200 can use for processing. FIG. 8B shows a portion corresponding to the processing performed by the feature extraction section 610 and a portion corresponding to the processing performed by the estimator 650 . The estimator 650 gives a regression value for one feature amount obtained by connecting the output signals of each layer in the feature extraction unit 610 . Methods used by the estimator 650 include, for example, SVR (Support Vector Regression) and logistic regression, but are not particularly limited. The training images can then be used to train the regression function used by this estimator 650 . For example, the regression function can be learned so as to minimize the error function based on the error between the output signal and the teacher information. Alternatively, the CNN may be trained in advance using the configuration shown in FIG. 8A, and then only the estimator 650 may be trained using the feature amount based on the output signal of each layer of the CNN. Here, if the estimator 650 is composed of a fully-connected multi-layer neural network, the CNN and the estimator 650 can be trained simultaneously using the error backpropagation method, as in the configuration of FIG. 8(A). .

Also, the feature extraction unit 610 can extract feature amounts using another feature extraction method such as HOG or SIFT. The estimator can also use discriminant functions such as SVR, logistic regression, or multi-layer neural networks to estimate the mixture state. Thus, in one embodiment, any combination of feature extraction and estimation techniques can be used. Even in such a case, the estimator can be trained according to the conventional method. The estimator parameters obtained by learning in step S 2200 are stored in estimator storage section 5200 .

A method of identifying the mixed state of the input image using the estimator trained in this way will be described with reference to the flowchart of FIG. 2(B). In S1100, the image acquisition unit 1100 acquires an input image to be mixed state identification target. The image acquisition unit 1100 can also acquire image data before development obtained from an imaging device.

Hereinafter, an image that is an object for estimating a mixed state in a predetermined region of an input image is called a target image. The image acquisition section 1100 can set a plurality of areas in the input image according to a predetermined area setting pattern. A partial image of the input image included in each of the set regions is the target image. The target image is a partial image of a predetermined size according to the identification unit, and its setting method is not particularly limited. For example, as in learning, the input image can be divided into a plurality of rectangular regions of a predetermined size (eg, 16×16 pixels), and determination can be made for a plurality of target images in each rectangular region. On the other hand, the determination may be performed on the target image in a partial area of the input image.

In step S1200, estimation section 1200 uses feature extraction section 610 to extract a feature amount from the target image in a predetermined region of the input image obtained in S1100. In addition, the estimating unit 1200 reads the learned estimator 620 from the estimator storage unit 5200 and inputs the feature amount to the estimator 620, thereby obtaining information indicating the state of mixture between regions of different attributes in the target image. Generate. Thus, estimation section 1200 estimates the mixed state of the target image in the input image acquired in step S1100. FIG. 8A shows an example of CNN that can be used for processing by the estimator 1200 . FIG. 8A shows a portion corresponding to the processing performed by the feature extraction unit 610. Here, the signal in a predetermined region in the input image is forward propagated to each layer, and the feature amount X _i of the target image is extracted. Also, FIG. 8A shows a portion corresponding to the estimator 620, where the output signal at the output element 621 assigned to each element of the mixture state vector is determined from the obtained feature quantity X _i is generated. The value of the output signal of each element l is the value of each element y _l (X _i ) of the mixed state vector.

In step S1300, output section 1300 outputs the estimation result obtained in step S1200. The processing performed by the output unit 1300 depends on how the identification result is used, and is not particularly limited. An example of processing using information indicating a mixed state is given below.

For example, the image processing for each region of the input image can be changed according to the blending state in that region. In this case, the output unit 1300 can output the mixed state of each region to the image correction application.

As another example, camera focus control can be performed according to the mixed state. For example, a focus control device for an imaging device with multiple ranging points can include an acquisition unit and a control unit. The acquisition unit acquires information indicating an area ratio of a region having a specific attribute in an image obtained by the imaging device, with respect to each region corresponding to each of the plurality of ranging points. Then, the control unit weights the plurality of distance measuring points according to the area ratio, and performs focus control of the imaging device. More specifically, when performing multi-point ranging AF, it is possible to increase the weight of ranging points having more subject components to be focused. For example, when performing focus control that emphasizes the foreground, it is possible to increase the weight of AF points with more foreground components. The weight of the distance measurement point can be increased. Such a focus control device may acquire information indicating a mixed state from the above information processing device, may have the above configurations provided in the above information processing device, or may have the above configuration. Information indicating the mixed state generated by a method different from that may be obtained.

As yet another example, camera exposure control may be performed in response to the blend state. For example, an exposure control device for an imaging device can comprise an acquisition unit, a calculation unit, a selection unit, and a control unit. The obtaining unit can obtain an image obtained by the imaging device and information indicating an area ratio of a region having a specific attribute in each region of the image. The calculation unit can calculate the area ratio of the area of the specific attribute in the entire image. The selection unit can select an exposure control algorithm according to the calculated area ratio. The control unit can use the selected exposure control algorithm to control the exposure of the imaging device. More specifically, the area of the sky can be calculated based on the mixed state when different exposure controls are performed according to the area of the sky in the field of view. In this case, as in the conventional technique, most of the area where the sky and branches are mixed is determined to be the foreground, or most of the area is determined to be the sky, thereby reducing the area of the sky. It can be expected to reduce the possibility that the values differ greatly from the actual values.

Although the case where still images are used as learning images and input images has been described here, moving images can also be used as learning images and input images. In this case, the definition of mixed state is extended in the temporal direction. For example, if a predetermined region of 16×16 pixels and 5 frames are used as identification units, the mixed state can be defined with respect to 16×16×5 voxels. For example, by extending the above example of expressing mixed states using area ratios, it is possible to express mixed states using volume ratios.

In this embodiment, the input image (and training image) was divided into multiple regions, each containing multiple pixels, and the mixture state within the regions was estimated. According to such processing, compared to the case of estimating the class for each of all pixels, the number of times of estimation processing is reduced, and speeding up of processing can be expected. On the other hand, the mixture state can also be estimated for each pixel of the input image. That is, a plurality of subjects belonging to different classes may appear in one pixel, and it is possible to estimate the mixed state of the subjects of each class in the subject area corresponding to this one pixel.

In this embodiment, the information indicating the mixed state was obtained as a scalar value or a vector composed of multiple scalar values. On the other hand, the information indicating the mixed state can be information selected from three or more values. For example, the information indicating the mixed state of the classes "empty" and "non-empty" in the given area is a value indicating that the given area is composed of "empty", It can be a value that indicates, or a value that indicates that "empty" and "non-empty" are mixed in a given region. Information indicating such a mixed state can also be used in the above-described processing example and the fourth and fifth embodiments described later.

[Embodiment 2]
In the description of the first embodiment, it is assumed that the class label is set for each pixel of the learning image. However, it takes time to set the class label for each pixel. In the second embodiment, a method of reducing the user's work of inputting class labels for learning images will be described. In this embodiment, the data acquisition unit 2100 automatically calculates the class label for each pixel based on the class label input for each region of the learning image.

The basic configuration of the learning device according to this embodiment will be described below with reference to FIG. The configuration of the image processing apparatus according to this embodiment is the same as that of the first embodiment, and the description thereof will be omitted. In the present embodiment, the learning data storage unit 5100 stores, in addition to the identification image, a region of the first attribute, a region of the second attribute, and regions of the first attribute and the region of the second attribute in the identification image. information indicating a mixed area in which For example, the learning data storage unit 5100 stores learning data including a learning image and a class label assigned to each region on the learning image. Here, an area in which a plurality of classes are mixed is given a class label indicating that it is a mixed area.

The data acquisition unit 2100 reads learning data from the learning data storage unit 5100 . That is, in addition to the identification image, the data acquisition unit 2100 obtains an identification image including a first attribute area, a second attribute area, and a mixture of the first attribute area and the second attribute area in the identification image. Gets information indicating the mixed area

The refinement unit 2300 determines the attribute of each pixel in the mixed area based on the pixel value of the pixel included in the area of the first attribute and the pixel value of the pixel included in the area of the second attribute. For example, the detailing unit 2300 calculates supervised information indicating a mixed state for an area given a class label indicating a mixed area. Details will be described later. The learning unit 2200 performs learning of the estimator in the same manner as in the first embodiment using the learning image and the teacher information of the mixed state.

The flow of processing performed by the learning device in this embodiment will be described with reference to FIG. 2(C). In step S2100, the data acquisition unit 2100 reads the learning image and the class label data from the learning data storage unit 5100 as learning data. A plurality of learning images and class label data for each are prepared in advance in the learning data storage unit 5100 .

Here, the class label data in this embodiment will be explained. A learning image 500 is shown in FIG. 9A, and class label data 400 for the learning image 500 is shown in FIG. 9B. In this example, the training image 500 is composed of a sky region 410, a non-sky region 420, and a mixed region 430. Pixels in each region are assigned classes of “sky,” “non-sky,” and “mixed.” attached as a label. Thus, in the learning image 500, a single-class area and a multi-class area are set.

These class labels can be entered in advance by a human using a tool or the like. For example, the operator can determine sky and non-sky regions of the training images. At that time, in a place where the branches of the foreground tree are fine and complicated, accurately separating the sky area and the non-sky area requires a heavy workload for the operator. Therefore, the operator can assign a class label of "mixed" to an area in which a plurality of classes are mixed in this way.

Here, an area in which "empty" and "non-empty" are mixed has been described, but as described in the first embodiment, the class definition is not limited to such. Also, when there are three or more classes, it is possible to define as many types of mixed regions as there are combinations of classes. For example, as shown in FIG. 5, when three classes of "sky", "plant", and "artificial object" are defined, "space and plant mixed area" and "sky and artificial object mixed area" are defined. , “mixed area of plants and artifacts” and “mixed area of sky, plants and artifacts” can be defined. In the following, a case where two classes of "empty" and "non-empty" are defined will be described as an example.

In step S2300, the detailing section 2300 refines the class label for the mixed area. Specifically, the detailing section 2300 sets a class label for each pixel in the mixed area. Here, the refinement unit 2300 determines the attribute of each pixel in the mixed area based on the pixel value of the pixel included in the area of the first attribute and the pixel value of the pixel included in the area of the second attribute. do. For example, the detailing unit 2300 can determine the class label of the mixed area with reference to the color information of each class. As a specific example, the detailing unit 2300 extracts the RGB values of each pixel for each of the sky region and the non-sky region in the learning image _In and plots them in the RGB color space. Sky areas and non-sky areas other than mixed areas are shown in the learning data. Refinement section 2300 then estimates a mixed Gaussian distribution for each of the sky region and the non-sky region. Then, for each pixel in the mixed region, the likelihood of the sky region can be determined based on its RGB value and the Gaussian mixture of the sky region, and the likelihood of the non-sky region can be determined based on its RGB value and the Gaussian mixture of the non-sky region. We can estimate the likelihood of being in a region. The refinement unit 2300 can then assign a class label of "empty" or "non-empty", whichever has the higher likelihood, to the pixel. In this way, the refinement unit 2300 can refine the class labels in the mixed area. FIG. 9(C) shows class label data 450 refined in this way, in which refined empty regions 460 and non-empty regions 470 are represented.

Based on the class label data detailed in this way, the detailing section 2300 calculates teacher information representing the mixed state of the identification region that is the identification unit. The definition and calculation method of the identification area and the teacher information indicating the mixed state are as described in detail in the first embodiment, so detailed description is omitted here. Note that it is not essential that the detailing unit 2300 refines the class label. For example, it is possible to estimate the mixed state in the mixed region based on the RGB value distribution of the pixels in the mixed region within the identification region and the mixed Gaussian distribution of the sky region and the non-sky region. Teacher information representing the mixed state in the identification region may be calculated.

As a modification, a mixed state may be set for an area in which a plurality of classes are mixed in the learning data. For example, the operator can input information indicating the area ratio of the class, such as "the percentage of non-empty areas is 30%" for a specific area. In this case, the detailing section 2300 can calculate teacher information representing the mixed state for the identification region, which is the identification unit, without estimating the class label for each pixel. On the other hand, the refinement unit 2300 can also refer to the mixture state to estimate the class label of each pixel of the input image. In this case, similar to the fifth embodiment described later, the degree of similarity between the information indicating the mixed state that can be calculated from the learning data and the information indicating the mixed state that is calculated based on the estimated attribute of each pixel An estimate can be made using an evaluation value that increases as .

[Embodiment 3]
In the first and second embodiments, the description has been given on the premise that the identification area, which is the identification unit, is set in advance as a rectangular area or a small area. On the other hand, the size and cutting method of the identification area can be changed based on various shooting information. For example, in highly blurred areas, fine texture information is lost, so there is a possibility that the mixed state estimation accuracy can be improved by estimating a wider identification area.

The imaging information includes information specific to the imaging device and information specific to the captured image. The information specific to the imaging device includes the size of the sensor or the diameter of the permissible circle of confusion, the brightness of the optical system or the focal length, and the like. Information unique to the captured image includes aperture value, focus distance, Bv value, RAW image, exposure time, gain (ISO sensitivity), white balance coefficient, distance information, location information such as GPS, and time information such as date and time. , etc. In addition, the information specific to the captured image includes the gravity sensor value, acceleration, geomagnetic direction, temperature, humidity, atmospheric pressure, altitude, etc. at the time of capturing. In addition to visible light, there are imaging systems that can obtain information on infrared light and ultraviolet light. The imaging information obtained differs depending on the specifications of the imaging apparatus. The shooting information may be information attached in association with the input image when the input image was shot, information indicating the state of the imaging device when the input image was shot, or information measured by the imaging device when the input image was shot. Also, the shooting information may be information representing the characteristics of the input image detected by the imaging device when the input image was shot. Also, the shooting information is information different from the data of the input image itself.

The basic configuration of the learning device according to the third embodiment will be described along FIG. 1(D). Learning data is stored in the learning data storage unit 5100 in advance. In the present embodiment, the learning data includes learning images, shooting information corresponding to each learning image, and mixed teacher information given to regions of various sizes on the learning images. The data acquisition unit 2100 reads learning images, shooting information, and teacher information from the learning data storage unit 5100 . The learning unit 2200 learns an estimator for estimating the mixture state using the learning image and the teacher information of the mixture state, and stores the obtained estimator in the estimator storage unit 5200 . Here, the learning unit 2200 generates the first estimator by learning using the identification image in the predetermined area set according to the first area setting pattern, and the predetermined area set according to the second area setting pattern. A second estimator is generated by learning using the identification image at . The evaluation unit 2400 uses the confirmation data read from the confirmation data storage unit 5400 to evaluate the estimation accuracy of each estimator obtained by learning. Then, the evaluation section 2400 generates an area setter based on the imaging information and the estimation accuracy, and stores it in the setter storage section 5300 .

Next, an overview of the configuration of the image processing apparatus will be described with reference to FIG. The image acquisition unit 1100 acquires an input image and shooting information. The area setting unit 1400 selects an area setting pattern to be used for setting the target image from among a plurality of area setting patterns according to the imaging information. In the present embodiment, the area setting unit 1400 reads the area setter from the setter storage unit 5300 and sets the area serving as the identification unit according to the imaging information. The estimator 1200 reads the estimator from the estimator storage unit 5200, and uses the estimator to estimate the mixed state of the target image in the predetermined region set according to the set discrimination unit.

A detailed description of the processing in this embodiment is provided below. First, processing during learning will be described with reference to the flowchart of FIG. 2D). In step S2100, the data acquisition unit 2100 reads the learning image, the shooting information, and the teacher information of the mixed state from the learning data storage unit 5100 as learning data.

In step S2200, the learning unit 2200 uses the learning image acquired by the data acquisition unit 2100 and the teacher information of the mixed state to learn an estimator for estimating the mixed state. As described above, in the present embodiment, identification units are set according to each of a plurality of types of area setting patterns. That is, various areas are prepared as identification units. For example, it is possible to prepare a plurality of patterns of identification units having different sizes, such as rectangular areas of 3×3, 9×9, and 15×15. As described in the first embodiment, identification units are not limited to rectangular areas. For example, as described in the first embodiment, it is possible to prepare a plurality of parameters used when setting small regions by region division as a plurality of region setting patterns.

Even at the same position on the image, teacher information in a mixed state may change due to differences in area setting patterns. FIG. 3C shows rectangular regions 551, 552, and 553 of various sizes at the same position in the training image. In the smallest rectangular region 551, the empty:non-empty area ratio is r=1. On the other hand, the rectangular areas 552 and 553 each include a non-empty area, so the area ratios are r=0.9 and r=0.8, respectively.

A learning unit 2200 learns an estimator corresponding to each region setting pattern. That is, the learning section 2200 learns the estimator corresponding to the region-of-interest setting pattern based on the identification region set according to the region-of-interest setting pattern and the teacher information given for this identification region. As a result, learning section 2200 generates an estimator corresponding to each of the plurality of region setting patterns. For example, if the region setting pattern index is q and the total number of region setting patterns is Q, Q types of estimators yq can be obtained by learning. Training of the estimator can be performed in the same manner as in the first embodiment. As an example, each estimator yq can perform mixture state estimation according to a regression function fq(X) (q=1, . . . , Q). The estimator obtained by learning is stored in estimator storage section 5200 .

In step S2300, evaluation section 2400 evaluates the identification accuracy of the estimator obtained in step S2200 together with the imaging information, and generates an area setter. For example, the evaluation unit 2400 can evaluate the identification accuracy of each estimator using verification images associated with teacher information and imaging information. Then, the evaluation unit 2400 generates information indicating an estimator corresponding to specific imaging information so that good identification accuracy can be obtained when determining an identification image associated with predetermined imaging information. can be done.

The shooting information includes information obtained for each pixel of the learning image. Also, new photographic information can be obtained by combining the photographic information. For example, if the distance Z(p) from the lens surface to the object at the pixel position p and the focal length f of the optical system are obtained as shooting information, the image magnification S(p) can be calculated.

Further, when the F value of the optical system, the focal length f, the in-focus distance Z _f at the time of shooting, and the distance Z(p) to the subject at the pixel position p are obtained as shooting information, the blur amount B (p) can be obtained.

Furthermore, when the values r(p), g(p), and b(p) at each pixel position p of the RAW image, the exposure time T, the gain G, and the aperture value F are obtained as shooting information, the pixel position p can obtain the absolute value of the incident light amount BV(p) at .

A case of generating an area setter using the blur amount B(p) at the pixel position p as the imaging information will be described below. However, the photographic information to be used is not limited to this, and other photographic information such as the image magnification S(p) or the incident light amount BV(p) may be used. Also, a plurality of shooting information may be combined, for example, the amount of blur B(p) and the amount of incident light BV(p) may be combined and used.

First, the evaluation unit 2400 divides the blur amount B into a plurality of bins and generates a table regarding the region setting pattern q. In this example, the blur amount B is divided into four bins: less than 2, 2 or more and less than 3, 3 or more and less than 4, and 4 or more. Three types of 3×3, 9×9, and 15×15 are used as the area setting pattern q, and a 3×4 table is obtained.

Next, the evaluation section 2400 reads confirmation data from the confirmation data storage section 5400 . The confirmation data, like the learning data, includes a plurality of confirmation images, class label data for each confirmation image, and shooting information. Here, the total number of confirmation images is represented by _Nv , and the v-th confirmation image is represented by Iv ( _v =1, . . . , _Nv ).

The evaluation unit 2400 extracts the feature amount in the area i serving as the identification unit in the confirmation image according to each of the area setting patterns q, and inputs it to the corresponding estimator. In this way, it is possible to obtain the estimated value y _q (X ^ν _i ) of the mixed state of the region i in the verification image Iν when the region setting pattern q is used. At this time, the squared error for the mixed state teacher information c _q (ν, i) can be expressed as follows.

ここでδ_Ｂ（ν，ｉ）は、確認画像Ｉνの領域ｉの中心位置におけるボケ量がビンＢの範囲内であるときに１、そうでないときに０を返すものとする。 Further, the mean squared error MSE (B, q) in the bin (B, q) for the combination of the blur amount B and the region setting pattern q is expressed as follows.

Here, δ _B (ν, i) returns 1 when the amount of blur at the center position of the region i of the verification image Iν is within the range of the bin B, and 0 otherwise.

The confidence T(B,q) for bin (B,q) can then be defined as 1 minus the root mean square error.

In this way, the evaluator 2400 can obtain a table of confidences T(B, q) for each bin (B, q). An example of the table thus obtained is shown below. The evaluation unit 2400 stores the table thus obtained in the setter storage unit 5300 as an area setter.

In this embodiment, the obtained table is stored in the setter storage unit 5300 as the area setter. On the other hand, the evaluation unit 2400 uses the value of the reliability T (B, q) as teacher information, and applies a regression function g _q (B) that outputs the reliability T for the blur amount B as a regression value to each region setting pattern q. and use it as the area setter.

Processing for estimating the mixed state of the input image using the mixed state estimator and region setter thus obtained will be described with reference to the flowchart of FIG. 2(E). In step S1100, the image acquisition unit 1100 acquires image data and shooting information obtained by the imaging device.

In step S1400, the area setting section 1400 reads the area setting device from the setting device storage section 5300 and determines the area setting pattern to be used according to the imaging information. For example, according to the following formula, the region setting unit 1400 sets region setting patterns q _win can choose. Note that the blur amount B(i) refers to the blur amount at the center position of the region i of the input image I. Although the specific processing is not particularly limited, for example, when the input image I is divided into a plurality of regions according to one region setting pattern, and the one region is subdivided according to another region setting pattern, the reliability becomes higher. , this subdivision can be performed. As another example, it is possible to connect regions with similar amounts of blur, and perform region division using a region setting pattern according to the amount of blur for each connected region.

In step S1200, the estimation unit 1200 reads the estimator from the estimator storage unit 5200 and estimates the mixture state at each position of the input image. Specifically, the estimating unit 1200 extracts the feature amount of the image of the predetermined region set at each position p, and inputs the extracted feature amount to the estimator, thereby estimating the mixed state at this position p. can do. Here, the predetermined area for each position p is set according to the area setting pattern q _win determined in step S1400. As described above, in this embodiment, estimators corresponding to each of a plurality of area setting patterns are generated. Therefore, estimation section 1200 can use an estimator selected from a plurality of estimators according to the region setting pattern determined in step S1400. For example, y _qwin is chosen as the estimator at position p, and the mixture state estimate for the given region at position p is obtained as y _qwin (X _i ). FIG. 3D shows an example in which the region setting method is changed depending on the position on the image, and each rectangle represents one region that is input to the estimator.

Since the processing related to step S1300 is the same as that of the first embodiment, description thereof is omitted. As in the present embodiment, by changing the method of setting areas that serve as identification units for estimating a mixed state using photographing information, it is possible to estimate a mixed state with less error.

[Embodiment 4]
In Embodiments 1 to 3, the mixed state in a predetermined area serving as a discrimination unit is estimated. In the fourth embodiment, a method of obtaining a detailed segmentation result by subdividing a region using the obtained mixture state estimation result will be described. The basic configurations of the learning device and the image processing device are the same as those of the first embodiment, and descriptions thereof are omitted.

The processing during learning will be described below with reference to the flowchart of FIG. 2(A). In step S2100, the data acquisition unit 2100 reads from the learning data storage unit 5100 a learning image, teacher information in a mixed state, and learning data.

In step S2200, the learning unit 2200 performs processing similar to that of the third embodiment. That is, areas of various sizes are prepared as identification units. For example, it is possible to prepare a plurality of patterns of rectangular areas of different sizes, such as 1×1, 3×3, 9×9, and 15×15, according to each of the plurality of area setting patterns. Then, the learning unit 2200 can learn the estimator corresponding to each region size using the mixed state teacher information obtained for each region size, as in the third embodiment. That is, if the region size index is q and the total number of region sizes is Q, Q types of estimators yq (q=1, . . . , Q) can be obtained by learning. As an example, each estimator yq can make an estimation of the mixture state according to the regression function fq(X). The estimator yq obtained by learning is written in the estimator storage unit 5200 .

Next, processing at the time of determination will be described according to the flowchart of FIG. 2(F). In step S1100, the image acquisition unit 1100 acquires an input image. In step S1200, the estimator 1200 estimates the mixed state in a predetermined region in the input image using an estimator. Here, estimation section 1200 performs region setting using a first region setting pattern among the plurality of region setting patterns. That is, the estimating unit 1200 determines the mixed state for the first target image having a size that follows the first area setting pattern. In this embodiment, the largest size among the Q types of area sizes is used as the identification unit. In the above example, 15×15 pixels is selected as the discrimination unit, and an estimator corresponding to 15×15 pixels is used as the estimator.

Then, the estimation unit 1200 determines whether or not to redetermine the mixture state of the first portion according to the information indicating the mixture state estimated for the first target image in the first portion of the input image. For example, the estimating unit 1200 determines whether or not to re-determine the mixed state for a predetermined region in which the mixed state has been estimated. For example, the estimating unit 1200 adopts this class estimation result for regions where the class purity is equal to or greater than the threshold.

On the other hand, the estimating unit 1200 re-determines the mixed state for the regions where the class purity is lower than the threshold. In response to the determination to re-determine, the estimating unit 1200 outputs information indicating the mixed state of the second target image having the size according to the second area setting pattern in the first portion. Here, the second target image is smaller than the first target image. That is, the estimating unit 1200 subdivides the regions whose class purity is lower than the threshold according to smaller discrimination units, and estimates the mixture state for each of the subdivided regions again using the estimator. The estimating unit 1200 can, for example, perform repartitioning using a region size that is one step smaller. As described above, in this embodiment, estimators corresponding to each of a plurality of area setting patterns are generated. Therefore, the estimator 1200 can use an estimator selected from a plurality of estimators according to the region setting pattern used for re-segmentation.

Here, the class purity indicates the ratio of pixels to which the same class label is assigned in the area. For example, when the value of the area ratio r shown in the first embodiment is 0.8 or more or 0.2 or less, it can be defined that the class purity is high. Using the map shown in FIG. 7, one can also define high class purity when p1≧0.9 and p2≦0.8.

In this way, by subdividing and re-estimating the mixed state for regions with low class purity, detailed region division can be performed. If the region can no longer be subdivided, or if the class purity of all regions is greater than or equal to the threshold, processing may proceed to step S1300. Since the processing in step S1300 is the same as that of the first embodiment, the description is omitted. Detailed region division results obtained in this way can be used for image quality enhancement processing such as tone mapping or white balance adjustment for each region.

[Embodiment 5]
In the fourth embodiment, a detailed area division result is calculated by subdividing the identification unit, but the area division method is not limited to this method. In the fifth embodiment, a method of obtaining a detailed segmentation result by performing class determination on a pixel-by-pixel basis using the mixed state estimation result for each region will be described.

FIG. 1F shows the basic configuration of the image processing apparatus according to this embodiment. The functions of the image acquiring unit 1100 and the estimating unit 1200 are the same as those of the first embodiment, so description thereof will be omitted. The determination unit 1500 determines attributes of each pixel of the target image. The determination unit 1500 determines the attribute of each pixel based on the evaluation value, and the evaluation indicated by the evaluation value is based on the information indicating the mixture state calculated based on the attribute of each pixel and the information obtained by the estimation unit 1200. The higher the similarity between the information indicating the mixed state, the higher. In this embodiment, the determination unit 1500 estimates the class label of each pixel of the input image based on the mixed state estimation result and the image information. The output unit 1300 outputs information indicating the class label estimated for each pixel of the input image.

Details of the determination processing according to this embodiment will be described with reference to FIG. The processing in steps S1100 and S1200 is the same as that of the first embodiment, so the description is omitted. In step S1500, the determination unit 1500 estimates the class of each pixel of the input image using the mixture state of each region estimated in step S1200. For example, the class of each pixel can be estimated such that the mixed state obtained according to the estimated class of each pixel is close to the mixed state estimated in step S1200. The class of each pixel can be estimated by using the color information of each pixel, for example, so that the colors of pixels belonging to the same class are similar or the colors of pixels belonging to different classes are not similar. can.

As an example of a method of estimating the class of each pixel of an input image, a case of using iterative processing such as CRF (Conditional Random Field) will be described below. CRF considers the pairwise potential based on the similarity between pairs of nodes and the unary potential of each node for a graph consisting of multiple nodes, and sequentially transitions the state of each node to a stable state. It is a way to go. If a CRF is used for image pixel discrimination, a CRF model can be used in which each node corresponds to each pixel in the image.

ここで、右辺第一項のψはｕｎａｒｙ ｐｏｔｅｎｔｉａｌ、右辺第二項のφはｐａｉｒｗｉｓｅ ｐｏｔｅｎｔｉａｌを表す。θ_ψ及びθ_φはそれぞれのパラメータであり、後述する学習処理において算出される。ε_ｉは、画素ｉの近傍画素の集合である。ｇ_ｉｊは画素ｉと画素ｊとの相互関係を表す関数であり、Ｚは正規化項である。判定部１５００は、このモデル式に従って各画素のクラスラベルを更新していくことによって、画像全体のポテンシャルが高い状態へと判定結果を収束させていく。 The conditional probability of class label ci of pixel i on input image I can be expressed by the following equation.

Here, ψ in the first term on the right side represents unary potential, and φ in the second term on the right side represents pairwise potential. θ _ψ and θ _φ are respective parameters, which are calculated in a learning process to be described later. ε _i is the set of neighboring pixels of pixel i. g _ij is a function representing the correlation between pixel i and pixel j, and Z is a normalization term. The determination unit 1500 updates the class label of each pixel according to this model formula, thereby converging the determination result to a state where the potential of the entire image is high.

ここで、ｘ_ｉ及びｘ_ｊはそれぞれ画素ｉ及び画素ｊの色情報であり、ＲＧＢ値を持つ３次元ベクトルで表わされる。βはユーザの定義するハイパーパラメータであって、β＝１などと設定することができる。このように、ｐａｉｒｗｉｓｅ ｐｏｔｅｎｔｉａｌは、時刻ｔにおいて違うクラスに属する画素の色が類似する場合に評価が低くなるように設定することができる。 The pairwise potential can be expressed by the following formula.

Here, x _i and x _j are color information of pixel i and pixel j, respectively, and are represented by three-dimensional vectors having RGB values. β is a user-defined hyperparameter and can be set such as β=1. In this way, the pairwise potential can be set so that the evaluation is low when the colors of pixels belonging to different classes at time t are similar.

ここでｙ_ｃ（Ｘ_ｉ）は、画素位置ｉにおけるクラスｃに関する混合状態推定値である。ｙ_ｃ（Ｘ_ｉ）は、画素位置ｉが含まれる所定領域について推定部１２００が推定した混合状態に基づいて算出することができ、例えば所定領域内における、クラスｃの面積比、エッジ画素率、又はクラスラベル配置パターン等でありうる。このように、ｕｎａｒｙ ｐｏｔｅｎｔｉａｌは、時刻ｔにおいて各画素の属性に基づいて計算される混合状態と、推定部１２００により得られた混合状態と、の類似度が大きいほど評価が高くなる。 The unary potential can be expressed as follows.

where y _c (X _i ) is the mixture state estimate for class c at pixel location i. y _c (X _i ) can be calculated based on the mixture state estimated by the estimating unit 1200 for the predetermined region including the pixel position i. Or it can be a class label arrangement pattern or the like. In this way, the unary potential is evaluated higher as the degree of similarity between the mixed state calculated based on the attribute of each pixel at time t and the mixed state obtained by the estimation unit 1200 increases.

L ⁱ _c (t) is the mixed state of class c in a predetermined region including pixel i at time t when the class label for each pixel transitions according to the CRF. L ⁱ _c (t) is the same kind of information as the mixed state estimated by the estimation unit 1200, and can be calculated by the estimation unit 1200 with reference to the class estimated for each pixel in the predetermined region at time t. Specific examples will be given below according to the example of the mixed state described in the first embodiment. For example, the area ratio r(t) of the class c can be obtained by counting the pixels assigned the class label c in the predetermined region including the pixel i at the time t in the middle of the transition. Further, the edge pixel ratio e(t) can be obtained by extracting and counting edge pixels according to the arrangement of class labels in a predetermined area. Further, the class label arrangement pattern p(t) can be obtained by determining which of the maps shown in FIG. 7 the class label arrangement in the predetermined area is closest to. As described in Embodiment 1, L ⁱ _c (t) can also be expressed by a combination of these mixed states at time t.

In this way, the similarity between the mixture state in the predetermined region at time t determined based on the pixel-level class label arrangement at time t during the transition and the mixture state estimate value for the predetermined region is expressed as unary It can be expressed as potential. Specifically, the unary potential can be expressed such that the higher the similarity between the mixed state in the predetermined region at time t and the mixed state estimated value for the predetermined region, the higher the evaluation.

Learning processing in this embodiment will be described with reference to FIG. In step S2100, data acquisition section 2100 acquires learning images and teacher data. In step S2200, the learning unit 2200 learns the estimator as in the first embodiment. The learning unit 2200 also determines values of parameters (for example, θ _ψ and θ _φ described above) used when estimating the class of each pixel of the input image. The learning unit 2200 can perform this process using a plurality of learning images and class label data indicating the class of each pixel in the learning image. As class label data, for example, data created as shown in FIG. 9C according to the second embodiment can be used. In this embodiment, the learning unit 2200 can calculate the values of θ _ψ and θ _φ such that the potential for all training images is maximized. That is, the values of θ _ψ and θ _φ that maximize the following expressions can be obtained by the gradient method or the like.

Learning section 2200 stores the obtained parameters in estimator storage section 5200 together with the estimator. In this embodiment, the values of θ _ψ and θ _φ are stored in the estimator storage unit 5200 and used by the decision unit 1500 as described above. The class label data for each pixel thus obtained can be used, for example, when image quality improvement processing is performed for each region in the same manner as in the fourth embodiment.

The method of performing class determination on a pixel-by-pixel basis using the mixed state estimation result is not limited to the above method. For example, similar to the second embodiment, class determination is performed on a pixel-by-pixel basis based on the mixed Gaussian distribution of each class obtained using the region where the class is determined and the similarity of the mixed state. can also

The processing according to the present embodiment can use any of the area ratio, the edge pixel ratio, and the class label arrangement pattern as the mixed state, and usable mixed states are not limited to these. Also, by using a mixed state expressed by combining a plurality of expressions, it is possible to improve the judgment accuracy. For example, by using the edge pixel ratio in addition to the area ratio, we distinguish between simple contours such as the boundary between a building and the sky and complex contours such as the boundary between a branch and the sky. becomes possible.

In this embodiment, the information indicating the mixed state is obtained by the processing of the estimator 1200 . However, the determination unit 1500 can acquire information indicating the mixed state obtained by a different method and determine the attribute of each pixel by a similar method.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

1100: image acquisition unit, 1200: estimation unit, 1300: output unit, 1400: area setting unit, 1500: determination unit, 2100: data acquisition unit, 2200: learning unit, 2300: detailing unit

Claims (20)

an acquisition means for acquiring an input image;
extracting means for extracting features from the input image;
output means for outputting information corresponding to the area of a region belonging to a specific class in the input image from the estimator to which the features extracted from the input image are input;
The image processing apparatus, wherein the estimator has parameters learned using learning images. 2. The image processing apparatus according to claim 1, wherein said output means outputs information specifying the ratio estimated by said estimator. 2. The image processing apparatus according to claim 1, further comprising control means for controlling imaging by the imaging means based on the result of estimation by said estimator. 4. The image processing apparatus according to claim 3, wherein said control means controls said imaging by said imaging means with respect to exposure, focus, or white balance based on the result of estimation by said estimator. 3. The ratio estimated by the estimator is a ratio of (1) the number of pixels in the region belonging to the particular class, and (2) the total number of pixels in the input image. The image processing device according to . 2. The image processing apparatus according to claim 1, wherein said estimator estimates a ratio of regions belonging to a sky class, a plant class, or a skin class to said input image. obtaining an input image;
extracting features from the input image;
outputting information corresponding to the area of a region belonging to a specific class in the input image from the estimator to which the features extracted from the input image are input;
A method of image processing, wherein the estimator has parameters learned using training images. A program for operating a computer as the image processing apparatus according to any one of claims 1 to 6. 7. The image processing apparatus according to any one of claims 1 to 6, wherein said input image is a divided image obtained by dividing an entire image. 7. The image processing apparatus according to any one of claims 1 to 6, wherein said input image has a plurality of areas belonging to each class. The estimator has parameters learned based on (a) features extracted from a training image and (b) teacher information representing a quantity corresponding to the area of a region belonging to a particular class in the training image. ,
7. The image processing apparatus according to any one of claims 1 to 6, wherein said teacher information represents a numerical value corresponding to a total area area of a plurality of areas belonging to said specific class in said learning image. . wherein the estimator outputs, as an output for the input image having a plurality of regions belonging to the specific class, a numerical value corresponding to a total area area of the plurality of regions belonging to the specific class in the input image. 7. The image processing apparatus according to any one of claims 1 to 6, wherein the image processing apparatus is learned. The information corresponding to the area of the region belonging to the specific class in the input image is a ratio of the area of the region belonging to the specific class in the input image to the area of the input image. 7. The image processing apparatus according to any one of claims 1 to 6. a first acquisition means for acquiring training images for training the estimator;
extracting means for extracting features from the training image;
a second acquiring means for acquiring, as teacher information, information corresponding to the area of a region belonging to a specific class in the learning image;
learning means for learning the estimator using a combination of the features and the teacher information;
A learning device comprising: obtaining training images for training the estimator;
extracting features from the training images;
a step of acquiring, as teacher information, information corresponding to the area of a region belonging to a specific class in the learning image;
and a step of learning the estimator using a combination of the features and the teacher information. an acquisition means for acquiring divided images obtained by dividing an input image;
extraction means for extracting features from the divided images;
output means for outputting information corresponding to the area ratio belonging to a specific class in the divided image from the estimator to which the feature extracted from the divided image is input;
The image processing apparatus, wherein the estimator has parameters learned using learning images. 17. The image processing apparatus according to claim 16, further comprising control means for controlling imaging by the imaging means based on the result of estimation by said estimator. 18. The image processing apparatus according to claim 17, wherein said control means controls said imaging by said imaging means with respect to exposure, focus, or white balance based on the result of estimation by said estimator. an obtaining step of obtaining divided images obtained by dividing an input image;
an extraction step of extracting features from the divided images;
an output step of outputting information corresponding to the area ratio belonging to a specific class in the divided image from the estimator to which the feature extracted from the divided image is input;
A method of image processing, wherein the estimator has parameters learned using training images. A program for operating a computer as the image processing apparatus according to any one of claims 16 to 18.

Images