JP2016099668A - Learning method, learning device, image recognition method, image recognition device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that erroneous detection is sometimes performed regardless of high reliability in a method for identifying the class of each area in an image on the basis of a feature amount extracted from each small area in the image.SOLUTION: A learning image is used to learn a first discriminator, an identification result of a learning evaluation image by the first discriminator is evaluated, and a small area that is hardly identified by the first discriminator is selected. Next, a second discriminator for identifying an area including the selected small area is learned. Lastly, an integrated discriminator is learned by integrating an identification result by the first discriminator and an identification result by the second discriminator.SELECTED DRAWING: Figure 10

Description

  The present invention relates to a technique for detecting a subject in an input image and dividing an area for each subject.

  2. Description of the Related Art Conventionally, processing for dividing a region for each subject and identifying a class related to subject classification is known for subsequent processing such as image scene recognition and image quality correction according to the subject. In the method described in Non-Patent Document 1, an input image is first divided into small regions called SP (superpixels) based on color information and texture information. And the class of each divided | segmented small area | region is identified using the discriminator called Recursive-Neural-Networks (RNNs).

R. Socher, “Parsing Natural Scenes and Natural Language with Recursive Neural Networks”, International Conference on Machine Learning 2011. P. Krahenbuhl, “Efficient Inference in Full Connected CRFs with Gaussian Edge Potentials”, Neural Information Processing Systems 2011. J. et al. Tighe, S .; Lazebnik, “Finding Things: Image Parsing with Regions and Per-Explorer Detectors”, CVPR2013. A. Oliva and A.M. Torralba, “Modeling the shape of the scene: a holistic representation of the spatial envelope”, International Journal of Computer Vision, 2001.

  However, the method of identifying the class of each region in the image based on the feature amount extracted from the small region as in the method of Non-Patent Document 1 has high reliability (the identification score and the identification likelihood are high). ) May be erroneously detected. For example, it is difficult to discriminate a small region having a similar feature amount by a discriminator, such as a small region obtained by cutting a part of the sky and a small region obtained by cutting a part of a blue wall.

  In order to solve the above-mentioned problem, according to the learning method of the present invention, a first learning step of learning a first classifier for identifying a class for each region of an image using a learning image; A learning identification step for identifying a class for each region of the learning evaluation image by the learned first discriminator, and a misidentification region in which the class identification result for the learning evaluation image by the first discriminator is incorrect A selection step of selecting a learning step, a generation step of generating learning data using the selected region including the misidentification region, and a second classifier of learning a second classifier for identifying a class of the generated learning data And 2 learning steps.

  According to the above configuration, according to the present invention, when the recognition target image is recognized by the image recognition device, even if the recognition target image includes a small region that is difficult to identify the class, the erroneous detection can be reduced, and the image can be accurately displayed. Can be recognized.

1 is a configuration diagram of an image recognition system according to a first embodiment. The figure explaining the recognition object image in connection with 1st Embodiment. The figure which illustrates notionally the process with respect to the recognition target image in 1st Embodiment. 1 is a diagram illustrating a hardware configuration of an image recognition apparatus according to a first embodiment. 1 is a diagram illustrating a functional configuration of an image recognition apparatus according to a first embodiment. The flowchart which shows the image recognition process in connection with each embodiment. The figure explaining the detection method of the example in connection with 1st Embodiment. The figure which illustrates notionally the image recognition process in 1st Embodiment. The figure explaining the integration process in connection with 1st Embodiment. The figure which shows the function structure of the learning apparatus in connection with each embodiment. The flowchart which shows the learning process in connection with each embodiment. The figure which shows the example of the image at the time of learning the 1st discriminator in 1st Embodiment. The flowchart of a 1st discriminator learning process in 1st Embodiment. The figure explaining the process of the misidentification area | region selection process in connection with 1st Embodiment. The flowchart explaining the process which produces | generates a negative example in 1st Embodiment. The figure which shows the learning data learned with a 2nd discriminator in 1st Embodiment. The figure explaining the process which produces | generates a negative example in 1st Embodiment. The flowchart of the process in an integrated discriminator learning process in 1st Embodiment. The flowchart of the process in an integrated discriminator evaluation process in 3rd Embodiment.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a configuration diagram illustrating an image recognition system according to the present embodiment. In the image recognition system of this embodiment, the camera 10 and the image recognition device 20 are connected via a network. Note that the camera 10 and the image recognition device 20 may be configured integrally. An image captured by the camera 10 is output to the image recognition device 20, and the image recognition device 20 acquires an image output from the camera 10. In the present embodiment, a mode in which the camera 10 captures a scene 30 as shown in FIG. 1 and the image recognition apparatus 20 processes the recognition target image will be described.

  FIG. 2 is a diagram illustrating a recognition target image according to the present embodiment. FIG. 2A shows a recognition target image 100 obtained by photographing the scene 30 with the camera 10. In order to recognize an image, the image recognition apparatus 20 according to the present embodiment identifies a class of each small region of the acquired recognition target image and divides the recognition target image into regions. In the present embodiment, the class is a class category name relating to the classification of the subject such as sky, tree, car, etc., as shown in FIG. 2B, and the class is for each pixel of the recognition target image. Assigned.

  FIG. 3 is a diagram conceptually illustrating processing for identifying a class of each region of the recognition target image in the present embodiment. As shown in FIG. 3A, a class is identified by setting a region obtained by dividing a recognition target image in the vertical direction and the horizontal direction as a unit region for processing. In this embodiment, the unit of the class identification process is a pixel unit. FIG. 3B is an enlarged view of the upper left part of FIG. 3A, and shows a state where the sky category is assigned to each pixel 103. As described above, in this embodiment, a region is divided by assigning a class to each pixel 103 of the recognition target image 100.

  FIG. 4 is a block diagram illustrating a hardware configuration of the image recognition apparatus 20. The CPU 401 controls the entire image recognition apparatus 20. When the CPU 401 executes a program stored in the ROM 403, the HD 404, or the like, the functional configuration of the image recognition device 20 and the processing of the flowchart relating to the image recognition device 20 described later are realized. The RAM 402 has a storage area that functions as a work area where the CPU 401 develops and executes a program. The ROM 403 has a storage area for storing programs executed by the CPU 401. The HD 404 has a storage area for storing various types of data including various programs required when the CPU 401 executes processing, data relating to threshold values, and the like. The operation unit 405 receives an input operation by the user. The display unit 406 displays information of the image recognition device 20. A network I / F 407 connects the image recognition apparatus 20 and an external device.

  FIG. 5 is a diagram illustrating a functional configuration of the image recognition device 20 according to the present embodiment. As described above, the image recognition apparatus 20 of the present embodiment is connected to the camera 10 via a network. The image recognition apparatus 20 includes an acquisition unit 501, an identification unit 502, and a detection unit 504. Further, the image recognition apparatus 20 includes a first discriminator holding unit 503, a second discriminator holding unit 505, and a first integrated discriminator holding unit 507 as means for storing and holding necessary information. The first discriminator holding unit 503, the second discriminator holding unit 505, and the first integrated discriminator holding unit 507 may be provided in a non-volatile storage device that is separate from the image recognition device 20. Details of these functions of the image recognition apparatus 20 will be described later with reference to FIG.

  FIG. 6A is a flowchart showing an image recognition process when processing an image to be recognized in the present embodiment. First, an outline of the process in each step will be described.

  In the acquisition step S110, the acquisition unit 501 receives the recognition target image captured by the camera 10 as input data. In the detection step S120, the detection unit 504 uses the second discriminator stored in the second discriminator holding unit 505 to detect a case learned in advance by the second discriminator. A case detection method and a learning method will be described later. When there are a plurality of second discriminators, the case is detected by applying a plurality of times. The detection result is transmitted to the integrated identification unit 506.

  In the identifying step S130, the identifying unit 502 identifies the class of each region of the recognition target image using the first classifier stored in the first classifier holding unit 503. The class identification result of each area is transmitted to the integrated identification unit 506. In the integrated identification step S140, the integrated identification unit 506 integrates the detection result by the second identifier and the identification result by the first identifier, and executes class identification for each region of the recognition target image.

  Next, more specific processing of each processing will be described according to the flowchart shown in FIG.

  In the acquisition step S110, the acquisition unit 501 receives the recognition target image captured by the camera 10 as input data. This recognition target image may be captured in advance and stored in an external device. In this case, the acquisition unit 501 acquires the recognition target image from the external device.

  Next, in the detection step S120, the detection unit 504 uses the second discriminator stored in the second discriminator holding unit 505 to detect a case that has been learned in advance by the second discriminator. A case study method will be described when explaining the learning process.

  FIG. 7 is a diagram for explaining a case detection method according to this embodiment. As shown in FIG. 7A, the case is detected by scanning the detection window 110 for the recognition target image 100. FIG. 7B shows a state in which a case to be detected is detected in the detection window 110.

  Then, a mask 111 for indicating the existence position of the case is stored at the time of learning, and an area corresponding to the case is obtained by superimposing the mask 111 on the detection position of the recognition target image 100 as shown in FIG. Is extracted. In the mask 111, 1 is recorded for pixels in which cases are present, and 0 is recorded for pixels in which no cases are present. Alternatively, a real value of 0 to 1 may be recorded as the existence probability.

The result of performing the detection process in the detection window is output as a detection score (or likelihood, a real value of 0 to 1) in each pixel. Specifically, the detection result (score) in each pixel is obtained by multiplying the score (likelihood) output from the detector by the existence probability of the case recorded in the mask 111. When the detection result (score) in each pixel is S _D (x, y), the following equation 1 is obtained.
S _D (x, y) = S (x ₀ , y ₀ ) · Mask (x−x ₀ , y−y ₀ )...
Here, S (x ₀ , y ₀ ) is a score (likelihood) output by the detector (second discriminator) at x ₀ , y ₀ . Mask (x−x ₀ , y−y ₀ ) represents the existence probability of cases in x and y. When there are a plurality of detection results (when a large number of detection positions are detected by one detector and a plurality of detection results exist), the plurality of detection results may be averaged at each pixel. Thereby, a real value of 0 to 1 is assigned to each pixel of the recognition target image. Hereinafter, this is referred to as a detection score map.

  When detecting the case, the contour may be refined using a graph cut or the like after the mask 111 is superimposed on the detection position of the recognition target image 100. In the case of refinement, a detection score may be calculated for the recognition target image 100 as shown in Equation 1 after that. Although an example in which only one case is detected by the detector has been described here, a plurality of cases may be detected. In that case, a detection score map is held for each case.

  Next, in the identifying step S130, the identifying unit 502 classifies the recognition target image using the first classifier stored in the first classifier holding unit 503. FIG. 8 is a diagram showing the process of the identification step S130 for the recognition target image. In the present embodiment, as shown in FIGS. 8A and 8B, a class is identified for each small region 101 formed by dividing the captured recognition target image 100. Here, the small area means an area composed of one or more pixels and a predetermined value or less in the image. In this embodiment, the image is divided into small areas called SP (superpixels) as described in Non-Patent Document 3. Other block divisions may be used.

  In the present embodiment, the first discriminator corresponds to a discriminator that extracts a feature amount from the small area 101 and receives the feature amount. As such a discriminator, for example, Recursive-Neural-Networks (RNNs) shown in Non-Patent Document 1 can be used. Alternatively, it may be a discriminator that inputs a feature value such as SupportVectorMachines (SVMs) and outputs a discrimination result. The identification result of the present embodiment takes a value range from 0 to 1 for each class defined in advance, and the higher the value, the higher the reliability. The learning method of the first discriminator will be described when explaining the processing at the time of learning.

  In the present embodiment, a method has been described in which the recognition target image is divided into small areas in advance, and class identification is performed by the first classifier for each small area. However, the present invention is not limited to this. For example, region division and class identification may be performed simultaneously using a conditional random field CRF (Conditional-Random-Field) as shown in Non-Patent Document 2. Further, in the present embodiment, as shown in FIG. 8B, each small region is arranged on the recognition target image without overlapping, but may be overlapped. The integration method in that case will be described later.

  Next, in the integrated identification step S140, the integrated identification unit 506 integrates the detection result in the detection step S120 and the class identification result in the identification step S130 and outputs the final result. FIG. 9 is a diagram for explaining the integration processing in the present embodiment. 9A is a recognition target image, FIG. 9B is a detection result by the second discriminator, FIG. 9C is a discrimination result by the first discriminator, and FIG. 9D is a final image by integration. Represents the class identification result. The detection result here corresponds to a case where the mask 111 is superimposed on the recognition target image at the detection position of the case detected by the detection window 110 as shown in FIG. The detection score is calculated for each pixel as shown in Equation 1 above.

  As the integration method in the present embodiment, one of the following two methods can be used.

  As a first method, a mask corresponding to a detection result equal to or higher than a predetermined threshold is superimposed on the score (likelihood) output by the detector (second discriminator) in the detection step S120, and the others The result of discrimination by the first discriminator is adopted for the area of. As shown in FIG. 9B, a mask 111 corresponding to the detected case is superimposed as shown in FIG. 9D. After all the masks of the detected cases are superimposed, the final class identification result is obtained by superimposing the identification result by the first classifier shown in FIG. 9C on the other regions. If the small areas overlap each other, once the class is identified for each small area, the class identification results of the small areas belonging to each pixel may be determined by averaging or voting. In addition, when the mask obtained by the detection process and the small area used for the first discriminator overlap, either result may be preferentially used, or the reliability of only the overlapping area High results may be employed.

  As the second method, an integrated classifier learned in advance is used. A learning method of the integrated classifier will be described later, and here, a method of using the integrated classifier at the time of identification will be described. In the present embodiment, an integrated discriminator for identifying the class of each pixel is used with the detection result of the detection step S120 for each pixel and the class identification result for the identification step S130 for each pixel as inputs. The identification result in each pixel is a result of class identification for a small area including the pixel.

  Assuming that the number of second classifiers that detect cases is Nd and the number of classes identified by the first classifier is C, the number of dimensions of the input vector input to the integrated classifier at each pixel is Nd + C dimensions. For the input vector, a C-dimensional output value corresponding to the number of classes C to be finally output is output by the integrated discriminator. Here, the number of classes identified by the first discriminator is the same as the number of classes finally output, but may be different.

  Here, the integrated identification is performed for each pixel. However, the integrated identification may be performed for each small area used in the identification step S130, or the small area and the block may be defined by defining the small area and the block for the integrated identification. You may perform an integration process for every. In that case, before inputting to the integrated classifier, the discrimination result by the detection score map and the first classifier is averaged for each small region or block.

  By the method described above, the detection result in the detection step and the identification result in the identification step are integrated, and finally the result of FIG. 9D is obtained. Thereafter, the class of each pixel may be re-estimated using a conditional random field CRF (Conditional-Random-Field) as disclosed in Non-Patent Document 2.

  Next, a learning method for the first discriminator and the second discriminator used in the detection step S120 and the discrimination step S130 in the present embodiment will be described.

  FIG. 10A is a diagram illustrating a functional configuration of the learning device 300 according to the present embodiment. Note that the hardware configuration of the learning apparatus 300 is the same as that of the image recognition apparatus 20 shown in FIG. Here, the learning apparatus 300 is described as being configured separately from the image recognition apparatus 20 of FIG. 5, but the learning apparatus 300 is configured integrally with the image recognition apparatus 20, and the learning apparatus 300 is included in the image recognition apparatus 20. These functional units may be included. That is, the CPU 401 of the image recognition apparatus 20 may execute a program stored in the ROM 403, the HD 404, or the like, so that the functional configuration of the learning apparatus 300 and the processing of the flowchart related to the learning apparatus 300 may be realized.

  The learning device 300 includes a first discriminator learning unit 301, a learning-time discriminating unit 302, an erroneous discrimination region selection unit 303, a second discriminator learning data generation unit 304, a second discriminator learning unit 305, and an integrated discriminator learning unit 306. Have Furthermore, the learning device 300 stores learning data as a means for storing and holding data, a learning image holding unit 351, a learning evaluation image holding unit 352, a first classifier holding unit 353, and a second classifier learning data holding unit 354. The second classifier holding unit 355 and the second integrated classifier holding unit 356 are provided. Details of each function of the learning device 300 will be described later with reference to FIG.

  FIG. 11A is a flowchart showing processing related to learning in the present embodiment.

  First, in the first discriminator learning step T110, the first discriminator learning unit 301 learns the first discriminator using the learning image held in the learning image holding unit 351. FIG. 12 is a diagram for explaining a learning image used for learning of the first discriminator. In the present embodiment, as learning images, for example, an image 50 as shown in FIG. 12A and a correct answer in which the class name of each pixel of the image 50 as shown in FIG. 12B is defined. Data (hereinafter referred to as GT (Grand Truth)) is used. At this time, a plurality of discriminators may be learned. However, here, for simplification of explanation, one discriminator is learned. The first discriminator learned by the first discriminator learning unit 301 is transmitted to the first discriminator holding unit 353.

  Next, in the learning time identification step T120, the learning time identification unit 302 uses the first classifier learned in the first classifier learning step T110, and the learning evaluation image held in the learning evaluation image holding unit 352. Class identification of the area. Here, the learning evaluation image and the learning image 50 described above are distinguished from each other, but the learning evaluation image and the learning image may be the same data. The result of class identification by the learning time identification unit 302 is transmitted to the erroneous identification region selection unit 303.

  Next, in the erroneous identification region selection step T130, the erroneous identification region selection unit 303 selects an erroneous identification region from the class identification result identified in the learning-time identification step T120. The misidentification region selection unit 303 selects the misidentification region by comparing the class identification result with the GT of the learning evaluation image held in the learning evaluation image holding unit 352. The method for selecting the erroneous identification area will be described in detail later.

  Next, in the second discriminator learning data generation step T140, the learning data that the second discriminator learning data generation unit 304 learns with the second discriminator based on the misidentification region selected in the misidentification region selection step T130. Is generated. This generation method will be described in detail later. The learning data generated by the second discriminator learning data generation unit 304 is transmitted to the second discriminator learning data holding unit 354.

  Next, in the second discriminator learning step T150, the second discriminator learning unit 305 learns the second discriminator using the learning data generated in the second discriminator learning data generation step T140.

  Finally, in the integrated discriminator learning step T160, the integrated discriminator learning unit 306 performs the identification result of the first discriminator learned in the first discriminator learning step T110 and the second discrimination learned in the second discriminator learning step T150. The integrated classifier or parameter that integrates the classifier result is learned.

  Next, specific processing of each step will be described according to the flowchart shown in FIG.

  First, in the first discriminator learning step T110, the first discriminator learning unit 301 learns the first discriminator. The first classifier may be any classifier as long as it can identify the class of each pixel as described above. In the present embodiment, description will be made using Recursive-Neural-Networks (RNNs), which is one of classifiers that extract feature amounts from small regions and input the feature amounts. RNNs are described in detail in Non-Patent Document 1.

  FIG. 13 shows a detailed flow of the process of the first discriminator learning process executed by the first discriminator learning unit 301. M in the figure indicates the number of learning evaluation images used for learning of the first discriminator.

  First, at T1201, a list of learning images used for learning of the first discriminator is set.

  Next, in T1202, based on the learning image list set in T1201, each learning image used for learning by the first discriminator is divided into small regions. For example, the image is divided into small regions called SP (superpixels) as described in the image recognition processing identification step S130.

  Next, in T1203, the feature amount of each small area divided in T1202 is extracted. Alternatively, the feature amounts of all the learning images may be extracted in advance, and the feature amounts may be loaded based on the learning image list in this step. The processes of T1202 and T1203 are performed for all small regions of all learning images. As an example of the feature amount, a statistic of color feature or texture feature in each small region may be used. For example, RGB, HSV, Lab, YCbCr color space components, Gabor filter, LoG filter responses are used. be able to. The color feature has 12 dimensions of 4 (color space) × 3 (component). The filter response has a dimension number corresponding to the number of Gabor filters and LoG filters.

  Further, since the characterization is performed for each small area, the statistic is obtained from the characteristic amount obtained for each pixel in each small area. Assume that four statistics are used: average, standard deviation, skewness, and kurtosis. Skewness indicates the degree of asymmetry of the distribution, and kurtosis is a statistic indicating the degree to which the distribution is close to the average. Therefore, the color feature has 48 dimensions of 4 (color space) × 3 (component) × 4 (statistic), and the number of dimensions of the texture feature is (filter response number) × 4 (statistic). In addition, the center of gravity coordinates of the small area, the area of the small area, and the like may be used as the feature amount.

  Next, in T1204, the class definition and the number of classes of the area learned by the first discriminator are set. The number of classes may be two or more. For example, sky, building, tree, load, and body are defined in the learning image in FIG. In this case, the number of classes may be five, or a classifier that identifies two classes, that class and sky, may be learned by combining building, tree, load, and body as one class.

  Next, in T1205, a first classifier that identifies the class defined in T1204 is learned. The learned first discriminator is stored in the first discriminator holding unit 353.

  In the learning time identification step T120, the learning time identification unit 302 identifies the class for the learning evaluation image using the first classifier learned in the first classifier learning step T110. Here, the number of first classifiers is 1, the number of learning evaluation images is N, and class identification is performed a total of N times. In this class identification, the learning evaluation image is divided into small regions defined in the first discriminator learning step T110, and the class is identified based on the feature amount of each small region. Recursive-Neural-Networks (RNNs), which is the first discriminator in this embodiment, outputs the likelihood for each class defined above.

When the number of classes identified by the first identifier is C, classifier result S _R of the first discriminator for each small region is expressed by the following equation (2).
S _R = {S ₁ , S ₂ ,..., S _c }.
Here, each S _c (c = 1, 2,..., C) is a likelihood for each class. In identifying the class, the class with the highest likelihood is assigned to each small region based on the above equation (2). The class identification result for each learning evaluation image is transmitted to the misidentification region selection unit 303.

  Next, in the misidentification region selection step T130, the misidentification region selection unit 303 selects the misidentification region by comparing the identification result for the learning evaluation image with GT (ground truth). FIG. 14 is a diagram for explaining the process of the erroneous identification region selection step. FIG. 14A shows a small area in which a class is identified by the first classifier in the learning evaluation image 120 and the identification result. FIG. 14B shows the GT 130 of the learning evaluation image. The misidentification area selection unit 303 selects the misidentification area 121 by comparing the identification result of FIG. 14A and the GT of FIG. The erroneous identification area selection unit 303 acquires an erroneous identification area for each learning evaluation image.

  Next, in the second discriminator learning data generation step T140, learning in which the second discriminator learning data generation unit 304 learns with the second discriminator based on the misidentification region selected in the misidentification region selection step T130. Generate data. FIG. 15 is a flowchart of detailed processing of the second discriminator learning data generation step T140 executed by the second discriminator learning data generation unit 304. L in the figure is the number of misidentification areas to be selected, and corresponds to the number of learning data to be generated.

  In T1401, the misidentification areas selected in the misidentification area selection step T130 are sorted. Although the learning data may be generated by the second discriminator for all the misidentification areas, here the misidentification areas are sorted in order to generate learning data using a part of the misidentification areas. To do. As the sorting method, for example, the likelihood when classifying each small region of the learning evaluation image or the area of the small region may be used. For example, if a region with a high likelihood is selected, a small region in which the class is mistakenly identified even though the likelihood is high can be selected, and it is difficult to identify only with a feature amount of the small region. A small area can be selected.

  In T1402, one misidentification area sorted in T1401 is selected. In T1403, learning data (positive case) learned by the second discriminator is generated. FIG. 16 shows learning data learned by the second discriminator. FIG. 16A shows an example in which learning data 122 is generated by connecting several small areas close to the misidentification area 121 from the learning evaluation image 120. The small area to be connected is an area close to the misidentification area, and any number of areas of the same class may be connected. FIG. 16B shows an example in which several small regions are further connected. In FIG. 16C, based on the GT 130 as shown in FIG. 16D, an area of the same class including the misidentification area (the car area in the figure) is selected as learning data for the second discriminator. . As a method for selecting learning data, if misidentification is included, the learning data may be cut out by using a rectangle or the like instead of connecting small areas.

  In T1404, a negative case for the learning data (positive case) generated in T1403 is generated. FIG. 17 is a diagram for explaining the processing of T1404 for generating a negative case. FIG. 17A shows learning data 122 generated for the learning evaluation image 120, and FIG. 17B shows a state where a negative case 123 is generated from another learning evaluation image 150. FIG. The negative example is by cutting out a part of the learning evaluation image that is different from the learning evaluation image that detected the misidentification area, using a mask that represents the shape of the learning data generated by connecting several small areas to the misidentification area. Can be generated. In this way, a region cut out in the same shape as the learning data from a plurality of learning evaluation images is created as a negative case. In addition, it is more effective that the negative case is generated from an area of the same class as the class erroneously identified in the misidentification area.

In T1405, it is evaluated whether the discrimination performance is improved by learning the second discriminator from the learning data (correct case) generated in T1403. Specifically, the feature amount is acquired from the learning data (positive case) generated in T1403 and the negative case data generated in T1404, and the distance is calculated. Here, the same feature quantity as that used in the first discriminator learning is used, but other feature quantities may be used. When the generated learning data is S _Posi and the negative case data is S _{Nega_i} , S _Posi may be set to satisfy Equation 3.
arg _i min dist (f (S _Posi ), f (S _{Nega — i} ))> l ₁
In equation (3), dist calculates the distance between two feature quantities, and a histogram distance, Eugrid distance, or the like can be used. Further, f indicates a feature amount in the small area. As expressed by Equation 3, if the distance to the closest negative case data is greater than or equal to the predetermined distance l ₁ , the learning data is adopted, and if the distance is equal to or smaller than the predetermined distance l ₁ , learning data is generated again. To do. l ₁ may be an arbitrary value equal to or greater than 0. For example, the distance between small regions that are not misidentified in the learning evaluation image may be calculated, and the distance having the smallest value may be used.

In addition to comparing the learning data with the negative case data, the learning data is also compared with the area of the same class as the learning data (in this case, the car area), the distance between the learning data and the negative case data is separated, and the same class Avoid distances from other areas. Thereby, the size of the area indicated by the learning data can be prevented from becoming too large. Assuming that another region of the same class is S (c = c (S _Posi )), S _Posi may be set so as to satisfy Equation 4.
dist (f (S _Posi ), f (S _Nega ))> l ₁ , dist (f (S _Posi ), f (S (c = c (S _Posi )))) <l _2.
Here, l _{2 in} Equation 4 is a predetermined value, for example, the distance between small regions that are not misidentified in the learning evaluation image is calculated, and the smallest value among them is set to l ₁ . The largest value is l ₂ . The above process is performed on all selected erroneous identification areas.

  Next, in the second discriminator learning step T150, the second discriminator learning unit 305 learns the discriminator using the learning data generated in the second discriminator learning data generation step T140. For example, it is only necessary to learn SupportVectorMachines (SVMs) using negative cases generated for each learning data. Each learned discriminator is held in the second discriminator learning data holding unit 354 and used in the discrimination processing.

  Finally, in the integrated discriminator learning step T160, the integrated discriminator learning unit 306 performs the identification result of the first discriminator learned in the first discriminator learning step T110 and the second discrimination learned in the second discriminator learning step T150. The integrated discriminator or parameter for integrating the discriminator results is learned.

  As described above, there are two integration methods for integrating the identification result of the first classifier and the identification result of the second classifier. Here, the method using the integrated classifier mentioned as the second method is used, and a learning method of this integrated classifier will be described. When the first method is used, a threshold for determining a score to be adopted as a final result from the score (likelihood) output from the detector (second discriminator) is set for the learning evaluation image. decide. In that case, a plurality of threshold values may be set, and a threshold value with the highest area division accuracy may be adopted for the learning evaluation image.

  FIG. 18 is a detailed flowchart of processing performed by the integrated classifier learning unit 306 in the integrated classifier learning step T160. N in the figure indicates the number of learning evaluation images. Here, the class is identified again using the first discriminator for the learning evaluation image, but the identification result performed in the learning-time identification step T120 may be held and loaded.

  In T2601, a list of learning evaluation images used for learning of the integrated classifier is acquired. In the following, the results of all the pixels of the learning evaluation image are used for learning of the integrated discriminator, but may be thinned out to shorten the learning time. Although an example using a learning evaluation image is described here, another learning image may be prepared, or a learning image used during learning of the first discriminator may be used.

  In T2602, according to the list of learning evaluation images, the learning evaluation images are loaded one by one. Next, in T2603, the second classifier learned by the second classifier learning unit is loaded, and detection processing is performed on the target position of the learning evaluation image. The target position is a position where a detection process is performed on the learning evaluation image, and usually a raster scan is performed from the upper left to the lower right of the learning evaluation image. Also in the present embodiment, the detection process is performed by raster scanning from the upper left to the lower right of the learning evaluation image. In the present embodiment, the description will be made assuming that there is only one second discriminator. However, when there are a plurality of discriminators, the detection process is performed using the plural discriminators. A learning method of the integrated classifier when a plurality of classifiers is used will be described later.

  After performing the detection process at all positions, the process proceeds to T2605. In T2605, class identification of the learning evaluation image is performed using the first classifier.

In T2606, the detection process performed in T2603 and the result of the class identification performed in T2605 are combined. When the combined result corresponding to each pixel is S (x, y), the integrated result is expressed by the following equation (5).
S (x, y) = {S _{(x, y) εR} , S _D (x, y)} _Equation 5
Here, S (x, y) εR represents the discrimination result by the first discriminator for the small region including the target pixel. S _D (x, y) is an identification score of the second classifier corresponding to the target pixel (x, y) appearing in Equation 1. For example, when the number of classes identified by the first classifier is 10, and the number of second classifiers is 5, the number of dimensions of S (x, y) is 15. The above operation is performed on all the pixels of each learning evaluation image according to the list of learning evaluation images.

  In T2607, the classifier that identifies the class of each pixel is learned by using the result of combining in T2606 expressed by Equation 5 as an input. In the case of FIG. 12B, since sky, building, tree, load, and body are defined, it is necessary to learn a classifier that outputs class likelihood corresponding to five classes. A multi-class classifier that identifies which class is one of the five classes may be learned, and for each class, five 2-class classifiers that identify the class or the other class are learned. Of the five classifiers, the class having the highest class likelihood may be assigned. In the case of FIG. 12B, two-class identification is performed to determine whether the class is sky or any of the four classes including building, tree, load, and body that are other classes. This is also performed for the other four classes, and at the time of identification, the class identified by the two-class classifier that outputs the highest likelihood is assigned.

  The integrated discriminator learning step T160 is thus completed, and the integrated discriminator learned in this step is held in the second integrated discriminator holding unit 356 and provided in the integrated discriminating step S140 of the discriminating process. In the above description, the first integrated discriminator holding unit 507 and the second integrated discriminator holding unit 356 have been described as separate configurations. However, when the image recognition device 20 and the learning device 300 are configured integrally, the first integration discriminator The classifier holding unit 507 and the second integrated classifier holding unit 356 may be a single holding unit.

  In the present embodiment, an example in which integrated identification is performed on each pixel has been described. However, a combination result in each pixel expressed by Equation 5 may be averaged and input for each small region or block. In that case, the teacher value given when learning the classifiers to be integrated may be given the class having the largest number of pixels in the small area, or the support-vector-regulation is learned by using the number of pixels relative to the area of the small area as a regression value. Also good.

  Moreover, in this embodiment, although the example which learns an integrated discriminator using all the learning evaluation images was demonstrated, instead of using all the learning evaluation images, you may select a learning evaluation image at random, You may select the pixel in a learning evaluation image at random.

  As described above, according to the image recognition device 20 of the present embodiment, first, a region where a class is to be identified is detected by the second discriminator, and for each region of the recognition target image using the first discriminator. Identifies the class. Then, the detection result using the second classifier and the class identification by the first classifier are integrated to identify the class of each region of the recognition target image. With this configuration, in this embodiment, it is possible to reduce erroneous detection of a small region where class identification is difficult, and to recognize an image with high accuracy.

  In addition, the learning-time identification unit 302 evaluates the identification result of the learning evaluation image by the first discriminator, and the misidentification region selection unit 303 selects the misidentification region as a small region that is difficult to identify in the first discriminator. Then, the second discriminator learning unit 305 learns a second discriminator for identifying a region composed of a plurality of small regions including the selected erroneous identification region. Finally, the discrimination result by the first discriminator and the discrimination result by the second discriminator are integrated to learn the integrated discriminator. In the image recognition apparatus 20 using the first classifier and the second classifier learned as described above, the image identification accuracy can be increased.

[Second Embodiment]
Next, as a second embodiment, incidental information such as position information is associated with a second discriminator at the time of learning to acquire the incidental information at the time of recognition, and there are a plurality of second identifications based on the information. A mode in which a necessary second discriminator is selected from among the discriminators to perform discrimination will be described.

  The accompanying information is various information that can be attached to an image at the time of photographing. For example, position information such as GPS of the position where the image was taken, information obtained by the camera that took the image to be recognized such as color temperature and various parameters set by the camera, autofocus information obtained for each block or pixel, There are scene feature values obtained from captured images. The scene feature value is a feature value uniquely obtained for an image, and uses a spatial pyramid matching kernel described in Non-Patent Document 5 or a GIST feature value described in Non-Patent Document 4. be able to. Alternatively, it may be a feature amount obtained by dividing an image into a plurality of blocks and histogramating the color distribution of each block. In addition, any feature amount that represents the entire image or a feature amount obtained from each part of the image is aggregated as a statistic.

  The functional configurations of the image recognition device 20 and the learning device 300 in the present embodiment are the same as those of the image recognition device 20 and the learning device 300 shown in FIGS. 5 and 10A in the first embodiment, but partly. The processing content in the functional unit is different from that of the first embodiment. Specifically, the acquisition unit 501 in the image recognition apparatus 20 acquires incidental information together with the recognition target image in the image recognition processing. Further, with respect to the learning image and learning evaluation image used by the learning apparatus 300, supplementary information is necessary for at least data used for learning of the second discriminator.

  Next, image recognition processing in the present embodiment will be described. FIG. 6B is a flowchart showing the recognition processing when processing the image to be recognized in the present embodiment. Since the processing content of acquisition process S210 is the same as that of S110 in 1st Embodiment, description is abbreviate | omitted.

  In the incidental information acquisition step S220, the acquisition unit 501 acquires the incidental information of the recognition target image acquired in the acquisition step S210, and transmits the acquired incidental information to the detection unit 504. As the auxiliary information to be acquired, as described above, for example, if the camera 10 is equipped with a GPS, position information may be acquired, or parameters used for imaging by the camera 10 may be acquired. You may make it do.

  In the detection step S230, the detection unit 504 loads a predetermined second discriminator from among a plurality of second discriminators based on the incidental information obtained in the incidental information acquisition step S220, and performs detection processing. In the present embodiment, in the second discriminator learning step of the learning process, the auxiliary information is held together with the second discriminator, and a predetermined number of second discriminators whose auxiliary information is close are loaded during the recognition process. . Or you may make it define the distance between incidental information and load a 2nd discriminator according to distance.

  Furthermore, the result of each detector (second discriminator) may be weighted according to the distance from the incidental information. For example, in the case of a scene feature amount, the distance between the scene feature amount of the learning evaluation image obtained from the learning data of the second discriminator and the scene feature amount of the recognition target image is calculated, and each detection is performed according to the distance to the auxiliary information. Weight the result of the vessel. As such a distance, for example, a histogram distance or the like can be used. When information uniquely corresponding to an image such as various shooting parameters and color temperature is used as supplementary information, the numerical values may be directly compared. In addition, in the case of autofocus information associated with each block or pixel, the obtained values may be compared by vectorization or histogram formation. Alternatively, all these values may be combined and vectorized, and compared with a vector obtained from a learning evaluation image obtained by acquiring learning data of the second discriminator.

  Since the processing content of the detection step S230 after loading the second discriminator is the same as the processing content in the first embodiment, the description thereof is omitted.

  The processing contents of the identification step integration processing S240 and the integration identification processing step S250 are the same as S130 and S140 in the first embodiment, and thus description thereof is omitted.

  As described above, in the present embodiment, the learning apparatus 300 also holds incidental information when learning the second discriminator, and the second discriminator is based on the incidental information of the recognition target image during the recognition process for recognizing the image. To load. Thereby, it is possible to omit unnecessary detection processing by the second discriminator, and it is possible to improve the accuracy and speed of class identification by the image recognition device 20 and to save memory.

[Third Embodiment]
Next, as a third embodiment, the accuracy of identification results by the second classifier and the integrated classifier is improved by evaluating the classification result of the integrated classifier learned in the learning process using the learning evaluation image. The form to be made is demonstrated. In the present embodiment, the learning evaluation image used for evaluating the integrated discriminator may be the same as the learning evaluation image used in other steps, or may be prepared separately.

  Since the image recognition apparatus 20 and its processing flow in the present embodiment are the same as those in the first embodiment, description thereof will be omitted. Next, the learning apparatus 300 and its processing flow in this embodiment will be described. FIG. 10B is a diagram illustrating a functional configuration of the learning device 300 according to the present embodiment. The learning device 300 according to the present embodiment includes an integrated classifier evaluation unit 307 in addition to the configuration of each functional unit of the learning device 300 illustrated in FIG. 10A in the first embodiment. Detailed description of the integrated discriminator evaluation unit 307 will be described later with reference to FIG. The configuration of the other functional units is the same as that shown in FIG.

  Next, the learning process in this embodiment will be described. FIG. 11B is a flowchart illustrating a learning process executed by the learning device 300 in the present embodiment. The processing from the first discriminator learning step T310 to the integrated discriminator learning step T360 is the same as the first discriminator learning step T110 to the integrated discriminator learning step T160 in the first embodiment, and thus description thereof is omitted. .

  In the integrated discriminator evaluation step T370, the integrated discriminator evaluation unit 307 evaluates the integrated discriminator learned in the integrated discriminator learning step T360. FIG. 19 is a flowchart showing details of the integrated discriminator evaluation step T370 executed by the integrated discriminator evaluating unit 307.

  In T3701, the detection processing of the case learned by the second classifier is performed on the learning evaluation image. Since the method of this detection process is the same as the processing content of the detection step S120 in the recognition process, description thereof is omitted.

  In T3702, the class is identified by the first classifier for the learning evaluation image. Since the processing content here is the same as the processing content of the identification step S130 in the recognition processing, the description thereof is omitted.

  In T3703, the detection result of T3701 and the identification result of T3702 are integrated, and the learning evaluation image is identified using the integrated classifier stored in the second integrated classifier holding unit 356.

  In T3704, accuracy is evaluated by comparing the identification result by the first classifier performed in T3702 with GT of the learning evaluation image. For example, Pixel Accuracy is used for this accuracy evaluation. PixelAccuracy is an evaluation value often used for evaluation of region division in Non-Patent Document 1 or the like, and is a value obtained by counting whether or not the class identification result of each pixel is correct.

  In T3705, the integrated identification result by the integrated classifier performed in T3703 is compared with the GT of the learning evaluation image to evaluate the accuracy. The accuracy evaluation here is also performed using Pixel Accuracy as in T3704.

  In T3706, the discrimination accuracy evaluated in T3704 and T3705 is compared. Then, if the accuracy of the integrated identification result is not higher than a predetermined value with respect to the identification accuracy by the first identifier, the second identifier is learned again. That is, if the Pixel Accuracy as the identification result by the integrated classifier for the learning evaluation image is not higher than the Pixel Accuracy as the identification result by the first classifier, the second classifier is learned again.

  When learning the second discriminator again, the integrated discriminator evaluation process starts from the erroneous discrimination area selection step T330 until PixelAccuracy, which is the discrimination result by the integrated discriminator, becomes higher than the discrimination result by the first discriminator by a predetermined value or more. Repeat until T370. In the second and subsequent erroneous identification region selection step T330, an erroneous identification region is selected for the integrated identification result by the integrated classifier, not by the identification result by the first identifier. If the selected misidentification area is an area that has been selected before, the size of the learning data in the second discriminator is changed or the feature amount is changed. When changing the size of the learning data, it suffices to combine a large number of adjacent small regions as compared with the previous learning data. In the case of a misidentification region that has not been selected before, a detector (second identifier) for identifying the region may be added. After re-learning the second discriminator, the integrated discriminator is learned again, and the integrated discriminating result is reevaluated.

  As described above, according to the present embodiment, by using the learning evaluation image to evaluate the integrated discriminator that integrates the discrimination result by the first discriminator and the detection result by the second discriminator, the discrimination accuracy by the integrated discriminator. Can be increased.

[Fourth Embodiment]
Next, as a fourth embodiment, a configuration will be described in which a user defines a class for a region misidentified with respect to a learning evaluation image, thereby learning a second classifier corresponding to a case in that region. As a preferred configuration, the first region discriminator identifies the class of the learning evaluation image registered by the user, and selects a necessary portion as a misidentification region by viewing the result. Then, the second discriminator is learned based on the selected misidentification region.

  Since the image recognition apparatus 20 and its processing flow in the present embodiment are the same as those in the first embodiment, description thereof will be omitted. Next, the learning apparatus 300 and its processing flow in this embodiment will be described. FIG. 10C is a diagram illustrating a functional configuration of the learning device 300 according to the present embodiment. The learning device 300 in the present embodiment includes a display control unit 308 and a learning evaluation image acquisition unit 309 in addition to the configuration of each functional unit of the learning device 300 illustrated in FIG. 10A in the first embodiment. Detailed descriptions of the display control unit 308 and the learning evaluation image acquisition unit 309 will be described later with reference to FIG. The other configuration is the same as that shown in FIG.

  Next, the learning process in this embodiment will be described. FIG. 11C is a flowchart showing a learning process executed by the learning device 300 in the present embodiment. Since the processing of the first discriminator learning step T410 is the same as the first discriminator learning step T110 in the first embodiment, the description thereof is omitted.

  In the learning evaluation image acquisition step T420, the learning evaluation image acquisition unit 309 acquires the learning image registered in the learning device 300 by the user. The acquired learning evaluation image is transmitted to and stored in the learning evaluation image holding unit 352.

  In the learning time identification step T430, the first discriminator learning unit 301 identifies the class using the first discriminator for the learning evaluation image acquired in the learning evaluation image acquisition step T420. Since the specific processing content is the same as the processing content of the learning time identification step T120 in the first embodiment, the description is omitted.

  In the erroneous identification region selection step T440, the display control unit 308 causes the display unit 406 to display the learning image used in the learning time identification step T430. The learning image displayed on the display unit 406 can be selected in units of small areas divided during class identification in the learning time identification step T430, and the user operates the operation unit 405 (such as a mouse). This makes it possible to select a misidentification area. The misidentification area selection unit 303 selects the misidentification area by acquiring information related to the misidentification area selected and instructed by the user. In addition, class definition for the selected misidentification area is also performed. The class definition may be selected by the misidentification area selection unit 303 from predetermined classes, or selected by the user, and the misidentification area selection unit 303 acquires the selected information. May be.

  The processing of the second discriminator learning data generation step T450 and the second discriminator learning step T460 is the same as the processing contents of the second discriminator learning data generation step T140 and the second discriminator learning step T150 in the first embodiment. Therefore, the description thereof is omitted.

  As described above, in the present embodiment, the first discriminator learning unit 301 learns the first discriminator using the learning evaluation image registered by the user, and the erroneous classification region selection unit 303 selects the error that the user has instructed to select. By acquiring information related to the identification area, the erroneous identification area is selected. Thereby, it is possible to extract a small region that is difficult to be identified by the first classifier. In particular, in the present embodiment, since the erroneous identification region is selected based on the user's selection instruction, a learning evaluation image without GT can be used, and the identification accuracy of the image that the user wants to recognize is high. A high classifier can be learned.

  In addition, the present invention supplies software (program) for realizing the functions of the above-described embodiments to a system or apparatus via a network or various storage media, and the computer of the system or apparatus (or CPU, MPU, etc.) programs Is read and executed. Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. The present invention is not limited to the above embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. is not. That is, the present invention includes all the combinations of the above-described embodiments and modifications thereof.

DESCRIPTION OF SYMBOLS 300 Learning apparatus 301 1st discriminator learning part 302 Discrimination at the time of learning 303 Misidentification area selection part 304 2nd discriminator learning data generation part 305 2nd discriminator learning part 306 Integrated discriminator learning part

Claims (16)

A first learning step of learning a first discriminator for identifying a class for each region of an image using a learning image;
A learning time identifying step of identifying a class for each region of the learning evaluation image by the learned first discriminator;
A selection step of selecting a misidentification area in which the class identification result for the learning evaluation image by the first classifier is incorrect;
A generation step of generating learning data using a region including the selected misidentification region;
And a second learning step of learning a second discriminator for identifying the class of the generated learning data.   The said generation process selects the area | region containing the said misidentification area | region based on the correct data of the said learning evaluation image, The said learning area | region is produced | generated using the selected area | region. Learning method.   The learning method according to claim 1, wherein the generation step selects a region including the misidentification region based on an instruction from a user, and generates the learning data using the selected region. .   In the generation step, the learning evaluation image having a region including the misidentification region as a correct example, having the same shape as the region including the misidentification region, and having the class erroneously identified by the first classifier The learning method according to any one of claims 1 to 3, wherein the learning data is generated with a region of a negative example as a negative example.   The generating step sets a size of a region including the misidentification region based on a distance between a feature amount extracted from the positive case region and a feature amount extracted from the negative case region. The learning method according to claim 4.   The learning method according to claim 1, wherein the learning image and the learning evaluation image are data of the same image.   7. The method according to claim 1, further comprising a third learning step of learning an integrated discriminator that integrates the discrimination result obtained by the first discriminator and the discrimination result obtained by the second discriminator. The learning method according to the section. An evaluation step for obtaining an evaluation value as a result of identifying a class for each area of the learning evaluation image by the integrated classifier and an evaluation value as a result of identifying a class for each area of the learning evaluation image by the first classifier Further comprising
Until the evaluation value of the identification result of the integrated discriminator becomes higher than the evaluation value of the identification result of the first discriminator by a predetermined value or more, the size of the region including the misidentification region is made different, and the second learning is performed. The learning method according to claim 7, wherein the steps are repeated. A detection step of detecting a region whose class is to be identified by the second classifier from the recognition target image;
An identifying step of identifying a class for each region of the recognition target image using a first identifier;
An image recognition method comprising: an integrated identification step of identifying a class for each region of the recognition target image based on a detection result of the detection step and an identification result of the identification step.   The first classifier and the second classifier learned by the learning method according to any one of claims 1 to 6 are used as the first classifier and the second classifier. The image recognition method according to claim 9. It further includes an acquisition step of acquiring incidental information incidental to the recognition target image,
The image according to claim 9 or 10, wherein the detection step selects a second discriminator to be used in the detection step from a plurality of second discriminators based on the acquired auxiliary information. Recognition method.   The supplementary information includes position information when the recognition target image is captured, information obtained by a camera that captured the recognition target image, parameters used by the camera when the recognition target image is captured, The image recognition method according to claim 11, wherein the image recognition method is one of camera autofocus information at the time of capturing a recognition target image and a scene feature amount of the recognition target image. First learning means for learning a first discriminator for identifying a class for each region of an image using a learning image;
A learning time identifying means for identifying a class for each region of the learning evaluation image by the learned first discriminator;
Selecting means for selecting a misidentification region in which the class identification result for the learning evaluation image by the first classifier is incorrect;
Generating means for generating learning data using a region including the selected misidentification region;
A learning apparatus comprising: a second learning unit that learns a second classifier that identifies a class of the generated learning data. Detecting means for detecting a region in which a class should be identified by the second classifier from the recognition target image;
Identifying means for identifying a class for each region of the recognition target image using a first identifier;
An image recognition apparatus comprising: an integrated identification unit that identifies a class for each region of the recognition target image based on a detection result of the detection unit and an identification result of the identification unit.   The program for making a computer perform the learning method of any one of Claim 1 to 8.   A program for causing a computer to execute the image recognition method according to any one of claims 9 to 12.

Images