JP2009069996A - Image processing device and image processing method, recognition device and recognition method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To separate a background from image data to extract an area corresponding to an object to be recognized. <P>SOLUTION: In an image obtained by passing background-included image data taken by imaging with a focus camera through a 3×3 filter that the coefficient of a target pixel is 8 and the coefficient of all of 8 pixels in the vicinity thereof is -1, a black area indicates an out-of-focus pixel that the difference between it and the vicinity pixel is nearly zero and a bright (white) area indicates an in-focus pixel having a sharp pattern. A filtering process of determining an average between the target pixel and its vicinity area is performed on the image to bind an area looking like the object to be recognized thereto. After a result of filtering has been binarized, Morphological processing is performed or masking is performed on a part which is not the black area from which the value is generated as the result of filtering to divide the area to separate the background from the image data. The present invention is applied to the image processing system, a learning device or a recognition device. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a processing device, an image processing method, a recognition device, a recognition method, and a program, and in particular, a processing device, an image processing method, a recognition device, and a recognition suitable for performing recognition processing using an image. The present invention relates to a method and a program.

  When object recognition is performed using an image (for example, an image captured using a camera or the like) in a real environment, the image includes not only the object to be recognized but also the background. In recognizing where the object to be recognized is located in a miscellaneous scene including the background (localization), the background pattern (shape, color, etc.) can cause misrecognition. The object recognition is very difficult.

  For example, conventionally, there has been a method of segmenting (cutting out) a recognized object based on the distance from the camera of an object being imaged using stereo calculation from an image imaged by a compound eye.

  In addition, various methods for object recognition by image processing have been proposed in recent years, and have improved dramatically in the last 10 years.

  Object recognition by image processing requires three features: a feature value, a geometric connection, and a classifier. That is, the method of object recognition by image processing depends on how to select a feature quantity, a method of geometric connection, and a classifier.

  First of all, as to what to use as the feature quantity, there are choices depending on whether to use a global feature quantity, a local feature quantity, or what kind of calculation to use among local feature quantities. There are many.

  For example, when using local region features, it is necessary to select a method for geometrically connecting them, and roughly speaking, a method of explicitly preparing a template (explicit template) and “voting” (implicit voting) method).

  Finally, it is necessary to consider what classifier is used to represent the object (target to be recognized) using the feature amount.

  Speaking of features, in recent methods, the entire image is divided into several small regions called local regions, and the object is based on local information such as feature points and features obtained from the local regions. Recognition is becoming mainstream. The expression “local region” has various names such as a local descriptor, a component, a part, and a fragment.

  Examples of such local information include Gabor Jet, Haar Wavelet, Gaussian Derivatives, SIFT features, and the like. As a discriminator, a statistical learning machine is often used. Examples thereof include a support vector machine (SVM), boosting, and Bayesian estimation.

  Conventionally, as described above, recognition processing has been performed using various feature amounts, geometric connection methods, and discriminators (see, for example, Non-Patent Document 1 to Patent Document 11).

Object Recognition with Cortex-like MechanismsSerre, T., L. Wolf, S. Bileschi, M. Riesenhuber and T. Poggio.IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 3, 411-426, 2007.

R. Fergus, P. Perona, and A. Zisserman. Object class recognition by unsupervised scale-invariant learning.In Proceedings of International Conference on Computer Vision and Pattern Recognition (CVPR), pages: II-264- II-271 vol.2 2003

B. Leibe, A. Leonardis, and B. Schiele. Combined object categorization and segmentation with an implicit shape model.In European Conference on Computer Vision (ECCV'04) Workshop on Statistical Learning in Computer Vision, 2004.

Contour-Based Learning for Object Detection Jamie Shotton, Andrew Blake, and Roberto Cipolla International Conference on Computer Vision (ICCV 2005)

M. Vidal-Naquet and S. Ullman.Object recognition with informative features and linear classification.In ICCV, pages 281-288, 2003.

Using the forest to see the trees: a graphical model relating features, objects and scenes P. Murphy, A. Torralba and W. T. Freeman Adv. In Neural Information Processing Systems 16 (NIPS), Vancouver, BC, MIT Press, 2003.

The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features.K. Grauman and T. Darrell.International Conference on Computer Vision (ICCV 2005)

Discovering objects and their location in images Josef Sivic, B. Russell, A. Efros, A. Zisserman, W. Freeman International Conference on Computer Vision (ICCV 2005)

SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition Hao Zhang Alexander C. Berg Michael Maire Jitendra Malik International Conference on Computer Vision and Pattern Recognition (CVPR 2006)

G. Csurka, C. Bray, C. Dance, and L. Fan.Visual categorization with bags of keypoints.In Workshop on Statistical Learning in Computer Vision, ECCV, pages 1-22, 2004.

Ian R. Fasel, Learning to Detect Objects in Real-Time: Probabilistic Generative Approaches, PhD thesis, UCSD, June 2006

  Non-Patent Document 1 to Non-Patent Document 11 will be described with reference to FIG.

  The approach method used in the recognition processing described in Non-Patent Document 1 is called HMAX, and the feature value is C2 (gabor), which is a feature value that is strong against misalignment by a combination of Gabor filters. Using local MAX pooling, which is the best match feature amount, identification is performed by an RBF (Radial Basis Function) Network.

  The approach method used in the recognition processing described in Non-Patent Document 2 is called Constellation, and a gray patch that is a luminance image patch is used as a feature value, and a Gaussian distribution is used to calculate ML (Maximum Likelihood) ( (Bayes) is used for identification.

  The approach method used in the recognition process described in Non-Patent Document 3 is called ISM (Implicit Shape Mode), and a gray patch that is a luminance image patch is used as a feature quantity, and a template is voted (implicit voting method). By using the method of obtaining implicitly, recognition is performed by SVM (Support Vector Machines).

  The approach method used in the recognition process described in Non-Patent Document 4 is called Blake from the name of the author of this document, using edge images (edgels) as feature quantities, and using a star model, Recognition is performed by a boosting algorithm.

  The approach method used in the recognition process described in Non-Patent Document 5 is called Fragments, and the feature amount uses a gray patch that is a luminance image patch, and a template method is used to obtain mutual information ( Recognition by mutual info).

  The approach used in the recognition process described in Non-Patent Document 6 is called Torralba from the name of the author of this document, and GD (Gaussian Derivatives) Lap (Laplacians) Haar (Haar Features) is used as a feature quantity. Using a template-based method, recognition is performed by a boosting algorithm and Bayesian estimation.

  The approach method used for the recognition processing described in Non-Patent Document 7 is called PMK (Pyramid Match Kernel). This proposes a kernel function that calculates the degree of similarity based on partial matching between two bags, and uses Pyramid Histogram as a feature quantity to perform image classification using SVM. .

  The approach method used in the recognition process described in Non-Patent Document 8 is called pLSA (probabilistic Latent Semantic Analysis), and SIFT (Scale Invariant Feature Transform) is used as a feature quantity so that recognition is performed by pLSA. Has been made.

  The approach method used for the recognition process described in Non-Patent Document 9 is called SVM-kNN (k-NearestNeighbor), and the shape context is used as a feature quantity, and a template is used. First, a kNN search is performed, and if a predetermined number of objects have the same label, they are classified into that class. Otherwise, recognition is performed by executing a multi-class SVM.

  The approach method used for the recognition process described in Non-Patent Document 10 is called Bag-of-Features, and uses SIFT as a feature quantity and performs recognition by SVM.

  The approach method used in the recognition process described in Non-Patent Document 11 is called Ian from the name of the author of this document, and a Haar type feature quantity is used as a feature quantity. Recognition is performed by Bayesian estimation.

  However, it has not been possible to detect where an object to be recognized is in a miscellaneous scene including a background using a simple method. For example, in order to use segmentation using stereo calculation, which has been conventionally used, a compound eye camera must be used, which causes an increase in cost. Therefore, for example, a complicated operation for cutting out a necessary area from the image data obtained by a monocular camera is performed by a user operation or the like.

  Therefore, there is a need for a technique that can easily extract an object to be recognized from image data, that is, separate a background, without performing complicated operations while suppressing cost.

  Further, in the conventional techniques described in Non-Patent Documents 1 to 11 described above, the content of the feature amount obtained as local information differs depending on the type, and compatibility is not ensured. For example, a feature quantity relating to color and a feature quantity relating to shape generally have different vector dimensions and scales, and therefore cannot be compared with each other. Therefore, it is difficult to use different types of feature quantities for object recognition.

  The present invention has been made in view of such a situation, and makes it possible to easily extract an object to be recognized from an image and perform object recognition.

  An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that generates a recognizer for recognizing a recognition target in advance by a learning process, and acquires a learning image used for the learning process. Using the model image acquisition means for acquiring a model image corresponding to the recognition target, the learning image acquired by the learning image acquisition means, and the model image acquired by the model image acquisition means A recognizer generating unit that executes a process and generates a recognizer for recognizing the recognition target, and at least one of the learning image acquiring unit or the model image acquiring unit exists at a predetermined focal length An image acquisition unit that acquires image data that is in focus with the image of the subject to be imaged and that is not in focus with other objects, and the image acquisition unit acquires the image data. Image extraction means for extracting a portion corresponding to the subject in focus from the image data, and the portion corresponding to the subject extracted by the image extraction means is the learning image or the model. Obtain as an image.

  The image extraction unit includes a first calculation unit that executes a calculation process for extracting a pixel having a large difference from a neighboring pixel in each pixel of the image data acquired by the image acquisition unit; A second arithmetic unit that obtains an average of the target pixel and its neighboring region using a pixel having a large difference from a neighboring pixel extracted by the first arithmetic unit as a target pixel, and an operation of the second arithmetic unit Based on the result, the image data can include a region corresponding to the object to be detected and a dividing unit that divides the image data into regions considered to be the background.

  The dividing means binarizes the calculation result of the second calculating means with a predetermined threshold value, thereby dividing the result into an area corresponding to the object to be detected and an area considered to be the background. be able to.

  The dividing unit can be made to recognize a pixel whose calculation result of the second calculation unit is a positive value as a region corresponding to an object to be detected.

  The recognizer generating unit includes a model feature point generating unit that generates a plurality of feature points as model feature points from the model image acquired by the model image acquiring unit, and the model generated by the model feature point generating unit. Model feature quantity generation means for generating a feature quantity at each feature point as a model feature quantity, and learning feature point generation means for generating a plurality of feature points as learning feature points from the learning image acquired by the learning image acquisition means Learning feature amount generating means for generating, as a learning feature amount, a feature amount at each of the learning feature points generated by the learning feature point generating means, and the model feature amount generated by the model feature amount generating means. For each of the learning feature amounts generated by the learning feature amount generation means, the one with the highest correlation is selected. Learning correlation feature value generating means for generating a degree of correlation with the selected learning feature value as a learning correlation feature value, and acquiring correct / incorrect information indicating whether or not the learning image includes the recognition target Corrector information acquisition means, recognizer generation means for generating a recognizer based on the learned correlation feature value generated by the learned correlation feature value generation means, and the correctness information acquired by the correctness information acquisition means; Can be provided.

  The model feature point generated by the model feature point generation unit may be selected according to the type of the model feature amount in the model feature point, and is generated by the learning feature point generation unit. The learning feature point may be selected according to the type of the learning feature amount at the learning feature point.

  The model feature quantity generated by the model feature quantity generation means can be selected according to the type of the model feature quantity, and the learning feature quantity generated by the learning feature quantity generation means is The learning feature value may be selected according to the type.

  The recognizer generation means can generate the recognizer by a learning process based on weighted voting.

  The learning process based on the weighted voting may be a boosting algorithm.

  The image extracting unit extracts a portion of the image data acquired by the image acquiring unit that is out of focus, thereby extracting a portion corresponding to the subject in focus. be able to.

  The image extraction means uses FFT to analyze the frequency spectrum of each image area constituting the image data acquired by the image acquisition means, and the focus is matched in an area that contains sufficient high-frequency components. Therefore, it is possible to extract a portion corresponding to the subject in focus.

  A recognizer storage unit that stores the recognizer generated by the recognizer generation unit, and a selection feature amount storage unit that stores a selection feature amount corresponding to each of the recognizers stored by the recognizer storage unit. A recognition image acquisition means for acquiring a recognition image used for performing recognition processing, and a recognition feature point generation means for generating a plurality of feature points as recognition feature points from the recognition image acquired by the recognition image acquisition means A recognition feature value generation unit that generates a feature value at each of the recognition feature points generated by the recognition feature point generation unit as a recognition feature value, and a selection feature value stored in the selection feature value storage unit. For each of the recognition feature values generated by the recognition feature value generation means, the one having the highest correlation is selected, and the selected recognition feature is selected. A recognition correlation feature quantity generating means for generating a degree of correlation as a recognition correlation feature quantity, and the recognition correlation feature quantity generated by the recognition correlation feature quantity generation means generated by the recognizer generation means. By substituting in the recognizer, it is possible to further comprise recognition processing means for determining whether or not the recognition object is included in the recognition image acquired by the recognition image acquisition means.

  The recognition image acquisition unit may include the image acquisition unit and the image extraction unit, and a part corresponding to the subject extracted by the image extraction unit may be acquired as the recognition image. Can be.

  An image processing method according to a first aspect of the present invention is an image processing method of an image processing apparatus that generates a recognizer for recognizing a recognition target in advance by learning processing, and acquires a learning image used for the learning processing. Obtaining a model image corresponding to the recognition target, performing the learning process using the acquired learning image and the model image, and generating a recognizer for recognizing the recognition target, At least one of the step of acquiring the learning image or the step of acquiring the model image is focused on an image of a subject existing at a predetermined focal distance, and focused on other objects. A step of acquiring non-image data and extracting a portion corresponding to the focused subject from the acquired image data, and corresponding to the extracted subject Min, the learning image, or to obtain, as the model image.

  A program according to a first aspect of the present invention is a program for causing a computer to execute processing for generating a recognizer for recognizing a recognition target in advance by learning processing, and acquiring a learning image used for the learning processing. Control, acquisition of the model image corresponding to the recognition target is controlled, the learning process is executed using the acquired learning image and the model image, and a recognizer for recognizing the recognition target is generated. And at least one of the step of acquiring the learning image or the step of acquiring the model image is in focus on an image of a subject existing at a predetermined focal length, and other objects include Controlling the acquisition of image data that is out of focus, and extracting a portion corresponding to the subject in focus from the acquired image data; Look, the extracted part corresponding to the subject, the learning image, or to execute a process of acquiring, as the model image to the computer.

  In the first aspect of the present invention, a learning image used for learning processing is acquired, a model image corresponding to a recognition target is acquired, learning processing is executed using the acquired learning image and model image, and recognition is performed. A recognizer for recognizing the object is generated. In at least one of the acquisition of the learning image or the acquisition of the model image, there is image data in which the focus is matched to the image of the subject existing at a predetermined focal length and the focus is not matched to the other objects. A portion corresponding to the subject in focus is extracted from the acquired image data, and a portion corresponding to the extracted subject is acquired as a learning image or a model image.

  A recognition device according to a second aspect of the present invention is a recognition device that performs recognition processing for determining whether or not a recognition target is included in a recognition image, using a recognizer generated by learning processing. Recognition image acquisition means for acquiring the recognition image used for performing processing, recognition device storage means for storing the recognition device, and selection corresponding to each of the recognition devices stored by the recognition device storage means Using the selected feature quantity storage means for storing the feature quantity, the recognizer stored in the recognizer storage means, and the selected feature quantity stored in the selected feature quantity storage means, the recognized image acquisition means Recognition processing means for determining whether or not the recognition target is included in the recognition image acquired by the recognition image acquisition means, the recognition image acquisition means to the image of the subject existing at a predetermined focal length Image acquisition means for acquiring image data in which points match and other objects are not in focus, and a portion corresponding to the subject in focus from the image data acquired by the image acquisition means And a portion corresponding to the subject extracted by the image extraction means is acquired as the recognition image.

  The image extraction unit includes a first calculation unit that executes a calculation process for extracting a pixel having a large difference from a neighboring pixel in each pixel of the image data acquired by the image acquisition unit; A second arithmetic unit that obtains an average of the target pixel and its neighboring region using a pixel having a large difference from a neighboring pixel extracted by the first arithmetic unit as a target pixel, and an operation of the second arithmetic unit Based on the result, the image data can include a region corresponding to the object to be detected and a dividing unit that divides the image data into regions considered to be the background.

  The dividing means binarizes the calculation result of the second calculating means with a predetermined threshold value, thereby dividing the result into an area corresponding to the object to be detected and an area considered to be the background. be able to.

  The dividing unit may recognize a pixel whose calculation result of the second calculation unit is a positive value as a region corresponding to an object to be detected.

  The recognition processing means includes recognition feature point generation means for generating a plurality of feature points as recognition feature points from the recognition image acquired by the recognition image acquisition means, and the recognition generated by the recognition feature point generation means. Recognition feature value generation means for generating a feature value at each feature point as a recognition feature value, and the recognition feature value generated by the recognition feature value generation means for each of the selected feature values stored in the selected feature value storage means A recognition correlation feature value generating unit that selects a feature value having the highest correlation among the feature values and generates a degree of correlation with the selected recognition feature value as a recognition correlation feature value; and the recognition correlation feature value generation By substituting the recognized correlation feature quantity generated by the means into the recognizer stored by the recognizer storage means, the recognized image acquisition means It can be made to make a determination means for determining whether or not the recognition target on the acquired recognition image contains.

  The recognizer stored by the recognizer storage unit generates a plurality of feature points as model feature points from a predetermined model image, generates a feature amount at each of the model feature points as a model feature amount, and A plurality of feature points are generated as learning feature points from the learning image, and a feature amount at each of the learning feature points is generated as a learning feature amount. For each of the model feature amounts, the most correlated among the learning feature amounts Select a high one, generate a degree of correlation with the selected learning feature amount as a learning correlation feature amount, obtain correct / incorrect information indicating whether the learning image includes the recognition target, It may be a recognizer generated based on the learning correlation feature quantity and the correct / incorrect information.

  The recognition method according to the second aspect of the present invention uses a recognizer generated by learning processing and stored in a storage unit, and a selection feature amount corresponding to each of the recognizers stored in the storage unit. A recognition method of a recognition device that performs a recognition process for determining whether or not a recognition target is included in a recognition image, obtains the recognition image used for performing the recognition process, and A step of determining whether or not the recognition target is included in the acquired recognition image using a selected feature amount, and in the processing of the step of acquiring the recognition image, a subject existing at a predetermined focal length Obtaining image data that is in focus with the other image and not in focus with other objects, and extracting a portion corresponding to the subject in focus from the obtained image data Wherein the extracted part corresponding to the subject and obtained as the recognition image.

  A program according to a second aspect of the present invention uses a recognizer generated by learning processing and stored in a storage unit, and a selection feature amount corresponding to each of the recognizers stored in the storage unit. A program for causing a computer to execute a process of determining whether or not a recognition target is included in a recognition image, controlling acquisition of the recognition image used for performing the recognition process, the recognizer and the selection A step of determining whether or not the recognition target is included in the acquired recognition image using a feature amount, and in the processing of the step of acquiring the recognition image, a subject existing at a predetermined focal length Controls the acquisition of image data that is in focus on the image but not on any other object, and corresponds to the subject in focus from the acquired image data It comprises extracting a minute, the extracted part corresponding to the object to execute a process of acquiring, as the recognition image on the computer.

  In the second aspect of the present invention, a recognition image used for performing recognition processing is acquired, and whether or not a recognition target is included in the acquired recognition image using a recognizer and a selected feature amount. Is judged. Then, when the recognition image is acquired, image data in which the focus is matched with the image of the subject existing at a predetermined focal length and the focus is not matched with other objects is acquired, and the acquired image data is used. A portion corresponding to the subject in focus is extracted, and a portion corresponding to the extracted subject is acquired as a recognition image.

  The network is a mechanism in which at least two devices are connected and information can be transmitted from one device to another device. The devices that communicate via the network may be independent devices, or may be internal blocks that constitute one device.

  The communication is not only wireless communication and wired communication, but also communication in which wireless communication and wired communication are mixed, that is, wireless communication is performed in a certain section and wired communication is performed in another section. May be. Further, communication from one device to another device may be performed by wired communication, and communication from another device to one device may be performed by wireless communication.

  The image processing apparatus may be an independent apparatus or a block that performs recording processing of the information processing apparatus. In addition, the learning device and the recognition device may be independent devices or may be blocks that perform recording processing of the information processing device.

  As described above, according to the first aspect of the present invention, a recognizer can be generated, and in particular, imaging of a recognition target is automatically performed from at least a part of an image used for learning of the recognizer. Regions can be extracted.

  In addition, according to the second aspect of the present invention, recognition processing can be performed. In particular, an imaging region to be recognized can be automatically extracted from at least a part of an image used for recognition processing. it can.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  The image processing apparatus according to the first aspect of the present invention corresponds to an image processing apparatus (for example, the learning apparatus 71 in FIG. 19 or the image processing system 51) that generates a recognizer for recognizing a recognition target in advance by a learning process. A learning image acquisition unit (for example, a learning image acquisition unit 95 in FIG. 19) that acquires a learning image used in the learning process, and a model image acquisition unit that acquires a model image corresponding to the recognition target. (E.g., using the model image acquisition unit 91 of FIG. 19), the learning image acquired by the learning image acquisition means and the model image acquired by the model image acquisition means, Recognizer generating means for generating a recognizer for recognizing the recognition target (for example, model feature point generator 92, model feature generator 93, model feature storage in FIG. 19) 94, a learning feature point generation unit 96, a learning feature amount generation unit 97, a learning correlation feature amount generation unit 98, a correct / incorrect information acquisition unit 99, and a recognizer generation unit 100), and the learning image acquisition means or the model At least one of the image acquisition means is focused on image data of a subject existing at a predetermined focal length, and image data that is not focused on other objects (for example, as shown in FIG. 3A). Image acquisition means (for example, the image acquisition unit 21 in FIG. 4) for acquiring image data captured by such a focus camera, and the subject in focus from the image data acquired by the image acquisition means. Image extraction means (for example, the background separation processing unit 22 in FIG. 4) for extracting a corresponding part, and the part corresponding to the subject extracted by the image extraction means is Learning image, or to obtain, as the model image.

The image extraction unit is a first calculation unit that executes a calculation process for extracting a pixel having a large difference from a neighboring pixel in each pixel of the image data acquired by the image acquisition unit (for example, FIG. 5 is a pixel having a large difference between the neighboring pixel difference filter calculation processing unit 31) and the neighboring pixel extracted by the first calculation unit, and an average of the pixel of interest and its neighboring region is obtained. Based on the calculation result of the second calculation means (for example, the neighborhood region sum filter calculation processing unit 32 in FIG. 5) and the calculation result of the second calculation means, the area corresponding to the object to be detected, the background And a dividing unit (for example, a threshold processing unit 33 in FIG. 5) that divides the region into regions that are considered to be.
The image processing apparatus according to claim 1.

  The recognizer generation unit includes a model feature point generation unit (for example, a model feature point generation unit 92 in FIG. 19) that generates a plurality of feature points as model feature points from the model image acquired by the model image acquisition unit. Model feature value generation means (for example, the model feature value generation unit 93 in FIG. 19) that generates a feature value at each of the model feature points generated by the model feature point generation means as a model feature value; and the learning image A learning feature point generation unit (for example, a learning feature point generation unit 96 in FIG. 19) that generates a plurality of feature points as learning feature points from the learning image acquired by the acquisition unit, and the learning feature point generation unit. Further, learning feature value generation means for generating a feature value at each of the learning feature points as a learning feature value (for example, the learning feature value generation unit 9 in FIG. 19). ) And the model feature amount generated by the model feature amount generation unit, the learning feature amount generated by the learning feature amount generation unit is selected and selected. A learning correlation feature value generation unit (for example, a learning correlation feature value generation unit 98 in FIG. 19) that generates a degree of correlation with the learning feature value as a learning correlation feature value, and the learning image includes the recognition target. Correct / incorrect information acquisition means (for example, the correct / incorrect information acquisition unit 99 in FIG. 19) for acquiring correct / incorrect information indicating whether or not, the learned correlation feature quantity generated by the learned correlation feature quantity generation means, and the correct / incorrect information Recognizing device generating means for generating a recognizing device based on the correct / incorrect information acquired by the acquiring device.

  The model feature point generated by the model feature point generation means is selected according to the type of model feature amount (for example, shape, color, movement, texture, material, walking pattern, etc.) at the model feature point, The learning feature point generated by the learning feature point generation means is selected according to the type of the learning feature amount (for example, shape, color, movement, texture, material, walking pattern, etc.) at the learning feature point. Can be.

  The model feature quantity generated by the model feature quantity generation unit is selected according to the type of model feature quantity (for example, shape, color, movement, texture, material, walking pattern, etc.), and the learning feature quantity generation The learning feature amount generated by the means may be selected according to the type of the learning feature amount (for example, shape, color, movement, texture, material, walking pattern, etc.).

  The image extraction means extracts an out-of-focus area of the image data acquired by the image acquisition means (for example, Blur Detection for Digital Images Using Wavelet Transform; Hanghang Tong: Mingjing Li, Hongjiang Zhang: By using the technique described in Changshiui Zhang), a portion corresponding to the subject in focus can be extracted.

  Recognizer storage means for storing the recognizer generated by the recognizer generation means (for example, the recognizer storage unit 122 in FIG. 19) and the recognizer stored in the recognizer storage means respectively. Selection feature quantity storage means for storing the selected feature quantity (for example, the selection feature quantity storage section 121 in FIG. 19) and recognition image acquisition means for acquiring the recognition image used for performing the recognition processing (for example, in FIG. 19). A recognition image acquisition unit 123), and a recognition feature point generation unit that generates a plurality of feature points as recognition feature points from the recognition image acquired by the recognition image acquisition unit (for example, the recognition feature point generation unit 124 of FIG. 19). And a recognition feature value generation unit (for example, the recognition feature in FIG. 19) that generates a feature value at each of the recognition feature points generated by the recognition feature point generation unit as a recognition feature value. And the selected feature quantity stored by the selected feature quantity storage means is selected from the recognized feature quantities generated by the recognized feature quantity generation means for the selected feature quantity. A recognition correlation feature value generating means (for example, a recognition correlation feature value generation unit 126 in FIG. 19) that generates a degree of correlation with the selected recognition feature value as a recognition correlation feature value, and the recognition correlation feature value The recognition target is included in the recognition image acquired by the recognition image acquisition unit by substituting the recognition correlation feature amount generated by the generation unit into the recognition unit generated by the recognition unit generation unit. Recognition processing means (for example, the recognition processing unit 127 in FIG. 19) for determining whether or not the image is present.

  The image processing method according to the first aspect of the present invention corresponds to an image processing device (for example, the learning device 71 in FIG. 19 or the image processing system 51) that generates a recognizer for recognizing a recognition target in advance by learning processing. Image processing method for acquiring the learning image used in the learning process (for example, the process of step S16 in FIG. 29), and acquiring the model image corresponding to the recognition target (for example, in FIG. 29). Step S11), the learning process is executed using the acquired learning image and the model image, and a recognizer for recognizing the recognition target is generated (for example, the process of Step S20 in FIG. 29). At least one of the step of acquiring the learning image and the step of acquiring the model image includes an image of a subject existing at a predetermined focal length. Image data (for example, image data captured by a focus camera as shown in FIG. 3A) is acquired (for example, step in FIG. 30). (Step S41), including a step of extracting the portion corresponding to the subject in focus from the acquired image data (for example, step S42 in FIG. 30), and including the portion corresponding to the extracted subject. Acquired as the learning image or the model image.

  A program according to a first aspect of the present invention is a program for causing a computer to execute processing for generating a recognizer for recognizing a recognition target in advance by learning processing, and acquiring a learning image used for the learning processing. Control (for example, the process of step S16 in FIG. 29), control the acquisition of the model image corresponding to the recognition target (for example, the process of step S11 in FIG. 29), and acquire the acquired learning image and the model image. Using the learning process to generate a recognizer for recognizing the recognition target (for example, the process of step S20 in FIG. 29), and acquiring the learning image, or the model image At least one of the steps of obtaining the image is in focus on the image of the subject existing at the predetermined focal length, and in focus on the other objects. Control of acquisition of non-image data (for example, image data captured by a focus camera as shown in FIG. 3A) (for example, processing in step S41 of FIG. 30), and from the acquired image data, a focus is controlled. Including a step of extracting a portion corresponding to the subject that matches (for example, step S42 in FIG. 30), and acquiring a portion corresponding to the extracted subject as the learning image or the model image. Let the computer run.

  The recognition apparatus according to the second aspect of the present invention uses a recognizer generated by learning processing to perform a recognition process for determining whether or not a recognition target is included in a recognition image (for example, FIG. 19). A recognition image acquisition means (for example, a recognition image acquisition unit 123 in FIG. 19) for acquiring the recognition image used for performing recognition processing, and a recognition device storage for storing the recognition device. Means (for example, the recognizer storage unit 122 in FIG. 19) and selection feature amount storage means (for example, in FIG. 19) that stores selection feature amounts corresponding to each of the recognizers stored in the recognizer storage unit. A selected feature amount storage unit 121), the recognizer stored in the recognizer storage unit, and the selected feature amount stored in the selected feature amount storage unit; Recognition processing means for determining whether or not the recognition target is included in the acquired recognition image (for example, a recognition feature point generation unit 124, a recognition feature amount generation unit 125, a recognition correlation feature amount generation unit in FIG. 19). 126, a recognition processing unit 127), and the recognition image acquisition unit has image data (for example, a focus that matches an image of a subject existing at a predetermined focal length and a focus that does not match other objects) From image acquisition means (for example, the image acquisition unit 21 in FIG. 4) for acquiring image data captured by a focus camera as shown in FIG. 3A and the image data acquired by the image acquisition means Image extracting means (for example, the background separation processing unit 22 in FIG. 4) for extracting a portion corresponding to the subject that is in focus, and for the subject extracted by the image extracting means. A portion, which obtained as the recognition image.

  The image extraction unit is a first calculation unit that executes a calculation process for extracting a pixel having a large difference from a neighboring pixel in each pixel of the image data acquired by the image acquisition unit (for example, FIG. 5 is a pixel having a large difference between the neighboring pixel difference filter calculation processing unit 31) and the neighboring pixel extracted by the first calculation unit, and an average of the pixel of interest and its neighboring region is obtained. Based on the calculation result of the second calculation means (for example, the neighborhood region sum filter calculation processing unit 32 in FIG. 5) and the calculation result of the second calculation means, the area corresponding to the object to be detected, the background And a dividing unit (for example, a threshold processing unit 33 in FIG. 5) that divides the region into regions considered to be

  The recognition processing unit includes a recognition feature point generation unit (for example, a recognition feature point generation unit 124 in FIG. 19) that generates a plurality of feature points as recognition feature points from the recognition image acquired by the recognition image acquisition unit. Recognition feature value generation means (for example, the recognition feature value generation unit 125 in FIG. 19) that generates a feature value at each of the recognition feature points generated by the recognition feature point generation means as a recognition feature value; and the selected feature value For each of the selected feature quantities stored in the storage means, the recognition feature quantity generated by the recognition feature quantity generation means is selected from the recognition feature quantities having the highest correlation, and the selected feature quantity between the selected feature quantities is selected. A recognition correlation feature value generation unit (for example, a recognition correlation feature value generation unit 126 in FIG. 19) that generates the degree of correlation as a recognition correlation feature value, and the recognition correlation feature value generation unit. Whether the recognition target is included in the recognition image acquired by the recognition image acquisition unit by substituting the generated recognition correlation feature amount into the recognition unit stored by the recognition unit storage unit. Judgment means for judging whether or not (for example, the recognition processing unit 127 of FIG. 19) can be provided.

  The recognition method according to the second aspect of the present invention uses a recognizer generated by learning processing and stored in a storage unit, and a selection feature amount corresponding to each of the recognizers stored in the storage unit. A recognition method of a recognition apparatus (for example, the recognition apparatus 72 in FIG. 19) that performs a recognition process for determining whether or not a recognition target is included in the recognition image, and is used for performing the recognition process. An image is acquired (for example, the process in step S181 in FIG. 34 or the process in step S271 in FIG. 37), and the recognition object is included in the acquired recognition image using the recognizer and the selected feature amount. Whether or not (for example, the process of step S186 of FIG. 34 or the process of step S278 of FIG. 37) includes a step of acquiring a recognized image. Acquire image data (for example, image data picked up by a focus camera as shown in FIG. 3A) that is focused on an image of a subject that is far away and not focused on other objects. (For example, the process of step S41 of FIG. 30), including the step of extracting the portion corresponding to the subject in focus (for example, step S42 of FIG. 30) from the acquired image data. A portion corresponding to the subject is acquired as the recognition image.

  A program according to a second aspect of the present invention uses a recognizer generated by learning processing and stored in a storage unit, and a selection feature amount corresponding to each of the recognizers stored in the storage unit. A program for causing a computer to execute processing for determining whether or not a recognition target is included in a recognition image, and controlling acquisition of the recognition image used for performing recognition processing (for example, step of FIG. 34). S181 or the process of step S271 in FIG. 37), using the recognizer and the selected feature amount, it is determined whether or not the recognition target is included in the acquired recognition image (for example, FIG. 34). Step S186 or the process of step S278 of FIG. 37), and in the process of acquiring the recognized image, the object existing at a predetermined focal length is obtained. Controls the acquisition of image data (for example, image data captured by a focus camera as shown in FIG. 3A) that is in focus on the body image and not in focus on other objects ( For example, the process of step S41 of FIG. 30 includes a step of extracting a portion corresponding to the subject in focus (for example, step S42 of FIG. 30) from the acquired image data, and the extracted subject The computer is caused to execute a process of acquiring a portion corresponding to the above as a recognized image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In performing the image recognition process, both the learning process and the recognition process extract only the learning target or the part to be recognized from the acquired image data. In other words, it should not be used for learning or recognition. It is necessary to remove no background part.

  For example, as shown in FIG. 2, the image recognition processing is largely divided into learning processing for learning an object model for recognition processing and recognition processing using an object model obtained as a result of learning. Both the learning process and the recognition process are performed on an image from which the background is removed.

  That is, during the learning process, the object to be recognized is cut out from the input image data acquired for learning, that is, the background is separated to correspond to the object to be recognized cut out from the input image data. A learning process is executed using the partial image data. The object model data for recognition obtained as a result of the learning process is stored in an object model database (DB). During the recognition process, the object to be recognized is cut out from the input image data acquired for recognition, that is, the background is separated, and the portion corresponding to the object to be recognized cut out from the input image data Whether the object to be recognized is included in the input image data acquired for recognition processing, that is, recognition by referring to the object model data stored in the object model database. Is judged.

  At this time, in order to easily remove the background from the input image in the learning process and the recognition process, it is preferable that the input image data is captured by the focus camera.

  The image data picked up by the focus camera is focused only on an object existing in the vicinity of a predetermined focal length, and is not focused on other objects, that is, the background portion, and is in a so-called out-of-focus state.

  A case where image data including an object to be recognized is picked up by a focus camera and a case where the image data is picked up so as to be in focus to some extent will be described with reference to FIG.

  FIG. 3A is a diagram illustrating image data including an object to be recognized, which is captured by the focus camera. FIG. 3B is a diagram illustrating image data including an object to be recognized, which is captured by normal focus control.

  In the image data shown in FIG. 3B, not only the object to be recognized but also the object that appears in the background is in focus, and the object that appears in the background is clearly imaged. On the other hand, in the image data shown in FIG. 3A, only the object to be recognized is in focus, and the object that appears in the background is out of focus. In other words, in the image data captured by the focus camera, only the object to be recognized appears to be raised.

  FIG. 4 is a block diagram illustrating a configuration of the image processing unit 11 that executes a process of extracting a portion in focus from image data obtained by imaging with a focus camera.

  The image processing unit 11 includes an image acquisition unit 21 and a background separation processing unit 22.

  The image acquisition unit 21 acquires image data obtained by imaging with a focus camera from the outside, or includes an internal focus camera to execute an imaging process and supplies the image data to the background separation processing unit 22.

  The background separation processing unit 22 extracts a focused part from the image data obtained by imaging with the focus camera, and outputs the extracted part of the image data.

  In order to extract the in-focus part from the image data captured by the focus camera, a commonly used image blur detection technique (for example, Blur Detection for Digital Images Using Wavelet Transform) ; Technology described in Hanghang Tong: Mingjing Li, Hongjiang Zhang: Changshiui Zhang) can be applied. That is, it is possible to extract a portion in which the focus is matched by detecting a blurred portion that is out of focus in the image data and deleting that portion.

  Further, by applying FFT (Fast Fourier transform) to the image data obtained by imaging with the focus camera, the frequency spectrum of each image area of the imaged image data is analyzed, and high frequency components are sufficiently contained. On the other hand, it is possible to extract a portion corresponding to an object to be recognized by determining that a region is in focus and a region having a low frequency is out of focus.

  Further, in each pixel of the image data obtained by imaging with the focus camera, it is possible to extract a focused portion by using a luminance difference filter between the target pixel and its neighboring pixels.

  FIG. 5 shows further details of the background separation processing unit 22 when a portion corresponding to a substance to be recognized is extracted from image data including a background obtained by imaging with a focus camera using the above-described luminance difference filter. It is a block diagram which shows a structure.

  A background separation processing unit 22 in the case of extracting a part corresponding to a recognition target substance from image data including a background obtained by imaging with a focus camera includes a neighboring pixel difference filter calculation processing unit 31 and a neighboring region sum filter calculation. A processing unit 32 and a threshold processing unit 33 are included.

  The neighboring pixel difference filter calculation processing unit 31 calculates a difference between adjacent pixel values in order to obtain a feature of the out-of-focus area of the supplied image data. When the pixel is out of focus, the difference between adjacent pixels is small (close to 0). Therefore, the neighboring pixel difference filter calculation processing unit 31 considers differences from the eight neighboring pixels, sets the pixel point of interest as coordinates (x, y), and sets the pixel value at that point to I (x, y ), The following equation (1) is calculated.

(I (x-1, y-1) -I (x, y)) + (I (x, y-1) -I (x, y))
+ (I (x + 1, y-1) -I (x, y)) + (I (x-1, y) -I (x, y))
+ (I (x + 1, y) -I (x, y)) + (I (x-1, y + 1) -I (x, y))
+ (I (x, y + 1) -I (x, y)) + (I (x + 1, y + 1) -I (x, y))
= Σ (I (x + Δx, y + Δy) −I (x, y))
... (1)

  However, in the right side of the equation (1), Δx = −1, 0, 1 and Δy = −1, 0, 1.

  If this is considered as a convolution filter, as shown in FIG. 6, it is a 3 × 3 filter, in which the coefficient for the center, that is, the pixel of interest is 8 and all the coefficients of the neighboring 8 pixels are −1.

  FIG. 7 is a diagram illustrating an output example of the neighboring pixel difference filter calculation processing unit 31 when A in FIG. 3 is an input image. In the calculation result of FIG. 7, the black area is a so-called out-of-focus pixel whose difference from the neighboring pixels is close to 0, and the bright (white) area is a pixel whose pattern is sharp and not blurred.

  However, in FIG. 7, even a portion corresponding to an object to be recognized does not have a texture, for example, a doll's cheek or a doll that is an object to be recognized in the image data described with reference to FIG. 3. Even in this portion and the like, the difference value between adjacent pixels is close to 0, so that the area is black. Therefore, even if the threshold processing is directly performed on the output of the neighboring pixel difference filter calculation processing unit 31, the region corresponding to the object to be recognized cannot be cut out correctly.

  Therefore, the neighborhood region sum filter calculation processing unit 32 performs a filtering process for obtaining an average of the target pixel and its neighborhood region.

  In order to extract the entire object to be recognized from the background, it is necessary to combine “regions that are likely to be recognized” into one region. For this purpose, calculation is performed to obtain the sum of pixel values of the target pixel and the neighboring region. This is a calculation for obtaining the sum of areas of neighboring N × M pixels when attention is paid to a certain pixel, and is hereinafter referred to as a sum filter calculation. The sum filter calculation is synonymous with setting the pixel after filter processing to a value obtained by averaging the peripheral areas of the pixel. The actual calculation process of the sum filter calculation corresponds to applying a convolution filter in which all the elements of the window size N × M are 1. FIG. 8 shows an example of the sum filter when the filter window size is 3 × 3.

  As shown in FIG. 9, the neighborhood region sum filter calculation processing unit 32 performs a sum filter operation of a predetermined window size (corresponding to a square frame on the image in the figure), and a neighborhood pixel difference filter calculation processing unit 31. The calculation result is supplied to the threshold value processing unit 33.

  In the sum filter calculation by the neighborhood region sum filter calculation processing unit 32, the result varies depending on the window size to be applied. The window size and the result of the sum filter calculation will be described with reference to FIGS.

  FIG. 10 is an example of the result of the sum filter calculation when the window size is small. If the window size is too small, an object with little texture will leave a large black area in the small area of the texture (in this case, the doll's nose, cheeks, etc.). I can't.

  FIG. 11 is an example of the result of the sum filter calculation when the window size is larger than that in FIG. As the window size increases, the black areas of the smaller texture areas decrease.

  FIG. 12 is an example of the result of the sum filter calculation when the window size is larger than that in FIG. If the window size is too large, the peripheral area of the object to be recognized also becomes a white area, so that the background area may be cut out.

  As shown in FIGS. 10 to 12, the smaller the window size, the greater the possibility of erroneous detection of a portion with less texture in the object to be detected as the background, and the larger the window size, There is a possibility that the background portion is erroneously detected as a region corresponding to an object to be detected. Thus, there is a trade-off relationship between the window size and the area detection accuracy. Therefore, the window size can be appropriately changed or adjusted based on, for example, whether the texture of the object to be extracted is small or the proportion of the object to be extracted in the entire area of the image data. It is preferable to make it possible.

  The threshold processing unit 33 divides the image data into a region corresponding to the object to be detected and a region considered to be the background based on the sum filter calculation result by the neighborhood region sum filter calculation processing unit 32.

  In the region division, the sum filter calculation result may be separated by a predetermined threshold, that is, binarization processing may be performed, or a portion of the sum filter calculation result where a value is generated (not a black region) It is good also as what masks (part). Note that when binarization is performed by threshold processing, a black region may remain in a white region. Therefore, after binarization, Morphological processing such as expansion / reduction is performed. With this process, even if there is a black hole in the white area, the hole can be filled and the influence of noise can be reduced.

  FIG. 13 is an example of the output of the threshold processing unit 33 for the input image of FIG. 3A when the process of dividing the region by performing the Morphological process after the binarization process. FIG. 14 shows the input image of FIG. 3A when the region dividing process is performed by masking the portion where the value is generated (the portion that is not a black region) in the sum filter calculation result. It is an example of the output of the threshold value process part 33.

  In this way, the background area and the area where the object to be recognized is imaged are divided.

  15 to 18, a portion corresponding to a recognition target substance is extracted from image data including a background obtained by imaging with a focal distance fixed by a focus camera using a brightness difference filter with the vicinity. The output examples of the neighboring pixel difference filter calculation processing unit 31, the neighboring region sum filter calculation processing unit 32, and the threshold processing unit 33 of the background separation processing unit 22 are shown.

  Each of FIGS. 15 to 18 includes, from above, an input image to the background separation processing unit 22, an output from the neighborhood pixel difference filter calculation processing unit 31, an output from the neighborhood region sum filter calculation processing unit 32, and binarization after threshold processing. An output of the threshold processing unit 33 when the morphological processing is performed and an output of the threshold processing unit 33 when the analog mask processing is performed are illustrated.

  As shown in FIGS. 15 to 18, the sum filter calculation and the neighboring pixel difference filter calculation are performed on the image data captured by the focus camera with the focal length matched to the object to be recognized. By performing binarization after threshold processing or performing analog mask processing, the background in the image data and the object to be recognized can be separated.

  By separating the object to be recognized from the background by the process of the background separation processing unit 22, the part corresponding to the object to be recognized is extracted from the image data, thereby improving the efficiency of the learning process and the performance of the recognition process. Improvement can be expected.

  In other words, in object recognition by image processing (object recognition), even in the recognition processing, even when learning processing is performed in order to generate a recognizer, conventionally, an object to be recognized is detected from a human image. Cut out and used the data. On the other hand, if the portion corresponding to the object to be recognized as the background portion of the image data can be automatically separated as described with reference to FIGS. It is not necessary to perform the process of cutting out from the image, which is preferable.

  Next, a case where an object is actually recognized using the portion corresponding to the object to be recognized extracted from the image data by the above-described processing will be described.

  Various methods for object recognition by image processing have been proposed in recent years, and have improved dramatically in the last decade. In many cases, these methods can be applied to, for example, “pet” recognition by using a more flexible recognition method than the conventional methods. Here, “pet” means, for example, one kept in a general household, such as dog, cat, bird, fish, chameleon, hamster, guinea pig, mouse, squirrel, rabbit, turtle, snake, etc. Any kind of animal may be used.

  When the types of pets to be recognized are different, it is conceivable that the recognition accuracy is higher when the recognition is performed using different feature amounts. For example, “four legs”, “joint”, “hair”, “tail”, etc. can be the feature amount for dogs, and the texture of “wings” can be recognized as feature amounts for birds. It is preferable to use it. Therefore, if the device can configure the discriminator by selecting the feature amount specific to the pet without explicitly giving what to adopt as the feature amount, image recognition with a high degree of freedom is achieved. It becomes possible.

  Then, referring to FIG. 19 to FIG. 37, a recognizer that actually recognizes an object is generated by using the portion corresponding to the object to be recognized extracted from the image data by the above-described processing. A specific example of learning processing for the purpose and recognition processing using a recognizer that is the result of learning will be described. The method described below can flexibly cope with the case of recognizing a pet as described above by changing parameters.

  FIG. 19 is a diagram showing a configuration example of the image processing system 51 in the embodiment of the present invention. The image processing system 51 includes a learning device 71 used in the learning phase and a recognition device 72 used in the recognition phase. Further, although the image processing system 51 is illustrated here as being configured by the learning device 71 and the recognition device 72, it may be configured by a single device having the same function. Needless to say.

  The learning device 71 includes a model image acquisition unit 91, a model feature point generation unit 92, a model feature amount generation unit 93, a model feature amount storage unit 94, a learning image acquisition unit 95, a learning feature point generation unit 96, and a learning feature amount generation unit. 97, a learning correlation feature value generation unit 98, an error information acquisition unit 99, and a recognizer generation unit 100.

  The model image acquisition unit 91 has the same configuration as the image processing unit 11 described with reference to FIG. 4, and acquires image data to be a model image obtained by imaging with a focus camera from the outside, Alternatively, a model image is taken with a focus camera inside, and a part in focus is extracted from the model image data obtained by the focus camera, and the extracted part of the image data is used as a model feature point. The data is output to the generation unit 92 and the model feature amount generation unit 93.

  Further, the model image acquisition unit 91 is not configured similarly to the image processing unit 11 described with reference to FIG. 4, and acquires a model image showing only a portion to be recognized, or a user operation, etc. Thus, only the portion to be recognized may be extracted from the predetermined image data and output to the model feature point generation unit 92. In the case where the number of model images is small, the latter configuration may be used, and a portion used for recognition as a model image may be surely cut out by a user operation or the like.

  The model feature point generation unit 92 generates model feature points from the model image supplied from the model image acquisition unit 91 and supplies the model feature points to the model feature amount generation unit 93. As the feature points, any point in the image can be used, and what kind of points are used can be defined depending on the type of feature amount. Specifically, for example, when color is used as a feature point, it is preferable that the feature point is generated in a flat area without texture, and when a shape, movement, texture, or the like is used as the feature point. It is preferable that feature points are generated at the edge portions. In this way, it is possible to appropriately use feature points suitable for the type of feature amount.

  The model feature amount generation unit 93 generates a model feature amount at the model feature point generated by the model feature point generation unit 92 and supplies the model feature amount to the model feature amount storage unit 94. This feature may be either a local feature or a global feature, and its type defines various things related to shape, color, movement, texture, material, walking pattern, etc. Can do. The model feature can be selected according to its type. For example, as the local shape information, the first derivative, second derivative, third derivative or color distribution of the Gauss derivative is selected. Can do. Thereby, the feature amount suitable for the type of feature amount is appropriately used.

  The model feature amount storage unit 94 stores the model feature amount at the model feature point generated by the model feature amount generation unit 93.

  The learning image acquisition unit 95 has the same configuration as that of the image processing unit 11 described with reference to FIG. 4, and acquires learning image data used for learning processing obtained by imaging with a focus camera from the outside. Or, a focus camera is provided inside to capture a learning image, and a focus-matched part is extracted from learning image data acquired by the focus camera, and the extracted part of the image data is learned. The result is output to the feature point generation unit 96 and the learning feature amount generation unit 97.

  The learning feature point generation unit 96 generates a learning feature point from the learning image and supplies the learning feature point to the learning feature amount generation unit 97. As the feature points, any point in the image can be used, and what kind of points are used can be defined depending on the type of feature amount. Specifically, for example, when color is used as a feature point, it is preferable that the feature point is generated in a flat area without texture, and when a shape, movement, texture, or the like is used as the feature point. It is preferable that feature points are generated at the edge portions. Thereby, the feature point suitable for the kind of feature-value can be utilized suitably.

  The learning feature amount generation unit 97 generates a learning feature amount at the learning feature point generated by the learning feature point generation unit 96 and supplies the learning feature amount to the learning correlation feature amount generation unit 98. This feature amount may be any one of a local feature amount and a global feature amount, and various types of shapes, colors, movements, textures, materials, walking patterns, and the like can be defined. The learning feature can be selected according to its type. For example, the first, second, third, and color distributions of Gaussian derivatives are selected as local shape information. Can do. Thereby, the feature amount suitable for the type of feature amount can be used as appropriate.

  The learning correlation feature value generation unit 98 obtains a correlation between each model feature value and each learning feature value, and generates a learning correlation feature value. Details of the correlation feature generation method will be described later with reference to FIGS. 25 and 26.

  The correct / incorrect information acquisition unit 99 acquires information indicating whether each of the learning images is an image including a recognition target included in the model image. The correct / incorrect information is input by a user who supplies a model image or learning image for learning processing to the learning device, or instructs processing for instructing imaging of the model image or learning image, for example.

  The recognizer generation unit 100 performs statistical learning of the recognizer based on the learning correlation feature amount and the correct / incorrect information generated by the learning correlation feature amount generation unit 98, and selects the model feature amount selected in the process as the selected feature amount. And a recognizer obtained as a result of learning is supplied to the recognition device 72. For example, a boosting algorithm can be used to generate the recognizer. This boosting algorithm is based on weighted voting, and for example, Discrete AdaBoost Algorithm, Gentle AdaBoost Algorithm, etc. can be used. Details of the generation of the recognizer will be described later with reference to FIGS. 27 and 28.

  Next, the recognition device 72 includes a selection feature amount storage unit 121, a recognizer storage unit 122, a recognition image acquisition unit 123, a recognition feature point generation unit 124, a recognition feature amount generation unit 125, a recognition correlation feature amount generation unit 126, a recognition A processing unit 127 and a recognition result output unit 128 are included.

  The selected feature amount storage unit 121 corresponds to the model feature amount selected in the course of the learning process in the learning device 71, that is, the recognizer generated by the recognizer generation unit 100 stored in the recognizer storage unit 122. The selected feature quantity, which is the feature quantity to be received, is supplied and stored.

  The recognizer storage unit 122 stores the recognizer generated in the recognizer generation unit 100 by the learning process in the learning device 71.

  The recognition image acquisition unit 123 has the same configuration as that of the image processing unit 11 described with reference to FIG. 4, and acquires learning image data used for recognition processing obtained by imaging with a focus camera from the outside. Alternatively, the camera is equipped with a focus camera inside to capture a recognition image, and a portion in focus is extracted from the recognition image data obtained by capturing with the focus camera, and the extracted portion of the image data is recognized. The result is output to the feature point generation unit 124 and the recognized feature amount generation unit 125.

  The recognition feature point generation unit 124 generates a recognition feature point that is a feature point from the recognition image. As the feature points, any point in the image can be used, and what kind of points are used can be defined depending on the type of feature amount. Specifically, for example, when color is used as a feature point, it is preferable that the feature point is generated in a flat area without texture, and when a shape, movement, texture, or the like is used as the feature point. It is preferable that feature points are generated at the edge portions. Thereby, the feature point suitable for the kind of feature-value can be utilized suitably.

  The recognition feature value generation unit 125 generates a recognition feature value that is a feature value at the recognition feature point generated by the recognition feature point generation unit 124. This feature amount may be any one of a local feature amount and a global feature amount, and various types of shapes, colors, movements, textures, materials, walking patterns, and the like can be defined. The recognition feature can be selected according to its type. For example, as the local shape information, the first derivative, second derivative, third derivative or color distribution of the Gaussian derivative is selected. Can do. Thereby, the feature amount suitable for the type of feature amount can be used as appropriate.

  The recognition correlation feature value generation unit 126 calculates a correlation between each recognition feature value for each of the selection feature values stored in the selection feature value storage unit 121, and generates a recognition correlation feature value. Details of the correlation feature generation method will be described later with reference to FIGS. 25 and 26, as in the learning process.

  The recognition processing unit 127 substitutes the recognition correlation feature value generated by the recognition correlation feature value generation unit 126 to the recognizer stored in the recognizer storage unit 122, so that each recognition image data includes a recognition target. Is recognized, and the recognition result is supplied to the recognition result output unit 128.

  Then, the recognition result output unit 128 displays the recognition result supplied from the recognition processing unit 127, for example, on the display unit, outputs it as voice data, or notifies the user using an LED, Alternatively, the data is output to another device through a predetermined transmission path or recorded on a predetermined recording medium.

  Here, at least one of the model image acquisition unit 91, the learning image acquisition unit 95, and the recognition image acquisition unit 123 has the same configuration as the image processing unit 11 described with reference to FIG. In addition to taking a recognition image with a focus camera, learning processing and recognition that can be realized by making it possible to extract a focused part from the recognition image data obtained by the focus camera. Processing will be described.

  For example, when acquiring image data used in learning processing or recognition processing, an object to be recognized by the user is placed at the focal length set in the focus camera, and then an imaging instruction is given to the device to capture the image. Processing may be performed. However, in such a case, for example, when a moving object such as a pet is recognized or when there are many objects to be recognized, the user recognizes an object recognized at a focal length set in the focus camera. If it is installed one by one, the operation becomes complicated.

  Therefore, the recognized object is appropriately moved so as to fall within the imaging range of the focus camera, and the imaging process is continuously executed during that time. In this way, when the focus is not matched to the recognition target, the extraction process is not performed correctly, or an image that does not include an object to be recognized in the learning process is input, or the recognition is performed. In the processing, an image determined to be an image that does not include a recognized object is input. On the other hand, when the background separation processing is performed based on the image captured when the recognized object is at the focal length set in the focus camera, the area where the object to be recognized is captured It is possible to easily obtain image data obtained by cutting out only the image data without performing complicated operations.

  For example, when the object to be recognized is far from the focal length of the focus camera, the obtained image data may be out of focus as a whole, or may be out of focus at all irrelevant things. Even if the background of such image data is separated, the result is that there is no region of the object to be recognized, and the learning process or the recognition process cannot be executed, or there is something unrelated to it. A part is extracted as a region of an object to be recognized. When a part that has nothing to do with the relationship is extracted as an object region to be recognized, the learning process corresponds to the input of an image that does not include the object to be recognized. Since the feature amount obtained from the part where the non-existent image is captured does not match the stored model feature amount, the result is that the image does not include the object to be recognized. When the object to be recognized matches the focal length of the focus camera, correct extraction processing is performed, and useful model images, learning images, or recognition images are obtained, and learning processing or recognition processing is performed. be able to.

  In this way, for example, even if the pet is moving in front of the camera when it is desired to recognize the pet, the recognition object is correctly extracted only when the focus is matched to the target. The learning process and the recognition process are correctly executed by the obtained image data. In addition, when the user wants to recognize a movable object, for example, by gripping it, the user moves the area appropriately without setting the recognition object exactly at the focal point of the focus camera. If the position matches the focal length of the focus camera, correct extraction processing is performed, and useful model images, learning images, or recognition images are obtained, and learning processing or recognition processing is performed correctly. Can do.

  In particular, when a recognition image is obtained in the recognition processing, it is preferable to appropriately move the recognized object so as to fall within the imaging range of the focus camera and continuously execute the imaging processing during that time. Can be considered. For example, when it is desired to recognize a person passing through the passage, the correct extraction process is automatically performed when the person passing through the passage passes a predetermined position corresponding to the focus position of the focus camera. Thus, the correct recognition process can be performed without requiring the start of imaging.

  As a method of outputting the recognition results, for example, all the recognition results of the recognition processing executed on each of the recognition images captured continuously may be output, or the recognition images captured continuously. The recognition result of the recognition process executed for each of the above is held for a predetermined number of times or for a predetermined time, the transition is observed, and the recognition result at the peak value that seems to be recognized is output. good. Similarly, the transition of the recognition result is observed, and if there is a value equal to or greater than the threshold value, the result may be output as being recognized.

  The outline of the learning phase executed in the learning device 71 will be described with reference to FIG.

Here, feature quantities (model feature points) of N (N is an integer of 2 or more) feature points (model feature points) generated from _X (X is an integer of 2 or more) model images (PM _{1 to} PM _X ). It is assumed that (feature amount) is accumulated in the model feature amount storage unit 94 (feature amount pool). All model images include recognition targets. That is, the feature amount at the feature points of the entire image including the recognition target is accumulated in the model feature amount storage unit 94. In this example, a pet dog is included as a recognition target.

On the other hand, M learning images (PI _{1 to} PI _M ) (M is an integer equal to or greater than 2) include those that include a recognition target and those that do not. Whether or not the recognition target is included is indicated by correct / incorrect information acquired by the correct / incorrect information acquisition unit 99. In the example of FIG. 20, “+1” is assigned when the recognition target is included, and “−1” is assigned when the recognition target is not included. That is, in the learning device 71, when the learning image acquisition unit 95 receives the supply of the learning image including the recognition target, the learning feature amount generation unit 97 obtains the feature amount at the feature points of the entire image including the recognition target. At the same time, the correct / incorrect information acquiring unit 99 receives supply of correct / incorrect information “+1” indicating that the learning image includes a recognition target. Further, in the learning device 71, when the learning image acquisition unit 95 receives the supply of the learning image that does not include the recognition target, the learning feature amount generation unit 97 determines the feature amount at the feature point of the entire image that does not include the recognition target. In addition, the correct / incorrect information acquisition unit 99 is supplied with correct / incorrect information “−1” indicating that the recognition target is not included in the learning image.

  The learning correlation feature value generation unit 98 stores the feature values (learning feature values) at a plurality of feature points (learning feature points) generated for each of the M learning images and the model feature value storage unit 94. The correlation value between the N model feature values is generated, and the learning feature value having the highest correlation is selected for each of the N model feature values, and the N correlation values generated at that time are selected. The value is the correlation feature amount. This correlation feature amount is generated for each of the M learning images, and constitutes M learning correlation feature amounts.

  The recognizer generation unit 100 learns the recognizer based on the learning correlation feature quantity and the correct / incorrect information obtained in this way. This recognizer is for determining whether or not a recognition target is included in the input recognition image in the recognition phase following the learning phase.

  The feature quantity used in the image processing system 51 may be either a local feature quantity or a global feature quantity, and the type of the feature quantity is shape, color, movement, texture, material, walking pattern, etc. Various things can be defined. For example, as the local feature amount related to the shape, the luminance information of the partial area may be used as it is, and Laplacian (secondary derivative), Gaussian derivative function (Gaussian Derivatives), steerable filter (Steerable Filters), Gabor filter ( Gabor Filters) or SIFT (Scale-Invariant Features Transform) may be used. Further, as the local feature amount related to the color, the color information (RGB, HSV, etc.) of the partial area may be used as it is, or information collected as a histogram may be employed. Furthermore, a motion vector (so-called optical flow) can be used as a local feature amount related to motion.

  Animals have characteristics in movement depending on their types, for example, how to carry their feet and how to move. Therefore, in particular, when trying to recognize a pet, if moving image data supplied by a plurality of frame image data can be acquired as model image data, learning image data, and recognition image data, It is considered useful to use the movement of the object as a feature amount.

  Specifically, for example, by using a method for describing movement represented by optical flow, it is possible to use the movement of the recognition target object as a feature amount. The optical flow is a representation of the motion of an object as a vector in a visual expression (usually temporally continuous image data).

  As the feature point, any point in the image can be used, but in general, an edge or a corner point is often used. This feature point can be defined by the type of feature amount. For example, as for the feature amount related to the shape, it is desirable to employ the edge or corner point as the feature point because the feature easily appears at the edge or corner point. On the other hand, with regard to feature quantities related to color, since features tend to appear in the object area, random points can be adopted as feature points without being limited to specific points, or feature points can be found from parts without texture far from the edge part. It is desirable to adopt.

  In order to obtain an edge or a corner point as a feature point related to the shape, a Harris corner detector can be used. In this Harris corner point detector, first, the luminance gradient is obtained at each pixel point I (x, y) in the image data, and the second moment matrix M in the local region is expressed by the following equation (2). calculate.

  Assuming that the two eigenvalues of the second moment matrix M are α and β, of the eigenvalues α and β, if both are greater than a predetermined threshold, a corner point, if one is greater than the predetermined threshold, both are predetermined thresholds. If it is smaller, there is nothing. Therefore, in order to make this determination, a determinant det (M) and a trace (sum of diagonal components) trace (M) of the second moment matrix M are calculated, and a corner is calculated using the following equation (3). A response function CR is obtained.

        CR = det (M) -k (trace (M)) 2 (3)

  Here, in Equation (2), it can be assumed that k = 0.04.

  If this corner response function CR is a positive number, it means a corner point, and if it is a negative number, it means an edge. However, there is nothing when the corner response function CR is smaller than a certain value. A corner point or an edge can be extracted by such a procedure.

  Although the corner response function CR by subtraction is used here to determine the corner point or edge, division may be used as in the following equation (4).

        CR = det (M) / (k (trace (M)) 2) (4)

Also, when SteerableFilters (Gausian Derivatives) are used as feature quantities related to the shape, calculation of the base kernel is performed by the Gaussian function and its differential function expressed by the following equations (5) to (10), and the linear combination is performed. Expressed. G is a Gaussian function, G ₁ is a first derivative, G ₂ is a second derivative, and G ₃ is a third derivative function. θ is the direction of the filter to be calculated. For example, the feature amount can be obtained by equally dividing pi into four directions or equally dividing into eight directions.

  FIG. 21 shows the shape of the SteerableFilter kernel in two dimensions calculated by the above formula.

  In order to strengthen the local feature quantity as the feature quantity, a combination of neighboring jets may be used. At this time, as shown in FIG. 22, it is preferable to pick up the jet from a location about 5 pixels away from the target pixel. If the jet used for combining is too far away from the pixel of interest, the local information becomes weak against deformation of the object. Conversely, if the jet used for combining is too close to the pixel of interest, the meaning of combining many jets will be reduced.

  Further, the local feature amount can be made invariant with respect to the rotation. For example, as shown in FIG. 23, by calculating the main direction α at the center pixel point and rotating the feature amount in the direction, the local feature amount can be made invariant to the rotation. it can. The main direction α is obtained by the following expression (11) from the x-direction and y-direction outputs of the first derivative of a certain Gaussian width σ.

  If this α is used, an output in four directions can be obtained using, for example, the following equation (12).

  On the other hand, when a color histogram is used as a color feature point, the color space is divided into predetermined color areas, and distributions in the respective color areas are obtained. FIG. 24 is an example of a histogram in the HSV space. In this HSV expression, H (Hue) represents hue, S (Saturation) represents saturation, and V (Value) represents lightness.

  In FIG. 24A, as a simple example, each HSV component is divided into two sections, and a total of eight (= 23) color regions are provided. A histogram shown in FIG. 24B is obtained by calculating the appearance frequency in each color region from the color distribution in the image region including the vicinity (for example, about 10 pixels) of a certain feature point. The eight frequency data of the histogram shown in B of FIG. 24 respectively correspond to any of the eight color regions shown in A of FIG.

  As described above, the feature points and feature quantities used in the learning process and the recognition process can be defined as appropriate according to the type of feature quantity. The feature quantity obtained in this way is converted into a correlation feature quantity in the learning correlation feature quantity generation unit 98 and the recognized correlation feature quantity generation unit 126. The learning correlation feature value generation unit 98 compares various feature values in the same dimension by obtaining a correlation with the model feature value for each of the learning feature values, and the result is compared with the recognition device in the recognizer generation unit 100. Supply for learning. In addition, the recognition correlation feature value generation unit 126 obtains a correlation between each of the recognition feature values and the selected feature value of the same dimension among various feature values stored in the selection feature value storage unit 121. Various feature values are compared in the same dimension, and the result is supplied to the recognition processing unit 127 for use in recognition processing.

  In general, the correlation value C between the two vectors v1 and v2 representing the feature amount is calculated by the following equation (13). Note that the upper line of the vector represents the average of the vectors.

  The correlation value C shown in the equation (13) is a value in the range from 0.0 to 1.0. The higher the correlation, the closer to 1.0, and the lower the correlation, the closer to 0.0.

  Further, when obtaining the correlation value, an elastic bunch graph matching (EBGM) method may be used. When this EBGM method is used, the learning correlation feature quantity generation unit 98 has the highest correlation point in the vicinity of the feature point corresponding to the model feature quantity stored in the model feature quantity storage unit 94 among the learning feature quantities ( (Correlation maximum point) is obtained, and the correlation value at the maximum correlation point is used as a learned correlation feature quantity. Further, when this EBGM method is used, the recognition correlation feature quantity generation unit 126 has the highest correlation in the vicinity of the feature point corresponding to the selection feature quantity stored in the selection feature quantity storage unit 121 among the recognition feature quantities. A point (maximum correlation point) is obtained, and the correlation value at the maximum correlation point is used as a recognized correlation feature quantity.

  A search example of the maximum correlation point by the EBGM method will be described with reference to FIG. Here, the case of the learning process in the learning device 71 will be described as an example.

  As shown in FIG. 25, when the feature point α is generated in the model image, the point α ′ on the learning image corresponding to the feature point α is determined. The learning correlation feature value generation unit 98 calculates a correlation value with the feature point α in the vicinity of the point α ′ on the learning image to obtain the maximum correlation point β. The correlation value at the maximum correlation point β becomes the learning correlation feature amount.

  As described above, by using the EBGM method when obtaining the correlation feature amount, the object distortion and viewpoint change are robust, and it becomes possible to deal with these disturbances more flexibly.

  Here, the case where the EBGM method is used when the learning correlation feature value generation unit 98 obtains the learning correlation feature value has been described, but the same applies when the recognition correlation feature value generation unit 126 obtains the recognition correlation feature value. The EBGM method can be applied to.

  Next, an example of correlation feature amount calculation using a plurality of types of feature amounts will be described with reference to FIG. Here, the case of the learning process in the learning device 71 will be described as an example.

  As shown in FIG. 26, the types of model feature values stored in the model feature value storage unit 94 include, for example, a model feature value related to color, a model feature value related to shape, and a model feature value related to motion. To do.

  The learning correlation feature value generation unit 98 calculates a correlation for each type of feature value. That is, in the case of FIG. 26, the learning correlation feature value generation unit 98 includes a correlation calculation unit 141 that calculates a color-related correlation, a correlation calculation unit 142 that calculates a shape-related correlation, and a correlation calculation unit 143 that calculates a motion-related correlation. including. The correlation calculation unit 141 extracts a learning feature amount related to color as a corresponding learning feature amount from the learning feature amount generated by the learning feature amount generation unit 97 for the model feature amount related to color, and The correlation value is calculated and output to the recognizer generation unit 100. Similarly, the correlation calculation unit 142 extracts the learning feature amount related to the shape as the corresponding learning feature amount from the learning feature amount generated by the learning feature amount generation unit 97 for the model feature amount related to the shape. Is calculated and output to the recognizer generation unit 100. In addition, the correlation calculation unit 143 extracts a learning feature amount related to the motion as a corresponding learning feature amount from the learning feature amount generated by the learning feature amount generation unit 97 for the model feature amount related to the motion. The correlation value between them is calculated and output to the recognizer generation unit 100.

  In this way, the correlation calculation units 141 to 143 calculate correlation values for different types of feature amounts. Since the original feature values themselves have different vector dimensions depending on the type of feature values, it is difficult to compare them as they are. However, since the learning correlation feature value generation unit 98 normalizes to a correlation feature value indicating a value in a certain range (0.0 to 1.0) according to the degree of correlation, it is a different type of feature value. Even compatible.

  And the recognizer production | generation part 100 implement | achieves the object recognition by the statistical learning using various kinds of feature-values by learning a recognizer using such a correlation feature-value, and performing recognition. it can.

  Here, the processing when the learning correlation feature value generation unit 98 converts the learning feature value into the learning correlation feature value has been described. However, the recognition correlation feature value generation unit 126 converts the recognition feature value into the recognition correlation feature value. In this case, basically the same processing is executed. Although the case where three types of feature quantities having different colors, shapes, and movements are used has been described here, it goes without saying that the types of feature quantities and the number of types are not limited thereto.

  Next, an example of learning processing executed in the recognizer generation unit 100 will be described with reference to FIGS. 27 and 28.

In FIG. 27, each of the correlation feature amounts of the _M learning images (PI _{1 to} PI _M ) is an N-dimensional vector corresponding to the number N of feature points of the model feature amount stored in the model feature amount storage unit 94. It is expressed as That is, the correlation feature quantity of the _first learning image PI ₁ is (A ₁ , A ₂ ,..., A _N ), and the correlation feature quantity of the _second learning image PI ₂ is (B ₁ , B _2, .., B _N ) The correlation feature amount of the _third learning image PI ₃ is represented as (C ₁ , C ₂ ,..., C _N ), and the M-th learning image PI in the same manner. correlation feature quantity of _M is represented as _{(M 1, M 2, ···} , M N).

At this time, assuming the group G _rk for the feature point k of the model feature amount, the correlation feature amount of the feature point k = 1 is indicated by the group G _r1 (A ₁ , B ₁ , C ₁ ,... M ₁ ), and similarly, the correlation feature quantity of the feature point k = 2 is (A ₂ , B ₂ , C ₂ ,..., M ₂ ) indicated by the group G _r2 . The correlation feature quantity of the feature point k = N is _represented by the group G _rN (A _N , B _N , C _N ,..., M _N ). That is, for each feature point k, a total of M correlation feature amount groups G _rk are defined corresponding to the _M learning images PI _{1 to} PI _M.

  Note that the value “+1” or “−1” at the left end is correct / incorrect information for each learning image that is supplied from the correct / incorrect information acquisition unit 99 and indicates whether or not the corresponding learning image includes a recognition target. .

For each feature point k, M correlation feature amounts are extracted by lottery according to the weight wi set for each learning image (PI _i ) (i is any one of 1 to M). In the first process, all the weights wi are equal, and if M pieces are selected, all the correlation feature quantities are selected probabilistically. Therefore, in the first process, all correlations are performed at each feature point k. It is assumed that a feature amount is selected. In the subsequent iterations, the same correlation feature amount may be selected redundantly.

The M input feature values sampled for each of the N input feature values are rearranged in ascending order or descending order. Then, based on correct / incorrect information indicating whether or not the learning target image from which the input feature amount is extracted includes the target object to be recognized, that is, (+1) or (−1) in FIG. Then, for each of the N input feature quantities rearranged in ascending order or descending order, a certain threshold th _jk is set to divide the feature quantity in the group G _rk into two. When the error rate e _jk of the group G _rk for each feature point k is calculated by the following equation (14) while changing the threshold value, whether or not right and wrong are correctly divided between the threshold value and the threshold value is The threshold is set so that the error rate e _jk is minimized. Here, j is a counter that counts the number of L weak recognizers f _jk (x) (L is an integer of 1 or more) with respect to the correlation feature quantity vector x at the feature point k, and is an integer indicating a range from 1 to L It is.

  Here, y ≠ fjk indicates the condition of the feature point k in error, and Ew indicates that the weight at the feature point k where the error has occurred is added.

This threshold th _jk is set as a weak recognizer.

In the example shown in FIG. 28, an example of setting the threshold th _{11 at} the first feature point k = 1 is shown with J = 1. Specifically, for example, M feature amounts corresponding to the feature point k = 1 should rise to L ₁ , A ₁ , C ₁ , B ₁ ,..., M ₁ as shown in FIG. If the range is smaller than the threshold value, it is recognized that there is no target object to be recognized, and if it is larger than the threshold value, it is recognized that there is a target object to be recognized. Here, the teacher label y (that is, correct / incorrect information) and the weak recognizer f _jk (x) indicate a value of “+1” or “−1” depending on the presence / absence of a recognition target. It shows what was right. As shown in FIG. 28, when the threshold th ₁₁ is set between the feature amounts A ₁ and C ₁ , the feature amount A ₁ surrounded by the dotted line in the figure includes the target object to be recognized. On the other hand, the feature amount C ₁ and the feature amount M ₁ are regarded as errors because they are the feature amounts of the learning image that do not include the target object to be recognized. Then, the value of Ew is set by accumulating the number of errors as if an error has occurred when the prediction is wrong.

In this way, the weight of the learning image from which the feature amount considered to be an error is extracted based on the correctness information of the learning image (information on whether or not the target object to be recognized is included) The error rate e _jk is calculated by adding Wi.

When the error rate e _jk is calculated in this way, the weak recognizer f _jk (x) having the smallest error rate e _jk is then selected from the set weak recognizers f _jk (x). The Then, the reliability c _{j of} the weak recognizer f _jk (x) is calculated by the following equation (15) using the error rate e _jk .

Further, the weight wi (i is an integer indicating a range from 1 to N) of the learning image is calculated using the following expression (16) based on the reliability c _j thus obtained, and the sum of wi Is further normalized and then updated.

  As a result, the weight of the learning image including the correlation feature quantity in which the error has occurred becomes large, and the learning image that needs to be learned again is clearly distinguished.

The weak recognizer f _jk (x) selected in this way is weighted by the reliability c _j shown in Expression (15), and the recognizer R (x) for the correlation feature vector x is expressed by the following expression ( It is updated as in 17).

That is, the weighted weak recognizer f _jk is added to the already held recognizer R (x), and is updated as a new recognizer R (x). That is, the generated recognizer R (x) includes a plurality of weak recognizers f _jk having a relatively low error rate.

  The recognizer generation unit 100 repeats such learning processing, and as a result, if R (x) is a positive number, it indicates that the recognition target is included, and if it is a negative number, it indicates that the recognition target is not included. The shown recognizer R (x) can be generated. That is, this recognizer is a function that outputs the presence / absence of a target object to be recognized by the majority decision of weak recognizers. The recognizer generation unit 100 supplies the generated recognizer to the recognizer storage unit 122 of the recognition device 72 to be stored.

Then, the recognizer generation unit 100 selects the model feature quantity of the feature point k to be used in each weak recognizer f _jk that minimizes the error rate e _jk and outputs it as the selected feature quantity. The output selected feature amount is stored in the selected feature amount storage unit 121 of the recognition device 72.

  The learning process for generating a recognizer by repeatedly adding a weak recognizer while weighting it by the learning process is a kind of boosting (weighted voting) algorithm and is called “Discrete AdaBoost Algorithm”. . In this learning process, the recognizer and error rate are calculated for each model feature so that the weight of the learning feature with a high error rate increases sequentially and the weight of the learning feature with a low error rate decreases. Is repeated. Therefore, the learning correlation feature amount selected when setting the recognizer in the iterative process is gradually selected with a high error rate, and is selected so that the learning correlation feature amount that is difficult to recognize is repeated. Since learning is repeated, more correlation feature amounts of learning images that are difficult to recognize are selected, and a high recognition rate can be finally achieved.

Also, according to this boosting algorithm, the model feature quantity of the feature point k that is the smallest among the N error rates e _jk is selected and stored in the selected feature quantity storage unit 121. Learning and feature quantity selection can be performed at the same time, and feature quantities suitable for recognition can be efficiently used without using all the feature quantities stored in the model feature quantity storage unit 94 in the recognition phase. it can.

  Next, processing executed by the image processing system 51 in FIG. 19 will be described with reference to flowcharts in FIGS. 29 to 37.

  First, the learning process executed by the learning device 71 of the image processing system 51 will be described with reference to the flowchart of FIG.

  In step S <b> 11, the model image acquisition unit 91 acquires a model image used for extracting the feature amount of the model image and supplies the model image to the model feature point generation unit 92.

  Here, the model image acquisition unit 91 acquires a model image indicating only a portion to be recognized, or extracts only a portion to be recognized from predetermined image data by a user operation or the like, Although it may be output to the model feature point generation unit 92, it may have the same configuration as the image processing unit 11 described with reference to FIG.

  In step S <b> 12, the model feature point generation unit 92 generates a feature point of the model image supplied from the model image acquisition unit 91 and supplies the feature point to the model feature amount generation unit 93. For example, N model feature points are generated for one model image.

  In step S <b> 13, the model feature quantity generation unit 93 generates a feature quantity at the model feature point of the model image and supplies it to the model feature quantity storage unit 94. For example, when N model feature points are generated, N model feature amounts at the N model feature points are generated by the model feature amount generation unit 93.

  In step S <b> 14, the model feature quantity storage unit 94 stores the model feature quantity at the model feature point generated by the model feature quantity generation unit 93.

  In step S15, it is determined whether or not the feature values of all model images are stored in the model feature value storage unit 94. If it is determined that the feature values are not stored, the process returns to step S11, and thereafter. The process is repeated.

  Here, it has been described that the processing of step S12 to step 15 is executed for the acquisition of one model image, and these processes are repeated when a plurality of model images are acquired. However, in step S11, A plurality of model images may be acquired, and the subsequent processing may be executed sequentially or in parallel for each model image.

  If it is determined in step S15 that the feature values of all model images have been stored in the model feature value storage unit 94, a learning image acquisition process described later with reference to FIG. 30 is executed in step S16.

  In step S <b> 17, the learning feature point generation unit 96 generates a feature point of the learning image acquired by the process of step S <b> 16 and supplies the learning feature amount generation unit 97.

  In step S <b> 18, the learning feature amount generation unit 97 generates a feature amount of the learning image at the learning feature point generated by the learning feature point generation unit 96, and supplies it to the learning correlation feature amount generation unit 98.

  In step S19, a learning correlation feature value generation process, which will be described later with reference to FIG. 32, is executed. In this process, for example, for each of N model feature amounts stored in the model feature amount storage unit 94, a correlation value between learning feature amounts of learning feature points in each of the learning images is a learning correlation. The process that is generated by the feature quantity generation unit 98 and has the highest correlation is the learning correlation feature quantity.

  In step S20, a recognizer generation process to be described later with reference to FIG. 33 is executed. In this process, statistical learning is performed by the recognizer generation unit 100 based on the learning correlation feature amount generated in step S19.

  In step S21, it is determined whether or not processing has been completed for all supplied learning images. If it is determined in step S21 that the processing for all supplied learning images has not been completed, the processing returns to step S16, and the subsequent processing is repeated. If it is determined in step S21 that the processing for all supplied learning images has been completed, the recognizer generated by the processing in step S20 is supplied to and stored in the recognizer storage unit 122 of the recognition device 72. At the same time, the model feature value selected in the process is supplied to and stored in the selected feature value storage unit 121 of the recognition device 72, and the process is terminated.

  Through such processing, learning processing is executed, and a recognizer capable of recognizing an object included in the image is generated. In this process, the learning feature value is converted into a learning correlation feature value and learning of the recognizer is performed, so that different types of feature values can be handled under the same scale and statistical learning can be performed.

  Next, the learning image acquisition process executed in step S16 in FIG. 29 will be described with reference to the flowchart in FIG. This learning image acquisition process has the same configuration as that of the image processing unit 11 described with reference to FIG. 4, and is obtained by capturing a learning image with a focus camera therein and by capturing with a focus camera. This is executed by the learning image acquisition unit 95 that can extract the portion in focus from the learning image data. Therefore, in the flowchart of FIG. 30, it is assumed that the learning image acquisition unit 95 has the configuration of the image processing unit 11 described with reference to FIGS.

  In step S <b> 41, the image acquisition unit 21 of the learning image acquisition unit 95 executes an imaging process at a predetermined focal length, and supplies the obtained image to the background separation processing unit 22.

  In step S42, the background separation processing unit 22 of the learning image acquisition unit 95 performs background separation processing described later using the flowchart of FIG.

  In step S43, the background separation processing unit 22 of the learning image acquisition unit 95 determines whether there is an object to be recognized in the image from which the background is separated. In step S43, when it is determined that there is no object to be recognized, for example, when the entire image data is out of focus, the process returns to step S41, and the subsequent processes are repeated.

  If it is determined in step S44 that there is an object to be recognized, in step S43, the background separation processing unit 22 of the learning image acquisition unit 95 performs image data on which the background has been separated, as necessary. Image processing such as alignment is performed.

  In step S45, the background separation processing unit 22 of the learning image acquisition unit 95 outputs an image corresponding to the object to be recognized from which the background is separated to the learning feature point generation unit 96 and the learning feature amount generation unit 97, The process returns to step S16 in FIG. 29 and proceeds to step S17.

  Through such processing, learning image data is acquired. Since the image data is picked up by the focus camera, it is possible to separate the object to be recognized existing at the position where the focus is matched from the background by an easy process.

  Next, the background separation process executed in step S42 of FIG. 30 will be described with reference to the flowchart of FIG. Also in this flowchart, processing will be described using the configuration of the image processing unit 11 described with reference to FIGS. 4 and 5.

  In step S81, the neighboring pixel difference filter calculation processing unit 31 of the background separation processing unit 22 performs the calculation of Expression (1), performs the calculation process using the neighboring pixel difference filter as shown in FIG. The output as described above is supplied to the neighborhood region sum filter calculation processing unit 32.

  In step S82, the neighborhood region sum filter calculation processing unit 32 obtains an average of the pixel of interest and the neighborhood region by performing a computation process using the neighborhood region sum filter as described with reference to FIG. 33. In the sum filter calculation by the neighborhood region sum filter calculation processing unit 32, the result changes depending on the window size to be applied. Therefore, an optimum window size determined by the size of the texture of the object to be recognized can be used. Is preferred.

  In step S83, the threshold processing unit 33 separates the sum filter calculation result by a predetermined threshold, that is, performs binarization processing, or a portion where a value is generated in the sum filter calculation result (in a black region) The background portion is separated from the portion corresponding to the object to be recognized, for example, by performing threshold processing for masking the non-existing portion.

  By such processing, it is possible to separate an object to be recognized existing at a position where the focus is matched from the background by an easy processing from the image captured by the focus camera.

  Next, the learning correlation feature value generation process executed in step S19 in FIG. 29 will be described with reference to the flowchart in FIG.

  In step S111, the learning correlation feature value generation unit 98 sets k = 1 as a variable k indicating the number of processed feature values.

  In step S <b> 112, the learning correlation feature value generation unit 98 learns the feature value corresponding to the feature point of the learning image with respect to the model feature value of the feature value k that is the k-th feature value among the N feature values of the model image. A correlation value is generated.

  In step S113, the learning correlation feature value generation unit 98 selects the learning feature value having the highest correlation from the generated correlation values.

  In step S114, the learning correlation feature value generation unit 98 sets the correlation value of the learning feature value selected in step S113 as the learning correlation feature value of the feature value k.

  In step S115, the learning correlation feature value generation unit 98 determines whether or not the variable k is the total number N of feature values for one image data.

  If it is determined in step S115 that the variable k is not N, that is, it has not reached N, in step S116, the learning correlation feature value generation unit 98 increments the variable k by 1, and the process proceeds to step S112. Return to, and the subsequent processing is repeated.

  If it is determined in step S115 that the variable k is N, the process returns to step S19 in FIG. 29 and proceeds to step S20.

  By such processing, for example, the learning correlation feature amount is generated by using the method described with reference to FIG. In addition, as described with reference to FIG. 26, the learning correlation feature quantity can be obtained by calculating the correlation of different types of feature quantities (for example, shape, color, motion, etc.).

  Next, the recognizer generation process executed in step S20 of FIG. 29 will be described with reference to the flowchart of FIG.

  In step S141, for example, the recognizer generation unit 100 initializes all the weights Wi for each learning image to 1 / M, and initializes the counter Q to 1 and the recognizer R (x) to 0, respectively. . Here, i identifies each of the plurality of learning input images PIi, and 1 <i <M. Therefore, all the learning images PIi are set to the same normalized weight (= 1 / M) by the processing in step S141.

  In step S142, the recognizer generation unit 100 supplies each local feature amount combination of the feature points k (k = 1, 2, 3,... N), that is, one learning image. For each of the N × P feature amounts, M feature amounts are selected according to the weight Wi of the learning input image PIi.

In this case, the feature quantity in the first combination of local feature quantities at the feature point k = 1 is represented by a group G _r1-1 (A _1-1 , B _1-1 , C _1-1 ,... M _1-1 ). Similarly, the feature quantities in the second combination of local feature quantities at the feature point k = 1 are represented by a group G _r1-2 (A _1-2 , B _1-2 , C _1-2 ,... M _1-2 ), and similarly, the feature amount in the combination of the P-th local feature amount at the feature point k = N is indicated by a group G _rN-P (A _NP , _BNP , _CNP ,... _MNP ).

  That is, a group of M feature amounts based on the learning image PIi is set for the P types of combinations of the local feature amounts of the feature points k.

  The recognizer generation unit 100 extracts M feature quantities by lottery according to the weight set for each learning image PIi for each P type combination of local feature quantities of each feature point k. In the first process, since all the weights Wi are equal, when M pieces are selected, all feature quantities are selected stochastically. Here, in the first process, each feature point k is selected. It is assumed that all feature quantities are selected in each combination of local feature quantities. Of course, in practice, the same feature amount may be selected in duplicate.

  In step S143, the recognizer generation unit 100 increases or decreases the feature amount for each group of M feature amounts sampled for each combination of local feature amounts of the N feature points. Sort in order of power.

In step S144, the recognizer generation unit 100 determines whether the feature point k is based on information indicating whether the target object to be recognized is included in the learning image from which the input feature amount is extracted. for each combination of each of the P type local feature quantity for each of M feature quantity, while changing the threshold value, to calculate the error rate e _jk as shown in the above expression (14), the error rate e _jk The threshold value is set so that is minimized. Here, the threshold th _jk for each combination of local feature quantities of the feature point k is one weak recognizer f _jk . That is, N × P weak recognizers f _jk are set for each combination of P types of local feature amounts of N feature points k, that is, according to N × P feature amounts, The error rate e _jk is obtained for each of N × P (for each weak recognizer f _jk ). Here, the recognizer f _jk is a function that outputs “+1” when the target object to be recognized is included, and outputs “−1” when the target object to be recognized is not included.

That is, in the same manner as described with reference to FIG. 28, feature amounts (obtained correlation coefficients) corresponding to combinations of a certain local feature amount at a certain feature point are arranged in ascending or descending order. The target value to be recognized as the feature quantity corresponding to the learning image including the position of the set threshold th _jk and the target object to be recognized on either side with respect to the threshold value. Whether or not there is an error is determined based on whether or not the feature amounts corresponding to the non-learning images are arranged. The recognizer generation unit 100 is an error based on correct / incorrect information (information on whether or not a target object to be recognized is included) of the input image for learning, as indicated by the above-described equation (14). An error rate e _jk is calculated by adding the weights Wi of the learning input image from which the feature amount regarded as is extracted.

In step S145, the recognizer generation unit 100 selects a weak recognizer f _jk having a minimum error rate e _{jk from} among the N weak recognizers f _jk .

In step S146, the recognizer generation unit 100 calculates the reliability c _jk based on the minimum error rate e _jk of the selected weak recognizer as represented by the above-described equation (15).

In step S147, the recognizer generation unit 100 recalculates the weights Wi for each learning input image based on the supplied reliability c _jk as shown in the above-described equation (16), The weight Wi is normalized and updated. And the recognizer production | generation part 100 sets the weight for every learning input image based on the update result of a weight.

In step S148, the recognizer generation unit 100, based on the recognizer f _jk selected, temporarily stores the Q-th recognizer f _Q. In other words, learner generation block 100 causes the updated (Q-1) -th recognizer recognizers f _Q-1, Q-th recognizer f _Q plus recognizer f _jk selected.

That is, the recognizing device generating unit 100 updates the recognizing device R (x) as represented by the above-described equation (17). In this way, the weighted weak recognizer f _jk is added to the recognizer R (x).

In step S149, the recognizer generation unit 100 temporarily stores a model feature amount in a combination of local feature amounts corresponding to the feature point k of the weak recognizer f _jk as a selected feature amount.

  In step S150, the recognizer generation unit 100 determines whether or not the value of the counter Q is larger than the number of repetitions L for generating the recognizer.

  If it is determined in step S150 that the value of the counter Q is not greater than L, in step S151, the recognizer generation unit 100 increments the counter Q by 1, and then the process returns to step S142 and thereafter. The process is repeated. If it is determined in step S150 that the counter Q is greater than L, the currently stored recognizer R (x) is supplied to the recognizer storage unit 122 of the recognizer 72 and is currently stored. The selected feature amount is supplied to the selected feature amount storage unit 121 of the recognition device 72, and the process returns to step S20 in FIG. 29 and proceeds to step S21.

Through the above processing, a recognizer R (x) including L weak recognizers f _Q (1 <Q <L) having a relatively low error rate is generated and stored in the recognizer storage unit 122 of the recognizer 72. with the model feature quantity of feature point k to be used in each weak learner f _Q as the selected feature quantity, it is stored in the selected feature amount storage unit 121 of the recognition device 72. Here, the number of repetitions L is L ≦ N × P.

  In addition, it can be said that the recognizer of Formula (17) is a function which outputs the presence or absence of the target object to recognize by the majority decision of L weak recognizers. Also, learning processing for repeatedly generating weak recognizers by repeatedly adding weak recognizers while weighting them by learning processing is referred to as Discrete Adaboost Algorithm.

  That is, by the above recognizer generation processing, the weight of the learning input feature amount of the learning input image having a high error rate is sequentially increased, and the weight of the learning input feature amount having a low error rate is decreased. For each quantity, the process of calculating the recognizer and the error rate is repeated. Accordingly, the learning input feature quantity (learning feature quantity selected in step S142) selected when setting the recognizer in the iterative process (the processes in steps S142 to S150) gradually has a high error rate. Since it becomes easy to select, the learning input feature quantity that is difficult to recognize is repeated and learning is repeated, so that more feature quantities of the learning input image that are difficult to recognize are selected. As a result, a recognizer having a high recognition rate can be finally generated.

  In the iterative process (the processes in steps S142 to S150), the recognizer generation unit 100 always selects the weak recognizer corresponding to the model feature having the lowest error rate, so the learning process is repeated. Thus, the weak recognizer for the model feature having the highest reliability is always selected and added to the recognizer, and the weak recognizer with high accuracy is sequentially added each time it is repeated.

That is, by the learning process described above, a recognition made up of L weak recognizers f _{jk having} a low error rate e _jk is used for each feature point and each combination using a feature value obtained by adding a geometric constraint to the feature value. A device R (x) will be generated. As a result, since a recognizer consisting only of a weak recognizer with high reliability is configured, it becomes possible to configure a highly reliable recognizer with a limited number of weak recognizers, It is possible to improve recognition accuracy while reducing the number of arithmetic processes in recognition processing described later.

  Further, if the number of weak recognizers is increased (L is increased as described above), the recognition accuracy by the recognizer can be improved. On the other hand, even if the number of weak recognizers is small (even if L described above is reduced), the weak recognizers that are selected perform recognition processing using only weak weak recognizers with high reliability. Therefore, it is possible to reduce the number of arithmetic processes in the recognition process while maintaining the reliability. In other words, if necessary, it is possible to generate higher-accuracy recognizers by increasing the number of recognizers generated by spending time on learning processing, and conversely, generating them without taking effort in learning. Even if learning is close to one-shot learning by reducing the number of recognizers to be performed, it is possible to generate a relatively highly accurate recognizer.

  As described above, in the learning device 71, the recognizer is learned using the learning correlation feature value generated by the learning correlation feature value generation unit 98, and the recognition correlation generated by the recognition correlation feature value generation unit 126 is used. Since recognition processing is performed using feature amounts, different types of feature amounts can be compared with each other under the same scale to determine whether a recognition target exists. That is, the learning device 71 can appropriately use various feature amounts when performing object recognition. For this reason, the learning device 71 can automatically select and use types of feature amounts suitable for recognition from various types of feature amounts prepared in advance, and can also use various features prepared in advance. A feature quantity suitable for recognition can be automatically selected from the quantities and used. Furthermore, the learning device 71 can automatically statistically learn feature points suitable for recognition.

  Here, as an example of the boosting algorithm, the application example of the Discrete AdaBoost Algorithm has been described. However, other boosting algorithms may be applied, for example, the Gentle AdaBoost Algorithm may be used. According to this Gentle AdaBoost Algorithm, each weak recognizer outputs a continuous variable value including reliability, so that corresponding weighting is performed and calculation of reliability can be omitted.

  Next, the recognition process 1 which is an example of the recognition process which the recognition apparatus 72 performs is demonstrated with reference to the flowchart of FIG.

  In step S181, a recognition image acquisition process described later with reference to FIG. 35 is executed.

  In step S <b> 182, the recognition feature point generation unit 124 generates a feature point of the recognition image acquired by the processing in step S <b> 181 and supplies the feature point to the recognition feature amount generation unit 125.

  In step S <b> 183, the recognition feature value generation unit 125 generates a feature value of the recognition image at the recognition feature point generated by the recognition feature point generation unit 124, and supplies it to the recognition correlation feature value generation unit 126.

  In step S184, recognition correlation feature value generation processing, which will be described later with reference to FIG. 36, is executed. This process is a process for obtaining a correlation between each selected feature quantity stored in the selected feature quantity storage unit 121 and each recognized feature quantity.

  In step S 185, the recognition processing unit 127 performs calculation processing for substituting the recognition correlation feature value generated by the recognition correlation feature value generation unit 126 into the recognizer stored in the recognizer storage unit 122.

  In step S186, the recognition processing unit 127 determines whether or not an object to be recognized is included in the recognition image based on the calculation result in step S185, and supplies the result to the recognition result output unit 128.

  In step S187, the recognition result output unit 128 displays the recognition result supplied from the recognition processing unit 127 on the display unit, outputs it as audio data, or notifies the user using an LED or the like. Or by recording the data on a predetermined recording medium through a predetermined transmission path or the like, and outputting to another device, and the processing is terminated.

  By such processing, the recognition process is executed by converting the recognition feature amount into the recognition correlation feature amount using the recognizer learned by converting the learning feature amount into the learning correlation feature amount. In this way, it is possible to determine whether or not there is a recognition target by comparing different types of feature quantities with each other under the same scale.

  Next, the recognition image acquisition process executed in step S181 in FIG. 34 will be described with reference to the flowchart in FIG. This recognition image acquisition process has the same configuration as that of the image processing unit 11 described with reference to FIG. 4, and is obtained by capturing a recognition image with a focus camera inside and capturing it with the focus camera. This is executed by the recognition image acquisition unit 123 that can extract a focused part from the recognized image data. Therefore, in the flowchart of FIG. 35, the configuration of the image processing unit 11 described with reference to FIGS. 4 and 5 is described as having the recognition image acquisition unit 123.

  In step S <b> 211, the image acquisition unit 21 of the recognition image acquisition unit 123 executes an imaging process at a predetermined focal length, and supplies the obtained image to the background separation processing unit 22.

  In step S212, the background separation processing unit 22 of the recognized image acquisition unit 123 executes the background separation process described with reference to the flowchart of FIG.

  In step S213, the background separation processing unit 22 of the recognized image acquisition unit 123 determines whether there is an object to be recognized in the image from which the background is separated. In step S213, when it is determined that there is no object to be recognized, for example, when the entire image data is out of focus, the process returns to step S211 and the subsequent processes are repeated.

  If it is determined in step S213 that there is an object to be recognized, in step S214, the background separation processing unit 22 of the recognized image acquisition unit 123 performs image data on which the background has been separated, as necessary. Image processing such as alignment is performed.

  In step S215, the background separation processing unit 22 of the recognition image acquisition unit 123 outputs an image corresponding to the object to be recognized from which the background is separated to the recognition feature point generation unit 124 and the recognition feature amount generation unit 125, The process returns to step S181 in FIG. 34 and proceeds to step S182.

  Through such processing, recognition image data is acquired. Since the image data is picked up by the focus camera, it is possible to separate the object to be recognized existing at the position where the focus is matched from the background by an easy process.

  Next, the recognition correlation feature value generation process executed in step S184 of FIG. 34 will be described with reference to the flowchart of FIG.

  In step S241, the recognized correlation feature value generation unit 126 sets a variable k indicating the number of processed feature values to k = 1.

  In step S242, the recognition correlation feature value generation unit 126 calculates the selected feature value of the feature value k, which is the k-th feature value among the N selected feature values, from the recognized feature value at the corresponding feature point of the recognized image. Generate correlation values.

  In step S243, the recognition correlation feature value generation unit 126 selects the recognition feature value having the highest correlation from the generated correlation values.

  In step S244, the recognition correlation feature value generation unit 126 sets the correlation value of the recognition feature value selected in step S243 as the recognition correlation feature value of the feature value k.

  In step S245, the recognized correlation feature value generation unit 126 determines whether or not the variable k is the total number N of feature values for one image data.

  If it is determined in step S245 that the variable k is not N, that is, it has not reached N, in step S246, the recognized correlation feature value generation unit 126 increments the variable k by 1, and the process proceeds to step S242. Return to, and the subsequent processing is repeated.

  If it is determined in step S245 that the variable k is N, the process returns to step S184 in FIG. 34 and proceeds to step S185.

  By such processing, for example, the recognition correlation feature amount is generated by using the method described with reference to FIG. In addition, as described with reference to FIG. 26, the recognized correlation feature quantity and the different types of feature quantities (for example, shape, color, motion, etc.) can be obtained by calculating the respective correlations. .

  Next, with reference to the flowchart of FIG. 37, the recognition process 2 which is a different process example of the recognition process which the recognition apparatus 72 performs is demonstrated.

  The recognition process 1 described using the flowchart of FIG. 34 outputs all the recognition results of the recognition process executed on each of the captured recognition images. This is because, when acquiring image data used in learning processing and recognition processing, an object to be recognized by the user is placed at the focal length set in the focus camera, and then an imaging instruction is given to the device. It may be a case where imaging processing is performed, or a case where an object recognized so as to fall within the imaging range of the focus camera is appropriately moved, and during that time, the imaging processing is continuously executed. Even if it exists, it is an executable process.

  On the other hand, in the process described using the flowchart of FIG. 37, when the recognition image is obtained in the recognition process, the recognized object is appropriately moved so as to fall within the imaging range of the focus camera. In the case where the imaging process is continuously executed, not all the recognition results of the recognition process executed for each of the continuously captured recognition images are output, but continuously captured. Processing for holding the recognition result of the recognition processing executed for each recognition image for a predetermined number of times or for a predetermined time, observing the transition, and outputting the recognition result at the peak value that seems to be recognized It is.

  In steps S271 to S275, processing similar to that in steps S181 to S185 in FIG. 34 is executed.

  That is, a recognition image is acquired, a feature point of the recognition image is generated, and a feature amount at the feature point is generated. And the recognition correlation feature-value production | generation process demonstrated using FIG. 36 is performed, and the calculation process which substitutes the acquired recognition correlation feature-value to a recognizer is performed.

  In step S276, the recognition processing unit 127 temporarily holds the obtained calculation result. The held calculation result is held for a predetermined number of times or a predetermined time in the calculation process.

  In step S277, the recognition processing unit 127 retains the calculation result of the recognition process executed for each of the continuously captured recognition images for a predetermined number of times or for a predetermined time, and observes the transition. Thus, it is determined whether or not a peak value has been obtained, in other words, whether or not a result most likely to be recognized has been obtained.

  If it is determined in step S277 that no peak value has been obtained, the process returns to step S271, and the subsequent processes are repeated.

  If it is determined in step S277 that a peak value has been obtained, in step S278, the recognition processing unit 127 determines whether or not an object to be recognized is included in the recognized image based on the peak value. The result is supplied to the recognition result output unit 128.

  In step S278, the recognition result output unit 128 displays the recognition result supplied from the recognition processing unit 127 on the display unit, outputs it as audio data, or notifies the user using an LED or the like. Or, the data is output to another device through a predetermined transmission path or recorded on a predetermined recording medium, and the processing is terminated.

  By using a recognizer that has been learned by converting learning feature quantities to learning correlation feature quantities through such processing, the recognition feature quantities are converted to recognition correlation feature quantities and recognition processing is performed. Can be compared with each other under the same scale to determine whether a recognition target exists.

  Furthermore, such processing eliminates the need for the user to perform complicated operations such as installing the recognized objects one by one at the exact focal length set in the focus camera. For example, when it is desired to recognize a pet, even if the pet is moving in front of the camera, the recognition process can be executed correctly. In addition, when the user wants to recognize a movable object, for example, by gripping the object, it is generally moved appropriately around the focal point of the focus camera without setting the recognized object exactly. Then, when the position matches the focal length of the focus camera, correct extraction processing is performed, and a useful recognition image can be obtained and recognition processing can be performed. Further, for example, when it is desired to recognize a person passing through the passage, a correct extraction process is automatically performed when the person passing through the passage passes a predetermined position corresponding to the focal position of the focus camera. It is possible to perform a correct recognition process without having to stand at a predetermined position and instruct the start of imaging.

  The recognition processing approach described above is called Collage-of-Feat, and any feature can be used. Then, recognition is performed by a boosting algorithm using local MAX pooling.

  By the above-described processing, learning of an object model in online execution learning, that is, registration of an object model obtained as a result of learning can be performed using image data with a separated background, so that learning performance can be improved. Furthermore, since the background can be deleted without performing complicated processing in the recognition image acquisition even during recognition execution, it can be expected that false recognition is reduced, and the performance is improved. Furthermore, both the learning process and the recognition process need only be performed on the cut out region, leading to a reduction in calculation time.

  Note that image data acquisition and recognition object region extraction in these processes can be performed only with a monocular camera without using a compound eye, so that an increase in cost can be suppressed.

  Even if the recognition process and the learning process are different from the processes described above, the image data captured using the focus camera is processed using the image processing unit 11 described with reference to FIG. Thus, it is possible to obtain an effect that a part necessary for recognition can be extracted without performing a complicated operation. That is, for example, in the learning process or the recognition process described as the conventional technique, even if the image processing unit 11 described with reference to FIG. 4 is used, a part necessary for recognition can be extracted without performing a complicated operation. The effect that it is possible can be obtained.

  Here, the image processing system 51 has been described as configured by the learning device 71 and the recognition device 72, but the learning processing is performed by one device having both functions of the learning device 71 and the recognition device 72. Needless to say, both the recognition processing and the recognition processing may be performed. In other words, the learning processing system 51 may be configured by a single device. Furthermore, the learning processing system 51 may be configured by two or more devices having both functions of the learning device 71 and the recognition device 72.

  Needless to say, the learning process and the recognition process may not be performed continuously, and the learning device 71 and the recognition device 72 may be installed apart from each other. In other words, the recognition device 72 in which the selected feature value and the recognizer generated by the learning device 71 are stored in the selected feature value storage unit 121 and the recognizer storage unit 122, respectively, is installed at a location that is different from the learning device 71. However, the recognition process can be performed independently.

  Further, in any process of obtaining the model image, obtaining the learning image, and obtaining the recognition image, a region corresponding to the object to be recognized from the image data may be automatically extracted. At least one of these image acquisition processes may be acquired using another method. For example, when the number of model images is small, the model image acquisition unit 91 is not configured similarly to the image processing unit 11 described with reference to FIG. Only a portion to be recognized may be extracted from predetermined image data by acquiring a model image or by a user operation, and may be output to the model feature point generation unit 92.

  The series of processes described above can be executed by hardware or can be executed by software. The software is a computer in which the program constituting the software is incorporated in dedicated hardware, or various functions can be executed by installing various programs, for example, a general-purpose personal computer For example, it is installed from a recording medium. In this case, the processing described above is executed by a personal computer 500 as shown in FIG.

  38, a CPU (Central Processing Unit) 501 performs various processes according to a program stored in a ROM (Read Only Memory) 502 or a program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. Execute. The RAM 503 also appropriately stores data necessary for the CPU 501 to execute various processes.

  The CPU 501, ROM 502, and RAM 503 are connected to each other via an internal bus 504. An input / output interface 505 is also connected to the internal bus 504.

  The input / output interface 505 includes an input unit 506 including a keyboard and a mouse, a display including CRT and LCD, an output unit 507 including a speaker, a storage unit 508 including a hard disk, a modem, a terminal adapter, and the like. A communicator 509 is connected. A communication unit 509 performs communication processing via various networks including a telephone line and CATV.

  A drive 510 is also connected to the input / output interface 505 as necessary, and a removable medium 521 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted, and a computer program read therefrom is It is installed in the storage unit 508 as necessary.

  When a series of processing is executed by software, a program constituting the software is installed from a network or a recording medium.

  As shown in FIG. 38, this recording medium is not only composed of a package medium consisting of a removable medium 521 on which a program is recorded, which is distributed to provide a program to the user, separately from the computer. These are configured by a hard disk including a ROM 502 storing a program and a storage unit 508 provided to the user in a state of being pre-installed in the apparatus main body.

  Further, in the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but may be performed in parallel or It also includes processes that are executed individually.

  In the present specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure for demonstrating the conventional recognition technique. It is a figure for demonstrating the relationship between a learning process and a recognition process. It is a figure for demonstrating the characteristic of the image imaged with the focus camera. It is a figure which shows the structure of the image process part which can isolate | separate a background from the image imaged with the focus camera. It is a figure which shows the further detailed structure of the background separation process part of FIG. It is a figure for demonstrating a neighborhood pixel difference filter. It is a figure which shows the example of an output of a near pixel difference filter calculation process part. It is a figure for demonstrating a neighborhood area | region sum filter. It is a figure for demonstrating the process of a neighborhood area | region sum filter calculation process part. It is a figure which shows the example of an output of a neighborhood area | region sum filter calculation process part. It is a figure which shows the example of an output of a neighborhood area | region sum filter calculation process part. It is a figure which shows the example of an output of a neighborhood area | region sum filter calculation process part. It is a figure which shows the example of an output of a threshold value process part. It is a figure which shows the example of an output of a threshold value process part. It is a figure for demonstrating the flow of a process by a background separation process part. It is a figure for demonstrating the flow of a process by a background separation process part. It is a figure for demonstrating the flow of a process by a background separation process part. It is a figure for demonstrating the flow of a process by a background separation process part. It is a block diagram for demonstrating the structure of an image processing system. It is a figure for demonstrating the outline | summary of the learning phase performed in a learning apparatus. It is a figure which shows the shape of the kernel of SteerableFilter on two dimensions. It is a figure for demonstrating the process which couple | bonds the jet of a vicinity. It is a figure for demonstrating rotation of a local feature-value. It is a figure which shows the example of the histogram in HSV space. It is a figure for demonstrating the example of a search of the correlation maximum point by EBGM method. It is a figure for demonstrating the example of the correlation feature-value calculation by multiple types of feature-value. It is a figure for demonstrating the example of the learning process performed in a recognizer production | generation part. It is a figure for demonstrating the example of the learning process performed in a recognizer production | generation part. It is a flowchart for demonstrating a learning process. It is a flowchart for demonstrating a learning image acquisition process. It is a flowchart for demonstrating a background separation process. It is a flowchart for demonstrating a learning correlation feature-value production | generation process. It is a flowchart for demonstrating a recognizer production | generation process. 10 is a flowchart for explaining recognition processing 1; It is a flowchart for demonstrating recognition image acquisition processing. It is a flowchart for demonstrating a recognition correlation author amount production | generation process. It is a flowchart for demonstrating the recognition process 2. FIG. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Image processing part, 21 Image acquisition part, 22 Background separation processing part, 31 Neighboring pixel difference filter calculation processing part, 32 Neighboring area sum filter calculation processing part, 33 Threshold processing part, 51 Image processing system, 71 Learning apparatus, 72 Recognition Apparatus, 91 model image acquisition unit, 92 model feature point generation unit, 93 model feature amount generation unit, 94 model feature amount storage unit, 95 learning image acquisition unit, 96 learning feature point generation unit, 97 learning feature amount generation unit, 98 Learning correlation feature generation unit, 99 correct / incorrect information acquisition unit, 100 recognizer generation unit, 121 selected feature amount storage unit, 122 recognizer storage unit, 123 recognition image acquisition unit, 124 recognition feature point generation unit, 125 recognition feature amount generation , 126 recognition correlation feature value generation unit, 127 recognition processing unit, 128 recognition result output unit, 41 to 143 correlation calculation unit

Claims (23)

In an image processing apparatus that generates a recognizer for recognizing a recognition target in advance by learning processing,
Learning image acquisition means for acquiring a learning image used in the learning process;
Model image acquisition means for acquiring a model image corresponding to the recognition target;
Recognition that executes the learning process using the learning image acquired by the learning image acquisition unit and the model image acquired by the model image acquisition unit, and generates a recognizer for recognizing the recognition target Generator generating means, and
At least one of the learning image acquisition means or the model image acquisition means is:
Image acquisition means for acquiring image data in which the focal point matches an image of a subject existing at a predetermined focal length and the focal point does not match other objects;
Image extracting means for extracting from the image data acquired by the image acquiring means a portion corresponding to the subject in focus;
An image processing apparatus that acquires a portion corresponding to the subject extracted by the image extraction means as the learning image or the model image. The image extracting means includes
First calculation means for executing calculation processing for extracting a pixel having a large difference from neighboring pixels in each pixel of the image data acquired by the image acquisition means;
Second computing means for obtaining an average of the target pixel and its neighboring area, with a pixel having a large difference from the neighboring pixel extracted by the first computing means as a target pixel;
2. The image according to claim 1, further comprising: an area corresponding to an object to be detected based on a calculation result of the second calculation means; and a dividing means for dividing the image data into areas considered to be the background. Processing equipment. The division unit divides the calculation result of the second calculation unit into a region corresponding to an object to be detected and a region considered to be the background by binarizing with a predetermined threshold value. The image processing apparatus described. The image processing apparatus according to claim 2, wherein the dividing unit recognizes a pixel whose calculation result of the second calculation unit is a positive value as a region corresponding to an object to be detected. The recognizer generating means includes
Model feature point generation means for generating a plurality of feature points as model feature points from the model image acquired by the model image acquisition means;
Model feature value generating means for generating a feature value at each of the model feature points generated by the model feature point generating means as a model feature value;
Learning feature point generation means for generating a plurality of feature points as learning feature points from the learning image acquired by the learning image acquisition means;
Learning feature value generation means for generating a feature value at each of the learning feature points generated by the learning feature point generation means as a learning feature value;
For each of the model feature quantities generated by the model feature quantity generation means, the learning feature quantity generated by the learning feature quantity generation means is selected, and the selected learning feature is selected. A learning correlation feature value generating means for generating a degree of correlation with a quantity as a learning correlation feature value;
Correct / incorrect information acquisition means for acquiring correct / incorrect information indicating whether or not the learning image includes the recognition target;
The recognizing device generating unit that generates a recognizing device based on the learning correlation feature amount generated by the learning correlation feature amount generating unit and the correct / incorrect information acquired by the correct / incorrect information acquiring unit. The image processing apparatus described. The model feature point generated by the model feature point generation means is selected according to the type of the model feature amount in the model feature point,
The image processing apparatus according to claim 5, wherein the learning feature point generated by the learning feature point generation unit is selected according to a type of the learning feature amount at the learning feature point. The model feature value generated by the model feature value generation means is selected according to the type of the model feature value,
The image processing apparatus according to claim 5, wherein the learning feature amount generated by the learning feature amount generation unit is selected according to a type of the learning feature amount. The image processing apparatus according to claim 5, wherein the recognizer generation unit generates the recognizer through a learning process based on a weighted vote. The image processing apparatus according to claim 8, wherein the learning process based on the weighted voting is a boosting algorithm. 2. The image extracting unit extracts a portion corresponding to the subject that is in focus by extracting a region that is out of focus from the image data acquired by the image acquisition unit. The image processing apparatus described. The image extraction means analyzes the frequency spectrum of each image area that constitutes the image data acquired by the image acquisition means using FFT, and the focus is matched in an area that sufficiently contains high-frequency components. The image processing apparatus according to claim 1, wherein a portion corresponding to the subject in focus is extracted by determining that the subject is in focus. Recognizer storage means for storing the recognizer generated by the recognizer generation means;
Selection feature quantity storage means for storing a selection feature quantity corresponding to each of the recognizers stored by the recognizer storage means;
Recognition image acquisition means for acquiring a recognition image used for performing recognition processing;
Recognition feature point generation means for generating a plurality of feature points as recognition feature points from the recognition image acquired by the recognition image acquisition means;
Recognition feature value generation means for generating a feature value at each of the recognition feature points generated by the recognition feature point generation means as a recognition feature value;
For each of the selected feature amounts stored by the selected feature amount storage unit, the recognition feature amount having the highest correlation among the recognition feature amounts generated by the recognition feature amount generation unit is selected and selected. A recognition correlation feature value generating means for generating a degree of correlation with the recognition correlation feature value;
By substituting the recognition correlation feature quantity generated by the recognition correlation feature quantity generation unit into the recognizer generated by the recognition unit generation unit, the recognition image acquired by the recognition image acquisition unit is added to the recognition image. The image processing apparatus according to claim 1, further comprising: a recognition processing unit that determines whether or not a recognition target is included. The image processing according to claim 12, wherein the recognition image acquisition unit includes the image acquisition unit and the image extraction unit, and acquires a portion corresponding to the subject extracted by the image extraction unit as the recognition image. apparatus. In an image processing method of an image processing apparatus for generating a recognizer for recognizing a recognition target in advance by learning processing,
Obtaining a learning image used in the learning process;
Obtaining a model image corresponding to the recognition target;
Performing the learning process using the acquired learning image and the model image, and generating a recognizer for recognizing the recognition target,
At least one of the step of acquiring the learning image or the step of acquiring the model image includes:
Obtain image data that has a focal point that matches the image of the subject that exists at the specified focal length and that does not match the focal point of other objects,
Extracting a portion corresponding to the subject in focus from the acquired image data,
An image processing method for acquiring a portion corresponding to the extracted subject as the learning image or the model image. A program for causing a computer to execute a process of generating a recognizer for recognizing a recognition object in advance by a learning process,
Controlling acquisition of learning images used in the learning process;
Controlling the acquisition of a model image corresponding to the recognition target;
Performing the learning process using the acquired learning image and the model image, and generating a recognizer for recognizing the recognition target,
At least one of the step of acquiring the learning image or the step of acquiring the model image includes:
Controls the acquisition of image data that is in focus with the image of the subject present at a given focal length and not in focus with other objects,
Extracting a portion corresponding to the subject in focus from the acquired image data,
A program that causes a computer to execute a process of acquiring a portion corresponding to the extracted subject as the learning image or the model image. In a recognition apparatus that performs a recognition process that determines whether or not a recognition target is included in a recognition image using a recognizer generated by a learning process,
Recognition image acquisition means for acquiring the recognition image used for performing recognition processing;
Recognizer storage means for storing the recognizer;
Selection feature quantity storage means for storing a selection feature quantity corresponding to each of the recognizers stored by the recognizer storage means;
The recognition target is included in the recognition image acquired by the recognition image acquisition unit using the recognition unit stored in the recognition unit storage unit and the selected feature amount stored in the selection feature amount storage unit. A recognition processing means for judging whether or not it is included,
The recognized image acquisition means includes
Image acquisition means for acquiring image data in which the focal point matches an image of a subject existing at a predetermined focal length and the focal point does not match other objects;
Image extracting means for extracting from the image data acquired by the image acquiring means a portion corresponding to the subject in focus;
A recognition apparatus that acquires, as the recognition image, a portion corresponding to the subject extracted by the image extraction means. The image extracting means includes
First calculation means for executing calculation processing for extracting a pixel having a large difference from neighboring pixels in each pixel of the image data acquired by the image acquisition means;
Second computing means for obtaining an average of the target pixel and its neighboring area, with a pixel having a large difference from the neighboring pixel extracted by the first computing means as a target pixel;
17. The image according to claim 16, further comprising: an area corresponding to an object to be detected based on a calculation result of the second calculation means; and a dividing means for dividing the image data into an area considered to be a background. Processing equipment. The division unit divides the calculation result of the second calculation unit into a region corresponding to an object to be detected and a region considered to be the background by binarizing with a predetermined threshold value. The image processing apparatus described. The image processing apparatus according to claim 17, wherein the dividing unit recognizes a pixel whose calculation result of the second calculation unit is a positive value as a region corresponding to an object to be detected. The recognition processing means includes
Recognition feature point generation means for generating a plurality of feature points as recognition feature points from the recognition image acquired by the recognition image acquisition means;
Recognition feature value generation means for generating a feature value at each of the recognition feature points generated by the recognition feature point generation means as a recognition feature value;
For each of the selected feature amounts stored in the selected feature amount storage unit, the recognition feature amount having the highest correlation among the recognition feature amounts generated by the recognition feature amount generation unit is selected and selected. A recognition correlation feature value generating means for generating a degree of correlation with the recognition correlation feature value;
By substituting the recognition correlation feature quantity generated by the recognition correlation feature quantity generation means into the recognizer stored by the recognizer storage means, the recognition image acquired by the recognition image acquisition means is added to the recognition image. The recognition apparatus according to claim 16, further comprising: a determination unit that determines whether or not a recognition target is included. The recognizer stored by the recognizer storage means is:
Generate multiple feature points as model feature points from a given model image,
Generating a feature quantity at each of the model feature points as a model feature quantity,
A plurality of feature points are generated as learning feature points from a predetermined learning image,
Generating a feature quantity at each of the learning feature points as a learning feature quantity;
For each of the model feature amounts, the one having the highest correlation among the learning feature amounts is selected, and a degree of correlation with the selected learning feature amount is generated as a learning correlation feature amount,
Correct / incorrect information indicating whether or not the learning image includes the recognition target,
The recognition device according to claim 16, wherein the recognition device is a recognizer generated based on the learning correlation feature quantity and the correct / incorrect information. Whether the recognition target is included in the recognition image using the recognition unit generated by the learning process and stored in the storage unit, and the selected feature amount corresponding to each of the recognition units stored in the storage unit In a recognition method of a recognition device that performs recognition processing to determine whether or not,
Obtaining the recognition image used for performing the recognition process;
Determining whether the recognition target is included in the acquired recognition image using the recognizer and the selected feature amount;
In the process of acquiring the recognition image,
Obtain image data that has a focal point that matches the image of the subject that exists at the specified focal length and that does not match the focal point of other objects,
Extracting a portion corresponding to the subject in focus from the acquired image data,
A recognition method for acquiring a portion corresponding to the extracted subject as the recognition image. Whether the recognition target is included in the recognition image using the recognition unit generated by the learning process and stored in the storage unit, and the selected feature amount corresponding to each of the recognition units stored in the storage unit A program for causing a computer to execute a process for determining whether or not,
Controlling the acquisition of the recognition image used to perform recognition processing;
Determining whether the recognition target is included in the acquired recognition image using the recognizer and the selected feature amount;
In the process of acquiring the recognition image,
Controls the acquisition of image data that is in focus with the image of the subject present at a given focal length and not in focus with other objects,
Extracting a portion corresponding to the subject in focus from the acquired image data,
A program for causing a computer to execute a process of acquiring a portion corresponding to the extracted subject as the recognized image.

Images