JP2016103094A - Image processing method, image processor, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor, image processing method and image processing program which recognize an area in which an object corresponding to a previously set detection object to an input image, and can make determination even about a state of the recognized object corresponding to the detection object.SOLUTION: The image processing method includes a step for determining whether a detection object is included in an input image on the basis of a first probability about end nodes which respective partial input images respectively reach when a plurality of partial input images obtained from the input image are given to a determination tree group, and determining whether the detection object included in the input image meets a predetermined condition on the basis of a second probability about the respective end nodes.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing method, an image processing apparatus, and an image processing program directed to object recognition.

  Various automation techniques using image processing techniques have been developed at manufacturing sites and the like. For example, Japanese Patent Application Laid-Open No. 2004-188562 (Patent Document 1) mounts a three-dimensional visual sensor on a robot, recognizes the position and orientation of a workpiece existing in a container with an opening, and takes out the workpiece based on the recognition. Disclosed is a workpiece picking device to perform.

  As such an object recognition technique, various methods have been proposed. For example, Japanese Patent Laid-Open No. 2013-003919 (Patent Document 2) collates captured image data acquired by a camera with a code book, selects a closest small area image pattern from among a plurality of small area image patterns, Disclosed is a method for recognizing an object by extracting a class related to a node having the smallest weight among nodes whose weight is equal to or greater than a threshold value for the region image pattern, and voting position information of the small region image pattern to the class. To do.

  Also, an identification method called a Random Forests method is known. In the Random Forests method, as a learning process, a plurality of subsets are extracted from a data set, and a decision tree (simple Bayes classifier) is constructed for each subset. That is, a decision tree group for each subset is constructed as supervised learning. For example, Japanese Patent Laying-Open No. 2012-042990 (Patent Document 3) discloses an image identification information providing apparatus in which a random forest method using a decision tree group as a classifier is applied to an image annotation technology (image identification information providing technology).

JP 2004-188562 A JP 2013-003919 A JP 2012-042990 A

  In the above-described prior art, mainly the position / orientation of the object to be detected, whether the object to be detected exists, or which of the plurality of candidates matches the object. Determine. However, in an actual manufacturing site or the like, it is necessary to evaluate a situation where a recognized object is placed. However, in the above-described prior art, it is not assumed that such a situation is evaluated.

  Therefore, for an input image acquired by imaging or the like, an image that recognizes a region where an object corresponding to a preset detection target exists and can also determine the status of the object corresponding to the recognized detection target There is a demand for a processing apparatus, an image processing method, and an image processing program.

  An image processing method according to an aspect of the present invention includes a step of constructing a decision tree group having a hierarchical structure from a root node to a plurality of terminal nodes using a plurality of partial learning images obtained from a learning image. The plurality of partial learning images include a first sample that indicates a portion that matches a predetermined condition among detection targets, a second sample that indicates a portion that does not match a predetermined condition among detection targets, And a third sample indicating a detection target. The step of constructing a decision tree group is a step of determining a branch function indicating to which of the child nodes branched from the node the given partial learning image should be classified for each node that is not a terminal node. And sequentially branching each of the partial learning images until reaching one of the terminal nodes according to the determined branch function, and for each terminal node, the sum of the first sample and the second sample and the first sample Determining a first probability indicating a ratio of three samples and a second probability indicating a ratio of the first sample and the second sample. The image processing method further includes an input based on a first probability for a terminal node to which each partial input image arrives when a plurality of partial input images obtained from the input image are given to the decision tree group. Whether or not a detection target is included in the image and whether or not the detection target included in the input image meets a predetermined condition based on the second probability for each terminal node Determining whether or not.

  Preferably, the step of constructing the decision tree group is performed for each node based on a result of classifying a plurality of partial learning images given according to the determined branch function into any one of the child nodes at each node that is not a terminal node. The method further includes determining a first weight indicating the discrimination capability between the detection target and the non-detection target, and a second weight indicating the discrimination capability between the first sample and the second sample.

  Alternatively, preferably, in the step of constructing the decision tree group, the first sample and the second sample are the same when the plurality of partial learning images given according to the determined branch function are classified into any child node, respectively. The step of lowering the weight indicating the discriminating ability between the first sample and the second sample is further included as the ratio of being classified into the child nodes is higher.

  Preferably, in the image processing method, a partial learning image and a partial input image that is misidentified or likely to be misidentified among the partial input images are further given to the decision tree group, and each of the images is selected. According to the step of identifying the terminal node that each image reaches by branching sequentially until reaching the terminal node of the image, and the identification probability between the partial learning image and the partial input image at the terminal node that the partial input image has reached, Adding a child node that branches off from the end node.

  Preferably, the image processing method further includes a step of setting a plurality of partial learning images indicating detection targets extracted from regions adjacent to each other in a single group, and the steps of constructing the decision tree group are the same Determining a weight in common for a plurality of partial learning images belonging to the group.

  Preferably, the image processing method further includes a step of rotating the learning image by a predetermined angle to generate a plurality of learning images and extracting a plurality of partial learning images from the generated learning images.

  An image processing apparatus according to another aspect of the present invention includes means for constructing a decision tree group having a hierarchical structure from a root node to a plurality of end nodes using a plurality of partial learning images obtained from a learning image. The plurality of partial learning images include a first sample that indicates a portion that matches a predetermined condition among detection targets, a second sample that indicates a portion that does not match a predetermined condition among detection targets, And a third sample indicating a detection target. The means for constructing a decision tree group is a means for determining, for each node that is not a terminal node, a branch function indicating to which of the child nodes branched from the node the given partial learning image should be classified. And sequentially branching each of the partial learning images until reaching one of the terminal nodes according to the determined branch function, and for each terminal node, the sum of the first sample and the second sample and the first sample Means for determining a first probability indicating a ratio of three samples and a second probability indicating a ratio of the first sample and the second sample. The image processing apparatus further inputs an input based on a first probability for each terminal node to which each partial input image reaches when a plurality of partial input images obtained from the input image are given to the decision tree group. Whether or not a detection target is included in the image and whether or not the detection target included in the input image meets a predetermined condition based on the second probability for each terminal node Means for determining whether or not.

  According to still another aspect of the present invention, an image processing program executed by a computer is provided. The image processing program causes a computer to execute a step of constructing a decision tree group having a hierarchical structure from a root node to a plurality of terminal nodes using a plurality of partial learning images obtained from the learning image. The plurality of partial learning images include a first sample that indicates a portion that matches a predetermined condition among detection targets, a second sample that indicates a portion that does not match a predetermined condition among detection targets, And a third sample indicating a detection target. The step of constructing a decision tree group is a step of determining a branch function indicating to which of the child nodes branched from the node the given partial learning image should be classified for each node that is not a terminal node. And sequentially branching each of the partial learning images until reaching one of the terminal nodes according to the determined branch function, and for each terminal node, the sum of the first sample and the second sample and the first sample Determining a first probability indicating a ratio of three samples and a second probability indicating a ratio of the first sample and the second sample. Based on the first probability for the terminal node to which each partial input image arrives when a plurality of partial input images obtained from the input image are given to the decision tree group, And whether or not the detection target included in the input image meets a predetermined condition based on the second probability for each terminal node. The step of determining is executed.

  According to some aspects of the present invention, an input image acquired by imaging or the like recognizes a region where an object corresponding to a preset detection target exists and corresponds to the recognized detection target. It is also possible to determine the situation of the object to be performed.

It is the schematic which shows the structural example of the image processing system containing the image processing apparatus which concerns on this Embodiment. It is a schematic diagram which shows the structural example of the image processing apparatus which concerns on this Embodiment. It is a figure explaining the background of the image recognition method which concerns on this Embodiment. It is a figure explaining the learning image used in the object recognition method which concerns on this Embodiment. It is a figure explaining the effect of the weight in the image recognition method which concerns on this Embodiment. It is a figure explaining the effect by the update of the weight in the image recognition method which concerns on this Embodiment. It is a figure explaining the effect by the update of the weight in the image recognition method which concerns on this Embodiment. It is a figure explaining the production | generation of Bag of the learning image used in the object recognition method which concerns on this Embodiment. It is a figure for demonstrating the construction process of the decision tree group in the learning process in the object recognition method which concerns on this Embodiment. It is a flowchart which shows the process sequence of the learning process of the object recognition method which concerns on this Embodiment. It is a schematic diagram for demonstrating the process which determines the branch function in the learning process in the object recognition method which concerns on this Embodiment. It is a schematic diagram for demonstrating the information of the terminal node obtained by the learning process in the object recognition method which concerns on this Embodiment. It is a flowchart which shows the recognition process sequence in the object recognition method which concerns on this Embodiment. It is a figure explaining the recognition process in the object recognition method which concerns on this Embodiment. It is a figure which shows an example of the recognition result by the object recognition method which concerns on this Embodiment. It is a figure explaining the additional learning process in the object recognition method which concerns on this Embodiment. It is a flowchart which shows the additional learning process procedure in the object recognition method which concerns on this Embodiment.

  Embodiments of the present invention will be described in detail with reference to the drawings. In addition, about the same or equivalent part in a figure, the same code | symbol is attached | subjected and the description is not repeated.

<A. Configuration example of image processing system>
FIG. 1 is a schematic diagram illustrating a configuration example of an image processing system 1 including an image processing apparatus 100 according to the present embodiment. FIG. 1 shows an image processing system 1 directed to a bin picking system as an example. The bin picking system recognizes the position / posture of an object (hereinafter, also referred to as “work”) designated for product selection and parts assembly from the input image, and according to the recognized position / posture information, The recognized workpiece is gripped (picked).

  More specifically, the image processing system 1 includes an imaging device 2, an image processing device 100, and a picking robot 200. The image processing apparatus 100 recognizes the position / posture of the workpiece 4 corresponding to the detection target registered in advance from the input image from the imaging device 2, and sends the detected position / posture information of the workpiece 4 to the picking robot 200. Output. The picking robot 200 grips the target workpiece 4 and moves it to a predetermined position in accordance with information from the image processing apparatus 100.

  The application destination of the image processing method, the image processing apparatus, and the image processing program according to the present invention is not limited to the bin picking system shown in FIG. 1, but can be applied to various systems using image recognition technology.

<B. Configuration example of image processing apparatus>
Next, a configuration example of the image processing apparatus 100 illustrated in FIG. 1 will be described. FIG. 2 is a schematic diagram illustrating a configuration example of the image processing apparatus 100 according to the present embodiment. FIG. 2 illustrates a form in which a processor executes an image processing program as a typical implementation example of the image processing apparatus 100.

  More specifically, the image processing apparatus 100 includes a processor 102, a main memory 104, an HDD (Hard Disk Drive) 106, a network interface 110, an image input interface 112, an input unit 114, and a display unit 116. And an output interface 118. These components are communicably connected to each other via the internal bus 120.

  The processor 102 is a processing entity that executes processing to be described later, and develops and executes the image processing program 108 stored in the HDD 106 in the main memory 104. The processor 102 typically includes a CPU (Central Processing Unit) and an MPU (Micro-Processing Unit). The HDD 106 may store a decision tree obtained as a result of learning processing described later, a result of recognition processing, and the like.

  The network interface 110 mediates communication with other devices and servers via an external network. The image input interface 112 includes a circuit that complies with an arbitrary communication protocol, and receives a learning image and / or an input image from the imaging device 2. The input unit 114 includes a keyboard, a mouse, and the like, and accepts an input operation from the user. The display unit 116 includes a display or the like, and notifies the user of processing processes and results such as learning processing and recognition processing. The output interface 118 includes a circuit that complies with an arbitrary communication protocol, and outputs the result obtained by the recognition process to the outside (for example, the picking robot 200).

  The imaging device 2 is a means for generating an input image by imaging a subject. As an example, in addition to an optical system such as a lens, a device such as a CCD (Coupled Charged Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor is used. Including.

  All or part of the functions of the image processing apparatus 100 may be realized using circuit elements such as SoC (System on a chip), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). Good. The image processing program 108 illustrated in FIG. 2 may be installed in the image processing apparatus 100 through an arbitrary recording medium (for example, an optical disk or a flash memory), or may be distributed via a network. Furthermore, the image processing apparatus 100 and the server apparatus may be linked to realize processes and functions described later. In this case, one or both of the image processing apparatus 100 and the server apparatus has a function necessary for realizing this embodiment.

  The imaging device 2 and the image processing device 100 may be configured integrally, or the image processing device 100 that is not directly connected to the imaging device 2 may be employed. In the latter case, an image may be generated or acquired using an arbitrary imaging unit, and the generated or acquired image may be captured into the image processing apparatus 100 via a network or an arbitrary recording medium.

<C. Overview>
Next, an outline of the object recognition method according to the present embodiment will be described.

  In the bin picking system as shown in FIG. 1, it is necessary to determine whether or not the recognized workpiece can be gripped in addition to the recognition of the position and orientation of the workpiece corresponding to the detection target. FIG. 3 is a diagram for explaining the background of the image recognition method according to the present embodiment.

  As shown in FIG. 3 (a), assuming a bin picking system in which a beverage container is an example of a workpiece 4, if the recognized workpiece 4 is separated from another workpiece 4 (that is, with the other workpiece 4). If there is no interference), it can be easily gripped, but if the recognized workpiece 4 is adjacent to another workpiece 4 (that is, if it interferes with another workpiece 4), it is difficult to grip. .

  In the object recognition method according to the related art, as shown in FIG. 3B, in addition to the process of recognizing the position and orientation of the work (work recognition), the process of determining whether or not the recognized work can be gripped. It was necessary to perform (grip determination). That is, in the object recognition method according to the related art, it is necessary to execute the workpiece gripping operation 16 after executing the workpiece recognition 12 and the gripping determination 14 on the input image 10.

  In the case of the processing procedure as shown in FIG. 3B, it is necessary to set a rule in advance in the gripping determination 14 using the input image and its distance image. More specifically, it is necessary to define a rule between a three-dimensional map indicating the situation around the recognized workpiece and a hand map indicating the gripping operation of the picking robot 200. That is, when executing the processing procedure as shown in FIG. 3B, it is necessary to set rules in advance in order to appropriately handle the grip determination 14, and it takes human costs to set these rules. There is a problem.

  On the other hand, in the object recognition method according to the present embodiment, as shown in FIG. 3C, a recognition process 13 that integrally includes workpiece recognition and gripping determination is executed. In the recognition process 13, the recognition process for the workpiece 4 is executed including the contents of the grip determination. Therefore, it is not necessary to set a rule necessary for executing the grip determination 14 in advance.

  That is, in the learning process for constructing a discriminator for recognizing the work 4, an image of a case that should be recognized as the work 4 and an image of a case that should not be recognized as the work 4 are given and recognized as the work 4 The image of the case that indicates the situation where it is easy to grip and the image of the case that indicates the situation where gripping is difficult are included in the image of the case that should be held. By performing learning processing using these images, it is possible to determine whether the workpiece is easily gripped or difficult to grip simultaneously with the recognition of the position / posture of the workpiece. That is, in the object recognition method according to the present embodiment, it is not necessary to set in advance the rules necessary for the grip determination 14 shown in FIG.

  In order to facilitate understanding, a specific example of a bin picking system has been described as an example. However, the present invention is not limited to this, and the object recognition method according to the present embodiment recognizes the recognition of the position / posture of a workpiece and the recognition. The present invention is applicable to various applications in which it is necessary to simultaneously determine whether or not a workpiece meets a predetermined condition (a situation where a recognized workpiece is placed).

  For example, assuming an application that sequentially holds a plurality of stacked workpieces according to a certain rule, it recognizes the position / posture of the workpiece and whether or not the recognized workpiece should be held preferentially. The present invention can also be applied to a picking system that makes a determination. Furthermore, the present invention can be applied not only to gripping a workpiece but also to an application for attracting a workpiece and an application for performing some processing on the workpiece.

  Furthermore, not only the determination of the two categories of whether or not the workpiece meets a predetermined condition, but it may also be determined which of the more categories it belongs to. For example, in the above-mentioned application for gripping a workpiece, it is “easy to grip”, “can be gripped (not easy)”, or “difficult to grip”. It can also be determined.

  As described above, the object recognition method according to the present embodiment determines whether or not a detection target is included in the input image, and the detection target included in the input image conforms to a predetermined condition. It is determined whether or not (a situation where the recognized workpiece is placed).

  The object recognition method according to the present embodiment recognizes a workpiece from an input image using a statistical learning method. In particular, the object recognition method according to the present embodiment is directed to a local patch-based statistical learning method. The local patch base is a technique of using a partial input image cut out from an input image (hereinafter also referred to as “input patch”) for object recognition. Since the voting process is performed using a plurality of input patches cut out from the input image, robust object recognition can be realized even if a part of the work is hidden or deformed.

  As an example of such a local patch-based statistical learning method, a Hough Forests method using Random Forests (decision tree group) for object recognition of an input patch has been proposed. In the Hough Forests method, a plurality of input patches cut out from an input image are input to Random Forests, and the class probability of the terminal node reached by each input patch is calculated using an offset amount (offset vector) to the center of the detection target. An object recognition method for voting. In the Hough Forests method, a decision tree group is constructed by learning processing from a subset of a partial image cut out from a sample image (hereinafter also referred to as “learning patch”), and the input patch cut out from the input image is determined as a decision tree group. To enter. It is determined which leaf node (hereinafter also referred to as “terminal node”) in each decision tree has reached each input patch, and the information held in each terminal node (that is, obtained in advance by learning processing). The position / orientation of the workpiece corresponding to the detection target is recognized in the input image.

  The object recognition method according to the present embodiment is based on the Hough Forests method. However, the object recognition method according to the present embodiment includes a novel process as described in detail below, and thus has a remarkable effect that cannot be obtained by a general Hough Forests method. There is an effect.

  The object recognition method according to the present embodiment uses a learning process according to supervised learning and a learning result obtained by the learning process to recognize the position / posture of the workpiece corresponding to the detection target from the input image, and Recognition processing for determining whether or not the recognized workpiece meets a predetermined condition. Furthermore, in the object recognition method according to the present embodiment, if the learning result obtained by executing the learning process cannot obtain an appropriate recognition result, the additional learning process corrects the learning result. Can be included. Hereinafter, the object recognition method according to the present embodiment will be described in detail in the order of “learning process”, “recognition process”, and “additional learning process”.

<D. Learning process>
(D1: Learning image used for learning processing)
In the learning process of the object recognition method according to the present embodiment, Random Forests (hereinafter also referred to as “decision tree group”) for identifying input patches are constructed using a plurality of partial learning images. As will be described later, the decision tree group has a hierarchical structure from a root node to a plurality of end nodes. Thus, the learning process includes a step of constructing a decision tree group having a hierarchical structure from a root node to a plurality of terminal nodes using a plurality of partial learning images obtained from the learning image.

  In the recognition process, the input patch cut out from the input image is input to the decision tree group, and the class probability of the input patch that has reached each terminal node is used to recognize the position / posture of the workpiece corresponding to the detection target. Then, it is determined whether or not the recognized workpiece meets a predetermined condition.

  In the learning process of the object recognition method according to the present embodiment, in order to recognize the position / posture of the work, a learning image (at least a part of the work of the detection target (that is, the target whose position / posture should be recognized) is shown ( Hereinafter, it is also referred to as a “positive image”) and a learning image (hereinafter also referred to as a “negative image”) in which at least a part of the work of a non-detection target (that is, a target whose position / posture should not be recognized) is shown. .). The partial learning image cut out from the positive image is also referred to as “positive patch”, and the partial learning image cut out from the negative image is also referred to as “negative patch”. Hereinafter, “positive patch” and “negative patch” may be simply referred to as “learning patch”. Since the object recognition method according to the present embodiment is a local patch-based statistical learning method, learning processing and recognition processing are executed using these learning patches.

  As will be described later, in the object recognition method according to the present embodiment, a plurality of patches may be handled as an aggregate (Bag). Therefore, for convenience of explanation, one or a plurality of patches acquired from a learning image is represented by “ It may be collectively referred to as “learning sample”. Similarly, one or more positive patches may be collectively referred to as “positive samples” and negative patches may be collectively referred to as “negative samples”. The input patch may also be collectively referred to as “input sample”.

  An arbitrary type of image can be used as the learning image, but in object recognition, it is preferable to use a distance image, that is, an image having a pixel value as a distance from a certain imaging point to each point on the workpiece surface. Of course, a normal image, that is, an image with the brightness of each point on the workpiece surface as the pixel value can also be used.

  FIG. 4 is a diagram for explaining a learning image used in the object recognition method according to the present embodiment. FIG. 4 shows (a) a positive image and (b) a negative image. The positive image 20 shown in FIG. 4 assumes a beverage container as a detection target. In order to identify a workpiece to be detected, a decision tree group for distinguishing between a detection target and a non-detection target is constructed using a negative image 30 obtained by copying another workpiece that does not have similar appearance characteristics. To do.

  In the object recognition method according to the present embodiment, in order to determine whether or not the detection target included in the input image meets a predetermined condition, the positive image 20 is determined according to the condition to be determined. Classify into two or more categories (hereinafter also referred to as “subclasses”). In other words, learning images of respective subclasses corresponding to the number of conditions to be determined are prepared. In the example shown in FIG. 4, in order to determine whether one condition, that is, the recognized workpiece as described above can be gripped, an input image indicating easy gripping and an input image indicating gripping difficulty are It is included in the positive image 20.

  A positive image 20 shown in FIG. 4 is a learning image 22 ((a-1) easy to grip) showing a state (subclass 1) in which the beverage container is disposed so as to be graspable, and the beverage container is disposed difficult to grasp. And a learning image 24 ((a-2) difficulty in gripping) showing a state (subclass 2) being present. A learning image generation method as shown in FIG. 4 will be described later.

  In the learning process, a decision tree group is constructed using a plurality of learning patches cut out from the positive image 20 (including the learning image 22 and the learning image 24) and the negative image 30, respectively. The plurality of learning patches (partial learning images) do not match the learning image 22 (first sample) indicating a portion that matches a predetermined condition among detection targets and the predetermined condition among detection targets. A learning image 24 (second sample) showing a portion and a negative image 30 (third sample) showing a non-detection target are included.

  When the decision tree group to be constructed corresponds to the positive image 20 and the probability that the input patch cut out from the input image is likely to be similar to the positive image 20 or the negative image 30 This is used to calculate the probability of being likely to be similar to subclass 1 or subclass 2.

(D2: Introduction of the concept of “weight”)
In the object recognition method according to the present embodiment, the concept of “weight” is introduced for the nodes of the decision tree, and a learning patch that is likely to cause misidentification is substantially It is preferable to adopt a new mechanism that can be automatically selected.

  As described above, in the object recognition method according to the present embodiment, identification (class identification) between a positive sample and a negative sample, and learning image 22 ((a-1) easy grasping: subclass 1) and learning are performed. Since it is necessary to perform identification (subclass identification) with the image 24 ((a-2) difficulty in gripping: subclass 2), the first weight related to identification between the positive sample and the negative sample, and subclass 1 And two types of weights, the second weight related to the discrimination between subclass 2 and subclass 2.

  The “weight” according to the present embodiment is used when the decision tree group is constructed when the feature indicated by the learning patch obtained from one learning image is similar to the other learning image. Used to relatively reduce class probabilities for patches. The class probability is a probability used for voting processing for an input patch cut out from the input image. By reducing the probability of the learning patch cut out from the learning image and having low reliability, erroneous identification can be suppressed in the recognition processing.

  More specifically, among the learning samples cut out from the positive image 20, the weight (first weight) of the learning sample similar to the negative image 30 is relatively low. Among the learning samples cut out from the learning image 22 ((a-1) easy to grasp), the learning sample similar to the learning image 24 ((a-2) difficult to grasp) is weighted (second). ) Is relatively low. Each weight is sequentially calculated or updated when the decision tree group is constructed.

  Thus, for the learning sample cut out from the positive image 20, the first weight and the second weight are set independently. On the other hand, the first weight is set for the learning patch cut out from the negative image 30.

  FIG. 5 is a diagram for explaining the effect of weighting in the image recognition method according to the present embodiment. FIG. 5A conceptually shows an example of the distribution of weights determined by the learning process, and FIG. 5B shows an example of the voting result by the recognition process.

As a first weight, a weight w _1ij between the positive sample and the negative sample (class) is introduced, and as a second weight, between the subclass 1 (easy gripping) and the subclass 2 (difficult gripping) (subclass) The weight w _2ij is introduced. Here, the subscript “ij” means the j-th learning patch belonging to the i-th Bag. Bag is a group of a plurality of learning patches similar to each other, and details thereof will be described later.

As shown in FIG. 5A, it can be seen that the weight w _1ij between the positive sample and the negative sample is high in the region showing the characteristics not included in the negative image 30. In addition, it can be seen that the weight w _2ij between the subclass 1 and the subclass 2 is high in a region showing a feature that is not included in the learning image 22 of the subclass 1.

That is, in the recognition process, whether or not a work corresponding to the detection target is included in the input image is determined by weighting the information of the learning patch cut out from the region having the high weight w _1ij , and the detection target Whether or not the workpiece determined to correspond to can be gripped is determined with emphasis on the information of the learning patch cut out from the region having the high weight w _2ij .

  FIG. 5B conceptually shows an example of voting results obtained when recognition processing is performed using a decision tree group reflecting the distribution of weights as shown in FIG. As shown in FIG. 5B, for a work corresponding to a detection target, voting on a positive sample is negative regardless of whether the voting result 42 is equivalent to easy grasping or the voting result 44 is equivalent to difficult grasping. It exceeds the vote for the sample.

  On the other hand, in the voting result 42 corresponding to easy gripping, the vote for subclass 1 (easy gripping) exceeds the voting for subclass 2 (difficult gripping), and in the voting result 44 corresponding to gripping difficulty, the subclass The vote for 2 (difficult to grasp) exceeds the vote for subclass 1 (easy to grasp).

  As shown in the voting results 42 and 44, according to the object recognition method according to the present embodiment, whether the image corresponds to the positive image 20 or the negative image 30 by one recognition process, and the subclass 1 It can be determined at the same time whether the sub-class 2 is applicable.

As shown in FIG. 5, according to the object recognition method according to the present embodiment, the weights w _1ij and w _2ij for the region showing the feature effective for identification included in the learning image are automatically increased. Since it is calculated and updated, it is not necessary to set a rule for determining whether the recognized workpiece can be gripped in advance.

  In the object recognition method according to the present embodiment, each weight is sequentially updated when a decision tree group is constructed. The effects obtained by updating the respective weights will be described below.

  6 and 7 are diagrams for explaining the effect of updating the weights in the image recognition method according to the present embodiment. FIG. 6 shows an example in which a learning patch in a certain node is classified into a plurality of branch nodes in the process of building a decision tree.

In the left branch node in FIG. 6, negative samples and positive samples are mixed, and this state indicates that the possibility (reliability) of identifying positive samples and negative samples is relatively low. means. Therefore, if a branch result at such a branch node occurs, the weight w _1ij between the positive sample and the negative sample is decreased. That is, in the decision tree group building process, the weight w _{1ij of the} positive sample similar to the negative sample is decreased.

In the center branch node in FIG. 6, a learning patch cut out from the learning image 22 of the subclass 1 and a learning patch cut out from the learning image 24 of the subclass 2 are mixed, and this state is easy to grasp. This means that the possibility (reliability) of identifying (subclass 1) and difficult to grasp (subclass 2) is relatively low. Therefore, if a branch result at such a branch node occurs, the weight w _2ij between the subclass 1 and the subclass 2 is decreased. The same applies to the right branch node in FIG. In other words, in the decision tree group building process, the weight w _{2ij of the} positive samples that are similar between the subclasses is reduced.

  That is, after the branch node is determined, if the learning patch classified into each branch node is non-unique, the weight is decreased.

As shown in FIG. 7, in the recognition process, input patches cut out from the input image are input to the respective decision trees. In each decision tree, identification is performed by voting the information of the terminal node that the input patch has reached, but by _{copying the} weight w _2ij as described above, a work that is easy to grasp is copied. For input patches cut out from the input image, the voting value from the subclass 1 is relatively high, and for input patches cut out from the input image showing a work that is difficult to grasp, the voting value from the subclass 2 is relative. Become expensive. Based on the relative difference of the vote values, it can be determined which subclass it corresponds to.

(D3: Generation of learning image)
Next, a procedure for generating a learning image will be described.

  A positive image 20 at each rotation angle is generated by rotating an image obtained by imaging the workpiece to be detected. This is because in the object recognition method according to the present embodiment, it is necessary to recognize the position / posture of the workpiece, and it is necessary to learn each posture (the state after rotation) of the detection target. Typically, 360 types of learning images are generated by rotating an image acquired by imaging 360 ° by 1 °.

  In addition, in the object recognition method according to the present embodiment, an image showing a state (subclass 1) in which a beverage container to be detected is arranged to be grippable according to a subclass to be recognized, and the beverage container grips Two types of images, which are images of difficultly arranged states (subclass 2), are acquired and rotated for each to generate positive images 20 (learning image 22 and learning image 24) at each rotation angle. To do.

  That is, the object recognition method according to the present embodiment rotates a learning image by a predetermined angle to generate a plurality of learning images, and also generates a plurality of learning patches / learning samples (partial learning images) from the generated learning images. A step of extracting.

  On the other hand, the negative image 30 is generated from an image obtained by capturing an image of a non-detection target work. Since the negative image 30 is information used for identification with the workpiece to be detected, it is not necessary to rotate and generate a learning image for each rotation angle.

  In the object recognition method according to the present embodiment, it is preferable to introduce the concept of Bag. By introducing the concept of Bag, it is possible to suppress the influence of fluctuation of the detection target appearing in the learning image.

  FIG. 8 is a diagram for explaining generation of a bag of a learning image used in the object recognition method according to the present embodiment. Referring to FIG. 8, a positive patch and a negative patch are cut out from positive image 20 and negative image 30, respectively. As described above, since the positive image 20 for each rotation angle is generated, a plurality of positive patches are generated from each positive image 20 by grid sampling. From these learning patches, positive BagB1, B2, B3, B4,... Indicating the positive image 20 and negative BagNB1, NB2, NB3, NB4, NB5,.

  In the object recognition method according to the present embodiment, since a bag is generated from a learning image with a teacher signal in advance, a bag in which a positive patch and a negative patch are mixed is not generated, and each bag includes only a positive patch, Or only negative patches will be included.

  A positive bag is generated from a plurality of learning patches cut out from regions close to each other. That is, each positive bag includes a plurality of learning patches whose positions and / or angles approximate each other in the positive image 20. By generating a positive bag by such a method, a group of a plurality of learning patches having appearances similar to each other is handled as one bag. As described above, the object recognition method according to the present embodiment includes a step of setting a plurality of learning patches (partial learning images) indicating detection targets extracted from regions close to each other in a single group.

On the other hand, in the object recognition method according to the present embodiment, the negative patch cut out from the negative image 30 is mainly used to lower the weight w _1ij for a region that is not effective for recognition of the detection target in the positive image 20. It is not necessary to consider the influence of fluctuations appearing in the learning image. Therefore, one negative patch cut out from the negative image 30 is set as one negative bag.

  As will be described later, in the learning process of the object recognition method according to the present embodiment, when a decision tree group is constructed, a weight is commonly applied to a plurality of learning patches (partial learning images) belonging to the same group. To decide. That is, learning patches included in the same Bag are assumed to have similar information to each other, and therefore, the processing is simplified by setting a common weight.

(D4: Construction of decision tree group)
Next, the decision tree group construction process in the learning process will be described. FIG. 9 is a diagram for explaining a decision tree group construction process in a learning process in the object recognition method according to the present embodiment. Referring to FIG. 9, positive patch 21 (positive patch 23 belonging to subclass 1 and positive patch 25 belonging to subclass 2) cut out from positive image 20 and negative patch 31 cut out from negative image 30 are shown. Randomly selected, a plurality of subsets 62 are generated. Since one decision tree 60 is constructed (ie, learned) from one subset 62, the same number of subsets 62 as the number of decision trees 60 included in the decision tree group to be constructed is generated.

In the learning process, a branch function is determined for each subset (node) for the subset 62, and the degree of separation between patches branched to each child node according to the determined branch function, that is, the class likelihood is calculated. The Based on the calculated class likelihood, the weight in the corresponding node (hierarchy) is calculated or updated. In this case, the weight w _1ij between positive samples and negative _samples, and both the weights w _2ij between subclasses 1 and subclass 2 is updated. A series of processes such as determination of a branch function, calculation of class likelihood, and update of weight are repeated until all learning patches included in the subset 62 reach the end node.

  Add up the weights assigned to the nodes 51-1, 51-2, 52-1, 52-2, 52-3, 52-4 on the route from the root node 50 to each of the end nodes 54-1 to 54-N. By doing so, the weight for each of the end nodes 54-1 to 54-N is finally determined.

  Each terminal node 54-1 to 54-N included in the constructed decision tree 60 holds the class probability and the offset vector obtained by the learning process. The offset vector is a vector quantity indicating the direction and distance from the patch center to the object center.

  Further, in each of the end nodes 54-1 to 54-N, a histogram about category information (for example, information about detection target / non-detection target distinction, position, angle, etc.) attached to the learning patch that has arrived there is a histogram. You may associate. Note that after the complete decision tree 60 is constructed, the weights (or class probabilities) for the end nodes 54-1 to 54-N may be adjusted.

The terminal node 54-N illustrated in FIG. 9 is a node that the negative patch 31 is likely to reach. In the process of reaching the terminal node 54-N, the positive patch 21 (subclass 1 is included in the subset 62). The weights w _1ij for the positive patches 23 that belong and the positive patches 25) that belong to subclass 2 will decrease sequentially. FIG. 9 conceptually shows the decrease in the weight w _{1ij in terms} of area size.

(D5: learning process procedure)
Next, a processing procedure of learning processing of the object recognition method according to the present embodiment will be described. FIG. 10 is a flowchart showing the processing procedure of the learning process of the object recognition method according to the present embodiment. Each step shown in FIG. 10 is typically realized by the processor 102 of the image processing apparatus 100 executing the image processing program 108.

  Referring to FIG. 10, the image processing apparatus 100 acquires learning images (positive image 20 and negative image 30) used for learning processing (step S2). Subsequently, the image processing apparatus 100 rotates the acquired positive image 20 to generate a learning image at each rotation angle, cuts out the positive patch and the negative patch, and generates a positive bag and a negative bag (step S4). ). Since the processing contents of steps S2 and S4 have been described in detail with reference to FIG. 8, detailed description will not be repeated.

Then, the image processing apparatus 100 initializes the weights w _1ij and the weights w _2ij (Step S6), and generates a predetermined number of subsets from the plurality of Bags generated in Step S4 (Step S8). Each of the subsets is generated by randomly selecting (random sampling) from a plurality of generated positive Bags and negative Bags.

  Then, the process (step S10-S20) which builds a decision tree group is started. One decision tree is constructed by performing the processing of steps S10 to S20 once. It is preferable that the processes of steps S10 to S20 are executed in parallel for the number of generated subsets. Of course, you may make it repeat the process of step S10-S20 in multiple times in series.

  In the process of building a decision tree, the image processing apparatus 100 first generates a branch function candidate at a node in the hierarchy 1 (step S10), and selects an optimum one of the generated branch function candidates in the hierarchy 1 A branch function is determined (step S12). Then, the image processing apparatus 100 determines whether or not there remains a node for which a branch function is not determined in the same hierarchy (step S14). If there is a node that has not yet determined a branch function in the same hierarchy (YES in step S14), the processes in and after step S10 are repeated.

If the branch function has been determined for all the nodes in the same hierarchy (NO in step S14), the image processing apparatus 100 updates the weights (weights w _1ij and weights w _2ij ) for each node ( Step S16).

  Thereafter, the image processing apparatus 100 determines whether or not a predetermined condition relating to completion of the construction of the decision tree group is satisfied (step S18). If the predetermined condition relating to the completion of the construction of the decision tree group is not satisfied (NO in step S18), the processes in and after step S10 are repeated.

  When the predetermined condition relating to the completion of the construction of the decision tree group is satisfied (in the case of YES in step S18), the image processing apparatus 100 associates with the end node of each decision tree and the learning patch image itself reached. Information such as an offset vector and a weighted probability is stored (step S20). Then, the learning process ends.

Hereinafter, more detailed contents of the learning processing procedure will be described.
(D6: weight initialization)
The image processing apparatus 100 initializes the weight w _1ij and the weight w _2ij before constructing the decision tree (step S6). More specifically, the image processing apparatus 100 _initializes the weight w _1ij and the weight w _2ij to 1 / N. Here, the constant N can be set to an arbitrary value.

(D7: generation of branch function candidates)
As the first stage of the decision tree construction process, the image processing apparatus 100 generates a branch function candidate in the hierarchy 1 (step S10). Here, the branch function candidate h ^(d) _{T, τ} (I) in the hierarchy d uses the similarity S (x, T) between the learning patch x extracted from the learning image and the template T and the threshold value τ. Thus, the following equation (1) is defined.

  Here, the parameter τ is a threshold value for evaluating the similarity between the learning patch x and the template T.

(D8: Determination of branch function)
Subsequently, the image processing apparatus 100 determines an optimal one from the generated branch function candidates as a branch function in the target hierarchy (step S12).

  FIG. 11 is a schematic diagram for explaining processing for determining a branch function in learning processing in the object recognition method according to the present embodiment. Referring to FIG. 11, the template T in the formula (1) is randomly selected from a plurality of positive patches given to the target branch node. In the example shown in FIG. 11, each learning patch x has a similarity with the template T calculated, and if the calculated similarity is less than the threshold, it is branched to the left child node. Branches to the right child node.

In the branch function determination process, the template T and the parameter τ are selected at random, and the evaluation value is calculated using the evaluation function U _* defined by the following equation (2). However, * ε {1, 2}, which corresponds to the weights w _1ij and w _2ij (the same applies hereinafter).

  A template T and a parameter τ that minimize the evaluation value calculated by the equation (2) are determined, and an optimal branch function is determined from these determined values.

Here, {x | h (I _i ) = 0} in the expression (2) indicates a set of learning patches divided into child nodes on the left side of FIG. 11, and {x | h (I _i ) = 1} Indicates a set of learning patches divided into child nodes on the right side of FIG.

For the evaluation of the evaluation function U, the following three criteria are switched for each layer and used. The first is evaluation functions U ₁ (A) and U ₂ (A) that evaluate the entropy of the class label, and the following (3) and (4) ) Is defined according to the formula.

The evaluation function U ₁ (A) indicates the ease (reliability) of discriminating between the positive sample and the negative sample, and the evaluation function U ₂ (A) is easy to grasp (subclass 1) and difficult to grasp (subclass 2). The ease of identification (reliability) is shown.

Here, c in the evaluation function U ₁ (A) represents the weighted probability of the positive sample included in the learning patch set A, and b in the evaluation function U ₂ (A) represents the learning patch set A. Of these, the weighted probabilities for the positive sample teacher attributes are shown.

The third is an evaluation function U ₃ (A) for evaluating the variation (dispersion) of the offset vector d _ij and is defined according to the following equation (5). The offset vector _dij indicates the position where the target learning patch is cut out from the learning image (positive image 20).

Here, d _A indicates an average value of A's obset vector.
Furthermore, the processes in steps S10 and S12 are repeated until all necessary nodes in the target hierarchy d are generated in each decision tree. That is, in steps S10 to S14 in FIG. 10, the image processing apparatus 100 classifies a given learning sample (partial learning image) as a child node branched from the node for each node that is not a terminal node. Processing for determining each branch function indicating whether or not to be performed is executed.

(D9: weight update)
When all the nodes for the target hierarchy d are generated, the image processing apparatus 100 updates the weights (weight w _1ij and weight w _2ij ) for each node (step S16).

The weight w _1ij and the weight w _2ij are determined or updated based on the respective class likelihood and the Bag class likelihood p _i . In addition, since it is not essential to introduce the concept of Bag, when the concept of Bag is not introduced, the weight may be determined or updated only from the class likelihood.

The update of the weight w _1ij is performed only for positive samples. Among the positive samples included in the set A, the class likelihood is relatively lowered for those that are similar to the negative samples included in the same set A. Therefore, a first class likelihood p _1ij indicating ease (reliability) of discriminating between a positive image and a negative image is _represented by the following expression (6).

The weight w _2ij is updated for the positive samples (positive patch 23 belonging to subclass 1 and positive patch 25 belonging to subclass 2) included in set A. Among the positive samples belonging to the subclass 1 included in the set A, the class likelihood is relatively lowered for those similar to the positive sample belonging to the subclass 2 included in the same set A. Similarly, among the positive samples belonging to the subclass 2 included in the set A, those that are similar to the positive samples belonging to the subclass 1 included in the same set A have a relatively low class likelihood. Therefore, the second class likelihood p _2ij indicating the ease (reliability) of discrimination between easy grip (subclass 1) and hard grip (subclass 2) is shown in the following equation (7).

In the equation (6), if defined as F (x) = 1−2c (where c is the ratio of positive samples in the set A), the class likelihood p _1ij is higher as the positive sample ratio c is higher at each node. Conversely, the lower the positive sample ratio c, the lower the class likelihood p _1ij .

Similarly, in the equation (7), a positive sample belonging to subclass 1 is defined as G ₁ (x) = 1−2b ₁ (where b ₁ is the proportion of positive samples belonging to subclass 1 in set A), A positive sample belonging to subclass 2 is defined as G ₂ (x) = 2b ₂ −2 (where b ₂ is the proportion of positive samples belonging to subclass 2 in set A).

The class likelihood p _{* i} of Bag is calculated according to the following equation (8) using the class likelihood p _{* ij} of the p set belonging to Bag.

The weight w _{* ij} is calculated according to the equation (9) using the Bag class likelihood p _{* i} and the class likelihood p _{* ij} .

It is preferable to normalize the finally calculated weights w _1ij and w _2ij .
As described above, in the learning process for constructing the decision tree group, each of the learning samples given according to the determined branch function is classified into one of the child nodes at each node that is not the terminal node. Based on the result of classifying the plurality of learning samples, the weight w _1ij (first weight) indicating the discrimination ability between the detection target and the non-detection target for each node, and the subclass 1 and the subclass 2 The weights w _2ij (second weights) indicating the discriminating ability are respectively determined.

That is, in the learning process for constructing a decision tree group, when a plurality of learning samples given according to the determined branch function are classified into any child node, the positive sample and the negative sample are classified into the same child node. The higher the ratio (the lower the degree of separation), the lower the weight w _1ij (first weight) indicating the discrimination ability between the detection target and the non-detection target. Further, the higher the ratio that subclass 1 and subclass 2 are classified into the same child node (the lower the degree of separation), the weight w _2ij (second weight) indicating the discrimination ability between subclass 1 and subclass 2 make low.

(D10: Generation of terminal node)
The image processing apparatus 100 repeats the branch function determination and weight update processing described above until a predetermined condition is satisfied. The predetermined condition includes, for example, whether the hierarchy reaches a specified depth, or whether the number of learning patches included in each node set is less than a certain number. The lowest node when the predetermined condition is satisfied is the terminal node.

  The image processing apparatus 100 stores information such as the reached learning patch image itself, the offset vector, and the weighted probability in association with each terminal node (step S20).

  FIG. 12 is a schematic diagram for explaining terminal node information obtained by learning processing in the object recognition method according to the present embodiment. Referring to FIG. 12, the image itself, the offset vector, and the weighted probability are stored for positive patch 23 and positive patch 25 that have reached the terminal node in association with the terminal node. For the negative patch 31 that has reached the end node, the image itself and the weighted probability are stored. That is, in steps S16 to S18 of FIG. 10, the image processing apparatus 100 sequentially branches each of the learning samples (partial learning images) until reaching one of the end nodes according to each determined branch function. For each terminal node, the sum of the positive patch 23 (first sample) cut out from the learning image 22 and the positive patch 25 (second sample) cut out from the learning image 24 and the negative cut out from the negative image 30 The ratio (first probability) with the patch 31 (third sample) and the ratio (second probability) between the positive patch 23 (first sample) and the positive patch 25 (second sample). The process to determine is executed.

  Through the learning process as described above, a decision tree group including a plurality of decision trees is constructed. In the recognition process, the voting process is performed using the weighted probability to determine the recognition of the position / orientation of the detection target and the appropriateness to the designated subclass.

<E. Recognition process>
Next, the recognition process of the object recognition method according to the present embodiment will be described. In the recognition processing, a plurality of partial images (input patches) cut out from the input image are input to the decision tree group, and each input tree is subjected to voting processing using the information of the terminal node in each decision tree. From the result, in addition to the recognition of the position and orientation of the detection target included in the input image, it is determined whether or not the recognized workpiece can be gripped. In this voting process, voting planes (XY coordinates) are prepared only for rotation angle types (360 in the above example), and these are combined to obtain a three-dimensional likelihood (similarity). A map is constructed. Then, the XY coordinates and the rotation angle of the detection target are determined from the high likelihood region in the three-dimensional likelihood map.

  Furthermore, in the recognition processing of the object recognition method according to the present embodiment, the recognized detection target is easy to grasp (subclass 1) based on the subclass probability voted for the area determined to correspond to the detection target. It is determined whether there is a gripping difficulty (subclass 2).

  FIG. 13 is a flowchart showing a recognition processing procedure in the object recognition method according to the present embodiment. Each step shown in FIG. 13 is typically realized by the processor 102 of the image processing apparatus 100 executing an image processing program.

  Referring to FIG. 13, the image processing apparatus 100 acquires an input image that is a target of recognition processing (step S102). Subsequently, the image processing apparatus 100 cuts out a plurality of input patches from the acquired input image (step S104), inputs each input patch to the decision tree group, and identifies a terminal node to which each input patch arrives ( Step S106). The image processing apparatus 100 performs a voting process using information held in the terminal node reached by each input patch in each decision tree (step S108). It should be noted that terminal nodes having a positive sample ratio less than the threshold value may not be subject to voting processing. In other words, in the recognition process, only the terminal nodes whose positive sample ratio after reflecting the weight is equal to or higher than a predetermined threshold may be subject to the voting process.

  When the voting process for all input patches is completed, the following search process is executed. Specifically, the image processing apparatus 100 specifies one or a plurality of local regions by scanning a voting plane having a certain rotation angle θ (step S110). Subsequently, the image processing apparatus 100 calculates the sum for each local area (step S112) and searches for the maximum value of the sum of the local areas on the voting plane (step S114). The image processing apparatus 100 determines whether or not the search process has been completed for all the rotation angles θ (step S116). If there is a rotation angle for which the search process has not been completed (NO in step S116), the image processing apparatus 100 selects a new rotation angle θ (step S118), and repeats the processes in and after step S110.

  When the search processing has been completed for all the rotation angles θ (YES in step S116), the image processing apparatus 100 detects the rotation angle θ corresponding to the voting plane that maximizes the sum of the local areas. And the point of interest in the local region is determined as the position to be detected (step S120). Note that when the total sum of the local regions is less than a predetermined threshold value, it may be determined that there is no detection target in the input image.

  Subsequently, the image processing apparatus 100 determines the subclass probability of the positive sample voted for the attention point (region) determined in step S120 (that is, the weighted probability voted for subclass 1 and the weight voted for subclass 2). Based on the ratio to the attachment probability, it is determined whether it is easy to grasp (subclass 1) or difficult to grasp (subclass 2) (step S122).

  FIG. 14 is a diagram for explaining recognition processing in the object recognition method according to the present embodiment. Referring to FIG. 14, a plurality of input patches cut out from an input image are input to decision trees 60-1, 60-2,..., 60-N. In each decision tree 60-1, 60-2,..., 60-N, it is specified which terminal node the input patch has reached.

  Then, voting processing to a three-dimensional likelihood map is performed using information associated with each terminal node. More specifically, based on the offset vector associated with the terminal node that the input patch has reached, the voting destination (position to be detected) of the three-dimensional likelihood map is identified and associated as a voting value. Weighted positive probabilities are used.

  When the voting process is performed for each of the plurality of input patches cut out from the input image, and the voting value for any position exceeds a predetermined threshold value, the voting plane (rotation angle) at that position And a position (XY coordinate) is specified as a position corresponding to a detection target.

  Furthermore, it is determined whether it is easy to grasp (subclass 1) or difficult to grasp (subclass 2) based on the ratio of the subclass of the positive samples voted to the position corresponding to the detection target.

  As described above, in the recognition processing, the image processing apparatus 100 uses the terminal nodes to which the input patches reach when a plurality of input patches (partial input images) obtained from the input images are given to the decision tree group. Based on the weighted probability (first probability) of the positive sample, it is determined whether or not a work corresponding to the detection target is included in the input image, and subclass 1 or subclass 2 for each terminal node. Based on the weighted probability (second probability), it is determined whether or not the workpiece corresponding to the detection target included in the input image meets a predetermined condition (easy gripping / hard gripping).

<F. Recognition result>
A recognition result using the object recognition method according to the present embodiment will be described. FIG. 15 is a diagram illustrating an example of a recognition result obtained by the object recognition method according to the present embodiment. In the recognition result shown in FIG. 15, an image captured in a state where a beverage container that is three works is arranged is used as an input image. (A) As shown in the recognition result, all of the three beverage containers are recognized as detection targets. On the other hand, as shown in the (b) likelihood map, the two drink containers on the right side have a relatively high value in the likelihood map of the subclass 2, and may be difficult to grasp, i.e., the subclass 2. It is determined that the nature is high.

  That is, according to the object recognition method according to the present embodiment, in addition to the recognition of the position / posture of the workpiece to be detected, it is possible to simultaneously determine the subclass (easy gripping / hard gripping).

<G. Additional learning process>
Next, a process for improving the discrimination performance of the decision tree group constructed by the learning process of the object recognition method according to the present embodiment will be described. Typically, when the learning image used in the learning process is not suitable for an environment such as an actual site, the decision tree group may not exhibit the original discrimination performance. In such a case, a new node (branch function: discriminator) is added to the decision tree group using the misidentification sample. The additional learning process of the object recognition method according to the present embodiment includes such a new node addition process. By adopting such additional learning processing, it is possible to improve the discrimination performance of the decision tree group.

(G1: Overview)
In the additional learning process of the object recognition method according to the present embodiment, the learning sample and the misidentification sample are input to all the decision trees that are already constructed. Then, all decision trees are scanned, and child nodes are added to the terminal nodes having a low class probability of misidentified samples. In this specification, misidentified samples include input samples that are likely to be misidentified as classes or subclasses in addition to those that are misidentified as classes or subclasses as a result of actual recognition processing. means.

  That is, for a terminal node that is considered to be difficult to identify misidentified samples (that is, with low reliability), a branch function for increasing the reliability of identification is added. Note that either a feature selection type or a case type can be used as the branch function to be added. Specific contents of these branch functions will be described later.

  FIG. 16 is a diagram for explaining additional learning processing in the object recognition method according to the present embodiment. With reference to Fig.16 (a), the misidentification sample to which the correct label is provided is input with respect to the decision tree 60 constructed | assembled by the learning process. In the example shown in FIG. 16A, it is assumed that the misidentification sample has reached the end node 54-2. In the class likelihood at the terminal node 54-2, it is assumed that the negative sample is higher than the positive sample. Therefore, the misidentification sample is separated from the learning sample by adding a branch function to the terminal node 54-2 where the misidentification sample has arrived.

  FIG. 16B shows an example in which the end node 54-2 is changed to a normal node, and end nodes 55-1 and 55-2 are added as its child nodes. In this way, by adding child nodes, the decision performance of the decision tree group can be improved.

  If the class probability of the misidentification sample is high among the end nodes that have reached the misidentification sample, a child node may not be added to the end node. In this case, it is considered that class identification between the learning sample and the misidentification sample is sufficiently performed.

  FIG. 17 is a flowchart showing the additional learning processing procedure in the object recognition method according to the present embodiment. Each step shown in FIG. 17 is typically realized by the processor 102 of the image processing apparatus 100 executing an image processing program.

  Referring to FIG. 17, the image processing apparatus 100 acquires a learning sample and a misidentification sample used for the additional learning process (step S202). The image processing apparatus 100 inputs each of the acquired learning sample and misidentification sample to each of the constructed decision tree groups, and identifies the end node to which each sample reaches in each decision tree (step S204). In other words, in steps S202 and S204, the image processing apparatus 100 inputs a misidentified sample (misidentified or misidentified) among the learning sample (partial learned image) and the input sample (partial input image). (Sample) is given to the decision tree group, and each sample is sequentially branched until reaching any one of the end nodes, thereby specifying the end node to which each sample reaches.

  Subsequently, the image processing apparatus 100 determines in each decision tree whether the class probability of the misidentification sample at the terminal node where the misidentification sample has reached meets a predetermined condition (step S206). If the class probability of the misidentification sample does not meet a predetermined condition (YES in step S206), the image processing apparatus 100 determines a branch function (step S208) and is generated by the determined branch function. The weight for the child node is updated (step S210).

  After updating the weight for the generated child node (after step S210), or when the class probability of the misidentified sample does not meet a predetermined condition (YES in step S206), image processing is performed. The apparatus 100 determines whether or not the evaluation has been completed for all terminal nodes that have reached the misidentification sample in the target decision tree (step S212). If the evaluation has not been completed for all terminal nodes that have reached the misidentification sample (NO in step S212), the image processing apparatus 100 selects another terminal node that has reached the misidentification sample (step S214). ), And the processing from step S206 is repeated.

  That is, in steps S206 to S214, the image processing apparatus 100 determines the end node according to the discrimination probability between the learning sample (partial learning image) and the misidentification sample (partial input image) at the end node where the misidentification sample has arrived. Add a child node that branches from.

  If the evaluation has been completed for all the end nodes to which the misidentification sample has arrived (YES in step S212), the image processing apparatus 100 determines all the decision trees included in the constructed decision tree group. It is determined whether the evaluation is completed (step S216). If the evaluation has not been completed for all the decision trees included in the constructed decision tree group (NO in step S216), the image processing apparatus 100 determines whether another decision tree group included in the constructed decision tree group includes another decision tree group. A decision tree is selected (step S218), and the processing from step S206 is repeated.

  If the evaluation has been completed for all decision trees included in the constructed decision tree group (YES in step S216), image processing apparatus 100 ends the additional learning process.

(G2: Collection of misidentified samples and labeling)
In order to perform additional learning processing, it is necessary to collect misidentification samples obtained as a result of actual recognition processing. Since a work (a teacher signal) is not given to a work that is a target of an actual recognition process, it is preferable to collect a misidentification sample by visually confirming a recognition result. However, since such a technique of collecting erroneously identified samples by visual inspection requires high human costs, it may be possible to evaluate the vote entry and automatically collect erroneously identified samples. “Vote Entropy” indicates the reliability of the identification result in the ensemble classifier. The value VE (x) of the vote entropy is defined as the following equation (10).

However, T is the number of decision trees, V (y _m) indicates the number of decision trees that predict the label y _m. For input samples with a high value of Vote Entropy, it can be determined that the identification by the decision tree group is ambiguous. Therefore, input samples whose Vote Entropy value is greater than or equal to a predetermined threshold value are automatically collected. Then, by applying a visual correct label to the automatically collected input sample, a misidentification sample used for the additional learning process is generated.

(G3: Feature selection type branch function)
The feature selection type branch function used for adding a child node is a method of selecting a feature dimension to be referenced and its threshold value based on information gain obtained from randomly selected candidates. For calculation of the feature amount F, a Haar-like filter can be used. As a branch function, the filter pattern, size, and position of the Haar-like filter are selected. Further, the branch function is defined as the following equation (11).

  Here, F (x) indicates a value obtained by extracting features from an input sample by a Haar-like filter, and τ indicates a threshold value.

(G4: Case type branch function)
In the case type branch function used for adding child nodes, a template selected from the misidentification sample is used as a case. The case type bifurcation function calculates the distance between the learning sample and the misidentification sample of the same end node using a certain misidentification sample as a template, and thresholds the midpoint between the sample with the closest distance and a different class. Determine as a value and determine the branch function. After the additional learning, if there is a misidentification sample, additional learning is further performed. The Euclidean distance is used for the distance calculation. A combination of template and threshold is selected based on information gain. The branch function is defined as the following equation (12).

  However, D shows the distance of a template and an input sample, (tau) shows a threshold value. Further, the threshold value of the case type branch function is defined as the following equation (13).

However, d _i denotes the sample closest distances among the templates and classes of different learning samples, T is shows a template.

<H. Advantage>
The object recognition method according to the present embodiment constructs a decision tree group by performing learning processing at a time using an input sample for identifying a class and an input sample for further identifying the class as a subclass. In addition, class discrimination and subclass discrimination can be simultaneously performed by a single recognition process using a common decision tree group. Therefore, the time and human cost required for the learning process and the recognition process can be reduced.

  As a more specific effect, it is determined whether or not a detection target is included in the input image, and whether or not the detection target included in the input image meets a predetermined condition (recognized) The situation where the workpiece is placed) can be judged in parallel. In addition, it is not necessary to visually determine a rule for determining whether or not a detection target included in the input image meets a predetermined condition, and the human cost for determining these rules can be reduced. .

  In addition, the object recognition method according to the present embodiment, when erroneous identification occurs in the recognition process using the constructed decision tree group (or when there is a high possibility of erroneous identification), the decision tree group Can be further learned, and the possibility of such misidentification can be reduced.

<I. Other Embodiments>
One of the concepts included in the object recognition method according to the present embodiment is that the weight of a positive sample similar to a negative sample is relatively low, and the weight of subclass 2 similar to subclass 1 (and subclass 2) The weight of the subclass 1 similar to the above may be relatively low. The mathematical processing of the weights for that can be arbitrarily adopted. That is, the weight of the positive sample that is not similar to the negative sample is relatively high, or the weight of the subclass 2 that is not similar to the subclass 1 (and the weight of the subclass 1 that is not similar to the subclass 2) May be relatively high.

  In the above description, the description has been made on the assumption that the influence is reduced as the “weight” is lower. However, the concept of “weight” may be used in the opposite concept to the above description. That is, the concept may be that the higher the “weight”, the less the target of voting processing.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  1 image processing system, 2 imaging device, 4 work, 10 input image, 12 work recognition, 13 recognition processing, 14 gripping determination, 16 gripping operation, 20 positive image, 21, 23, 25 positive patch, 22, 24 learning image, 30 negative images, 31 negative patches, 42, 44 voting results, 50 root nodes, 51-1, 51-2, 52-1, 52-2, 52-3, 52-4 nodes, 54-1 to 54-N 55-1, 55-2 End node, 60 decision tree, 62 subset, 100 image processing device, 102 processor, 104 main memory, 108 image processing program, 110 network interface, 112 image input interface, 114 input unit, 116 display Part, 118 output interface, 120 internal bar Su, 200 picking robot.

Claims (8)

Using a plurality of partial learning images obtained from the learning image, and constructing a decision tree group having a hierarchical structure from a root node to a plurality of terminal nodes,
The plurality of partial learning images include a first sample indicating a portion that satisfies a predetermined condition among detection targets, and a second sample indicating a portion that does not conform to the predetermined condition among the detection targets. And a third sample indicating a non-detection target,
The step of constructing the decision tree group includes:
For each node that is not a terminal node, determining a branch function that indicates to which of the child nodes that the given partial learning image should branch from the node;
According to the determined branch function, each of the partial learning images is sequentially branched until reaching one of the end nodes, so that the sum of the first sample and the second sample is obtained for each end node. Determining a first probability indicating a ratio between the first sample and the third sample, and a second probability indicating a ratio between the first sample and the second sample, and further comprising: When a plurality of partial input images obtained from the above are given to the decision tree group, a detection target is included in the input image based on a first probability for each terminal node to which each partial input image arrives. And whether the detection target included in the input image is based on the second condition based on the second probability for each terminal node. An image processing method comprising a step of determining whether or not the information conforms to.   The step of constructing the decision tree group is based on the result of classifying a plurality of partial learning images given according to the determined branch function into any one of the child nodes at each node that is not a terminal node. Determining a first weight indicating the discrimination capability between the detection target and the non-detection target and a second weight indicating the discrimination capability between the first sample and the second sample, respectively. The image processing method according to claim 1, further comprising:   The step of constructing the decision tree group is such that when the plurality of partial learning images given according to the decided branch function are classified into any child node, the first sample and the second sample are the same. The image processing method according to claim 1, further comprising a step of lowering a weight indicating the discrimination ability between the first sample and the second sample as the ratio of being classified as a child node is higher. The partial learning image and the partial input image that is misidentified or highly likely to be misidentified among the partial input images are given to the decision tree group until each image reaches one of the end nodes. Identifying the end node to which each image arrives by sequentially branching;
The method according to claim 1, further comprising: adding a child node that branches from the terminal node according to a discrimination probability between the partial learning image and the partial input image at the terminal node reached by the terminal image. The image processing method according to claim 1. Further comprising a step of setting a plurality of partial learning images indicating the detection targets extracted from regions close to each other in a single group,
The image processing method according to claim 1, wherein the step of constructing the decision tree group includes a step of determining a weight in common for a plurality of partial learning images belonging to the same group. .   6. The method according to claim 1, further comprising: generating a plurality of learning images by rotating the learning image by a predetermined angle; and extracting a plurality of partial learning images from the generated plurality of learning images. The image processing method as described. A means for constructing a decision tree group having a hierarchical structure from a root node to a plurality of terminal nodes using a plurality of partial learning images obtained from a learning image,
The plurality of partial learning images include a first sample indicating a portion that satisfies a predetermined condition among detection targets, and a second sample indicating a portion that does not conform to the predetermined condition among the detection targets. And a third sample indicating a non-detection target,
The means for constructing the decision tree group includes:
Means for determining for each node that is not a terminal node, a branch function that indicates to which of the child nodes branched from the node the given partial learning image should be classified;
According to the determined branch function, each of the partial learning images is sequentially branched until reaching one of the end nodes, so that the sum of the first sample and the second sample is obtained for each end node. And a means for determining a first probability indicating a ratio between the first sample and the third sample, and a second probability indicating a ratio between the first sample and the second sample, and an input image When a plurality of partial input images obtained from the above are given to the decision tree group, a detection target is included in the input image based on a first probability for each terminal node to which each partial input image arrives. The detection target included in the input image is suitable for the predetermined condition based on the second probability for each terminal node. An image processing apparatus comprising means for determining whether or not to match. An image processing program executed on a computer, the image processing program being stored in the computer
Using a plurality of partial learning images obtained from the learning image, to execute a step of building a decision tree group having a hierarchical structure from a root node to a plurality of terminal nodes,
The plurality of partial learning images include a first sample indicating a portion that satisfies a predetermined condition among detection targets, and a second sample indicating a portion that does not conform to the predetermined condition among the detection targets. And a third sample indicating a non-detection target,
The step of constructing the decision tree group includes:
For each node that is not a terminal node, determining a branch function that indicates to which of the child nodes that the given partial learning image should branch from the node;
According to the determined branch function, each of the partial learning images is sequentially branched until reaching one of the end nodes, so that the sum of the first sample and the second sample is obtained for each end node. Determining a first probability indicating a ratio between the first sample and the third sample, and a second probability indicating a ratio between the first sample and the second sample, and further comprising: When a plurality of partial input images obtained from the above are given to the decision tree group, a detection target is included in the input image based on a first probability for each terminal node to which each partial input image arrives. And whether the detection target included in the input image is based on the second condition based on the second probability for each terminal node. An image processing program for executing a step of determining whether or not the information conforms to the above.