JP2016115179A - Information processing unit, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a holding object even when there is no holding object in an image of holding means holding a holding object.SOLUTION: A recognition device 100 includes an image acquisition section 110, a holding feature generation section 112, and a holding object recognition section 116. The image acquisition section 110 acquires an image of holding means 501 holding a holding object 502. The holding feature generation section 112 generates holding features representing a physical relationship of the holding means 501 and a plurality of predetermined sites in the image. The holding object recognition section 116 recognizes the holding object 502 on the basis of the holding features.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing device, an information processing method, and a program, and more particularly, to an information processing device, an information processing method, and a program for recognizing an object held by a holding unit.

  Using an image of a state where a person or the like (hereinafter also referred to as a gripper) is holding an object (hereinafter also referred to as a gripping target) with a finger or the like (hereinafter also referred to as gripping means), A technique for recognizing a gripping object is known.

  For example, in the technique disclosed in Patent Document 1, when matching a partial region of an object in an image with an article shape in a database, the matching is performed using the size of a space calculated from a hand gripping posture. Limit the items of interest.

  Further, in the technique disclosed in Non-Patent Document 1, the feature of the shape pair of the hand region and the object region in the image is stored for each grip pattern, and the grip pattern is recognized using the feature.

  As a related technique, Patent Document 2 discloses a technique for specifying a virtual key to be operated using a finger position detected from an image. Patent Document 3 discloses a technique for recognizing an object included in an image using a color histogram. Patent Document 4 discloses a technique for specifying an object based on the characteristics of changes in the area and outer peripheral length of the object in a moving image. Non-Patent Document 2 discloses a technique for detecting a finger region by searching for a designated color in an image photographed by wearing a glove (glove) having a designated color for each finger.

JP 2010-244413 A JP2013-143082A JP 2012-150552 A JP 2010-244440 A

Hiromasa Kasahara, 3 others, “Restoring Missing Pixels and Object Recognition Based on Grasping Pattern Image Learning”, Image Recognition and Understanding Symposium (MIRU2008), July 2008, p.623-628 Ken Watanabe and three others, "Recognition of finger characters using color gloves", IEICE Transactions, D-II, 1997, vol. J80-D-2, no. 10, p.2713-2722

  As described above, when the gripping target is recognized by an image obtained by capturing the state where the gripping means is gripping the gripping target, for example, the gripping target is covered by a finger or palm, and there is no gripping target in the image. There is a case. However, in the techniques described in Patent Literature 1 and Non-Patent Literature, since the gripping target is recognized using a partial image of the gripping target, there is no gripping target in the image in this way. The gripping target cannot be recognized.

  An information processing apparatus, an information processing method, and an information processing apparatus capable of recognizing a gripping target even when the gripping target does not exist in an image of a gripping unit that grips the gripping target. Is to provide a program.

  The information processing apparatus of the present invention generates an image acquisition unit that acquires an image of a gripping unit that is gripping a gripping target, and a gripping feature that indicates a positional relationship between a plurality of predetermined parts of the gripping unit in the image. Gripping feature generating means; and gripping object recognition means for recognizing the gripping object based on the gripping feature.

  The information processing method of the present invention acquires an image of a gripping means that grips a gripping target, generates a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping means in the image, and Based on the above, the object to be grasped is recognized.

  The program of the present invention acquires, on a computer, an image of a gripping unit that grips a gripping target, generates a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping unit in the image, and the gripping feature Based on the above, a process for recognizing the gripping object is executed.

The effect of the present invention is that the gripping target can be recognized even when the gripping target does not exist in the image of the gripping means that grips the gripping target.

It is a block diagram which shows the fundamental structure of embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 100 in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 100 implement | achieved by the computer in the 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the recognition apparatus 100 in the 1st Embodiment of this invention. It is a figure which shows the example of the acquired image in the 1st Embodiment of this invention. It is a figure which shows the example of the production | generation method of the grasping feature-value in the 1st Embodiment of this invention. It is a figure which shows the other example of the production | generation method of the grasping feature-value in the 1st Embodiment of this invention. It is a figure which shows the example of the holding | grip characteristic information 115 in the 1st Embodiment of this invention. It is a figure which shows the example of extraction of the instance in the 1st Embodiment of this invention. It is a figure which shows the example of the calculation result of the object likelihood in the 1st Embodiment of this invention. It is a figure which shows the other example of the grip characteristic information 115 in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 200 in the 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the recognition apparatus 200 in the 2nd Embodiment of this invention. It is a figure which shows the example of the object characteristic information 225 in the 2nd Embodiment of this invention. It is a figure which shows the example of the calculation result of integrated likelihood in the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 200 in the 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of the recognition apparatus 200 in the 3rd Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 200 in the 4th Embodiment of this invention. It is a flowchart which shows operation | movement of the recognition apparatus 200 in the 4th Embodiment of this invention. It is a figure which shows the example of calculation of the grasping feature-value containing a movement feature in the 4th Embodiment of this invention.

<First Embodiment>
First, a first embodiment of the present invention will be described.

  In the first embodiment of the present invention, the gripping target 502 is recognized based on the positional relationship between a plurality of predetermined parts of the gripping means 501. In the embodiment of the present invention, a case where the gripping means 501 is a human hand will be described as an example.

  First, the configuration of the first exemplary embodiment of the present invention will be described.

  FIG. 2 is a block diagram showing the configuration of the recognition apparatus 100 in the first embodiment of the present invention. The recognition apparatus 100 is an embodiment of the information processing apparatus of the present invention.

  Referring to FIG. 2, the recognition apparatus 100 according to the first embodiment of the present invention includes an image acquisition unit 110, a gripping means detection unit 111, a gripping feature generation unit 112, an object likelihood calculation unit 113, and a gripping feature storage unit 114. And a grasping object recognition unit 116.

  The image acquisition unit 110 acquires an image of the gripping unit 501 that is gripping the gripping target 502. The image acquisition unit 110 may be an RGB camera that can acquire three-color information of red, blue, and green. The image acquisition unit 110 may be a camera that can acquire other wavelength signal information, such as a far-infrared camera or a multispectral camera. Further, the image acquisition unit 110 may be a distance camera (distance sensor) that can store distance information from the camera to an object in each pixel in the image. Furthermore, the image acquisition unit 110 may be a camera that can simultaneously acquire one or more of the above-described three-color information, other wavelength signal information, and distance information.

  The gripping means detection unit 111 detects the position or position and direction of each of a plurality of predetermined parts of the gripping means 501 in the image acquired by the image acquisition unit 110. In the embodiment of the present invention, the finger of the gripping means 501 is used as the predetermined part. In addition, as the position of the predetermined part (finger position), a position designated on the finger such as a fingertip or a joint of each finger is used.

  The gripping feature generation unit 112 generates a gripping feature amount indicating a positional relationship between a plurality of predetermined parts (a positional relationship between fingers) as a gripping feature amount representing a gripping feature by the gripping unit 501. In the embodiment of the present invention, the coordinate value of each finger position, the coordinate value and direction of each finger position, the distance between finger positions, and the like are used as the positional relationship between the positions of a plurality of predetermined parts.

  The grip feature storage unit 114 stores grip feature information 115. The gripping feature information 115 is information for calculating “object likelihood based on gripping feature amount” for the category of the object to be recognized. In the gripping feature information 115, information corresponding to the object likelihood calculation method is set as will be described later.

  The object likelihood calculation unit 113 uses the gripping feature amount generated by the gripping feature generation unit 112 and the gripping feature information 115 stored in the gripping feature storage unit 114 to set the gripping feature amount for each object category. Based on the object likelihood.

  The gripping target recognition unit 116 recognizes the category of the gripping target 502 using the object likelihood calculated by the object likelihood calculation unit 113.

  Note that the recognition apparatus 100 may be a computer that includes a CPU (Central Processing Unit) and a storage medium that stores a program, and that operates by control based on the program.

  FIG. 3 is a block diagram showing a configuration of the recognition apparatus 100 realized by a computer according to the first embodiment of the present invention.

  The recognition apparatus 100 includes a CPU 101, a storage device (storage medium) 102 such as a hard disk and a memory, a communication device 103 that communicates with other apparatuses, an input device 104 such as a mouse and a keyboard, and an output device 105 such as a display. Including.

  The CPU 101 executes a computer program for realizing the functions of the image acquisition unit 110, the gripping means detection unit 111, the gripping feature generation unit 112, the object likelihood calculation unit 113, and the gripping target recognition unit 116. The storage device 102 stores data of the gripping feature storage unit 114. The input device 104 acquires an image of the gripping unit 501 that is gripping the gripping target 502 from a user or the like. The output device 105 outputs the recognition result (the category of the object of the gripping target 502) to the user or the like. Further, the communication device 103 may acquire an image from another device or the like, and output a recognition result to the other device or the like.

  Next, the operation of the first exemplary embodiment of the present invention will be described.

  FIG. 4 is a flowchart showing the operation of the recognition apparatus 100 in the first embodiment of the present invention.

  First, the image acquisition unit 110 acquires an image of the gripping unit 501 that is gripping the gripping target 502 from a user or the like (step S101).

FIG. 5 is a diagram illustrating an example of an acquired image in the first embodiment of the present invention.
For example, the image acquisition unit 110 acquires an image as illustrated in FIG.

  The gripping means detection unit 111 detects the position of each finger of the gripping means 501 or the position and direction of each finger in the image acquired by the image acquisition unit 110 (step S102). Here, the grip means detection unit 111 detects the position and direction of the finger based on, for example, the color, shape, and arrangement of the finger. Further, the gripping means detection unit 111 may detect the position of the finger using a technique described in Non-Patent Document 2. Furthermore, the gripping means detection unit 111 may detect the position of the finger for a portion where the finger is present in the image, or by estimating the shape based on the detection result of the portion where the finger is present. You may estimate also about the part which does not exist. Further, the position of the finger may be specified by two-dimensional coordinates in the image, or may be specified by three-dimensional coordinates in the real space. In addition, the coordinate value may be an absolute coordinate value with a specific point as an origin, or a relative coordinate value from an arbitrary point. Further, the gripping means detection unit 111 may detect a plurality of positions for each finger.

The grip means detection unit 111 determines whether the number of fingers detected in the image is 2 or more (step S103).
In step S103, when the number of detected fingers is two or more (step S103 / Y), the gripping feature generation unit 112 determines the position of each finger or the position and direction of each finger. Then, a gripping feature amount is generated (step S105). Here, the gripping feature generation unit 112 generates a gripping feature amount, for example, according to any of the following gripping feature amount generation methods 1 to 3.

(Grip feature generation method 1)
FIG. 6 is a diagram illustrating an example of a method for generating a gripping feature amount according to the first embodiment of the present invention. Here, it is assumed that the position of n fingers is detected by the gripping means detection unit 111. In this case, a line segment connecting the positions of the n fingers is obtained as shown in FIG. The number N _l of line segments is calculated by the equation (1).

The length l _i (i = 1,..., N _l ) of each of the N _l line segments represents the detected j-th (j = 1,..., N) finger position as P _j = (x _j , Y _j , z _j ), it is calculated by equation (2).

Here, if the lengths l _i of the line segments are arranged in descending order, the vectorized gripping feature value V _A can be defined as shown in Equation 3.

Further, if the maximum value of the elements of the gripping feature amount V _A is mx, the gripping feature amount V ′ _A as shown in Equation 4 can be defined.

  The gripping feature generation unit 112 generates gripping feature amounts such as Equation 3 or Equation 4 based on the length of the line segment in FIG.

For example, the object likelihood calculating unit 113 generates a gripping feature value V _A = (1.5, 1.0, 0.3,...) From the image of FIG.

(Grip feature generation method 2)
FIG. 7 is a diagram illustrating another example of a method for generating a gripping feature value according to the first embodiment of this invention. Here, it is assumed that the position of n fingers is detected by the gripping means detection unit 111. In this case, as shown in FIG. 7, n−1 line segments connecting the detected positions of the fingers and the positions of the other fingers are obtained. A set of lengths of these segments _{Gr i = (l i, 1} , l i, 2, ..., l i, n-1) to. Also _, Gr ′ _i = (l ′ _{i, 1} , l ′ _{i, 2} ,..., L ′ _{i, n−1} ) is obtained by rearranging the elements of Gr _i in descending order of size (length). . Furthermore, 'a _i, the first element l' Gr those rearranged in descending order of _{i, 1 (Gr "0,} Gr" 1, ..., Gr "n) and then, _{Gr" i} elements Gr _"i = the ( l ″ _{i, 1} , l ″ _{i, 2} ,..., l ″ _{i, n−1} ). By arranging the elements of each Gr ″ _i , a gripping feature value V _B as shown in Equation 5 can be defined.

Further, using the maximum value mx of the elements of the gripping feature value V _B , a gripping feature value V ′ _B as shown in Equation 6 can be defined.

  The gripping feature generation unit 112 generates gripping feature amounts such as Equation 5 or Equation 6 based on the length of the line segment in FIG.

(Grip feature generation method 3)
It is assumed that the gripping means detection unit 111 can identify the finger among the thumb, the index finger, the middle finger, the ring finger, and the little finger in addition to the position and direction of each finger. In this case, for example, in the order of thumb, index finger, middle finger, ring finger, and little finger, the coordinate values P _j = (x _j , y _j , z _j ), (j = 1,..., N) and The direction D _j = (a _j , b _j , c _j ) is obtained. Here, for example, the direction of the normal of the fingertip position obtained by the distance sensor may be used as the direction of each finger, or the direction from the first joint to the fingertip may be used. Also, other orders may be used as the order of the fingers (thumb, index finger,...).

These coordinates, the direction, the coordinates on a predetermined coordinate system Z, by expressing in the direction, the gripping feature amount such as equation (7) V _C, or gripping feature quantity V _D as equation (8) is defined it can.

  As the coordinate system Z, for example, a coordinate system having a coordinate axis parallel to a predetermined direction of one finger may be used.

  The gripping feature generation unit 112 generates gripping feature amounts of Formula 7 or Formula 8 based on the position and direction of each finger.

  Next, the object likelihood calculating unit 113 calculates an object likelihood based on the gripping feature amount for each category of the object (step S106). Here, the object likelihood calculating unit 113 calculates the object likelihood according to any of the following object likelihood calculating methods 1 to 3, for example.

(Object likelihood calculation method 1)
FIG. 8 is a diagram illustrating an example of the gripping feature information 115 according to the first embodiment of this invention. In the gripping feature information 115 in FIG. 8, for each object category, an instance indicating the gripping feature amount when the object is gripped is registered. Here, a pair of each feature and a gripping feature amount registered for the category is referred to as an instance. Multiple instances may be registered for one category. Here, the category of the object is C _i (i = 1,..., M, M is the number of categories), and the gripping feature amounts corresponding to the category C _i are V _ij (j = 1,..., R (i), R If (i) is the number of gripping feature values for category C _i ), the instance is represented as (C _i , V _ij ).

The object likelihood calculating unit 113 calculates the distance between the gripping feature amount generated by the gripping feature generating unit 112 and the gripping feature amount V _{ij of} each instance registered in the gripping feature information 115. The distance may be Euclidean distance, Manhattan distance, or other distance measure. The object likelihood calculating unit 113 extracts an instance whose calculated distance is equal to or less than a predetermined threshold from the instances registered in the gripping feature information 115. Then, the object likelihood calculating unit 113 calculates the object likelihood L _i of each category C _i using, for example, Equation 9 using the extracted instance.

Here, k is the number of extracted instances, and Nc _i is the number of instances (C _i , V _ij ) corresponding to the category C _i out of k instances.

  FIG. 9 is a diagram illustrating an instance extraction example according to the first embodiment of this invention. For example, the object likelihood calculating unit 113, as shown in FIG. 9, among the instances registered in the gripping feature information 115 of FIG. 8, the gripping feature amount V = (1.5, 1.0) generated by the gripping feature generating unit 112. , 0.3,...), 10 instances whose distances are below the threshold are extracted.

  FIG. 10 is a diagram illustrating an example of the calculation result of the object likelihood in the first embodiment of the present invention. For example, the object likelihood calculating unit 113 calculates the object likelihood as shown in FIG. 10 based on the number of extracted instances.

  Note that the object likelihood calculating unit 113 may extract a predetermined number of instances from a smaller distance instead of extracting instances whose distance is equal to or smaller than a predetermined threshold.

(Object likelihood calculation method 2)
FIG. 11 is a diagram illustrating another example of the gripping feature information 115 in the first exemplary embodiment of the present invention. In the gripping feature information 115 in FIG. 11, the object likelihood at each point in the gripping feature amount space is registered for each object category. In this case, the object likelihood at each point is calculated in advance by a nearest neighbor density estimation method, a kernel density estimation method, or the like.

Object likelihood calculating unit 113 refers to the grip characteristic information 115, by obtaining the object likelihoods corresponding to the grip characteristic quantity generated by the gripping feature generation unit 112, the object likelihood Li for each category C _i calculate.

(Object likelihood calculation method 3)
In the gripping feature information 115, for example, a learning result obtained by machine learning such as Support Vector Machine or Random Forest may be registered.

For example, when Support Vector Machine is used, an identification boundary surface in the gripping feature space is registered as a learning result. In this case, the object likelihood calculating unit 113 calculates the object likelihood L _i of each category C _{i based on} a function of the distance between the gripping feature amount generated by the gripping feature generating unit 112 and the identification boundary surface.

When Random Forest is used, the learning boundary and the weight of the classifier are registered as learning results. In this case, the object likelihood calculating unit 113 calculates the object likelihood L _i of each category C _i using a function of the weighted vote result.

  Next, the gripping target recognition unit 116 recognizes the category of the gripping target 502 using the calculated object likelihood (step S107). Here, the gripping target recognizing unit 116 identifies the category idx of the gripping target 502 according to, for example, Formula 10.

Note that L _th is a preset threshold value for the maximum value of the object likelihood L _i .

For example, the gripping target recognition unit 116, based on the object likelihood calculation results of FIG. 10, the category of the gripping target 502, object likelihood is identified as Category C ₁ is the largest.

  The gripping target recognition unit 116 sets the category index of the gripping target 502 or “not applicable” in the recognition result according to the result of step S106 (step S108).

For example, the gripping target recognition unit 116 sets “category C ₁ ” as the recognition result.

  In step S103, when the number of detected fingers is less than two (step S103 / N), “not applicable” is set as the recognition result (step S104).

  Finally, the gripping target recognition unit 116 outputs the recognition result to the user or the like (step S109).

  Thus, the operation of the first exemplary embodiment of the present invention is completed.

  In the above description, the gripping target recognizing unit 116 identifies the category of the object based on the object likelihood for each category calculated by the object likelihood calculating unit 113. However, the present invention is not limited to this, and the grasping target recognition unit 116 specifies the object shape based on the object likelihood with respect to the object shape, and further acquires the category of the object associated with each object shape. A category may be specified.

  Further, in the embodiment of the present invention, the case where the gripping unit 501 is a human hand has been described as an example. However, the present invention is not limited to this. A hand such as a robot may be used.

  Further, as long as an object can be gripped by moving a plurality of movable parts, the gripping means 501 may have a shape other than a hand. In this case, for example, each movable part of the gripping means 501 is used as the predetermined part, and a specified position such as the tip, center, or joint of each movable part is detected as the position of the predetermined part (position of the movable part). The Further, as the positional relationship between the positions of a plurality of predetermined parts (positional relationship between the movable parts), the coordinate value of the position of each movable part, the distance between the positions of the movable parts, and the like are used.

  Next, the basic configuration of the embodiment of the present invention will be described. FIG. 1 is a block diagram showing a basic configuration of an embodiment of the present invention.

  Referring to FIG. 1, the recognition apparatus 100 includes an image acquisition unit 110, a gripping feature generation unit 112, and a gripping target recognition unit 116. The image acquisition unit 110 acquires an image of the gripping unit 501 that is gripping the gripping target 502. The grip feature generation unit 112 generates a grip feature indicating the positional relationship between a plurality of predetermined parts of the grip means 501 in the image. The grip target recognition unit 116 recognizes the grip target 502 based on the grip feature.

  According to the embodiment of the present invention, the gripping target 502 can be recognized even when the gripping target 502 does not exist in the image of the gripping means 501 gripping the gripping target 502. The reason is that the gripping target recognition unit 116 recognizes the gripping target 502 based on gripping characteristics indicating the positional relationship between a plurality of predetermined parts of the gripping means 501 in the image. Thereby, even if the gripping object 502 is shielded in the image of the gripping means 501 gripping the gripping target 502, the gripping target 502 can be recognized.

<Second Embodiment>
Next, a second embodiment of the present invention will be described.

  In the second embodiment of the present invention, in addition to the gripping feature amount, an object feature amount is generated from the object region of the gripping target 502 on the image, and the gripping object is detected using the gripping feature amount and the object feature amount. recognize. In the second embodiment of the present invention, the object likelihood based on the gripping feature amount is referred to as a first object likelihood, and the object likelihood based on the object feature amount is referred to as a second object likelihood.

  First, the configuration of the second exemplary embodiment of the present invention will be described.

  FIG. 12 is a block diagram showing the configuration of the recognition apparatus 200 in the second embodiment of the present invention.

  Referring to FIG. 12, a recognition apparatus 200 according to the second embodiment of the present invention includes an image acquisition unit 210, a gripping means detection unit 211, a gripping feature generation unit 212, a first object likelihood calculation unit 213, and a gripping feature storage. Part 214. The recognition apparatus 200 further includes a gripping target detection unit 221, an object feature generation unit 222, a second object likelihood calculation unit 223, an object feature storage unit 224, an integrated likelihood calculation unit 230, and a gripping target recognition unit 240. .

  The image acquisition unit 210, the gripping unit detection unit 211, and the gripping feature generation unit 212 are respectively the image acquisition unit 110, the gripping unit detection unit 111, and the gripping feature generation unit in the first exemplary embodiment of the present invention. 112. The gripping feature storage unit 214 stores gripping feature information 215 similar to the gripping feature information 115 (information for calculating “object likelihood based on gripping feature amount (first object likelihood)”). Similar to the object likelihood calculating unit 113, the first object likelihood calculating unit 213 calculates a first object likelihood for each category of the object.

  The grip target detection unit 221 detects the object area of the grip target 502 in the image acquired by the image acquisition unit 110. Here, the gripping target detection unit 221 may detect the object region using, for example, a background subtraction method that detects a moving object when the background is fixed. Further, the gripping target detection unit 221 may detect a pixel whose distance is smaller (closer) than a predetermined threshold as an object region. Further, the gripping target detection unit 221 may detect the object region by regarding peripheral pixels similar to a certain gripping target candidate region (for example, the center of the image) as a region belonging to the gripping target.

  The object feature generation unit 222 generates an object feature amount indicating a feature related to a color, a pattern, or the like of the gripping target 502 as an object feature amount representing the feature of the object of the gripping target 502. Here, for example, color appearance frequency or color arrangement may be used as the object feature. Further, as the object feature, a partial light / dark pattern of the luminance value of the image, a change direction of the luminance value, and a response intensity to the filter may be used.

  The object feature storage unit 224 stores object feature information 225. The object feature information 225 is information for calculating the “object likelihood based on the object feature amount (second object likelihood)” for the category of the object to be recognized.

  The second object likelihood calculation unit 223 uses the object feature amount generated by the object feature generation unit 222 and the object feature information 225 stored in the object feature storage unit 224 for each object category. Calculate object likelihood.

  The integrated likelihood calculating unit 230 calculates the integrated likelihood using the first object likelihood and the second object likelihood.

  The gripping target recognition unit 240 recognizes the category of the gripping target 502 using the integrated likelihood calculated by the integrated likelihood calculation unit 230.

  Next, the operation of the second exemplary embodiment of the present invention will be described.

  FIG. 13 is a flowchart showing the operation of the recognition apparatus 200 in the second embodiment of the present invention.

  First, the image acquisition unit 210 acquires an image of the gripping means 501 that is gripping the gripping target 502 (step S201). In this case, it is assumed that the image includes at least a part of the object region of the gripping target 502.

  The gripping means detection unit 211 detects the position of each finger of the gripping means 501 or the position and direction of each finger in the image acquired by the image acquisition unit 210 (step S202).

  The grip target detection unit 221 detects the object area of the grip target 502 in the image acquired by the image acquisition unit 210 (step S203).

  The gripping means detection unit 211 determines whether the number of fingers detected in the image is two or more (step S204).

  In step S204, when the number of detected fingers is two or more (step S204 / Y), the gripping feature generation unit 212 determines the position of each finger or the position and direction of each finger. Then, a gripping feature amount is generated (step S206).

  The first object likelihood calculating unit 213 calculates the first object likelihood for each object category based on the generated gripping feature amount (step S207).

  FIG. 15 is a diagram illustrating an example of the calculation result of the integrated likelihood in the second exemplary embodiment of the present invention.

  For example, the first object likelihood calculation unit 213 extracts an instance whose distance from the gripping feature amount generated by the gripping feature generation unit 112 is equal to or less than a threshold from the instances registered in the gripping feature information 115 in FIG. To do. Then, the first object likelihood calculating unit 213 calculates the first object likelihood as shown in FIG. 15 based on the number of extracted instances.

  In step S204, when the number of detected fingers is less than two (step S204 / N), 1 is set as the first object likelihood for all categories (step S205).

  Next, the object feature generation unit 222 generates an object feature amount from the detected object region of the gripping target 502 (step S208).

  The second object likelihood calculating unit 223 calculates the second object likelihood for each category of the object based on the generated object feature amount (step S209). Here, the second object likelihood calculating unit 223 calculates the second object likelihood by, for example, the same method as the object likelihood (first object likelihood) calculation method based on the gripping feature amount.

  FIG. 14 is a diagram illustrating an example of the object feature information 225 according to the second embodiment of the present invention. In the object feature information 225 in FIG. 14, an instance indicating the object feature amount of the object is registered for each object category. Here, the object feature amount represents, for example, an object feature such as an object color or a pattern.

  For example, the second object likelihood calculating unit 223 extracts an instance whose distance from the object feature amount generated by the object feature generating unit 222 is equal to or less than a threshold from the instances registered in the object feature information 225 of FIG. To do. Then, the second object likelihood calculating unit 223 calculates the second object likelihood as shown in FIG. 15 based on the number of extracted instances.

Next, the integrated likelihood calculating unit 230 calculates an integrated likelihood using the first object likelihood and the second object likelihood (step S210). The integrated likelihood calculating unit 230 calculates the integrated likelihood L _comb (i) of each category C _i using, for example, Equation 11 or Equation 12.

Here, L _A (i) and L _B (i) are the first object likelihood and the second object likelihood of category C _i , respectively.

  For example, the integrated likelihood calculating unit 230 calculates the integrated likelihood as shown in FIG.

  Next, the gripping target recognition unit 240 recognizes the category of the gripping target 502 using the calculated integrated likelihood (step S211). Here, the gripping target recognition unit 240 identifies the category idx of the gripping target 502 according to, for example, Equation 13.

Here, L _{th_comb} is a threshold for a maximum value of the integrated likelihood L _comb (i) set in advance.

For example, the gripping target recognition unit 240, based on the integrated likelihood calculation results of FIG. 15, the category of the gripping target 502, integrated likelihood is specified as category C ₃ is the maximum.

  The gripping target recognition unit 116 sets the category index of the gripping target 502 or “not applicable” in the recognition result according to the result of step S211 (step S212) and outputs the result (step S213).

  Thus, the operation of the second exemplary embodiment of the present invention is completed.

  According to the second embodiment of the present invention, the recognition accuracy of the grasped object 502 can be improved as compared with the first embodiment of the present invention. The reason is that the gripping target recognition unit 240 determines the gripping target 502 based on the integrated likelihood calculated using the first object likelihood based on the gripping feature amount and the second object likelihood based on the object feature amount. This is for recognition. Thereby, for example, in the image of the gripping means 501 that is gripping the gripping target 502, most of the gripping target 502 is shielded, but the recognition accuracy of the gripping target 502 is improved when a part of the gripping target 502 exists. it can.

<Third Embodiment>
Next, a third embodiment of the present invention will be described.

  In the third embodiment of the present invention, a gripping feature amount is generated for a part in contact with the gripping target 502 (finger in contact with the gripping target 502) among a plurality of predetermined parts of the gripping means 501. .

  First, the configuration of the third exemplary embodiment of the present invention will be described.

  FIG. 16 is a block diagram showing the configuration of the recognition apparatus 200 in the third embodiment of the present invention.

  Referring to FIG. 16, a recognition device 200 according to the third embodiment of the present invention includes a contact detection unit 250 in addition to the components of the recognition device 200 according to the second embodiment of the present invention.

  The contact detection unit 250 detects the gripping of the detected finger based on the object area of the gripping target 502 detected by the gripping target detection unit 221 and the position of each finger detected by the gripping means detection unit 111. The finger (contact finger) in contact with the target 502 is specified. For example, when the distance between the coordinate value indicating the position of the finger of interest and the object region closest to the coordinate value is less than a predetermined threshold, the contact detection unit 250 determines that the finger is a contact finger.

  The gripping feature generating unit 212 generates a gripping feature amount that represents the positional relationship between the predetermined parts related to the contact finger.

  Next, the operation of the third exemplary embodiment of the present invention will be described.

  FIG. 17 is a flowchart showing the operation of the recognition apparatus 200 in the third embodiment of the present invention.

  First, the image acquisition unit 210 acquires an image of the gripping means 501 that is gripping the gripping target 502 (step S301).

  The grip means detection unit 211 detects the position of each finger or the position and direction of each finger in the image acquired by the image acquisition unit 210 (step S302).

  The grip target detection unit 221 detects the object region of the grip target 502 in the image acquired by the image acquisition unit 210 (step S303).

  The gripping means detection unit 211 determines whether the number of fingers detected in the image is two or more (step S304).

  In step S304, when the number of detected fingers is two or more (step S304 / Y), the contact detection unit 250 identifies a contact finger among the detected fingers (step S306).

  The contact detection unit 250 determines whether the number of contact fingers is two or more (step S307).

  In step S307, when the number of contact fingers is two or more (step S307 / Y), the gripping feature generation unit 212 determines the position of each contact finger or the position and direction of each contact finger. Then, a gripping feature amount representing the positional relationship between the contact fingers is generated (step S308).

  The first object likelihood calculating unit 213 calculates the first object likelihood for each object category based on the generated gripping feature amount (step 309).

  If the number of fingers detected in step S304 is less than 2 (step S304 / N), or if the number of contact fingers is less than 2 in step S307 (step S307 / N), all categories The first object likelihood for is set to 1.

  Thereafter, the generation of the object feature amount, the calculation of the second object likelihood, the calculation of the integrated likelihood, and the recognition of the category of the gripping target 502 (steps S310 to S315) are performed in the second embodiment (step S208 to S213) are performed.

  Thus, the operation of the third embodiment of the present invention is completed.

  According to the third embodiment of the present invention, the recognition accuracy of the grasped object 502 can be improved as compared with the first embodiment of the present invention. The reason is that the gripping feature generation unit 212 indicates the positional relationship between the parts that are in contact with the gripping target 502 (the finger that is in contact with the gripping target 502) among the plurality of predetermined parts of the gripping means 501. This is to produce a quantity. As a result, it is possible to exclude information relating to a part that is not in contact with the gripping target 502 (a finger that is not in contact) from the gripping feature amount, and to influence the position of the part that does not contribute to gripping by the gripping means 501 The category of the gripping target 502 can be specified without receiving it.

<Fourth embodiment>
Next, a fourth embodiment of the present invention will be described.

  In the fourth embodiment of the present invention, a gripping feature indicating a temporal change in the positional relationship between a plurality of predetermined parts of the gripping means 501 is generated.

  First, the configuration of the fourth embodiment of the present invention will be described.

  FIG. 18 is a block diagram showing the configuration of the recognition device 200 in the fourth embodiment of the present invention.

  Referring to FIG. 18, in the configuration of the fourth embodiment of the present invention, the gripping feature generation unit 212 is replaced with a gripping feature generation unit 260 in the second embodiment of the present invention.

  The gripping feature generation unit 260 generates a gripping feature amount including a motion feature that indicates a temporal change in the positional relationship between a plurality of predetermined parts of the gripping unit 501. The gripping feature generation unit 260 includes a frame feature generation unit 261, a frame feature storage unit 262, and a motion feature extraction unit 263.

  The frame feature generation unit 261 generates a grip feature amount for each frame of the image by the same method as the grip feature generation unit 212.

  The frame feature storage unit 262 stores the gripping feature amount for each frame generated by the frame feature generation unit 261 for a predetermined number of frames.

  The motion feature extraction unit 263 extracts a motion feature based on the difference between grip feature amounts for each frame, and generates a grip feature amount including the motion feature.

  Next, the operation of the fourth exemplary embodiment of the present invention will be described.

  FIG. 19 is a flowchart showing the operation of the recognition apparatus 200 in the fourth embodiment of the present invention.

  First, the frame feature generation unit 261 of the gripping feature generation unit 260 sets 1 to a variable t indicating a frame (t = 1,..., Nf, where Nf is the number of frames for generating a gripping feature amount including a motion feature). Set (step S401).

  The image acquisition unit 210 acquires one frame of the image of the gripping means 501 that is gripping the gripping target 502 (step S402).

  The grip means detection unit 211 detects the position of each finger or the position and direction of each finger in the frame (target frame) acquired by the image acquisition unit 210 (step S403).

  The grip target detection unit 221 detects the object region of the grip target 502 in the target frame acquired by the image acquisition unit 210 (step S404).

  The grip means detection unit 211 determines whether the number of fingers detected in the target frame is two or more (step S405).

In step S405, when the number of detected fingers is two or more (step S405 / Y), the frame feature generation unit 261 determines the position of each finger or the position and direction of each finger. Then, a gripping feature amount V (t) at the target frame t is generated (step S407). Here, the frame feature generation unit 261 uses the gripping feature amounts V _A , V _{B, and} V shown in the gripping feature amount generation method according to the first embodiment of the present invention as the gripping feature amount V (t), for example. _C, and generate either of _{V D.}

  The frame feature generation unit 261 determines whether the variable t is greater than or equal to Nf (step S408).

  If the variable t is less than Nf in step S408 (step S408 / N), the frame feature generation unit 261 stores the generated gripping feature amount V (t) in the frame feature storage unit 262, and adds 1 to the variable t. (Step S409). Then, the processing from step S402 is repeated.

  On the other hand, if the variable t is greater than or equal to Nf in step S408 (step S408 / Y), the motion feature extraction unit 263 calculates the difference between the gripping feature amounts for each frame stored in the frame feature storage unit 262 ( Step S410).

  The motion feature extraction unit 263 generates a gripping feature amount including a motion feature based on the calculated difference (step S411).

Here, the motion feature extraction unit 263 generates the gripping feature amount V _mov including the motion feature using, for example, Equation 14.

FIG. 20 is a diagram illustrating a calculation example of the gripping feature amount including the motion feature according to the fourth embodiment of the present invention. FIG. 20 shows an example when Nf is 3. For example, the motion feature extraction unit 263 generates a gripping feature amount V _mov including a motion feature as illustrated in FIG. 20 based on the gripping feature amount V (t) at t = 1, 2, and 3.

  Next, the first object likelihood calculating unit 213 calculates the first object likelihood for each category of the object based on the gripping feature amount including the generated motion feature (step S412).

  In step S405, when the number of detected fingers is less than 2 (step S405 / N), 1 is set as the first object likelihood for all categories (step S406).

  Next, the object feature generation unit 222 generates an object feature amount from the detected object region of the gripping target 502 (step S413). Here, the object feature generation unit 222 generates an object feature amount from the object region of the gripping target 502 detected in the first frame or the Nf-th frame among the Nf frames. Further, the object feature generation unit 222 may calculate the average value of the object feature amounts generated from the object region of the gripping target 502 detected in each of the Nf frames.

  Thereafter, the calculation of the second object likelihood, the calculation of the integrated likelihood, and the recognition of the category of the gripping target 502 (steps S414 to S418) are the same as in the second embodiment (steps S209 to S213) of the present invention. To be done.

  Thus, the operation of the fourth exemplary embodiment of the present invention is completed.

  According to the fourth embodiment of the present invention, when there is a temporal change in gripping by the gripping means 501, the recognition accuracy of the gripping target 502 can be improved. The reason is that the gripping feature generation unit 260 generates a gripping feature amount including a motion feature that indicates a temporal change in the positional relationship between a plurality of predetermined parts of the gripping means 501. Thereby, for example, a gripping target 502 whose shape changes with time during gripping by the gripping means 501 such as a soft object can be distinguished from a gripping target 502 whose shape does not change with time, such as a hard object. In addition, even when a predetermined part (finger) of the gripping unit 501 moves on the grip target 502 such as a smartphone operation, the recognition accuracy of the grip target 502 can be improved.

  While the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  For example, as described in the third embodiment of the present invention, the generation of the gripping feature amount for the part in contact with the gripping target 502 among the plurality of predetermined parts of the gripping unit 501 is performed by the contact with the gripping target 502. May be applied to the first embodiment of the present invention.

  In addition, the generation of the gripping feature amount including the movement feature described in the fourth embodiment of the present invention may be applied to the first embodiment or the second embodiment of the present invention.

100 recognition device 101 CPU
DESCRIPTION OF SYMBOLS 102 Storage device 103 Communication device 104 Input device 105 Output device 110 Image acquisition part 111 Grasping means detection part 112 Grasping feature production | generation part 113 Object likelihood calculation part 114 Grasping feature memory | storage part 115 Grasping feature information 116 Grasping object recognition part 200 Recognition apparatus 210 Image acquisition unit 211 Grasping means detection unit 212 Grasping feature generation unit 213 First object likelihood calculation unit 214 Grasping feature storage unit 215 Grasping feature information 221 Grasping target detection unit 222 Object feature generation unit 223 Second object likelihood calculation unit 224 Object Feature storage unit 225 Object feature information 230 Integrated likelihood calculation unit 240 Grasping object recognition unit 250 Touch detection unit 260 Grasping feature generation unit 261 Frame feature generation unit 262 Frame feature storage unit 263 Motion feature extraction unit 501 Grasping means 502 Grasping object

Claims (10)

An image acquisition means for acquiring an image of the grip means holding the grip target;
A gripping feature generating means for generating a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping means in the image;
A gripping object recognition means for recognizing the gripping object based on the gripping features;
An information processing apparatus comprising: Furthermore, an object feature generating means for generating an object feature indicating at least one of the color and pattern of the grip target in the image,
The information processing apparatus according to claim 1, wherein the gripping target recognition unit recognizes the gripping target based on the gripping feature and the object feature. The gripping feature generating means generates the gripping feature indicating a positional relationship between parts in contact with the gripping target among the plurality of predetermined parts.
The information processing apparatus according to claim 1 or 2. The gripping feature generating means generates a gripping feature indicating a temporal change in a positional relationship between the plurality of predetermined parts;
The information processing apparatus according to claim 1. The gripping feature indicates a positional relationship between the plurality of predetermined portions by a coordinate value indicating the position of each of the plurality of predetermined portions.
The information processing apparatus according to claim 1. The gripping feature indicates a positional relationship between the plurality of predetermined portions by a coordinate value indicating the position of each of the plurality of predetermined portions and a direction of the predetermined portion.
The information processing apparatus according to claim 5. The gripping feature indicates a positional relationship between the plurality of predetermined portions by a distance between the plurality of predetermined portions.
The information processing apparatus according to claim 1. The predetermined part is a finger of the gripping means.
The information processing apparatus according to claim 1. Obtain an image of the gripping means that grips the gripping target,
Generating a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping means in the image, and recognizing the gripping object based on the gripping feature;
Information processing method. On the computer,
Obtain an image of the gripping means that grips the gripping target,
Generating a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping means in the image, and recognizing the gripping object based on the gripping feature;
A program that executes processing.

Images