JP6739896B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program for recognizing an object held by a holding means.

Using an image obtained by capturing a state in which a person or the like (hereinafter, also referred to as a gripper) grips an object (hereinafter, also referred to as a gripping target) with a finger or the like (hereinafter, also described as gripping means), A technique for recognizing a grip target is known.

For example, in the technique disclosed in Patent Document 1, when matching a partial area of an object in an image with an article shape in a database, matching is performed with the size of the space calculated from the gripping posture of the hand. Limit the target items.

In addition, in the technique disclosed in Non-Patent Document 1, the feature of the pair of shapes of the hand region and the object region in the image is stored for each gripping pattern, and the gripping pattern is recognized using the feature.

As a related technique, Patent Document 2 discloses a technique of identifying a virtual key to be operated by using the position of a finger detected from an image. Patent Document 3 discloses a technique for recognizing an object included in an image using a color histogram. Patent Document 4 discloses a technique for identifying an object based on the characteristics of the area of the object in the moving image and the change in the outer peripheral length. Non-Patent Document 2 discloses a technique for detecting a finger area by searching for a designated color in an image taken by wearing gloves (gloves) having a designated color for each finger.

JP, 2010-244413, A JP, 2013-143082, A JP, 2012-150552, A JP, 2010-244440, A

Hiromasa Kasahara, 3 others, “Reconstruction of missing pixels and object recognition based on learning of grip pattern image”, Image Recognition and Understanding Symposium (MIRU2008), July 2008, p.623-628 Ken Watanabe and 3 others, "Recognition of finger characters using color gloves", IEICE Transactions, D-II, 1997, vol. J80-D-2, no. 10, p. 2713-2722

As described above, when the gripping unit recognizes the gripping target in the image of the state in which the gripping target grips the gripping target, for example, the gripping target is covered by the finger or the palm, and the gripping target does not exist in the image. There are cases. However, in the techniques described in Patent Document 1 and Non-Patent Document, since the gripping target is recognized using a partial image of the gripping target, when the gripping target does not exist in the image in this way, , The gripping target cannot be recognized.

The present invention solves the above-mentioned problems, and an information processing apparatus, an information processing method, and an information processing method capable of recognizing a grip target even when the grip target does not exist in an image of a grip unit that grips the grip target. , To provide the program.

An information processing apparatus of the present invention generates an image acquisition unit that acquires an image of a gripping unit that grips a grip target, and a gripping characteristic that indicates a positional relationship between a plurality of predetermined parts of the gripping unit in the image, A gripping feature generation unit and a gripping target recognition unit that recognizes the gripping target based on the gripping feature.

An information processing method of the present invention obtains an image of a gripping means gripping a gripping target, generates a gripping characteristic indicating a positional relationship between a plurality of predetermined parts of the gripping means in the image, and determines the gripping characteristic. First, the grip target is recognized.

The program of the present invention causes a computer to acquire an image of a gripping means gripping a gripping target, generate a gripping characteristic indicating a positional relationship between a plurality of predetermined parts of the gripping means in the image, Based on the above, processing for recognizing the grip target is executed.

The effect of the present invention is that the gripping target can be recognized even when the gripping target does not exist in the image of the gripping means that grips the gripping target.

It is a block diagram showing a basic composition of an embodiment of the invention. It is a block diagram which shows the structure of the recognition apparatus 100 in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 100 implement|achieved by the computer in the 1st Embodiment of this invention. 5 is a flowchart showing an operation of the recognition device 100 according to the first embodiment of the present invention. It is a figure which shows the example of the acquired image in the 1st Embodiment of this invention. It is a figure which shows the example of the generation method of the grasping feature-value in the 1st Embodiment of this invention. It is a figure which shows the other example of the production|generation method of the holding feature-value in the 1st Embodiment of this invention. It is a figure which shows the example of the holding|grip characteristic information 115 in the 1st Embodiment of this invention. It is a figure which shows the example of extraction of the instance in the 1st Embodiment of this invention. It is a figure which shows the example of the calculation result of object likelihood in the 1st Embodiment of this invention. It is a figure which shows the other example of the holding|grip characteristic information 115 in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 200 in the 2nd Embodiment of this invention. It is a flowchart which shows operation|movement of the recognition apparatus 200 in the 2nd Embodiment of this invention. It is a figure which shows the example of the object feature information 225 in the 2nd Embodiment of this invention. It is a figure which shows the example of the calculation result of integrated likelihood in the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 200 in the 3rd Embodiment of this invention. It is a flowchart which shows operation|movement of the recognition apparatus 200 in the 3rd Embodiment of this invention. It is a block diagram which shows the structure of the recognition apparatus 200 in the 4th Embodiment of this invention. It is a flowchart which shows operation|movement of the recognition apparatus 200 in the 4th Embodiment of this invention. It is a figure in the 4th Embodiment of this invention which shows the example of calculation of the holding|grip characteristic amount containing a motion characteristic.

<First Embodiment>
First, the first embodiment of the present invention will be described.

In the first embodiment of the present invention, the grip target 502 is recognized based on the positional relationship between a plurality of predetermined parts of the grip means 501. In the embodiment of the present invention, the case where the gripping means 501 is a human hand will be described as an example.

First, the configuration of the first embodiment of the present invention will be described.

FIG. 2 is a block diagram showing the configuration of the recognition device 100 according to the first embodiment of the present invention. The recognition device 100 is an embodiment of the information processing device of the present invention.

Referring to FIG. 2, the recognition device 100 according to the first embodiment of the present invention includes an image acquisition unit 110, a gripping unit detection unit 111, a gripping feature generation unit 112, an object likelihood calculation unit 113, and a gripping feature storage unit 114. , And a gripping target recognition unit 116.

The image acquisition unit 110 acquires an image of the grip means 501 that grips the grip target 502. The image acquisition unit 110 may be an RGB camera that can acquire three-color information of red, blue, and green. The image acquisition unit 110 may be a camera that can acquire other wavelength signal information, such as a far infrared camera or a multispectral camera. Further, the image acquisition unit 110 may be a distance camera (distance sensor) that can store the distance information from the camera to the object in each pixel in the image. Further, the image acquisition unit 110 may be a camera capable of simultaneously acquiring one or more of the above-mentioned three-color information, other wavelength signal information, and distance information.

The gripping unit detection unit 111 detects the position, or position and direction, of each of the plurality of predetermined portions of the gripping unit 501 in the image acquired by the image acquisition unit 110. In the embodiment of the present invention, the finger of the grip means 501 is used as the predetermined portion. Further, as the position of the predetermined part (the position of the finger), the position designated on the finger such as the fingertip or joint of each finger is used.

The gripping characteristic generation unit 112 generates a gripping characteristic amount indicating a positional relationship between a plurality of predetermined parts (positional relationship between fingers) as a gripping characteristic amount representing a gripping characteristic by the gripping unit 501. In the embodiment of the present invention, as the positional relationship between the positions of the plurality of predetermined parts, the coordinate value of the position of each finger, the coordinate value and direction of the position of each finger, the distance between the positions of the fingers, and the like are used.

The gripping characteristic storage unit 114 stores the gripping characteristic information 115. The gripping characteristic information 115 is information for calculating the “object likelihood based on the gripping characteristic amount” for the category of the object to be recognized. As described below, the gripping characteristic information 115 is set with information according to the method of calculating the object likelihood.

The object likelihood calculation unit 113 uses the gripping feature amount generated by the gripping feature generation unit 112 and the gripping feature information 115 stored in the gripping feature storage unit 114 to determine the gripping feature amount for each category of object. Based on the calculated object likelihood.

The grip target recognition unit 116 recognizes the category of the grip target 502 using the object likelihood calculated by the object likelihood calculation unit 113.

The recognition device 100 may be a computer that includes a CPU (Central Processing Unit) and a storage medium that stores a program and that operates under the control of the program.

FIG. 3 is a block diagram showing the configuration of the computer-implemented recognition device 100 according to the first embodiment of the present invention.

The recognition device 100 includes a CPU 101, a storage device (storage medium) 102 such as a hard disk and a memory, a communication device 103 that communicates with other devices, an input device 104 such as a mouse and a keyboard, and an output device 105 such as a display. Including.

The CPU 101 executes a computer program for realizing the functions of the image acquisition unit 110, the gripping unit detection unit 111, the gripping feature generation unit 112, the object likelihood calculation unit 113, and the gripped target recognition unit 116. The storage device 102 stores the data of the gripping characteristic storage unit 114. The input device 104 acquires an image of the grip unit 501 that holds the grip target 502 from a user or the like. The output device 105 outputs the recognition result (the category of the object of the grip target 502) to the user or the like. The communication device 103 may acquire an image from another device or the like and output the recognition result to the other device or the like.

Next, the operation of the first exemplary embodiment of the present invention will be described.

FIG. 4 is a flowchart showing the operation of the recognition device 100 according to the first embodiment of the present invention.

First, the image acquisition unit 110 acquires an image of the grip means 501 gripping the grip target 502 from a user or the like (step S101).

FIG. 5 is a diagram showing an example of an acquired image according to the first embodiment of the present invention.
For example, the image acquisition unit 110 acquires an image as shown in FIG.

The gripping unit detection unit 111 detects the position of each finger of the gripping unit 501 or the position and direction of each finger in the image acquired by the image acquisition unit 110 (step S102). Here, the gripping means detection unit 111 detects the position and direction of the finger, for example, based on the color, shape, and arrangement of the finger. In addition, the gripping unit detection unit 111 may detect the position of the finger by using the technique described in Non-Patent Document 2. Further, the gripping unit detection unit 111 may detect the position of the finger in a portion where the finger exists in the image, or the finger may be detected by shape estimation based on the detection result of the portion where the finger exists. It is also possible to estimate the part that does not exist. Further, the position of the finger may be designated by two-dimensional coordinates in the image, or may be designated by three-dimensional coordinates in the real space. Further, the coordinate value may be an absolute coordinate value having a certain point as an origin, or a relative coordinate value from an arbitrary point. Further, the gripping means detection unit 111 may detect a plurality of positions for each finger.

The gripping unit detection unit 111 determines whether the number of fingers detected in the image is two or more (step S103).
In step S103, when the number of detected fingers is two or more (step S103/Y), the gripping-feature generation unit 112 uses the detected position of each finger or the position and direction of each finger. , And generate gripping feature amounts (step S105). Here, the gripping feature generation unit 112 generates the gripping feature amount according to any of the following gripping feature amount generation methods 1 to 3, for example.

(Grip feature amount generation method 1)
FIG. 6 is a diagram showing an example of a method of generating a gripping characteristic amount according to the first embodiment of the present invention. Here, it is assumed that the gripping unit detection unit 111 detects the positions of n fingers. In this case, a line segment connecting the positions of the n fingers is obtained as shown in FIG. The number N ₁ of line segments is calculated by the equation (1).

The length l _i (i=1,..., N _l ) of each of the N _l line segments corresponds to the detected position of the j-th (j=1,..., n) finger as P _j =(x _j , Y _j , z _j ) is calculated by the equation (2).

Here, by arranging the lengths l _i of the line segments in the descending order, the gripping feature amount V _{A in the} vector format as in the equation 3 can be defined.

Further, when the maximum value of the element of the gripping characteristic amount V _A is mx, the gripping characteristic amount V′ _A can be defined by the equation (4).

The gripping-feature generation unit 112 generates a gripping-feature amount such as Equation 3 or Equation 4 based on the length of the line segment in FIG.

For example, the object likelihood calculation unit 113 generates a gripping feature amount V _A =(1.5, 1.0, 0.3,...) From the image in FIG.

(Grip feature amount generation method 2)
FIG. 7 is a diagram showing another example of the method of generating the gripping characteristic amount according to the first embodiment of the present invention. Here, it is assumed that the gripping unit detection unit 111 detects the positions of n fingers. In this case, as shown in FIG. 7, n-1 line segments connecting the detected positions of the respective fingers and the positions of the other fingers are obtained. _Let the set of lengths of these line segments be Gr _i =(l _i,1 , l _i,2 ,..., l _i,n−1 ). _Also elements the size of Gr _i Gr those sorted in descending order _{(length) 'i = (l' i} , 1, l 'i, 2, ..., l' i, n-1) to .. Furthermore, 'a _i, the first element l' Gr those rearranged in descending order of _{i, 1 (Gr "0,} Gr" 1, ..., Gr "n) and then, _{Gr" i} elements Gr _"i = the ( l″ _i,1 , l″ _i,2 ,..., L″ _i,n−1 ). By arranging the elements of each Gr″ _i, the gripping characteristic amount V _B as in the formula 5 can be defined.

Further, using the maximum value mx of the elements of the gripping feature quantity V _B, it can be defined gripping feature quantity V _'B, such as equation (6).

The gripping-feature generating unit 112 generates a gripping-feature amount such as Equation 5 or Equation 6 based on the length of the line segment in FIG. 7.

(Grip feature amount generation method 3)
In addition to the position and direction of each finger, it is assumed that the gripping unit detection unit 111 can identify which of the thumb, the index finger, the middle finger, the ring finger, and the little finger each finger is. In this case, for example, the coordinate values P _j =(x _j , y _j , z _j ), (j=1,..., N) of each finger in the order of thumb, index finger, middle finger, ring finger, and little finger, and The direction D _j =(a _j , b _j , c _j ) is obtained. Here, as the direction of each finger, for example, the direction of the normal line of the fingertip position obtained by the distance sensor may be used, or the direction from the first joint to the fingertip may be used. Further, as the order of the fingers (thumb, index finger,...), another order may be used.

By expressing these coordinate values and directions by the coordinates and directions on the predetermined coordinate system Z, the gripping feature amount V _C as in the formula 7 or the gripping feature amount V _D as in the formula 8 is defined. it can.

As the coordinate system Z, for example, a coordinate system having a predetermined coordinate axis parallel to the direction of one finger may be used.

The gripping characteristic generation unit 112 generates a gripping characteristic amount of Expression 7 or Expression 8 based on the position and direction of each finger.

Next, the object likelihood calculating unit 113 calculates the object likelihood based on the gripping feature amount for each object category (step S106). Here, the object likelihood calculation unit 113 calculates the object likelihood according to any of the following object likelihood calculation methods 1 to 3, for example.

(Object likelihood calculation method 1)
FIG. 8 is a diagram showing an example of the gripping characteristic information 115 according to the first embodiment of the present invention. In the gripping characteristic information 115 of FIG. 8, an instance indicating a gripping characteristic amount when the object is gripped is registered for each category of the object. Here, a pair of each category and the gripping feature amount registered for the category is called an instance. Multiple instances may be registered for one category. Here, the object category is C _i (i=1,..., M, M is the number of categories), and the gripping feature amount corresponding to the category C _i is V _ij (j=1,..., R(i), R When (i) is the number of gripping feature quantities for the category C _i ), the instance is represented as (C _i , V _ij ).

The object likelihood calculation unit 113 calculates the distance between the gripping characteristic amount generated by the gripping characteristic generating unit 112 and the gripping characteristic amount V _{ij of} each instance registered in the gripping characteristic information 115. The distance may be Euclidean distance, Manhattan distance, or any other distance measure. The object likelihood calculating unit 113 extracts, from the instances registered in the gripping characteristic information 115, the instances whose calculated distance is equal to or less than a predetermined threshold value. Then, the object likelihood calculating unit 113 calculates the object likelihood L _i of each category C _i by using the extracted instance, for example, by the formula (9).

Here, k is the number of extracted instances, and Nc _i is the number of instances (C _i , V _ij ) corresponding to the category C _i among the k instances.

FIG. 9 is a diagram showing an example of instance extraction in the first embodiment of the present invention. For example, as shown in FIG. 9, the object likelihood calculation unit 113, among the instances registered in the gripping characteristic information 115 of FIG. 8, the gripping characteristic amount V=(1.5, 1.0) generated by the gripping characteristic generating unit 112. , 0.3,...) 10 instances whose distance to the threshold is less than or equal to the threshold value are extracted.

FIG. 10 is a diagram showing an example of the calculation result of the object likelihood in the first embodiment of the present invention. For example, the object likelihood calculation unit 113 calculates the object likelihood as shown in FIG. 10 based on the number of extracted instances.

Note that the object likelihood calculation unit 113 may extract a predetermined number of instances from the smaller distance instead of extracting the instances whose distance is equal to or less than the predetermined threshold value.

(Object likelihood calculation method 2)
FIG. 11 is a diagram showing another example of the gripping characteristic information 115 according to the first embodiment of the present invention. In the gripping feature information 115 of FIG. 11, the object likelihood at each point in the gripping feature amount space is registered for each object category. In this case, the object likelihood at each point is calculated in advance by the nearest neighbor density estimation method, the kernel density estimation method, or the like.

The object likelihood calculation unit 113 refers to the gripping characteristic information 115 and acquires the object likelihood corresponding to the gripping characteristic amount generated by the gripping characteristic generation unit 112, thereby calculating the object likelihood Li of each category C _i. calculate.

(Object likelihood calculation method 3)
In the gripping characteristic information 115, learning results obtained by machine learning such as Support Vector Machine and Random Forest may be registered.

For example, when the Support Vector Machine is used, the identification boundary surface in the gripping feature space is registered as the learning result. In this case, the object likelihood calculation unit 113 calculates the object likelihood L _i of each category C _i by the function of the distance between the gripping feature amount generated by the gripping feature generation unit 112 and the identification boundary surface.

Further, when Random Forest is used, the discrimination boundary in the discriminator and the weight of the discriminator are registered as the learning result. In this case, the object likelihood calculation unit 113 calculates the object likelihood L _i of each category C _i by the function of the weighted voting result.

Next, the gripping target recognition unit 116 recognizes the category of the gripping target 502 using the calculated object likelihood (step S107). Here, the gripping target recognition unit 116 specifies the category idx of the gripping target 502, for example, according to Expression 10.

Note that L _th is a preset threshold value for the maximum value of the object likelihood L _i .

For example, the gripping target recognition unit 116 identifies the category of the gripping target 502 as the category C ₁ having the maximum object likelihood based on the object likelihood calculation result of FIG. 10.

The gripping target recognition unit 116 sets an index of the category of the gripping target 502 or "not applicable" in the recognition result according to the result of step S106 (step S108).

For example, the grip target recognition unit 116 sets “category C ₁ ”in the recognition result.

When the number of detected fingers is less than two in step S103 (step S103/N), "not applicable" is set in the recognition result (step S104).

Finally, the gripping target recognition unit 116 outputs the recognition result to the user or the like (step S109).

With the above, the operation of the first exemplary embodiment of the present invention is completed.

In the above description, the gripping target recognition unit 116 specifies the object category based on the object likelihood for each category calculated by the object likelihood calculation unit 113. However, the present invention is not limited to this, and the gripping target recognition unit 116 identifies the object shape based on the object likelihood with respect to the object shape, and further acquires the category of the object associated with each object shape. You may specify the category.

Further, in the embodiment of the present invention, the case where the gripping means 501 is a human hand has been described as an example, but the present invention is not limited to this, and if the gripping means 501 can grip an object similarly to a human hand, the gripping means 501 can A hand such as a robot may be used.

Further, the gripping means 501 may have a shape other than the hand as long as the object can be gripped by moving the plurality of movable parts. In this case, for example, each movable part of the gripping means 501 is used as the predetermined part, and as the position of the predetermined part (position of the movable part), a designated position such as a tip, a center, or a joint of each movable part is detected. It Further, as the positional relationship between the positions of the plurality of predetermined parts (positional relationship between the movable parts), the coordinate value of the position of each movable part, the distance between the positions of the movable parts, etc. are used.

Next, a basic configuration of the embodiment of the present invention will be described. FIG. 1 is a block diagram showing the basic configuration of the embodiment of the present invention.

Referring to FIG. 1, the recognition device 100 includes an image acquisition unit 110, a gripping feature generation unit 112, and a gripping target recognition unit 116. The image acquisition unit 110 acquires an image of the grip means 501 that grips the grip target 502. The gripping characteristic generation unit 112 generates a gripping characteristic indicating a positional relationship between a plurality of predetermined parts of the gripping means 501 in the image. The grip target recognition unit 116 recognizes the grip target 502 based on the grip characteristics.

According to the embodiment of the present invention, even if the gripping target 502 does not exist in the image of the gripping unit 501 that grips the gripping target 502, the gripping target 502 can be recognized. The reason is that the gripping target recognition unit 116 recognizes the gripping target 502 based on the gripping characteristics indicating the positional relationship between the plurality of predetermined parts of the gripping means 501 in the image. Thereby, even if the gripping target 502 is shielded in the image of the gripping unit 501 gripping the gripping target 502, the gripping target 502 can be recognized.

<Second Embodiment>
Next, a second embodiment of the present invention will be described.

In the second embodiment of the present invention, in addition to the gripping feature amount, the object feature amount is generated from the object region of the gripping target 502 on the image, and the gripping object is determined using the gripping feature amount and the object feature amount. recognize. In the second embodiment of the present invention, the object likelihood based on the gripping feature amount is referred to as a first object likelihood, and the object likelihood based on the object feature amount is referred to as a second object likelihood.

First, the configuration of the second embodiment of the present invention will be described.

FIG. 12: is a block diagram which shows the structure of the recognition apparatus 200 in the 2nd Embodiment of this invention.

Referring to FIG. 12, the recognition device 200 according to the second exemplary embodiment of the present invention includes an image acquisition unit 210, a gripping unit detection unit 211, a gripping feature generation unit 212, a first object likelihood calculation unit 213, and a gripping feature storage. The unit 214 is included. The recognition apparatus 200 further includes a grip target detection unit 221, an object feature generation unit 222, a second object likelihood calculation unit 223, an object feature storage unit 224, an integrated likelihood calculation unit 230, and a grip target recognition unit 240. ..

The image acquisition unit 210, the gripping unit detection unit 211, and the gripping feature generation unit 212 are respectively the image acquisition unit 110, the gripping unit detection unit 111, and the gripping feature generation unit according to the first embodiment of the present invention. Similar to 112. The gripping characteristic storage unit 214 stores gripping characteristic information 215 (information for calculating “object likelihood based on gripping characteristic amount (first object likelihood)”) similar to the gripping characteristic information 115. The first object likelihood calculation unit 213, like the object likelihood calculation unit 113, calculates the first object likelihood for each category of object.

The grip target detection unit 221 detects the object region of the grip target 502 in the image acquired by the image acquisition unit 110. Here, the gripping target detection unit 221 may detect the object region using, for example, a background subtraction method that detects a moving object when the background is fixed. In addition, the gripping target detection unit 221 may detect pixels whose distance is smaller (closer) than a predetermined threshold as the object region. The gripping target detection unit 221 may detect the object region by regarding the peripheral pixels similar to a certain gripping target candidate region (for example, the center of the image) as regions that belong to the gripping target.

The object feature generation unit 222 generates, as the object feature amount representing the feature of the object of the grip target 502, the object feature amount representing the feature of the color or pattern of the grip target 502. Here, for example, the appearance frequency of colors or the arrangement of colors may be used as the object feature. Further, as the object feature, a partial bright/dark pattern of the brightness value of the image, the changing direction of the brightness value, or the response strength to the filter may be used.

The object feature storage unit 224 stores the object feature information 225. The object feature information 225 is information for calculating the “object likelihood based on the object feature amount (second object likelihood)” for the category of the object to be recognized.

The second object likelihood calculation unit 223 uses the object feature amount generated by the object feature generation unit 222 and the object feature information 225 stored in the object feature storage unit 224 to determine the second object likelihood for each object category. Calculate the object likelihood.

The integrated likelihood calculating unit 230 calculates the integrated likelihood using the first object likelihood and the second object likelihood.

The grip target recognition unit 240 recognizes the category of the grip target 502 using the integrated likelihood calculated by the integrated likelihood calculation unit 230.

Next, the operation of the second exemplary embodiment of the present invention will be described.

FIG. 13 is a flowchart showing the operation of the recognition device 200 according to the second embodiment of the present invention.

First, the image acquisition unit 210 acquires an image of the grip means 501 that grips the grip target 502 (step S201). In this case, it is assumed that the image includes at least a part of the object region of the grip target 502.

The gripping unit detection unit 211 detects the position of each finger of the gripping unit 501 or the position and direction of each finger in the image acquired by the image acquisition unit 210 (step S202).

The grip target detection unit 221 detects the object region of the grip target 502 in the image acquired by the image acquisition unit 210 (step S203).

The gripping unit detection unit 211 determines whether the number of fingers detected in the image is two or more (step S204).

In step S204, when the number of detected fingers is two or more (step S204/Y), the gripping characteristic generation unit 212 determines the position of each finger or the position and direction of each finger based on the detected position. , And generate gripping feature amounts (step S206).

The first object likelihood calculation unit 213 calculates the first object likelihood for each object category based on the generated gripping feature amount (step S207).

FIG. 15 is a diagram showing an example of the integrated likelihood calculation result in the second exemplary embodiment of the present invention.

For example, the first object likelihood calculation unit 213 extracts, from the instances registered in the gripping characteristic information 115 in FIG. 8, instances whose distance to the gripping characteristic amount generated by the gripping characteristic generation unit 112 is equal to or less than a threshold value. To do. Then, the first object likelihood calculation unit 213 calculates the first object likelihood as shown in FIG. 15 based on the number of extracted instances.

When the number of detected fingers is less than two in step S204 (step S204/N), 1 is set to the first object likelihood for all categories (step S205).

Next, the object feature generation unit 222 generates an object feature amount from the detected object region of the grip target 502 (step S208).

The second object likelihood calculation unit 223 calculates the second object likelihood for each object category based on the generated object feature amount (step S209). Here, the second object likelihood calculating unit 223 calculates the second object likelihood, for example, by the same method as the method of calculating the object likelihood (first object likelihood) based on the gripping feature amount.

FIG. 14 is a diagram showing an example of the object feature information 225 according to the second embodiment of the present invention. In the object feature information 225 of FIG. 14, an instance indicating the object feature amount of the object is registered for each category of the object. Here, the object feature amount represents, for example, an object feature such as a color or a pattern of the object.

For example, the second object likelihood calculation unit 223 extracts an instance whose distance from the object feature amount generated by the object feature generation unit 222 is equal to or less than a threshold value among the instances registered in the object feature information 225 of FIG. To do. Then, the second object likelihood calculation unit 223 calculates the second object likelihood as shown in FIG. 15 based on the number of extracted instances.

Next, the integrated likelihood calculating unit 230 calculates the integrated likelihood using the first object likelihood and the second object likelihood (step S210). The integrated likelihood calculating unit 230 calculates the integrated likelihood L _comb (i) of each category C _i by, for example, Expression 11 or Expression 12.

Here, L _A (i) and L _B (i) are the first object likelihood and the second object likelihood of category C _i , respectively.

For example, the integrated likelihood calculating unit 230 calculates the integrated likelihood as shown in FIG.

Next, the gripping target recognition unit 240 recognizes the category of the gripping target 502 using the calculated integrated likelihood (step S211). Here, the gripping target recognition unit 240 specifies the category idx of the gripping target 502, for example, according to Expression 13.

Here, L _{th_comb} is a preset threshold value for the maximum value of the integrated likelihood L _comb (i).

For example, the gripping target recognition unit 240 identifies the category of the gripping target 502 as the category C ₃ having the largest integrated likelihood based on the integrated likelihood calculation result of FIG. 15.

The gripping target recognition unit 116 sets the category index of the gripping target 502 or “not applicable” in the recognition result according to the result of step S211 (step S212) and outputs it (step S213).

With the above, the operation of the second exemplary embodiment of the present invention is completed.

According to the second embodiment of the present invention, the recognition accuracy of the grip target 502 can be improved as compared with the first embodiment of the present invention. The reason is that the gripping target recognition unit 240 determines the gripping target 502 based on the integrated likelihood calculated using the first object likelihood based on the gripping feature amount and the second object likelihood based on the object feature amount. This is for recognition. Thereby, for example, when most of the gripping target 502 is shielded in the image of the gripping unit 501 gripping the gripping target 502, but a part thereof is present, the recognition accuracy of the gripping target 502 is improved. it can.

<Third Embodiment>
Next, a third embodiment of the present invention will be described.

In the third embodiment of the present invention, a gripping characteristic amount is generated for a part of the plurality of predetermined parts of the gripping means 501 that is in contact with the grip target 502 (finger that is in contact with the grip target 502). ..

First, the configuration of the third embodiment of the present invention will be described.

FIG. 16: is a block diagram which shows the structure of the recognition apparatus 200 in the 3rd Embodiment of this invention.

Referring to FIG. 16, the recognition device 200 according to the third exemplary embodiment of the present invention includes a contact detection unit 250 in addition to the components of the recognition device 200 according to the second exemplary embodiment of the present invention.

The contact detection unit 250 grasps the inside of the detected fingers based on the object region of the grasped object 502 detected by the grasped object detection unit 221 and the position of each finger detected by the grasping means detection unit 111. A finger (contact finger) in contact with the target 502 is specified. For example, when the distance between the coordinate value indicating the position of the target finger and the object region closest to the coordinate value is less than a predetermined threshold value, the contact detection unit 250 determines that the finger is the contact finger.

The gripping-feature generating unit 212 generates a gripping-feature amount that represents a positional relationship between predetermined parts related to the contact finger.

Next, the operation of the third exemplary embodiment of the present invention will be described.

FIG. 17 is a flowchart showing the operation of the recognition device 200 according to the third embodiment of the present invention.

First, the image acquisition unit 210 acquires an image of the grip means 501 that holds the grip target 502 (step S301).

The gripping unit detection unit 211 detects the position of each finger or the position and direction of each finger in the image acquired by the image acquisition unit 210 (step S302).

The grip target detection unit 221 detects the object area of the grip target 502 in the image acquired by the image acquisition unit 210 (step S303).

The gripping unit detection unit 211 determines whether the number of fingers detected in the image is two or more (step S304).

In step S304, when the number of detected fingers is two or more (step S304/Y), the contact detection unit 250 identifies the contact finger among the detected fingers (step S306).

The contact detection unit 250 determines whether the number of contact fingers is two or more (step S307).

In step S307, when the number of contact fingers is two or more (step S307/Y), the gripping feature generation unit 212 determines the position of each contact finger or the position and direction of each contact finger based on the detected position. , And generates a gripping feature amount indicating a positional relationship between the contact fingers (step S308).

The first object likelihood calculation unit 213 calculates the first object likelihood for each object category based on the generated gripping feature amount (step 309).

If the number of detected fingers is less than 2 in step S304 (step S304/N), or if the number of contact fingers is less than 2 in step S307 (step S307/N), all categories The first object likelihood for is set to 1.

After that, the generation of the object feature amount, the calculation of the second object likelihood, the calculation of the integrated likelihood, and the recognition of the category of the grip target 502 (steps S310 to S315) are the second embodiment (step of the present invention). This is performed in the same manner as S208 to S213).

With the above, the operation of the third exemplary embodiment of the present invention is completed.

According to the third embodiment of the present invention, it is possible to improve the recognition accuracy of the grip target 502 as compared with the first embodiment of the present invention. The reason is that the gripping feature generation unit 212 indicates the gripping feature that represents the positional relationship between the parts that are in contact with the grip target 502 (the fingers that are in contact with the grip target 502) among the plurality of predetermined parts of the grip means 501. This is because it produces a quantity. As a result, it is possible to exclude, from the gripping feature amount, information related to a part that is not in contact with the gripping target 502 (a finger that is not in contact), and the influence of the position of the part that does not contribute to gripping by the gripping unit 501 is affected. The category of the grip target 502 can be specified without receiving it.

<Fourth Embodiment>
Next, a fourth embodiment of the present invention will be described.

In the fourth embodiment of the present invention, a gripping feature indicating a temporal change in the positional relationship between a plurality of predetermined parts of the gripping means 501 is generated.

First, the configuration of the fourth embodiment of the present invention will be described.

FIG. 18: is a block diagram which shows the structure of the recognition apparatus 200 in the 4th Embodiment of this invention.

Referring to FIG. 18, in the configuration of the fourth exemplary embodiment of the present invention, in the second exemplary embodiment of the present invention, the gripping characteristic generation unit 212 is replaced with the gripping characteristic generation unit 260.

The gripping feature generation unit 260 generates a gripping feature amount including a motion feature, which indicates a temporal change in the positional relationship between a plurality of predetermined parts of the gripping unit 501. The grip feature generation unit 260 includes a frame feature generation unit 261, a frame feature storage unit 262, and a motion feature extraction unit 263.

The frame feature generation unit 261 generates the gripping feature amount for each frame of the image by the same method as the gripping feature generation unit 212.

The frame feature storage unit 262 stores the gripping feature amount for each frame generated by the frame feature generation unit 261 for a predetermined number of frames.

The motion feature extraction unit 263 extracts the motion feature based on the difference in the grip feature amount for each frame, and generates the grip feature amount including the motion feature.

Next, the operation of the fourth exemplary embodiment of the present invention will be described.

FIG. 19 is a flowchart showing the operation of the recognition device 200 according to the fourth embodiment of the present invention.

First, the frame feature generation unit 261 of the gripping feature generation unit 260 sets 1 to a variable t (t=1,..., Nf. Nf is the number of frames for generating a gripping feature amount including a motion feature) indicating a frame. It is set (step S401).

The image acquisition unit 210 acquires one frame of the image of the grip unit 501 that holds the grip target 502 (step S402).

The gripping unit detection unit 211 detects the position of each finger or the position and direction of each finger in the frame (target frame) acquired by the image acquisition unit 210 (step S403).

The grip target detection unit 221 detects the object area of the grip target 502 in the target frame acquired by the image acquisition unit 210 (step S404).

The gripping unit detection unit 211 determines whether the number of fingers detected in the target frame is two or more (step S405).

In step S405, when the number of detected fingers is two or more (step S405/Y), the frame feature generation unit 261 determines the position of each finger or the position and direction of each finger based on the detected position. The gripping feature amount V(t) in the target frame t is generated (step S407). Here, the frame feature generation unit 261 uses, as the gripping feature amount V(t), for example, the gripping feature amounts V _A , V _{B, and} V shown in the gripping feature amount generation method according to the first embodiment of the present invention. Generate either _{C or} V _D.

The frame feature generation unit 261 determines whether the variable t is Nf or more (step S408).

If the variable t is less than Nf in step S408 (step S408/N), the frame feature generation unit 261 saves the generated gripping feature amount V(t) in the frame feature storage unit 262, and adds 1 to the variable t. Yes (step S409). Then, the processing from step S402 is repeated.

On the other hand, in step S408, when the variable t is Nf or more (step S408/Y), the motion feature extraction unit 263 calculates the difference in the gripping feature amount for each frame stored in the frame feature storage unit 262 ( Step S410).

The motion feature extraction unit 263 generates a gripping feature amount including a motion feature based on the calculated difference (step S411).

Here, the motion feature extraction unit 263 generates the gripping feature amount V _mov including the motion feature, for example, by using Expression 14.

FIG. 20 is a diagram showing an example of calculation of a gripping feature amount including a motion feature according to the fourth embodiment of the present invention. FIG. 20 is an example when Nf is 3. For example, the motion feature extraction unit 263 generates a grip feature amount V _mov including a motion feature based on the grip feature amount V(t) at t=1, 2, 3 as shown in FIG.

Next, the first object likelihood calculation unit 213 calculates the first object likelihood for each category of the object based on the generated gripping feature amount including the motion feature (step S412).

In addition, in step S405, when the number of detected fingers is less than two (step S405/N), 1 is set to the first object likelihood for all categories (step S406).

Next, the object feature generation unit 222 generates an object feature amount from the detected object region of the grip target 502 (step S413). Here, the object feature generation unit 222 generates an object feature amount from the first frame of the Nf frames or the object region of the grip target 502 detected in the Nfth frame. Further, the object feature generation unit 222 may calculate an average value of the object feature amounts generated from the object region of the grip target 502 detected in each of the Nf frames.

After that, the calculation of the second object likelihood, the calculation of the integrated likelihood, and the recognition of the category of the grip target 502 (steps S414 to S418) are the same as those in the second embodiment of the present invention (steps S209 to S213). To be done.

With the above, the operation of the fourth exemplary embodiment of the present invention is completed.

According to the fourth embodiment of the present invention, the recognition accuracy of the grip target 502 can be improved when the grip by the grip means 501 changes with time. The reason is that the gripping feature generation unit 260 generates a gripping feature amount including a motion feature, which indicates a temporal change in the positional relationship between the plurality of predetermined portions of the gripping unit 501. Accordingly, for example, a gripping target 502 whose shape changes with time during gripping by the gripping unit 501, such as a soft object, can be distinguished from a gripping target 502 whose shape does not change with time, such as a hard object. Further, even when a predetermined part (finger) of the grip means 501 moves on the grip target 502 such as a smartphone operation, the recognition accuracy of the grip target 502 can be improved.

Although the present invention has been described with reference to the exemplary embodiments, the present invention is not limited to the above exemplary embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

For example, as described in the third embodiment of the present invention, the generation of the gripping characteristic amount of the part in contact with the gripping target 502 among the plurality of predetermined parts of the gripping means 501 is performed by the contact with the gripping target 502. If it can be detected, it may be applied to the first embodiment of the present invention.

Further, the generation of the gripping feature amount including the motion feature described in the fourth embodiment of the present invention may be applied to the first embodiment or the second embodiment of the present invention.

100 recognition device 101 CPU
Reference numeral 102 storage device 103 communication device 104 input device 105 output device 110 image acquisition unit 111 gripping means detection unit 112 gripping feature generation unit 113 object likelihood calculation unit 114 gripping feature storage unit 115 gripping feature information 116 gripping target recognition unit 200 recognition device 210 Image acquisition unit 211 Gripping means detection unit 212 Gripping feature generation unit 213 First object likelihood calculation unit 214 Gripping feature storage unit 215 Gripping feature information 221 Gripping target detection unit 222 Object feature generation unit 223 Second object likelihood calculation unit 224 Object Feature storage unit 225 Object feature information 230 Integrated likelihood calculation unit 240 Gripping target recognition unit 250 Contact detection unit 260 Gripping feature generation unit 261 Frame feature generation unit 262 Frame feature storage unit 263 Motion feature extraction unit 501 Gripping unit 502 Gripping target

Claims (7)

A storage unit that stores an object and a gripping feature indicating a positional relationship between a plurality of predetermined portions of the gripping unit when the gripping unit grips the object in association with each other,
An image acquisition means for acquiring an image of said gripping means gripping the gripping target,
Generating the gripping characteristics of the gripping means in the image, a gripping feature generating means,
It said gripping feature acquired from the storage unit, on the basis of, and the gripping features generated by the gripping feature generating means, recognizing, gripping target recognition means that the gripping target is the object,
Equipped with
The information processing apparatus, wherein the gripping feature indicates a positional relationship between the plurality of predetermined parts based on a distance between the plurality of predetermined parts. Furthermore, an object feature generation unit that generates an object feature indicating at least one of the color and the pattern of the grip target in the image is provided,
The gripping target recognizing means recognizes said gripping feature acquired from the storage means, the gripping feature generated by the gripping feature generating means and said object, wherein, on the basis of, the gripping target, wherein The information processing device according to item 1. The gripping characteristic generation means generates the gripping characteristic indicating a positional relationship between the portions of the plurality of predetermined portions that are in contact with the gripping target.
The information processing apparatus according to claim 1. The gripping characteristic generation means generates a gripping characteristic indicating a temporal change in the positional relationship between the plurality of predetermined parts,
The information processing apparatus according to claim 1. The predetermined portion is a finger included in the gripping means,
The information processing apparatus according to claim 1. An object and a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping means when the gripping means grips the object are stored in association with each other,
Acquiring an image of said gripping means gripping the gripping target,
It generates the gripping characteristics of the gripping means in the image,
It said gripping feature is the storage, on the basis of, and the gripping feature the generated, recognizing that the gripping target is the object,
An information processing method ,
The information processing method, wherein the gripping feature indicates a positional relationship between the plurality of predetermined parts based on a distance between the plurality of predetermined parts. On the computer,
An object and a gripping feature indicating a positional relationship between a plurality of predetermined parts of the gripping means when the gripping means grips the object are stored in association with each other,
Acquiring an image of said gripping means gripping the gripping target,
It generates the gripping characteristics of the gripping means in the image,
Based said gripping feature is the storage, and the gripping feature the generated, and recognizes that the gripping target is the object,
A program Ru to execute the processing,
The gripping feature is a program that indicates a positional relationship between the plurality of predetermined portions based on a distance between the plurality of predetermined portions.

Images