JP2010244413A - Method for recognizing object gripped by grip means - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for recognizing an object gripped by a grip means, which improves speed of recognizing an object having a hidden part. <P>SOLUTION: The method for recognizing the object 3 gripped by the grip means includes: an image acquisition step of acquiring an object image 1 including a hand 2 and the object 3 gripped by the hand 2; an object detection step of detecting the object displayed on the image 1; a hand detection step of detecting the hand 2 displayed on the image 1 and estimating gripping posture of the hand 2; and an object determination step of specifying the object 3 (partial area A) in the image 1 from information on the object detected by the object detection step and information on the hand 2 detected by the hand detection step, and recognizing the entire shape of the object 3 from information on the external appearance of the predetermined object 3 (partial area A) and information on the gripping posture of the hand 2 which is estimated by the hand detection step. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a method for recognizing an object that is held by a holding means and has a portion hidden by a holding hand or the like.

  In an image obtained by capturing an image of an object, the object includes a part that is partially hidden by the object itself and a shielding object present on the object and is not depicted on the image. And the method and apparatus which estimate the whole shape of the target object containing such a site | part based on the image which imaged the target object which has a hidden site | part are proposed.

For example, Patent Document 1 describes an information processing apparatus that recognizes a partially hidden object from an image acquired from a camera.
The information processing apparatus disclosed in Patent Document 1 accumulates an image of an object itself supplied from a camera in advance as a model image. From this model image, the degree of change in pixel value with respect to a change in position in the image is determined. An edge strength image having the edge strength shown is generated. In addition, the information processing apparatus generates an edge image indicating a size boundary with respect to the threshold value of the pixel value of the pixel of the image from the model image. Further, the information processing apparatus sets a plurality of feature points at the edge that is a boundary on the generated edge image, and sets a feature amount extraction region in the vicinity of the feature points. Then, the information processing apparatus extracts the edge strength in the feature amount extraction region of each feature point as a model feature amount based on the edge strength image, and associates the model feature amount with the positional relationship of each feature point for each model image. Register in the model dictionary.

Further, the information processing apparatus uses a real image recognized by the camera as a target image, generates an edge image from the target image, and sets a plurality of edge points on the generated edge image. Furthermore, the information processing apparatus extracts the edge strength at each edge point as a target feature amount.
Therefore, the information processing apparatus performs matching between the model feature amount registered in the model dictionary and the target feature amount, and identifies the target object included in the model image and the target object included in the target image, Recognize objects in the target image. Thereby, the shape of the target object including the hidden region of the target object in the target image is specified.

JP 2008-243175 A

  However, in the information processing apparatus of Patent Document 1, all of the model feature amounts corresponding to a plurality of feature points set for each of a large number of model images registered in the model dictionary and a plurality of set of target images are set. The model image and the target image are identified by matching the target feature amount at the feature point. For this reason, there is a problem that the processing amount in matching between the model feature amount and the target feature amount increases, and the processing speed related to the identification of the model image and the target image is reduced. Furthermore, if the number of feature points set in each image is increased in order to improve the identification accuracy between the model image and the target image, the time required for matching increases due to a further increase in the processing amount. There is a problem that the processing speed relating to the identification of the image quality further decreases.

  The present invention has been made to solve such problems, and provides a method for recognizing an object held by a grasping means capable of improving the recognition processing speed of an object having a hidden portion. The purpose is to do.

  An object recognition method according to the present invention includes a grasping means and an image acquisition step for obtaining an image of an object including the object grasped by the grasping means, and an object projected on the image. Detected by an object detection step to be detected, a gripping means detection step for detecting a gripping means projected in an image and estimating a gripping posture of the gripping means, and information on an object detected by the object detection step and a gripping means detection step. The object in the image is identified from the information regarding the gripping means, and the overall shape of the object is recognized from the information regarding the appearance of the identified object and the information regarding the gripping posture of the gripping means estimated by the gripping means detection step. An object determination step.

  Thereby, the spatial range formed inside the gripping means (grip side) can be calculated from the gripping posture estimated from the image of the gripping means projected on the image. And the area | region which the target object hidden by the holding means can take is limited from this space range. Therefore, not only the part projected in the image but also the area related to the appearance of the target object is limited by the region that can be taken by the target object hidden by the gripping means, and the entire shape of the target object is recognized. For this reason, the processing amount required for recognition of a target object is reduced. Therefore, the method for recognizing the object held by the holding means can improve the processing speed required for recognizing the object.

The grasping means detection step calculates an area surrounded by the grasping means from information on the estimated grasping posture of the grasping means, and specifies an area including a portion hidden by the grasping means in the image from the area surrounded by the grasping means. May further be included.
The object determination step may further include combining the specified area with the specified object and recognizing the entire shape of the object from the combined object. Thereby, the range of the elements related to the appearance of the entire object such as the area and the outer peripheral length is limited by combining the area surrounded by the grasping means hidden by the grasping means and the information on the object projected on the image. . Accordingly, the object is limited not only by the portion shown in the image but also by the elements related to the appearance of the object, and the entire shape of the object is recognized.

In the object determination step, a database including information on the object being grasped is accessed, the database is searched only within a range that satisfies the condition using the information regarding the combined object as a condition, and matching with the database is performed. It may further include. Accordingly, by limiting the search range of the database within the range of the condition on the condition regarding the information on the combined objects, the processing amount required for the search is reduced and the processing speed is improved. In the object determination step, the information related to the combined object may include at least one of the area and the outer peripheral length of the combined object.
The object detection step may further include specifying the target object from the object imaged on the image based on the information on the grip means detected by the grip means detection step. Thereby, the object detection step searches from the information about the position and shape of the gripping means in the vicinity of the gripping means in the image to detect the object in the vicinity of the gripping means, and further identifies the target object from the detected object. Therefore, the processing amount in the object detection step and the object determination step is reduced, and the processing speed related to the recognition of the object held by the holding means can be improved.

  The object detection step may further include specifying the target object from the object imaged on the image based on the information related to the holding posture of the holding means estimated by the holding means detection step. As a result, the object detection step detects only the region (grip side) that can be gripped by the gripping means from the information on the gripping posture of the gripping means, detects the object of the image, and further identifies the target from the detected object. To do. Therefore, the amount of processing in the object detection step and the object determination step is reduced, and the processing speed related to the recognition of the object held by the holding means can be improved.

In the object detection step, the object is identified from the object projected on the image based on the information on the gripping means detected in the gripping means detection step and the information on the gripping posture of the gripping means estimated in the gripping means detection step. May further be included. Thereby, the object detection step limits the search range of the object in the image to only the grippable area (grip side) of the gripping means from the information regarding the gripping posture of the gripping means, and further, in this search range, An object in the vicinity of the gripping means is detected by searching from the vicinity of the gripping means from information on the position and the like. In the object detection step, the target is specified from the detected object. Therefore, the amount of processing in the object detection step and the object determination step is reduced, and the processing speed related to the recognition of the object held by the holding means can be improved.
The program according to the present invention is a program for causing a computer to execute each step in the method for recognizing an object held by the holding means described above.

  According to the present invention, the method for recognizing an object held by the holding means can improve the recognition processing speed of an object having a hidden portion.

It is a figure which shows the image containing the hand holding the target object and the target object. In FIG. 1, it is a figure which classifies and shows the area | region detected by the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 1-3 and 6 of this invention. It is a figure which shows each area | region of FIG. 2 separately. It is a block diagram which shows the structure of the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 1 of this invention. It is a figure which shows an example of the database of a dictionary. It is a block diagram which shows the structure of the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 2 of this invention. It is a block diagram which shows the structure of the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 3 of this invention. In FIG. 1, it is a figure which classifies and shows the area | region detected by the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 4 of this invention. It is a block diagram which shows the structure of the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 4 of this invention. In FIG. 1, it is a figure which classifies and shows the area | region detected by the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 5 of this invention. It is a block diagram which shows the structure of the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 5 of this invention. It is a block diagram which shows the structure of the recognition apparatus of the target object currently hold | gripped by the holding means which concerns on Embodiment 6 of this invention.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
Embodiment 1 FIG.
The configuration and operation of the recognition apparatus 101 for an object gripped by the gripping means according to Embodiment 1 of the present invention will be described with reference to FIGS. In the following embodiments, recognition of an object when the object is held by a human hand will be described. Furthermore, the processing in the recognition apparatus 101 is executed by a program incorporated in the recognition apparatus 101.

First, referring to FIGS. 1 and 4, the recognition apparatus 101 is configured to include an image acquisition unit 11, an object detection unit 12a, a hand detection unit 12b, an object determination unit 13, and a dictionary 15 which is a storage unit. ing. The image acquisition unit 11 transmits information to the object detection unit 12a and the hand detection unit 12b, and the object detection unit 12a and the hand detection unit 12b transmit information to the object determination unit 13, respectively. It has become. The hand detection unit 12b and the object determination unit 13 can access the dictionary 15.
The hand detection unit 12b includes a hand detection unit 12b1, a hand shape estimation unit 12b2, and a limitation condition generation unit 12b3. The hand detection unit 12b1 generates a limitation condition from the hand shape estimation unit 12b2 and the hand shape estimation unit 12b2. Information is transmitted to the unit 12b3.
Furthermore, the object determination means 13 includes an information combination unit 13a and an object recognition unit 13b, and information is transmitted from the information combination unit 13a to the object recognition unit 13b.

The image acquisition means 11 is for acquiring an external image, and is constituted by a camera or the like. Then, the two-dimensional image 1 shown in FIG. 1 is acquired by the image acquisition unit 11, and the image acquisition unit 11 sends the acquired image 1 to the object detection unit 12a and the hand detection unit 12b1 of the hand detection unit 12b.
The object detection unit 12a detects a characteristic portion in the supplied image 1 and detects the shape of each object projected on the image 1. Note that the detection of the characteristic portion in the image 1 is performed by, for example, detecting an edge that is a portion where the pixel value of the pixel changes rapidly. Then, the object detection unit 12 a sends the shape information of each object detected in the image 1 to the information combining unit 13 a of the object determination unit 13. The object detection unit 12 a also detects the two-dimensional position of each object in the image 1 and sends this position information to the information combining unit 13 a of the object determination unit 13.

The hand detection unit 12b1 in the hand detection unit 12b receives the supply of the image 1 from the image acquisition unit 11, detects the human hand 2 as the gripping unit in the supplied image 1, and further detects the two-dimensional position of the hand 2 Is detected. Then, the hand detection unit 12b1 sends the detected position information and shape information of the hand 2 to the hand shape estimation unit 12b2 in the same hand detection unit 12b.
When the hand 2 is detected by the hand detection unit 12b, the hand 2 itself is detected by extracting the skin color area in the image 1 and the texture matching of the human skin, and the detected hand 2 and the dictionary 15 The two-dimensional shape of the hand 2 is detected by matching with a hand shape pattern registered in advance.

The hand shape estimation unit 12b2 includes information on the hand 2 including information such as the direction of the hand, the inclination angle of the hand, and the bending angle of the finger from the shape of the hand 2 on the two-dimensional image 1 detected by the hand detection unit 12b1. The finger shape, that is, the holding posture of the hand 2 is estimated. Then, the hand shape estimation unit 12b2 sends the estimated gripping posture information of the hand 2 and the position information and shape information of the hand 2 supplied from the hand detection unit 12b1 to the limited condition generation unit 12b3 in the same hand detection unit 12b. send.
Regarding the estimation of the gripping posture of the hand 2, for example, a study by Tanimoto et al. ("Image database using self-propagating SOM for robot hand control" which was made public in March 2006 by Takaaki Tanimoto et al. Using the method described in the University of Tsukuba Graduate School of Information Science and Technology Master's thesis) Estimation can be performed. In the research of Tanimoto et al., The joint joint angle information and hand image are acquired in advance, contour extraction and feature quantification are performed in the image, and a database is constructed using this feature value and angle as data. . Then, the actual image of the hand is converted into the same feature as when the database was constructed, and the angle of the joint of the hand is estimated by comparing the obtained feature with the feature of the database. Is estimated.
Further, by using a plurality of image acquisition means 11, the hand 2 can be three-dimensionally restored from the image of the hand 2 captured from different directions by the image acquisition means 11, and the gripping posture of the hand 2 can be measured. .

  The limiting condition generation unit 12b3 limits the range of the space formed by the finger, palm, and the like of the hand 2 on the side that holds the object 3 from the gripping posture information of the hand 2 estimated by the hand shape estimation unit 12b2. Further, in this space range, an area that is hidden and hidden from the hand 2 in the image 1 is specified.

  Therefore, the limiting condition generation unit 12b3 obtains a cross section having the maximum cross sectional area or a cross section having the maximum outer peripheral length from the cross sections parallel to the image 1 in the space range formed by the hand 2. Further, for the cross section having the maximum cross-sectional area, the limiting condition generation unit 12b3 projects this cross section on the image 1, specifies the region where the space range formed by the hand 2 overlaps the hand 2 of the image 1, Calculate the area of the identified region. Note that a region where the range of the space formed by the hand 2 in the cross section having the maximum cross-sectional area overlaps the hand 2 of the image 1 is indicated by a region B in the region diagram (b) of FIGS. In addition, the limiting condition generation unit 12b3 projects this cross section on the image 1 for the cross section having the maximum outer peripheral length, specifies an area where the space formed by the hand 2 overlaps the hand 2 of the image 1, and The outer peripheral length of the identified area is calculated. A region where the space formed by the hand 2 in the cross section having the maximum outer peripheral length overlaps with the hand 2 of the image 1 is indicated by a region B in the region diagram (b) of FIGS. In the embodiment, the range of the space formed by the hand 2 in the cross section having the maximum cross-sectional area and the region B where the hand 2 of the image 1 overlaps are the same.

Further, the area and the outer peripheral length of the region B are the limiting conditions for limiting the region of the part hidden in the image 1 by the hand 2 in the object 3.
Then, the limiting condition generating unit 12b3 sends the area value or the outer peripheral length value of the region B shown in the region diagram (b) of FIGS. 2 and 3 to the information combining unit 13a of the object determining unit 13. Further, the limiting condition generation unit 12b3 also sends the position information and shape information of the hand 2 detected by the hand detection unit 12b1 and sent via the hand shape estimation unit 12b2 to the information combination unit 13a.
The processing operation in the hand detection unit 12b and the processing operation in the object detection unit 12a are performed in parallel.

The information combining unit 13a in the object determining unit 13 includes the area B and the position information and shape information of each object including the hand 2 in the image 1 sent from the object detecting unit 12a, and the region B sent from the limiting condition generating unit 12b3. The value of the area or the outer peripheral length of [refer to the region diagram (b) of FIGS. 2 and 3] and the position information and shape information of the hand 2 are combined.
Therefore, the information combining unit 13a uses the object detection unit 12a to detect the object 3 held by the hand 2 based on the position information and shape information of each object in the image 1 and the position information and shape information of the hand 2. It identifies from each detected object. At this time, the region of the object 3 identified by the information combining unit 13a is indicated by a partial region A in the region diagram (a) of FIGS. 2 and 3, and the partial region A is a target imaged on the image 1. Object 3 is shown.

Next, the information combining unit 13a sets the area B [see FIG. 2] sent from the limiting condition generating unit 12b3 to the area value or the outer peripheral length value of the partial area A [see the area diagrams (a) in FIGS. 2 and 3]. 2 and the region diagram (b) of FIG. 3] are combined.
When combining the value of the area of the region B with the value of the area of the partial region A, the information combining unit 13a adds the value of the area of the region B to the value of the area of the partial region A. In addition, when combining the value of the outer peripheral length of the region B with the value of the outer peripheral length of the partial region A, the information combining unit 13a adds the value of the outer peripheral length of the region B to the outer peripheral length of the partial region A. A value that is twice the value of the outer peripheral length of the outer peripheral portions B1, B2, and B3 (refer to the region diagrams (c) in FIGS. 2 and 3), which are regions adjacent to each other, is reduced.

  Thereby, the value of the virtual area of the object 3 or the virtual outer circumference length obtained by combining the limited condition for limiting the portion of the object 3 hidden by the hand 2 in the image 1 by the area or the outer circumference length with the partial region A Is calculated. Then, the calculated value of the virtual area of the object 3 or the value of the virtual outer circumference length is sent to the object recognition unit 13b in the same object determination unit 13 by the information combination unit 13a. The information combining unit 13a also sends the shape information of the partial area A in the target object 3 to the target object recognition unit 13b.

The object recognition unit 13b specifies the overall shape of the object 3 from the value of the virtual area or the value of the virtual outer circumference of the object 3 sent from the information combining unit 13a.
Therefore, the object recognition unit 13b accesses the database 15a shown in FIG.
The database 15a tabulates a set consisting of the name of an article, the area of the article (projected area), the outer perimeter of the article (outer perimeter of the projection surface), and the overall shape of the article as data relating to the article. Includes data on other items. And in the database 15a, the data regarding articles | goods are arranged in ascending order about the area or outer periphery length of articles | goods.

When the value of the virtual area of the target object 3 is sent from the information combining unit 13a, the target object recognition unit 13b only has an article having an area that is equal to or smaller than the virtual area value of the sent target object 3 in the database 15a. Is included in the search range, and the overall shape of the article in the search range is matched with the shape of the partial area A of the object 3 sent from the information combining unit 13a [see the area diagram (a) in FIG. 3]. I do. At this time, the object recognition unit 13b searches for an article within the search range in descending order with respect to the area from the article having the area closest to the virtual area of the object 3 (in the direction in which the area decreases), and the article and the object. Matching with object 3 is performed.
For example, when the value of the virtual area of the object 3 is 7, matching is performed between the shape of the banana, the strawberry, or the like and the shape of the partial region A whose area is 7 or less in the database 15a.

  Further, when the value of the virtual outer circumference length of the target object 3 is sent from the information combining unit 13a, the object recognition unit 13b only has an article having an outer circumference length equal to or smaller than the value of the virtual outer circumference length sent in the database 15a. Is set so as to be included in the search range, and matching between the overall shape of the article within the search range and the shape of the partial region A of the object 3 is performed in the same manner as the matching using the above-described area.

As a result of this matching, the object recognition unit 13b recognizes an article having a shape similar to the shape of the partial area A of the object 3 as the object 3. Thereby, the whole shape of the target object 3 having a hidden part in the image 1 is specified. In addition, the object recognition unit 13b can also specify a gripping posture such as the orientation and inclination of the object 3 when matching the overall shape of the article in the database 15a and the shape of the partial area A of the object 3. .
Therefore, when the recognition apparatus 101 matches the shape of the partial region A of the object 3 with the shape of the article in the database 15a in the dictionary 15, the value of the area between the object 3 and the article in the database 15a or Since the number of articles in the database 15a to be matched is limited by the comparison of the values of the outer peripheral lengths and the quantity thereof is reduced, the processing amount related to matching is reduced and the processing speed is improved.

  As described above, according to the recognition apparatus 101 for the object gripped by the gripping means according to the first embodiment, an image for acquiring the image 1 of the object including the hand 2 and the target 3 gripped by the hand 2. Detected by an acquisition step, an object detection step for detecting an object imaged in image 1, a hand detection step for detecting hand 2 imaged in image 1 and estimating the gripping posture of hand 2, and an object detection step. The object 3 (partial area A) in the image 1 is identified from the information regarding the detected object and the information regarding the hand 2 detected by the hand detection step, and information regarding the appearance of the identified object 3 (partial area A) and the hand Recognizing the object 3 by performing the object determination step for recognizing the entire shape of the object 3 from the information related to the gripping posture of the hand 2 estimated by the detection step. It can be.

  Thereby, the spatial range formed inside the hand 2 (gripping side) can be calculated from the gripping posture estimated from the image of the hand 2 projected on the image 1. And the area | region which the target object 3 which is hidden with the hand 2 can take is limited from this space range. Therefore, the object 3 is limited not only in the part (partial area A) shown in the image 1 but also in the area that can be taken by the object 3 hidden by the hand 2, the elements on the appearance are limited. 3 overall shapes are recognized. For this reason, the processing amount required for recognition of the target object 3 is reduced. Therefore, according to the recognition apparatus 101, the processing speed required for recognition of the target object 3 can be improved.

The image acquisition step and the object detection step are respectively performed by the image acquisition unit 11 and the object detection unit 12a.
The hand detection step is performed by the hand detection means 12b. The hand detection unit 12b calculates a region surrounded by the hand 2 from the estimated information regarding the gripping posture of the hand 2, and includes a region (region B) including a portion hidden by the hand 2 in the image 1 from the region surrounded by the hand 2. ) Can also be specified.

  Further, the object determining step is performed by the object determining means 13. The object determination unit 13 combines the identified object 3 (partial area A) with a region (area B) corresponding to a portion hidden by the hand 2, and the overall shape of the object 3 from the combined object 3. Can also be recognized. At this time, by combining information related to the appearance of the area 3 corresponding to the part hidden by the hand 2 (area B) and the object 3 (partial area A) projected in the image 1 The range of elements related to the appearance of the entire object 3 is limited. Therefore, the target object 3 is limited not only by the part (partial area A) projected on the image 1 but also by elements relating to the appearance of the target object 3, and the entire shape of the target object 3 is recognized.

  Further, the object determination means 13 accesses the database 15a including information on the object 3 being gripped, and uses the information (the virtual area or the virtual outer circumference length of the object 3) regarding the combined object 3 as a condition. It is also possible to search the database only within a range that satisfies the conditions and perform matching with the database. Thereby, since the search range of the database 15a can be limited within the range of the condition on the condition of the information (the virtual area or the virtual outer circumference length of the target object 3) regarding the combined target object 3, the processing amount required for the search Is reduced and the processing speed is improved.

  Further, a plurality of image acquisition means 11 may be provided. When a plurality of images are supplied by the plurality of image acquisition units 11, the object detection unit 12a and the hand detection unit 12b1 are based on the plurality of images and the positions of the object included in the image 1 and the hand 2 in the three-dimensional space. Can be detected. Further, the image acquisition unit 11 may be a stereo camera. In the image captured by the stereo camera, not only the planar position of the object included in the image 1 and the hand 2 but also the distance related to perspective can be indicated, that is, the three-dimensional position can be detected. Thereby, when specifying the target object 3 held by the hand 2 in the information combination unit 13a of the target object determination unit 13, the positions of the hand 2 and the object in the image 1 can be compared in three dimensions. Therefore, the processing amount required for specifying the object 3 is reduced, and the processing speed can be improved.

  In addition, in the object recognition unit 13b of the object determination means 13, when matching the overall shape of the article in the database 15a and the shape of the partial area A of the object 3, the search range of the article in the database 15a You may limit so that only the articles | goods which have the area and outer periphery length which are below the value of both the virtual area of the thing 3, and the value of virtual outer periphery length may be included. Thereby, the data amount in the database 15a used for matching is further reduced, and the processing speed can be further improved.

Embodiment 2. FIG.
The object recognition apparatus 102 grasped by the grasping means according to the second embodiment of the present invention transmits information from the hand detection unit 12b1 of the recognition apparatus 101 according to the first embodiment to the object detection means 12a. It is comprised as follows.
In the following embodiments, the same reference numerals as those in the previous drawings are the same or similar components, and thus detailed description thereof is omitted.

Referring to FIG. 6, in the recognition apparatus 102, information is transmitted from the image acquisition unit 11 to the object detection unit 22a and the hand detection unit 22b1 of the hand detection unit 22b as in the first embodiment. ing.
Further, information is transmitted from the hand detection unit 22b1 to the hand shape estimation unit 22b2 and the object detection unit 22a.
Therefore, referring also to FIG. 1, the hand detection unit 22b1 receives the supply of the image 1 from the image acquisition unit 11, and detects the position information and the shape information of the hand 2 detected in the supplied image 1 as a hand shape estimation unit. 22b2 and the object detection means 22a.

The object detection unit 22a detects the shape and position of each object in the image 1 supplied from the image acquisition unit 11, and the position information of the hand 2 is sent from the hand detection unit 22b1. Therefore, based on this information, the object detection unit 22a starts searching for an object from the vicinity of the hand 2, and detects the position and shape of the object located in the vicinity of the hand 2. That is, the object detection unit 22 a expands the search range from the vicinity of the periphery of the hand 2 on the outside of the hand 2 and detects the position and shape of the object detected near the periphery of the hand 2.
Further, the object detection means 22a detects the object 3 held by the hand 2 based on the detected position information and shape information of the object and the position information and shape information of the hand 2 sent from the hand detection unit 22b1. Identify from the detected object. Then, the object detection unit 22a sends the specified position information and shape information of the target object 3 to the information combining unit 23a of the target object determination unit 23.

The hand shape estimation unit 22b2 and the limitation condition generation unit 22b3 in the hand detection unit 22b operate in the same manner as the hand shape estimation unit 12b2 and the limitation condition generation unit 12b3 of the first embodiment, and the information combination unit 23a 2 and the area value of the region B shown in the region diagram (b) of FIG.
The information combination unit 23a in the object determination unit 23 includes the position information and shape information of the object 3 sent from the object detection unit 22a and the region B [from FIG. 2 and FIG. 3 sent from the limiting condition generation unit 22b3. The area value or the perimeter length value of the area diagram (b)] is combined.

In the second embodiment, since the object 3 has already been specified, the information combining unit 23a determines the area value of the partial area A of the object 3 [see the area diagrams (a) in FIGS. 2 and 3]. Alternatively, the value of the perimeter length and the value of the area or the perimeter length value of the region B [see FIG. 2 and FIG. And the value is sent to the object recognition unit 23b.
Further, since the other configuration and operation of the object recognition apparatus 102 grasped by the grasping means according to the second embodiment of the present invention are the same as those in the first embodiment, description thereof will be omitted.

Thus, according to the recognition apparatus 102 of the target object gripped by the gripping means in the second embodiment, the same effect as the recognition apparatus 101 in the first embodiment can be obtained.
In the recognition device 102, the object detection unit 22a can also specify the target 3 from the object shown in the image 1 based on the information about the hand 2 detected by the hand detection unit 22b1 of the hand detection unit 22b. . Therefore, the object detection unit 22a starts searching for an object in the image 1 from the vicinity of the hand 2 from information on the position and shape of the hand 2 in the processing operation of the object detection unit 22a, and selects the object to be detected by the hand 2. The object is limited to those in the vicinity of and the object 3 is specified from the detected object. Therefore, the object detection unit 22a reduces the object search range and the number of objects to be detected from those in the first embodiment. For this reason, since the processing amount in the object detection unit 22a and the information combining unit 23a is reduced, the recognition apparatus 102 has a higher processing speed than the recognition apparatus 101 in the first embodiment.

Embodiment 3 FIG.
In the configuration of the recognition apparatus 103 for the object gripped by the gripping means according to Embodiment 3 of the present invention, information is transmitted from the hand shape estimation unit 12b2 of the recognition apparatus 101 according to Embodiment 1 to the object detection means 12a. It is configured to be transmitted.

Referring to FIG. 7, in the recognition apparatus 103, information is transmitted from the image acquisition unit 11 to the object detection unit 32 a and the hand detection unit 32 b 1 of the hand detection unit 32 b as in the first embodiment. ing. Further, information is transmitted from the hand detection unit 32b1 to the hand shape estimation unit 32b2.
Information is transmitted from the hand shape estimation unit 32b2 to the limiting condition generation unit 32b3 and the object detection unit 32a.

Referring also to FIG. 1, the hand shape estimation unit 32b2 receives the position information and shape information of the hand 2 detected in the image 1 from the hand detection unit 32b1, as in the first embodiment. The holding posture of the hand 2 is estimated from the shape information of 2. Then, the hand shape estimation unit 32b2 sends the estimated gripping posture information of the hand 2 and the position information and shape information of the hand 2 to the limiting condition generation unit 32b3 and the object detection unit 32a.
Based on the information sent from the hand shape estimation unit 32b2, the limiting condition generation unit 32b3 calculates the area value or the outer circumference length value in the region B shown in the region diagram (b) of FIGS. The calculated value and the position information and shape information of the hand 2 are sent to the information combination unit 33a of the object determination unit 33.

  In addition, the object detection unit 32 a detects the shape and position of each object in the image 1 supplied from the image acquisition unit 11. However, the hand posture estimation unit 32b2 supplies the gripping posture information of the hand 2, and the position information and shape information of the hand 2 to the object detection unit 32a. For this reason, when detecting the shape and position of the object in the image 1, the object detection unit 32 a can start searching for the object from the position information of the hand 2 from the vicinity of the hand 2. It is possible to search for an object limited to a region where the hand 2 can be gripped (grip side region) from the information. That is, the object detection unit 32a starts searching for an object from the vicinity of the hand 2 in the grippable region (grip side region) of the hand 2, and is in the grippable region (grip side region) of the hand 2 and in the vicinity of the hand 2 The position and shape of the object located at is detected. For example, the area on the grip side includes the area inside the finger and palm of the hand 2 and does not include the area on the back side of the hand 2.

  Furthermore, the object detection unit 32a detects the object 3 held by the hand 2 based on the detected position and shape of the object and the position information and shape information of the hand 2 sent from the hand shape estimation unit 32b2. Identify from detected objects. Then, the object detection unit 32a sends the specified position information and shape information of the target object 3 to the information combining unit 33a of the target object determination unit 33.

The information combining unit 33a receives the position information and shape information of the target 3 sent from the object detection unit 32a and the region B sent from the limiting condition generating unit 32b3 [see the region diagram (b) in FIGS. 2 and 3]. ] Are combined in the same manner as in Embodiment 2 to calculate the value of the virtual area or the value of the virtual outer circumference of the object 3, and this value is used as the object recognition unit. To 33b.
Moreover, since the other structure and operation | movement of the recognition apparatus 103 of the target object currently hold | gripped by the holding means which concerns on Embodiment 3 of this invention are the same as that of Embodiment 1, description is abbreviate | omitted.

Thus, according to the recognition apparatus 103 of the target object gripped by the gripping means in the third embodiment, the same effect as the recognition apparatus 101 in the first embodiment can be obtained.
In the recognition device 103, the object detection unit 32a includes information on the position and shape of the hand 2 detected by the hand detection unit 32b1 of the hand detection unit 32b, and the gripping posture of the hand 2 estimated by the hand shape estimation unit 32b2. The object 3 can be specified from the object shown in the image 1 on the basis of the information on the information. Therefore, in the processing operation by the object detection unit 32a, the position and shape of the object in the image 1 are detected from the information regarding the gripping posture of the hand 2 and the position and shape of the hand 2 (the gripping side region). The search for the object is started from the vicinity of the hand 2 in (), and the object to be detected is limited to the object that is in the grippable region (grip side region) of the hand 2 and is located in the vicinity of the hand 2. And the object detection means 32a specifies the target object 3 from the detected object. For this reason, the object search range and the number of objects to be detected are reduced as compared with the second embodiment. Therefore, the processing speed of the recognition apparatus 103 is improved compared to the recognition apparatus 102 of the second embodiment because the processing amount in the object detection unit 32a is reduced.

Embodiment 4 FIG.
The configuration of the recognition apparatus 104 for an object gripped by the gripping means according to Embodiment 4 of the present invention is obtained by changing the region B calculated by the limiting condition generation unit 12b3 of the recognition apparatus 101 in Embodiment 1. The section of the space itself formed by the hand 2 on the side where the object 3 is gripped is a region B.

Referring to FIG. 9, the recognition device 104 includes the image acquisition unit 11, the object detection unit 42 a, the hand detection unit 42 b, and the object determination unit 43, as in the first embodiment.
Similarly to the first embodiment, the limited condition generating unit 42b3 of the hand detecting unit 42b detects the image 1 (see FIG. 1) from the hand shape estimating unit 42b2 of the same hand detecting unit 42b by the hand detecting unit 42b1. The position information and shape information of the hand 2 (see FIG. 1) and the grip posture information of the hand 2 estimated by the hand shape estimation unit 42b2 are sent.

  Referring also to FIG. 8, the limiting condition generation unit 42 b 3 limits the range of the space formed by the hand 2 on the side where the object 3 is gripped from the gripping posture information of the hand 2. Further, the limiting condition generation unit 42b3 obtains a cross section having the maximum cross sectional area or a cross section having the maximum outer peripheral length from the cross sections parallel to the image 1 (see FIG. 1) in the space range formed by the hand 2. Therefore, the limiting condition generation unit 42b3 projects the obtained cross section onto the image 1, and the projected area is shown as a region Bd in FIG. In the fourth embodiment, the cross section having the maximum cross-sectional area is the same as the cross section having the maximum outer peripheral length. Further, the limiting condition generation unit 42b3 detects the position and shape of the region Bd, calculates the value of the cross-sectional area or the outer peripheral length of the region Bd, and uses these pieces of information as the information combination unit 43a of the object determination unit 43. Send to. At the same time, the limiting condition generation unit 42b3 sends the position information and shape information of the hand 2 sent from the hand shape estimation unit 42b2 to the information combining unit 43a.

  The information combining unit 43a includes position information and shape information of each object, position information and shape information of the hand 2, and position information of the region Bd in the image 1 (see FIG. 1) detected and sent by the object detection unit 42a. Then, based on the shape information, the object 3 held by the hand 2 is specified from among the objects detected by the object detection means 42a. Further, the information combining unit 43a specifies a region obtained by removing a region that overlaps the region Bd from the region of the specified object 3, and detects the position and shape of this region. Note that the region of the object 3 that is identified by the information combining unit 43a and from which the region overlapping the region Bd is removed is the partial region Ad shown in FIG.

Next, the information combining unit 43a calculates the value of the area of the partial region Ad or the value of the outer peripheral length, and sends the calculated value of the area of the partial region Ad or the value of the outer peripheral length and the limit condition generating unit 42b3. The area value or the outer circumference length value of the region Bd is combined to calculate the virtual area value or the virtual outer circumference length value of the object 3. That is, the value of the virtual area of the target object 3 is the value of the area of the entire region where the partial region Ad and the region Bd are combined, and the value of the virtual outer circumference length of the target object 3 is a combination of the partial region Ad and the region Bd This is the value of the outer peripheral length in the entire region.
Then, the information combining unit 43a sends the calculated virtual area value or virtual outer peripheral length value of the target object 3 and the shape information of the partial region Ad of the target object 3 to the target object recognition unit 43b.

Further, since the other configuration and operation of the object recognition device 104 held by the holding means according to the fourth embodiment of the present invention are the same as those in the first embodiment, description thereof will be omitted.
Thus, according to the recognition apparatus 104 of the target object gripped by the gripping means in the fourth embodiment, the same effect as the recognition apparatus 101 in the first embodiment can be obtained.
Further, in the recognition device 104, the limiting condition generation unit 42b3 of the hand detection unit 42b calculates the value of the area or the outer peripheral length in the region Bd that is a cross section of the space formed by the hand 2 on the side where the object 3 is gripped. This value is used as a limiting condition for specifying the overall shape of the object 3. As a result, the limiting condition generation unit 42b3 calculates the length of the edge portion of the finger of the hand 2 for the calculation of the region where the hand 2 and the region Bd overlap or the length of the outer periphery of the hand 2 in the region Bd. Detailed information processing work such as calculation is reduced. Therefore, the recognition apparatus 104 can improve the processing speed as compared with the recognition apparatus 101 according to the first embodiment because the processing amount in the limiting condition generation unit 42b3 is reduced.

Embodiment 5 FIG.
In the configuration of the recognition apparatus 105 for the object gripped by the gripping means according to Embodiment 5 of the present invention, information is sent from the object detection means 12a of the recognition apparatus 101 in Embodiment 1 to the limiting condition generation unit 12b3. It is configured to be transmitted.

Referring to FIG. 11, in the recognition apparatus 105, as in the first embodiment, information is transmitted from the image acquisition unit 11 to the object detection unit 52a and the hand detection unit 52b1 of the hand detection unit 52b. ing. Further, information is transmitted from the hand detection unit 52b1 to the hand shape estimation unit 52b2, and from the hand shape estimation unit 52b2 to the limiting condition generation unit 52b3. Further, information is transmitted from the object detection means 52a to the limiting condition generation section 52b3 and the information combination section 53a of the object determination means 53.
Accordingly, the hand shape estimation unit 52b2 performs position information and shape information of the hand 2 (see FIG. 1) detected in the image 1 (see FIG. 1) by the hand detection unit 52b1 as well as the first embodiment, and the hand. The holding posture information of the hand 2 estimated by the shape estimating unit 52b2 is sent to the limiting condition generating unit 52b3. Further, the object detection means 52a sends the position information and shape information of each object detected in the image 1 to the limiting condition generation unit 52b3 and the information combination unit 53a.

Referring to FIG. 10 as well, the limiting condition generation unit 52b3, as in the first embodiment, the position information and shape information of each object in the image 1 (see FIG. 1) sent from the object detection unit 52a, Based on the position information and shape information of the hand 2 sent from the hand shape estimation unit 52b2, the target 3 held by the hand 2 is specified from among the objects detected by the object detection means 52a. This identified object 3 is indicated by a partial area A in FIG.
Further, the limiting condition generation unit 52b3 calculates a region B shown in FIG. 10 based on the information sent from the hand shape estimation unit 52b2. Region B is the same as region B in the first embodiment.

Next, the limiting condition generation unit 52b3 detects the positions and shapes of the partial areas A and B, and is surrounded by the partial areas A (target 3) and the areas B based on the detected position information and shape information. A region C shown in FIG. 10 which is a region is specified.
Further, the limiting condition generation unit 52b3 detects a boundary line C1 between the region C and the partial region A. Then, the limiting condition generation unit 52b3 extends both ends of the boundary line C1 toward the inside of the region B in accordance with the curvature of the boundary line C1 until reaching the outer peripheral portion B4 of the region B, and extends to both ends of the boundary line C1. Line parts C2 and C3 are set. Note that the extended line portions C2 and C3 may be straight lines extending in the tangential direction of the boundary line C1.

Therefore, the limiting condition generation unit 52b3 adopts only the region on the partial region A (target 3) side in the region B as the new region Be1 with the extended line portions C2 and C3 as the boundary, and the extended line portions C2 and C3 as the boundary. The region Be2 on the opposite side to the partial region A (object 3) in the region B is deleted. For this reason, the region Be1 is limited to a part of the region B, and has a shape and area closer to those of the region B than the actual region hidden by the hand 2 in the object 3.
Further, the limiting condition generation unit 52 b 3 calculates the area value or the outer peripheral length value in the region Be 1, and sends this information to the information combining unit 53 a of the object determination unit 53. At the same time, the limiting condition generation unit 52b3 sends the position information and shape information of the partial area A to the information combining unit 53a.

The information combining unit 53a calculates the value of the area of the partial region A or the value of the outer peripheral length from the shape information of the partial region A sent from the limiting condition generating unit 52b3, and calculates the value of the area of the partial region A or the outer peripheral length. The value is combined with the area value or the outer peripheral length value of the region Be1 sent from the limiting condition generating unit 12b3, and the virtual area value or the virtual outer peripheral length value of the object 3 is calculated. That is, the value of the virtual area of the object 3 is the value of the entire area where the partial area A and the area Be1 are combined, and the value of the virtual outer circumference length of the target object 3 is the combined area of the partial area A and the area Be1. This is the value of the outer peripheral length in the entire region. Note that the calculated virtual area value or virtual outer circumference length value of the object 3 is the virtual area value or virtual outer circumference length of the object 3 obtained using the region B as shown in the first embodiment. Is closer to the actual area value of the entire object 3 or the outer peripheral length.
Then, the calculated value of the virtual area of the object 3 or the value of the virtual outer circumference length is sent to the object recognition unit 53b by the information combining unit 53a together with the shape information of the partial area A of the object 3.

Further, since the other configuration and operation of the object recognition apparatus 105 gripped by the gripping means according to the fifth embodiment of the present invention are the same as those in the first embodiment, description thereof will be omitted.
Thus, according to the recognition apparatus 105 of the target object gripped by the gripping means in the fifth embodiment, the same effect as the recognition apparatus 101 in the first embodiment can be obtained.
In addition, since the value of the virtual area or the virtual outer circumference length of the object 3 in the recognition apparatus 105 is calculated by limiting the region B in the first embodiment to the region Be1 that is a part of the region B, the first embodiment. Is closer to the actual area value or the outer peripheral length of the entire target object 3 than the value of the virtual area or the virtual outer peripheral length value of the target object 3 calculated in (1). Thereby, since the search range of the articles in the database 15a (see FIG. 5) is narrowed, the processing amount at the time of matching with the database 15a in the object recognition unit 53b of the object determination means 53 is reduced. Therefore, the recognition apparatus 105 can improve the processing speed as compared with the recognition apparatus 101 of the first embodiment.

Embodiment 6 FIG.
The configuration of the object recognition device 106 gripped by the gripping means according to Embodiment 6 of the present invention is such that the limited condition generation unit 12b3 of the recognition device 101 in Embodiment 1 uses the area B area value or the outer circumference. The position information and the shape information of the region B are sent from what has been sent the long value to the information combining unit 13a.

Referring to FIG. 12, the recognition device 106 includes an image acquisition unit 11, an object detection unit 62 a, a hand detection unit 62 b, and an object determination unit 63, as in the first embodiment.
The limited condition generation unit 62b3 of the hand detection unit 62b includes the position information and shape information of the hand 2 (see FIG. 1) detected by the hand detection unit 62b1, and the posture of the hand 2 estimated by the hand shape estimation unit 62b2. Information is sent.
Referring to FIGS. 2 and 3 together, the limiting condition generation unit 62b3 creates the area B shown in the area diagram (b) of FIGS. 2 and 3 based on the sent information in the same manner as in the first embodiment. calculate. Further, the limiting condition generating unit 62b3 detects the position and shape of the region B, and sends the detected position information and shape information to the information combining unit 63a of the object determining unit 63. At the same time, the limiting condition generation unit 62b3 sends the position information and shape information of the hand 2 to the information combining unit 63a.

The information combining unit 63a includes the position information and shape information of each object in the image 1 (see FIG. 1) detected and sent by the object detection unit 62a, the position information of the hand 2 sent from the limiting condition generation unit 62b3, and Based on the shape information, the object 3 held by the hand 2 is specified from among the objects detected by the object detection means 62a. The specified object 3 is indicated by a partial area A in the area diagram (a) of FIGS. 2 and 3.
Further, the information combining unit 63a divides the partial region A [see the region diagram (a) in FIG. 3] and the region B [see the region diagram (b) in FIG. 3] sent from the limiting condition generating unit 62b3. Are combined based on the position information and shape information (see FIG. 2). Then, the information combining unit 63a calculates the value of the virtual area or the value of the virtual outer circumference length in the region where the partial region A and the region B are combined.
The calculated value of the virtual area or the value of the virtual outer circumference length is sent to the object recognition unit 63b by the information combining unit 63a. At the same time, the shape information of the partial area A is also sent to the object recognition unit 63b by the information combining unit 63a.

Further, since the other configuration and operation of the object recognition apparatus 106 gripped by the gripping means according to Embodiment 6 of the present invention are the same as those in Embodiment 1, description thereof is omitted.
Thus, according to the recognition apparatus 106 of the target object gripped by the gripping means in the sixth embodiment, the same effect as the recognition apparatus 101 in the first embodiment can be obtained.

  Moreover, in the object determination means 13-63 of Embodiments 1-6, the value of the virtual area of the target object 3 or the value of the virtual perimeter length obtained by combining the limiting conditions with the partial regions A and Ad of the target object 3 Is used to limit the search range in the database 15a and match the shape of the article within the search range with the shape of the object 3. However, the present invention is not limited to this. Items relating to the softness (tactile sensation) of the article and the color of the article are added to the database 15a, and the contents of the database 15a are determined according to the softness (tactile sensation) detected from the object 3 and the color of the object 3 detected from the image 1. The search range of the article may be further limited. Thereby, since the processing amount at the time of matching with the shape of the articles | goods in the database 15a and the shape of the target object 3 is further reduced, the processing speed of the recognition apparatuses 101-106 can be improved.

In the recognition apparatuses 101 to 106 according to the first to sixth embodiments, the grasping means for the object 3 is the human hand 2. However, the present invention is not limited to this, and a robot hand may be used.
In the first to sixth embodiments, the entire shape of the object 3 having a hidden portion in the image 1 is recognized by the hand 2, that is, the gripping means, but is not limited thereto. Even if what hides the object 3 is something other than gripping means such as cloth, the recognition method of the object 3 by the recognition devices 101 to 106 can be applied.

  1 image, 2 hands (gripping means), 3 object, 11 image acquisition means, 12a, 22a, 32a, 42a, 52a, 62a object detection means, 12b, 22b, 32b, 42b, 52b, 62b hand detection means, 13 , 23, 33, 43, 53, 63 Object determination means, 15a database, 101, 102, 103, 104, 105, 106 Recognizing apparatus for objects gripped by the gripping means.

Claims (9)

A method for recognizing an object held by a holding means,
An image acquisition step of acquiring an image of an object including the object being gripped by the gripping means and the gripping means;
An object detection step of detecting the object projected in the image;
A gripping means detection step of detecting the gripping means projected in the image and estimating a gripping posture of the gripping means;
From the information related to the object detected in the object detection step and the information related to the gripping means detected in the gripping means detection step, the object in the image is identified.
A grasping means comprising: an object determination step for recognizing the overall shape of the object from information relating to the appearance of the identified object and information relating to a grasping posture of the grasping means estimated by the grasping means detection step. A method for recognizing a grasped object. The gripping means detection step includes
From the information about the estimated gripping posture of the gripping means, calculate a region surrounded by the gripping means,
The method for recognizing an object gripped by the gripping means according to claim 1, further comprising: specifying an area including a portion hidden by the gripping means in the image from an area surrounded by the gripping means. The object determining step includes:
Combining the identified region with the identified object;
The method for recognizing an object held by the holding means according to claim 2, further comprising recognizing an overall shape of the object from the combined objects. The object determining step includes:
Accessing a database containing information about the object being grasped, searching the database only within a range that satisfies the condition on the condition that the information about the combined object is a condition, and performing matching with the database Furthermore, the recognition method of the target object hold | gripped by the holding means of Claim 3. In the object determining step,
The method for recognizing an object held by the holding means according to claim 4, wherein the information related to the combined object includes at least one of an area and an outer peripheral length of the combined object. The object detection step includes:
The gripping according to any one of claims 1 to 5, further comprising: specifying the target object from the object projected on the image based on information on the gripping means detected by the gripping means detection step. A method for recognizing an object held by a means. The object detection step includes:
6. The method according to claim 1, further comprising: specifying the target object from the object projected on the image based on information on a gripping posture of the gripping means estimated by the gripping means detection step. A method for recognizing an object gripped by the gripping means. The object detection step includes:
Based on the information on the gripping means detected by the gripping means detection step and the information on the gripping posture of the gripping means estimated by the gripping means detection step, the object is identified from the object projected on the image. A method for recognizing an object gripped by the gripping means according to any one of claims 1 to 5, further comprising:   The program for making a computer perform each step in the recognition method of the target object currently hold | gripped by the holding means as described in any one of Claims 1-8.

Images