JP4646943B2 - robot - Google Patents

Description

  The present invention relates to a robot that holds an article according to the condition of the article.

  As a conventional holding device, there is known a device including a handling unit configured to limit a holding object to one type such as vegetables and cloth and to hold the holding object optimally. (For example, refer to Patent Documents 1 and 2).

  On the other hand, a plurality of handling units corresponding to each type of holding object are provided so that a plurality of types of holding objects can be held optimally, and an optimum handling unit is selected according to the type of holding object. A holding device is also known (see, for example, Patent Document 3).

  Further, when there are a plurality of types of holding objects, the feature data of each type of holding object and the operation information for the holding object are stored in association with each other, and the visual sensor is based on the feature data. There is also known a holding device that recognizes the type of the holding object and holds it with an operation according to the recognized holding object (see, for example, Patent Document 4).

Japanese Patent No. 1976349 JP-A-8-132379 Japanese Patent No. 2710856 Japanese Patent Laid-Open No. 64-2882

  Since the holding devices disclosed in the above documents are used in the field of so-called FA (Factory Automation) in factories, warehouses, etc., the types of holding objects are limited and are necessary when holding by holding devices. Information such as shape data, material data, holding position data in the holding object, and the like are obtained in advance for each type of holding object.

  On the other hand, the above holding device is mounted on a robot, for example, and the robot performs work in a space in which a person lives (such as a home, office, hotel, store, and hospital). It can be considered. However, since there are a great many types of articles in the living space, the number of types of objects to be held by the holding device increases accordingly.

  Therefore, it may be difficult to obtain information necessary for holding all types of articles in advance, and it may be impossible to hold articles without information.

  Moreover, in a living space, there are cases where information necessary for holding cannot be obtained in advance. For example, when holding a dish on which a dish is arranged, it is necessary to hold (grip) a portion where the dish is not arranged, but it is not possible to determine in advance where the dish is arranged. That is, information necessary for holding cannot be obtained in advance.

  Further, the situation of the holding object does not change in a factory or the like, but the situation of the article may change every moment in the living space. For example, paper usually has a sheet-like form, but the form may change completely when crumpled. Then, if the information necessary for holding the article has been obtained in advance, that is, in the above example, even if the information for holding the sheet-like paper is obtained, the paper on which the information is rounded is used. Useless when holding.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a robot capable of optimally holding a wide variety of articles according to the situation. .

  In order to achieve the above object, the present invention is configured as follows.

According to the first aspect of the present invention, a recognition device for recognizing the status of a holding object among articles in a living space;
A holding method database for storing different article holding methods depending on the status of the holding object;
An article holding method specifying means for specifying one article holding method from different article holding methods according to the situation recognized by the recognition device;
A holding device that performs holding of the holding object with reference to the article holding method specified by the article holding method specifying means,
The recognition device recognizes a holding operation of the article performed by a person , including a moving speed or position of the article,
Learning means for learning and setting a holding method for each article or each situation of the article based on a recognition result of the holding operation of the article performed by the person by the recognition device;
The learning means refers to a holding operation of the article performed by the person, learns a holding method of the article, and stores the learned holding method in the holding method database in association with the article. provide.

  According to the robot of the present invention, in order to recognize the situation of the holding object and specify the holding method according to the recognition result, the optimum holding method according to the situation of various articles is specified, and the holding object is It can be held by an optimal holding method.

  Before continuing the description of the present invention, the same parts are denoted by the same reference numerals in the accompanying drawings.

  Before describing embodiments of the present invention, various aspects of the present invention will be described first.

According to a first aspect of the present invention, there is provided a holding robot control method for holding an article in a living space using a robot holding device.
A recognition step for recognizing a situation of a holding object among the articles in the living space;
A specific step of identifying one article holding method from among different article holding methods according to the situation recognized in the recognition step;
A holding robot control method including a holding step of holding the holding object by the holding device means according to the article holding method specified in the specifying step.

According to the second aspect of the present invention, the recognition device for recognizing the status of the holding object among the articles in the living space;
A holding method database for storing different article holding methods depending on the status of the holding object;
An article holding method specifying means for specifying one article holding method from different article holding methods according to the situation recognized by the recognition device;
With reference to the article holding method specified by the article holding method specifying means, there is provided a robot comprising a holding device for holding the holding object.

According to the third aspect of the present invention, the holding object is an article whose shape, temperature, and form including contents change,
The recognition device recognizes the form of the holding object as the situation of the holding object,
The article holding method specifying means specifies one article holding method from the different article holding methods in the holding method database according to the form as the situation recognized by the recognition device. The robot described in is provided.

  According to the fourth aspect of the present invention, the holding method database stores article holding methods for a plurality of different holding objects as the holding objects, and the different holding objects share the same article ID information. The robot according to the second aspect, wherein the article holding method is stored so that the holding method is changed by changing the form of the article having different shapes such as shape, temperature, or contents. I will provide a.

  According to a fifth aspect of the present invention, there is provided the robot according to any one of the second to fourth aspects, wherein the recognition device recognizes the state of the holding object that changes with time.

According to the sixth aspect of the present invention, the holding object is an article that can accommodate an object therein,
The recognizing device recognizes whether or not an object is contained in the holding object;
The article holding method specifying means selects one article holding method from among different article holding methods depending on whether or not an object is accommodated in the holding object recognized by the recognition device. The robot according to the fifth aspect is specified.

According to the seventh aspect of the present invention, the holding object has light transmission characteristics,
The recognizing device recognizes whether an object is accommodated inside the holding object using refraction of light in the holding object,
The article holding method specifying means selects one article holding method from among different article holding methods depending on whether or not an object is accommodated in the holding object recognized by the recognition device. The robot according to the sixth aspect is specified.

According to the eighth aspect of the present invention, the recognition device recognizes the temperature distribution of the holding object,
In the fourth aspect, the article holding method specifying means specifies one article holding method from among different article holding methods, which is recognized by the recognition device, according to the temperature distribution of the holding object. Provide robots.

According to the ninth aspect of the present invention, the holding object is a cooking utensil or tableware,
The holding method specifying means provides the robot according to the eighth aspect, wherein the holding position of the cooking utensil or tableware is specified based on the temperature distribution of the cooking utensil or tableware recognized by the recognition device.

  According to a tenth aspect of the present invention, there is provided the robot according to the fifth aspect, wherein the holding object is paper or clothing.

According to the eleventh aspect of the present invention, the recognition device recognizes whether an object is placed on the holding object,
The article holding method specifying means selects one article holding method from among different article holding methods according to whether or not an object is placed on the holding object recognized by the recognition device. The robot according to the fifth aspect is specified.

  According to a twelfth aspect of the present invention, there is provided the robot according to the eleventh aspect, wherein the holding object is tableware.

According to a thirteenth aspect of the present invention, further comprising learning means for learning the holding method,
The learning means refers to a holding operation of an article being performed by a person, learns a holding method of the article, and stores the learned holding method in association with the article in the holding method database. The robot described in is provided.

According to the fourteenth aspect of the present invention, the holding method database stores a holding method set in advance for each article,
The holding method specifying means selects the holding method of the holding object from the holding methods preset in the holding method database with reference to the recognition result by the recognition device,
When the holding method corresponding to the recognition result is not preset in the holding method database, the holding method specifying means further includes a holding method set for an article that approximates the material or shape of the holding object, The robot according to the second aspect, which is selected from the holding method database and used as the holding method of the holding object, is provided.

According to the fifteenth aspect of the present invention, the recognition device recognizes the material of the holding object,
The robot according to the second aspect, wherein the article holding method specifying means specifies one article holding method from among different article holding methods according to the material of the holding object recognized by the recognition device. provide.

According to a sixteenth aspect of the present invention, the recognition device comprises:
An imaging device for imaging the holding object;
A robot according to a fifteenth aspect is provided, comprising: a material recognizing unit that recognizes a material of the holding object based on an image picked up by the image pickup device using an optical characteristic of the holding object.

According to a seventeenth aspect of the present invention, further comprising a light source for irradiating the holding object with light,
The robot according to the sixteenth aspect, wherein the material recognizing means recognizes the material using the reflection characteristic of the holding object.

According to an eighteenth aspect of the present invention, the light source is provided so that the position relative to the holding object can be changed,
The imaging device captures a plurality of images while changing the position of the light source,
The robot according to the seventeenth aspect, wherein the material recognition means recognizes the material of the object to be held by processing the plurality of images.

According to a nineteenth aspect of the present invention, there is provided an article holding system for holding an article in a living space,
A recognition device for recognizing the status of a holding object among the articles in the living space;
A robot for holding the holding object based on the recognition result by the recognition device,
The robot
A holding method specifying means for receiving a situation recognized by the recognition device and specifying a holding method according to the situation;
There is provided a holding system including a holding device that holds the holding object in accordance with the holding method specified by the holding method specifying means.

According to a twentieth aspect of the present invention, in the article holding system according to the nineteenth aspect,
The holding device provides a holding system mounted on the robot.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram showing a basic configuration of an article holding system 10 by a robot 2 having a holding device 103 according to the first embodiment of the present invention. The basic components of the article holding system 10 are mounted on a human-type home robot 2 as shown in FIG. 2A, for example. The robot 2 is a device that receives a work command and executes a work in accordance with the command. The article holding system 10 executes holding of an article related to the work.

  As shown in FIG. 1, the article holding system 10 includes a recognition device 101, a holding method specifying unit 102, a holding method database 102 d, and a holding device 103.

  The recognition device 101 recognizes the status of the holding object. Specific processing of situation recognition by the recognition apparatus 101 will be described later.

  The holding method specifying unit 102 refers to the situation recognized by the recognition device 101 and specifies an optimum article holding method for the situation from a plurality of article holding methods stored in advance. In this “article holding method”, as will be described later, a gripping hand 103 or a suction hand 103 s that grips or sucks an article is selected, and how the held article is moved (a gripping or suction angle, Moving speed, moving direction, etc.).

  The holding method database 102d stores in advance different article holding methods depending on the status of the holding object.

  For example, at least one holding device 103 is disposed on the main body 390 of the robot 2, and preferably a pair is disposed on both sides of the main body 390 of the robot 2. Each holding device 103 performs holding of the holding object in accordance with the holding method specified by the holding method specifying means 102. For example, as shown in FIG. 2A, each holding device 103 may include a hand 103a that holds an article and a bendable arm 103b that has a hand 103a attached to the tip. The hand 103a includes a pneumatic hand, a hydraulic hand, a motor hand, a hand composed of a plurality of fingers such as an artificial muscle, a suction hand 103s, etc., which are selected according to a specified holding method as described later. Is done.

  FIG. 2B is a schematic diagram of the holding device 103 for performing posture control when holding an article. 2B, the arm 103b of the holding device 103 includes an arm on the proximal end side having a length L1, an arm on the distal end side having a length L2, and a base end portion of the arm on the proximal end side of the robot 2 by indirect P0. The distal end of the proximal arm and the proximal end of the distal arm are connected to bendable by indirect P1 while being pivotally connected to the side of the main body 390. A hand 103a is connected to the tip of the arm via indirect P2. Here, in order to simplify the explanation, it is considered to control the posture of the contact indirect P2 between the hand 103a and the arm 103b on a two-dimensional plane.

  FIG. 2C shows a drive control system of an arbitrary drive unit such as a motor control type actuator of the moving device 414 and the holding device 103 described later. A similar drive control system is configured for all the drive units.

  The rotation axis of the motor 304 is driven forward and backward by the current supplied from the motor driver 303 to rotate, for example, four wheels 394 forward and backward. The motor 304 is provided with an angle sensor for detecting the rotation angle of the driven rotating shaft. An output signal from the angle sensor 307 is input to the CPU 301 through the encoder 306 and the encoder board 305.

  The CPU 301 controls the driving of the motor 304. The CPU 301 is connected to a memory 310 that stores information necessary for control.

  Further, the hand 103a is provided with a pressure sensor 309 for detecting whether or not the object to be held is properly held, for example, on the grip surface, and the sensor output signal from the pressure sensor 309 is A / D. The data is input to the CPU 301 through the converter 308.

  The output signal from the angle sensor 307 and the sensor output signal from the pressure sensor 309 are received as input signals, and in response to the received input signal, the CPU 301 generates a control signal for the motor 304, and the control signal is D / A It is sent to the motor driver 303 through the converter 302. In response to this control signal, the motor 304 is driven forward and backward to realize arm and hand posture control.

  Here, the length of the arm 103b is L1, L2, and the angle sensor 307 is acquired by the angle sensor (here, an encoder) at the indirect P0 and P1, and the angles are θ and φ, respectively. At this time, the position of the tip position P2 of the arm 103b is obtained by the following relational expression.

この関係式から、保持対象物の位置を推定し、任意の位置へ移動させることができるため、姿勢制御を実現することができる。姿勢制御とは、以上のような角度センサなどの情報を元に微調整を行い、位置や向きを正確に制御することである。また、ここでは簡略化のために２次元平面における位置制御のみについて説明したが、３次元空間における位置・姿勢制御についても同様の処理が可能である。

From this relational expression, the position of the object to be held can be estimated and moved to an arbitrary position, so that posture control can be realized. Posture control is to perform fine adjustment based on information such as the angle sensor as described above, and to accurately control the position and orientation. In addition, here, only the position control in the two-dimensional plane has been described for the sake of simplicity, but the same processing can be performed for the position / posture control in the three-dimensional space.

  For example, when the glass GL containing a liquid is held by a hand, the posture control in the three-dimensional space is to control the posture so that the liquid does not spill. Here, the coordinate system is set as shown in FIG. 2D. That is, with respect to the position of the object, the horizontal floor composed of the Xw axis and the Yw axis orthogonal to each other is set to the Xw-Yw plane, and the Zw axis is set vertically upward from this plane. Regarding the posture (rotation), the bottom surface of the glass GL is set to the xy plane composed of the x-axis and the y-axis orthogonal to each other, and the z-direction is set in the vertical direction from this plane toward the mouth direction of the glass GL. Then, the directions of α, β, and γ are respectively set clockwise from the origin side with respect to the x, y, and z axes. When controlling the attitude of the glass GL containing the liquid LD inside, only the directions α and β need not be changed with respect to the six axes.

FIG. 2E is a table showing allowable variation ranges in the α and β directions with respect to the holding level, and shows information on the holding method stored in the holding method database 102d. The “holding level” is determined in advance depending on the amount of the liquid LD contained in the glass GL and the size of the glass GL. As shown in FIG. 2D, the height of the glass GL is h, the radius of the mouth of the glass GL is r, the height of the liquid LD in the glass GL is hi, and the allowable angle of the glass GL with respect to the water inclination is θ. _{Assuming w} , the retention level is determined as follows.

グラスＧＬの高さh、グラスＧＬの口の半径rは、グラスＧＬに付された電子タグに書き込んでおけばよい。また、グラスＧＬに入っている液体ＬＤの高さhiは、後述する重量センサを利用して測定することができる。すなわち、グラスＧＬの重量w_g及び、グラスＧＬの形状情報を電子タグに書き込んでおく。重量センサによって測定された対象物の重さをw_wとすると、内部に入った液体ＬＤの重量はw_w−w_gとして求められる。ここで、内容物が水であるとすると、内部に入った液体ＬＤの量が推定できる。ここで、電子タグ情報より、グラスＧＬの形状情報を取得できため、これらの情報からグラスＧＬに入っている液体ＬＤの高さhiを推定することができる。

The height h of the glass GL and the radius r of the mouth of the glass GL may be written on an electronic tag attached to the glass GL. Further, the height hi of the liquid LD contained in the glass GL can be measured using a weight sensor described later. That is, the weight w _g of the glass GL and the shape information of the glass GL are written in the electronic tag. When the weight of the object measured by the weight sensor and w _w, the weight of the liquid LD entering the interior is determined as w _w -w _g. Here, when the content is water, the amount of the liquid LD that has entered can be estimated. Here, since the shape information of the glass GL can be acquired from the electronic tag information, the height hi of the liquid LD contained in the glass GL can be estimated from the information.

  Next, an environmental system in which the robot 2 is placed will be described.

  FIG. 2F is a block diagram showing a configuration of an environmental system in which the robot 2 according to the first embodiment of the present invention is active. As shown in FIG. 2F, the environmental system is roughly divided into a work robot 402 functioning as an example of the robot 2 in which the article holding system 10 is installed, a sensing device for grasping the situation in the environment (an example of the recognition device 101). An environment management server 401 serving as a server, and an operation terminal 403 serving as an operation device. Each of the subsystems 401 to 403 includes transmission / reception means 409, and these transmission / reception means 409 exchange information, data, signals, and the like via a wireless or wired network under the control of each control means. Can be performed independently of each other. In addition, in order to perform a common process, each transmission / reception means 409 attaches | subjects the same code | symbol 409 in FIG. 2F.

  In the description here, “environment” is, for example, a room in a house, but the present invention is not limited to this, and means a living space in which articles are arranged.

  The configuration and operation of each subsystem will be described in order.

<Configuration of environmental management server>
The environment management server 401 includes articles and mobile body search / management means 405 that manage articles existing in the environment, a moving object including a person and a robot 402, and articles detected by the sensing device 404, and the article. And an article and moving object database 406 for storing data of the moving object, an environment map managing means 407 for managing the status of the entire environment other than the article and the moving object, and an environment map database 408 for storing data of the entire environment. I have. Output signals from the sensing device 404 are connected so as to be input to the article and moving body search / management means 405 and the environment map management means 407, respectively. The article and moving body search / management means 405 is connected to the article and moving body database 406 and the control means 410. The environment map management unit 407 is connected to the environment map database 408 and the control unit 410. The transmission / reception unit 409 is connected to the control unit 410.

  Based on the control of the control means 410, the transmission / reception means 409 receives an inquiry (signal) of the data of the article and the moving body database 406 and the data of the environment map 408 from the outside, or sends the response signal to the outside. Also, a control command for the robot 402 is transmitted. The control unit 410 independently controls the operations of the article and moving object search / management unit 405, the environment map management unit 407, and the transmission / reception unit 409.

  The sensing device 404 is the position (position coordinates) and state (sitting, going out, standing, sleeping, walking, holding, adsorbing) of articles such as furniture existing in the environment and people and robots 402 existing in the environment. (Grip, release, etc.) is constantly monitored.

  The sensing device 404 can also detect that an article has been brought into the environment or taken out of the environment by a person or a robot 402. Specifically, the sensing device 404 includes a camera (image sensor), a tag sensor, and the like installed in the environment, and details thereof will be described later. When detecting the article or the moving body, the sensing device 404 transmits the detection information to the article and moving body search / management unit 405 and the environment map management unit 407. The information transmitted from the sensing device 404 to the article and moving body search / management unit 405 and the environment map management unit 407 includes, for example, the detection time of the article, the position (positional coordinates) and the direction of the article.

  The article and moving body search / management unit 405 accumulates and manages the article and moving body information detected by the sensing device 404 in the article and moving body database (DB) 406. The information managed in the article and moving body database 406 includes at least information on the current positions (position coordinates) of the article and the moving body. Details of the article and mobile object database 406 and the updating method thereof will be described later.

  Further, the article and moving body search / management unit 405 selects a moving body (person / robot) that is operating the article (handling and / or moving of the article) based on the information from the sensing device 404. Inference is performed, and the inference result is stored in the article and mobile object database 406.

  Further, the article / moving body search / management unit 405 receives an inquiry about the article / moving body database 406 from the control unit 410 to the article / moving body search / management unit 405 as a result of reception by the transmission / reception unit 409 or the like. In accordance with the contents of the inquiry, necessary information is extracted from the article and moving body database 406 and sent to the control means 410.

  The environment map management unit 407 creates an environment map based on the information from the sensing device 404 and stores it in the environment map database 408, and manages the created environment map in the environment map database 408. The environment map stored in the environment map database 408 is used when the robot 402 moves in the environment. The robot 402 acquires the environment map from the server 401 and makes a movement route plan.

  Furthermore, when the environment map management unit 407 receives an inquiry about the environment map database 408 from the control unit 410 to the environment map management unit 407 as a result of reception by the transmission / reception unit 409 or the like, it is necessary depending on the content of the inquiry. Information is extracted from the environment map database 408 and sent to the control means 410.

  The control means 410 is an element that controls the entire environment management server 401. The main control contents include the following.

  1) When the transmission / reception unit 409 receives an inquiry about various data in the environment management server 401, the control unit 410 determines the content of the inquiry, and according to the determination result, the control unit 410 searches for articles and moving objects. A data reference request is issued to the management unit 405 and the environment map management unit 407.

  2) Inquire about the result sent from the article and moving object search management means 405 or the environment map management means 407 to the control means 410 via the transmission / reception means 409 under the control of the control means 410. Send to the original.

  3) Interpret the work content message of the robot 402 transmitted from the operation terminal 403 to the control unit 410 via the transmission / reception unit 409, and generate a robot control command sequence for causing the robot 402 to execute an operation. Then, the data is transmitted from the control unit 410 to the robot 402 via the transmission / reception unit 409. The robot control command sequence will be described later.

  4) As necessary, transmission / reception means for a certain period of time, a part or all of the articles managed in the article and moving body database 406 and the situation of the environment map managed in the environment map database 408 The information is broadcast to the robot 402 and the user (operation terminal 403) by the control unit 410 via 409.

(Specific examples of sensing devices)
The environment targeted by the robot article holding system according to the first embodiment is a living space such as a home. Therefore, in this environment, articles are placed in various places, and their positions are not regular, and their positions change constantly. In addition, a moving body such as a person or a robot can move freely without any restriction on its moving path. For this reason, in the article holding system using the robot, a sensing technique capable of accurately detecting the situation of articles and moving bodies in the environment is required.

Image sensor One of the most commonly used sensors for detecting articles is an image sensor (camera). In order to efficiently monitor a relatively wide area such as the whole room with a small amount of equipment, the image sensor, that is, the camera is fixed to the ceiling or wall of the room, and the camera image (captured image) is used to It is common to detect articles and the like.

  There is a background subtraction method as a general method for detecting articles and moving objects in the environment using camera images. The background difference method is a method of detecting an object by preparing a model image as a background in advance and taking the difference between the current camera image and the model image. In the article holding system using the robot described above, it is necessary to detect and monitor articles and moving objects in the environment. Therefore, if there are few changes in the environmental situation, the model / image contains objects / moving objects in the environment. An image captured when not being used may be used. Further, when the environmental situation changes greatly, an image obtained by averaging a plurality of images taken with a predetermined time interval may be used as a model image.

  However, in article detection using an image sensor, in general, it is weak against changes in brightness, resolution is low, the article is hidden behind other objects (the problem of blind spots), and a plurality of overlapping articles are one article. There are many problems. For example, the problem of blind spots can be solved by arranging a plurality of cameras in the environment substantially evenly so that all articles existing in the environment can be imaged by any one of the cameras. However, it is not always possible to reliably detect an article simply by eliminating the blind spot. In other words, no matter how much the number of cameras is increased, the problem of resolution and the problem of overlapping of articles are not solved, so it is not always possible to identify what is the part that is raised in the background difference image.

-Use of an electronic tag In recent years, a technique for detecting the position (positional coordinates) of an article or a moving body using an electronic tag is being developed. An electronic tag is a device composed of an IC that stores data and an antenna that transmits and receives data wirelessly. A device called a reader / writer can read information written on an electronic tag in a contactless manner or write information in an electronic tag in a contactless manner. Such a reader / writer is called a tag reader.

  Therefore, an electronic tag is attached to each article, and data relating to the article, for example, data such as the kind of article (or article ID information), shape, weight, image of the article, date of manufacture, etc. are embedded in the electronic tag. . In addition, an electronic tag is also attached to a moving body (a person and a robot), and data relating to the moving body, for example, information such as a person's name and date of birth is written. In the case of a person, an electronic tag may be attached to an object that is always carried (for example, a wristwatch or glasses). On the other hand, many tag readers are installed in the environment. By reading the information of the electronic tag attached to the article or the moving body by the tag reader, it is possible to detect the article or the like existing in the environment without a camera.

  In addition, when a camera is used, only the presence of an article is detected. When an electronic tag is used, not only the presence of an article can be detected, but also data of the article embedded in the electronic tag can be used. it can. For example, by using the shape data of the article, the article can be easily held by the robot 402 as described later. In addition, the use of manufacturing date data makes it possible to manage the quality deadline, and the use of article type data makes it easy to find a search item, which brings great advantages to the user.

  However, the article detection using the electronic tag has a problem that the communication distance is short. That is, in order to exchange data between the electronic tag and the tag reader, a very weak radio wave that does not affect the human body must be used, so the communication distance is as short as several tens of cm at most. In order to solve the problem of the communication distance, it may be possible to install a large number of tag readers in the environment. However, since the tag reader is more expensive than the camera, it is not realistic.

-Combination of image sensor and electronic tag As described above, the method using the image sensor and the method using the electronic tag have advantages and disadvantages. Therefore, a method using both an image sensor and an electronic tag can be considered. That is, a hybrid process is performed in which an approximate position (position coordinates) of an article in the environment is specified by the background subtraction method described above, and the article is specified using an electronic tag.

  Two examples of specific processing will be given.

  One example is that a camera is installed on the ceiling or wall in the environment, and the tag reader 404 a is attached to the work robot 402. Moreover, an electronic tag is attached to each article and the moving body. First, the position (position coordinates) of an article in the environment is specified by a background subtraction method using a camera image. Then, the robot 402 is moved to the vicinity of the identified article, and information is read from the electronic tag attached to the article by the tag reader 404a attached to the robot 402 to identify the article.

  Another example is that a camera is installed on a ceiling, a wall, or the like in the environment, and a plurality of tag readers are installed almost uniformly in the environment. This tag reader has directivity with respect to data reading of the electronic tag, and its reading direction is variable. First, the position (position coordinates) of an article in the environment is specified by a background subtraction method using a camera image. Next, a tag reader installed at a position (position coordinates) closest to the identified article is selected, and the reading direction of the tag reader is directed to the article. And information is read from the electronic tag attached to the article, and the article is specified. In this example, since the distance between the tag reader and the electronic tag may be long, it is necessary to use a relatively strong radio wave. For this reason, it is preferable to read information from the electronic tag after confirming that there is no person in the environment, for example, by the background subtraction method.

  For sensing in this system, a method other than the method using the image sensor or the electronic tag described here may be adopted.

(Goods and moving body search management means)
FIG. 2G is a diagram conceptually showing the internal configuration of the article and moving object search / management means 405. In FIG. 2G, the article handling detection means 431 detects that the article is handled by the moving body (article handling state), and the handler identification means 432 handles the article according to the detection result of the article handling detection means 431. Identify the moving body (handler)

  The article handling detection unit 431 detects the article handling state based on information from the sensing device 404. For example, when article detection is performed using the image sensor and the background subtraction method as described above, when a portion having a difference between the camera image and the model image is generated, Detects that an object is being handled. Of course, the article handling state may be detected using, for example, an electronic tag.

  When the article handling detection unit 431 detects the article handling state, the handler identification unit 432 identifies the handler (person or robot) handling the article, and the information on the handler is stored in the article and mobile database 406. To accumulate.

  Specifically, the operator may be specified as follows. When a camera is used as the sensing device 404, an image of the area where the article handling state is detected is captured by the camera. And a face authentication process is performed with respect to a captured image, and a moving body is specified. Since the identified mobile body is considered to have been in the vicinity of the article that has been handled, the mobile body is estimated to be a handler. Note that a wide-angle camera is usually used as the sensing device 404 to capture a wide area, but the resolution of a captured image of the wide-angle camera is relatively low and may not be sufficient for performing face authentication processing. Therefore, apart from the wide-angle camera, a narrow-angle high-resolution camera may be installed in the environment or in the robot 402 as a camera for face authentication processing. By picking up an area where the article handling detection means 431 detects the article handling state with this narrow-angle camera and performing face authentication processing on the captured image, the handler can be identified with high accuracy.

  The identification of the article handler is not limited to the face authentication process, but may be performed by other authentication processes such as iris authentication. Further, the camera image itself may be stored in the article and moving object database 406 without performing the authentication process. This may be performed only when the moving object cannot be identified by the authentication process. Further, the operator may be specified using an electronic tag.

(Article and mobile database)
The article and mobile object database 406 is a database that stores information on articles and mobile objects. For example, an article database (DB) 406b that handles articles as shown in FIG. 2H and a mobile object as shown in FIG. 2I are handled. And a mobile database (DB) 406m.

  The article database 406b in FIG. 2H includes three sub-databases 406b-1, 406b-2, and 406b-3 that store article data, article history data, and article attribute data, respectively.

1) Article Data The article data stored in the article data sub-database 406b-1 includes ID information for distinguishing individual articles (for example, “kan_small — 0001 assigned as an ID for a small can in FIG. 2H). ”), A pointer to the article history data (eg,“ List 1 ”as the position history of a small can in FIG. 2H), and a pointer to the article attribute data (eg, a small can in FIG. Attribute data, such as “kan_small”). In the example of FIG. 2H, different ID information is assigned to different physically different articles even if they are of the same type, and are handled as different articles. However, since the same type of articles have the same physical attributes, the pointers to the ID information and the position history data are different, but the pointers to the article attribute data are common. Thereby, the capacity of the database can be saved.

2) Article history data The article history data stored in the article history data sub-database 406b-2 represents the history of handling articles. In FIG. 2H, the handling time (for example, the position history in FIG. 2H). Time “t1” for list 1), handling contents (for example, operation contents “new” etc. for position history list 1 in FIG. 2H), handler (for example, position history list 1 in FIG. 2H) On the other hand, the operator “dad” or the like, and the position (position coordinates) after handling (for example, the position “x1, y1, z1, l1, m1, n1” after the operation with respect to the position history list 1 in FIG. ), 4 items are included. Although various representations of position data (position coordinate data) can be considered, in the example of FIG. 2H, three parameters (x1, y1, z1) representing the position of the article (using the barycentric position coordinates, etc.) and The position (positional coordinates) is expressed by six parameters including the three parameters (l1, m1, n1) representing the direction of the article. The handler is a moving body specified by the handler specifying means 432.

3) Article attribute data The article attribute data stored in the article attribute data sub-database 406b-3 represents physical attribute information of the article. For example, as shown in FIG. Weight (for example, weight “100 g” for kan_small in FIG. 2H), shape (for example, a perspective view of a cylinder as a shape model size for kan_small in FIG. 2H), appearance image data (for example, FIG. In 2H, for example, image data of an appearance image with respect to kan_small).

  2I includes two sub-databases 406m-1 and 406m-2 that store mobile data and mobile history data, respectively.

1) Mobile data The mobile data stored in the mobile data sub-database 406m-1 includes ID information (for example, “dad” as an ID in FIG. 2I) for distinguishing individual mobiles, and A pointer to moving body history data (for example, “list 3” as the position history in FIG. 2I) is included. The mobile body stored in the mobile body data may be manually registered in advance by the user.

2) Moving object history data The moving object history data stored in the moving object history data sub-database 406m-2 includes the time (for example, time “t1” as the position history list 3 in FIG. 2I) and the position at that time. (Position coordinates) (for example, the position “(x4, y4, z4)” as the position history list 3 in FIG. 2I) and the state at the time (for example, the state “sit” as the position history list 3 in FIG. 2I) Etc.). A moving body (a robot other than the robot 402 or a person), unlike an article, has a large volume in the space, and thus tends to be an obstacle to the movement of the robot 402. For this reason, it is preferable to represent the position (positional coordinates) of the moving body as realistic as possible so that the robot 402 can move while avoiding such obstacles. In FIG. 2I, the area occupied by the moving body on the floor surface is approximated by a circle in order to represent it with the minimum necessary information, and the position (positional coordinates) of the moving body is represented by the center coordinates and the radius of the circle. Of course, it may be expressed more strictly. For example, the outline of the region occupied by the moving object on the floor may be approximated using a plurality of line segment vectors.

  The state of the moving object is represented by general human actions such as “sitting”, “standing”, “sleeping”, “walking”, etc. in the case of a person, and “holding” (or “ This is expressed by an operation performed on the article by the robot 402 such as “gripping” or “adsorption”) or “release”. The robot 402 is expressed as “article ID: operation content” together with the work ID information as well as the operation. Specifically, for example, “kan_small_0001 grasp” is indicated as time t4 in the position history list 5 of FIG. 2I. For example, a plurality of moving object state candidates are prepared in advance in the moving object history data sub-database 406m-2, and the state of the moving object is determined based on the detection result by the sensing device 404, for example. The article and moving object search / management means 405 may determine whether the condition candidate is applicable.

  The article and moving body search / management unit 405 stores information on the article and the moving body in the article and moving body database 406, and stores the information every time the position (positional coordinate) of each article or moving body is changed. Update. Note that the update timing is not limited to this, and the update timing may be set as appropriate, for example, updating every predetermined time.

  The article history data and the moving body history data are preferably stored for as long as possible. Thereby, the history can be examined more retroactively. Further, it is preferable that the mobile history data is accumulated at a time interval as short as possible. Thereby, it is possible to manage the moving path of a moving body such as a person or a robot more finely. However, since the capacity of the database is limited, data for a predetermined period may be stored, and data past that period may be deleted as needed. Further, the time interval for accumulating data may be shortened when the state change of the moving body is severe, and the time interval may be lengthened when the state change is small.

(Environmental map and equipment database)
2J to 2L are examples of environment maps in the environment map database 408. FIG. 2J is an example of an actual environment, FIG. 2K is an environment map obtained by simplifying the actual environment of FIG. 2J with a three-dimensional model, and FIG. 2L is an environment map further simplified with a planar model.

  The environment map may be created according to its use and the time (labor) required for creation. For example, when it is necessary to create an environment map composed of a three-dimensional model in a very short time, a three-dimensional object existing in the environment may be modeled with a minimum rectangular parallelepiped covering it as shown in FIG. 2K. In FIG. 2K, the table Tb and the bookshelf Bs are each modeled as a rectangular parallelepiped, and the trash can Ts is modeled as a substantially cylinder. The same applies to the environment map composed of a planar model. In FIG. 2L, the table Tb and the bookshelf Bs are each modeled by a rectangular area (hatched area) orthogonally projected onto the plane, and the trash can Ts is a circular area (hatched area). Area). These two rectangular areas and circular areas are set as areas where the robot 402 cannot move. Further, an environment map obtained by directly converting a real environment as shown in FIG. 2J into a three-dimensional model may be used.

  FIG. 2M is a diagram showing an example of the equipment database 408e attached to the environment map, and corresponds to the environment of FIG. This equipment database is composed of two sub-databases 408e-1 and 408e-2 that store equipment data and equipment attribute data, respectively.

1) Equipment data The equipment data stored in the equipment data sub-database 408e-1 is the environment itself and the individual equipment in this environment (unlike goods, fixed or installed in the environment, ID information (for example, “room_0001” as an ID in FIG. 2M) and a pointer to equipment attribute data (for example, as attribute data in FIG. 2M) "Room01"). More specifically, in FIG. 2M9, “room_0001” is assigned to the environment (room) as ID information, and “table_0001” is provided as ID information to the table Tb, the bookshelf Bs, and the trash can Ts existing in the environment. , “Bookshelf_0001” and “trash_0001” are attached.

2) Equipment attribute data The equipment attribute data stored in the equipment attribute data sub-database 408e-2 includes the equipment attribute data related to the environment itself and the floor data in the environment. For example, when there are a plurality of floor surfaces having different heights in the environment, the floor surface data (for example, “floor surface 1, floor surface 2” in FIG. Etc.) are accumulated. For example, the floor data is expressed as follows.
((X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3), (X4, Y4, Z4), 2200, 0)
Here, the first four sets of coordinate values represent the real world coordinates of each vertex constituting the floor surface, and the next value (2200) represents the distance (mm) from the floor surface to the ceiling. The last value (0) means the floor material. For example, “0” may be a flooring, “1” may be a tatami mat, “2” may be a carpet, or the like.

Equipment attribute data related to equipment, such as furniture, is recorded on the data (surface 1, surface 2) of each surface constituting the facility, the type of facility, and when the facility has a surface on which an article can be placed. This includes the shape and posture of the main article placed. Specifically, for example, the data of the surfaces constituting the equipment are expressed as follows.
((X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3), 1,400)
Here, the first three sets of coordinate values represent the real world coordinates of each vertex constituting the surface. The next value (1) is a flag indicating whether or not an article can be placed on the surface. “1” indicates that the article can be placed and “0” indicates that the article cannot be placed. The last value (400) indicates the upper limit height (mm) of an article that can be placed when the article can be placed on the surface. For example, when the surface is a table top, the distance from the top to the ceiling is the upper limit height, and when the surface is a shelf on the bookshelf, The distance is the upper limit height.

  The “main article shape” in the equipment attribute data is the shape of the article accommodated in the equipment. If the type of equipment is a bookshelf, it will be “book shape”. That is, a rectangular parallelepiped whose depth and height are extremely long compared to the width is the shape of the main article on the bookshelf. Further, the “posture of the main article” is the posture of the article when accommodated in the facility. If the type of equipment is a bookshelf, it is the posture in which the book is placed on the shelf surface of the bookshelf, and usually the posture in which the book is set up. By accumulating “main article shape and posture” data in the equipment attribute data, for example, when the work robot 402 is designated to move a book to a bookshelf, the work robot 402 is given a “main article shape”. Based on the “and posture” data, it is possible to place the designated book on the shelf of the bookshelf in a posture in which the book is set up.

  However, this “main article shape and attitude” data may not be provided depending on the type of equipment. For example, a table and a trash can have no limitation on the shape or posture of the article. For this reason, the equipment attribute data of the table and the trash can does not have data of “shape and posture of main article”.

<Work robot configuration>
The work robot 402 performs an article handling operation in the environment. Here, in particular, it is assumed that an operation of moving an article in the environment is performed according to a user instruction.

  2A and 2F, the robot 402 is an obstacle sensor that detects an obstacle or the like in the vicinity of the robot 402 (an apparatus that can function as another example of the recognition device 101 of the article holding system 10). 411, the article holding system 10 including the holding device 103 that holds the article, the transmission / reception means 409 and the control means 415 of the robot 402, the transmission / reception means 409 and the control means 410 of the environment management server 401, and the article / moving object search / management A movement plan creating means 413 for making a movement plan for the robot 402 by referring to the environment map in the environment map database 408 of the environment management server 401 via the means 405 and the like, and a moving device 414 for moving the robot 402 itself are provided. Yes. The transmission / reception unit 409 transmits / receives various data to / from the environment management server 401 and the operation terminal 403 under the control of the control unit 415. The control means 415 is connected to the obstacle sensor 411, the transmission / reception means 409, the article holding system 10, the movement plan creation means 413, and the movement device 414, and controls each operation independently.

  FIG. 2A is a schematic diagram illustrating an example of the structure of the robot 402. As described above, the robot 402 includes the substantially box-shaped main body 390 that accommodates the movement plan creation unit 413, the control unit 415, and the like. I have. Hereinafter, the front side of the paper surface in FIG. 2A is referred to as the front side, the back side of the paper surface is referred to as the rear side, the right side of the paper surface is referred to as the left side, and the left side of the paper surface is referred to as the right side.

  As described above, each holding device 103 of the article holding system 10 independently drives the articulated arm 103b, the hand 103a disposed at the tip of the arm 103b, and each arm 103b and the hand 103a independently. It is comprised from the drive device, and is attached to the upper part of the both sides of the main-body part 390. The driving device for each arm 103b and hand 103a may be configured by an actuator controlled by a motor, or may be configured by another actuator, for example, an actuator by an artificial muscle. Further, when the holding device 103 further includes a suction hand 103 s that holds an article by air suction, the holding device 103 is configured by a suction device that controls the suction operation and the suction release operation of the suction hand 103 s and the suction hand 103 s. It doesn't matter.

  Further, in the article holding system 10 using the robot, an electronic tag is attached to each article existing in the environment, and each holding device 103 has a tag reader 404a as an example of the sensing device 404 of the environment management server 401. (See FIG. 2A). Thereby, when the holding device 103 holds the article, the tag reader 404a reads the information written on the electronic tag of the held article, and by the read information, what is the article held by the holding device 103, The article and moving object search / management unit 405 of the environment management server 401 can specify the item and moving object database 406 with reference to the article and moving object database 406. Note that the tag reader 404a attached to the holding device 103 may be omitted, and the tag reader 404a installed in the environment may be used.

  The moving device 414 includes four wheels 394, a motor 304 that drives the four wheels 394 to rotate in the forward and reverse directions, a motor driver 303 that drives and controls the motor 304, and the like (see FIG. 2C). Two 394 are attached to each of the left and right sides of the main body 390 (illustration of the wheel 394 on the right rear side is omitted in the example of FIG. 2A). Note that the configuration of the moving device 414 may be selected in accordance with the environment in which the robot 402 is used. For example, when the undulation of the floor of the environment is severe, it is preferable to configure the moving device 414 in a crawler type or a multi-legged walking type.

  Here, the obstacle sensor 411 includes an ultrasonic sensor 411a, a pair of cameras 105 as a visual sensor, and a collision sensor 411c. Each ultrasonic sensor 411a calculates the approximate distance to the obstacle by measuring the time until the reflected wave is received after emitting the ultrasonic wave. To detect. As an example, three ultrasonic sensors 411a are attached to the lower part of each side surface (front surface, rear surface, left and right side surfaces) of the main body 390. Each camera 105 inputs the situation around the robot 402 as an image. By performing recognition processing or the like on this image, it is possible to determine the presence or absence of an obstacle or to obtain more accurate information about the article to be held. The camera 105 is attached to the upper part of the front part of the main body 390. Further, the collision sensor 411c detects that a predetermined impact force is applied to the robot 402. For example, the collision sensor 411c detects that an obstacle has collided with the robot 402 or that the robot 402 itself has collided with the obstacle while moving. The collision sensor 411c is attached to the lower part of the front surface and the rear surface of the main body 390, respectively.

  The movement plan creation means 413 indicates the movement path from the position coordinates of the current position of the robot 402 to the position coordinates of the target position of the environment map database 408 when the movement of the article or movement associated with other work is designated. Create by referring to the map. In the environment map, as shown in FIG. 2L, a non-movable area (a hatched area) is set. For this reason, if a movement route is created in an area other than the non-movable area, a movement path avoiding an obstacle can be created. For example, in FIG. 2L, when moving from point A to point B, a route that avoids a non-movable region as shown by an arrow line is created in consideration of the size of the robot 402. The most common Dijkstra method may be used to create a movement route. If the environment is complicated, a route search algorithm improved from the Dijkstra method may be used. As a countermeasure when the travel route cannot be calculated because the environment is too complex, or the calculation takes a long time, the user uses the operation terminal 403 to create a travel plan for the travel route of the robot 402. A mode designated in the means 413 may be provided.

(Robot control commands)
The control means 415 of the work robot 402 mainly interprets the robot control command sequence sent from the environment management server 401 via the environment management server side and the transmission / reception means 409, 409 on the work robot side, and sequentially outputs the control commands. Execute.

  The robot control command is a command for controlling the holding of the article and the movement of the robot 402 itself, and is roughly divided into three types: “move”, “hold”, and “release”. These three types of commands will be briefly described.

1) Movement: (move, coordinates) or (move, equipment ID)
This is a command to move from the position coordinates of the current position of the robot 402 to the position specified by the position coordinates or the position coordinates of the equipment position specified by the equipment ID information. The position coordinates are specified in the world coordinate system, and the movement plan creation means 413 plans the movement path from the position coordinates of the current position to the position coordinates of the target position. In addition, when moving to the position coordinates of the position of the equipment specified by the equipment ID information, a route that approaches the equipment to a predetermined distance is created. In this case, the equipment attribute data in the environment map is Use.

2) Retention: (grab, item ID)
This is a command for holding the article specified by the article ID information by the hand 103b as an example of the holding device 103. The position (position coordinates) of the article is obtained with reference to the article and moving body database 406, and the holding plan and the holding release plan are created by the holding method specifying means 102 of the article holding system 10.

3) Release: (release)
This is a command for releasing the hand 103b based on the holding release plan.

For example, when the user gives an instruction to move a certain article to a certain place, the operations are “movement (to the position B1 of the article)”, “(holding of the article)”, “(destination). It is broken down into four units of work: “movement to B2” and “(holding of goods) release”. The robot control command sequence in this case is
move, B1 (moves the robot 402 to the position B1 where the article is placed)
grab, article ID (the article at position B1 is held by holding device 103)
move, B2 (the robot 402 moves to the destination position B2 (with the article held by the holding device 103))
release (releases the item held by the holding device 103)
It becomes. When instructed to move a plurality of articles, the command sequence is arranged by the number of articles with the above four commands as a set, and the control means 415 of the robot 402 executes the control commands in order according to the order.

  Of course, the robot control commands are not limited to these three types, and needless to say, they may be increased as necessary.

  FIG. 21 is a flowchart showing an operation example of the control means 415. The control unit 415 determines which unit of work the control command sequence transmitted from the environment management server 401 is received by the robot-side transmission / reception unit 409 or when the control command sequence is input from the article holding system 10. (Steps S101 to S103 in FIG. 21), and processing according to the unit of work is executed.

  First, when the unit of work is “move” (YES in step S101 in FIG. 21), the route to the designated position (position coordinates) is created by the movement plan creation unit 413 (step S104). Then, a movement control command is sent to the moving device 414 according to the route created by the movement plan creating means 413, and the movement process to the designated position (position coordinates) is executed (step S105).

  When the unit of work is “hold” (NO in step S101 and YES in step S102), the obstacle sensor 411 detects the posture of the object to be held (step S106), and the arm according to the detection result. The operations of 103a and hand 103b are calculated (step S107). Then, a holding control command is sent from the control means 415 to the holding device 103 of the article holding system 10, and an article holding process by the holding device 103 is executed (step S108). In step S106, since the posture of the article is recorded in the article and mobile object database 406 of the environment management server 401, the attitude of the article may be inquired of the environment management server 401.

  When the unit of work is “release” (NO in step S101, NO in step S102, and YES in step S103), the operation of the arm 103a and the hand 103b so that the article is placed at the designated destination. Is calculated (step S109). Then, a release control command is sent from the control means 415 to the holding device 103, and an article release process is executed (step S110).

  When the movement, holding, and release operations are completed, a message to that effect is sent to the environment management server 401, which is the work instruction source, under the control of the control means 415 (step S111). In this way, the work content instructed by the operation terminal 403 is executed by the robot 402.

<Configuration of operation terminal>
The operation terminal 403 is a user interface in the article holding system by the robot, and is a terminal that the user operates to instruct the robot 402 to handle the article and to inquire about the article.

  As shown in FIG. 2F, the operation terminal 403 displays an operation screen, for example, a display device 417 composed of a CRT or a liquid crystal display, and for instructing the robot 402 on the operation content on the operation screen of the display device 417. For example, an input device 416 including a pointing device and display control means 418 for performing display control such as creation of an operation screen displayed on the display device 417 are provided. The transmission / reception means 409 transmits the work contents and inquiry contents of the robot 402 input to the input device 416 to the server 401 and receives an answer to the inquiry from the environment management server 401 under the control of the control means 419. The control means 419 is connected to the transmission / reception means 409, the input device 416, the display device 417, and the display control means 418, and controls each operation independently.

  For example, a general-purpose PC (personal computer) can be used as the operation terminal 403. In this case, the general-purpose PC can be used as the operation terminal 403 by reading a control program that executes each process.

  The display control means 418 includes information transmitted from the environment management server 401, specifically, data of an image captured by the camera as the sensing device 404 in the environment, data stored in the article and mobile object database 406, environment An operation screen is created based on the environment map stored in the map database 408. The created operation screen is displayed on the display device 417.

  That is, in the system as described above, the operation terminal 403 displays an image representing a predetermined living space on the display device 417, and the user gives an operation instruction to the article included in the image from the operation terminal 403. The environment management server 401 refers to the article and moving body database 406 and converts the user instruction transmitted from the terminal 403 into a control command that can be executed by the robot 402 and transmits the control command to the robot 402. By such an article operation method, the user can easily cause the work robot, in other words, the life support robot 402 to perform an accurate operation in a complicated situation in the living space.

  In addition, by the article holding system 10, each holding device 103 causes the object information such as the position (position coordinates), ID information, and shape of the object to be held, and the person information such as the position (position coordinates) and ID information of the person. It is possible to perform a holding operation using

  Hereinafter, the holding of the article by the article holding system 10 will be described with a specific example.

(1. Holding articles that can accommodate objects inside)
As described above, when holding an article such as a glass that contains a liquid, it is preferable to change the holding method depending on whether or not the liquid is inside. In other words, when holding a glass that contains liquid, tilting the glass or applying vibration to the glass will spill liquid, so the glass is as horizontal as possible and does not vibrate. It is preferable to perform such attitude control. On the other hand, when holding a glass in which no liquid is contained, such attitude control is not particularly necessary.

  Therefore, the holding device 103 of the article holding system 10 accommodates an object in the interior thereof by a camera as an example of the recognition apparatus 101 when holding an article (here, a glass) that can accommodate the object. Is recognized (in this case, whether the glass has water). An example of this recognition method is a recognition method using light refraction.

  FIG. 3 is a diagram illustrating a configuration of a recognition apparatus 101 that recognizes the contents of the glass 201 using light refraction. The recognition apparatus 101 includes a light source 104 that projects light onto the glass 201, an imaging apparatus 105 that captures an image projected on the glass 201, and the recognition processing unit that performs recognition processing on information captured by the imaging apparatus 105. 101p. The light source 104 is preferably one that emits light having high directivity. For example, a laser irradiation apparatus can be used. The imaging device 105 may be a camera, for example, as long as it can detect the locus of the laser beam emitted from the laser irradiation device 104. As shown in FIG. 2A, the laser irradiation device 104 is preferably attached to the arm 103g of the robot 402 so that its position (positional coordinates) can be changed. Further, the camera 105 may be configured to be embedded in the face of the humanoid robot 402.

  When recognizing the contents of the glass 201, the recognizing device 101 irradiates the object to be held (here, the light-transmitting glass 201) with the laser beam 104 g by the laser irradiation device 104.

  Usually, the refractive index of gas (air) is different from the refractive index of liquid (water) (for example, the refractive index of air is 1.0 and the refractive index of water is about 1.33). The trajectory of the laser beam 104g changes depending on whether or not it is present. That is, as shown in FIG. 3A, when the glass 201 is empty, both the inside of the glass 201 and the outside of the glass 201 are filled with air. Go straight as shown in 3A. On the other hand, as shown in FIG. 3B, when water 201e is placed in the glass 201, the refractive index of air and the refractive index of water 201e are different, so that the irradiated laser beam 104g is refracted.

  While the laser irradiation device 104 is irradiating the laser beam 104 g, the recognition device 101 images the glass 201 with the camera 105. Thereby, the locus of the laser beam 104g is detected. The recognition processing unit 101p recognizes whether or not the liquid 201e is contained in the glass 201 based on the detection result. Since the refractive index of light varies depending on the object (material), it is possible to detect the material of the object placed inside the glass 201 by detecting the refraction angle of the laser beam 104g.

  The holding method specifying unit 102 specifies the holding method based on the recognition result of the recognition processing unit 101p. Although this specification is performed using the holding method database 102d, the article and mobile object database 406 may be referred to as necessary.

  Specifically, in the holding method database 102d, as described above with reference to FIG. 2E, the holding method is determined for each holding level, but here, for the sake of simplicity, the contents are stored for each article to be held. It is assumed that a table in which different holding methods are described depending on the presence or absence of is provided (see FIG. 4A). Each table stores a situation that can be taken by the article (for example, the presence or absence of contents) and a holding method that is optimal for the situation in association with each other. FIG. 4A shows an example of a table corresponding to the glass 201, and the situations that the glass 201 can take include “contents present” (see FIG. 4D) and “contents absent” (see FIG. 4B). “Use finger-type hand to perform posture control” is set as the holding method corresponding to “contents present” (see FIG. 4E), and “holding hand is used” as the holding method corresponding to “no contents” The posture control is not performed ”(see FIG. 4C).

  Therefore, when the recognition device 101 recognizes that the contents of the glass 201 are present, the holding method specifying unit 102 refers to the holding method database 102d to hold the posture control using the finger type hand. When specifying (selecting) (see FIG. 4E) and recognizing that the contents of the glass 201 are absent, the holding method (FIG. 4C) does not perform posture control using the suction hand by referring to the holding method database 102d. (See) is specified (selected).

  The holding device 103 performs holding of the holding object under the control of the control unit 415 in accordance with the holding method specified by the holding method specifying unit 102. Therefore, when the glass 201 is filled with water (see FIG. 4D), the holding device 103 holds the glass 201 while controlling the posture so that the glass 201 maintains a horizontal posture by the finger type hand 103a. (See FIG. 4E). On the other hand, when the glass 201 is empty (see FIG. 4B), the glass 201 is held by the suction hand 103s (see FIG. 4C). At this time, the attitude control of the glass 201 is not performed.

  Note that in order for the holding device 103 to hold the holding object, the position (position coordinates), shape, and orientation information of the holding object are required. Such information can be acquired by various known methods using, for example, a range finder or stereo vision. Details of the method of acquiring the position (position coordinates), shape, and orientation information of the holding object will be described later, but the article and moving object database 406 may be referred to.

  As described above, an article that can accommodate an object, such as the glass 201 exemplified above, changes its state with respect to time depending on whether the object is accommodated or not accommodated therein. When such an article is held by the holding device 103, the recognition apparatus 101 recognizes whether or not an object is contained inside the article (holding object), and the holding method specifying unit 102 determines whether the object is inside. The holding method is specified depending on whether or not it is accommodated. This makes it possible to optimally hold an article that can accommodate an object in accordance with the situation.

(Modification)
The holding method database 102d referred to by the holding method specifying unit 102 does not need to be included in the article holding system 10. For example, a tag such as an electronic tag is attached to the holding object (each article), Information such as a holding method according to the status of the holding object may be written in advance. In this case, the holding method specifying unit 102 can specify the holding method according to the recognition result by referring to information such as the holding method in the tag attached to the holding object.

  Further, instead of pre-writing information such as the holding method in the tag, the tag may store the address (Internet address) of a server having a database in which information such as the holding method is accumulated. . In this case, the holding method specifying unit 102 specifies the holding method according to the recognition result by accessing the address and referring to information such as the holding method in the database. The medium for storing these information is not limited to the electronic tag. For example, an optical tag, a barcode, a two-dimensional barcode, etc. may be used.

  Further, in FIG. 3A and FIG. 3B, the article holding system 10 recognizes the presence or absence of the contents of the glass 201 based on the locus of the laser beam 104g, but recognition using light refraction is not limited to this. Absent. For example, an object whose relative position (positional coordinates) is known with respect to the camera 105 is imaged through the glass 201, and the position (positional coordinates) on the glass 201 is detected to detect the position of the glass 201. The presence or absence of contents can be recognized. In this case, an object existing in the environment (a room in a general home) can be used as the photographing object. Here, for example, as shown in FIG. 23, the relative position (positional coordinates) is created by attaching a camera 505 to the ceiling 500 of the room and creating map information from the image of the camera 505. The position coordinates in the map information can be obtained by storing in advance in a database such as the article and moving body database 406 and the holding method database 102d.

  Furthermore, the presence / absence of the contents of the glass 201 can be recognized other than using refraction of light.

  For example, the presence or absence of the contents of the glass 201 may be recognized by measuring the weight of the glass 201 using a weight sensor. In this case, the weight of the empty glass 201 needs to be known in advance, but the weight of the empty glass 201 may be embedded in a tag attached to the glass 201, for example. Further, the weight sensor may be attached to the holding device 103 and the glass 201 may be weighed by holding the glass 201 by the holding device 103 before specifying the holding method, or the glass 201 is placed. The weight of the glass 201 may be measured by attaching a weight sensor to a shelf, a table, or the like. Recognition using a weight sensor has the advantage that the object to be held need not be light-transmitting and can be recognized for any article. Moreover, the recognition of the situation using a weight sensor can also be applied when recognizing whether a dish or the like is on the plate when holding the plate or the like.

  For example, the presence or absence of the contents of the glass 201 may be recognized using a water drop sensor. The water drop sensor may be attached to the tip of the arm 103b of the robot 402 so that the sensor can contact the outer surface of the glass 201. In this case, the holding method can be determined by referring to the table of FIG. 4A as “contents present” when it is determined that water droplets are attached, and “no contents” when water droplets are not attached.

(2. Identification of holding method corresponding to changes in form)
Next, holding of an article whose form changes will be described. When holding (holding) an article (for example, paper waste) 202a having a three-dimensional shape, as shown in FIG. 5A, the article is generally sandwiched from above by a finger-type hand 103a.

  On the other hand, when the article is a sheet-like article 202b, for example, a piece of paper or cloth placed on the floor, it is extremely difficult to execute the above holding method. When gripping such paper or cloth, as shown in FIG. 5B, it is efficient to hold it by suction with the suction hand 103s.

  Further, even when the same paper is in the form of a bundle of 100 sheets, it is difficult to hold by suction in consideration of the weight. In this case, as shown in FIG. 5C, it is efficient to sandwich the paper bundle 202c with the finger type hand 103a from the thickness direction.

  Furthermore, even when the same paper is in a form that is crumpled, it is efficient to sandwich the paper from above with a finger-type hand 103a as shown in FIG. 5A.

  Thus, even if the article is the same paper, the optimum holding method varies depending on the form. Therefore, when holding the holding object, the article holding system 10 recognizes the form of the holding object and specifies (selects) an optimal holding method according to the recognition result.

  Recognition of the form of the holding object can be performed by using a tag reader (tag reader, for example, 404a) that receives information of an electronic tag attached to the holding object and a range finder. That is, in this case, the recognition device 101 of the article holding system 10 further includes a tag reader (for example, tag reader 404a) and a range finder (not shown). In addition, here, what is the article (information for identifying the article), and the situation (form) that the article (paper here) can take as shown in FIG. It is assumed that a table associated with a method is stored.

  When recognizing the form of the holding object, the recognition device 101 of the article holding system 10 first reads the information of the holding object by a tag reader (for example, tag reader 404a). By making the electronic tag compatible with anti-collision, the information of the electronic tag attached to each paper can be recognized at the same time even if the holding objects are bundled. In this way, the information on the electronic tag identifies what the holding object is and obtains a table.

  Next, the recognition device 101 of the article holding system 10 detects the shape of the holding object with a range finder. From the detection result, the form of the holding object is recognized by the recognition processing unit 101p.

  The holding method specifying unit 102 specifies the holding method corresponding to the recognized form of the holding object by referring to the table based on the recognition result.

  The holding device 103 holds the holding object according to the holding method specified by the holding method specifying means 102. That is, when the holding object is a sheet of paper 202b placed flat, the holding device 103 holds the paper 202b by the suction hand 103s, and when the holding object is a rolled paper 202a, When the rolled paper 202a is pinched from above by the finger-type hand 103a and the holding object is a bundle of plural sheets 202c, the paper bundle 202c is pinched from the side by the finger-type hand 103a.

  In this way, the object to be held can be held by an optimum holding method corresponding to the form.

  Of course, such a paper holding method is not limited to this, and for example, a combination of a plurality of holding devices 103 may be used. An example of holding this in combination with the suction hand 103s and the finger type hand 103a when the object to be held is a piece of paper placed on the floor will be described with reference to FIGS. 2A and 5D.

  FIG. 2A shows this state. Even if the robot 402 tries to pinch it with the finger type hand 103, this operation is very difficult when the paper 202b is placed on the floor. Therefore, first, the robot 402 holds the paper 202b by suction with the suction hand 103s as shown in FIG. 5D. Thereafter, the robot 402 is controlled so that the suction hand 103s and the finger-type hand 103a are brought closer to each other, and the end of the sucked paper 202b is sandwiched by the finger-type hand 103a from above and below (FIG. 5E). Thereafter, the suction of the suction hand 103s is stopped (for example, the suction device is stopped by the control of the control unit 415), whereby the gripping of the paper 202b by the finger type hand 103a is realized (FIG. 5F).

  FIG. 5G shows a list of holding methods in the robot 402 shown in FIG. 2A, which are stored in advance in the holding method database 102d. As described above, the robot 402 has two finger-type hands 103a and one suction hand 103s.

  First, when using the finger-type hand 103a, a method of holding it with a single-armed finger-type hand 103a (for example, FIG. 5A and FIG. 5C), and holding with a two-armed finger-type hand 103a There are methods (for example, FIG. 9 and FIG. 5H). This holding method using the two-armed two-fingered hand 103a is effective when holding a large object that is difficult to stabilize with a single-armed finger-type hand 103a or a heavy object.

  As another holding method, there is a holding method using the suction hand 103s (for example, FIG. 5B). This is effective for paper placed on the floor surface, which is difficult to hold with the finger type hand 103a, and for an object having a surface with less unevenness.

  A holding method combining the finger type hand 103a and the suction hand 103s is also conceivable. For this, a method of first sucking with the aforementioned suction hand 103s and sandwiching the sucked object between the single-armed finger-type hand 103a or the two-fingered two-handed hands 103a (for example, FIG. 5E), double-arm There is a method of holding the object placed on the two finger-type hands 103a while holding the object with the suction hand 103s. FIG. 5I shows this state. This is effective as a holding method when a large number of books or documents delivered from a person or another robot are transported to another place like a cart.

  Another example of specifying the holding method corresponding to the form of the holding object is clothing. Next, a case where the holding object is clothing will be described with reference to FIG.

  In other words, the holding method of “holding with a finger-type hand” when the clothes are placed for washing, for example, is efficient. On the other hand, it is necessary to avoid a holding method in which a garment is wrinkled when the garment is washed and hung on a hanger or when it is folded by ironing.

  Even in the case of clothing, recognition of the shape of the holding object can be performed using a tag reader (for example, tag reader 404a) that receives information on an electronic tag attached to the holding object and a range finder. . In this case, for example, a table having information as shown in FIG. 7 may be stored in the holding method database 102d in the electronic tag. In the form in which the clothes 210 are hung on the hanger 211 by this table, the portion of the hanger 211 is held by the finger-type hand 103a (for example, the hook portion of the hanger 211 is held so as to be gripped or hooked by the finger-type hand 103a). Is set. On the other hand, when the garment 210a is folded, for example, the holding operation is performed such that the arm 103b or the two finger type hands 103a are used to lift the garment 210a in the folded state. In addition, when the clothing 210 is in another form (for example, a rounded state or the like) 210b, it is set to be held so as to be picked by the finger type hand 103a.

  Note that the process of recognizing the form of the object to be held by the recognition apparatus 101 and the process of specifying the holding method by the holding method specifying unit 102 are as described above, and thus the description thereof is omitted here.

  As described above, the paper and clothing exemplified above are articles whose form changes with time. When the article is held by the holding device 103, the recognition apparatus 101 recognizes the form of the article. The holding method specifying unit 102 specifies a holding method according to the form. This makes it possible to optimally hold an article whose form changes according to the form.

(3. Specifying and changing the holding position according to the condition of the article)
Next, the specification and change of the holding position (position coordinates) according to the condition of the article will be described. This is effective when the robot 402 (for example, a housework support robot) holds the plate 203 on which dishes (food) 204 are arranged. That is, when the dish 203 on which the dish 204 is arranged is held, a portion of the dish 203 where the dish 204 does not exist is held. For this purpose, when the holding device 103 holds the plate 203 that is the holding object, the holding device 103 recognizes the situation of the plate 203 that is the holding object, and specifies an optimum holding method according to the recognition result. Here, when the dish 203 is held, the position (position coordinates) of the dish 203 is recognized at which position the dish 204 exists, and the holding position (position coordinates) is specified according to the recognition result.

  In this case, the status of the article is recognized using a tag reader (for example, tag reader 404a) and a camera (for example, camera 105). That is, a tag reader (for example, tag reader 404a) and a camera (for example, camera 105) as examples of the recognition apparatus 101 are used.

  The recognition of the state of the dish 203 using the tag reader (for example, the tag reader 404a) and the camera (for example, the camera 105) will be described with reference to FIG. 8A. FIG. 8G is a diagram showing the flow of processing.

First, the article holding system 10 captures an image of the plate 203 on which the dish 204 is served by the camera at time t = t ₀ (step S2001). 8A and 8D are views of the dish 203 and the food 204 placed on the dish 203, taken in this manner. It is assumed that an electronic tag is attached to the plate 203, and shape data and texture information (information such as the shape of the plate 203 shown in FIG. 8B and the pattern 203a of the plate 203) is stored in the electronic tag. And

  When recognizing the status of the holding object, the article holding system 10 first reads the information on the holding object by the tag reader 404a (step S2002). Thereby, as shown in FIG. 8B, data of the dish 203 in a state where no dish is placed is obtained. The small circle is the pattern 203a of the plate 203.

  Then, the recognition processing unit 101p of the recognition apparatus 101 executes a difference process between the data shown in FIG. 8A and the data shown in FIG. 8B (step S2003). By this difference processing, as shown in FIG. 8C, only the region 81 of the food 204 placed on the plate 203 is extracted. In FIG. 8C, 82 is an area other than the food area 81 and is an area that can be gripped by the finger type hand 103a.

Next, imaging is performed again at time t = t ₁ when a certain time has elapsed (see FIG. 8E), and a difference image is created (see FIG. 8F). By calculating the difference between the difference images at time t = t ₀ and time t = t ₁ obtained in this way by the recognition processing unit 101p, the state change of the object to be held (here, the sauce applied to the dish is moved) The state change that occurs is estimated (step S2004). FIG. 8F shows the difference area thus obtained.

  Using the above results, the holding position (position coordinates) of the plate 203 by the holding device 103 is estimated (step S2005).

  More specifically, it is processed as follows.

  First, holding position candidates (position coordinate candidates) as shown in FIG. 8H are stored in advance as holding position information in the electronic tag of the plate 203, the holding method database 102d, the article and moving body database 406, and the like. . In FIG. 8H, each of the areas H1 to H12 shows a holding position candidate area 2101 that is a peripheral portion of the plate 203 that can be gripped by the finger type hand 103a of the holding apparatus 103. As an example, the holding position candidate regions H1 to 12 are regions in which the peripheral portion of the plate 203 is equally divided. The holding position information is preferably held as shape information in the electronic tag of the plate 203, the holding method database 102d, the article and moving body database 406, or the like.

  Next, the difference image obtained in step S2003 is superimposed on the holding position information.

  At this time, an area that overlaps the food area 81 in the holding position candidate area 2101 is excluded from the holding position candidates. 8I and 8J are diagrams showing this state. In FIG. 8I, since the region H2 and the food region 81 overlap each other, the H2 region is excluded from the holding position candidates and is set as a holding impossible region.

  At this time, it is desirable to use the latest food region 81 to be superimposed.

  Next, the difference area information obtained in step S2004 is superimposed on the holding position information. Similarly, the barycentric position (positional coordinates of the barycentric position) Pf of the food area 81 is also superimposed on the holding position information. Here, the region (the hatched region 2102 in FIG. 8J) that connects each point of the difference region and the gravity center position Pf of the food region and further extends to the edge of the dish 203 is a region that is likely to become the food region 81 in the future. It is thought that. Therefore, these areas 2102 are also set as non-holdable areas.

  From the above processing, a holdable area is determined from the remaining holding position candidates. In FIG. 8J, it is the regions H4 to H12. Preferably, if the regions H6 to H9 opposite to the non-holdable region 2102 are set as the holdable regions, even if the food moves during the movement, the finger-type hand 103a The possibility of contact with can be further reduced.

  Of course, the holdable area does not need to be determined by the above method. For example, the area other than the food area 81 in the entire area of the dish is specified as the holdable area 82 (the hatched area in FIG. 8K). It does not matter (see FIG. 8L). That is, FIG. 8L is a diagram in the case where the difference area is the holdable area and the no difference area is the non-holdable area in the entire area of the dish. Further, the holding position candidate area 2101 (the area H7 in FIG. 8I) that is farthest from the food area 81 may be selected as the holding position (position coordinate).

  The holding device 103 holds the holdable area 82 by the arm 103b and the hands 103a and 103b according to the specified holdable area 82 (see FIG. 9). In this way, the dish 203 on which the dish is arranged can be held without touching the dish.

  By specifying the food area by the difference process, the food area 81 can be accurately recognized even when the pattern 203a is drawn on the plate 203, for example. In addition, since the luminance information of the plate 203 changes depending on the illumination state, it is desirable to use a background difference method corresponding to the change of the light source. As such a method, for example, there is a method of obtaining a difference after extracting an edge of an image.

  Thus, for example, an object such as a dish may have an object placed on it, and the situation changes with time. When such an article is held by the holding apparatus 103, the recognition apparatus 101 recognizes whether or not an object is placed on the article (holding object) and where the object is placed, and a holding method. The specifying unit 102 specifies the holding method (here, the holding position (position coordinates)) according to the position (position coordinates) where the object is placed. Thus, the article on which the object is placed can be optimally held according to the situation.

  In addition, while holding the holding object by specifying the optimum holding position (position coordinates) as described above, the holding position ( (Position coordinates) may be changed.

  That is, as shown in FIG. 9, while the region other than the food region 81 is held and the robot 402 is carrying the plate 203, the sauce, ketchup, etc. included in the dish move, and as shown in FIG. 10. In this way, it is assumed that the food area 81 has changed. In that case, the hand 103a may touch food.

  Therefore, the recognition apparatus 101 captures the dish 203 on which the dish is arranged by the camera 105 while holding the holding object, and sequentially executes the above difference processing. Thus, the food area 81 is sequentially recognized.

  Then, the holding method specifying means 102 re-specifies the holdable area 82 when the food area 81 changes based on the recognition result. For example, as shown in FIG. 11, the holding position by the hand 103a as shown in FIG. Change (Position coordinates).

  As described above, even after the optimum holding method is once specified, the holding method can be always optimized in accordance with the change of the situation by continuing the recognition of the situation of the holding object.

(4. Specifying the holding method according to the temperature condition of the article)
Next, description will be given of the holding method, particularly the specification of the holding position (position coordinates) by recognizing the temperature state of the article. This is effective when an electronic tag or the like is not attached to the article and a holding method (holding position) of the article is not obtained in advance. That is, when the holding method is obtained by an electronic tag or the like, the holding position (position coordinates) of the article can be specified.

  Here, the case where the robot 402 (household support robot) performs the operation of lowering the pan or the frying pan 121 hung on the stove 511 from the stove 511 will be described as an example of specifying the holding method according to the temperature state of the article. To do.

  Here, as shown in FIG. 23, a recognition apparatus 101, a holding method specifying unit 102, a holding apparatus 103, and a holding method database 102d or a holding information database 510 functioning as an article and moving body database 406 are provided independently. Consider a configuration in which these are connected to each other via a network. In the system shown in FIG. 23, the robot 402 is the holding device 103 itself. The holding method specifying unit 102 receives a recognition result from the recognition device 101, specifies a carrying method, and outputs a carrying instruction to the holding device 103 (robot 402) based on the specified method. It becomes.

  First, the recognition apparatus 101 recognizes the shape of the frying pan 121 applied to the stove 511 using a three-dimensional measurement method such as range finder or stereo vision. That is, the article holding system 10 further includes a range finder or a stereo camera (for example, 105). Specifically, the recognition apparatus 101 acquires, for example, the three-dimensional shape information of the stove 511 on which the frying pan 121 is not placed in advance, and compares the stove 511 with the actually measured three-dimensional information. The shape can be specified (see FIG. 12A).

  Next, the recognition apparatus 101 measures the temperature distribution of the holding object using a temperature sensor 520 such as an infrared sensor. That is, the article holding system 10 further includes a temperature sensor 520. Such a temperature sensor 520 is widely known as one using a thermopile element, a thermography device, or the like, and thus detailed description thereof is omitted. Essentially, such a temperature sensor 520 cannot estimate an accurate temperature when the emissivity of the object is not known. Therefore, the radiation rate data of the target object may be stored in a tag 521 such as an electronic tag attached to the holding target object (each article).

  FIG. 2A is a schematic diagram in which a temperature sensor 520 is installed in the article holding system 10. As shown in FIG. 2A, the temperature sensor 520 may be operated by the hand 103a instead of incorporating the temperature sensor 520 into the article holding system 10.

  A cooking utensil used by a person is provided with a part held by the person (in the case of the frying pan 121, a handle part 121a), and the part is designed so that the temperature does not increase. Therefore, by measuring the temperature distribution of the object to be held, as shown in FIG. 12B, it is detected that the portion of the frying pan 121 that is lit is hot and the handle 121 a is relatively cold.

Based on this temperature distribution, the holding method specifying means 102 uses the holdable region 122 (in this case, the handle 121a where the temperature T is lower than the threshold temperature Th _TH (for example, 30 ° C.)) as the holding position (position coordinate). Specify (see FIG. 12D). A portion where the temperature T is higher than the threshold temperature Th _TH is a non-holdable region.

  As shown in FIG. 12C, the holding device 103 holds the handle 121a of the frying pan 121 by the hand 103a and the arm 103b according to the specified holdable area 122.

  In this way, the robot 402 holds the same position (position coordinates) as the position (position coordinates) held by a person. This is a necessary process when, for example, the configuration of the hand 103a is weak against heat. Even if the configuration of the hand 103a does not need to consider heat, for example, if the position (position coordinates) held by the robot 402 is a position (position coordinates) that is not normally held by a person, it may not be possible for some people. Although it is anticipated that a pleasant feeling will be memorized, there exists an advantage that such discomfort is reduced when the robot 402 holds the same position (position coordinate) as a person.

  The holding position (gripping position) (positional coordinates) based on the temperature distribution can be specified not only on the cooking utensil but also on a plate where dishes are served. In other words, the temperature distribution of the dish on which the dish is arranged is measured, and the holding method specifying means 102 may specify the part having a low temperature, in other words, the part of the dish on which the dish is not placed, as the holding position (position coordinate). .

  In addition, in order to acquire the temperature state of the holding object, the recognition apparatus 101 may use history information or position information of the article. This will be described in detail with reference to FIG.

  When the holding object is the above-described frying pan 121, the temperature of the holding object rises because it is put on a gas stove. Therefore, it is possible to estimate the temperature state of the holding object by confirming whether or not the holding object has recently been present on the gas range. Specifically, by referring to the table in FIG. 12F, the holding method is determined by specifying the holding area 122 as the holding position. Such a method for acquiring the history information of the article may use an RFID, a camera, or the like, and is known from, for example, Japanese Patent Application Laid-Open No. 2004-249389, and thus the description of the method is omitted here.

  Moreover, you may make it change the temperature of the holding | maintenance apparatus 103 according to the temperature condition of a holding target object. This is particularly effective when holding a low temperature. For example, when holding ice, if the temperature of the holding device 103 is sufficiently high, the ice melts and cannot be held.

  Therefore, when the temperature of the holding object is sufficiently low, a process of cooling the temperature of the holding device 103 to a temperature sufficiently close to the temperature of the holding object is performed. Such a cooling process may use a known Peltier element or the like.

  As described above, when the holding device 103 holds an article such as a cooking utensil or tableware in which temperature distribution occurs, that is, an article whose state changes with time, the recognition apparatus 101 recognizes and holds the temperature distribution of the article. The method specifying unit 102 optimizes holding by specifying the holding position (position coordinates) according to the recognized temperature article.

Note that the specification of the holding method according to the temperature state of the article is not limited to the specification of the holding position (position coordinates) as described above. For example, when holding a pan containing hot water, an optimal holding method according to the temperature of the hot water may be selected. That is, for example, when hot water is boiling (as shown in FIG. 12E, the temperature t is lower than the predetermined upper limit temperature Th _TH and is equal to or higher than the predetermined lower limit temperature Th _TL ), the holding method specifying means 102 While the holding method is selected such that the moving method is moved at a slow moving speed while maintaining a horizontal state, when the temperature t is lower than the predetermined lower limit temperature Th _TL , the holding method specifying means 102 moves so as not to spill hot water. Set to speed. When the temperature t is higher than the predetermined upper limit temperature Th _TH , the holding is impossible. Moreover, since the hot water temperature changes with time, the hot water temperature is sequentially measured by the recognition device 101 while holding the hot water pot, and the holding method specifying means 102 changes the temperature. The holding method may be optimized accordingly.

(5. Specifying the holding method according to the relationship between the article and surrounding articles)
Next, the specification of the holding method according to the relationship between the article and the surrounding articles will be described. This relates to a holding method when holding a plurality of articles together. When the robot 402 carries a plurality of articles, it is possible to hold and carry the articles one by one. However, since the work becomes inefficient, the plurality of articles are held together. Carry it. Then, the holding method is optimized in each of the case where the holding object is a single article and the case where a plurality of articles are collected as a holding object.

  The recognition of the status of the object to be held, here, whether or not a plurality of articles are stacked, can be recognized by reading information of an anti-collision compatible electronic tag attached to the article. That is, the recognition apparatus 101 may recognize whether another article exists in the vicinity of the holding object based on the information of the electronic tag. In this case, the holding device 103 further includes a tag reader 404a.

  The recognition device 101 can also recognize whether or not a plurality of articles are stacked based on an image captured by the camera 105.

  For example, when holding only one plate, the holding method specifying unit 102 selects a holding method such as picking with a finger type hand 103a, while when a plurality of plates are stacked, Using the hand 103a, a holding method is selected so as to support the stacked dishes from below (see FIG. 24).

  The holding device 103 holds the holding object in accordance with the holding method specified by the holding method specifying means 102.

Whether or not an article is stacked is a situation that changes with time, but by recognizing such a situation, a holding method according to the situation is specified, and an article is held with an optimum holding method according to the situation. Retention is performed.
(6. Specifying retention method using learning)
It is difficult to set a holding method for each article in advance, and for example, it is further difficult to set a holding method corresponding to the article situation shown in FIGS. 4A, 6 and 7 in advance for all articles. . On the other hand, in order to optimize the holding method for each article and for each article situation, it is necessary to set the holding method for each article and each article situation.

  Therefore, in order to easily and accurately set the holding method, the article holding method may be learned and set based on an action performed by a person.

  FIG. 13 is a modified example of the above embodiment of the present invention, and shows a block diagram of the article holding system 11 that performs learning. The article holding system 11 is compared with the article holding system 10 shown in FIG. Learning means 106 is further included. The learning means 106 recognizes the holding operation of the article performed by a person, and learns and sets a holding method for each article or each article situation based on the recognition result. The recognition device 101 recognizes a holding operation of an article performed by a person.

  A holding operation of an article performed by a person can be recognized using a stereo camera and a tag reader. That is, the article holding system 11 further includes a stereo camera 105 and a tag reader 404a.

Next, the two learning methods will be specifically described. It is assumed that an electronic tag storing information such as object shape information and texture information is attached to all articles that can be held objects. Further, since the person who operates the article always carries the electronic tag, the tag reader 404a can acquire the position (positional coordinates) and ID information of the article and the person.

  One learning method is a learning setting of a holding method based on the moving speed of an article. FIG. 25 shows the flow of processing at this time. In this method, first, when a person is holding an article, the recognition apparatus 101 captures an image of the held state with a stereo camera and obtains an optical flow from the captured image (step S201).

  Here, if there is no movement within the imaging range (no movement of the electronic tag), all detected optical flows are very small.

  On the other hand, when there is a motion within the imaging range (the electronic tag has moved), the optical flow in the region where the motion has occurred becomes large. Therefore, the size of the optical flow in the region is checked (step S202). If there is no sufficiently large optical flow, it is determined that there is no movement in the imaging range, and the process ends (No in step S202). ).

  On the other hand, if a sufficiently large optical flow exists, an area where the large optical flow exists is extracted as a motion area (Yes in step S202).

  Next, the recognition device 101 uses a tag reader to detect what is moving in the area where the movement has occurred. That is, by receiving the information of the electronic tag, it is specified what the article is being moved by a person. If both the article and the person's electronic tag are not detected in the motion area, it is considered that it is not currently necessary to perform learning in the motion area, and the process ends (No in step S203). On the other hand, when both the article and the electronic tag of the person are detected (Yes in step S203), learning is performed because the article that can be held is currently moved by the person in the movement region.

  First, object shape information and texture information are acquired by acquiring information on an electronic tag existing in the motion region (step S204). Only the object is extracted from the captured image of the motion region, and the movement, specifically, the moving speed of the object is measured (step S205). Detecting the position (positional coordinates) of an object in the image by performing pattern matching processing on the stereo camera image using the object shape information and texture information acquired from the electronic tag, and measuring its movement Can do. Since a known method can be used for this, details are omitted (see, for example, Japanese Patent No. 2961264).

  When the object is small, there may be a case where the person is holding the object and the object is hidden behind the person's hand and cannot be recognized in the image. In such a case, an area of a human hand may be extracted from the image and its moving speed may be measured. The measured moving speed may be the movement of the object. For the extraction of the human hand, for example, only the skin color region in the image may be extracted using color information. While performing this extraction process in each frame, the movement of the object can be measured by comparing the positions (positional coordinates) between the frames.

  Of course, the stereo camera and the tag reader as described above do not need to be directly installed in the article holding system 11. Information installed in the environment management server 401 and detected by the environment management server 401 may be transmitted and received using the transmission / reception means 409.

  Through the above processing, it is possible to measure the moving speed of the article while the person is holding the article. For example, as shown in FIGS. 14A and 14B, the moving speed of the stuffed animal (see FIG. 14A) It turns out that it is higher than the moving speed (see FIG. 14B). In other words, the person is moving the stuffed animal quickly while the vase is moving slowly. The moving speed of the article can be acquired by using the above-described tag sensor using ultrasonic waves.

  The learning means 106 sets the holding method of each article in a table format based on the recognition result, for example, as shown in FIG. 26, and the holding method specifying means 102 determines the result (the holding object is recognized) by the recognition device 101. Based on the recognition result), the holding method learned and set by the learning means 106 is specified. Then, as shown in FIG. 26, when the holding object is a stuffed animal, a relatively fast moving speed holding method is specified, and when the holding object is a vase, a relatively slow moving speed holding method is specified. Identified. As described above, this is a database such as the article and moving object database 406 and the holding method database 102d in a table format as shown in FIG. This is realized by using the database.

  Another learning method is a learning setting of a holding method based on where the position (positional coordinates) of a human hand exists with respect to an article.

  This learning method will be described with reference to FIGS. 15A and 15B. FIG. 15A shows a situation when a person is holding a stuffed animal 151, and FIG. 15B shows a situation when a person is holding a vase 152.

  When the person is holding an article, the recognition device 101 causes the stereo camera to capture an image of the held state. As described above, by identifying the positions (position coordinates) of the human hands 153 and 154 and the position (position coordinates) of the article in the captured image, it is detected how the person holds the article. The

  In FIG. 15A, one human hand 153 exists in the vicinity of the stuffed animal 151, but the other hand 154 exists at a position (positional coordinates) away from the stuffed animal 151. From this, the person can recognize that the stuffed animal 151 is held with one hand. On the other hand, in FIG. 15B, two human hands 153 and 154 exist in the vicinity of the vase 152. From this, it is recognized that the person holds the vase 152 with both hands.

  Based on the recognition result, the learning means 106 sets a holding method for holding the stuffed animal 151 by one hand, and sets a holding method for holding two portions of the vase by two hands for the vase 152. To do. This learning method is desirably employed when the holding device 103 has a double-arm configuration similar to that of a person.

  27 and 28 show an example in which the holding method learned in this way is stored in a table format in a database such as the article and moving object database 406 and the holding method database 102d. In the electronic tag of the object, one holding level among the holding levels shown in FIG. 28 is stored as shown in FIG. By learning such a database, a flexible holding operation can be realized for each holding object.

  Further, in order to use the single arm / double arm and the two holding methods properly according to the scene, a database such as the article and moving body database 406 and the holding method database 102d may be created as shown in FIG. This is effective, for example, when another object is held while an existing object is held by a single arm. That is, when holding one object, a double arm is used, but when holding two objects, a single arm is used to hold the objects separately. Thereby, even when there are a plurality of holding objects, the object can be held by an optimum holding method.

  The learning by the learning means 106 described above is only learning the holding method for each article. However, by further recognizing the situation of the article when a person is holding, the situation of the article and the holding method It is also possible to learn in correspondence. For example, when a person is holding a glass, by further recognizing whether or not the glass is filled with water, the holding method when the glass is filled with water, or when the glass is not filled with water You can learn how. By performing such learning, a database as shown in FIGS. 4A, 6 and 7 can be created by learning.

  By setting the holding method by learning, it is not necessary to preset the holding method for each article or each situation of the article, while by learning the holding method based on the holding operation performed by a person, Appropriate holding method can be set.

  The learning by the learning means 106 may be performed when a person hands over an article to the robot 402 and instructs a work on the article. In other words, the robot 402 recognizes the operation when the person hands over the article to the robot 402 and learns the holding method, and the robot 402 performs the subsequent operation with the same operation as that person.

  By performing learning at this timing, there is an advantage that learning accuracy increases. That is, when a person hands over an article to the robot 402, the person often faces the robot 402. Since the stereo camera is usually designed so that the measurement of the directly facing object is highly accurate, the position of the hand (positional coordinates) is obtained by performing learning when a person hands the holding object to the robot 402. In addition, the detection accuracy of the position (positional coordinates) of the object to be held increases and the learning accuracy increases.

  Moreover, although the learning method mentioned above is performed based on the image which imaged the condition where the person is holding the articles | goods, it is not restricted to this. For example, it is possible to attach an ID tag capable of detecting a three-dimensional position (position coordinates) to each article that can be a held countermeasure, and perform learning based on the tag information. That is, since it is possible to detect the movement trajectory of the object based on the tag information, it is possible to detect how the object is held (the movement speed) based on the movement trajectory. As a tag capable of detecting a three-dimensional position (position coordinates), a tag using an ultrasonic wave is known (for example, “a robust measurement of a person's daily activities using an ultrasonic three-dimensional tag—a redundant sensor” Robust location estimation based on information ", Yoshifumi Nishida, Hiroshi Aizawa, Toshio Hori, Masayoshi Sasakura, Proc. 20th Annual Conference of the Robotics Society of Japan, 3C18, 2002).

  Also, for example, “Recognition of grasped shape by visual volume intersection method using infrared image”, Kentaro Hashimoto, Koichi Ogawara, Satoshi Takamatsu, Katsushi Ikeuchi. Computer vision and image media 135-10, pp.55-62, 2002) can also be used to detect and recognize how a person is holding an article. is there. Even when this method is employed, it is desirable that the holding device 103 includes the arm 103b and the hand 103a in the same manner as a person.

  Furthermore, learning by the learning means 106 is preferably performed only when a specific person is a learning target and the person holds the article. This is because the holding operation differs depending on the person even for the same article. For example, even if the article to be held is the same stuffed animal 151, the child often holds it messy, while the mother holds it carefully, and the grandmother holds it slowly. In other words, since there are individual differences in the holding operation of an article by a person, the individual difference is included when learning the holding method of the article based on the action performed by the person. From this, a specific person is set as a learning target, and learning is performed when the person holds the article. As a result, individual differences can be excluded from the holding method that is set for learning.

  In contrast to this, it is possible to individually learn the holding operation of the article by each of a plurality of people, and to select an optimal holding method from among them. Various methods can be considered as a method for selecting the optimum holding method. For example, various learned holding methods may be actually tested and selected by evaluating the stability of the holding method. Further, various learned holding methods may be actually tried, and selection may be made according to the reactions of the people present at that time. That is, it is possible to recognize the pleasantness / discomfort of a person existing in the surroundings from the face image and select a holding method that the person feels most comfortable.

  Furthermore, the holding | maintenance operation | movement of the goods by each of several persons may be learned separately, and the holding | maintenance method corresponding to the person who becomes the object at the time of the goods holding system 11 performing holding | maintenance of goods may be selected. For example, when the robot 402 performs an operation of handing an article to a person, the article is held according to the holding method learned from the holding operation of the person to whom the hand is delivered. By doing so, the robot 402 holds the article as the other person normally holds, and the holding operation of the robot 402 can be individually applied.

(7. Analogy of holding method)
Next, specifying the holding method by analogy will be described. This process also has an advantage that it is not necessary to set a holding method in advance for all holding objects.

  This process will be described with a specific example. Here, the holding method of the dish and the holding method of the stuffed toy are each set in advance (may be set by learning described above, or may be set in advance by the user or the manufacturer). Suppose that the cup retention method has not yet been set.

  Then, when the recognition apparatus 101 recognizes that the object to be held is a cup, the holding method specifying unit 102 holds the cup that is considered to be the best holding method among the already set holding methods. Select a method. In this selection, the holding method specifying means 102 may use, for example, the reflection parameter or shape data of the article. Here, the reflection parameter is a parameter that depends on the material of the article, and the closer this parameter is, the closer the material can be determined (the reflection parameter will be described later). In other words, articles having similar materials or shapes are also considered to have the same holding method, and the holding method specifying unit 102 can use a material or an item for an article (unset article) for which no holding method is set. A holding method set for an article having an approximate shape is specified as the holding method for an unset article.

  Specifically, in the above example, the holding method specifying unit 102 compares the reflection parameter of the cup, the reflection parameter of the dish, and the reflection parameter of the stuffed animal. Since the reflection parameter of the cup is closer to the reflection parameter of the dish than the reflection parameter of the stuffed animal, the holding method specifying means 102 employs a dish holding method as the cup holding method. In this way, by specifying the holding method using analogy, it is possible to optimally hold various articles by the holding device 103 even if the holding method is not set in advance for all holding objects. it can.

(Second Embodiment)
FIG. 16 is a block diagram showing a basic configuration of the article holding system 12 in the second embodiment of the present invention. The article holding system 12 of the present embodiment includes a recognition device 101, a holding method specifying unit 102, and a holding device 103. The recognizing device 101 includes an imaging device 105 that captures an object to be held, and a material recognizing unit 107 that recognizes the material of the object from the captured image.

  In the article holding system 12 of the second embodiment, the same means and devices as those of the article holding system 10 of the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  The imaging device 105 captures an object to be held using an imaging element such as the camera 105.

  The material recognizing means 107 first estimates the reflection parameter of the object from the image captured by the imaging device 105, and then recognizes the material from the estimated reflection parameter. This utilizes the fact that the reflection characteristics change depending on the material.

  The holding method specifying unit 102 specifies a holding method based on the material recognized by the material recognizing unit 107.

  The article holding system 12 of this embodiment is also mounted on a robot 402 as shown in FIG. 2A. In this robot 402, the camera as the imaging device 105 may be embedded in the face of the humanoid robot 402. Further, by attaching a light source 104 (in this embodiment, it is not necessary to use a laser irradiation apparatus as in the first embodiment) to the arm 103g of the robot 402, the position (positional coordinates) thereof is illuminated. It is desirable to change arbitrarily.

  Next, a method for estimating the reflection parameter of the object from the photographed image by the material recognition means 107 and recognizing the material from the estimated reflection parameter will be described in detail. For this purpose, the Torrance-Sparrow model, which is a widely used reflection model, will be described first.

  The Torrance-Sparrow model, which is one of the dichroic reflection models, expresses object reflected light as the sum of specular reflection components and diffuse reflection components. (Expression 1) is a simplified Torrance-Sparrow model expression.

（式１）における第１項は拡散反射成分を表し、第２項は鏡面反射成分を表している。ここで、ｃはＲＧＢ（赤緑青）のいずれか、ｉ_cは画像面の明るさ、Ｌ_cは光源の光度、Ｒは光源から物体表面の各点までの距離であり、照明光強度がこの距離の２乗に反比例して減衰する特性を考慮して導入されたものである。Ｋ_d,cとＫ_s,cとはそれぞれ、拡散反射成分と鏡面反射成分に対応する反射係数、σは表面粗さを示す。また、θ_iは物体表面の点における入射角であって、法線方向と光源方向とがなす角度、θ_rは法線方向と視線方向とがなす角度、αは光源方向と視線方向の２等分方向と法線方向とがなす角度である。

The first term in (Expression 1) represents a diffuse reflection component, and the second term represents a specular reflection component. Here, c is one of RGB (red, green and blue), i _c is the distance in the brightness of the image plane, L _c is the light source intensity, R represents a light source to each point of the object surface, the illumination light intensity is the This is introduced in consideration of the characteristic of attenuation in inverse proportion to the square of the distance. K _{d, c} and K _{s, c} are reflection coefficients corresponding to the diffuse reflection component and the specular reflection component, respectively, and σ is the surface roughness. Θ _i is an incident angle at a point on the object surface, and is an angle formed by the normal direction and the light source direction, θ _r is an angle formed by the normal direction and the line-of-sight direction, and α is 2 of the light source direction and the line-of-sight direction. It is an angle formed by the equally dividing direction and the normal direction.

とすると（式１）は以下のように変形される。

つまり、Ｋ_d、Ｋ_s及びσのパラメータを推定することができれば、対象物の反射特性を記述することができ、対象物の材質を推定することができる。そこで、これらのパラメータを反射パラメータと呼ぶ。 These relationships are shown in FIG. In this figure, reference numerals 200, 105, and 104 denote a holding object, a camera, and a light source, respectively, vector N denotes a normal direction of the surface of the holding object 200, and vectors V, L, and H denote a camera direction, a light source direction, and The bisector direction between the camera direction and the light source direction is shown. here,

Then, (Equation 1) is modified as follows.

That is, if the parameters K _d , K _s, and σ can be estimated, the reflection characteristics of the object can be described, and the material of the object can be estimated. Therefore, these parameters are called reflection parameters.

The estimation of the reflection parameter consists of the following three steps.
1) Precise position (position coordinates), shape and orientation estimation of holding object 2) Separation of diffuse reflection area and specular reflection area 3) Estimation of reflection parameters Each step will be described below in order.

1) Estimating the exact position (positional coordinates), shape, and orientation of the holding object Assume that an electronic tag storing the shape is installed on the holding object.

  Here, a method for obtaining the exact position (position coordinates), shape, and orientation of the object by obtaining the correspondence between the shape information and the three-dimensional position information of the holding object obtained from the stereo vision will be described. The three-dimensional position information of the holding object is information necessary not only for estimating the material of the holding object but also for holding the holding object as described above.

  Stereo vision using a block matching method is widely used as a three-dimensional position detection method using an image. In this method, a target is photographed with a plurality of cameras, and correspondence between images is obtained for each divided block (point). A three-dimensional position (position coordinates) is obtained from the corresponding points thus obtained.

  When the number of corresponding points to be obtained is sufficiently large, the shape of the object can be obtained by this method, but due to processing time and matching accuracy, only a three-dimensional position (positional coordinates) that is sparse with respect to the object is obtained. There are many cases where it cannot be obtained. Therefore, while the necessary information is the position (position coordinates), shape, and orientation information of the object, there is a problem that what is obtained by this method is a three-dimensional position (position coordinates) of a plurality of points. That is, a means for obtaining the exact position (position coordinates), shape, and orientation of the holding object from the set of points for which the three-dimensional position (position coordinates) has been obtained is required. Therefore, the surface shape information stored in the electronic tag is used to estimate its position (positional coordinates) and posture.

  18A to 18E are conceptual diagrams regarding this processing. Each point in FIG. 18A indicates a point whose three-dimensional position (positional coordinate) is obtained by the block matching method. FIG. 18B shows the shape information of the holding object stored in the electronic tag. Here, it is assumed that the object to be held is a cylinder having a radius r and a height h.

  All points for which the three-dimensional position (positional coordinates) is obtained by the block matching method are considered to be points on the surface of the object. Therefore, the shape data obtained from the electronic tag is virtually arranged in order in the three-dimensional space while gradually changing the position and orientation (see FIGS. 18C, 18D, and 18E). Then, the number of points existing on the surface of the virtual shape data among the points whose three-dimensional positions (position coordinates) are obtained by the block matching method is counted (circled in FIGS. 18C, 18D, and 18E). Point). Then, it is estimated that the position (positional coordinate) and orientation at which the number of counted points is the maximum are the position (positional coordinate) and orientation of the object (FIG. 18D in the example).

  Even if the electronic tag is not installed on the object to be held and its shape is unknown, high precision 3 such as laser range finder, ultrasonic sensor, optical cutting method, high precision stereo vision, etc. By using the dimension measurement method, it is also possible to directly determine the position (positional coordinates), shape, and orientation of the object.

2) Separation of diffuse reflection area and specular reflection area Next, an image obtained by photographing an object is separated into a diffuse reflection area and a specular reflection area. Here, a method of dividing a region using a plurality of images (see FIGS. 19A to 19D) captured while fixing the camera 105 and moving the light source 104 is used (“image using optical phenomenon classification criteria”). ”, Ikui Ishii, Fukui Kotaro, Mukaikawa Yasuhiro, Shaku Naga Takeshi, Image Recognition and Understanding Symposium (MIRU2002), vol.2, pp.167-176).

  This method uses the fact that an image assuming a Lambertian model can represent an image of an arbitrary light source by linearly combining three images with different light sources, and classifies the image into a completely diffuse reflection region, a specular reflection region, etc. It is.

  20A to 20D show diagrams obtained by dividing the image shown in FIGS. 19A to 19D into regions by this method, respectively. 20A to 20D, the filled areas 1301 to 1304 indicate specular reflection areas.

  At this time, in order to change the position of the light source 104, it is desirable to install the light source 104 on the arm 103b of the robot 402 as described above. For example, the light source 104 is installed in infrastructure such as lighting installed in the home. You may use what is provided. In that case, home appliance cooperation may be performed so that the position of the light source is changed.

  Instead of fixing the camera 105 and changing the position of the light source 104, the light source 104 may be fixed and the position of the camera 105 may be changed. In this case, the correspondence between the pixels needs to be calculated between the images captured by changing the positions.

  Note that the specular reflection region estimation method is not limited to the method described above, and may be an estimation method using a deflection filter, for example. Alternatively, the estimation may be performed by cutting out a high luminance area in the image. Specifically, the specular reflection region is assumed to be a pixel having sufficiently high luminance, and when the luminance i (x) of a certain pixel satisfies the following expression, the pixel may be used as the diffuse reflection region.

ここで、Th_dとは、鏡面反射と拡散反射を区別する閾値である。

Here, Th _d is a threshold value for distinguishing between specular reflection and diffuse reflection.

3) Estimation of reflection parameter Next, a method for estimating the reflection parameter using the information of the separated diffuse reflection area and specular reflection area will be described.

First, the reflection parameter K _d is estimated using the information of the diffuse reflection region. In the diffuse reflection region, since the specular reflection component is considered to be almost zero, the following relationship is established from (Equation 2).

上述したように、光源１０４はアーム１０３ｂによって位置が制御されるため、光源１０４の方向は既知となる。また、保持対象物の位置（位置座標）と形状、姿勢も得られているため、対象物表面の法線方向、つまりｃｏｓ（θ_i）も既知である。

As described above, since the position of the light source 104 is controlled by the arm 103b, the direction of the light source 104 is known. Further, since the position (positional coordinates), shape, and orientation of the holding object are also obtained, the normal direction of the object surface, that is, cos (θ _i ) is also known.

In the position all of the plurality of images taken by changing the light source 104 for these pieces of information is known, in (Equation 3), a known change in I _c when changing the at each point of the object I understand that. Therefore, K _d of each point can be estimated by solving (Equation 3) using a method such as a least square method for each point of the object. At this time, since each luminance value often becomes an outlier due to the influence of noise or a three-dimensional position estimation error, the outlier value such as RANSAC (RANdom Sample Consensus) or SVDMD (Singular Value Decomposition with Missing Data) is used. It is desirable to use an optimization technique that can be eliminated.

Instead of obtaining the reflection parameter _Kd for each point, the image may be divided into regions for each material as preprocessing, and the reflection parameter _Kd may be obtained for each region.

  As such a region dividing method, the regions having the same color may be divided on the basis of the color on the assumption that the regions are made of the same material. Moreover, you may divide | segment using the shape information. For example, it is possible to divide by assuming that “the areas on a certain plane are all of the same material”.

Next, the reflection parameters K _s and σ are estimated using the information of the specular reflection region. Assuming that the reflection parameter of the specular reflection region is the same as that in the vicinity thereof, the parameter K _d indicating the diffuse reflection component of the specular reflection region can be obtained from the surrounding region. Therefore, (Equation 2) can be modified as follows.

ここで、右辺とｃｏｓ（θ_r）、αは全て既知であるため、鏡面反射成分を表す反射パラメータＫ_s，σを推定することができる。

Here, since the right side, cos (θ _r ), and α are all known, the reflection parameters K _s and σ representing the specular reflection component can be estimated.

  The material of the object is estimated by comparing and classifying the reflection parameter obtained in this way and the reflection parameter of each material stored in the database by a known method such as SVM (Support Vector Machine).

  In the above processing, a camera is used as the imaging device 105, but the imaging device 105 may be an infrared camera or the like.

  It is also possible to recognize the material of the holding object using a spectral image in the infrared region or visible light region. The former method is known as infrared spectroscopic analysis (FTIR), and the latter method is, for example, “(Distinction of material by spectral image for object recognition”, Yoshiaki Manabe, Kosuke Sato, Seiji Iguchi, Electronic Information Communication) Academic Journals D-II, Vol. J79-D-II, No. 1, pp. 36-44, 1996).

  As described above, the article holding system 12 of this embodiment specifies a holding method based on the recognized material.

  The article holding system 12 that recognizes the material of the holding object as in the present embodiment can be used when holding tableware. That is, it is assumed that the robot 402 performs an operation of transporting each tableware to the dishwasher in order to clean up the tableware after the meal. At this time, when there is leftovers in the tableware, the leftovers are thrown away and then put into the dishwasher, and when there is no leftovers in the tableware, they are put into the dishwasher as they are.

  When holding the tableware, the material recognition device 107 recognizes the material of the surface of the tableware to determine the presence or absence of leftover food. Specifically, the reflection parameter of the tableware is stored in advance in an electronic tag attached to each tableware. Then, as described above, the reflection parameter of the tableware is actually detected, and the detected parameter is compared with the storage parameter. If both parameters are the same, it can be determined that there is nothing on the tableware, and if both parameters are sufficiently different, it can be determined that something exists on the tableware. Alternatively, the reflection parameter of the leftover may be obtained in advance, and the actually detected reflection parameter may be compared with the reflection parameter of the leftover.

  When the material recognition device 107 recognizes the presence or absence of leftover food, the holding method specifying unit 102 specifies a holding method corresponding to the subsequent processing. In other words, when leftovers are present in the tableware, the most suitable holding method is selected for the operation of discarding the remaining rices, while when there is no leftovers, the optimal holding method is selected for the operation of putting into the dishwasher. .

  The holding device 103 holds the tableware according to the holding method specified by the holding method specifying means 102.

とする。推定された反射パラメータが十分に皿の反射パラメータに近い場合、その点は皿の領域であると判断し、その点を保持可能領域とし保持を行なう。一方、推定された反射パラメータが皿の反射パラメータと大きく異なる場合、その点は食品領域であると判断し、保持を行なわない。 Further, as described in the first embodiment, the article holding system 12 according to the present embodiment can also be used when specifying a holding position (positional coordinates) according to the condition of the article. That is, when holding the dish on which the dish (food) is arranged, the material recognition device 107 uses the difference between the reflection parameter of the dish and the reflection parameter of the dish, so that the dish placed on the dish is displayed. An area can be specified. In response to the identification result, the holding method specifying means 102 can specify an area other than the food area in the entire area of the dish as the holdable area. Therefore, the holding method specifying unit 102 specifies the holding method based on the recognition result of the recognition device 101. This specification is performed using a database (for example, in a table format) shown in FIG. For this purpose, the parameter estimation result based on the above-described (Equation 1) may be used. First, dish reflection parameters K _dp , K _sp , and σ _p are acquired in advance. Next, the reflection parameter at each point of the grasped object is estimated by the method described above. The estimated reflection parameter is

And If the estimated reflection parameter is sufficiently close to the reflection parameter of the dish, the point is determined to be a dish area, and the point is held as a holdable area. On the other hand, if the estimated reflection parameter is significantly different from the reflection parameter of the dish, it is determined that the point is a food area and is not held.

  Furthermore, the article holding system 12 of this embodiment is also effective when an electronic tag or the like is not attached to the article and a method for holding the article is not obtained in advance. For example, when an electronic tag or the like is not attached to a cup that is an object to be held and information (position (position coordinates), shape, and posture information) necessary for holding the cup cannot be obtained, stereo viewing is performed. Or the range finder, the recognition apparatus 101 obtains the shape of the cup. Further, the material recognition device 107 recognizes the material of the cup. Thereby, it is specified whether the cup is made of glass, stainless steel or paper, for example.

On the other hand, the article holding system 12 is provided with a database storing holding methods (including torque information for holding) corresponding to each material as shown in FIG. Select the holding method corresponding to the selected material. Then, the holding device 103 holds the cup according to the holding method specified by the holding method specifying means 102. Thus, for example, it is avoided that the torque is increased excessively when the glass cup is held, and that the paper cup is crushed when the paper cup is held. In other words, when holding a glass cup, it is held at a torque below the limit torque _TLg , and when holding a paper cup, it is held at a torque below the limit torque _TLp. To prevent.

  In this way, the state of the article is recognized by recognizing the material of the holding object, and the holding object can be held by a holding method optimal for the situation.

(Other embodiments)
In each of the above embodiments, the holding devices 103 to 12 have the recognition device 101, but the recognition device 101 may be provided outside the article holding system 10 to 12. In this case, a holding system in which the recognition apparatus 101 and the article holding systems 10 to 12 (robot 402) are connected to each other via a wireless and / or wired network is configured, and the recognition result by the recognition apparatus 101 is transmitted via the network. What is necessary is just to transmit to a holding | maintenance apparatus.

  In the database such as the holding method database, article holding methods for a plurality of different holding objects are stored as the holding objects, and the different holding objects have the same article ID information, and the shape, temperature, It is an article having different forms such as contents or moisture content, and the holding method changes depending on the form. Articles with different forms such as the shape, temperature, contents, or moisture content of the object to be held, or articles with different forms such as the shape, temperature, contents, or moisture content of the object to be held , ID information is the same, but it is an article that cannot be changed even if it is attached with an electronic tag. The above-mentioned paper and paper waste, laundry before washing, laundry folded after washing, empty cup and liquid There are cups, dry paper and wet paper.

  In addition, this invention is not limited to the said embodiment, It can implement in another various aspect.

  For example, the robot 402 may not include the moving device 414. FIG. 22 shows a holding device 103 having a robot arm 103b and a hand 103a without a moving device 414. The robot 402 is fixed to the ceiling 500 or the wall surface of the room and cannot move by itself. Such a robot 402 is effective when the movement range is limited such as cooking assistance in a kitchen or the like.

  It is to be noted that, by appropriately combining arbitrary embodiments of the various embodiments described above, the effects possessed by them can be produced.

  As described above, the present invention recognizes the situation of an article and then holds the object to be held by the holding method according to the situation, so that information necessary for holding cannot be obtained in advance. Even when information obtained in advance due to changes in the situation becomes useless or when it is difficult to set information necessary for holding each item in advance, it is possible to optimally hold a wide variety of items. Therefore, it is useful for various robots including a holding device, a housework support robot, and a cooking assistance robot used in a living space.

  Although the present invention has been fully described in connection with preferred embodiments with reference to the accompanying drawings, various variations and modifications will be apparent to those skilled in the art. Such changes and modifications are to be understood as being included therein, so long as they do not depart from the scope of the present invention according to the appended claims.

These and other objects and features of the invention will become apparent from the following description taken in conjunction with the preferred embodiments with reference to the accompanying drawings.
FIG. 1 is a block diagram showing a configuration of an article holding system in the first embodiment of the present invention. FIG. 2A is a schematic view illustrating a robot equipped with the article holding system in the first embodiment of the invention. FIG. 2B is a schematic diagram of a holding device of the article holding system for performing posture control. FIG. 2C is a block diagram illustrating a drive control system of an arbitrary drive unit such as an actuator of a motor control type of a moving device or a holding device. FIG. 2D is an explanatory diagram of a coordinate system set when controlling the posture. FIG. 2E is a table showing allowable variation ranges in the directions of α and β with respect to the holding level and stored in the database. FIG. 2F is a block diagram showing a configuration of an environmental system in which the robot according to the first embodiment of the present invention is active. FIG. 2G is a diagram conceptually showing the internal configuration of the article and moving object search / management means. FIG. 2H is an explanatory diagram of a table stored in an article database that handles articles. FIG. 2I is an explanatory diagram of a table stored in a moving object database that handles moving objects. FIG. 2J is an explanatory diagram of an example of an actual environment of the environment map of the environment map database. FIG. 2K is an explanatory diagram of an environment map obtained by simplifying the real environment of FIG. 2J of the environment map of the environment map database with a three-dimensional model. FIG. 2L is an explanatory diagram of an environment map obtained by further simplifying the real environment of FIG. 2J of the environment map of the environment map database with a planar model. FIG. 2M is a diagram illustrating an example of a table stored in the facility database associated with the environment map. FIG. 3A is an explanatory diagram illustrating an example of a sensing device using a laser beam according to the first embodiment of the present invention. FIG. 3B is an explanatory diagram illustrating an example of a sensing device using a laser beam according to the first embodiment of the present invention. FIG. 4A is a diagram illustrating a database that stores a holding method set for a glass in a table format. FIG. 4B is a diagram illustrating an empty glass. FIG. 4C is a diagram illustrating an example of sucking the empty glass of FIG. 4B with a suction hand. FIG. 4D is a diagram illustrating a glass containing a liquid. FIG. 4E is a diagram illustrating an example in which the glass containing the liquid in FIG. 4D is gripped by a finger-type hand. FIG. 5A is a diagram illustrating an example of a holding method in which an article is sandwiched from above by a finger-type hand when holding the paper of each aspect. FIG. 5B is an explanatory diagram illustrating a state in which paper is held by suction using a suction hand. FIG. 5C is an explanatory diagram showing a state in which a stack of paper is pinched by a finger-type hand from the thickness direction. FIG. 5D is an explanatory diagram illustrating a state in which paper is held by suction using a suction hand. FIG. 5E is an explanatory diagram illustrating a state where the end of the sucked paper is sandwiched from above and below by a finger-type hand. FIG. 5F is an explanatory diagram illustrating a state in which the gripping of the paper by the finger type hand is realized by stopping the suction of the suction hand. FIG. 5G is a diagram showing a list of holding methods in the robot shown in FIG. 2A. FIG. 5H is a diagram showing a method of holding the two fingers with two arms. FIG. 5I is a diagram illustrating a method of holding an object placed on two finger-type hands with two arms while holding the object with a suction hand. FIG. 6 is a diagram illustrating a table stored in a database that stores the holding method set for paper. FIG. 7 is a diagram illustrating a table stored in a database that stores a holding method set for clothing. FIG. 8A is a photographed image for explaining dish and food recognition processing based on background difference. FIG. 8B is a diagram of a dish with a pattern when there is no dish when explaining the recognition processing of the dish and food by background difference. FIG. 8C is a diagram of a background difference image when explaining dish and food recognition processing based on background difference. FIG. 8D is a view similar to FIG. 8A and is a photographed image of a dish and food placed on the dish at time t = t ₀ . FIG. 8E is a photographed image when photographing is performed again at time t = t ₁ when a fixed time has elapsed from time t = t ₀ . FIG. 8F is a diagram of a difference image between the captured image of FIG. 8D and the captured image of FIG. 8E. FIG. 8G is a flowchart of a dish status recognition process using a tag reader and a camera. FIG. 8H is a diagram illustrating holding position candidates (position coordinate candidates). FIG. 8I is a diagram illustrating holding position candidates (position coordinate candidates) and food. FIG. 8J is a diagram illustrating a non-holdable region by superimposing the difference image on the holding position information. FIG. 8K is an explanatory diagram when the region other than the food region is specified as the holdable region in the entire region of the dish. FIG. 8L is a diagram in a case where, among all the regions of the dish, a region with a difference is set as a holdable region, and a region without a difference is set as a non-holdable region. FIG. 9 is a diagram showing a holding state of the dish by the holding device. FIG. 10 is a diagram illustrating a state in which the state of the dish has changed with respect to FIG. 9. FIG. 11 is a diagram illustrating a state in which the holding position is changed in accordance with the change in the state of the dish. FIG. 12A is a diagram illustrating a holding method according to the temperature distribution of the frying pan. FIG. 12B is a diagram illustrating a holding method according to the temperature distribution of the frying pan. FIG. 12C is a diagram illustrating a holding method according to the temperature distribution of the frying pan. FIG. 12D is an explanatory diagram in a case where the portion where the temperature T is lower than the threshold temperature Th _TH is a holdable region, and the high portion is a non-holdable region. FIG. 12E is an explanatory diagram in a case where whether or not to hold an object is selected and posture control and moving speed are selected in accordance with a relationship between a predetermined upper limit temperature and a lower limit temperature. FIG. 12F is a diagram of a table when the recognition apparatus uses the history information and position information of the article. FIG. 13 is a block diagram showing a holding device further having learning means. FIG. 14A is a diagram illustrating an article movement speed history when a person holds the article. FIG. 14B is a diagram illustrating an article movement speed history when a person holds the article. FIG. 15A is a diagram illustrating an example of how a person is holding an article. FIG. 15B is a diagram illustrating another example of how a person is holding an article. FIG. 16 is a block diagram showing a configuration of an article holding system according to the second embodiment of the present invention. FIG. 17 is a conceptual diagram for explaining the Torrance-Sparrow model. FIG. 18A is a conceptual diagram for explaining a method of estimating the position, shape, and orientation of a holding object. FIG. 18B is a conceptual diagram for explaining a method of estimating the position, shape, and orientation of the holding object. FIG. 18C is a conceptual diagram for explaining a method of estimating the position, shape, and orientation of the holding object. FIG. 18D is a conceptual diagram for explaining a method of estimating the position, shape, and orientation of the holding object. FIG. 18E is a conceptual diagram for explaining a method of estimating the position, shape, and orientation of the holding object. FIG. 19A is a view of a spherical object photographed while moving the light source. FIG. 19B is a diagram in which a spherical object is photographed while moving the light source. FIG. 19C is a diagram in which a spherical object is photographed while moving the light source. FIG. 19D is a view of a spherical object photographed while moving the light source. FIG. 20A is a diagram in which only the specular reflection area is extracted from the image of FIG. 19A. FIG. 20B is a diagram in which only the specular reflection area is extracted from the image of FIG. 19B. FIG. 20C is a diagram in which only the specular reflection region is extracted from the image of FIG. 19C. FIG. 20D is a diagram in which only the specular reflection area is extracted from the image of FIG. 19D. FIG. 21 is a flowchart showing an operation example of the control means of the work robot. FIG. 22 is a diagram illustrating a robot that does not have a moving device. FIG. 23 is an explanatory diagram for explaining how to specify a holding method according to the temperature state of an article when the robot performs an operation of lowering a pan or frying pan on the stove from the stove. In FIG. 24, when only one plate is held, a holding method such as picking with a finger-type hand is selected. On the other hand, when a plurality of plates are stacked, the stacking is performed using an arm and a hand. It is a figure explaining the case where the holding | maintenance method which supports the done dish from the bottom is selected. FIG. 25 is a flowchart for explaining learning setting of a holding method based on the moving speed of an article as one of learning methods. FIG. 26 is a diagram illustrating an example in which the holding method of each article is set in a table format based on the result learned by the learning unit. FIG. 27 is a diagram illustrating an example of a table that associates each article with a holding level that specifies a learned holding method for each article. FIG. 28 is a diagram showing an example in which the holding method and holding level of each article are set in a table format. FIG. 29 is a diagram showing an example of a table for properly using the single arm / double arm holding method according to the scene. FIG. 30 shows an example of specifying a dish area placed on a plate and specifying a holdable area and a non-holdable area of the dish by using the difference between the reflection parameter and the reflection parameter of the dish. FIG. FIG. 31 is a diagram illustrating an example of a holding method corresponding to a recognized material in a table format.

Explanation of symbols

2 Robots 10 to 12 Article holding system 101 Recognizing means 102 Holding method specifying means 103 Holding apparatus 104 Light source 105 Imaging apparatus 106 Learning means 107 Material recognition means

Claims (1)

A recognition device for recognizing the status of an object to be held among items in the living space;
A holding method database for storing different article holding methods depending on the status of the holding object;
An article holding method specifying means for specifying one article holding method from different article holding methods according to the situation recognized by the recognition device;
A holding device that performs holding of the holding object with reference to the article holding method specified by the article holding method specifying means,
The recognition device recognizes a holding operation of the article performed by a person , including a moving speed or position of the article,
Learning means for learning and setting a holding method for each article or each situation of the article based on a recognition result of the holding operation of the article performed by the person by the recognition device;
The learning means learns a holding method of the article by referring to the holding operation of the article performed by the person, and stores the learned holding method in the holding method database in association with the article.

Images