JP2004243499A - Article handling system for living space, article handling method, and robot operating device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily designate work contents to be performed by a robot 102 in an article handling system for non-industrial use for making the robot 102 perform the article handling work in a living space wherein a person acts such as an ordinary home, an office, a hotel, a store or a hospital. <P>SOLUTION: An operation screen constituted by a virtual space in accordance with actual circumstances of the living space is displayed on a display part 117 of an operation terminal 103, and an input part 116 is operated by an user. Thus, the work contents to be performed by the robot in the virtual space are designated. Based on the designated work contents, the robot 102 performs the article handling work in the living space. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

【０１３７】
ここで式（１）におけるＲは、仮想視点座標系を構成する３つの軸周りの回転行列であり、この行列に仮想視点座標系における座標値を乗じることにより、その座標値が実世界座標系における座標値に変換されるように、回転行列の値が設定されている。また式（２），（３）におけるｆは仮想視点の焦点距離を示す。
【０１３８】
さて、仮想視点画像によって構成される操作画面の画面マスクデータの作成を、具体的な処理として言い換えると、センシング手段１０４ によって得られた物品の占める領域を、仮想視点で見たときに物品の占める領域に変換し、その領域を操作画面上の座標値に変換して、それをマスクデータとするということである。これはすなわち、
１）実世界座標系の座標値で表された物品領域の点（ｘ ，ｙ ，ｚ ）を、式（１）によって仮想視点座標系における座標値（ｘｅ，ｙｅ，ｚｅ）に変換し、
２）上記変換された座標値（ｘｅ，ｙｅ，ｚｅ）を、式（２），（３）によって仮想視点画像からなる操作画面の座標値（ｕ ，ｖ ）に変換する
という計算を、実世界座標系の座標値で表された物品領域の全ての点に対して行うことに他ならない。
【０１３９】
上記変換により仮想視点画像からなる操作画面の画面マスクデータを作成することができるが、より正確なマスクデータを作成するには、２）で得られた（ｕ ，ｖ）の座標値をプロットし、それらを囲む最小の多角形領域をマスクデータとしてもかまわない。
【０１４０】
尚、本実施形態では、操作端末１０３ の表示制御手段１１８ が、操作画面を作成するとしたが、こうした操作画面は、環境管理サーバ１０１ が作成してもよい。この場合、サーバ１０１ が作成した操作画面は、ネットワークを介して操作端末１０３ に送信し、操作端末１０３ は、受信した操作画面を表示部１１７ に表示すればよい。また、入力部１１６ の操作によって、仮想空間の視点の切り替え操作がされたときには、操作端末１０３ からサーバ１０１ に、視点切り替え要求信号を送信し、それに応じて上記サーバ１０１ が視点を切り替えた操作画面を作成し、これを操作端末１０３ に送信するようにすればよい。
【０１４１】
−ロボットの作業内容の指定手順−
次に、操作端末１０３ の表示部１１７ に表示される操作画面上で、ロボット１０２ に実行させる物品の移動作業を指定する手順について、２つの例を挙げて具体的に説明する。
【０１４２】
先ず、図１４を参照しながら、部屋の床のＢ１位置に置かれているバナナを、テーブルの手前側であるＢ２位置に移動させる場合を例に、ロボット１０２ の移動作業の指定手順を説明する。
【０１４３】
この移動操作の指定は、操作画面上で、バナナ（物品アイコン）を選択する操作と、この選択したバナナアイコンを、仮想空間内の所望の移動先に移動（ドラッグ）させると共に、その移動先位置でバナナアイコンをドロップする、というアイコンのドラッグアンドドロップ操作によって行われる。
【０１４４】
（操作画面の表示ステップ：Ｐ１００１）
操作端末１０３ の表示部１１７ には、操作画面が表示される。ここでは、操作画面を構成する仮想空間を、カメラが撮像したカメラ視点画像によって構成している。この仮想空間は部屋の実状況に則しているため、操作画面には、部屋内に存在する設備（テーブル等）が表示される。また、部屋内に存在する物品（バナナ等）は、物品アイコンとして仮想空間内に配置されている。
【０１４５】
ユーザは、操作端末１０３ の入力部１１６ を操作することにより、操作画面上でカーソルを移動させることができる。このカーソルが物品アイコンを指しているときには、その物品アイコンを強調表示する。具体的に、カーソルがバナナアイコンを指しているときには、そのバナナアイコンの色を変える、また、図１６（ａ）に示すように、バナナアイコンの輪郭を強調して表示する、さらに、図１６（ｂ）に示すように、物品アイコンに対応する物品の名称として、「バナナ」の文字をポップアップ表示する。一方、上記カーソルがテーブル等の設備を指していても、これらは物品アイコンではないため強調表示はされない。こうすることでユーザは、操作画面上で、例えばバナナがロボット１０２ の取扱い可能な物品であることを認識することができ、ロボットの作業内容の指定が容易になる。その結果、ユーザの利便性が向上する。
【０１４６】
尚、物品の名称をポップアップ表示する場合は、その物品に関する他の情報を併せて表示するようにしてもよい。例えば、その物品が食料品であれば、賞味期限や品質保持期限、購入年月日等の情報をポップアップ表示してもよい。こうすることで、ユーザは、その情報に応じてロボットに対する作業内容を指定することができる（例えばその物品の移動場所を変える等）。また、その物品に関する情報の内容に応じて、その物品に対するお勧めの作業内容をポップアップ表示してもよい。例えばカーソルが指している物品アイコン（物品）が、賞味期限切れ間近の食料品であるときには、「当該物品を冷蔵庫内に移動させる」旨の作業内容をポップアップ表示する、等である。
【０１４７】
また、カーソルが物品アイコンを指しているときに、物品の領域の色を変えたり、物品の領域の輪郭を強調して表示するだけでなく、その物品を拡大表示するようにしてもよい。また、カーソルが物品アイコンを指しているときには、例えば「○○があります」というように、その物品の名称を音声でユーザに報知してもよい。こうすることで、目の不自由なユーザにも使いやすいシステムとなる。
【０１４８】
また、ここでは物品アイコンの表示形態をかえるようにしているが、例えばカーソルが物品アイコンを指しているときには、そのカーソルの形態を変えるようにしてもよい。こうすることでも、そのカーソルが指している物品アイコンが、ロボットが取扱い可能な物品であることを、ユーザに認識させることが可能になる。
【０１４９】
さらに、ここでは、カーソルが物品アイコンを指しているときに、物品アイコンを強調表示するようにしているが、例えば、仮想空間内に配置されている全ての物品アイコンを所定時間だけ強調表示してもよい。特に、ユーザが操作画面における操作を開始する際に、全ての物品アイコンを所定時間だけ強調表示してもよいし、ユーザの要求操作に応じて、全ての物品アイコンを所定時間だけ強調表示してもよい。こうすることで、ユーザは、仮想空間内の物品アイコン、つまり、ロボット１０２ が取扱い可能な物品が、環境内のどこに置かれているかを一目で確認することができる。
【０１５０】
（物品の指定操作ステップ：Ｐ１００２）
ロボット１０２ に移動させる物品は、カーソルをその物品アイコンの位置に移動させて、その物品アイコンをクリック操作することによって指定する。この物品アイコンの指定操作がされたときには、「対象物品はバナナでよろしいですか？」等の確認を促す表示を行い、ユーザに「ＯＫ／キャンセル」の操作を行わせるようにしてもよい。
【０１５１】
こうして物品アイコンが指定されれば、そのことを明確にするために、その物品アイコンを強調表示する。この強調表示としては種々の表示形態が考えられるが、例えば、図１４に示すように、物品アイコンの色を変える、そのアイコンを塗りつぶす等の、その物品が指定されていることが一目で判るような形態であればどのような表示形態でもよい。但し、対象物品を指定したときの強調表示の形態は、カーソルが物品アイコンを指しているときに行う強調表示とは異ならせることが好ましい。これは、その物品がロボット１０２ の取扱い可能な物品であることと、その物品がロボット１０２ の作業対象として指定されたこととを区別するためである。
【０１５２】
（移動先指定ステップ：Ｐ１００３）
ロボット１０２ に移動させる物品を指定すれば、その物品の移動先を指定する。
この移動先の指定は、図１４に示すように、カーソルの操作により、物品アイコンを仮想空間内の所望の移動先（ここではテーブルの手前側のＢ２位置）に、ドラッグアンドドロップ操作することによって行う（同図の矢線参照）。この操作が行われたときには、「移動先はここでよろしいですか？」等の確認表示を行ってもよい。こうして、ユーザによる移動作業の指定が完了することになる。
【０１５３】
このように、物品の移動作業は、環境内に存在する物品に対応して仮想空間内に配置される物品アイコンを、その仮想空間内の所望の移動先にドラッグアンドドロップ操作するという、直感的な操作によって指定されるため、ユーザは、極めて容易にロボットに実行させる作業内容を指定することができる。
【０１５４】
次に、図１５を参照しながら、部屋の床のＢ１位置に置かれているバナナを、テーブルの奥側であるＢ２位置に移動させる場合を例に、ロボット１０２ の移動作業の指定手順を説明する。
【０１５５】
（操作画面の表示ステップ：Ｐ２００１）
操作画面の表示ステップは、上記のＰ１００１と同様であり、カメラ視点画像によって構成された仮想空間からなる操作画面が、操作端末１０３ の表示部１１７ に表示される。
【０１５６】
（物品の指定操作ステップ：Ｐ２００２）
物品の指定操作ステップも、上記のＰ１００２と同様であり、物品アイコンをクリック操作することによってロボット１０２ に移動させる物品を指定する。これにより、指定された物品アイコンは強調表示される。
【０１５７】
（視点切り替えステップ：Ｐ２００３）
ロボット１０２ に移動させる物品を指定すれば、その物品の移動先を指定する。ここで、本例では、移動先がテーブルの奥側であるＢ２位置であり、カメラ視点画像からなる操作画面にはＢ２位置が表示されていない。そこで、操作画面を構成する仮想空間の視点を切り替える。
【０１５８】
視点の切り替え時には、図１５に示すように、操作画面に視点位置にカメラアイコンが表示される。ユーザは、このカメラアイコンをクリック操作することによって、操作画面を構成する仮想空間の視点を切り替えることが可能である。尚、ここでは、仮想空間の視点が２つであるため、カメラアイコンは一つしか表示されないが、仮想空間の視点が３以上であるときには、各視点位置に対応して、複数のカメラアイコンを操作画面に表示すればよい。
【０１５９】
（移動先指定ステップ：Ｐ２００４）
仮想空間の視点を切り替えることにより、操作画面を構成する仮想空間は、カメラ画像を変形合成した仮想視点画像に切り替わる。そして、この新たな操作画面において、カーソルの操作により、物品アイコンを仮想空間内の所望の移動先（ここではテーブルの奥側のＢ２位置）に、ドラッグアンドドロップ操作する（同図の矢線参照）。こうして、ユーザによる移動作業の指定が完了することになる。
【０１６０】
このように、操作画面を構成する仮想空間の視点が複数あることで、ある視点では死角になるときでも、視点を切り替えることでそれが解消される。こうして、一般家庭等の生活空間では、設備の配置や物品の置かれる位置の自由度が高く、カメラ画像に死角が生じやすいが、ユーザは、ロボットに実行させる作業内容を仮想空間からなる操作画面上で的確に指定することができる。
【０１６１】
−ロボットの作業内容の実行制御−
以上のようにして、ユーザが操作端末１０３ を操作することにより、ロボット１０２ に実行させる作業内容が指定されれば、操作端末１０３ は環境管理サーバ１０１ にその作業内容のメッセージを送信する。このメッセージには、少なくとも、作業内容が物品の移動作業である旨と、移動対象となる物品の情報と、移動先の座標とを含めればよい。尚、操作端末１０３ からサーバ１０１ に送信する移動先の座標は、操作画面上の座標でもよいし、操作画面上の座標を環境の実世界座標に変換した座標でもよい。
【０１６２】
上記サーバ１０１ はメッセージを受信したときには、そのメッセージ内容に従ってロボット制御コマンド列を生成し、生成したロボット制御コマンド列を上記ロボット１０２ に送信する。物品の移動作業は、「（物品の位置への）移動」、「（物品の）把持」、「（移動先への）移動」、「（物品の）開放」という４つの作業単位に分解される。従って、上記の例では、ロボット制御コマンド列は、
ｍｏｖｅ，Ｂ１（バナナが置かれている位置Ｂ１にロボットを移動する）
ｇｒａｂ，バナナ（Ｂ１の位置にあるバナナを把持する）
ｍｏｖｅ，Ｂ２（（バナナを把持した状態で）移動先であるＢ２に移動する）
ｒｅｌｅａｓｅ（把持しているバナナを解放する）
となる。尚、複数の物品の移動が指定されたときには、コマンド列は、上記４つの作業単位を一組として、物品の数だけ並べる。
【０１６３】
図１７は作業ロボット１０２ の制御手段１１５ に係るフローチャートを示す。作業ロボット１０２ の送受信手段１０９ がサーバ１０１ から送信された制御コマンド列を受信したときには、上記制御手段１１５ は、ステップＳ２４０１〜ステップＳ２４０３で、いずれの作業単位であるかを判定し、その作業単位に応じた処理を実行する。
【０１６４】
先ずステップＳ２４０１で作業単位が「移動」であったときには、ステップＳ２４０４に移行し、指定された位置（ここでは、現在地から位置Ｂ１、又は位置Ｂ１から位置Ｂ２）までの経路を、移動計画作成手段１１３ に作成させる。
【０１６５】
続くステップＳ２４０５では、上記移動計画作成手段１１３ で作成した経路に応じて移動制御コマンドを移動手段１１４ に送り、上記指定された位置までの移動処理を実行する。
【０１６６】
一方、上記ステップＳ２４０２で作業単位が「把持」であったときには、ステップＳ２４０６に移行し、障害物センサ１１１ によって把持対象物品の姿勢を検出すると共に、続くステップＳ２４０７で、その検出結果に応じてアーム１２ａ とハンド１２ｂ の動作を計算する。そして、ステップＳ２４０８で、把持制御コマンドを把持手段１１２ に送り、物品の把持処理を実行する。尚、上記ステップＳ２４０６では、障害物センサ１１１ によって物品の姿勢を検知するが、物品の姿勢は、環境管理サーバ１０１ の物品ＤＢ１０６ に記録されているため、サーバ１０１ に物品の姿勢を問い合わせるようにしてもよい。
【０１６７】
上記ステップＳ２４０３で作業単位が「解放」であったときには、ステップＳ２４０９に移行し、指定された移動先に物品が設置されるようにアーム１２ａ とハンド１２ｂ の動作を計算する。そして、ステップＳ２４１０で、解放制御コマンドを把持手段１１２ に送り、物品の解放処理を実行する。
【０１６８】
ステップＳ２４１１では、移動、把持及び解放の各動作が終了したらその旨をメッセージとして作業指示元（サーバ１０１ ）に送信する。
【０１６９】
こうして、操作端末１０３ で指定した作業内容が、ロボット１０２ によって実行され、位置Ｂ１に置かれたバナナが、位置Ｂ２に移動されることとなる。
【０１７０】
上記ロボット１０２ が動作をしているときには、操作端末１０３ の表示部１１７ に表示される操作画面をカメラ画像と連動させることで、そのロボット１０２ が作業をしている様子を操作画面において映し出すようにしてもよい。こうすることで、ユーザは、指定した作業内容がロボット１０２ によって実行されているか否かを操作画面上で確認することができ、例えばロボット１０２ が間違った作業をしているときには、その作業をすぐに中止させることも可能になる。
【０１７１】
これに対し、ロボット１０２ が動作をしているときには、上記操作画面をロボット１０２ の動作開始前のカメラ画像で固定してもよい。こうすることで、新たな作業を指定しようとしても、操作画面内でロボット１０２ が動作していると指定操作がし難くなる場合があるが、それが解消される。
【０１７２】
これらの表示形態の切り替えは、ユーザの指定に応じて切り替えればよい。また、ロボット１０２ が動作をしているときには、カメラ画像と連動した操作画面の変更頻度を調節可能にしてもよい。
【０１７３】
以上説明したように、本発明による物品取扱いシステムは、所定の生活空間（環境）内に存在する物品の取扱い作業を行うロボット１０２ と、操作画面を表示する表示部１１７ 及びユーザが操作をする入力部１１６ を有する操作手段（操作端末１０３ ）とを備え、上記表示部１１７ に、上記生活空間の実状況に則した仮想空間からなる操作画面を表示し、ユーザに上記入力部１１６ を操作させることによって、上記仮想空間内でロボット１０２ に行わせる作業内容を指定させ、上記指定された作業内容に基づいて、上記ロボット１０２ が上記生活空間内で物品の取扱い作業を実行するものである。
【０１７４】
このように本発明では、操作画面が生活空間の実状況に則した仮想空間によって構成されている。このため、一般家庭等、生活空間の実状況は、工場や倉庫の状況よりも複雑であるが、その複雑な状況がそのまま操作画面として表示される。例えば、生活空間内では物品の配置は定まっていないが、その生活空間内の物品の配置に対応して、仮想空間内にも物品が配置される。こうして、複雑な生活空間内の状況が的確にかつ分かりやすくユーザに提示される。
【０１７５】
さらに本発明では、ロボット１０２ の作業内容は、この仮想空間内で指定する。このことで、複雑な状況下の生活空間内におけるロボット１０２ の作業内容を、容易に指定可能になる。
【０１７６】
また、ロボットが実行する作業が、生活空間内での物品の移動作業であるときに、本発明による物品取扱いシステムでは、操作画面を、生活空間内に存在する物品に対応して仮想空間内に配置される物品アイコンを有するものとして、物品の移動作業を、上記物品アイコンの操作によって指定する。
【０１７７】
このように、生活空間に存在する物品に対応して物品アイコンを仮想空間内に配置することにより、生活空間内で様々な位置に置かれる物品の中から、ロボットに移動させる物品を容易に指定することが可能になる。また、物品アイコンの操作によってロボット１０２ の作業内容を指定することで、的確かつ容易に作業内容を指定可能になる。
【０１７８】
また、本発明による物品取扱いシステムでは、物品の移動作業を、所望の物品アイコンを上記仮想空間内の所望の移動先にドラッグアンドドロップ操作することによって指定する。このためユーザは、物品の移動作業を指定するときには、操作画面の仮想空間内に配置される物品アイコンの中から、所望の物品アイコンを選択すると共に、その選択した物品アイコンを、仮想空間内の所望の移動先にドラッグアンドドロップ操作することになる。この操作によってロボット１０２ は、指定された物品アイコンに対応する物品を、仮想空間内で指定された移動先に対応する生活空間内の位置に移動させる作業を行うことになる。
【０１７９】
このように、指定した物品アイコンを、仮想空間内の所望の移動先にドラッグアンドドロップ操作するという、直感的な操作によって物品の移動作業が指定されるため、ロボット１０２ に実行させる作業内容を極めて容易に指定可能になる。
【０１８０】
また、本発明による物品取扱いシステムは、生活空間を撮像する撮像手段（センシング手段１０４ としてのカメラ）をさらに備えており、操作画面の仮想空間は、上記撮像手段が撮像した画像のデータによって構成される。
【０１８１】
こうすることで、生活空間の実状況に則した仮想空間を簡便に作成可能になる。尚、仮想空間は、例えばコンピュータグラフィックスによって構成してもよい。
【０１８２】
さらに、本発明による物品取扱いシステムでは、操作画面を、仮想空間を構成する画像のデータと、生活空間内に存在する物品に対応して仮想空間内に配置される物品アイコンと、上記画像のデータに対応する座標値を有しかつ、上記仮想空間内における物品アイコンの位置を特定する画面マスクデータとから構成する。
【０１８３】
すなわち、仮想空間を画像のデータによって構成したときに、操作画面上で物品アイコンを操作するには、その画像のどの領域が物品アイコンに対応するかを特定する必要がある。このため、画像のデータと、物品アイコンと、画面マスクデータとから操作画面を構成することにより、画面マスクデータによって、仮想空間内における物品アイコンの位置が特定され、その結果、画像データから構成される操作画面上で物品アイコンを操作することが可能になる。
【０１８４】
本発明による物品取扱いシステムは、生活空間内に存在する物品の、その生活空間内における位置を検出するセンシング手段１０４ をさらに備えている。そして、画面マスクデータは、上記センシング手段１０４ により検出された物品の位置に基づいて作成される。
【０１８５】
センシング手段１０４ によって生活空間内に存在する物品の位置を検出すれば、その検出した位置に対応する仮想空間内の位置が特定されるため、仮想空間内における物品アイコンの位置を特定する画面マスクデータが作成可能になる。
【０１８６】
本発明による物品取扱いシステムのように、一般家庭等の生活空間を対象とするシステムでは、その生活空間内には種々の物品が存在する。例えばその空間に固定・設置された固定・設置物等も物品である。しかしながら、こうした固定・設置物等は、ロボット１０２ によって移動させることができない。このように、生活空間内に存在する物品は、ロボット１０２ が取扱い可能な物品と、取扱い不可能な物品とに分けることができるが、仮想空間からなる操作画面には、ロボット１０２ が取扱い可能な物品と、取扱い不可能な物品との双方が表示される。
【０１８７】
そこで、本発明による物品取扱いシステムでは、操作画面上でポインタ（カーソル）が、ロボット１０２ の取扱い可能な物品に対応する物品アイコンを指しているときには、その物品アイコンを強調表示する、又は、その物品に関する情報を表示する。
【０１８８】
こうすることでユーザは、操作画面内に表示される多数の物品の中から、ロボット１０２ の取扱い作業可能な物品を容易に特定することができる。それによって、ロボット１０２ に実行させる作業内容を容易に指定することが可能になる。尚、ロボット１０２ の取扱い作業可能な物品と、取扱い作業不可能な物品との分別基準は、その取扱い作業の内容によって異なる。例えば物品の移動作業については、ロボット１０２ が移動させることのできない固定・設置物や、大型物・重量物は、ロボット１０２ の取扱い作業不可能な物品となり、それ以外の物品はロボット１０２ の取扱い作業可能な物品となる。
【０１８９】
また、そのシステムにおける作業ロボット１０２ が変更されたときには、そのロボット１０２ の変更に伴い取扱い可能な物品が変更される場合がある。ここで、ロボット１０２ の変更には、例えばロボット１０２ が新機種に置き換わるというハードウエア上の変更だけでなく、ロボット１０２ の制御プログラムが変更されるというソフトウエア上の変更も含む。
【０１９０】
このように、ロボット１０２ の取扱い可能な物品が変更されたときには、そのことをユーザに報知すべく、仮想空間内に配置されている全ての物品アイコンを所定時間だけ強調表示してもよい。こうすることでユーザは、ロボット１０２ が取扱い可能な物品を、一目で把握することができる。
【０１９１】
本発明による物品取扱いシステムは、ロボット１０２ の取扱い可能な物品が変更されたときには、そのロボット１０２ の取扱い可能な物品に対応する全ての物品アイコンを強調表示するように構成してもよい。
【０１９２】
また、本発明による物品取扱いシステムでは、上記操作画面上で物品アイコンが指定されることによりロボット１０２ の作業対象となる物品が指定されたときには、その物品アイコンを強調表示する。
【０１９３】
こうすることでユーザは、物品アイコンの指定を確認することができ、これにより、ロボット１０２ が誤って指定された物品の取扱い作業を実行してしまうことが、未然に防止される。
【０１９４】
本発明による物品取扱いシステムは、上述したように、一般家庭、オフィス、ホテル、店舗及び病院等の生活空間における物品取扱いシステムである。このため、その生活空間内には、例えば家具等の固定・設置物が存在していると共に、ロボットの取扱い作業の対象である物品も様々な位置に存在する。
【０１９５】
この場合、操作画面を生活空間の実状況に則した仮想空間から構成するときに、その仮想空間の視点が一つだけに定められていると、例えばその視点からは家具等に隠れて物品が見えない、又は、その物品を所望の移動先に移動させる場合にその移動先が死角になっている、等の問題が生じる。この問題は、工場や倉庫等の産業用のシステムでは通常生じない問題である。なぜなら、産業用のシステムでは、物品が定められた位置に配置されることが多く、さらには、空間の状況を見る視点が一つであっても、死角が生じないようにその空間の状況を予め設定しているためである。
【０１９６】
そこで、本発明による物品取扱いシステムでは、上記操作画面を構成する仮想空間は、その視点を切り替え可能にする。
【０１９７】
仮想空間の複数の視点が相互に切り替え可能であることで、ある視点では死角になるときでも、視点を切り替えることでそれが解消される。こうして、状況が複雑な生活空間での物品取扱いシステムにおいて、ロボット１０２ に実行させる作業内容を操作画面上で的確に指定することが可能になる。
【０１９８】
また、本発明によるロボット操作装置（操作端末１０３ ）は、所定の生活空間内に存在する物品の取扱い作業を行うロボット１０２ の作業内容を指定するための装置である。
【０１９９】
上記ロボット操作装置は、操作画面を表示する表示部１１７ と、ユーザが操作をする入力部１１６ とを備え、上記表示部１１７ に、上記生活空間の実状況に則した仮想空間からなる操作画面を表示し、ユーザに上記入力部１１６ を操作させることによって、上記仮想空間内でロボット１０２ に行わせる作業内容を指定させるものである。
【０２００】
この構成により、上述したように、表示部１１７ には、生活空間の実状況に則した仮想空間からなる操作画面が表示される。これにより、生活空間内の複雑な状況が的確にかつ分かりやすくユーザに提示される。
【０２０１】
また、ロボット１０２ の作業内容の指定は、その仮想空間からなる操作画面において行われるため、ユーザは、ロボット１０２ の作業内容を容易に指定可能になる。
【０２０２】
本発明による物品取扱い方法は、所定の生活空間内に存在する物品の移動作業を行うロボット１０２ を用いた方法である。
【０２０３】
そして、上記物品取扱い方法は、上記生活空間の実状況に則した仮想空間からなり、その生活空間内に存在する物品に対応して上記仮想空間内に配置される物品アイコンを有する操作画面を表示するステップと、上記操作画面上で、所望の物品アイコンを指定するステップと、上記指定した物品アイコンを、上記仮想空間内の所望の移動先にドラッグアンドドロップ操作するステップと、上記生活空間内で上記ロボット１０２ が、上記指定された物品アイコンに対応する物品を、仮想空間内で指定された移動先に対応する位置に移動するステップとを含む。
【０２０４】
本発明による物品取扱い方法は、操作画面を構成する仮想空間の視点を切り替えるステップをさらに含む。この視点を切り替えるステップは、上記実施形態では、仮想空間内の所望の移動先に物品アイコンをドラッグアンドドロップ操作するステップの前としたが、これに代えて、又はこれと併せて、所望の物品アイコンを指定するステップの前としてもよい。
【０２０５】
−他の実施形態−
尚、本実施形態では、環境管理サーバ１０１ が操作端末１０３ からの作業内容メッセージに応じてロボット制御コマンドを作成し、これを作業ロボット１０２ に送信するようにしたが、例えば、操作端末１０３ が操作画面上で指定された作業内容に応じてロボット制御コマンドを作成し、これを作業ロボット１０２ に送信するようにしてもよい。
【０２０６】
また、本実施形態では、物品取扱いシステムを、環境管理サーバ１０１ 、ロボット１０２ 、操作端末１０３ の３つのサブシステムからなり、それらサブシステム１０１ 〜１０３ が無線又は有線等のネットワークを介して互いに情報をやりとりするような構成とした。しかし、物品取扱いシステムはこの構成に限らず、例えば操作端末１０３ が環境管理サーバ１０１ に一体となった構成でもよい。
【０２０７】
またロボット１０２ も１台ではなく複数台が協調しながら作業を並行して行う構成でも構わない。
【０２０８】
さらに、本実施形態では、ロボット１０２ は物品の移動作業を行うものとし、環境の実状況に則した仮想空間からなる操作画面において、物品アイコンのドラッグアンドドロップ操作により、ロボット１０２ の作業内容を指定するようにした。しかし、仮想空間からなる操作画面において指定するロボット１０２ の作業内容は、物品の移動作業に限るものではなく、物品の移動を伴わない作業を含め、その他の作業としてもよい。例えば、操作画面上で、ロボット１０２ に作業をさせる対象となる物品アイコンを指定したときに、ロボット１０２ が実行し得る作業内容をメニュー形式で操作画面に表示し、そのメニューから作業内容を選択させることによって、ロボット１０２ に実行させる作業内容を指定するようにしてもよい。
【０２０９】
【発明の効果】
以上説明したように、本発明による生活空間用の物品取扱いシステム、物品取扱い方法、及びロボット操作装置では、生活空間の実状況に則した仮想空間からなる操作画面が、操作端末（ロボット操作装置）の表示部に表示される。これによりユーザは、生活空間内の複雑な状況を、的確に把握することができる。また、ユーザはこの操作画面上で、ロボットに実行させる作業内容を指定することができる。例えば物品の移動作業を指定するときには、生活空間内に存在する物品に対応して仮想空間内に配置される物品アイコンを操作することにより、例えば仮想空間内の所望の移動先にドラッグアンドドロップ操作をすることにより、移動作業の指定ができる。このような直感的な操作によって、ロボットに実行させる作業内容が指定できるため、誰もが容易に、作業内容の指定をすることができる。
【０２１０】
また、操作画面上で、カーソルがロボットの取扱い可能な物品アイコンを指しているときには、そのことをユーザに明示する表示がなされるため、ユーザはロボットに作業させる物品の指定を確実に行うことができる。
【０２１１】
さらに、ロボットに作業させる物品（物品アイコン）を指定したときには、操作画面上で、そのことをユーザが認識可能となる表示がされるため、ロボットが誤って指定された物品の取扱い作業を実行してしまうことを、未然に防止することができる。
【０２１２】
加えて、操作画面を構成する仮想空間の視点は切り替え可能に構成されているため、例えば、ある視点からは死角になっている物品を、視点を切り替えることで見ることができ、ある視点からは死角になっている位置を、視点を切り替えることで見ることができる。すなわち、操作端末の表示部に、生活空間の複雑な実状況に則した仮想空間内全てを、確実に表示することができる。
【０２１３】
また、生活空間の実状況に則した仮想空間を、カメラが撮像した画像データによって構成することで、仮想空間を容易に作成することができる。
【図面の簡単な説明】
【図１】本実施形態に係る物品取扱いシステムの全体構成を示すブロック図である。
【図２】背景差分法の原理を示す説明図である。
【図３】物品／移動体検索・管理手段の構成を示すブロック図である。
【図４】ゲート型のリーダライタを環境内のドアや窓に配置した例を示す図である。
【図５】物品／移動体データベースの内、物品データベースの構成と記載内容例を示す図である。
【図６】物品／移動体データベースの内、移動体データベースの構成と記載内容例を示す図である。
【図７】持出・持込データベースの構成と記載内容例を示す図である。
【図８】環境内で物品が移動する一例を説明する説明図である。
【図９】環境の実状況と、それに対応する環境マップとを示す図である。
【図１０】環境マップに付随する設備データベースの構成と記載内容例を示す図である。
【図１１】作業ロボットの構成の一例を示す概略図である。
【図１２】画像データと、画面マスクデータとからなる操作画面の構成を示す図である。
【図１３】環境内に設定された実世界座標系と仮想視点座標系との関係を示す図である。
【図１４】操作画面上で、ロボットに実行させる物品の移動作業を指定する手順を示す図である。
【図１５】図１４とは異なる物品の移動作業を指定する手順を示す図である。
【図１６】操作画面における物品アイコンの表示形態の例を示す図である。
【図１７】作業ロボットの制御手段に係るフローチャートである。
【符号の説明】
１０１・・・環境管理サーバ
１０２・・・作業ロボット
１０３・・・操作端末（操作手段、ロボット操作装置）
１０４・・・センシング手段（撮像手段）
１０６・・・物品／移動体データベース
１０８・・・環境マップ
１１０，１１５，１１９・・・制御手段
１１６・・・入力部
１１７・・・表示部
１１８・・・表示制御手段[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a living space article that allows a robot to perform an article handling operation in a space where a person performs an activity such as a general home, office, hotel, store, and hospital (hereinafter, these are collectively referred to as a living space). The present invention relates to a handling system, an article handling method, and a robot operation device.
[0002]
[Prior art]
Conventionally, robots have been effectively used in many fields. For example, robots are used to grip and transport parts in automated product assembly lines in factories and transport and manage inventory in automated warehouses, taking advantage of the robot's gripping function for articles. There is no point to mention.
[0003]
Such industrial robots are often limited to a rectangular parallelepiped or the like at best even if the shapes and sizes of the articles to be handled are the same, or even if they have a little degree of freedom. A technology that makes use of this fact to improve the efficiency of carrying work using a robot has also been disclosed. As a typical example, Patent Literature 1 discloses a robot system for loading and unloading articles.
[0004]
[Patent Document 1]
JP-A-7-237159
[0005]
[Problems to be solved by the invention]
Nowadays, robots that have matured for industrial use have been leveraged to achieve the high goal of supporting human life while coexisting with humans in ordinary households. R & D is becoming increasingly active. For example, entertainment robots that heal the human mind by behaving like a pet, or sensors that detect obstacles in the room with sensors and automatically clean the room while avoiding them, etc. Robots for housework support have been actually developed and commercialized. In addition, as element technologies, handling technologies for freely grasping various articles and sensing technologies for recognizing a state in a space in which a robot moves, which are indispensable for performing housework support, are also being actively developed. As the development of such technology progresses, robots that perform various household tasks on behalf of humans will be realized in the future.
[0006]
Now, one of the basic tasks of housekeeping support is an operation of moving (transporting) an article. As a specific application example of the robot system for moving an article, there is a system in which a self-propelled robot capable of gripping an article moves an article on behalf of a person, such as an elderly person, who cannot carry the article. Conceivable.
[0007]
As an operation interface of such a system, for example, remote operation (remote control operation) of the operation of the robot can be considered as conventionally realized in an industrial robot.
[0008]
However, in order for the robot to move an article located at a certain first point to a second point, the robot moves 1) from the current position to the first point, and 2) exists at the first point. At least four operations of gripping the article, 3) moving to the second point while holding the article, and 4) releasing the article gripped at the second point must be performed. Therefore, when the robot performs a moving operation by remote operation, all of the above four operations 1) to 4) must be operated, which is extremely complicated and requires time and effort for learning the remote operation. It takes.
[0009]
For example, the robot system disclosed in Patent Document 1 is configured to specify the work content of the robot on an operation screen. More specifically, in this system, when the robot loads a rectangular parallelepiped cargo on a pallet, a virtual article diagram that virtually indicates the cargo is displayed in the operation screen, and the operation screen is displayed. Above, by moving the virtual article diagram to a desired position and orientation, the work content to be executed by the robot is specified.
[0010]
However, in this system, the robot merely moves an article present at a predetermined position (for example, on a conveyor) to a predetermined position (on a pallet). On the operation screen, the position and orientation of the article on the pallet are determined. Just specify. On the other hand, for example, when a robot performs an operation of moving an article in a space where a person, such as a general household, performs an activity, the position where the article is present is not determined, and the position where the article is moved is also determined. Absent.
[0011]
In other words, the operation interface of the industrial robot system disclosed in Patent Document 1 cannot specify a complicated and highly flexible robot operation content, so that non-industrial robots used in ordinary households and the like cannot be specified. It is not possible to use the operation interface of the industrial robot system disclosed in the above-mentioned Patent Document 1 in the above-mentioned robot system.
[0012]
In a living space robot system, since the situation in a living space is more complicated than the situation in a factory or warehouse, an operation interface that accurately presents the complicated situation in the living space to a user is required.
[0013]
In addition, considering that it is used in ordinary households, the operation interface does not require much labor to learn as in the remote operation described above, and anyone can easily specify the work content to be executed by the robot. Preferably it is possible.
[0014]
The present invention has been made in view of such circumstances, and an object of the present invention is to provide an object to a robot in a living space where a human is active, such as a general home, office, hotel, store, and hospital. It is an object of the present invention to provide a non-industrial article handling system for executing a handling operation, in which a situation in a living space is accurately presented to a user, and a task to be executed by the robot can be easily specified.
[0015]
[Means for Solving the Problems]
An article handling system according to the present invention includes a robot for handling an article existing in a predetermined living space, an operation unit having a display unit for displaying an operation screen and an input unit for a user to operate, and Displaying an operation screen consisting of a virtual space in accordance with the actual situation of the living space on the unit, and prompting the user to operate the input unit, thereby causing the robot to specify the operation content to be performed by the robot in the virtual space; The robot performs a task of handling articles in the living space based on the performed work content.
[0016]
According to this configuration, the operation screen is displayed on the display unit of the operation means. The operation screen displayed on this display unit is a virtual space that conforms to the actual situation of the living space. Here, that the virtual space conforms to the actual situation of the living space means that the situation of the virtual space and the actual situation of the living space match each other. By operating the input unit, the user specifies the work content to be executed by the robot in the virtual space of the operation screen. When the work content is designated in this way, the robot executes an article handling work in the living space based on the work content.
[0017]
As described above, in the present invention, the operation screen is configured by the virtual space conforming to the actual situation of the living space. For this reason, the actual situation of a living space such as a general home is more complicated than the situation of a factory or a warehouse, but the complicated situation is displayed as it is on the operation screen. For example, the arrangement of the articles in the living space is not fixed, but the articles are also arranged in the virtual space corresponding to the arrangement of the articles in the living space. Thus, the situation in the complicated living space is presented to the user accurately and easily.
[0018]
Further, in the present invention, the work content of the robot is specified in this virtual space. This makes it possible to easily specify the work content of the robot in the living space under a complicated situation.
[0019]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings. The article handling system according to the present invention causes a work robot to execute an article handling operation in a space (living space) where a person performs an activity, such as a general home, an office, a hotel, a store, and a hospital. System.
[0020]
In the present embodiment, one room in a building such as a general home is defined as a target space (hereinafter, referred to as an environment) of the article handling system. In addition, in the present embodiment, the work robot performs a moving operation of moving an article specified by a user to a specified position. It shall be.
[0021]
As shown in FIG. 1, the present system includes an environment management server 101 (hereinafter, sometimes simply referred to as a server), a work robot 102 (hereinafter, sometimes simply referred to as a robot), and an operation terminal 103. It is divided into three subsystems. These three subsystems 101 to 103 are connected via a wireless or wired network, and are configured to exchange information via this network.
[0022]
Each of the three subsystems 101 to 103 includes control means 110, 115, 119 and a transmission / reception means 109. Here, since the transmission / reception means 109 of each of the subsystems 101 to 103 has the same processing, the same reference numerals are given.
[0023]
-Configuration of environmental management server-
The environment management server 101, which is the first subsystem, includes a sensing unit 104 for grasping the situation in the environment, and among the situations grasped by the sensing means 104, the articles existing in the environment (the robot 102 can perform the handling work). Object / moving object search / management means 105 for managing the situation of a moving object (a person or a work robot 102 or the like that mainly handles articles) and an article / moving object for storing data of the article and the moving object. A body database 106; an environment map management means 107 for managing the situation of the entire environment other than the article and the mobile body; an environment map 108 for storing data of the entire environment; and data and an environment map 108 of the article / mobile body database 106 Data inquiry (signal) from the outside, or send a response signal to the outside Ri also includes a receiving means 109 and transmits a control command for the robot 102 consists of these means 104, 105, 107, 109 control to the control means 110..
[0024]
The sensing means 104 is for constantly monitoring any and all articles and furniture existing in the environment, and the position and state of a person or the robot 102 existing in the environment. The sensing means 104 also detects that an article has been brought into the environment by an individual or the robot 102 and that an article has been taken out of the environment (the details of this will be described later).
[0025]
One of the characteristic components of this system is the sensing means 104. In other words, in the system according to the present embodiment, the target environment is limited to one room in a specific building, but the position where the article is placed in the environment and the movement route of the moving object carrying the article. There are no restrictions on. For this reason, in this system, a sensing technology capable of accurately detecting the status of articles and moving objects in the environment is required.
[0026]
Therefore, first, a general method of detecting an article using an image sensor, which is one of the most frequently used sensors, and its problems will be briefly described. Next, a method of detecting an article using an electronic tag, which is another sensor, and its problems will be briefly described.
[0027]
In order to efficiently monitor a relatively large area, such as the entire room, with a small number of facilities, an image sensor, that is, a camera is fixed to a ceiling or a wall of a room, and articles in the room are taken using the camera image (captured image). Is generally detected.
[0028]
Here, there is a background subtraction method as a general method for detecting an article or a moving object in an environment using a camera image. The background subtraction method is a method in which a model image as a background is prepared in advance, and a target object is detected by calculating a difference between a current camera image and the model image. The purpose of the sensing means 104 in the present system is to detect and monitor articles and moving objects in the environment. For this reason, when there is little change in the situation of the environment, an image captured when no article or moving object exists in the environment may be used as the model image. Further, when the environmental situation fluctuates greatly, an image obtained by averaging a plurality of images taken at predetermined time intervals may be used as a model image.
[0029]
Specifically, an article detection method using the background subtraction method will be described with reference to FIG. Here, FIG. 2A is a diagram illustrating an example of a model image, FIG. 2B is a diagram illustrating an image (input image) captured by a camera at a certain point in time, and FIG. FIG. 14 is a diagram illustrating an example of a background difference image obtained by subtracting a model image from an input image. As can be seen from FIG. 2C, in the background difference image, a part having a difference between the input image and the model image appears (see a shaded part in the figure). Therefore, it is possible to detect an article present in the environment by taking out only the protruding portion. Further, the image is subjected to image processing to specify what the article is. As described above, by using the background difference method, it is possible to detect the status of articles and moving objects in the environment.
[0030]
However, when using a camera to detect objects, it is difficult to change the brightness, the resolution is low, the objects are hidden by other objects and cannot be seen, and if the objects overlap, they are detected as one article In general, there are many problems.
[0031]
For example, the problem of blind spots can be solved by arranging a plurality of cameras substantially uniformly in an environment and enabling all of the articles present in the environment to be imaged by any one of the cameras. However, simply installing a large number of cameras so as to eliminate blind spots does not always ensure that articles can be detected. In other words, even if the number of cameras is increased, the problem of the resolution and the problem of the overlapping of the articles cannot be solved, so that it is not possible to judge what the protruding portion is in the background difference image (the article cannot be specified).
[0032]
Next, a method for detecting an article using an electronic tag will be described. In recent years, a method of detecting the position of an article or a moving object using an electronic tag has been developed, and it is fully conceivable that the position detection using a tag may be replaced with the position detection using a camera image.
[0033]
An electronic tag is a device composed of an IC for storing data and an antenna for transmitting and receiving data wirelessly. The electronic tag reads information written on the electronic tag by a device called a reader / writer and writes information on the electronic tag. be able to.
[0034]
Therefore, the electronic tag is attached to each article, and data relating to the article, for example, data such as the type, shape, and weight of the article, an image of the article, and date of manufacture are embedded in the electronic tag. In addition, an electronic tag is also attached to the moving object (human / robot 102). In the case of a person, the electronic tag may be embedded in a portable device (for example, a wristwatch). Data related to the moving object, for example, information such as a person's name and date of birth is written in the electronic tag attached to the moving object. On the other hand, many reader / writers are installed in the environment. By doing so, the reader / writer reads information on the article and the electronic tag attached to the moving body, so that an article or the like existing in the environment can be detected without a camera. When using a camera, it simply detects the presence of an article.When using an electronic tag, in addition to detecting the presence of an article, it uses shape data of the article embedded in the electronic tag. As will be described later, the robot 102 can easily hold the article, the date of manufacture can be used to control the quality expiration date, and the article type data can be used to easily find the item to be searched. Brings great benefits to the user.
[0035]
However, when exchanging data between the electronic tag and the reader / writer, a very weak radio wave which does not affect the human body has to be used, so that the communication distance is as short as several tens cm at most. One of the problems of article detection using an electronic tag is its short communication distance. In addition, it may be possible to solve the problem of communication distance by installing many reader / writers in the environment.However, reader / writers are more expensive than cameras, and it is not possible to install many readers / writers in the environment like cameras. Not realistic.
[0036]
As described above, a method using an image sensor and a method using an electronic tag have been described as methods for detecting an article and a moving object existing in the environment, but each method has advantages and disadvantages.
[0037]
Therefore, a method of detecting an article using both the image sensor and the electronic tag can be considered. That is, a hybrid process is performed in which the approximate position of an article in the environment is specified by the above-described background subtraction method, and the article is specified using an electronic tag. The following are two specific examples of the processing.
[0038]
In one example, a camera is installed on a ceiling or a wall in an environment, and a reader / writer is attached to the work robot 102. Also, an electronic tag is attached to each article and the moving body. Then, the position of the article in the environment is specified by the background subtraction method using the camera image. Next, the robot 102 is moved to the vicinity of the specified article, and information is read from the electronic tag attached to the article by a reader / writer attached to the robot 102 to specify the article.
[0039]
In another example, a camera is installed on a ceiling, a wall, or the like in an environment, and a plurality of reader / writers are installed substantially uniformly in the environment. It is assumed that this reader / writer has directivity for reading data of the electronic tag and the reading direction is variable. Then, the position of the article in the environment is specified by the background subtraction method using the camera image. Next, the reader / writer installed at the position closest to the specified article is selected, and the reading direction of the reader / writer is directed to the article. Thus, the information is read from the electronic tag attached to the article, and the article is specified. In this example, since the distance between the reader / writer and the electronic tag may be long, a relatively strong radio wave is used. Therefore, it is necessary to confirm that there is no person in the environment, for example, by the background subtraction method.
[0040]
As described above, a method using a camera and / or an electronic tag has been described as a specific example of the sensing means 104 in the present system. However, the sensing means 104 in the present system may employ a method other than that described here.
[0041]
As will be described in detail later, a reader / writer serving as sensing means 104 is installed at the doorway between the environment and the outside, such as a door or a window, to detect the entry or removal of articles into the environment.
[0042]
When the sensing means 104 detects an article or a moving object, it transmits the information to the article / moving object search / management means 105. The information to be transmitted includes, for example, the detection time of the article, the position and orientation of the article, and the like.
[0043]
The article / moving object search / management means 105 accumulates information on each article and moving object existing in the environment detected by the sensing means 104 in an article / moving object database (DB) 106. Although the details of the DB 106 will be described later, the information stored in the DB 106 includes at least the position of the article and the moving object.
[0044]
The article / moving object search / management means 105 infers a moving object (people / robot) performing article handling (article movement) based on the information from the sensing means 104 and outputs the inference result. It is configured to accumulate in the article / moving object DB 106.
[0045]
As shown in FIG. 3, the article / moving object search / management means 105 determines that the article (article registered in the article / moving object DB 106) is handled by the moving object, and here, in particular, An article handling detecting means 31 for detecting that the object is being moved, and a handler specifying means 32 for specifying a moving body (handler) handling the article according to the detection result of the article handling detecting means 31. Have.
[0046]
The article handling detecting means 31 detects that the article is handled by the moving body (article handling state) based on the information from the sensing means 104. For example, when an article is detected using the above-described background subtraction method, when an input image (camera image) and a model image are compared and a portion having a difference between the two occurs, an object is detected at that portion. It can be detected that it is being handled. Note that, instead of this method, for example, an electronic tag may be used to detect that the article is being handled by the moving object.
[0047]
When the article handling detecting means 31 detects the article handling state, the handler identifying means 32 identifies the handler who is handling the article, and stores the information of the handler in the article / mobile object DB 106. .
[0048]
The above-mentioned handling person may be specifically specified as follows. In other words, when the sensing means 104 uses a camera, it captures an image of the area where the article handling state is detected by the camera. Then, a face authentication process is performed on the captured image, and the moving object is specified by the authentication process. Since the moving body specified in this manner is considered to have been near the handled article, the moving body can be estimated as the handler. Here, the camera used by the sensing means 104 is a wide-angle camera for photographing a wide area because it is used for article detection by the background subtraction method. The image captured by the wide-angle camera has a relatively low resolution, and the resolution is often insufficient for performing face authentication processing. Therefore, apart from the camera used for the background subtraction method, a narrow-angle high-resolution camera as a camera for face authentication processing may be installed in the environment or on the robot 102. The area in which the article handling detection means 31 has detected the article handling state is imaged by a narrow-angle camera, and a face authentication process is performed on the captured image, whereby the operator can be specified with high accuracy.
[0049]
The method of specifying the article handler is not limited to the face authentication process, and may be performed by other authentication processes such as iris authentication. Further, the camera image itself may be stored in the article / moving object DB 106 without performing the authentication process on the camera image. This may be performed only when the moving object cannot be identified by the authentication processing. Further, the identification of the handler may be performed using an electronic tag.
[0050]
In addition, when managing the handling of articles in a specific place of the environment (room) such as a door or a window, specifically, when managing the bringing in and taking out of articles in the environment, an electronic tag and a reader are required. It is better to use a writer.
[0051]
That is, as shown in FIG. 4, gate-type reader / writers (RF antennas) 41 and 42 are disposed at windows 51 and doors 52 which are entrances and exits between the environment and the outside. Then, when the article and the moving body pass through the window 51 and the door 52, the reader / writer reads the information from the electronic tag attached to the article and the moving body, and the article handling detecting means 31 detects that the article and the moving body It is detected that the object is being handled by the moving body (carrying into or taking out of the environment). Further, based on the information read by the reader / writer, the handler specifying means 32 allows the mobile object that has passed through the window 51 or the door 52 together with the article to be handled by the article handler (a person who carries in or takes out the article). What is necessary is just to specify and store the information of the handler in the article / mobile object DB 106. This makes it possible to automatically manage the carry-in / take-out of goods by simple processing.
[0052]
In FIG. 4, the reader / writers 41 and 42 are installed so as to surround the upper, lower, left, and right sides of the opening of the window 51 or the door 52, but this is for performing highly accurate detection regardless of the direction of the electronic tag. The reader / writers 41 and 42 may be installed only at the vertical position of the window 51 or the door 52, only at the left and right positions, or at the center position.
[0053]
In addition, by installing such a reader / writer double with respect to the window 51 and the door 52, it is possible to easily determine whether the article has been brought into the environment or taken out of the environment. That is, although not shown, the first reader / writer is located outside the opening of the window 51 or the door 52 outside the environment, while the second reader / writer is located outside the opening of the window 51 or the door 52. Installed inside the opening of. By doing so, when the inner second reader / writer detects information from the electronic tag after the outer first reader / writer detects information from the electronic tag, the article to which the electronic tag is attached becomes Can be identified as being brought into the environment. Conversely, when the outer first reader / writer detects information from the electronic tag after the inner second reader / writer detects information from the electronic tag, the article to which the electronic tag is attached is removed from the environment. Can be identified as being taken outside.
[0054]
The management of articles in a specific place using such a reader / writer is not limited to the management of articles in the window 51 and the door 52, but can be applied to other places. For example, if a similar reader / writer is installed in an opening of an object that accommodates an article such as a refrigerator or a cabinet, it is possible to manage an article to be put in the refrigerator or a cabinet and an article taken out from the refrigerator or the cabinet. .
[0055]
In the present system, the handling of articles is not limited to humans, and the robot 102 also handles articles in accordance with the work contents specified by the human at the operation terminal 103 as described later. When the robot 102 handles an article in this way, the article handler may be the robot 102 or a person who operates the robot 102 may be the article handler. For example, when the person who operates the robot 102 is an article handler, the operator of the robot 102 is subjected to biometrics authentication processing such as fingerprint authentication, face authentication, voiceprint authentication, and iris authentication, and is performed based on the authentication result. The operator (article handler) of the robot 102 may be specified. Further, when user authentication is performed by so-called LOGIN and LOGON processing at the operation terminal 103 at the start of the operation of the robot 102, the operator (article handler) of the robot 102 may be specified from the user information.
[0056]
Further, as will be described later in detail, when the article / moving object search / management means 105 receives an inquiry from the control means 110 to the article / moving object DB 106, the article / moving object DB 106 sends necessary information in accordance with the contents of the inquiry. The information is also extracted from the mobile object DB 106 and the information is sent to the control means 110.
[0057]
The article / moving object DB 106 of the environment management server 101 is a DB that stores information on articles and moving objects, and is configured as shown in FIGS. 5 and 6, for example. That is, the article / moving object DB 106 includes an article database (FIG. 5) for handling articles and a moving object database (FIG. 6) for handling moving objects.
[0058]
The article DB is made up of sub-databases each storing three types of data: article data, article history data, and article attribute data, and the data contents stored in each sub-database are as follows.
[0059]
1) Article data: An ID for distinguishing individual articles, a pointer to article history data, and a pointer to article attribute data are stored. Here, if a plurality of articles are of the same type but are physically different, they are treated as different articles, so that different IDs are assigned. On the other hand, since articles of the same type have the same physical attributes, they have the same pointer to the article attribute data even if different IDs are assigned. This saves database capacity.
[0060]
2) Article history data: Stores the history of handling of articles, and consists of four items: time when the article was handled, handling details, handler, and position after handling. Among these, various expressions are conceivable as a method of expressing the position data. Here, as shown in FIG. 5, the position is expressed by six parameters. That is, the first three parameters (x1, y1, z1) represent the position of the article (using the center of gravity or the like), and the last three parameters (11, m1, n1) represent the orientation of the article. The handler in the article history data is the moving object specified by the handler specifying unit 32.
[0061]
3) Article attribute data: This stores physical attribute information possessed by an article. Examples of the attribute information include, as shown in FIG. 5, image data of the weight / shape of the article and its appearance.
[0062]
As one of the article history data, as described above, the article handling history at a specific place in the environment may be managed by another sub-database. For example, FIG. 7 is an example of a carry-out / carry-in DB that stores a history of taking out and carrying-in of articles through a door. This DB is composed of four items: the time of handling, the article handled, the handling content, and the operator. Here, "Article name @ bag C" in the item of the handled article indicates that the article has passed through the door in a state of being put in the bag C. This can be detected by installing a reader / writer in the bag. From this carry-out / carry-in DB, it can be seen that at time t2, the dad put clothes A, towel C, and bulk B into bag C and took them out of the environment (room).
[0063]
Further, the destination and purpose (for example, business trip) of the father holding the bag C may be registered in the carry-in / carry-in DB. In this way, if a destination and a purpose are specified in the future, a necessary article (for example, an article necessary for a business trip) can be specified by referring to the carry-out / carry-in DB. It is also possible to execute the operation of putting the necessary articles into the bag C.
[0064]
As shown in FIG. 6, the moving object DB is composed of sub-databases each storing two types of data, a moving object list and moving object history data, and the data contents stored in each sub-database are as follows. is there.
[0065]
1) Moving object data: An ID for distinguishing each moving object and a pointer to moving object history data are stored. The mobile object stored in the mobile object data may be manually registered in advance by the user.
[0066]
2) Moving body history data: The moving body history data includes three items: a position of the moving body at a certain time and a state at the time. Here, the moving object, unlike the article, occupies a large volume in the space, and becomes an obstacle when the robot 102 moves in the environment. For this reason, it is desirable to represent the position of the moving object as realistically as possible. Here, the approximate area occupied by the moving object on the floor is represented by a circle in order to represent the position of the moving object with the minimum necessary information so that the robot 102 can create a route while avoiding obstacles. I do. That is, the position of the moving object is represented by the XY coordinates (x4, y4) of the center position of the circle and the radius (r1) of the circle. Note that the area occupied by the moving object may be represented more strictly. For example, the position of the moving object may be represented using a plurality of line segment vectors that approximate the outline of the area occupied on the floor of the moving object.
[0067]
Further, the state of the moving object in the moving object history data indicates a general motion of a person such as “sit,” “stand,” “sleep,” “walk,” if the moving object is a person. 102 indicates an operation performed by the robot 102 on an article, such as “gripping” or “release”. For example, a plurality of state candidates that can be the state of the moving object may be prepared in advance, and it may be determined which state candidate the state of the moving object corresponds to based on the detection result by the sensing means 104 or the like. In the moving body history data relating to the robot 102, not only the operation but also the article ID of the work target is stored as "article ID: operation content". Specifically, it is “kan_small_0001: grip”.
[0068]
Here, how the information detected by the sensing means 104 and the information of the moving object such as the person and the robot 102 are stored in the article / moving object DB 106 by the article / moving object search / management means 105 is described. Further, how the data of the article / mobile object DB 106 is updated will be specifically described with reference to FIGS.
[0069]
FIG. 8 is a diagram showing an environment (room), and shows how two can juices 21 and 22 are brought into the environment and the can juices 21 and 22 move in the environment. In the figure, the movement of the can juices 21 and 22 is indicated by arrows, and the characters t1, t2, t5 and t6 attached to these arrows indicate the time at which the can juice was moved according to the arrows. Show. It is assumed that the time has advanced in the order of t1 to t6.
In the environment shown in FIG. 8, a reader / writer of an electronic tag is provided as sensing means 104 (not shown), and each can juice 21 and 22 and a person and a robot 102 (shown in and out of the environment). (Omitted) are each provided with an electronic tag. The article attribute data of each can juice 21 and 22 is to be read by a reader / writer from the electronic tag, and the registration of each moving object in the moving object data of the article / moving object DB 106 and the initialization of the moving object history data have already been performed. It is assumed that Further, it is assumed that the article data in the article / mobile object DB 106 is an empty state in which nothing is recorded.
[0070]
First, a father (not shown), which is a mobile object, enters a room (environment) with a can juice 21. When the sensing means 104 detects the father and the can juice 21, the detection result is sent to the article / moving object search / management means 105. The article / moving object search / management means 105 receiving the information assigns an ID [kan_small_0001] to the can juice 21 and stores the ID in the article data in association with a pointer to the article attribute data. At the same time, the article attribute data of the can juice 21 is stored. Further, “position history list 1” for storing the movement history of the can juice is created (see FIG. 5). At this point, the contents of the position history list 1 are still empty. On the other hand, to update the father's moving body history detected by the sensing means 104, the "position history list 3" is called by referring to the position history of the ID "dad" in the moving body data (see FIG. 6). ).
[0071]
At time t1, the dad sits at a position P4 (x4, y4) near the table and places the can juice 21 that he had at P1 (x1, y1, z1) on the table. When the sensing means 104 detects this, the article / moving object search / management means 105 sets the contents of the position history list 1 of the article history data (see FIG. 5). In particular,
Time: t1
Handling contents: New
Handler: Dad
Position after handling: P1 (x1, y1, z1, l1, m1, n1)
Is set. Here, the handling content being “new” means that an article that did not exist in the environment until then has been newly brought in from the outside. The article / moving object search / management unit 105 sets the contents of the position history list 3 of the moving object history data (see FIG. 6). In particular,
Time: t1
Position: (x4, y4, r1)
Condition: Sit
Is set.
[0072]
At time t2, another moving body, a son (not shown), moves the can juice 21 placed on P1 (x1, y1, z1) to P2 (x2, y2, z2) on the floor. Let it. When the sensing means 104 detects this, the article / moving object search / management means 105 sets new contents of the position history list 1 of the article history data (see FIG. 5). In particular,
Time: t2
Handling contents: Move
Handler: Son
Position after handling: P2 (x2, y2, z2, l2, m2, n2)
Is set. Here, the handling content being “moved” means that an article already registered in the article history data has been moved. At time t2, since the position of the son changes, the article / moving object search / management unit 105 sets the contents of the son's moving object history data (position history list 4), but illustration thereof is omitted here. .
[0073]
At time t3, the dad goes out of the room. When the sensing means 104 detects this, the article / moving object search / management means 105 sets new contents of the position history list 3 of the moving object history data (see FIG. 6). In particular,
Time: t3
Position:-
Condition: going out
Is set. Here, the position being “-” means that the position has gone out of the environment and has thus become out of the management of the present system.
[0074]
When the father goes out of the room, the father operates the operation terminal 103 to return the can juice 21 moved by the son to the original table P1 (x1, y1, z1). An instruction is given (the details of the instruction to the robot 102 will be described later). The robot 102 receiving the instruction moves to the position P2 (x2, y2, z2) of the can juice, and grasps the can juice at time t4. The article / moving object search / management means 105 sets the contents of moving object history data (position history list 5) of the robot 102 (see FIG. 6). In particular,
Time: t4
Position: (x2, y2, r2)
Status: [kan_small_0001]: grip
Is set. Note that the operation of the robot 102 may be detected by the sensing means 104, but the operation of the robot 102 may be detected by the server 101 receiving operation information from the robot 102 via a network.
[0075]
At time t5, the robot 102 moves to the vicinity P4 (x4, y4) of the table while holding the can juice, and releases the held can juice at P1 (x1, y1, z1) on the table. The article / moving object search / management means 105 sets new contents of the position history list 5 of the moving object history data (see FIG. 6). In particular,
Time: t5
Position: (x4, y4, r2)
Status: [kan_small_0001] released
Is set.
[0076]
Further, the article / moving object search / management unit 105 sets new contents of the position history list 1 of the article history data (see FIG. 5). In particular,
Time: t5
Handling contents: Move
Handler: Robot
Position after handling: P1 (x1, y1, z1, l1, m1, n1)
After a while, another moving body, mother, enters the room with a new can juice 22. When the sensing means 104 detects the mother and the new can juice 22, the article / moving object search / management means 105 assigns an ID of [kan_small_0002] to the new can juice 22 and associates the new can juice 22 with a pointer to the article attribute data. And store it in the article data. The article attribute data of the new can juice is the same as the can juice 21 described above. Further, a “position history list 2” for storing the movement history of the new can juice 22 is created (see FIG. 5). At this point, the contents of the position history list 2 are still empty.
[0077]
At time t6, the mother puts a new can juice 22 on P3 (x3, y3, z3) on the floor. When the sensing means 104 detects this, the article / moving object search / management means 105 sets the contents of the position history list 2 of the article history data (see FIG. 5). In particular,
Time: t6
Handling contents: New
Handler: Mother
Position after handling: P3 (x3, y3, z3, l3, m3, n3)
Is set. In FIG. 6, the moving object data and the moving object history of the mother are omitted.
[0078]
As described above, the article / moving object search / management unit 105 stores / updates information of the article / moving object in the article / moving object DB 106. Here, an example has been described in which the database is updated each time the position of each article or moving object registered in the article / moving object DB 106 is changed, but the update timing of the database is not limited to this. What is necessary is just to set suitably.
[0079]
It is desirable that the article history data and the moving body history data in the article / moving body DB 106 be accumulated for as long as possible so that the histories can be checked retroactively. Further, it is desirable that the moving object history data be stored at as short a time interval as possible so that the moving path of a moving object such as a person or a robot can be managed.
[0080]
However, since the capacity of the article / moving object DB 106 is limited, data for a certain period is stored, and data past that period may be deleted at any time.
Further, when the state change of the moving object is severe, the time interval for accumulating data may be shortened, and when the state change is small, the time interval may be lengthened.
[0081]
The environment map management means 107 in the environment management server 101 creates an environment map 108 based on the information from the sensing means 104 and manages the created environment map 108. The environment map 108 is used when the robot 102 moves in the environment. The robot 102 acquires the environment map 108 from the server 101 and makes a movement route plan, as described later.
[0082]
The environment map management means 107 also sends necessary information to the control means 110 in accordance with the contents of the inquiry when the environment map 108 is inquired from the control means 110, which will be described in detail later. .
[0083]
For example, in the case where the actual situation of the environment is shown in FIG. 9A, the environment map 108 may be obtained by simplifying the actual situation with a three-dimensional model (FIG. 9B). Further, as shown in FIG. 9C, a further simplified model with a plane model may be used as the environment map 108. Further, a three-dimensional model of the actual situation of the environment shown in FIG. 9A may be used as the environment map 108. That is, the environment map 108 may be created in accordance with the use of the map and the time (time and labor) required for creating the map. For example, when it is necessary to create an environment map composed of a three-dimensional model in a very short time, a three-dimensional object existing in the environment may be modeled with a minimum rectangular parallelepiped covering the three-dimensional object. The environment map 108 shown in FIG. 9B is an example of this, in which the table and the bookshelf are each modeled by a rectangular parallelepiped, and the trash can is modeled by a substantially cylindrical column. The same is true of the environment map composed of a plane model. In the work map 108 shown in FIG. (Shaded area). In the map 108 composed of the plane model, the two rectangular areas and the circular area are set as areas in which the robot 102 cannot move.
[0084]
FIG. 10 is a diagram showing an example of a facility database attached to the environment map, which corresponds to the environment shown in FIG. This equipment database is composed of sub-databases that respectively store two types of data: equipment data and equipment attribute data.
[0085]
1) Equipment data: To identify the environment itself and individual equipment in this environment (things that are fixed or installed in the environment, unlike articles, and are not subject to robot handling work). An ID and a pointer to equipment attribute data are stored. Here, an ID of “room_0001” is assigned to the environment (room), and IDs of “table_0001”, “bookshelf_0001”, and “flash_0001” are assigned to the tables, bookshelves, and trash cans existing in the environment.
[0086]
2) Facility attribute data: The floor data in the environment is accumulated in the facility attribute data relating to the environment itself. For example, when there are a plurality of floors having different heights in the environment, the floor data is accumulated by the number of the floors (floor 1, floor 2,...). This floor data is represented, for example, as follows.
((X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3), (X4, Y4, Z4), 2200, 0)
Here, the first four values (coordinates) indicate the coordinates (coordinates in real world coordinates) of each vertex constituting the floor, and the next value (2200) indicates the distance (ceiling) from the floor to the ceiling. mm). The last value (0) means the material of the floor surface. For example, “0” may be flooring, “1” may be tatami, and “2” may be carpet or the like.
[0087]
The equipment attribute data for equipment (furniture, etc.) includes data of each surface constituting the equipment (Surface 1, Surface 2,...), The type of equipment, and if the equipment has a surface on which articles can be placed, The shape and posture of the main article placed on the surface are accumulated. Specifically, the data of the surface constituting the equipment is represented, for example, as follows.
((X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3), 1,400)
Here, the first three values (coordinates) indicate the coordinates (coordinates in real world coordinates) of each vertex constituting the surface, and the next value “1” indicates whether an article can be placed on the surface. This is the flag. The value is "1" when the article can be placed, and "0" when the article cannot be placed. The last value “400” indicates the upper limit height (mm) of the mountable article when the article can be placed on the surface. For example, when the surface is a table top, the distance from the top to the ceiling is the upper limit height, and when the surface is a single shelf surface on a bookshelf, the distance from the shelf surface to the shelf directly above it. Is the upper limit height.
[0088]
The “main article shape” in the equipment attribute data is the shape of the article housed in the equipment. If the type of equipment is a bookshelf, it will be a "book shape". That is, a rectangular parallelepiped whose depth and height are extremely longer than its width is the shape of the main article of the bookshelf. Further, the “posture of the main article” is the posture of the article when housed in the facility.
If the type of equipment is a bookshelf, this is the posture in which the book is to be placed on the shelf surface of the bookshelf. Usually, the book is in a standing position. Such data of “shape and posture of main items” include, for example, when the type of equipment is a shoe shelf (in this case, the main item is shoes) or in a dish dryer (in which case, the main item is tableware). Stored in equipment attribute data. However, depending on the type of equipment, there is a case where this “main article shape and attitude” data is not included. When the type of equipment is, for example, a table or a trash can, there is no limitation on the shape and posture of the article. For this reason, the equipment attribute data of the table or the trash can does not have the data of “the shape and posture of the main article”.
[0089]
In this way, by accumulating the data of “the shape and posture of the main article” in the equipment attribute data, when a work to move a book to a bookshelf, for example, is designated to the work robot 102, the work robot 102 The book specified on the basis of the data can be placed on the bookshelf in an upright position.
[0090]
The control means 110 in the environment management server 101 is a part for controlling the entire server, and the main control contents are as follows.
[0091]
1) When the transmission / reception unit 109 receives an inquiry about various data in the server, it determines the content of the inquiry, and sends the data to the article / mobile object search / management unit 105 and the environment map management unit 107 according to the result of the determination Issue a reference request.
[0092]
2) The result sent from the article / mobile object search management means 105 or the environment map management means 107 in response to the request is sent to the inquiry source via the transmission / reception means 109.
[0093]
3) Interpret the robot work content message transmitted from the operation terminal 103 via the transmission / reception means 109, generate a robot control command sequence for causing the robot 102 to execute an operation, and transmit the generated sequence to the robot 102. The robot control command sequence will be described later.
[0094]
4) If necessary, the status of part or all of the articles managed in the article / moving object DB 106 and the status of the environment map 108 are transmitted to the robot 102 or the user (operation terminal) via the transmission / reception means 109 at regular intervals. 103).
[0095]
For example, the controls 1) and 2) are used when the user operates the operation terminal 103 to search for an article. At this time, as the content of the inquiry to the server 101, the user may give a search condition with a natural language sentence such as “Who has put the money placed on the table in the living room at about △ / △ ××”. Alternatively, a search key for narrowing down attributes of an article that the user wants to know, such as a date, a handler, and a type of the article, may be input.
[0096]
In response to such an inquiry, the environment management server 101 searches the article / mobile object DB 106 for the article. Then, for example, the operation terminal 103 may be made to answer by voice, for example, "It has been changed to △△," or a map in the environment may be presented to indicate the current location of the article. May be.
[0097]
When the article is taken out of the environment, the information of the person who brought the article may be output. At this time, if the exporter is a person, the server 101 may inquire the exporter by telephone or e-mail about the current location of the article to be taken out.
Further, when the present location of the exporter is in an environment managed by another environmental management server, the server 101 may directly inquire the other environmental management server about the current location of the articles to be taken out.
[0098]
-Work robot configuration-
The work robot 102, which is the second subsystem, performs a work of handling articles in the environment. Here, in particular, the work robot 102 performs a moving operation for moving an article in the environment according to the operation content specified by the user.
[0099]
As shown in FIG. 1, the robot 102 has, as its basic components, an obstacle sensor 111 for detecting an obstacle or the like in the vicinity of the robot, a gripper 112 for gripping an article, and a travel plan for making a travel plan using an environment map 108. Creating means 113, moving means 114 for moving the robot 102 itself, transmitting / receiving means 109 for transmitting / receiving various data to / from the environment management server 101 and the operation terminal 103, these sensors 111 and respective means 109, 112 to 114 Is controlled by the control means 115.
[0100]
FIG. 11 is a schematic diagram showing an example of the structure of the robot 102 in the present system. The robot 102 has a substantially box-shaped main body 10 for accommodating the movement plan creating means 113 and the control means 115. . Hereinafter, the right side of FIG. 11 is referred to as a front side, the left side is referred to as a rear side, the back side of the page is referred to as a left side, and the near side of the page is referred to as a right side.
[0101]
The gripping means 112 comprises an articulated arm 12a and a hand 12b disposed at the tip of the arm 12a, and is attached to the upper surface of the main body 10. The arm 12a and the hand 12b may use an actuator controlled by a motor, or may use another actuator, for example, an actuator using artificial muscle.
[0102]
The moving means 114 is constituted by wheels 14, and two wheels 14 are attached to each of the left and right sides of the main body 10 (the left wheel is not shown in the figure). Here, the moving means 114 is constituted by the wheels 14, but the structure of the moving means 114 may be selected in accordance with the environment in which the robot is used. For example, when the floor of the environment is highly uneven, it is preferable to configure the moving means 114 as a crawler type or a multi-legged type.
[0103]
In the present system, the obstacle sensor 111 includes an ultrasonic sensor 11a, a camera 11b as a visual sensor, and a collision sensor 11c.
[0104]
The ultrasonic sensor 11a calculates an approximate distance from the sensor 11a to an obstacle by measuring a time until an ultrasonic wave is emitted and a reflected wave thereof is received. It is for detecting before colliding with it. Three ultrasonic sensors are attached to each side surface (front surface, rear surface, left and right side surfaces) of the main body.
[0105]
The camera 11b is for inputting the situation around the robot 102 as an image and performing processing such as recognition of the image to determine the presence or absence of an obstacle and to obtain more accurate information of the object to be grasped. is there. The camera 11b is attached to the front of the main body 10.
[0106]
The collision sensor 11c is a sensor that detects that a predetermined impact force is applied to the robot 102. For example, the collision sensor 11c detects that an obstacle has collided with the robot 102 or that the robot 102 itself has collided with the obstacle while moving.
The collision sensor 11c is attached to each of the front and rear surfaces of the main body.
[0107]
The movement plan creation means 113, when the movement of an article or other work is specified to the robot 102, specifies the movement path from the current position of the robot 102 to the specified position (target position). It is created using the map 108. At this time, there must be no obstacles on the movement path from the current position to the destination position. However, as described above, the environment map 108 indicates that the robot cannot move (for example, the hatched area in FIG. 9C). Area) is set. Therefore, if a movement route is created in an area other than the immovable area, a movement path that avoids obstacles is created. For example, when the robot is moved from the point A1 to the point A2 using the environment map 108 using the plane model of FIG. 9C, a route that avoids the immovable area in consideration of the size of the robot 102 (see FIG. 9C). 9 (c) is created). In creating such a moving route, the Dijkstra method, which is the most general method, may be used, or if the environment is complicated, a route search algorithm obtained by improving the Dijkstra method may be used.
[0108]
In addition, as a countermeasure in a case where the environment condition is too complicated to calculate the movement route of the robot 102 or a long time is required for the calculation, a mode in which the user specifies the movement route of the robot 102 is provided. You may.
[0109]
The control means 115 of the work robot 102 mainly interprets a robot control command sequence sent from the environment management server 101 via the transmission / reception means 109 and executes the robot control commands in order.
[0110]
Here, the robot control command is a command for gripping an article or controlling the movement of the robot 102 itself. There are mainly three types of "movement", "gripping", and "release". These three types of commands will be briefly described.
[0111]
1) Movement: (move, coordinate) or (move, equipment ID)
This is a command for moving from the current position of the robot 102 to a position specified by coordinates or a position of equipment specified by equipment ID. The coordinates are designated in the world coordinate system, and the movement route from the current position to the destination position is planned by the movement plan creating means 113.
[0112]
Further, when moving to the position of the equipment specified by the equipment ID, a route is created so as to approach the equipment to a predetermined distance, but the coordinates of the equipment use the equipment attribute data in the environment map.
[0113]
2) Grasping: (grab, article ID)
This is a command for holding the article specified by the article ID with the hand 12b. The location of the article is referred to the article DB, and the grasping plan is created by the grasping means 112.
[0114]
3) Release: (release)
This is a command for releasing the hand 12b.
[0115]
The three types of robot control commands have been briefly described above. Of course, the robot control commands are not limited to these three types, and needless to say, may be increased as necessary.
[0116]
In the present embodiment, the work robot 102 is a self-propelled robot having the moving means 114 constituted by the wheels 14, but the work robot 102 is not limited to this structure. For example, the work robot 102 may be configured by assembling the gripping means 112 composed of the arm 12a and the hand 12b described above to a guide rail attached to a ceiling in the environment. The work robot 102 having this configuration can move to the designated position by the gripping means 112 being guided by the guide rail and thereby move the designated article in the environment to the designated position. Can be performed.
[0117]
Further, the work robot 102 may be configured by fixedly installing a plurality of gripping means 112 (composed of the arm 12a and the hand 12b) at predetermined positions in the environment. In this case, each gripping means 112 is arranged such that all the articles existing in the living space can be gripped by any of the gripping means 112. When moving the designated article to the designated position, the work robot 102 having this configuration selects the grasping means 112 that can reach the hand 12b from the plurality of grasping means 112, and The selected gripping means 112 may grip the article and move the article gripped by the arm 12a and the hand 12b to a designated position. When the specified position is a position where the arm 12a of the selected gripping means 112 cannot reach, the article may be moved to the specified position while handing the article between the plurality of gripping means 112.
[0118]
-Configuration of operation terminal-
An operation terminal 103, which is a third subsystem, is a user interface in the present system, and is a terminal operated by a user mainly to instruct a robot 102 to handle an article.
[0119]
As shown in FIG. 1, the operation terminal 103 has, as its basic configuration, a display section 117 for displaying an operation screen to be described later, for example, a CRT or a liquid crystal display. For example, an input unit 116 composed of a pointing device, a display control unit 118 for performing display control such as creation of an operation screen displayed on the display unit 117, and a work content of the robot 102 input by the input unit 116 are stored in an environment management server The transmission / reception means 109 for sending the information to the communication means 101, and the control means 119 for controlling these means 109, 116 to 118.
[0120]
For example, a general-purpose PC can be used as the operation terminal 103. In this case, the PC reads a control program for executing each process to be described later, so that the PC can be used as the operation terminal 103.
[0121]
The display control unit 118 includes information transmitted from the server 101, specifically, data of an image of the environment captured by a camera as the sensing unit 104, data stored in the article / mobile object database 106, An operation screen is created based on the map 108.
[0122]
Here, an operation screen created by the display control means 118 and displayed on the display unit 117 will be described. FIG. 12 shows an example of the operation screen, and this operation screen is configured by a virtual space conforming to the actual situation of the environment.
[0123]
Further, the virtual space has a plurality of different viewpoints. In the present embodiment, as shown in FIG. 12A, a viewpoint in which the virtual space is viewed in a substantially horizontal direction, and in FIG. As shown, it has two viewpoints, a viewpoint overlooking the virtual space. The viewpoint of the virtual space forming the operation screen is switched by the user operating the operation terminal 103.
[0124]
These virtual spaces forming the operation screen are created based on image data captured by a camera (not shown) installed in the environment. As described above, in order to make a plurality of viewpoints in the virtual space based on the camera image, for example, the following method is used.
[0125]
One is to place a plurality of cameras in an environment, and configure a virtual space with image data captured by each camera. For example, a camera is installed on each of the side wall and the ceiling of the environment (room), and a virtual space of a viewpoint shown in FIG. The virtual data of the viewpoint shown in FIG. 12B may be configured by the image data captured by the installed camera.
[0126]
The other is to create an image (virtual space) from an arbitrary viewpoint by installing at least one camera at an appropriate place in the environment and deforming and synthesizing the camera image. Since a technique of synthesizing an image viewed from an arbitrary virtual viewpoint using such a camera image is described in Japanese Patent No. 3286306, its detailed description is omitted here.
[0127]
In this embodiment, a virtual space shown in FIG. 12A is formed by image data (hereinafter also referred to as a camera viewpoint image) captured by a camera installed as a sensing unit 104 on a side wall in the environment. A virtual space shown in FIG. 12B is configured by image data obtained by transforming and synthesizing image data captured by the camera (hereinafter also referred to as a virtual viewpoint image). As described above, it is not necessary to match the number of viewpoints of the virtual space forming the operation screen with the number of camera viewpoints for capturing image data forming the virtual space.
[0128]
The operation screen also has a cursor (pointer) and an article icon (in FIG. 12, a banana is an article icon) arranged in the virtual space corresponding to an article present in the environment. The article icon may be any icon as long as the user can recognize that the article is an article. For example, the article icon may be configured as an icon that animates (illustrates) the article, or a part of the article may be automatically extracted from a camera image of the article. Alternatively, it may be constituted by an image cut out manually.
[0129]
The user operates the cursor in the operation screen by operating the input unit 116 of the operation terminal 103 1, and designates the operation content to be executed by the robot by the icon operation. Although the procedure for specifying the work content on the operation screen will be described later, in the present embodiment, the article icon is configured to be able to move in a virtual space configuring the operation screen by a so-called drag operation, When the robot 102 is to perform a moving operation of an article, the cursor is used to specify a desired article icon in the virtual space, and a desired destination in the virtual space is specified by the cursor, thereby specifying the moving work. Is made. Note that facilities such as furniture existing in the environment are displayed on the operation screen, but they do not become article icons. Items that are regarded as article icons on the operation screen are articles that can be handled by the robot 102.
[0130]
Here, as described above, since the virtual space forming the operation screen is image data captured by the camera or image data obtained by transforming and synthesizing the image data, any area of the image data corresponds to the article icon. Need to be identified. Therefore, the operation screen has screen mask data for specifying the position of the article icon in the virtual space, in addition to the image data (and the deformed and synthesized image data) constituting the virtual space. The screen mask data corresponds to the image data (virtual space) on a one-to-one basis. Here, "corresponding one-to-one" means that the coordinate values of both data correspond.
[0131]
FIG. 12 shows an example of screen mask data that has a one-to-one correspondence with data of an image forming a virtual space (operation screen). In the screen mask data, mask data (see the shaded portion in the figure) is set corresponding to the area where the article icons are arranged on the operation screen, and the coordinates of the mask data correspond to the coordinates of the article database. A pointer to the article data (see FIG. 5) is described. This pointer corresponds to the article related to the article icon, and thereby, it is possible to determine which article each article icon included in the operation screen represents.
[0132]
Then, when the position (coordinate value) of the article icon is designated by the cursor on the operation screen composed of the virtual space, the same coordinate value is referred to in the screen mask data, whereby the designated area becomes the article (article icon). Is determined. At the same time, when the position of the article icon is designated, the article data is referred to using the pointer described at the coordinates as a key, whereby the article icon pointed by the cursor on the operation screen indicates which article. To identify
[0133]
When the virtual space forming the operation screen is configured by an image captured by a camera as the sensing means 104 (see FIG. 12A), a screen mask corresponding to the virtual space on a one-to-one basis by a background difference image. Data can be created. This is because the position of the article in the image captured by the camera is specified in the background difference image.
[0134]
On the other hand, when the virtual space forming the operation screen is configured by an image obtained by transforming and synthesizing a camera image, in other words, a virtual viewpoint image (see FIG. 12B), the screen mask data corresponding to the one-to-one correspondence is It can be created as follows.
[0135]
FIG. 13 is an explanatory diagram for creating image mask data of an operation screen composed of virtual viewpoint images. In FIG. 13B, a coordinate system including the origin at Ow, the X axis, the Y axis, and the Z axis is a real world coordinate system. In this coordinate system, the position of a virtual viewpoint (this virtual viewpoint corresponds to the ceiling of the environment) is represented by (x1, y1, z1). Also, a virtual viewpoint coordinate system having the virtual viewpoint Oe as the origin is set. Further, coordinate values (x 1, y 2, z 2) shown in FIG. 13A are coordinate values in the real world coordinate system of any one point in the area constituting the article detected by the sensing means 104. Further, the coordinate values (u, v) shown in FIG. 12B are coordinate values when the above coordinate values (x, y, z) are converted into an operation screen including a virtual viewpoint image. The following equations (1), (2), and (3) are equations for conversion.
[0136]
(Equation 1)

[0137]
Here, R in equation (1) is a rotation matrix around three axes constituting the virtual viewpoint coordinate system, and by multiplying this matrix by the coordinate values in the virtual viewpoint coordinate system, the coordinate values are converted into the real world coordinate system. The value of the rotation matrix is set so as to be converted into the coordinate value of. F in the expressions (2) and (3) indicates the focal length of the virtual viewpoint.
[0138]
Now, the creation of the screen mask data of the operation screen composed of the virtual viewpoint image, as a specific process, occupies the area occupied by the article obtained by the sensing means 104 when viewed from the virtual viewpoint. That is, the image data is converted into an area, the area is converted into coordinate values on the operation screen, and the converted values are used as mask data. This means that
1) Convert the point (x 1, y 2, z 2) of the article area represented by the coordinate value of the real world coordinate system into the coordinate value (xe, ye, ze) of the virtual viewpoint coordinate system by equation (1),
2) The converted coordinate values (xe, ye, ze) are converted into the coordinate values (u, v) of the operation screen composed of the virtual viewpoint images by equations (2) and (3).
Is performed for all points in the article area represented by the coordinate values of the real world coordinate system.
[0139]
Screen mask data of an operation screen composed of virtual viewpoint images can be created by the above conversion. To create more accurate mask data, the coordinate values of (u, v) obtained in 2) are plotted. Alternatively, the smallest polygonal area surrounding them may be used as mask data.
[0140]
In this embodiment, the display control means 118 of the operation terminal 103 creates the operation screen. However, such an operation screen may be created by the environment management server 101. In this case, the operation screen created by the server 101 may be transmitted to the operation terminal 103 via the network, and the operation terminal 103 may display the received operation screen on the display unit 117. Also, when a viewpoint switching operation in the virtual space is performed by the operation of the input unit 116, a viewpoint switching request signal is transmitted from the operation terminal 103 to the server 101, and the server 101 switches the viewpoint accordingly. May be created and transmitted to the operation terminal 103.
[0141]
-Robot work content designation procedure-
Next, a procedure for designating an article moving operation to be executed by the robot 102 on the operation screen displayed on the display unit 117 of the operation terminal 103 will be specifically described with reference to two examples.
[0142]
First, referring to FIG. 14, a procedure for designating a moving operation of the robot 102 will be described by taking as an example a case where a banana placed at a position B1 on the floor of a room is moved to a position B2 on the near side of a table. .
[0143]
This movement operation is specified by an operation of selecting a banana (article icon) on the operation screen, moving the selected banana icon to a desired destination in the virtual space (drag), and setting the destination position. The drop of the banana icon is performed by a drag-and-drop operation of the icon.
[0144]
(Operation screen display step: P1001)
An operation screen is displayed on the display unit 117 of the operation terminal 103. Here, the virtual space forming the operation screen is configured by a camera viewpoint image captured by the camera. Since this virtual space conforms to the actual situation of the room, the operation screen displays facilities (tables and the like) existing in the room. Articles (bananas and the like) existing in the room are arranged in the virtual space as article icons.
[0145]
The user can move the cursor on the operation screen by operating the input unit 116 of the operation terminal 103. When the cursor points to an article icon, the article icon is highlighted. Specifically, when the cursor points to the banana icon, the color of the banana icon is changed, and as shown in FIG. 16A, the outline of the banana icon is displayed in an emphasized manner. As shown in b), the character "banana" is displayed in a pop-up as the name of the article corresponding to the article icon. On the other hand, even if the cursor points to equipment such as a table, these are not product icons and are not highlighted. By doing so, the user can recognize on the operation screen that, for example, a banana is an article that can be handled by the robot 102, and the operation content of the robot can be easily specified. As a result, user convenience is improved.
[0146]
When the name of the article is displayed in a pop-up, other information about the article may be displayed together. For example, if the article is a food item, information such as the expiration date, the quality retention period, and the date of purchase may be displayed in a pop-up. By doing so, the user can specify the work content for the robot in accordance with the information (for example, change the moving location of the article). Further, a recommended work content for the article may be displayed in a pop-up according to the content of the information on the article. For example, when the item icon (article) pointed to by the cursor is a food item whose expiration date is about to expire, a work content of "moving the item into the refrigerator" is displayed in a pop-up.
[0147]
When the cursor is pointing at an article icon, not only the color of the article area is changed, or the outline of the article area is emphasized, but also the article may be enlarged and displayed. When the cursor is pointing at an article icon, the name of the article may be notified to the user by voice, for example, "there is XX". This makes the system easy to use for the visually impaired user.
[0148]
Although the display mode of the article icon is changed here, for example, when the cursor is pointing at the article icon, the form of the cursor may be changed. This also allows the user to recognize that the article icon pointed to by the cursor is an article that can be handled by the robot.
[0149]
Furthermore, here, when the cursor is pointing at the article icon, the article icon is highlighted, but, for example, all the article icons arranged in the virtual space are highlighted for a predetermined time. Is also good. In particular, when the user starts an operation on the operation screen, all the article icons may be highlighted for a predetermined time, or according to the user's requested operation, all the article icons may be highlighted for a predetermined time. Is also good. By doing so, the user can confirm at a glance where the article icon in the virtual space, that is, the article that can be handled by the robot 102 is located in the environment.
[0150]
(Article specifying operation step: P1002)
The article to be moved by the robot 102 is designated by moving the cursor to the position of the article icon and clicking on the article icon. When the operation of specifying the article icon is performed, a display prompting confirmation such as “Is the target article a banana?” May be displayed, and the user may perform an “OK / Cancel” operation.
[0151]
When an article icon is designated in this way, the article icon is highlighted to make it clear. Various display forms can be considered as the highlighting. For example, as shown in FIG. 14, it can be seen at a glance that the article is designated, such as changing the color of the article icon or filling the icon. Any display form may be used as long as it is an appropriate form. However, it is preferable that the form of highlighting when the target article is designated is different from the highlighting performed when the cursor points to the article icon. This is for distinguishing that the article is an article that can be handled by the robot 102 and that the article is designated as a work target of the robot 102.
[0152]
(Destination designation step: P1003)
If an article to be moved by the robot 102 is designated, a destination of the article is designated.
As shown in FIG. 14, the destination is specified by dragging and dropping the article icon to a desired destination in the virtual space (here, the B2 position on the near side of the table) by operating the cursor. (See the arrow in the figure). When this operation is performed, a confirmation display such as "Is the destination here?" Thus, the designation of the moving work by the user is completed.
[0153]
As described above, the operation of moving an article is performed intuitively by dragging and dropping an article icon arranged in the virtual space corresponding to an article present in the environment to a desired destination in the virtual space. Therefore, the user can very easily specify the work content to be executed by the robot.
[0154]
Next, referring to FIG. 15, a procedure for designating a moving operation of the robot 102 will be described by taking as an example a case where a banana placed at a position B1 on the floor of a room is moved to a position B2 on the back side of a table. I do.
[0155]
(Operation screen display step: P2001)
The display step of the operation screen is the same as that of P1001 described above, and the operation screen composed of the virtual space composed of the camera viewpoint images is displayed on the display unit 117 of the operation terminal 103.
[0156]
(Article Designation Operation Step: P2002)
The article designating operation step is also the same as in P1002 described above, and designates an article to be moved to the robot 102 by clicking on an article icon. As a result, the designated article icon is highlighted.
[0157]
(Viewpoint switching step: P2003)
If an article to be moved by the robot 102 is designated, a destination of the article is designated. Here, in this example, the movement destination is the B2 position on the far side of the table, and the B2 position is not displayed on the operation screen including the camera viewpoint image. Therefore, the viewpoint of the virtual space constituting the operation screen is switched.
[0158]
When the viewpoint is switched, a camera icon is displayed at the viewpoint position on the operation screen as shown in FIG. The user can switch the viewpoint of the virtual space forming the operation screen by clicking the camera icon. Here, since there are two viewpoints in the virtual space, only one camera icon is displayed. However, when there are three or more viewpoints in the virtual space, a plurality of camera icons are displayed in correspondence with each viewpoint position. It may be displayed on the operation screen.
[0159]
(Destination designation step: P2004)
By switching the viewpoint of the virtual space, the virtual space forming the operation screen is switched to a virtual viewpoint image obtained by transforming and synthesizing the camera image. Then, on this new operation screen, by operating the cursor, the article icon is dragged and dropped to a desired destination in the virtual space (here, the position B2 on the back side of the table) (see the arrow in the figure). ). Thus, the designation of the moving work by the user is completed.
[0160]
As described above, since there are a plurality of viewpoints in the virtual space that constitute the operation screen, even when a certain viewpoint becomes a blind spot, the problem is solved by switching the viewpoint. Thus, in a living space such as a general home, there is a high degree of freedom in the arrangement of equipment and the position where articles are placed, and blind spots are likely to occur in the camera image. You can specify exactly above.
[0161]
-Execution control of robot work contents-
As described above, when the user operates the operation terminal 103 to specify the work content to be executed by the robot 102, the operation terminal 103 transmits a message of the work content to the environment management server 101. This message may include at least the fact that the work content is the moving operation of the article, information on the article to be moved, and the coordinates of the movement destination. The coordinates of the movement destination transmitted from the operation terminal 103 to the server 101 may be coordinates on the operation screen, or coordinates obtained by converting the coordinates on the operation screen into real world coordinates of the environment.
[0162]
Upon receiving the message, the server 101 generates a robot control command sequence according to the content of the message, and transmits the generated robot control command sequence to the robot 102. The work of moving an article is broken down into four work units: “moving (to the position of the article)”, “gripping (to the article)”, “moving (to the destination)”, and “opening (to the article)”. You. Therefore, in the above example, the robot control command sequence is
move, B1 (moves the robot to the position B1 where the banana is placed)
grab, banana (grab banana at position B1)
move, B2 (moves to the destination B2 (while holding the banana))
release (releases the banana being gripped)
It becomes. When the movement of a plurality of articles is specified, the command string is arranged by the number of articles, with the above-mentioned four work units as one set.
[0163]
FIG. 17 shows a flowchart relating to the control means 115 of the work robot 102. When the transmission / reception unit 109 of the work robot 102 receives the control command sequence transmitted from the server 101, the control unit 115 determines which work unit it is in steps S2401 to S2403, and determines Execute the corresponding process.
[0164]
First, if the work unit is “move” in step S2401, the process moves to step S2404, and the route from the current position to the designated position (here, the position B1 or the position B1 to the position B2) is defined by the movement plan creation unit. 113.
[0165]
In the following step S2405, a movement control command is sent to the movement means 114 according to the route created by the movement plan creation means 113, and the movement processing to the designated position is executed.
[0166]
On the other hand, when the work unit is “gripping” in step S2402, the process proceeds to step S2406, where the obstacle sensor 111 detects the posture of the object to be grasped, and in step S2407, an arm is provided in accordance with the detection result. The operation of the hand 12a and the hand 12b is calculated. Then, in step S2408, a gripping control command is sent to the gripping means 112 to execute gripping processing of the article. In step S2406, the position of the article is detected by the obstacle sensor 111. Since the attitude of the article is recorded in the article DB 106 of the environment management server 101, the server 101 is inquired about the attitude of the article. Is also good.
[0167]
If the work unit is “released” in step S2403, the process moves to step S2409 to calculate the operations of the arm 12a and the hand 12b so that the article is set at the designated destination. Then, in step S2410, a release control command is sent to the gripper 112, and an article release process is executed.
[0168]
In step S2411, when the movement, gripping, and release operations are completed, a message to that effect is transmitted to the work instruction source (server 101).
[0169]
Thus, the work content specified by the operation terminal 103 is executed by the robot 102, and the banana placed at the position B1 is moved to the position B2.
[0170]
When the robot 102 is operating, the operation screen displayed on the display unit 117 of the operation terminal 103 is linked with the camera image so that the operation of the robot 102 is displayed on the operation screen. You may. By doing so, the user can check on the operation screen whether or not the specified work content is being executed by the robot 102. For example, when the robot 102 is performing an incorrect work, the work can be immediately performed. It will also be possible to cancel.
[0171]
On the other hand, when the robot 102 is operating, the operation screen may be fixed with a camera image before the operation of the robot 102 starts. By doing so, even if an attempt is made to designate a new work, the designation operation may become difficult if the robot 102 is operating within the operation screen, but this is eliminated.
[0172]
Switching between these display modes may be performed in accordance with the user's specification. When the robot 102 is operating, the change frequency of the operation screen linked to the camera image may be adjustable.
[0173]
As described above, the article handling system according to the present invention includes a robot 102 for handling an article in a predetermined living space (environment), a display unit 117 for displaying an operation screen, and an input for user operation. Operating means (operation terminal 103) having a unit 116, displaying an operation screen composed of a virtual space in accordance with the real situation of the living space on the display unit 117, and allowing the user to operate the input unit 116. Thus, the operation content to be performed by the robot 102 in the virtual space is designated, and the robot 102 executes the article handling operation in the living space based on the specified operation content.
[0174]
As described above, in the present invention, the operation screen is configured by the virtual space conforming to the actual situation of the living space. For this reason, the actual situation of a living space such as a general home is more complicated than the situation of a factory or a warehouse, but the complicated situation is displayed as it is on the operation screen. For example, the arrangement of the articles in the living space is not fixed, but the articles are also arranged in the virtual space corresponding to the arrangement of the articles in the living space. Thus, the situation in the complicated living space is presented to the user accurately and easily.
[0175]
Further, in the present invention, the work content of the robot 102 is specified in this virtual space. This makes it possible to easily specify the work content of the robot 102 in the living space under a complicated situation.
[0176]
Further, when the operation performed by the robot is a moving operation of an article in the living space, the article handling system according to the present invention displays the operation screen in the virtual space corresponding to the article existing in the living space. Assuming that the article icon is to be arranged, the operation of moving the article is designated by operating the article icon.
[0177]
In this way, by arranging the article icons in the virtual space corresponding to the articles existing in the living space, it is possible to easily specify the article to be moved by the robot from the articles placed at various positions in the living space. It becomes possible to do. In addition, by specifying the work content of the robot 102 by operating the article icon, the work content can be accurately and easily specified.
[0178]
Further, in the article handling system according to the present invention, the operation of moving the article is designated by dragging and dropping a desired article icon to a desired destination in the virtual space. For this reason, when the user specifies the work of moving an article, the user selects a desired article icon from among the article icons arranged in the virtual space on the operation screen, and displays the selected article icon in the virtual space. A drag and drop operation is performed to a desired destination. With this operation, the robot 102 moves the article corresponding to the designated article icon to the position in the living space corresponding to the designated destination in the virtual space.
[0179]
As described above, since the moving operation of the article is specified by an intuitive operation of dragging and dropping the specified article icon to a desired destination in the virtual space, the operation content to be executed by the robot 102 is extremely limited. It can be easily specified.
[0180]
Further, the article handling system according to the present invention further includes an imaging unit (a camera as the sensing unit 104) for imaging a living space, and the virtual space of the operation screen is configured by data of an image captured by the imaging unit. You.
[0181]
This makes it possible to easily create a virtual space according to the actual situation of the living space. Note that the virtual space may be constituted by, for example, computer graphics.
[0182]
Furthermore, in the article handling system according to the present invention, the operation screen includes data of an image constituting the virtual space, an article icon arranged in the virtual space corresponding to an article existing in the living space, and data of the image. And screen mask data for specifying the position of the article icon in the virtual space.
[0183]
That is, when the virtual space is configured by image data, in order to operate the article icon on the operation screen, it is necessary to specify which region of the image corresponds to the article icon. Therefore, by configuring the operation screen from the image data, the article icon, and the screen mask data, the position of the article icon in the virtual space is specified by the screen mask data, and as a result, the screen is configured from the image data. It becomes possible to operate the article icon on the operation screen.
[0184]
The article handling system according to the present invention further includes a sensing unit 104 for detecting the position of an article present in the living space in the living space. Then, the screen mask data is created based on the position of the article detected by the sensing means 104.
[0185]
If the position of the article present in the living space is detected by the sensing means 104, the position in the virtual space corresponding to the detected position is specified. Therefore, screen mask data for specifying the position of the article icon in the virtual space. Can be created.
[0186]
As in the article handling system according to the present invention, in a system that targets a living space such as a general home, various articles are present in the living space. For example, a fixed / installed object fixed / installed in the space is also an article. However, such fixed / installed objects cannot be moved by the robot 102. As described above, the articles present in the living space can be divided into articles that can be handled by the robot 102 and articles that cannot be handled. Both the article and the unhandled article are displayed.
[0187]
Therefore, in the article handling system according to the present invention, when the pointer (cursor) on the operation screen points to an article icon corresponding to an article that can be handled by the robot 102, the article icon is highlighted or the article icon is highlighted. Display information about
[0188]
By doing so, the user can easily specify an article that can be handled by the robot 102 from a large number of articles displayed on the operation screen. Thereby, it is possible to easily specify the work content to be executed by the robot 102. It should be noted that the criterion for distinguishing between articles that can be handled by the robot 102 and articles that cannot be handled depends on the contents of the handling work. For example, in the case of moving objects, fixed / installed objects or large / heavy objects that cannot be moved by the robot 102 are objects that cannot be handled by the robot 102, and other objects are handled by the robot 102. It becomes a possible article.
[0189]
Further, when the work robot 102 in the system is changed, there is a case where the articles that can be handled are changed with the change of the robot 102. Here, the change of the robot 102 includes not only a hardware change that the robot 102 is replaced with a new model, but also a software change that the control program of the robot 102 is changed.
[0190]
As described above, when an article that can be handled by the robot 102 is changed, all the article icons arranged in the virtual space may be highlighted for a predetermined time in order to notify the user of the change. By doing so, the user can grasp at a glance the articles that the robot 102 can handle.
[0191]
The article handling system according to the present invention may be configured such that when an article that can be handled by the robot 102 is changed, all the article icons corresponding to the article that can be handled by the robot 102 are highlighted.
[0192]
In the article handling system according to the present invention, when an article to be worked by the robot 102 is designated by designating the article icon on the operation screen, the article icon is highlighted.
[0193]
By doing so, the user can confirm the designation of the article icon, thereby preventing the robot 102 from erroneously performing the handling operation of the designated article.
[0194]
As described above, the article handling system according to the present invention is an article handling system in a living space such as a general home, office, hotel, store, and hospital. For this reason, in the living space, there are fixed / installed objects such as furniture, and articles to be handled by the robot also exist at various positions.
[0195]
In this case, when the operation screen is configured from a virtual space conforming to the actual situation of the living space, if only one viewpoint of the virtual space is set, for example, objects are hidden from furniture or the like from that viewpoint. There are problems such as being invisible or when the article is moved to a desired destination, the destination is in a blind spot. This problem does not normally occur in industrial systems such as factories and warehouses. This is because, in industrial systems, articles are often placed at fixed positions, and even if there is only one viewpoint to see the situation of the space, the situation of the space is controlled so that blind spots do not occur. This is because it is set in advance.
[0196]
Therefore, in the article handling system according to the present invention, the viewpoint of the virtual space constituting the operation screen can be switched.
[0197]
Since a plurality of viewpoints in the virtual space can be switched to each other, even when a certain viewpoint becomes a blind spot, the problem can be solved by switching the viewpoint. In this way, in the article handling system in a living space where the situation is complicated, it is possible to accurately specify the work content to be executed by the robot 102 on the operation screen.
[0198]
Further, the robot operation device (operation terminal 103) according to the present invention is a device for designating the operation contents of the robot 102 which performs an operation of handling an article present in a predetermined living space.
[0199]
The robot operation device includes a display unit 117 that displays an operation screen, and an input unit 116 that is operated by a user. The operation screen including a virtual space according to the actual situation of the living space is displayed on the display unit 117. By displaying the information and allowing the user to operate the input unit 116, the operation content to be performed by the robot 102 in the virtual space is designated.
[0200]
With this configuration, as described above, the display unit 117 displays an operation screen including a virtual space according to the actual situation of the living space. Thereby, the complicated situation in the living space is presented to the user accurately and easily.
[0201]
In addition, since the operation content of the robot 102 is specified on the operation screen including the virtual space, the user can easily specify the operation content of the robot 102.
[0202]
The article handling method according to the present invention is a method using a robot 102 for moving an article present in a predetermined living space.
[0203]
Then, the article handling method comprises a virtual space according to the real situation of the living space, and displays an operation screen having an article icon arranged in the virtual space corresponding to an article present in the living space. A step of specifying a desired article icon on the operation screen; a step of dragging and dropping the specified article icon to a desired destination in the virtual space; Moving the article corresponding to the designated article icon to a position corresponding to the designated destination in the virtual space.
[0204]
The article handling method according to the present invention further includes a step of switching viewpoints of a virtual space forming the operation screen. In this embodiment, the step of switching the viewpoint is performed before the step of dragging and dropping the article icon to the desired destination in the virtual space. However, instead of or in addition to this, the desired article is switched. It may be before the step of designating the icon.
[0205]
-Other embodiments-
In the present embodiment, the environment management server 101 creates a robot control command in response to a work content message from the operation terminal 103 and transmits the command to the work robot 102. For example, the operation terminal 103 operates A robot control command may be created in accordance with the work content specified on the screen, and transmitted to the work robot 102.
[0206]
Further, in the present embodiment, the article handling system is composed of three subsystems: an environment management server 101, a robot 102, and an operation terminal 103, and these subsystems 101 to 103 exchange information with each other via a wireless or wired network. It was configured to communicate. However, the article handling system is not limited to this configuration. For example, a configuration in which the operation terminal 103 is integrated with the environment management server 101 may be used.
[0207]
Also, the configuration may be such that a plurality of robots 102 work in parallel while cooperating, instead of one.
[0208]
Further, in the present embodiment, it is assumed that the robot 102 performs the work of moving the article, and the operation content of the robot 102 is specified by the drag and drop operation of the article icon on the operation screen including the virtual space according to the actual situation of the environment. I did it. However, the work content of the robot 102 specified on the operation screen including the virtual space is not limited to the work of moving the article, and may be other work including the work not involving the movement of the article. For example, when an article icon to be operated by the robot 102 is specified on the operation screen, the work contents executable by the robot 102 are displayed on the operation screen in a menu format, and the work contents are selected from the menu. By doing so, the work content to be executed by the robot 102 may be designated.
[0209]
【The invention's effect】
As described above, in the article handling system, article handling method, and robot operation device for living space according to the present invention, the operation screen composed of the virtual space according to the actual situation of the living space is displayed on the operation terminal (robot operation device). Is displayed on the display unit. Thus, the user can accurately grasp the complicated situation in the living space. Further, the user can specify the work content to be executed by the robot on this operation screen. For example, when designating a moving operation of an article, by operating an article icon arranged in the virtual space corresponding to the article existing in the living space, for example, a drag and drop operation to a desired destination in the virtual space By doing, it is possible to specify the moving work. With such intuitive operation, the work content to be executed by the robot can be specified, so that anyone can easily specify the work content.
[0210]
In addition, when the cursor is pointing to an item icon that can be handled by the robot on the operation screen, a display is displayed to clearly indicate this to the user, so that the user can reliably specify the item to be worked by the robot. it can.
[0211]
Further, when an article (article icon) to be operated by the robot is specified, a display is displayed on the operation screen so that the user can recognize the operation, so that the robot performs an operation for handling the erroneously specified article. Can be prevented beforehand.
[0212]
In addition, since the viewpoint of the virtual space forming the operation screen is configured to be switchable, for example, a blind spot item can be viewed from a certain viewpoint by switching the viewpoint, and from a certain viewpoint, You can see the blind spot by switching the viewpoint. That is, it is possible to reliably display the entire virtual space according to the complex real situation of the living space on the display unit of the operation terminal.
[0213]
In addition, a virtual space conforming to the actual situation of the living space is formed by image data captured by the camera, so that the virtual space can be easily created.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an overall configuration of an article handling system according to an embodiment.
FIG. 2 is an explanatory diagram showing the principle of the background subtraction method.
FIG. 3 is a block diagram illustrating a configuration of an article / moving object search / management unit.
FIG. 4 is a diagram showing an example in which a gate type reader / writer is arranged on a door or a window in an environment.
FIG. 5 is a diagram showing a configuration of a product database and an example of contents described in the product / moving object database;
FIG. 6 is a diagram showing a configuration of a moving object database and an example of contents described in an article / moving object database.
FIG. 7 is a diagram showing a configuration of a carry-out / carry-in database and an example of description contents;
FIG. 8 is an explanatory diagram illustrating an example in which an article moves in an environment.
FIG. 9 is a diagram showing an actual situation of an environment and an environment map corresponding thereto.
FIG. 10 is a diagram showing a configuration of a facility database attached to an environment map and an example of description contents;
FIG. 11 is a schematic diagram illustrating an example of a configuration of a work robot.
FIG. 12 is a diagram illustrating a configuration of an operation screen including image data and screen mask data.
FIG. 13 is a diagram illustrating a relationship between a real world coordinate system and a virtual viewpoint coordinate system set in an environment.
FIG. 14 is a diagram showing a procedure for designating an article moving operation to be executed by the robot on the operation screen.
FIG. 15 is a diagram illustrating a procedure for designating an article moving operation different from that of FIG. 14;
FIG. 16 is a diagram illustrating an example of a display mode of an article icon on an operation screen.
FIG. 17 is a flowchart according to control means of the work robot.
[Explanation of symbols]
101: Environmental management server
102 ・ ・ ・ Work robot
103 ... operation terminal (operation means, robot operation device)
104 sensing means (imaging means)
106 ・ ・ ・ Article / mobile object database
108 ・ ・ ・ Environment map
110, 115, 119 ... control means
116 ・ ・ ・ Input unit
117 ... display unit
118 ··· display control means

Claims (14)

A robot that performs an operation for handling articles existing in a predetermined living space;
An operation unit having a display unit that displays an operation screen and an input unit that is operated by a user,
On the display unit, display an operation screen including a virtual space according to the actual situation of the living space,
By allowing the user to operate the input unit, the user is allowed to specify the work to be performed by the robot in the virtual space,
An article handling system, wherein the robot performs an article handling operation in the living space based on the designated work content. In claim 1,
An article handling system, wherein the operation performed by the robot is an operation of moving an article in a living space. In claim 2,
The operation screen has an article icon arranged in the virtual space corresponding to the article existing in the living space,
The article handling system is characterized in that an article moving operation is designated by operating the article icon. In claim 3,
The article handling system is characterized in that the article moving operation is designated by dragging and dropping a desired article icon to a desired destination in the virtual space. In claim 1,
Further provided is an imaging means for imaging a living space,
The article handling system, wherein the virtual space of the operation screen is configured by data of an image captured by the imaging unit. In claim 5,
The operation screen is
Image data constituting the virtual space,
An article icon arranged in the virtual space corresponding to the article existing in the living space;
An article handling system, comprising: coordinate values corresponding to the image data; and screen mask data for specifying the position of the article icon in the virtual space. In claim 6,
Further provided is a sensing means for detecting a position of the article present in the living space in the living space,
An article handling system, wherein the screen mask data is created based on the position of the article detected by the sensing means. In claim 1,
The operation screen has an article icon arranged in the virtual space corresponding to an article existing in the living space, and a pointer,
When the pointer points to an article icon corresponding to an article that can be handled by the robot on the operation screen, the article icon is highlighted. In claim 1,
The operation screen has an article icon arranged in the virtual space corresponding to an article existing in the living space, and a pointer,
When the pointer points to an article icon corresponding to an article that can be handled by the robot on the operation screen, information on the article is displayed. In claim 1,
The operation screen has an article icon arranged in the virtual space corresponding to the article existing in the living space,
An article handling system, wherein when an article to be worked by the robot is designated by designating the article icon on the operation screen, the article icon is highlighted. In claim 1,
An article handling system, wherein the viewpoint of the virtual space constituting the operation screen is switchable. A robot operation device for designating the operation content of a robot that performs an operation of handling an article present in a predetermined living space,
A display unit that displays an operation screen, and an input unit that is operated by a user,
On the display unit, display an operation screen including a virtual space according to the actual situation of the living space,
A robot operating device, wherein a user operates the input unit to specify a task to be performed by the robot in the virtual space. An article handling method using a robot that performs a work of moving an article present in a predetermined living space,
Displaying an operation screen including an article icon arranged in the virtual space corresponding to an article existing in the living space, the virtual screen being formed of a virtual space in accordance with the actual situation of the living space;
A step of designating a desired article icon on the operation screen;
Dragging and dropping the designated article icon to a desired destination in the virtual space;
Moving the article corresponding to the designated article icon in the living space to a position corresponding to the designated destination in the virtual space. In claim 13,
An article handling method, further comprising a step of switching a viewpoint of a virtual space forming an operation screen.

Images