JP2006338059A - Article management system, and control program and control method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an article management system for managing articles within a living environment, capable of operating articles hidden in a facility (a refrigerator, a drawer, a closet, a box or the like) through an interface means performing operation of articles on a screen in comforming with the real situation. <P>SOLUTION: The article management system 100 comprises a display means 119 displaying the real situation within a room, an article operation means 118 performing designation of an article using the display means 119, a facility inside display means 125 generating data for displaying information for the articles within the facility 104 on the display means 119, and a control means 120 displaying, upon designating the facility 104 by the operation means 118, information for the articles hidden within the facility 104 on the display means 119. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an article management system that manages articles present in a living environment, a control program for the article management system, and a control method therefor, and more particularly, to a system, program, and control method that facilitate operations on articles by a user.

  It is actually performed in many fields to make a robot do some work by moving an article. For example, gripping and transportation of parts in an automatic product assembly line in a factory, transportation and management of inventory items in an automated warehouse. On the other hand, today, attempts are being made to allow robots to work at home using the robot technology that has matured for industrial use. In other words, research and development of robots for home use has been increasingly carried out toward the grand goal of supporting human life while coexisting with humans at home.

  For example, it is equipped with an entertainment robot that heals a person's heart by behaving like a pet in an environment where no pets can be kept, and a number of sensors that can detect obstacles in the room. However, so-called housework support robots, such as automatically cleaning the room, have been developed and commercialized. In addition, as elemental technologies, technologies that are indispensable for actually providing various housekeeping support, such as handling technologies for freely grasping and transporting items in the home, sensing technologies for recognizing the situation in the space where the robot moves, etc. It has also been developed extensively.

  However, even if the handling technology and sensing technology are matured, it is not always possible to realize housework support by robots immediately. This is because, in addition to the above-described technology, it is indispensable to develop a technology for easily giving instructions to the robot, that is, an interface technology. In other words, it is necessary to have a technique for improving the user-friendliness. For example, it is a technique that can easily instruct operations such as carrying an article in the home from one place to another, or throwing away trash.

  In the case of an industrial robot, such an operation instruction is relatively simple. For example, Patent Literature 1 discloses a method for simply limiting an article to be handled and instructing to load the article on a pallet. In this method, a virtual article diagram corresponding to the actual situation is displayed on the monitor, and the user gives an instruction while viewing the virtual article diagram. That is, the user instructs an operation procedure for loading a rectangular parallelepiped article on the pallet by moving the article on the virtual article diagram and inputting the position. Such a moving operation is performed by an intuitive operation such as mouse dragging and dropping. In the above method, the procedure for loading articles on the pallet is completed by repeating such instruction work.

  The system that executes the above method sequentially accumulates the location before moving and the location after moving all items as work information through these instruction operations. Then, when the instruction is finished, the robot goes to the place before the movement for each article, holds it, carries it to the place after the movement, and installs it by releasing the holding. When all the article transport operations are completed, the work is completed.

Therefore, in the above system, the input operation of the load handling setting can be performed intuitively while looking at the load status of the load handled article displayed on the monitor, and thus the operation is simplified.
JP 7-237159 A

  The technique disclosed in Patent Document 1 for setting a cargo handling procedure in a virtual article diagram is also applicable to instructing a household robot to grasp and carry an article. For example, the present invention can be applied to a system in which a person who cannot carry an article due to a decrease in muscle strength, such as an elderly person, can grip and move an article indoors while looking at a monitor screen.

  However, unlike the industrial field in which the shape and arrangement of articles are arranged so that uniform processing can be performed in advance, the articles are not always arranged in a visible position in the home. Specifically, home goods are not only placed on the floor or table, but also in equipment (refrigerator, drawer, push-in, box, etc.). In addition, the article may be hidden behind the equipment. For this reason, it is difficult to apply the conventional technology as it is in a living environment such as a home, and there is a demand for a technology that can easily operate an article or the like in the facility using an interface unit.

  The present invention has been made in view of the above points, and the object of the present invention is to easily perform an operation via an interface unit even on an article hidden inside or behind a facility. Is to make it.

  An article management system according to the present invention is a system for managing articles existing in a living environment, comprising a database for managing information on articles in the living environment, and structure information of facilities and spaces in the living environment. An environment map to be managed, a display means for displaying the actual status of the living environment, an article operation means for performing an operation for designating at least an article on the display by the display means, information on the database and the environment map Based on the display data generating means for generating data for displaying information on the article inside or behind the equipment on the display means, and when the equipment is designated by the article operating means, the inside of the equipment Or the display control means which displays the information of the article hidden behind on the said display means is provided.

  A program according to the present invention includes a database for managing information on articles in a living environment, an environment map for managing structural information of facilities and spaces in the living environment, and display means for displaying the actual status of the living environment; A program for controlling an article management system comprising at least an article operation means for performing an operation for designating an article on display by the display means, wherein the database and the environment map information are stored in the article management system. And generating data for displaying information on an article hidden inside or behind the equipment on the display means, and when the equipment is designated by the article operating means, the equipment inside or behind the equipment. And displaying the information of the article hidden in the display means on the display means.

  The control method of the article management system according to the present invention includes the above steps.

  In the article management system, when a user instructs an equipment on a display (display image, etc.) on a display means using an article operation means (such as a mouse), the display means displays an article hidden inside or behind the equipment. Information is displayed. Therefore, the user can obtain information on not only articles existing outside the equipment but also articles hidden inside or behind the equipment while viewing the display on the display means. Therefore, an operation can be easily performed even on articles that are irregularly or randomly arranged. Therefore, it is possible to perform operations on various articles existing in the living environment.

  Another article management system according to the present invention is a system for managing articles existing in a living environment, and a database for managing information on articles in the living environment, and structures of facilities and spaces in the living environment. An environment map for managing information, a display means for displaying the actual status of the living environment, an article operation means for performing an operation for designating at least an article on the display by the display means, the database, and the environment map Based on the information, when the equipment is designated by the display data generating means for generating data for displaying information on the goods inside and behind the equipment on the display means, and the equipment operating means, the equipment Display control means for selectively and switchably displaying on the display means either information on an article hidden inside the equipment or information on an article hidden behind the equipment It is one that is equipped with a.

  Another program according to the present invention includes a database that manages information on articles in a living environment, an environment map that manages structural information of facilities and spaces in the living environment, and a display that displays the actual status of the living environment An article operating means for performing an operation for designating at least an article on the display by the display means, and an article operating means for performing an operation for designating at least the article on the display by the display means A program for controlling a management system, wherein data for displaying information on articles inside and behind the equipment on the display means is displayed on the display unit based on the information of the database and the environment map. When the equipment is specified by the generating step and the article operation means, information on the article hidden inside the equipment and hidden behind the equipment. Comprising the steps of one of the article information selectively and switchable displayed on said display means is one in which to run.

  Another control method according to the present invention is a method including the above steps.

  Also in the article management system, the user can obtain information not only on articles existing outside the equipment but also on articles inside or behind the equipment while viewing the display on the display means. Furthermore, in the article management system, either the information on the article hidden inside the equipment or the information on the article hidden behind the equipment is selectively and switchably displayed. Even if an article exists in any of these parts, information on the article can be displayed separately. Therefore, the user can easily operate the article.

  According to the present invention, the user can obtain information on an article hidden inside or behind the equipment by a simple operation of specifying the equipment on the display of the display means using the article operating means. Therefore, since not only articles existing outside the equipment but also information on articles hidden inside or behind the equipment can be easily obtained, even for articles arranged irregularly and randomly, The operation can be easily executed. Therefore, the operation of the article in the living space can be easily executed via the article operation means (interface means).

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, the overall configuration of the article management system according to the present embodiment will be described, and then the operation of the article management system will be described together with a specific example.

-Overall configuration of the article management system-
FIG. 1 is a block diagram illustrating an example of the overall configuration of an article management system 100 according to the present embodiment. The article management system 100 is roughly divided into four subsystems, that is, an environment management server 101 (hereinafter also referred to simply as a server), a robot 102 as an article moving means, an operation terminal 103, and equipment 104. Composed. Hereinafter, the configuration and operation of each subsystem will be described in the order of the environment management server 101, the facility 104, the operation terminal 103, and the robot 102.

-Environmental management server-
The environment management server 101, which is the first subsystem, includes a sensing means 105 for grasping the situation in the living environment (hereinafter simply referred to as the environment), and articles and moving objects (for example, existing in the environment based on the grasped situation) Article search / management means 106 for managing and searching humans, robots, etc.), an article database 107 for storing data therefor, an environment map management means 108 for managing information on the entire environment other than articles, and for the management Environment map 109 that stores the data, the transmission / reception means 110 that receives inquiries about data from the article database 107, the environment map 109, etc. from the outside, and sends information to the outside in response thereto, and the control that controls these means Means 111 are provided.

  In addition, the living environment in this embodiment means the environment where people and goods exist in mutual relations, such as a house and an office, for example.

  The map information includes structural information of an object (non-animal body) that usually does not move, such as a room or furniture. The structure information is the area information (for example, the surface of the floor, if it is a room, and the shelf if it is stored) that exists in and above the space occupied by at least the inanimate object on which other objects can be placed. Of the circumscribed polygon). The area information is represented by a display using a coordinate system and a shape.

  Hereinafter, each component of the environment management server 101 will be described in order.

<Sensing means>
The sensing means 105 always monitors the position and state of all monitoring objects existing in the operating environment (for example, houses, offices, stores, etc.), that is, articles, furniture, people existing in the environment, the robot 102, etc. It is. The sensing means 105 also detects when a new article is newly brought into the environment by a person, a robot 102, or the like.

  Although the specific configuration of the sensing unit 105 is not particularly limited, for example, a unit using an image sensor or a unit using an electronic tag can be suitably used. Here, as a specific example of a method for detecting an article or a moving body, a method using an image sensor and a method using an electronic tag will be described.

  First, a method using an image sensor will be described. Although the kind of image sensor used here is not specifically limited, A camera (image sensor) can be used suitably as what monitors efficiently a wide range like indoors with few facilities. That is, a camera may be fixed to a ceiling or a wall, and articles may be detected using the captured images.

  A background difference method is known as a general method for detecting an article using a camera image installed in an environment. The background difference method is a method of obtaining a target object from an image by preparing a model image as a background in advance and taking the difference between the current input image and the model image. The sensing means 105 according to this embodiment is intended to detect and monitor articles and moving objects in the environment. Therefore, as a model image, for example, when there is no environmental change, a single image in which no article or the like exists in the environment can be used. On the other hand, when the environmental fluctuation is severe, an image obtained by averaging images taken continuously at a certain time may be used.

  2A to 2C are auxiliary diagrams for specifically explaining the background subtraction method, and FIG. 2B is a diagram illustrating an example of the model image. FIG. 2A is a diagram showing an input image at a certain point of time taken using the same camera as the camera that took the image of FIG. 2B, and FIG. FIG. 3 is a diagram illustrating an example of a background difference image obtained by subtracting the model image of FIG. 2B from the input image of FIG. As can be seen from FIG. 2C, only the difference between the input image and the model image appears in the background difference image. As described above, in the background difference method, it is possible to detect an article in the environment by taking out only the portions that are raised in the background difference image. Note that the number of cameras used may be one, but if two or three or more cameras are used, the shape and orientation information of the article can be acquired using a three-dimensional measurement technique based on stereo vision.

  Next, a method for detecting an article using an electronic tag will be described. In recent years, a method for detecting the position of an article or a moving body using an electronic tag has been developed. In the present invention, almost all of these methods can be applied. For example, the technique disclosed in Japanese Patent Laid-Open No. 2000-357251 can be used.

  Here, the electronic tag is a device including an IC for storing data and an antenna capable of transmitting data wirelessly, and information can be written to the tag or read by a device called a reader / writer. .

  The tag can store attribute data that characterizes the article, that is, data such as the kind of article, date of manufacture, shape, weight, and article image. By storing such data in an electronic tag and making the data freely referable, more advanced article management is possible. Then, for example, it is possible to use the shape and weight when grasping or installing the article, use the date of manufacture for managing the quality deadline, use the kind of article for searching for items, etc. This will bring great benefits to the user. In addition, the tag itself may be configured to store only an industry standardized product code (similar to a barcode) and inquire the manufacturer for the attribute data of the article using communication means such as the Internet. Good.

  As described above, the detection method of the article and the moving body using the camera and the electronic tag has been described as a specific example of the sensing technology, but the sensing means 105 may of course use other methods. The sensing means 105 includes at least one of a camera, an electronic tag, and other sensors.

  When a new article or moving object is detected by the sensing means 105, the information is registered / updated in the article database 107 via the article search / management means 106 described later.

<Article database>
The article database 107 is a database for storing data such as what kind of articles are placed in which place. The details will be described below with reference to the drawings.

  FIGS. 3A and 3B are conceptual diagrams showing the data structure of the article database 107 and examples of description contents. 3A and 3B show the same configuration, and only the data contents thereof are different. The reason why two types of databases are shown in FIG. 3 is to explain how data contents are changed over time.

  In this example, each item data constituting the item database 107 has five attributes described below.

  1) Article ID: An ID for distinguishing individual articles. Even if the same type of article exists physically differently, it is necessary to treat them as different articles. For this reason, different IDs are assigned even to the same type of article.

  2) Article name: a name indicating the type of the article. Unlike the article ID, if the type is the same, the name is the same.

  3) Time: Time when the article was most recently operated.

  4) Place: The place where the article has moved when the article was most recently operated. The place is specified by the ID number of environment attribute data 601 or facility attribute data 602 registered in an environment map 109 described later (see FIG. 6). In addition, a coordinate value is set for an object whose spatial position is difficult or cannot be specified only by the ID number. For example, if the place is “refrigeration room” or “freezer room”, it is possible to specify that the article exists in the room alone, and therefore no coordinate value is designated. On the other hand, when the designated place covers a wide range such as “floor” and the specific position of the article cannot be specified only by the place name, a coordinate value for specifying the position is given. It is preferable that the initial setting of the presence location value of the article, the update when moved, and the provision of the coordinate value as additional information are automatically performed by the sensing means 105. However, it goes without saying that it may be performed manually. Whether or not to give a coordinate value when indicating the location of the article may be determined by the performance of the robot 102 that holds and conveys the article. If the performance of the robot 102 is very low, for example, if an accurate coordinate is required even when an article in the refrigerator compartment is gripped, the location (coordinate) of the article in the refrigerator compartment may be given.

  5) Article image: An image of the article.

  Here, an example is shown in which five attributes are provided to distinguish the characteristics of each article, but other attributes may be provided as necessary. For example, the three-dimensional shape of the article, accurate position / posture data (not shown) of the article, and the like may be assigned as attributes. By giving such an attribute, the robot 102 can more easily grasp the article. Also, the time / location / article image can be recorded for each time and stored as a history. Thereby, it becomes possible to instruct the position of the article at a certain point in the past and to display an image.

  Next, how the contents of the data are changed as time passes will be described while comparing the contents of the article database 107 with the actual situation.

  FIGS. 4A and 4B are schematic diagrams obtained by photographing a certain environment at two different times. Here, it is assumed that FIGS. 4A and 4B correspond to FIGS. 3A and 3B, respectively. That is, it is assumed that the database storing the article data existing in the environment at each time coincides with the databases of FIGS. 3 (a) and 3 (b).

FIG. 3A shows the contents of the database at 9:00 on October 10, 2002 as an example. This database contains
Seven items of can juice, lunch box, notebook, banana, paper waste, ice cream and popsicle are registered. this house,
As can be seen from the example of FIG. 4 (a), the five articles of can juice, lunch box, notebook, banana, and waste paper are scattered on the floor. Therefore, as shown in FIG. 3A, as the value of the location of each article in the database, not only “floor” (floor # 0001) but also the coordinate position on each floor is given as additional information.

Meanwhile, the remaining items below,
Both ice cream and popsicles are stored in a freezer (not explicitly shown in FIG. 4A), and the location can be limited to some extent. Only “chamber” (freezer # 0001).

Next, it is assumed that the following environmental change occurs at 10 o'clock on October 10, 2002. That is,
1) The user instructed to clean up the articles on the floor (the respective clean-up destinations are as indicated by the curves with arrows in FIG. 4A), and in response to the instruction, the robot 102 cleaned up the articles.

  2) The user ate ice cream and popsicles in the freezer compartment.

  FIG. 3B shows the state of the database at 20:00 on October 10, 2002, after a while after such environmental changes.

As shown in FIG. 4B, the following articles scattered on the floor at the time of FIG.
The five items of can juice, lunch box, notebook, banana, and paper scraps were cleared according to the above instructions, and the value of the database location was rewritten accordingly (see FIG. 3B). In addition, the following items disappeared by eating,
It can be seen that ice cream and popsicles are deleted from the database (see FIG. 3B). This deletion process may be performed manually or automatically using a sensor such as a tag. In addition, since food is virtually disappeared from the real world when someone eats it, the place data may be written instead of describing the place.

  As described above, when the article moves or disappears from the environment, the data content of the article database 107 is sequentially updated, and the article database 107 always keeps the latest information. The contents of the article database 107 have been described above.

<Article search / management means>
The article search / management means 106 accumulates information on articles obtained by manual input by the sensing means 105 and the user for any article placed in the environment in the article database 107, or from outside the environment management server 101. When there is an inquiry about an article or the like via the transmission / reception means 110 and the control means 111, processing is performed in which necessary information is extracted from the article database 107 according to the content of the inquiry and sent to the inquiry destination. Note that access to the article database 107 is permitted only to the article search / management means 106, and the same article data is managed so that writing and reading are not performed simultaneously. When receiving an article information registration / update request from the robot 102 and the operation terminal 103, the article database 107 is registered / updated via the article search / management means 106. It is also possible to search for the location of the article being searched for by specifying a search key for narrowing down the attributes of the article to be known, such as date and article type, using the operation terminal 103 described later.

<Environmental map management means>
The environment map management means 108 manages maps in the environment. FIG. 5 is a conceptual diagram showing an example of the environment map 109 in comparison with the actual situation. FIG. 5A shows the actual situation, and FIG. 5B shows the actual situation as the environment map 109. FIG. 5C is a diagram showing a simplified three-dimensional model, and FIG. 5C is a diagram showing the actual situation further simplified by a plane model. The environment map 109 may be represented as three-dimensional data in this way, or may be planar data more easily. Data can be created according to the purpose of the map and the time required to create the map. That's fine. For example, the model in FIG. 5B is such an example. In FIG.5 (b), the table located in the center in Fig.5 (a) is modeled by the rectangular parallelepiped. The same applies to the plane data. In the model of FIG. 5C, the table located at the center is represented by a rectangular area (rectangular area of the hatched portion in FIG. 5C) that is orthogonally projected onto the plane. It is defined as a non-movable area. For convenience of the following description, a coordinate system constituted by the X axis, the Y axis, and the Z axis shown in FIGS. 5A to 5C is referred to as a world coordinate system.

<Environment map>
FIG. 6 is a diagram showing an example of the data of the environment map 109. The environment map 109 is roughly composed of two elements: environment attribute data 601 and equipment attribute data 602.

In other words, the environment attribute data 601 is detailed room data. In this example, two floor surface data (floor # 0001, floor # 0002) are recorded in the environment attribute data 601 (floor # 0001). 0002 data not shown). In the floor surface data, the coordinates of the vertex of the surface (the coordinate position in real world coordinates) are described, and the material of the floor surface is given for each surface. For example, in the case of square floor data, as shown in FIG.
((X1, Y1, Z1),
(X2, Y2, Z2),
(X3, Y3, Z3),
(X4, Y4, Z4), 0)
It is expressed. However, the lowest floor height in the room is set as 0 as a reference of coordinates. The first four coordinates indicate vertex coordinates, and the last numerical value “0” indicates the floor material. The material of the floor is, for example, “0” for flooring, “1” for tatami mat, “2” for carpet, etc., and corresponding numbers are predetermined for each material. When there are a plurality of floor surfaces having different heights in the room, these floor surface data may be prepared for the number of the floor surfaces.

  The facility attribute data 602 lists the facilities 104 existing in the environment (specifically, a room) configured by the environment attribute data 601. The equipment 104 here is household goods that are not moved and used in a normal state, and for example, furniture, large household appliances, and the like.

  In the example shown in FIG. 6, there are a table, a freezer compartment, a refrigerator compartment, a trash can, and the like in the room as the environment. Those attributes are stored. Originally, the freezer compartment and the refrigerator compartment are integrally called a refrigerator. However, in this example, the facilities are distinguished by the unit where the articles are stored, so the refrigerator is not handled as one facility. And the refrigerator compartment are distinguished as one independent facility.

In the equipment attribute data 602, as the attributes of each equipment, the data of a plurality of surfaces when the surface of the equipment is approximated by a polyhedron, the type of the equipment, the main article shape installed on the surface where the equipment can be installed And posture etc. are preserved. The equipment surface data describes the coordinates of the vertex of the surface (coordinate position in real world coordinates), and a flag indicating whether or not an article can be placed on the surface is attached to each surface. Yes. For example, if the number of vertices is 4 face data,
((X1, Y1, Z1),
(X2, Y2, Z2),
(X3, Y3, Z3),
(X4, Y4, Z4), 1)
It becomes. The first three coordinates indicate the coordinates of the vertex, and the last numerical value “1” means that the article can be placed. The surface having this numerical value “0” is a surface on which an article cannot be placed. Depending on the type of equipment, this flag can be switched according to the situation. Examples of such situations include whether the door is open or closed. FIG. 7 is an auxiliary diagram for showing such a typical example.

  In the example of FIG. 7, the attribute data of the equipment regarding the freezer compartment is shown. 7A shows attribute data when the freezer compartment door is closed, and FIG. 7B shows attribute data when the freezer compartment door is open. From these figures, it can be seen that the last value of the flag is changed according to the opening and closing of the freezer compartment. That is, when the door of the freezer compartment is closed, since the article cannot be stored inside as it is, the flag is “0”. On the other hand, when the door of the freezer compartment is open, the flag can be set to “1” because the article can be stored inside as it is. In the freezer compartment in FIG. 7B, the surface on which the article is installed has a mechanism that protrudes to the front when the door is opened so that the article can be taken in and out using the robot 102. For this reason, the coordinate values of the four corners of the surface in the protruding state are given to the coordinate values of the surface on which the article is placed. Then, the robot 102 takes in and out the article only when the surface is protruding (that is, when the door is open). It is also possible to perform an operation for placing an article on the surface and an operation for taking out an article placed on the surface with reference to the coordinate values of the surface. When the door is closed, the surface (article installation surface) is stored in the freezer compartment. Along with this, the actual coordinate value of the surface changes. However, when the door is closed, the robot 102 does not put in or take out articles from the freezer compartment, so the coordinate values described as the equipment attribute data 602 are left unchanged.

  In this example, only the identification flag indicating whether or not an item can be installed is shown as the equipment attribute data 602, but other information may be added as needed. For example, the surface material may be added in the same manner as the environment attribute data 601. Further, a locus of approach of the robot hand to the surface may be added to install an article on the surface or take out an object from the surface. Furthermore, a program for moving the robot hand can be stored and used. For example, when a standard program specification for moving the robot arm is determined in advance and the robot 102 capable of arm control according to the specification is used, the program stored as a part of the equipment attribute data 602 is stored in the robot. The robot 102 may be downloaded as appropriate, and the robot 102 may be moved by the downloaded program. As a result, the robot 102 does not have to prepare individual grip control programs for all the facilities, and the memory capacity for storing the programs can be reduced.

<Control means of environmental management server>
The control means 111 of the environment management server 101 is a part that controls the entire server 101. As described above, the main control contents include the following.

  That is, when there is an inquiry from the outside via the transmission / reception means 110 regarding the various data in the server 101, the control means 111, with respect to the article search / management means 106 or the environment map management means 108, depending on the content, A data reference request is issued.

  In addition, the control unit 111 sends a result sent from the article search / management unit 106 or the environment map management unit 108 to the inquiry source via the transmission / reception unit 110 in response to the request.

  Further, the control unit 111 interprets a request for registration / update regarding various data in the server 101 sent from the outside via the transmission / reception unit 110, and according to the contents, the article search / management unit 106 or the environment map A request to register / update the data is sent to the management means 108.

-Equipment-
The facility 104, which is the second subsystem of the article management system 100 of the present embodiment, is a facility having a place for storing articles with a certain purpose. Here, the phrase “having a purpose” means, for example, “store” for a refrigerator, “warm” for a microwave oven, and the like. The word “storage” is generally used as a meaning of storage or the like, but “storage” in the present invention includes temporarily placing an article in a place for the purpose. Therefore, for example, putting food in a refrigerator or microwave oven is also called storage.

  As shown in FIG. 1, the facility 104 has, as its basic configuration, facility operation information storage means 122 for operating the facility 104 in response to an operation instruction from the outside, and attribute data of articles in the facility. Sensing means 123 for obtaining, transmission / reception means 110 for receiving the operation instruction from the outside or transmitting the result of the operation to the instruction source, and control means 121 for controlling the means 110, 122, 123.

<Equipment sensing method>
Since the sensing means 123 is the same as the sensing means 105 of the environment management server 101, detailed description thereof is omitted here. However, it is assumed that sensed information and facility structure information are stored in the article database 107 and the environment map 109 of the environment management server 101 via the network. Further, by configuring the sensing means 123 with a camera, an intuitive GUI (Graphical User Interface) using an actual image in the facility can be realized.

<Facility operation information storage means>
The facility operation information storage means 122 mainly stores a command (facility operation command) for remotely operating the facility 104 from the outside. FIG. 8 is a diagram showing facility operation commands stored in the facility operation information storage means 122 in a table format. The description information of the table includes, in order from the left column, a facility ID for distinguishing facilities existing in the environment, a facility operation command name for controlling the facility from the outside, a processing procedure corresponding to the command, and the processing This is the return value of the result of performing.

  Here, as an example of equipment, there are three types: Refrigerator # 0001 (refrigerator room), Freezer # 0001 (freezer room), and Microwave # oven # 0001 (microwave oven) that are distinguished by equipment ID. Directive commands are shown. Next, the meaning of the description information will be described using the example.

In the first two examples, the refrigerator compartment and freezer compartment,
・ Door # open
・ Door # close
Two are prepared. When the refrigeration room and the refrigeration room receive these commands from the outside, the facility itself performs the processes of “opening the door” and “closing the door” as described in the processing procedure. For this reason, the doors of the refrigerator compartment and the freezer compartment are automatically opened and closed. Next, when processing of each equipment operation command ends normally, Ack is returned to the command transmission source, and when processing of the equipment operation command fails, Nack is returned to the command transmission source as a return value. If the equipment is a refrigerator or freezer, it is desirable to be able to view the interior without opening the door. Prepare a door that can be switched between transparent and non-transparent, door # transparent # on, door # transparent # off Two commands may be prepared. The door that can be switched between transmission and non-transmission can be realized, for example, by attaching a liquid crystal shutter or a blind to a transparent door.

In the third example, the microwave oven, as an equipment operation command,
・ Door # open
・ Door # close
・ Warm # start
・ Warm # end
・ Is # object # in
There are five types. Of these, door # open and door # close are the same as those in the refrigerator compartment and the freezer compartment, and thus description thereof is omitted.

  When the equipment operation command warm # start is received, warming starts as described in the processing procedure. At this time, if the article is in the range and the warming process can be started, Ack is returned to the command source, and Nack is returned to the command source in other cases as a return value. When the equipment operation command warm # end is received, it is checked whether the warming is completed. If warming is complete, True is returned, and if warming is still in progress, False is returned to the command source. When the equipment operation command is # object # in is received, it is checked whether or not there is an article in the microwave oven. If there is an article, True is returned, and False is returned to the command source as a return value. At this time, the presence or absence of the article may be confirmed using an image sensor, a weight sensor, or an electronic tag sensor if an electronic tag is attached to the article.

  As described above, three examples of the facility 104 are given and the example of the facility operation command is briefly described. However, the necessary facility operation command may be prepared according to the function of each facility 104. Also, when a new equipment operation command is prepared by the manufacturer of the equipment 104, it is written in the equipment 104 using some storage medium or the like, or if the equipment 104 is connected to the manufacturer via an external network, the above Equipment operation commands may be sent to the equipment 104 via a network.

-Operation terminal-
The operation terminal 103, which is the third subsystem of the article management system 100, is a terminal for the user to instruct article operations in the environment.

  As shown in FIG. 1, the operation terminal 103 has, as its basic configuration, an article operation means 118 for giving an article movement instruction by an operation for designating an article, an operation for moving the article to a predetermined location, and the like, and a display for displaying an operation screen. Environment management of means 119, in-facility display means 124 for generating and synthesizing a display form capable of instructing the article and location in the facility on the operation screen, and the article operation instruction content input by the article operation means 118 It comprises transmission / reception means 110 for sending to the server 101, and control means 120 for controlling these means.

<Article operation means>
In the article operation means 118, the user gives an instruction to operate the article while referring to the image displayed on the display means 119. For example, a real-time video obtained by photographing the actual situation with a camera or the like is displayed on the display means 119 as it is, and the user operates the article while watching the video. For this article operation, for example, a pointing device (mouse, touch panel, line-of-sight input) or cursor movement using a keyboard is used.

  4 (a) and 4 (b) are used to explain the contents of the article database 107 described above. On the other hand, this figure also corresponds to a real-time image obtained by photographing a real situation with a camera or the like. . 4A and 4B show examples of operation screens before and after the article operation, respectively.

Here, it is assumed that the following operation has been performed as an example of an operation for designating an article and an operation for moving the designated article to a predetermined location using the operation screens of FIGS. That is,
1) Move the mouse cursor to the item you want to operate on the screen.

  2) Confirm the article by dragging, move the cursor to the place you want to move, and drop it.

  The above operation is a very simple operation consisting of two steps. A curve with an arrow in FIG. 4A is an example of the movement locus of the mouse cursor during drag and drop. For example, when the notebook S # 0001 is to be placed on the table, the mouse cursor is first moved to the notebook S # 0001 and dragged, the cursor is moved onto the table in that state, and the mouse is dropped.

By the way, in order to convert such an operation into instruction data to be described later, it is not sufficient to simply display the real-time video of the camera on the screen as it is, and the following processes 1) and 2) are required. That means
1) Recognition processing of what is the item pointed to by the mouse on the screen and where it is dropped (operation outside the facility)
2) A place for indicating an article in an area that cannot be seen in the camera image, for example, an article hidden inside or behind the facility 104, or when the designated article is moved to an area that cannot be seen in the camera picture Processing (operation inside the equipment)
Is required. Next, examples of these two points will be described.

<Operation outside equipment>
First, a description will be given of how to designate articles and places other than the inside of the facility (such as in the refrigerator compartment). For example, in the example of moving the notebook just described, the user moves the mouse cursor to the location of the notebook S # 0001 on the screen and clicks the notebook S # 0001. At this time, in order for the system to understand that the article to be moved is the notebook S # 0001, the screen position data of the notebook S # 0001 corresponding one-to-one with the display screen is required. The simplest expression method of the position data is a method using article mask data corresponding one-to-one with the display screen. Here, “corresponding to one-to-one” means that coordinate values correspond.

  When the position of an article on the display screen is designated as a coordinate value, the same coordinate value is then referred to in the article mask data. In the article mask data, a pointer to the article data is described as an attribute of the coordinates where the article exists. By tracing the pointer and referring to the article data, it is possible to know what the article on the display screen is.

  The problem here is how to create the article mask data. If the camera image for detecting the article is used as it is on the display screen for article manipulation using the background difference method, the result of the background difference method may be used. This is because the position in the screen of the article is detected as a result of the background difference, and can be used as it is as mask data.

On the other hand, the process of recognizing the moving location is performed using, for example, a screen map in which the video and coordinates of the operation screen correspond in advance. FIG. 9 is a conceptual diagram showing an example of an operation screen and a screen map corresponding to the operation screen. FIG. 9A shows the operation screen, and FIG. 9B shows the screen map. As can be seen in this example, the screen map is data that distinguishes where the entity corresponding to the environment attribute data 601 and the equipment attribute data 602 specified in the environment map 109 is present on the operation screen. is there. The screen map can be stored as image data, for example. For example, the screen map is image data in which each pixel is represented by an integer value, and a pair of the value and location is
0: Nothing 1: Floor (floor # 0001)
2: Table (table # 0001)
3: Cold room (refrigerator # 0001)
4: Freezer room (freezer # 0001)
5: General trash (wastebasket # 0001)
6: Recycle trash (wastebasket # 0002)
It may be determined in advance as such. In order to check what is present at a suitable place on the operation screen, it is determined by examining the value of the pixel at the same place in the corresponding environment map 109. However, it goes without saying that this determination process is performed after first checking whether or not an article exists and confirming that there is no article at that location.

  Although the example given here includes a refrigerator compartment and a freezer compartment having an internal structure, an operation process inside the facility described later is required to indicate a more specific article or place inside the compartment. . Of course, when the place for storing the article may be any position in the refrigerator compartment or the freezer compartment, the location indication method described here is sufficient.

  The equipment 104 to be registered in the environment map 109 hardly moves as a general rule once installed. Therefore, if it is difficult to automate the creation of the environment map 109, it may be created manually. Of course, if the three-dimensional model of each facility 104 is known and the installation location can be calculated accurately, the creation of the environment map 109 can be automated.

<Operation inside equipment>
If the item or location you want to indicate on the screen is inside the facility 104 or hidden behind the facility 104, the item or location will not appear explicitly in the camera image, so you cannot specify it as it is. . Next, an example of a simple method for instructing an article and a place such as the inside of the facility 104 will be described.

  First, regardless of whether or not the article to be operated is dragged, whether or not the location specified by the mouse cursor is a facility can be known using the screen map (see FIG. 9B) as described above. .

  If the location designated by the mouse cursor is the facility 104, the internal structure of the facility 104 and the articles inside the facility 104 are displayed in some form so that the location and the article inside the facility can be indicated. The structure and articles inside the facility can be known by referring to the environment map 109 and the article database 107. As described above, in the article database 107, for example, when an article is stored in the facility 104, information on the article storage location in the article database 107 is updated automatically or manually each time. Accordingly, it is possible to easily grasp what items are stored in each facility 104 by simply referring to the item database 107. As the internal structure of the facility 104 and the display form of articles, for example, the following can be considered.

(1) First Display Form In the first display form, as shown in FIG. 10A, each article image inside the facility is displayed for a certain period of time, for example, in a pop-up display. Designation of a specific article may be performed by moving the mouse cursor to the display position of the specific article while each article is displayed for a certain period of time, or of course other methods may be used. FIG. 10A shows an example in which, when the mouse cursor is moved to the refrigerating room and the freezing room, an image of an article stored therein is displayed in a pop-up. This display example is suitable when an article in the facility is moved to another location. Of course, you may use a character instead of the said image. In this display mode, when an article is moved into the facility, it is left to the system to place the item in the facility.

(2) Second Display Form In the second display form, hierarchical display is performed for each type as shown in FIG. If the facility is a refrigerated room, first the food category such as meat and vegetables is displayed, and then the food corresponding to the hierarchically selected category is displayed. This display form is suitable for searching for a desired thing without making the operator aware of the structure in the facility.

  Here, some facilities have a plurality of storage locations (such as shelves), but they may be displayed hierarchically for each storage location. When displaying an image of an article as in the first display mode, it is assumed that a complicated screen is displayed when a large number of articles are in the facility. Therefore, it is effective to display hierarchically in this way.

  In addition, when the equipment is a refrigerator, ten pieces of food (articles) may be displayed in order from the earliest expiry date, or may be displayed with different colors and character sizes. It is also possible to devise the display so that it is easy to see, such as displaying from the one with the largest number of withdrawals. Displaying according to the number of withdrawals, for example, in the case of foods, the greater the number of withdrawals, the more severe the quality degradation. It also means that it is displayed with higher priority. Also, the operator's information can be input, and the displayed items may be filtered according to whether or not the operator's belongings are belonging to the user.

(3) Third display mode As shown in FIG. 10 (c), when a camera is installed in the facility to sense an article (for example, using the sensing means 123), an image in the facility is displayed, The article or location on the image in the facility is designated with the mouse cursor by the same technique as that for designating the article or location on the table. At this time, if the video in the equipment to be displayed is enlarged and displayed, the operability is improved. Moreover, since the inside of the facility is dark when the door is closed and the camera cannot be photographed, it is desirable to install illumination in the facility and turn on the illumination when photographing. Furthermore, by displaying the captured video of the equipment on the equipment door on the screen, it is possible to intuitively instruct and operate in the same way as usual.

(4) Fourth display mode In the fourth display mode, as shown in FIG. 10 (d), when an instruction operation such as hovering the mouse cursor over the equipment is performed, the equipment door is automatically opened to directly The inside of equipment is displayed on the display screen. As described above, an operation instruction command is prepared for each facility, and such processing can be realized by using this operation instruction command.

  Specifically, when a desired operation (for example, refrigeration room) is instructed with the mouse cursor, a door # open command is sent to Refrigerator # 0001 (refrigeration room), the door of the refrigeration room opens, and the display screen displays the inside of the refrigeration room Is projected directly. Then, when the instruction of the article or place is finished, a door # close command is transmitted. In addition, since this display form opens and closes an actual door, equipment that does not control temperature and humidity, such as a closet and a shoe box, is more suitable than a refrigerator or the like. As shown in Fig. 10 (e), in the case of facilities that manage temperature, humidity, etc., such as refrigerator compartments and freezer compartments, leave some or all of the walls as walls that can be switched between transmission and non-transmission. The wall surface may be transmitted / non-transmitted at the same timing as the door opening / closing described above. By installing such doors in the refrigerator compartment or freezer compartment and preparing door # transparent # on and door # transparent # off commands, the above display mode can be realized using such commands.

  It is also effective to switch these display forms according to the situation or the type of equipment. For example, the dragged article is displayed in the first display mode when it is placed in the facility, while it is displayed in one of the second to fourth display modes when the article in the facility is indicated. In order to realize such switching processing, a display form table as shown in FIG. 11 is prepared, and the in-facility display means 124 switches the display form according to the user's operation and equipment based on this table. The number of the display form in FIG. 11 corresponds to the notation in FIG. In addition to this, the display form may be changed according to the person operating, according to the equipment to be operated, or according to the article to be operated. That is, the display form may be appropriately selected according to various attributes.

  Next, a display form in the case where an article or place to be designated on the screen is hidden by the equipment 104 will be described. The area concealed in the facility can be designated with a mouse or the like as in the above-described interior of the facility. However, in this case, after that, it is necessary to indicate the position in the depth direction from the camera line of sight, such as whether it is inside the equipment, behind it, or inside the equipment behind it. As one method for realizing this, regarding the equipment in the forefront, other equipment hidden behind it, the floor of the room, etc. are regarded as a part of its own structure. The fourth display form can be applied as it is. FIG. 12 shows an example in which this state is realized in the pop-up list form of the first display form. When the mouse cursor is placed on (or clicked on) the cupboard, the cupboard and the contents of the refrigerator and floor over the pop-up pop up, and the desired item can be indicated.

  Here, a method is needed to know how much equipment is hidden behind the indicated point. As described above, there is a method using a screen map (see FIG. 9B) as a method for checking whether or not the designated point is equipment. Here, this is expanded, a sufficient number of screen maps (hierarchical screen maps) are prepared, and all the facilities are arranged without overlapping each screen map. Then, a plurality of facilities including the point designated by the mouse and a floor or wall concealed in any of the plurality of facilities are displayed as the contents of the pop-up list. FIG. 13 is a hierarchical screen map corresponding to the environment of FIG. 13 (a) is the screen map at the back, FIG. 13 (b) is the screen map from the back, FIG. 13 (c) is the screen map at the front, and FIG. 13 (d) is the screen at the front. It is a screen map. As can be seen from FIG. 13, the facilities do not overlap in each screen map.

  Further, the display in the equipment behind the point designated by the mouse is changed in order from the front or back every time another switch-like input means (for example, the wheel of the wheel mouse or the up / down arrow keys of the keyboard) is input once. It is also possible. For example, in the example of FIG. 12, when the mouse cursor is moved to the cupboard, the contents in the cupboard are displayed in a pop-up, and when the “up” arrow key is pressed once, the contents in the refrigerator beyond that one are displayed in a pop-up. Then, pressing the “up” arrow key again will pop up the contents of the item placed on the floor, etc. Then, by pressing the “up” arrow key again, the contents of the cupboard can be restored. Conversely, the contents can be popped up in the reverse order using the “down” arrow key.

<Control means for operation terminal>
The control unit 120 receives the above operation instruction from the article operation unit 118, generates instruction data, and transmits the instruction data to the robot 102. The instruction data is data on which the action plan of the robot 102 is created by the action plan creation means 117 of the robot 102. This instruction data is
(Articles to be operated, destination)
Have two sets of values. For example, when moving a notebook to the table,
(Notebook S # 0001, table)
Is the instruction data. Only the location registered in the environment attribute data 601 or the facility attribute data 602 can be specified as the destination. In addition, because the destination has a certain wide range, if the location can not be specified only by the destination name, add a specific coordinate value to the destination in the world coordinate system (the coordinate system is based on the environment, The coordinates may be specified in the coordinate system shown in FIG. For example, when placing an article on a predetermined place on the floor,
(Goods, floor (x1, y1,0))
And so on. The coordinates in the world coordinate system of the location indicated on the display screen are provided with, for example, the three-dimensional model environment map 109 shown in FIG. 5B as the environment map 109, and the camera parameters (cameras) displaying the three-dimensional model and the display screen. Position, orientation, angle of view, etc.). Since such a calculation method is a basic known technique of computer graphics, its description is omitted. If the three-dimensional model and the camera parameters are known, the above-described environment map 109 can be obtained by calculation.

-Robot-
The robot 102, which is the fourth subsystem, plays a role of actually grasping and carrying an article in the environment in the article management system 100 according to the present embodiment.

  As shown in FIG. 1, the robot 102 has, as its basic configuration, a sensor 112 that detects obstacles in the vicinity of the robot 102 and obtains information on an article to be grasped, a grasping means 113 that grasps an article, an environment, and the like. A movement plan creation means 114 for making a movement plan using the map 109; an action plan creation means 117 for drafting an action plan for the robot 102 in accordance with the content of the instruction to execute an instruction from the user; It includes a moving means 115, a transmitting / receiving means 110 for transmitting / receiving various data to / from the environment management server 101 and the operation terminal 103, and a control means 116 for controlling these means.

  When gripping and transporting an article by the robot 102, a user issues an instruction via the operation terminal 103, and instruction data in which the instruction content is encoded is transmitted to the robot 102 via the network. When the robot 102 receives the instruction data, the action plan creation means 117 generates a list of robot control commands for the robot 102 to act from the instruction data, and the robot 102 processes the robot control commands in order. Execute the gripping and transporting operation of the article.

The robot control command here is a command for gripping and moving the robot 102, and further controlling the equipment 104.
・ Movement ・ Grip ・ Release ・ Four types of equipment operation. These will be described below.

  “Move” is represented by (move, coordinates) or (move, equipment ID). Upon receiving this command, the robot 102 moves from the current position to the position designated by the coordinates or the equipment 104 designated by the equipment ID. Here, the coordinates are specified in the world coordinate system, and the movement plan creation means 114 drafts the movement route plan. When moving to the facility 104, a route that approaches the facility 104 to a predetermined distance is created. The location coordinates of the facility 104 can be obtained by referring to the facility attribute data 602 in the environment map 109.

  “Grip” is represented by (grab, article ID). Upon receiving this command, the robot 102 holds the article specified by the article ID. The location of the article is grasped by referring to the article database 107 described above, and a grasping plan by the grasping means 113 is executed.

  “Release” is represented by (release). Upon receiving this command, the robot 102 releases the hand 202 constituting the gripping means 113.

  “Equipment operation” is represented by (own ID, equipment ID, equipment operation command). Upon receiving this command, the robot 102 sends the designated equipment operation command to the equipment 104 designated by the equipment ID. The facility operation command is an operation instruction command received by each facility 104 from the outside, and each facility performs processing corresponding to the command when receiving the operation instruction command. Here, the reason for attaching its own ID in the command is that when a facility operation command is sent to a certain facility 104, the received facility 104 performs processing according to the facility operation command, and then the processing result is received from the facility 104. This is to have you reply yourself.

  As described above, four types of robot control commands have been described as typical examples. However, it goes without saying that the robot control commands are not limited to these four types, and may be increased or decreased as necessary.

  FIG. 14 is a schematic diagram showing an example of the robot 102. Hereinafter, each unit of the robot 102 will be described with the direction in which the tip of the arm is facing in FIG.

<Movement means>
Two moving means 115 are provided on one side of the robot body, and are constituted by a total of four wheels on both sides. In this example, an example of a wheel is shown as the moving unit 115, but an optimal unit may be selected as the moving unit 115 according to the place and environment where the robot 102 is used. For example, when moving on uneven ground, a crawler type or multi-legged moving means may be used. In addition, when the gripping means 113 including the arm 201 and the hand 202 makes the entire area of the house movable, the moving means 115 is not always necessary.

<Sensor>
The sensor 112 detects obstacles in the vicinity of the robot, and in this example, includes an ultrasonic sensor 112a, a stereo camera 112b as a visual sensor, and a collision sensor 112c. The ultrasonic sensor 112a is attached to each of three locations on the front, back, and left and right sides, and by measuring the time it takes to emit the ultrasonic wave and receive the reflected wave, the approximate distance from the sensor to the obstacle Is calculated. In this embodiment, the ultrasonic sensor 112a detects an obstacle at a short distance before the collision. The stereo camera 112b inputs the surrounding situation as an image, and performs processing such as recognition on the image, thereby determining the presence / absence of an obstacle and obtaining more accurate information on the object to be grasped. The collision sensor 112c is a sensor that detects that the sensor 112c has an impact of a predetermined force, and it has hit an obstacle that cannot be detected by other sensors, or the robot 102 itself moves. The collision sensor 112c detects that the collision has occurred.

<Movement plan creation means>
When the movement plan creation means 114 receives the robot control command for moving the robot 102 to the designated location, the movement plan creation means 114 uses the environment map 109 acquired from the environment management server 101 for the movement route from the current position to the designated location. Create. Naturally, if there is an obstacle between the current position and the destination, a route to avoid it is necessary, but the environment map 109 has an area where the robot 102 can move as described above. Is written in advance, and a movement route within the area may be created. If the sensor detects an obstacle after creating a movement route and starting moving, a new route for avoiding the obstacle is recreated each time. In creating the movement route, the Dijkstra method, which is the most common method, is used.

<Gripping means>
The gripping means 113 is a means for gripping an article. In the present embodiment, the gripping means 113 includes an arm 201 having a multi-joint and a hand 202 disposed at the tip of the arm 201 as shown in FIG. When a gripping position is instructed by a robot control command, which will be described later, the gripping means 113 moves the tip of the arm 201 to that position and performs a gripping operation of the hand 202. The arm control for moving the hand 202 to the gripping position may be performed by the gripping means 113. The gripping means 113 also performs the release operation of the hand 202 when instructed to release by the robot control command.

<Robot control means>
The control means 116 interprets a list of robot control commands sent from the outside via the transmission / reception means 110, and executes the robot control commands in order. If the robot control command that has been sent is the aforementioned instruction data, the instruction data is sent to the action plan creation means 117 to convert the instruction data into an executable robot control command for the robot 102, and processed there. The robot control commands are executed in order.

<Action plan creation means>
The action plan creation means 117 can give a work instruction to the robot 102 simply by performing a simple operation in which the user designates an article and moves the designated article to a predetermined location by the article manipulation means 118 in the operation terminal 103. It is means provided for doing so. Specifically, when the robot 102 receives instruction data from the operation terminal 103 via the network, the action plan creation unit 117 refers to the robot control command DB (database) as necessary, and stores the instruction data. Based on this, the robot 102 generates a list of robot control commands for executing a series of operations.

  Here, in the robot control command DB, a list of robot control commands when the robot 102 performs equipment operations is stored in advance for each equipment. The reason is as follows.

  As described above, the instruction data includes only two pieces of data, that is, two pieces of data including (article to be operated, destination of movement). The robot control command DB is not particularly required if the target article is in a state where it can be gripped as it is immediately before the robot 102, or the moving place is a spatially open place. However, in reality, the target article is rarely in front of the robot 102, and the robot 102 (or the gripping means 113) usually has to move to the vicinity of the article to be operated. In addition, when the article exists inside the facility closed by the door, the door must be opened to grasp the article, and then the door must be closed. In addition, some equipment 104 may require more complicated processing after the article is stored. However, it is difficult to say that the system is easy to use if the user instructs these processes one by one using the operation screen. If possible, it is desirable that the robot 102 can be instructed with only a simple instruction operation for generating the instruction data as described above, that is, only two operations of selecting an article and specifying a moving place. .

  Therefore, knowledge data for creating a robot control command for causing the robot 102 to perform a predetermined work including the operation of the facility 104 from the instruction data generated by a simple instruction operation is required. The robot control command DB stores the knowledge data.

FIG. 15 is a table format showing an example of a list of robot control commands stored in the robot control command DB. This figure includes two tables with different facilities. In the leftmost column of each table, the ID of the facility 104 operated by the robot 102 is described. The “location attribute” in the right column indicates the source or destination,
・ Movement source: When the article is stored in a certain facility 104 and wants to take out the article. ・ Destination: Store the item in a certain facility 104, and further use various functions of the facility 104 as necessary. This means that it is necessary to perform some processing on the stored article. The rightmost column shows a robot control command list corresponding to the location attribute.

As an example, a robot control command list when the equipment ID is Refrigerator # 0001 (refrigerator room) and the location attribute is the movement source will be described. This is a list of robot control commands when the article database 107 is referred to for the article included in the first value of the instruction data and is stored in the Refrigerator # 0001 (refrigerator room). The three commands here are
・ Open the door of Refrigerator # 0001 (refrigerator room)
・ Grip and take out articles
・ Close the door of Refrigerator # 0001 (refrigerator room),
It corresponds to the operation. Where the second robot control command,
(grab, $ object)
$ Object means that the ID of the article to be operated is entered. Information whose value changes depending on the situation is treated as a variable by adding $ to the head, and when the article to be handled is specifically determined by the instruction data, the value is set in the variable. In this way, the robot control command can be generalized.

Here, for the purpose of understanding the contents of the robot control command, a very simple example has been described. However, for practical use, a further necessary robot control command may be added. For example, the command list of the movement destination of the microwave oven is
(Robot # 0001, Microwave # oven # 0001, door # open)
(release, $ object)
(Robot # 0001, Microwave # oven # 0001, door # close)
(Robot # 0001, Microwave # oven # 0001, warm # start)
However, since there are actually no items in the microwave oven, there is an article in the microwave oven before these four commands. Command to check if
(Robot # 0001, Microwave # oven # 0001, is # object # in)
Is preferably provided. If the return value of this command is TRUE, that is, there is an object in the microwave oven, instead of executing the subsequent command, the user is informed that the object is in the microwave oven. It is also possible to communicate and end the process.

  Further, if this command sequence is applied to an article to be put in the microwave oven, there arises a problem that a process of “warming” is performed on any object. For this reason, for example, the microwave oven may be provided with a mechanism for recognizing the contents of the article and switching the specific warming method of the “warming” process accordingly. For example, when “warming” and “thawing” are provided as detailed functions of “warming” of the microwave oven, the object placed in the cabinet is given to some method, for example, image processing or an object. It may be recognized by an electronic tag or the like, and the “warming of cooking” and “defrosting” may be switched appropriately according to the result. Of course, other methods may be used for this switching. For example, when the microwave oven does not have the recognition function, the robot 102 may have the function, and the robot 102 may recognize the contents of the object and send the result to the microwave oven.

  As described above, in this embodiment, a series of robot control command lists for realizing instruction data is generated and executed using such a robot control command DB.

  In addition, after the generation of the robot control command list is completed, the process may be executed without confirmation by the user. However, there is a case where the content instructed by the article instruction operation unit is incorrect and the user does not notice it. It is possible. For example, in the environment of FIG. 4B, when trying to warm the lunch box in the refrigerator compartment by placing it in the microwave oven, the lunch box in the refrigerator compartment is placed directly below the microwave oven due to a user's mouse operation error. It may be put in As a result, it may happen that a warm lunch cannot be made. In order to prevent such a situation as much as possible, the action plan of the robot 102 created by the action plan creation means 117 is presented to the user before execution, thereby prompting the user to check whether there is an error in the article operation. It is preferable.

  FIG. 16 shows a screen for confirming an instruction to move an article. In this example, “Bento” that is a moving article, “Refrigerator” that is the source, “Microwave” that is the destination, and “Warm” that is the process at the destination are indicated by characters, respectively. Further, in order to clarify the movement source and the movement destination, a dotted curve with an arrow from the movement source to the movement destination is shown. If the source or destination is not equipment, nothing may be displayed. For example, when the lunch box is on the floor, the movement source is not equipment, so it is not displayed, and only the “microwave oven” of the movement destination is displayed. Of course, as a method of displaying this confirmation screen, equipment and articles may be represented by letters, or highlighted in the original drawing.

  Furthermore, if an error has occurred, correction may be performed in this confirmation process. For example, when the article moving to the microwave oven in the above example is “frozen meat” instead of “lunch”, the treatment in the microwave oven should be “thaw” rather than “warm”. However, as shown in FIG. 15, in the robot control command, if the last command in the robot control command sequence at the movement destination of the microwave oven is only “warm (start)”, it is executed. Therefore, even if the article placed in the microwave oven is “frozen meat”, if the user does nothing, the microwave oven “warms” this “frozen meat”. When such a processing error occurs, the user may correct it before executing the processing.

  Further, as described above, the present article management system 100 based on FIG. 1 includes four subsystems of the environment management server 101, the robot 102, the facility 104, and the operation terminal 103, and these subsystems are wireless or wired. It is configured to exchange information with each other via the network. However, a configuration in which the operation terminal 103 is attached to the environment management server 101, the facility 104, or the robot 102 may be employed. Further, the configuration may be such that not only one robot 102 but also a plurality of robots cooperate to perform work in parallel.

In the above description, the mechanism of the system operation when the user instructs only one operation of moving a certain article from one place to another and executes it is described. However, in practice, there are cases where the user wants to give two or more instructions. In this case, if one instruction is processed and then the next instruction is input, the user cannot leave the system until all desired instructions have been issued, and the system is difficult for the user to use. It becomes. Therefore, it is desirable that the system is designed to accept all operations that the user wants to perform at a time and process them sequentially. Thus, for example, if you require the system to warm the lunch in the refrigerator and then eat it, the following two instructions (not shown):
-Put the refrigerator lunch in the microwave and warm.-Bring the microwave lunch to your place. That is, after the user issues the above two instructions, the system automatically executes the above operation simply by waiting. As a result, the user can do other things until the warm lunch box comes to him, and can use his time efficiently.

  Furthermore, it is preferable to provide a scheduling function for improving the efficiency of a plurality of processes at that time. For example, if you take out multiple types of goods from a facility and move them to different locations, you can schedule all the processing to be efficient by performing only the process of taking out the multiple types of goods at a time. Good. For this purpose, it is not necessary to limit the number of arms 201 and the like of the work robot 102 to one, and a multi-arm configuration may be adopted so that a plurality of articles can be handled simultaneously.

  It should be noted that the above-described embodiment and its modifications can be realized using a computer program. The program according to the present invention includes a computer program that executes part or all of the operations of the above-described embodiment and its modifications.

  As described above, the present invention is useful for an article management system that manages articles in a living environment such as a house or an office, and a program for controlling the article management system. In particular, a system for instructing operations on an article using various electronic devices such as a computer or a portable information terminal (such as a mobile phone) as an article operating means, and moving the article by an article moving means such as a robot based on the instruction. And useful for programs.

It is a block diagram showing the whole article management system composition concerning an embodiment. (A)-(c) is explanatory drawing for demonstrating a background difference method. It is the conceptual diagram which showed the structure and description content example of article | item data, (a) shows the state before cleaning, (b) shows the state after cleaning. (A)-(b) is the schematic diagram which image | photographed the mode of a certain environment in two different time. It is a figure for demonstrating an environment map, (a) represents an actual condition, (b) represents a stereo model, (c) represents the plane model of (a). It is the figure which showed an example of the data of an environment map. (A)-(b) is a figure showing an example of equipment attribute data. It is the figure which showed the equipment operation command memorize | stored in the equipment operation information storage means in tabular form. (A) is the example of an operation screen, (b) is the conceptual diagram which showed the example of the screen map corresponding to the operation screen of (a). (A)-(e) is the figure which showed the example of the display form of the articles | goods accommodated in the inside of an installation. It is a figure showing the correspondence with the conditions which switch the display form inside an installation, and a display form. It is a figure showing an example of the display form of an installation, the place behind the installation, the floor surface, and articles | goods. (A)-(d) is the conceptual diagram which showed the example of the hierarchical screen map corresponding to the environment of FIG. It is a perspective view which shows the structure of a robot. It is the figure of the table format which showed the example of the list | wrist of the robot control command memorize | stored in robot control command DB. It is the figure which showed the example of the confirmation screen.

Explanation of symbols

101 Environmental management server
102 robot
103 Operation terminal
104 facilities
105 Sensing means
106 Goods retrieval and management means
107 Goods database (database)
108 Environmental map management means
109 Environmental Map
110 Transmission / reception means
111 Control method of environmental management server
112 sensors
113 Grasping means
114 Movement plan creation means
115 Traveling means
116 Robot control means
117 Action plan creation means
118 Article operation means
119 Display means
120 Control method of operation terminal
121 Equipment control means
122 Equipment operation information storage means
123 Sensing means
124 In-facility display means (display data generation means)
601 Environmental attribute data
602 Equipment attribute data

Claims (16)

A system for managing articles present in a living environment,
A database for managing information on articles in the living environment;
An environment map for managing structural information of facilities and spaces in the living environment;
Display means for displaying the actual situation of the living environment;
Article operation means for performing an operation of designating at least an article on the display by the display means;
Display data generating means for generating data for displaying information on articles hidden inside or behind equipment on the display means based on the information of the database and the environment map;
When the equipment is designated by the article operation means, display control means for displaying information on the article hidden inside or behind the equipment on the display means,
An article management system comprising: A system for managing articles present in a living environment,
Article operation means for performing an operation of designating at least an article on the display in the display means for displaying the actual situation of the living environment;
Based on information of a database for managing information on articles in the living environment and an environment map for managing structural information on equipment and space in the living environment, information on articles hidden inside or behind the equipment is displayed. Display data generating means for generating data to be displayed on the means;
When the equipment is designated by the article operation means, display control means for displaying information on the article hidden inside or behind the equipment on the display means,
An article management system comprising: A system for managing articles present in a living environment,
A database for managing information on articles in the living environment;
An environment map for managing structural information of facilities and spaces in the living environment;
Display means for displaying the actual situation of the living environment;
Article operation means for performing an operation of designating at least an article on the display by the display means;
Display data generating means for generating data for displaying information on the inside of the equipment and the hidden article on the display means based on the information of the database and the environment map;
When the equipment is designated by the article operating means, either one of the information on the article hidden inside the equipment and the information on the article hidden behind the equipment is selectively and freely displayed on the display means. Display control means for
An article management system comprising: A system for managing articles present in a living environment,
Article operation means for performing an operation of designating at least an article on the display in the display means for displaying the actual situation of the living environment;
Based on information of a database that manages information on articles in the living environment and an environment map that manages structural information on equipment and space in the living environment, information on articles hidden inside and behind the equipment is displayed. Display data generating means for generating data to be displayed on the means;
When the equipment is designated by the article operating means, either one of the information on the article hidden inside the equipment and the information on the article hidden behind the equipment is selectively and freely displayed on the display means. Display control means for
An article management system comprising: Sensing means for detecting information on the article;
Database control means for updating information in the database based on the detection result of the sensing means;
The article management system according to claim 1, further comprising: Article moving means for moving the article;
An article movement control means for causing the article movement means to move the article to the movement location when the article and the movement location of the article are instructed by the article operation means;
The article management system according to claim 1, further comprising: The article operation means displays a cursor for article operation on the display of the display means,
When the cursor is overlaid on the equipment on the display of the display means, the display control means displays information on the article hidden inside or behind the equipment alone or together with the structure information of the equipment in a pop-up list format. The article management system according to any one of claims 1 to 6. The article operation means displays a cursor for article operation on the display of the display means,
At least one of the facilities in the living environment is provided with photographing means for photographing the interior or shadow area of the facility,
The display control means displays an image of an article hidden inside or behind the equipment when the cursor is superimposed on the equipment on the display of the display means. Goods management system. The article operation means displays a cursor for article operation on the display of the display means,
At least one of the facilities in the living environment is provided with a transmission means for making the inside or shadow area of the facility visible.
The display control means transparently displays an article hidden inside or behind the equipment when the cursor is superimposed on the equipment on the display of the display means. Goods management system. The equipment consists of equipment that performs a predetermined operation,
The article management system according to claim 1, further comprising a control unit that causes the facility to execute the operation when the facility is designated by the article operating unit. The facility includes a door that can be opened and closed,
11. The article management system according to claim 10, wherein the control means opens or closes a door of the equipment when the equipment is designated by the article operation means. The display data generation means generates a plurality of data for displaying article information in a plurality of display forms on the display means,
The said display control means selects the said display form based on at least one of the attribute of the said equipment, the attribute of the said article, the attribute of an operator, and operation content, The Claim 1 any one of Claims 1-11. Goods management system. A database for managing information on articles in the living environment, an environment map for managing structural information on facilities and spaces in the living environment, display means for displaying the actual status of the living environment, and display on the display means A program for controlling an article management system comprising at least an article operation means for performing an operation of designating an article, wherein the article management system includes:
Generating data for displaying on the display means information on an article hidden inside or behind the facility based on the information of the database and the environment map;
When the equipment is designated by the article operating means, displaying information on an article hidden inside or behind the equipment on the display means;
A program that executes A database for managing information on articles in the living environment, an environment map for managing structural information on facilities and spaces in the living environment, display means for displaying the actual status of the living environment, and display on the display means A program for controlling an article management system comprising at least an article operating means for performing an operation for designating an article and an article operating means for performing an operation for designating at least an article on a display by the display means. In the article management system,
Generating data for displaying on the display means information on articles hidden inside and behind the facility based on the information of the database and the environment map;
When the equipment is designated by the article operating means, either one of the information on the article hidden inside the equipment and the information on the article hidden behind the equipment is selectively and freely displayed on the display means. And steps to
A program that executes A database for managing information on articles in the living environment, an environment map for managing structural information on facilities and spaces in the living environment, display means for displaying the actual status of the living environment, and display on the display means A method for controlling an article management system comprising at least an article operation means for performing an operation of designating an article,
Generating data for displaying on the display means information on an article hidden inside or behind the facility based on the information of the database and the environment map;
When the equipment is designated by the article operating means, displaying information on an article hidden inside or behind the equipment on the display means;
A control method comprising: A database for managing information on articles in the living environment, an environment map for managing structural information on facilities and spaces in the living environment, display means for displaying the actual status of the living environment, and display on the display means A method for controlling an article management system comprising at least an article operating means for performing an operation for designating an article and an article operating means for performing an operation for designating at least an article on a display by the display means. ,
Generating data for displaying on the display means information on articles hidden inside and behind the facility based on the information of the database and the environment map;
When the equipment is designated by the article operating means, either one of the information on the article hidden inside the equipment and the information on the article hidden behind the equipment is selectively and freely displayed on the display means. And steps to
A control method comprising:

Images