JP2012022457A - Task instruction system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide technology capable of instructing to a robot, sequential tasks consisting of a plurality of sequential tasks using information of an individual and concrete object in a user's use environment.SOLUTION: The task instruction system includes: a photographing section; a display section for displaying a photographed picture; a recognition section for recognizing an object; a task input section which allows a user to perform task input procedures including a step for photographing a picture including an object related to the task, a step for selecting the object or an electric household appliance related to the task in the photographed picture, and a step for inputting an operation of the robot or the electric household appliance with respect to the selected object or the operation of the selected electric household appliance; an annotated picture generation section which generates a text representing a task content, and combines it with the picture of the object or the electric household appliance that the user selects in the task input procedures to generate an annotated picture; and a control section which sends a control command of the task corresponding to the annotated picture.

Description

  The present invention relates to a technique for instructing a task to be executed by a robot or a home appliance.

  Computer-controlled robots for general households have appeared on the market. For example, cleaning robots are used in many households. Research on robots that perform various household tasks such as cooking and folding clothing is also underway.

  Housework can be understood as a sequence of a plurality of different tasks such as movement of objects and operation of home appliances, or a collection of similar tasks such as storing different types of tableware on different shelves. Execution procedures and objects (household appliances, tableware, etc.) of tasks constituting such housework vary widely depending on individual home environments. Therefore, it is preferable that the operation of the robot that performs housework can be customized so that the user can be adapted to his / her own usage environment. And it is desirable that the instruction to the robot can accurately reflect the execution procedure of the user's favorite task and the information of the individual specific object in the user environment. In addition, since the task execution procedure and the object easily change due to changes in the home environment, it is desirable that the instruction content to the robot can be easily confirmed and edited.

  Various methods for a user to give a unique instruction to a robot have been proposed. For example, a method using a programming language is typical, but this method is difficult to learn, and it is not easy to check and edit instructions. There is a visual programming environment that uses animation and icons (for example, Non-Patent Document 1), but the same abstraction work as in the case of general programming is required, and information on actual objects is used as it is. You cannot give instructions to the robot.

  Although there is a method of giving instructions to the robot using natural language, information unique to the user such as the execution procedure of the user's favorite task and individual specific objects in the user environment is expressed and transmitted unambiguously in natural language. This is not easy even in communication between humans, and it is difficult to give accurate instructions to the robot in natural language. Although there are methods of giving instructions to the robot using human demonstrations or photographs (for example, Non-Patent Document 2), there is no consideration for instructing a sequence of a plurality of tasks. A method for causing a robot to execute a sequence of a plurality of tasks by reinforcement learning has been proposed (for example, Non-Patent Document 3). However, in this method, the behavior of the robot when the usage environment changes is unknown, and the user can It is not easy to check and edit the instruction content.

Leifer, L. Van der Loos, M. Lees, D. 1991.Visual language programming: for robot command / control in unstructured environments.In Conf.on Advanced Robotics, 31-36. Hashimoto, S., Ostanin, A., Inami M, and Igarashi, T. Photograph-Based Interaction for Teaching Object Delivery Tasks to Robots. In HRI 2010 Late-Breaking, 153-154. Calinon, S., Guenter, F., and Billard, A. 2007. On Learning, Representing, and Generalizing a Task in a Humanoid Robot.In IEEE Trans.on Systems, Man, and Cybernetics, v. 37, 286-298 .

  It is an object of the present invention to provide a technique that can instruct a robot on a sequential task including a plurality of continuous tasks using information on individual specific objects in a user's usage environment. . It is another object of the present invention to provide a technique capable of easily confirming and editing the contents of a task executed by a robot.

In order to solve the above problem, the present invention is a task instruction system for instructing a task to a robot and a household electrical appliance,
A shooting section;
A display unit for displaying an image captured by the imaging unit;
A recognition unit for recognizing an object reflected in an image displayed on the display unit;
A task input unit that allows a user to input the contents of one task,
A step of photographing an image including an object related to a task by a photographing unit;
A step of selecting an object related to a task among objects recognized by a recognition unit in a captured image or an image of a home appliance;
A step of inputting an operation of the robot or household electrical appliance with respect to the selected object or an operation of the selected household electrical appliance;
A task input procedure that includes a task input procedure that causes the user to enter the contents of one task, and when the task input procedure for one task is completed, the task input unit enters a state waiting for the task input procedure for the next task;
An annotated image that is an image including the text representing the content of the task input in the task input unit and the generated text and an image showing the object or home appliance selected by the user in the task input procedure An annotated image generator that is generated in association with the input task;
A control unit that generates a control command for causing the robot or the home appliance to execute the task associated with the annotated image, and sends the control command to the robot or the home appliance;
Have

  According to this task instruction system, a user performs a task to be executed by a robot or a household electrical appliance by an intuitive operation of photographing an individual specific object in the user environment and selecting an image of the photographed actual object. Can be entered. Therefore, a task to be executed by the robot can be instructed using individual specific information specific to the user environment. Further, since the input procedure for the next task can be performed after completion of the input procedure for a plurality of one task, a sequential task consisting of a plurality of continuous tasks can be easily designated.

  In addition, since the annotated image includes text representing the task content and an image showing the target object of the task, the user can easily confirm the task content later on the basis of the annotated image. Can do.

  In the task instruction system of the present invention, the annotated image generation unit uses a text representing the name of the object or home appliance selected by the user in the task input procedure and a text representing the operation input by the user, Text for annotated images can be generated according to rules.

  With the text generated in this way, the user can easily grasp the content of the task and information about the target object without ambiguity.

  In the task instruction system of the present invention, when a task input procedure for a plurality of tasks is performed in the task input unit, the control unit sequentially executes the plurality of tasks in the order in which the task input procedure is performed in the task input unit. A control command can be generated and sent out to be executed by a home appliance.

  As a result, a sequential task can be instructed by inputting a plurality of tasks in the order in which the robot wants the robot to execute, and the sequential task can be instructed intuitively and easily.

  In the task instruction system of the present invention, the display unit may include a touch panel display that allows a user to select a displayed image by a touch-type pointing operation.

  As a result, the user can perform task input by an intuitive operation.

The task instruction system of the present invention includes:
A task input device having a photographing unit, a display unit, a recognition unit, a task input unit, and an annotated image generation unit;
A control unit having a control unit and configured to be communicable with a task input device, a robot, and a home appliance;
It can be set as the structure which has these.

  In this case, since the task execution command to the robot and the home appliance is performed by a control device separate from the task input device, the task input device is not necessary when the robot and the home appliance execute the task. The method for the control device to send a task execution command to the robot or home appliance is to start the task execution triggered by a wall switch or the operating state of other home appliances as well as a method for operating the control device. It is possible to adopt a simple configuration, and the degree of freedom of the method for executing the task is increased.

In the task instruction system of the present invention, the task input unit is a task input procedure for a task to be executed by a household electrical appliance that does not require an operation target.
Capturing an image including a home appliance that performs the task;
A step of causing a user to select an image of a household electrical appliance that instructs an operation among objects recognized by a recognition unit in a captured image;
Inputting an operation to be executed by the selected home appliance;
To the user,
The annotated image generation unit can generate the text for the annotated image using the text representing the name of the home appliance selected by the user in the task input procedure and the text representing the operation input by the user. .

  In the step of inputting an operation, for example, an instructable operation list can be presented to select one of the operations, or a user can input a desired numerical value. Home appliances that do not require an operation target are home appliances that do not affect the state of the target object, and examples thereof include lamps, radios, and watches.

In the task instruction system of the present invention, the task input unit is a task input procedure for a task to be executed by a household electrical appliance that requires an operation target.
A step of photographing an image including an electric appliance that executes a task and an object that can be an operation target of the electric appliance;
A step of causing a user to select an image of a home appliance that instructs the operation among objects recognized by the recognition unit in the captured image, and an object that is an operation target of the home appliance,
Inputting an operation to be executed by the selected home appliance;
To the user,
The annotated image generation unit includes a text representing the name of the home appliance selected by the user in the task input procedure, a text representing the name of the object to be operated by the home appliance, a text representing the operation input by the user, Can be used to generate text for an annotated image.

  Household appliances that require an operation target are household appliances that affect the state of the target object, and examples include a water heater that pours hot water into a cup and a dispenser that puts coffee powder into a cup. In order for these home appliances to operate correctly, it is necessary that an appropriate operation target exists. For example, when the spoon is the target of operation, the operation of the water heater is not appropriate. In the step of photographing, if the home appliance and the object that can be the operation target are not captured, a task to perform an inappropriate operation is created if it is not allowed to move to the next step of selecting. This can be suppressed.

In the task instruction system of the present invention, the task input unit, in the task input procedure of the object movement task to be executed by the robot,
Capturing an image including an object to be moved;
A step of selecting an image of an object to be moved by the robot among movable objects recognized by the recognition unit in the captured image;
Photographing an image including an object to be moved and an object to be moved; and
A step of selecting an image of an object to be moved by the robot among objects that can be specified as a movement destination recognized by the recognition unit in the captured image;
To the user,
The annotated image generation unit uses the text representing the name of the object to be moved selected by the user in the task input procedure, the text representing the name of the object to be moved, and the text meaning “move” Text for the annotated image can be generated.

  In order to prevent an inappropriate movement task from being instructed also in the movement task instruction, it may be configured to prohibit selection of an inappropriate movement destination in the step of selecting a movement destination. For example, when the object to be moved is a spoon, a water heater is inappropriate as the object to be moved, and therefore it is preferable that selection of the water heater is prohibited in the step of selecting.

In the task instruction system of the present invention, the task input unit is configured to cause the user to set a trigger for starting execution of a predetermined task by the robot or the home appliance.
Photographing an image including an object as a trigger;
Selecting an image of an object to be set as a trigger among objects that can be set as a trigger recognized by the recognition unit in the captured image;
Inputting a condition to be satisfied with respect to an object selected for starting execution of a predetermined task by a robot or a home appliance; and
Allows the user to perform a trigger setting procedure that includes
The annotated image generation unit uses an annotated image using text representing the name of the object selected by the user in the trigger setting procedure, text meaning the satisfaction of the condition input by the user, and text meaning “wait”. Generate text for, and generate an image including the generated text and an image of the object selected by the user in the trigger setting procedure as an annotated image associated with the trigger setting,
The control unit can generate a control command for controlling the robot or the home appliance according to the trigger setting associated with the annotated image, and send the control command to the robot or the home appliance.

  In a configuration where the operation of the central control device and the input device is necessary to execute the task, the convenience for the user is not good. By setting the trigger object, the task can be executed in various ways. According to the above configuration, the trigger object can be set in an intuitive manner while capturing an image of the actual trigger object. In addition, since an annotated image corresponding to the trigger setting is also generated, the user can easily confirm the contents of the trigger setting later.

In the task instruction system of the present invention, the task input unit causes the user to perform a standby time setting procedure including a step of inputting a standby time in order to cause the user to set a time for the robot or home appliance to wait for execution of the task. It is possible,
The annotated image generation unit generates text that means waiting for the elapse of the standby time input by the user in the standby time setting procedure, and generates an image including the generated text in association with the standby time setting as an annotated image. ,
The control unit can generate a control command for controlling the robot or the home appliance according to the standby time setting associated with the annotated image, and send the control command to the robot or the home appliance.

  According to this configuration, it is not necessary to perform shooting when setting the standby time. That is, it is possible to save the trouble of taking a clock image, for example, in order to instruct the standby time, which is highly convenient.

  In the task instruction system of the present invention, the task input unit can edit the contents of the task corresponding to the existing annotated image by causing the user to perform the task input procedure, the trigger setting procedure, or the waiting time setting procedure again. it can.

  Thereby, the user can edit the content of the existing task by intuitive operation similar to the time of task input. Even at the time of editing, as in the case of task input, it is possible to easily reflect information on individual specific objects specific to the user's use environment in the contents of the task without ambiguity.

  In the present invention, the recognition unit, task input unit, annotated image generation unit, and control unit can each be regarded as dedicated hardware. Also, it can be understood as a program that realizes each functional unit by being executed by the CPU.

  ADVANTAGE OF THE INVENTION According to this invention, the sequential task which consists of a several continuous task can be instruct | indicated with respect to a robot using the information of the individual specific target object in a user's use environment. In addition, it is possible to easily check and edit the contents of tasks to be executed by the robot.

1 is a diagram showing a configuration of a housework system including an embodiment of a task instruction system according to the present invention. It is a functional block diagram of a housework system and an input device. It is an example of the database with which a housework system is equipped. It is a figure which shows the correspondence of the icon displayed on the live view image of an input device, and an object. It is a transition diagram of the operating state of an input device. It is an example of GUI display of an input standby screen for waiting for a user to start a task creation process in the task creation mode of the input device. FIG. 7A is a GUI display example of the movement task input screen. FIG. 7B is a GUI display example of the movement destination instruction screen. FIG. 7C is a display example of an annotated image representing a movement task. FIG. 8A is a GUI display example of the home appliance operation input screen. FIG. 8B is a GUI display example of the parameter setting screen. FIG. 8C is a display example of an annotated image representing the home appliance operation task. FIG. 9A is a GUI display example of the standby time setting screen. FIG. 9B is a display example of an annotated image representing the standby time. FIG. 10A shows a GUI display example of the trigger setting screen. FIG. 10B is a display example of an annotated image representing the trigger setting. It is a display example of GUI of a task content display screen for displaying the content of an existing task in the task content confirmation mode. It is a flowchart showing the process in which a user inputs the content of a task with an input device. It is a figure which shows the task which comprises the "put coffee" task. FIG. 14 is an example of an annotated image obtained when the “put coffee” task of FIG. 13 is input by an input device.

  Hereinafter, embodiments of the present invention will be described in detail. The dimensions, materials, shapes, relative arrangements, and the like of the components described in the present embodiment are not intended to limit the technical scope of the invention to those unless otherwise specified.

<Housework system>
FIG. 1 is a diagram showing a configuration of a housework system including an embodiment of a task instruction system according to the present invention. This housework system is built in a room 11 and includes a ceiling camera 1, a robot 2, a water heater 3, a coffee powder dispenser 4, a tablet dispenser 5, a lamp 6, an alarm clock 7, a table 8, an input device 9, a computer 10, and a cup 12. . The water heater 3, the coffee powder dispenser 4, the tablet dispenser 5, the lamp 6, and the alarm clock 7 are examples of “home appliances” in the present invention, and their operations are controlled by control commands transmitted from the computer 10 wirelessly or by wire. The The cup 12 is an example of the “movable object” in the present invention.

  The ceiling camera 1 is a small camera including an image sensor such as a CCD or a CMOS installed on the ceiling of the room 11 and can capture the entire area of the room 11. In addition, the apparatus which image | photographs the household appliances and objects in the room 11 will be anything as long as it is an imaging apparatus that can keep the components constituting the housework system among the household appliances and objects in the room 11 within the angle of view. But it is not limited to the ceiling camera.

  The robot 2, the water heater 3, the coffee powder dispenser 4, the tablet dispenser 5, the lamp 6, the alarm clock 7, the table 8, and the cup 12 have markers 23, 33, 43, 53, 63 at positions that can be photographed from the ceiling camera 1, respectively. 73, 83 and 123 are attached. Each marker is a unique graphic that can identify these objects. The correspondence between the marker and the object is registered in the database of the computer 10, and the computer 10 can recognize each object and track the position based on the marker image photographed by the ceiling camera 1. Note that the object recognition method and the position tracking method are not limited to image recognition using such markers, and any recognition technique can be used.

The robot 2 is a robot that executes a task of moving the cup 12 on the table 8. The robot 2 is a self-propelled robot including a moving mechanism 21 composed of two wheels and a drive motor and a claw-like hand 22. The robot 2 can move the cup 12 by grabbing and lifting the cup 12 with the hand 22, moving with the moving mechanism 21, and lowering and releasing the cup 12 gripped with the hand 22.

  The operation of the robot 2 is controlled by a control command transmitted from the computer 10 wirelessly or by wire. The robot 2 includes a controller (not shown) that controls the operation of the moving mechanism 21 and the hand 22 based on a control command from the computer 10. Operations such as “forward”, “reverse”, “right rotation”, “left rotation” of the moving mechanism 21, “grab”, “release”, “lift”, “lower”, etc. of the hand 22 are preprogrammed in the controller 10. The operation of the robot 2 can be controlled by sending a control command corresponding to each operation. The position of the robot 2 is tracked by the computer 10 based on the image taken by the ceiling camera 1 and controlled so as not to fall from the table 8 based on the information on the boundary of the table 8. Alternatively, the robot 2 itself may determine the movable area.

  The water heater 3 can perform operations such as “pour hot water”, “heat”, “stop”, and the like according to control commands from the computer 10. For the “pour hot water” operation, it is possible to control the amount of pouring, such as “1 second”, “2 seconds”, “3 seconds”, “continue to pour”. The water heater 3 includes a push button configured to be able to pour an arbitrary amount of hot water when pressed, and the push button is pressed by a robot having a mechanism capable of pressing a person or the push button. It may be configured to be able to pour hot water. In this case, the operation of pouring hot water by the water heater 3 can be controlled by a control command transmitted from the computer 10 to the robot.

  The coffee powder dispenser 4 can perform operations such as “put coffee powder” by a control command from the computer 10. As for the operation of “adding coffee powder”, the amount of “1 second”, “2 seconds”, “3 seconds”, etc. can be controlled. The point which may have the structure which can put arbitrary quantity of coffee powder by human operation is the same as that of the water heater 3.

  The tablet dispenser 5 can perform operations such as “insert a small tablet”, “insert a medium tablet”, and “insert a large tablet” by a control command from the computer 10. The point which may have the structure which can put the tablet of arbitrary sizes by human operation is the same as that of the water heater 3 grade | etc.,.

  The water heater 3, the coffee powder dispenser 4, and the tablet dispenser 5 are examples of “home appliances that require an operation target” in the present invention. That is, the water heater 3 changes the amount of hot water in the object such as the cup 12, the coffee powder dispenser 4 changes the amount of coffee powder in the object such as the cup 12, and the tablet dispenser 5 is a small plate (not shown). Changing the amount of the tablet in the object causes a change in the state of the object.

  The lamp 6 can perform operations such as “lighting” and “lighting off” in accordance with a control command from the computer 10. In addition, the computer 10 can detect states such as “lit” and “off”. The alarm clock 7 can control operations such as a ringing time and a ringing method according to a control command from the computer 10. The lamp 6 and the alarm clock 7 are examples of “home appliances that do not require an operation target” in the present invention. That is, operations such as “turning on” and “off” of the lamp 6 are completed only by the lamp 6, and operations such as “ringing” of the alarm clock 7 are completed only by the alarm clock 7, and the state of some object is changed. It does not.

  As described above, a stationary general household electrical appliance has been exemplified as the household electrical appliance. However, an automatic machine that includes a transition mechanism, an arm, and the like like the robot 2 and performs more complicated and advanced operations can be the household electrical appliance of the present invention. Each of the robot and the home appliance in the present invention is an automatic machine that automatically executes a predetermined operation in the absence of a user in accordance with a control command from the computer 10, and can be regarded as a robot in a broad sense.

  The cup 12 is an object whose position can be moved by the robot 2, and is an example of the “movable object” in the present invention. Further, since the state of the amount of hot water or coffee powder changes depending on the operation of the water heater 3 or the coffee powder dispenser 4, it is an object that can be the operation target of the water heater 3 or the coffee powder dispenser 4.

  On the table 8, a robot 2, a water heater 3, a coffee powder dispenser 4, a tablet dispenser 5 and a cup 12 are arranged. A table 8 is defined as an actionable range of the robot 2. A marker for identifying the boundary of the region of the table 8 may be attached to the table 8 so that the robot 2 is controlled so as not to fall off the table 8.

  The input device 9 is a handheld device that includes a touch panel display 91 and a camera 92. When the user performs a predetermined operation on the input device 9, the contents of a task to be executed by the robot 2 and the home appliances (the water heater 3, the coffee powder dispenser 4, the tablet dispenser 5, the lamp 6, and the alarm clock 7) are input. be able to. The task input to the input device 9 is transmitted to the computer 10 wirelessly or by wire, converted into a control command for transmission to the robot 2 and the home appliance in the computer 10, and transmitted to the robot 2 and the home appliance.

<Input device>
FIG. 2 is a functional block diagram of the housework system and the input device 9. The same reference numerals and names are used for components equivalent to those described in FIG. As shown in FIG. 2, the input device 9 includes a touch panel display 91 (display unit), a camera 92 (imaging unit), an image capture unit 93, an object recognition unit 94 (recognition unit), a database 95, a GUI control unit 96, An annotated image generation unit 97 (annotated image generation unit), a display control unit 98, and a task input unit 99 (task input unit) are included.

  The camera 92 includes an image sensor such as a CCD or a CMOS, and outputs a captured image to the image capturing unit 93. The image capturing unit 93 captures an image captured by the camera 92 and outputs it to the object recognition unit 94 and the display control unit 98.

  The object recognition unit 94 recognizes an object shown in an image photographed by the camera 92 based on the above-described marker attached to each object. Information on the correspondence between the marker graphic and the object is stored in the database 95. Note that the method in which the object recognition unit 94 recognizes an object from the image captured by the camera 92 of the input device 9 and the method in which the computer 10 recognizes the object from the image captured by the ceiling camera 1 of the housework system are the same. Or different. When the same recognition technology is used in the housework system and the input device, the database is centrally managed in either the input device 9 or the computer 10, and the other is accessed by wireless or wired communication. Anyway. Any technique can be used as a method for recognizing an object in an image. The object recognition unit 94 outputs recognition result information to the GUI control unit 96.

The GUI control unit 96 acquires information such as the name of the object recognized by the object recognition unit 94, the type of the object, and an operation list that can be instructed when the user inputs the task content from the database 95, and text representing the name of the object In addition, an icon corresponding to the type of object, a GUI (graphical user interface) for the user to input task contents, and the like are output to the display control unit 98. Details of the task input by the user will be described later.

  The database 95 stores information on objects registered in the home appliance system. FIG. 3 shows an example of the database. The database illustrated in FIG. 3 stores, for each object, the name of the object, the type of the object, a list of operations that can be instructed when inputting the task content, and information on the object that can be an operation target. The types of objects include the above-described movable objects, home appliances that require an operation target, home appliances that do not require an operation target, and place types. Here, the “place” is an object that can be designated as a movement destination in the task of moving the object, and includes a table, a kitchen counter, and the like. The database 95 also stores information related to a range (boundary) that can be specified as a movement destination of an object by the robot 2 on a table or kitchen counter.

  Regarding household electrical appliances, in addition to the list of operations defined for each household electrical appliance described above, information on objects that can be designated as operational targets is also stored particularly for household electrical appliances that require operational targets. For example, a cup and a mug are registered as an operation target of a water heater, but a dish and a spoon are not registered.

  FIG. 4 shows an example of the correspondence between object types and icons. The GUI control unit 96 outputs the icons shown in FIG. 4 to the display control unit 98 according to the type of object acquired by inquiring of the database 95.

  The “trigger” icon is an icon displayed on the trigger object. The trigger object is an object that can be set so that the robot or home appliance starts executing a task when a predetermined condition is satisfied. Although not shown in FIG. 1, the trigger object is, for example, a wall switch. By setting the wall switch as the trigger object and setting the condition that the switch is pressed as the condition for the robot or home appliance to start executing the task, the user can press the wall switch to A command to start execution can be transmitted to the robot or the home appliance. That is, when instructing the execution of a task, the user may perform an operation that satisfies the task start condition on the trigger object instead of operating the computer 10 or the input device 9. Also, if the alarm clock 7 and the trigger object are used, and if the condition of “ring” is met, the robot or home appliance can be set to start the task of “adding coffee”, the alarm clock 7 will sound automatically In addition, the execution of the task of making coffee can be started.

  The "prohibited" icon cannot be specified as an operation target when inputting a task by a user, which will be described later, for example, an object that cannot be specified as a movement destination during input of an object movement task or an operation task of a home appliance This icon is displayed on a simple object.

  Returning to FIG. 2, the display control unit 98 appropriately performs image processing such as enlargement / reduction processing on an image captured by the camera 92 output from the image capturing unit 93 and also performs an object recognition unit in the captured image after image processing The text and icon output from the GUI control unit 96 are overlaid and displayed at the position where the object recognized by 94 is reflected. In addition, when a user inputs a task, the captured image after image processing is combined with a GUI for task input. Then, a display image to be finally displayed on the touch panel display 91 is generated and output to the touch panel display 91. The display control unit 98 can perform live view display in which an image currently captured by the camera 92 is output to the touch panel display 91 as a video in real time. It is also possible to output the captured image at the moment designated by the user to the touch panel display 91 as a still image.

  The touch panel display 91 includes a display device that displays a display image output from the display control unit 98 and a touch panel that is provided on the display device. As the touch panel, various types such as a capacitance type, a resistance film type, and a matrix switch type can be used. The input interface can use a keyboard or a pointing device instead of the touch panel, or a combination thereof.

  The task input unit 99 performs processing for causing the user to input the contents of a task to be instructed to the robot 2 or the home appliance. Input by the user is performed via the touch panel display 91. The user captures an image of an object or home appliance related to the task while viewing an actual object or home appliance image displayed in live view on the touch panel display 91, and inputs the task content via touch operation and GUI To do. Details of the task input procedure that the task input unit 99 causes the user to perform will be described later.

  The annotated image generation unit 97 generates a text representing the content of the task input in the task input unit 99, and shows the generated text and objects related to the task and home appliances taken by the user in the task input procedure. An annotated image that combines a photograph with The annotated image includes graphics such as icons and arrows output from the GUI control unit 96. Details of the annotated image will be described later.

  The annotated image is stored in the storage device of the input device 9 in association with the task. The annotated image generation unit 97 transmits the generated annotated image to the computer 10 of the housework system by wireless or wired communication. The computer 10 includes a control unit 101 that generates a control command for causing the robot 2 or the home appliance to execute a task associated with the annotated image acquired from the annotated image generation unit 97. The control unit 101 transmits the control command to the robot 2 and home appliances by wireless or wired communication. Thereby, the task corresponding to the annotated image is executed.

  Note that the control unit 101 that generates a control command for transmitting to the robot 2 or the home appliance from the annotated image may be provided in the input device 9 instead of the computer 10. Then, the input device 9 may transmit the control command information to the computer 10 wirelessly or by wire, and the computer 10 may transmit the control command to the robot 2 or the home appliance at an appropriate timing. Alternatively, the input device 9 itself may have a function of transmitting a control command to the robot 2 and the home appliance, and may directly transmit the control command to the robot 2 and the home appliance without using the computer 10. Examples of wireless communication methods include Bluetooth and infrared rays.

  Of the functional blocks of the input device 9, the functional units of the object recognition unit 94, the database 95, the GUI control unit 96, the display control unit 98, and the annotated image generation unit 97 may be implemented by dedicated hardware, respectively. It can also be realized by the CPU of the input device 9 loading and executing a program stored in the storage device. These functions can also be designed as an application program that operates on a device such as an existing PC or smartphone having a camera and a touch panel display. In this case, when the application program operates on these devices, the devices function as the input device 9. In addition, the image capturing unit 93 acquires an image captured by a fixed camera such as a web camera or a surveillance camera provided in the ceiling camera 1 or the separate room 11 via the computer 10 or an external server by wired or wireless communication. By configuring so, a device that does not include a camera can also function as the input device 9.

<Task input method>
A method of inputting task contents by the input device 9 will be described with reference to FIGS.

  FIG. 5 is a transition diagram of the operating state of the input device 9. 6 to 11 are examples of GUIs displayed on the touch panel display 91 of the input device 9. FIG. 12 is a flowchart showing a process in which the user inputs the contents of a task using the input device 9.

  As shown in FIG. 5, the state of the input device 9 is roughly divided into two states: a task creation mode and a task content confirmation mode. In the task creation mode, the user can create a new task. In the task content confirmation mode, the user can confirm and edit the content of an existing task. In the task creation mode, first, an input standby screen (FIG. 6) for waiting for the user to start the task creation process is displayed. When touching various object icons displayed in the live view image, the touched object is displayed. Transitions to the screen for entering tasks related to. The transition screen varies depending on the touched object. When the icon of the movable object is touched, the screen transits to the movement task input screen (FIG. 7). When the home appliance icon is touched, a transition is made to the home appliance operation input screen (FIG. 8). When the icon of the trigger object is touched, a transition is made to the trigger setting screen (FIG. 10). When the standby time setting button is touched, the screen transits to a standby time setting screen (FIG. 9). In the task content confirmation mode, a screen (FIG. 11) for displaying the task content is displayed. Switching from the input standby screen to the task content display screen is performed by touching a thumbnail image representing an existing task. The task content display screen is switched to the input standby screen by touching the live view button.

<Input standby screen>
FIG. 6 is a GUI display example of an input standby screen for waiting for the user to start the task creation process in the task creation mode. On the GUI of the input standby screen, thumbnail images 601 and 602, a live view button 603, a standby time setting button 604, and a live view image 605 are displayed.

  Thumbnail images 601 and 602 are GUIs for shifting to a task content confirmation mode for confirming the content of an existing task corresponding to each thumbnail image. If there is no existing task, the thumbnail image is not displayed.

  The live view button 603 is highlighted to indicate that the current mode is the task creation mode.

  The standby time setting button 604 is a GUI for making a transition to a standby time setting screen (FIG. 9) for inputting a time for which the robot or home appliance waits for execution of a task.

  In the live view image 605, an image captured by the camera 92 is displayed in real time. Further, the image of the object recognized by the object recognition unit 94 in the captured image is displayed with an overlay indicating text representing the name of the object and an icon representing the type of the object. When the user touches and selects an icon of any object, the screen transitions to a screen for inputting a task related to the selected object according to the type of the selected object.

  When the user touches an icon of a movable object in the live view image, the movable object is selected as a task input target, and a moving task input screen for inputting a moving task for moving the selected movable object by the robot 2 ( Transition to FIG.

When the user touches the home appliance icon in the live view image, the home appliance is selected as a task input target, and a transition is made to a home appliance operation input screen (FIG. 8) for inputting an operation to be executed by the selected home appliance. .

  When the user touches the trigger object icon in the live view image, the trigger object is selected as the trigger setting target, and the trigger for inputting conditions that the trigger object must satisfy in order for the robot or home appliance to start executing the task Transition to the setting screen (FIG. 10). Although the trigger object and the icon are not shown in the display example of FIG. 6, if the trigger object is captured in the image captured by the camera 92, the trigger object icon and the trigger object name are the same as those of the movable object and the home appliance. Overlaid on the trigger object image.

  In the live view image 605 illustrated in FIG. 6, the water heater 3 and the cup 12 are shown. According to the recognition result of the object recognition unit 94, the image of the water heater 3 includes an icon 662 representing “home appliance” and a “water heater”. The name is displayed as an overlay, and the image of the cup 12 is displayed with an icon 672 indicating “movable object” and the name “cup” as an overlay.

<Move task input screen>
FIG. 7A is a GUI display example of the movement task input screen. On the GUI of the movement task input screen, a movement destination instruction button 701, a final state instruction button 702, a redo button 703, an all selection button 704, and a departure point image 705 are displayed. On the movement task input screen, the user can input a movement task by two methods. There are a method of instructing a destination and a method of instructing a final state to be achieved by a movement task.

  The movement destination instruction button 701 is a GUI for shifting to a movement destination instruction screen for inputting the contents of the movement task by a method of instructing the movement destination.

  The final state instruction button 702 is a GUI for shifting to a final state instruction screen for inputting the contents of the movement task by a method of instructing the final state.

  The redo button 703 is a GUI for returning to the input standby screen (FIG. 6).

  The all selection button 704 is a GUI for selecting all the movable objects shown in the departure point image 705. In the default state when moving to the moving task input screen, the movable object selected by touching the user in the live view image 605 on the input standby screen is selected. A task for moving a plurality of movable objects can be easily input by instructing a destination after touching all selection buttons 704.

  The departure point image 705 is a still image in which both the movable object selected on the input standby screen (FIG. 6) and the departure point object or place are shown. The departure point image 705 is generated from the captured image displayed in the live view image 605 when the user touches the icon of the movable object on the input standby screen (FIG. 6).

  The display example of FIG. 7A is a case where the user touches the cup icon 672 in the state of the input standby screen illustrated in FIG. 6 (that is, the state where the cup 12 and the water heater 3 are reflected in the live view image 605). In the display example of the moving task input screen, the starting point image 705 shows the cup 12 as the object to be moved and the water heater 3 as the object at the starting point of the movement.

<Destination instruction screen>
FIG. 7B is a GUI display example of the movement destination instruction screen. On the GUI of the movement destination instruction screen, a live view image 706, a departure point image 707, and a redo button 708 are displayed.

  The departure point image 707 is a still image in which the selected object to be moved and the object or place of the departure point are shown, and the departure point image 705 displayed on the movement task input screen (FIG. 7A). Generated from

  The redo button 708 is a GUI for returning to the movement task input screen (FIG. 7A).

  The live view image 706 is a real-time image (video) taken by the camera 92. The user manually moves the movement target object to the movement destination, and captures the state with the camera 92. While viewing the live view image 706, the user takes an image so that both the movement target object and the movement destination object or location appear in the live view image 706, and the object recognition unit 94 causes the movement target object and the movement destination object or location to move. Recognize In this state, when an arbitrary place on the live view image 706 is touched, the object or place recognized together with the movement target object in the live view image 706 is determined as a movement destination by the movement task, and the movement task instruction is completed. In the live view image 706, a prohibition icon may be displayed instead of an icon indicating the type of object for an object that cannot be set as the movement destination of the movement target object. For example, since hot water is not poured from the water heater 3 into the dish, the dish is not registered as an operation target object of the water heater 3 in the database of the system. When a movement task in which the object to be moved is a dish is input, even if the water heater 3 is reflected in the live view image 706 on the movement destination instruction screen, a prohibition icon is displayed on the water heater 3 and the dish movement destination is displayed. Can not be specified as.

  The display example of FIG. 7B shows a state in which the user takes the cup 12 to the table 8 that is the movement destination and is taking a picture so that the cup 12 and the table 8 appear in the live view image 706 in that state. ing. In the live view image 706, a cup icon 672 and a table icon 682 are displayed. The table icon 682 is the “place” icon described above.

<Annotated image>
When the movement task instruction is completed, an annotated image representing the content of the instructed movement task is generated by the annotated image generation unit 97 and displayed on the touch panel display 91.

  FIG. 7C is a display example of an annotated image representing a movement task. In the annotated image of the moving task, a departure point image 709, a final state image 710, an arrow graphic 714, texts 711 to 713, a save button 715, and a redo button 716 are displayed.

  The departure point image 709 is a still image showing the object to be moved and the object or place of the departure point related to the input movement task, and the departure point displayed on the movement task input screen (FIG. 7A). It is generated from the point image 705.

  The final state image 710 is a still image in which the object to be moved and the object or location of the movement destination related to the input movement task are shown. The user can view the live view image on the movement destination instruction screen (FIG. 7B). It is generated from the captured image displayed on the live view image 706 when the touch button 706 is touched.

  An arrow graphic 714 connects the image of the moving target object in the starting point image 709 and the image of the moving target object in the final state image 710 and intuitively represents that this task is a task for moving the object. .

  The texts 711 to 713 are texts representing the contents of the input movement task, and are based on the user input contents on the movement task input screen (FIG. 7A) and the movement destination instruction screen (FIG. 7B). Automatically generated by the annotated image generation unit 97. The text generated for the annotated image of the movement task includes text 712 that represents the name of the object to be moved, text 713 that represents the name of the object to be moved, and text 711 that means “move”.

  The save button 715 is a GUI for saving the generated annotated image. When the user touches the save button 715, the input of the movement task is completed, and the screen returns to the input standby screen (FIG. 6).

  The redo button 716 is a GUI for returning to the movement destination designation screen (FIG. 7B). Returning to the movement destination designation screen (FIG. 7B), it is possible to redo the shooting of the movement destination object or place while viewing the live view image 706.

  The display example of FIG. 7C is an annotated image representing a moving task for moving the cup 12 from the water heater 3 to the table 8, and the starting point image 709 includes the cup 12 that is the moving target object and the starting point object. The final-state image 710 includes the cup 12 that is the object to be moved and the table 8 that is the destination location. The image of the cup 12 and the final state of the image 709 at the departure point. The image of the cup 12 in the image 710 is connected by an arrow graphic 714, and texts 711 to 713 meaning “the cup moves to the table” are displayed. The text included in the annotated image may be displayed as an image caption or title, as in the display example of FIG. 7C, or may be displayed as a bullet, or a normal sentence. May be displayed. Regardless of the display mode, the user can grasp the contents of the task without ambiguity by viewing it together with the image in which the object that is the target of the actual task is shown.

<Final status instruction screen>
Here, a moving task instruction method by a method of instructing the final state achieved by the moving task will be described.

  The final state desired to be achieved by the moving task is, for example, a state in which the room cleanup is completed. Take a picture of the messy room, move to the movement task input screen (FIG. 7A), touch the all selection button 704, or select a plurality of objects to be moved, and touch the final state instruction button 702. . Then, the screen shifts to a final state instruction screen (not shown) for instructing a final state in which each of the selected plurality of objects to be moved is stored in an appropriate place. As a final state instruction method, for example, a method of causing the user to select a room image in a tidy state can be exemplified. The image is configured to be accompanied by information on the correspondence between various objects and their storage locations. The user selects one image of the room in the final tidy state on the final state instruction screen, so that the plurality of objects selected on the movement task input screen (FIG. 7A) are stored in the respective storage locations. A plurality of parallel movement tasks of moving can be instructed at a time. Here, the plurality of parallel tasks is a set of tasks that can be executed in any order. According to the method of instructing the final state, the user can save the trouble of manually taking the object to be moved to the object or location of the destination and taking a picture as in the method of instructing the destination. The method of instructing the final state is suitable for instructing a fixed task in which the storage destination is constant. Note that even when one movable object is selected on the movement task input screen (FIG. 7A), the movement task can be input by the method of instructing the final state.

<Home appliance operation input screen>
FIG. 8A is a GUI display example of the home appliance operation input screen. On the GUI of the home appliance operation input screen, an operation list 801, a redo button 802, and an image 803 of the home appliance are displayed.

  The operation list 801 is a list of operations that can be instructed to the home appliance selected on the input standby screen (FIG. 6). Information on the operation list that can be instructed by the home appliance is stored in a database of the system (see FIG. 3). The user can instruct the home appliance to perform the touched operation by touching any of the instructable operation lists 801.

  The redo button 802 is a GUI for returning to the input standby screen (FIG. 6).

  The home appliance image 803 is a still image showing the home appliance selected on the input standby screen (FIG. 6). When the selected household electrical appliance is a household electrical appliance that requires an operation target object, it is a still image in which both the selected household electrical appliance and the operation target object are shown. The home appliance image 803 is generated from the captured image displayed in the live view image 605 when the user touches the home appliance icon on the input standby screen (FIG. 6).

  The display example of FIG. 8A is a case where the user touches the water heater icon 662 in the state of the input standby screen illustrated in FIG. 6 (that is, the state where the cup 12 and the water heater 3 are shown in the live view image 605). This is a display example of the home appliance operation input screen, and an image 803 of the home appliance shows a water heater 3 and a cup 12 as an operation target of the water heater. The operation list 801 lists “pour”, “heat”, “continue pouring”, and “pour stop” as operations that can be instructed to the water heater 3. Note that parameters can be set for the “pour” operation, and the pouring time can be selected from “1 second”, “2 seconds”, and “3 seconds”. Parameters can also be set for the “heating” operation, and a target temperature can be designated.

  When a button for an operation that allows parameter setting is touched in the operation list 801 of the home appliance operation input screen (FIG. 8A), a transition is made to the parameter setting screen (FIG. 8B). On the GUI of the parameter setting screen, a settable value list 804, a redo button 802, and an image 803 of home appliances are displayed. Information on parameter values that can be set for the operation of the home appliance is stored in a database of the system (see FIG. 3). The user can set a parameter value by touching one of the settable value lists 804.

  The display example of FIG. 8B is a display example of the parameter setting screen displayed when the “pour” button of the operation list 801 of FIG. 8A is touched, and “1 second” is settable. “2 seconds” and “3 seconds” are listed. In addition to the list selection GUI, the parameter setting screen may display a GUI that can display a GUI such as a numeric keypad and can input an arbitrary numerical value. For example, in the parameter setting screen displayed when the “heat” button in the operation list 801 in FIG. 8A is touched, the GUI of the numeric keypad is set so that an arbitrary temperature can be input as a heating operation parameter. Can be displayed.

  When one of the operation lists 801 on the home appliance operation input screen (FIG. 8A) is touched and the operation to be executed by the home appliance is confirmed (in the case of an operation capable of parameter setting, the parameter setting screen (FIG. 8 ( When the setting of the parameter value is confirmed in B))), the instruction of the home appliance operation task is completed.

<Annotated image>
When the home appliance operation task instruction is completed, an annotated image representing the content of the instructed home appliance operation task is generated by the annotated image generation unit 97 and displayed on the touch panel display 91.

  FIG. 8C is a display example of an annotated image representing the home appliance operation task. In the annotated image of the home appliance operation task, a home appliance image 805, texts 806 to 809, a save button 810, and a redo button 811 are displayed.

  The home appliance image 805 is a still image in which the home appliance related to the input home appliance operation task is shown. From the home appliance image 803 displayed on the home appliance operation input screen (FIG. 8A). Generated.

  Texts 806 to 809 are texts representing the contents of the input home appliance operation task, and include user input contents on the home appliance operation input screen (FIG. 8A) and parameter setting screen (FIG. 8B). Based on this, it is automatically generated by the annotated image generation unit 97. The text generated for the annotated image of the home appliance operation task includes a text 806 representing the name of the home appliance, a text 807 representing the operation of the home appliance, a text 808 representing a parameter value as necessary, and an action target It includes text 809 representing the name of the object.

  The save button 810 is a GUI for saving the generated annotated image. When the user touches save button 810, the input of the home appliance operation task ends, and the screen returns to the input standby screen (FIG. 6).

  The redo button 811 is a GUI for returning to the home appliance operation input screen (FIG. 8A). Returning to the home appliance operation input screen (FIG. 8A), the operation designation from the operation list 801 can be performed again.

  The display example of FIG. 8C is an annotated image representing the task of pouring hot water into the cup 12 for 2 seconds with the water heater 3, and the water heater 3 and the cup 12 that is the operation target are shown in the image 805 of the home appliance. And texts 806 to 809 meaning “pour hot water into the cup for 2 seconds with a water heater” are displayed. The display mode of the text is not limited to the display example of FIG. 8C, as in the case of the annotated image of the moving task.

  In the housework system of the present embodiment, the water heater 3 is registered as a home appliance that requires cups such as a cup 12 and a mug as an operation target (see FIG. 3), and an input standby screen (FIG. 6). Only when both the water heater 3 and an object (such as cup 12 or other cup-attribute tableware) that can be operated are shown in the live view image 605 of the home appliance operation input screen (FIG. 8A) ) May be permitted. In this case, when the tableware attributed with the water heater 3 is not reflected in the live view image 605 on the input standby screen (FIG. 6), the operation input screen of the water heater 3 is displayed even if the water heater icon 662 is touched. Transition to FIG. 8 (A)) is not performed. At this time, an alert display such as “Please copy the cup in the screen” or the like may be performed to prompt the user to copy both the cup attributed tableware and the water heater 3.

When the water heater 3 performs the operation of pouring hot water with the cup 12 as an operation target, it is obvious that the water heater 3 should be operated after the cup 12 has been moved to just below the spout of the water heater 3. Therefore, even if the user does not place the cup 12 directly below the spout of the water heater 3, if the water heater 3 and the cup 12 are shown in one screen, the home appliance operation input screen of the water heater 3 (FIG. 8 ( A)) is allowed, and when a task for pouring hot water into the cup 12 is input by the water heater 3 in that state, the task is “move the cup 12 to just below the pouring spout of the water heater 3. The moving task “Make” may be included behind the scenes. In this way, it is possible to reduce the trouble when the user takes an image while manually moving the object when inputting the operation of the home appliance.

<Standby time setting screen>
FIG. 9A is a GUI display example of the standby time setting screen. On the GUI of the standby time setting screen, a unit button 901, a numeric button 902, a display unit 903, an enter button 904, and a redo button 905 are displayed.

  The unit button 901 and the number button 902 are GUIs for the user to input numerical values and time units by a touch operation. The display unit 903 is a GUI for displaying the input time. The determination button 904 is a GUI for determining the standby time based on the time currently displayed on the display unit 903. The redo button 905 is a GUI for returning to the input standby screen (FIG. 6).

  In the setting of the standby time, it is not necessary to take an image as in the case of inputting a movement task or a home appliance operation task.

<Annotated image>
When the standby time is determined, an annotated image representing the standby time is generated by the annotated image generation unit 97 and displayed on the touch panel display 91.

  FIG. 9B is a display example of an annotated image representing the standby time. Text 907 to 909, a save button 910, and a redo button 906 are displayed on the annotated image of the waiting time. Unlike the annotated image of the moving task and the annotated image of the home appliance operation task, only the text is included and the image of the object or home appliance is not included.

  The save button 910 is a GUI for saving the generated annotated image. When the user touches the save button 910, the setting of the standby time is completed, and the screen returns to the input standby screen (FIG. 6).

  The redo button 906 is a GUI for returning to the standby time setting screen (FIG. 9A). Returning to the standby time setting screen (FIG. 9A), the standby time can be input again.

  Text 907 is a text representing the content of the standby time setting, and is automatically generated by the annotated image generation unit 97 based on the user input content on the standby time setting screen (FIG. 9A). The display example of FIG. 9B includes text 907 indicating “time”, text 908 indicating the length of the set standby time, and text 909 indicating “wait”. The display mode of the text is not limited to the display example of FIG. 9B, as in the case of the annotated image of the moving task.

<Trigger setting screen>
FIG. 10A shows a GUI display example of the trigger setting screen. On the GUI of the trigger setting screen, a trigger object image 102, a task execution condition list 103, and a redo button 104 are displayed.

  The trigger object image 102 is a still image in which the trigger object selected on the input standby screen (FIG. 6) is shown. The trigger object image 102 is generated from the captured image displayed in the live view image 605 when the user touches the trigger object icon on the input standby screen (FIG. 6).

  The task execution condition list 103 is a list of actions and states of trigger objects that can be set as task execution conditions (startup conditions). When the user touches and selects one of the conditions listed in the task execution condition list 103, the condition is set as the task execution condition.

  The redo button 104 is a GUI for returning to the input standby screen (FIG. 6).

  The display example of FIG. 10A is a table of the trigger object setting screen when the user touches the icon of the alarm clock 7 while the alarm clock 7 is reflected in the live view image 605 of the input standby screen (FIG. 6). In this example, an alarm clock 7 that is a trigger object and an icon 111 of the trigger object are displayed in the trigger object image 102. The task execution condition list 103 lists “time” and “ringing” as alarm clock operations and states that can be set as task execution conditions. When the user touches “Time”, a screen for setting the time to execute the task is displayed, and the trigger setting is set so that the task is executed when the time pointed by the alarm clock 7 reaches the set time. can do. When the user touches “ring”, the trigger can be set so that the task is executed when the alarm clock 7 rings.

  When one of the task execution condition lists 103 on the trigger setting screen (FIG. 10A) is touched to determine the task execution conditions, the trigger setting is completed.

<Annotated image>
When the trigger setting is completed, an annotated image representing the contents of the trigger setting is generated by the annotated image generation unit 97 and displayed on the touch panel display 91.

  FIG. 10B is a display example of an annotated image representing the trigger setting. In the annotated image of the trigger setting, a trigger object image 105, texts 108 to 110, a save button 106, and a redo button 107 are displayed.

  The trigger object image 105 is a still image in which the trigger object related to the trigger setting is captured, and is generated from the trigger object image 102 displayed on the trigger setting screen (FIG. 10A).

  Texts 108 to 110 are texts representing the contents of the trigger setting, and are automatically generated by the annotated image generation unit 97 based on the user input contents on the trigger setting screen (FIG. 10A). The text generated for the annotated image of the trigger setting includes text 108 representing the name of the trigger object, text 109 representing the task execution condition, and text 110 representing the content of the task to be executed.

  The save button 106 is a GUI for saving the generated annotated image. When the user touches the save button 106, the trigger setting ends and the screen returns to the input standby screen (FIG. 6).

  The redo button 107 is a GUI for returning to the trigger setting screen (FIG. 10A). Returning to the trigger setting screen (FIG. 10A), the task execution conditions can be specified again from the task execution condition list 103.

The display example of FIG. 10B is an annotated image representing a trigger setting in which “ringing” of the alarm clock 7 is an execution condition of the “coffee making task”, and the alarm clock 7 is shown in the image 105 of the trigger object. The texts 108 to 110 meaning “add coffee when the alarm clock rings” are displayed. The display mode of the text is not limited to the display example of FIG.

<Task content confirmation mode>
FIG. 11 is a GUI display example of a task content display screen for displaying the content of an existing task in the task content confirmation mode. In the GUI of the task content display screen, thumbnail images 601 and 602, a live view button 603, an annotated image 606, a delete button 607, and an edit button 608 are displayed.

  Thumbnail images 601 and 602 are GUIs for transitioning to a task content display screen that displays the contents of existing tasks corresponding to the thumbnail images. If there is no existing task, the thumbnail image is not displayed. The display example of FIG. 11 shows a state in which the contents of a task corresponding to the thumbnail image 602 are displayed. When the user touches the thumbnail image 601 in this state, the contents of the task corresponding to the thumbnail image 601 are displayed.

  The live view button 603 is a GUI for shifting to an input standby screen (FIG. 6) in the task creation mode.

  The annotated image 606 is an annotated image corresponding to the task whose contents are being displayed. In the display example of FIG. 11, an annotated image of the moving task is displayed.

  The delete button 607 is a GUI for deleting a task whose contents are being displayed.

  An edit button 608 is a GUI for editing a task whose contents are being displayed. When the edit button 608 is touched, for example, the redo button is touched on the annotated image display screen in the task creation mode (FIG. 7C, FIG. 8C, FIG. 9B, and FIG. 10B). A GUI similar to the screen displayed in this case is displayed, and the contents of the task can be changed.

<Flowchart>
FIG. 12 is a flowchart showing a process in which a user inputs the contents of a task using the input device 9. The user starts task input from the input standby screen (FIG. 6) (step S10). The user takes a picture with the camera 92 of the input device 9 so that the object related to the task to be input is reflected in the live view image on the input standby screen (step S20).

  The user selects an object related to a task to be input by touching the icon of the object shown in the live view image (step S30). On the touch panel display 91, a task input screen (any one of FIGS. 7 to 10) corresponding to the type of the object selected by the user is displayed (step S40). The user performs an operation of inputting task contents on the GUI of the task input screen (step S50).

  When the task input by the user is completed, an annotated image is generated based on the input task content and displayed on the touch panel display 91 (step S60). One task can be input by the above process. The user can input a plurality of tasks continuously by repeating the above process.

When inputting a sequential task composed of a plurality of continuous tasks, the above process is repeated for all the tasks constituting the sequential task (step S70). When the input of all the tasks constituting the sequential task is completed, an annotated image representing the contents of the sequential task is created and displayed (step S80). An annotated image of a sequential task is an image in which annotated images of individual tasks constituting the sequential task are arranged along the task execution procedure.

  The process for allowing the user to input task contents by the process described above is controlled by the task input unit 99. The task input unit 99 causes the user to capture an image including an object related to the task with the camera 92 (step S10 and step S20), and causes the user to recognize the object recognized by the object recognition unit 94 in the captured image. Of these, a step (step S30) of selecting an object related to the task or an image of the home appliance by a touch operation, and the user controls the operation of the robot 2 or the home appliance with respect to the selected object or the operation of the selected home appliance. The content of the task is input by causing a task input procedure including steps (step S40 and step S50) to be input via the GUI output by the unit 96 to be performed. When the user completes the task input procedure for inputting the contents of one task, the task input unit 99 waits for the task input procedure for the next task.

<Generation of comment text>
Here, the text generated for the annotated image by the annotated image generation unit 97 will be described.

  As described above, the annotated image generation unit 97 automatically generates text representing the task content based on the task content input by the user on the task input screen (FIGS. 7 to 10). An annotated image represents a task content by combining a text and an image, and the user can obtain a lot of information about an individual specific object or home appliance as a task target from the image. Therefore, it is sufficient that the text can express the minimum necessary information, and the text included in the annotated image illustrated in FIGS. 7 to 10 is automatically generated according to a simple rule.

  Text included in the annotated image is generated based on the image. A sentence is generated with a noun representing the name of an object or home appliance in the image as a subject or object and a verb representing a task content as a predicate. The verb representing the content of the task is “move” if it is a movement task, and is a verb corresponding to the action selected by the user from the action list if it is a home appliance operation task. When the home appliance requires an operation target, the verb representing the content of the task is a transitive verb, and the noun representing the name of the object to be operated or the home appliance is the object. When the home appliance does not require an operation target, the verb representing the content of the task is an intransitive verb. When the operation of the home appliance can set a parameter, the verb representing the task content is accompanied by the parameter.

The text included in the annotated image is described by, for example, the following context-free grammar.
(1) S * => S | SS *
(2) S => subject V object
(3) S => subject V
(4a) V => action
(4b) V => action parameter

A movable object follows (2). “Subject” is a movable object, “object” is a destination object or location, and “action” is “(move to destination)” “move (to final state)”. For example, the text “s: cup, v: move, o: table” (s is subject, v is action, o is object) expresses an instruction for the robot to move the cup to the table.

Home appliances comply with (2) or (3). Home appliances that affect the state of other objects, that is, home appliances that require an operation target, comply with (2). For example, the text “s: kettle, v: pour, o: cup” represents an instruction to the kettle to change the state of the cup. A household electrical appliance that does not change the state of another object, that is, a household electrical appliance that does not require an operation target, conforms to (3). For example, the text “s: lamp, v: lit” expresses an instruction to turn on the lamp.

The trigger object follows (3). For example, the text “s: push button, v: wait, p: pressed” (p is parameter) represents a trigger instruction to execute a task when the push button is pressed. The standby time setting which is a kind of trigger setting also follows (3). For example, the text “s: time, v: wait, p: 5 seconds” represents an instruction to wait for 5 seconds until the next task is executed. As described above, it is not necessary to take an image in the standby time setting. Accordingly, the subject “time” in this case is not determined based on the image, but is used when the standby time setting button 604 is touched on the input standby screen (FIG. 6).

  When the text has a plurality of subjects, the text is decomposed into a plurality of texts having one subject and converted into instructions to the robot. For example, the text “s: A, B, C, v: move, o: table” is “s: A, v: move, o: table” “s: B, v: move, o: Table ”“ s: C, v: move, o: table ”is decomposed into three texts, and the task of causing the robot to move the object A to the table, the task of moving the object B to the table, and the object C This is an expression that designates three tasks: a task to be moved to the table.

  On the task input screen, the user is forced to create an annotated image that can generate text with the correct syntax. In other words, it is necessary to create an annotated image so that all elements necessary for giving instructions to the robot and home appliance are reflected in the image. For example, in the case of an annotated image of a moving task, the object to be moved needs to be reflected in both the starting point image and the moving destination image. In the case of an annotated image of an operation task of a home appliance that requires an operation target, both the home appliance and the operation target object need to be shown. When such a condition is not satisfied, as described above, the user is prompted to make a correction by using a prohibition icon or an alert, and a text having an appropriate syntax is generated.

  On the task input screen, the user is forced to create an annotated image that can generate meaningful text. In other words, it is necessary to create an annotated image so that instructions to the robot and home appliances are semantically appropriate instructions. For example, in the operation task of a water heater, a cup is appropriate as an operation target, but a dish and a spoon are not appropriate. Therefore, a warning by a prohibition icon or alert is performed on the task input screen so as to prohibit the appearance of a dish or spoon in the object of the text representing the operation task of the water heater. Information on appropriate operation targets and inappropriate operation targets is registered in the home appliance and housework system databases.

  As described above, in the housework system of the present embodiment, the task input unit 99 can perform a syntax check and a semantic check in a simple manner, so that appropriate instructions can be given to the robot and the home appliance.

<Example of coffee making task>
Housework can be understood as a sequential task in which a plurality of two types of tasks, a task for moving an object and a task for operating a home appliance, are continued. For example, as shown in FIG. 13, the housework of “adding coffee” includes (1) moving the cup 12 to the coffee powder dispenser 4 and (2) operating the coffee powder dispenser 4 to supply a desired amount of coffee powder. (3) Move the cup 12 from the coffee powder dispenser 4 to the water heater 3, (4) Next, operate the water heater 3 to pour a desired amount of hot water into the cup 12, (5) 3 minutes When waiting (because it is hot), (6) the cup 12 can be moved from the water heater 3 to the table 8 and can be broken down into a sequence of movement tasks and household appliance operation tasks.

  According to the input device 9 of the housework system of this embodiment, the user can easily instruct a sequential task composed of a plurality of such continuous tasks according to the flowchart of FIG. When a plurality of tasks are input, the computer 10 transmits a control command so that the robot or home appliance sequentially executes the input tasks in the order in which the task input procedures are performed. In each task instruction, the user performs an intuitive operation of specifying an image by pointing operation while capturing an image of an individual actual task target object in the user environment. You can enter the contents. Therefore, it is possible to reflect the information about the target object and the home appliance unique to the user environment in the content of the task without ambiguity. Furthermore, the contents of the input task are collected as an annotated image combining the photographed image and the text, and can be easily confirmed later. FIG. 14 is an annotated image obtained by inputting the six tasks constituting the “coffee making task” in FIG. As shown in FIG. 14, an annotated image representing a sequential task is a sequence of a plurality of annotated images. 14A to 14F correspond to individual tasks (1) to (6) constituting the coffee making task shown in FIG.

  The annotated image shown in FIG. 14 is different from a task constructed by an abstract programming language, and includes an image showing an actual task object and text representing the task content. It can be grasped intuitively. Therefore, for example, when a water heater is replaced or a taste of coffee changes, it is possible to easily determine which task should be corrected in order to correct the coffee filling task accordingly. . When editing a task, it is possible to intuitively edit the contents of the task while viewing an actual image taken by the camera 92.

  Other household chores, such as cooking, move food from the counter to the food processor, then process the food in the food processor, then move the food from the food processor to the microwave, and then heat the food in the microwave , And so on, can be broken down into a sequence of movement tasks and household appliance operation tasks. Various housework such as washing, cleaning up rooms, assembling things, etc. can be broken down into such a combination of movement and operation of home appliances. And if it is a housework task which can be decomposed into a movement task and a home appliance operation task, the task contents can be input intuitively while photographing an actual task target object or home appliance with the input device 9 of this embodiment. An annotated image can be generated.

  In the housework system of the present embodiment, the movement task is instructed by specifying the object to be moved, the object or location of the departure point, and the object or location of the movement destination. The “location” of the departure point and the movement destination means an object having a “location” attribute, not a coordinate. That is, in this embodiment, the moving task is defined as a relationship between objects. From this, even if the object to be moved and the object to be moved are placed at different positions from when the moving task is created, if the computer 10 can detect the position of each object from the image of the ceiling camera 1, the moving task Is executed correctly.

9 Input device 92 Imaging device 94 Object recognition unit 91 Touch panel display 97 Annotated image generation unit 99 Task input unit 101 Control unit

Claims (11)

A task instruction system for instructing a task to a robot and home appliances,
A shooting section;
A display unit for displaying an image captured by the imaging unit;
A recognition unit for recognizing an object reflected in an image displayed on the display unit;
A task input unit that allows a user to input the contents of one task,
A step of photographing an image including an object related to a task by a photographing unit;
A step of selecting an object related to a task among objects recognized by a recognition unit in a captured image or an image of a home appliance;
A step of inputting an operation of the robot or household electrical appliance with respect to the selected object or an operation of the selected household electrical appliance;
A task input procedure that includes a task input procedure that causes the user to enter the contents of one task, and when the task input procedure for one task is completed, the task input unit enters a state waiting for the task input procedure for the next task;
An annotated image that is an image including the text representing the content of the task input in the task input unit and the generated text and an image showing the object or home appliance selected by the user in the task input procedure An annotated image generator that is generated in association with the input task;
A control unit that generates a control command for causing the robot or the home appliance to execute the task associated with the annotated image, and sends the control command to the robot or the home appliance;
A task instruction system. In claim 1,
The annotated image generation unit uses the text representing the name of the object or home appliance selected by the user in the task input procedure and the text representing the operation input by the user, for the annotated image according to a predetermined rule. A task instruction system that generates text.   In Claim 1, when the task input procedure of a plurality of tasks is performed in the task input unit, the control unit sequentially transfers the plurality of tasks to the robot or the home appliance in the order in which the task input procedure is performed in the task input unit. A task instruction system for generating and sending control instructions to be executed. In claim 1,
The display unit is a task instruction system having a touch panel display that allows a user to select an image to be displayed by a touch-type pointing operation. In claim 1,
A task input device having a photographing unit, a display unit, a recognition unit, a task input unit, and an annotated image generation unit;
A control unit having a control unit and configured to be communicable with a task input device, a robot, and a home appliance;
A task instruction system. In claim 1,
The task input unit is a task input procedure for a task to be executed by a home appliance that does not require an operation target.
Capturing an image including a home appliance that performs the task;
A step of causing a user to select an image of a household electrical appliance that instructs an operation among objects recognized by a recognition unit in a captured image;
Inputting an operation to be executed by the selected home appliance;
To the user,
The annotated image generation unit generates a text for an annotated image using text representing the name of the home appliance selected by the user in the task input procedure and text representing the operation input by the user . In claim 1,
The task input unit is a task input procedure for a task to be executed by a home appliance that requires an operation target.
A step of photographing an image including an electric appliance that executes a task and an object that can be an operation target of the electric appliance;
A step of causing a user to select an image of a home appliance that instructs the operation among objects recognized by the recognition unit in the captured image, and an object that is an operation target of the home appliance,
Inputting an operation to be executed by the selected home appliance;
To the user,
The annotated image generation unit includes a text representing the name of the home appliance selected by the user in the task input procedure, a text representing the name of the object to be operated by the home appliance, a text representing the operation input by the user, A task instruction system that generates text for annotated images using. In claim 1,
The task input unit is a task input procedure for an object moving task to be executed by the robot.
Capturing an image including an object to be moved;
A step of selecting an image of an object to be moved by the robot among movable objects recognized by the recognition unit in the captured image;
Photographing an image including an object to be moved and an object to be moved; and
A step of selecting an image of an object to be moved by the robot among objects that can be specified as a movement destination recognized by the recognition unit in the captured image;
To the user,
The annotated image generation unit uses the text representing the name of the object to be moved selected by the user in the task input procedure, the text representing the name of the object to be moved, and the text meaning “move” A task instruction system that generates text for annotated images. In claim 1,
The task input unit allows the user to set a trigger that causes the robot or home appliance to start executing a predetermined task.
Photographing an image including an object as a trigger;
Selecting an image of an object to be set as a trigger among objects that can be set as a trigger recognized by the recognition unit in the captured image;
Inputting a condition to be satisfied with respect to an object selected for starting execution of a predetermined task by a robot or a home appliance; and
Allows the user to perform a trigger setting procedure that includes
The annotated image generation unit uses an annotated image using text representing the name of the object selected by the user in the trigger setting procedure, text meaning the satisfaction of the condition input by the user, and text meaning “wait”. Generate text for, and generate an image including the generated text and an image of the object selected by the user in the trigger setting procedure as an annotated image associated with the trigger setting,
The control unit is a task instruction system that generates a control command for controlling the robot or the home appliance according to the trigger setting associated with the annotated image and sends the control command to the robot or the home appliance. In claim 1,
The task input unit allows the user to set the time for the robot or home appliance to wait for the task to be executed.
Allows the user to perform a waiting time setting procedure that includes the step of entering a waiting time,
The annotated image generation unit generates text that means waiting for the elapse of the standby time input by the user in the standby time setting procedure, and generates an image including the generated text in association with the standby time setting as an annotated image. ,
The control unit is a task instruction system that generates a control command for controlling the robot or the home appliance according to the standby time setting associated with the annotated image and sends the control command to the robot or the home appliance. In claim 1,
A task input system in which a task input unit edits the contents of a task corresponding to an existing annotated image by causing a user to perform a task input procedure, a trigger setting procedure, or a waiting time setting procedure again.

Images