JP2012135858A - System and method for construction of operating environment model - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the burden of modeling an introduction environment for a working robot.SOLUTION: The system 100 for constructing an operating environment model is configured to: generate geometry information indicating geometrical characteristics of an object to be operated by applying a geometric primitive to a three-dimensional image obtained by measuring the object to be operated; obtain position information indicating a space position of the object to be operated that is the generation target of the geometry information; and memorize operation information in association with the generated geometry information and the obtained position information, the operation information indicating operation contents that are to be performed by a working robot to the object to be operated. An operation tag can be set virtually to the object to be operated, thus reducing the burden of modeling an introduction environment for the working robot.

Description

  The present invention relates to an operation environment model construction system and an operation environment model construction method.

  Attempts have been made to apply robot technology to various environments.

  Patent Document 1 discloses a system that causes a robot to perform a moving operation of an article in a living space where a person is active such as a general household, office, hotel, store, and hospital. In Patent Document 1, a state in which an article can be identified using an electronic tag is constructed in advance (see, for example, paragraphs 0029 and 0066 to 0067 of the document). The movement of the article by the work robot is performed based on information obtained from the electronic tag (for example, see paragraph 0153 of the same document). Patent Document 2 discloses a procedure for moving an article via an operation screen, as shown in FIGS.

  Patent Document 3 discloses a robot work teaching system capable of giving an appropriate work instruction to a work robot. As shown in FIG. 3 of the document, an object model of an object is selected (see paragraph 0027 of the document), and then a work model is selected (see paragraph 0028 of the document). 4 of the same document is an example of a screen, FIG. 5 of the same document is an icon list for designating the type of work object, and FIG. 6 of the same document is an icon list for designating work for the object. .

  Patent Document 4 discloses a robot simulation apparatus that artificially performs an operation of a robot having a visual sensor offline. Patent Document 5 discloses that a specified elementary shape model is fitted to three-dimensional data, and an appropriate gripping pattern is acquired from a database based on the result (see FIG. 2 of the same document). .

Japanese Patent No. 3738256 Japanese Patent No. 3713021 JP 2008-68348 A Japanese Patent No. 4153528 JP 2009-214212 A

  In some cases, the environment in which the work robot is introduced is modeled in advance. In short, in order to facilitate the operation of the robot with respect to structures such as furniture and doors, they may be modeled in advance. This is because in order for the work robot to operate the structure, information on which part of the structure should be operated and how is necessary.

  However, the work of modeling such a surrounding environment three-dimensionally by CAD or the like is time-consuming even for an expert, and this work load cannot be ignored. Further, it may be necessary to perform modeling for each robot introduction environment, and the overall introduction cost / introduction burden can be significant. If the environment changes after the introduction of the robot (for example, furniture layout change, extension / renovation, etc.), the existing three-dimensional model needs to be appropriately corrected, and the maintenance burden may increase.

  As is clear from the above description, there is a strong demand to reduce the modeling burden of the work robot introduction environment.

  The operating environment model construction system according to the present invention generates geometric information indicating geometric characteristics of the operated object by adapting geometric primitives to a stereoscopic image obtained by measurement of the operated object, The position information indicating the spatial position of the operation target that is the generation target is acquired, and the operation information indicating the operation content to be performed by the work robot on the operation target is acquired, the generated geometric information and the acquired The position information is stored in association with the position information.

  The geometric characteristic of the object to be operated may be expressed by adaptation of the plurality of geometric primitives to the stereoscopic image.

  The geometric information may include information indicating shapes of the plurality of geometric primitives adapted to the stereoscopic image.

  The geometric information may include information indicating positions of the plurality of geometric primitives adapted to the stereoscopic image.

  The operation information preferably includes at least information for indicating a moving direction of the operated object.

  The acquisition of the position information and the generation of the geometric information may be executed in accordance with an assignment by a user of operation contents for the object to be operated.

  The operation information to be stored may be operation information selected by the user from a plurality of operation information prepared in advance according to the type of operation content.

  It is preferable to generate an image in which the geometric primitive is applied to the object to be operated.

  The content of the operation information is preferably configured to be changeable.

  Extracting the features around the object to be operated, storing the spatial positions of the features in advance, extracting the features again, and spatially fitting the registered features to the extracted features It is preferable to display information having a relative position with respect to the pre-registered feature at a position corresponding to the result of the fitting.

  The operating environment model construction system according to any one of the above includes a terminal device that can be carried by a user, and a server that is communicably connected to the terminal device, and the terminal device includes the position Information acquisition may be performed, and the server may generate the geometric information.

  A system according to the present invention is a system including the above-described operation environment model construction system and a work robot configured to be communicable with the operation environment model construction system, wherein the work robot includes the manipulation target Position information for specifying the position of the body is transmitted to the server, and the server transmits the operation information associated with the position information received from the work robot to the terminal device. The terminal device requests the user to input a work instruction to the work robot based on the operation information transmitted from the server.

  In the operation environment model construction method according to the present invention, the system generates geometric information indicating the geometric characteristics of the operated object by adapting a geometric primitive to a stereoscopic image obtained by measurement of the operated object. The system acquires position information indicating a spatial position of the object to be operated, which is an object for generating the geometric information, and the system generates operation information indicating an operation content to be performed by a work robot. The geometric information and the acquired position information are stored in association with each other.

  According to the present invention, it is possible to reduce the modeling burden of the work robot introduction environment.

1 is a schematic block diagram of a system according to a first exemplary embodiment. 1 is a schematic diagram of a terminal device according to a first embodiment; FIG. 3 is a schematic diagram illustrating a usage state of the terminal device according to the first embodiment; FIG. 3 is a schematic diagram illustrating an operation state of the work robot according to the first embodiment. 1 is a schematic block diagram of an environment modeling server according to a first embodiment. 1 is a schematic block diagram of a terminal device according to a first embodiment; 3 is a schematic block diagram of a 3D image acquisition unit according to Embodiment 1. FIG. FIG. 3 is a schematic block diagram of a position / orientation information acquisition unit incorporated in the terminal device according to the first exemplary embodiment; 1 is a schematic block diagram of a work robot according to a first embodiment. FIG. 2 is a schematic block diagram of a position / orientation information acquisition unit incorporated in the work robot according to the first exemplary embodiment. 3 is a flowchart showing a schematic operation of the system according to the first exemplary embodiment; 3 is a schematic flowchart showing an operation tag arrangement operation according to the first exemplary embodiment; FIG. 6 is a diagram showing a display image at the time of operation tag placement operation according to the first exemplary embodiment; FIG. 6 is a diagram showing a display image at the time of operation tag placement operation according to the first exemplary embodiment; FIG. 6 is a diagram showing a display image at the time of operation tag placement operation according to the first exemplary embodiment; FIG. 6 is a diagram showing a display image at the time of operation tag placement operation according to the first exemplary embodiment; FIG. 6 is a diagram showing a display image at the time of operation tag placement operation according to the first exemplary embodiment; FIG. 6 is a diagram showing operation icons displayed as images according to the first embodiment. It is a schematic diagram which shows the data structure of the operation tag stored in the operation tag database concerning Embodiment 1. FIG. 6 is a schematic diagram illustrating a relationship between geometric primitives and instruction operation contents according to the first embodiment; FIG. 10 is a schematic diagram illustrating a modification example of the combination of the geometric primitive and the instruction operation content according to the first embodiment. 4 is a schematic flowchart showing a procedure for changing operation information according to the first exemplary embodiment; FIG. 6 is a diagram showing a display image when changing operation information according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when changing operation information according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when changing operation information according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when changing operation information according to the first exemplary embodiment. FIG. 3 is a virtual diagram schematically showing an arrangement state of operation tags according to the first exemplary embodiment. 3 is a schematic flowchart showing an operation instruction procedure for the work robot according to the first embodiment; FIG. 6 is a diagram showing a display image when an operation instruction is given to the work robot according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when an operation instruction is given to the work robot according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when an operation instruction is given to the work robot according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when an operation instruction is given to the work robot according to the first exemplary embodiment. FIG. 6 is a diagram showing a display image when an operation instruction is given to the work robot according to the first exemplary embodiment. FIG. 10 is a diagram showing variations of geometric primitives according to the first exemplary embodiment. It is a schematic block diagram of the environment modeling server concerning Embodiment 2. FIG. It is a figure which shows the extraction state of the feature point around the to-be-operated body concerning Embodiment 2. FIG. It is a schematic diagram which shows the data structure of the operation tag stored in the operation tag database concerning Embodiment 2. 6 is a schematic flowchart showing a procedure for registering feature points according to the second embodiment; 9 is a schematic flowchart showing a procedure for displaying and correcting the position of an operation tag according to a second embodiment; FIG. 10 is another diagram showing an extraction state of feature points around the operated body according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each embodiment is not individually independent, but can be appropriately combined, and a synergistic effect based on the combination can also be claimed. The same elements are denoted by the same reference numerals, and redundant description is omitted.

Embodiment 1
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram of the system. FIG. 2 is a schematic diagram of the terminal device. FIG. 3 is a diagram illustrating a usage state of the terminal device. FIG. 4 is a diagram illustrating an operation state of the work robot. FIG. 5 is a schematic block diagram of the environment modeling server. FIG. 6 is a schematic block diagram of the terminal device. FIG. 7 is a schematic block diagram of the 3D image acquisition unit. FIG. 8 is a schematic block diagram of the position and orientation information acquisition unit. FIG. 9 is a schematic block diagram of the work robot. FIG. 10 is a schematic block diagram of the position / orientation information acquisition unit incorporated in the work robot. FIG. 11 is a flowchart showing a schematic operation of the system. FIG. 12 is a schematic flowchart showing an operation tag placement operation. 13 to 17 are diagrams showing display images at the time of operation tag placement operation. FIG. 18 is a diagram illustrating operation icons. FIG. 19 is a schematic diagram showing a data structure of operation tags stored in the operation tag database. FIG. 20 is a schematic diagram showing the relationship between geometric primitives and instruction operation contents. FIG. 21 is a schematic diagram illustrating a modification of the combination of the geometric primitive and the instruction operation content. FIG. 22 is a schematic flowchart showing a procedure for changing operation information. 23 to 26 are diagrams showing display images when the operation information is changed. FIG. 27 is a virtual diagram schematically showing the arrangement state of operation tags. FIG. 28 is a schematic flowchart showing an operation instruction procedure for the work robot. FIG. 29 to FIG. 33 are diagrams showing display images when an operation instruction is given to the work robot. FIG. 34 is a diagram showing variations of geometric primitives.

  As shown in FIG. 1, the system 100 is constructed from an environment modeling server 10, a terminal device 30, and a work robot 40 that are communicably connected to each other via a wireless network 20. A person skilled in the art can make system changes as appropriate. For example, it is possible to transfer some functions of the terminal device 30 and the work robot 40 to the environment modeling server 10 (and vice versa).

  The environment modeling server 10 is a general server device, and is configured to be able to execute advanced arithmetic processing and function as a database server. The terminal device 30 is composed of a computer having a general-purpose processing function. The work robot 40 also incorporates a computer having a general-purpose processing function. As is well known, the computer includes a CPU (Central Processing Unit), a hard disk, a memory, a motherboard, a daughter board, an external interface, and the like. The CPU executes a program stored in a storage area such as a hard disk or a memory. Various functions are implemented by such program processing by the CPU.

  As shown in FIG. 2, the terminal device 30 is configured to have a size that can be carried by the user. A display unit 30c is provided on the front surface 30a of the terminal device 30, and a three-dimensional measurement unit 30d and an image acquisition unit 30e are provided on the back surface 30b. A touch panel device is incorporated in the display unit 30c of the terminal device 30 (see also FIG. 3). The user 210 inputs an instruction to the terminal device 30 by designating an icon displayed on the touch panel of the terminal device 30 with a finger. The image acquisition unit 30e corresponds to an image acquisition unit 38b described later. The three-dimensional measurement unit 30d corresponds to a depth information acquisition unit 38c described later.

  As schematically shown in FIG. 3, the user 210 has the terminal device 30 and sequentially photographs the structures (objects to be operated) 90 a to 90 c fixed in the environment where the work robot 40 is introduced. As described above, an operation tag is virtually set for the stereoscopic image acquired by the terminal device 30. The user 210 can take an image of an arbitrary structure by making the position / posture of the terminal device 30 appropriate. Therefore, the target structure can be measured quickly and intuitively. In this case, the problem of blind spots as in the case of a fixed camera is reduced.

  The operation tag setting method executed by the system 100 will be briefly described as follows. First, "position data (position information)" to indicate the position of the structure, "structure data (geometry information)" to indicate the geometric shape of the structure, and the work robot should perform on the structure “Operation content data (operation information)” for specifying the content is generated and stored in association with the operation tag ID. In this way, the operation content to be performed by the work robot 40 for each structure existing in the introduction space is preset. Each operation tag is identified by an operation tag ID and includes the above-described data. In the present application, a data group associated with an operation tag ID may be simply referred to as an “operation tag”. This is a problem for convenience of explanation, and the technical scope of the present invention is not allowed to be narrowly interpreted based on this point.

  The above-described structure data is preferably obtained by applying geometric primitives to a stereoscopic image obtained by three-dimensional measurement of a structure. As a result, the system can grasp the actual shape of the structure to be operated by the work robot 40, and the operation accuracy of the structure by the work robot 40 is improved. The above-mentioned position data is acquired by GPS positioning, positioning using wireless LAN access as a positioning reference, positioning using a mobile phone base station, or the like. The operation content data is selected from pre-typed data.

  As exemplarily shown in FIG. 3, a structure 90 such as a chest 90 a, a container case 90 b, and a door 90 c is provided in the environment where the work robot is introduced. The drawer 90a is provided with a drawer that is opened and closed by a handle 91a and a drawer that is opened and closed by a handle 91b. The container case 90b is provided with a door that is opened and closed by a handle 91c. The door 90c is opened and closed by a handle 91d.

  As schematically shown in FIG. 4, the work robot 40 performs various operations in the introduction environment. For example, the work robot 40 pulls the handle 91a of the chest 90a toward the front, and images the contents of the upper drawer provided in the chest 90a. The work robot 40 pulls the handle 91d of the door 90c forward, and moves to the room on the other side of the door 90c. The operation of each structure by the work robot 40 is executed by the operation of the terminal device 30 by the user.

  The operation instruction for the work robot 40 is executed as follows when viewed from the user side. The terminal device 30 presents to the user an image in which the operation tag icon is superimposed on the acquired image of the work robot 40. The user touches the operation tag icon presented by the terminal device 30 with a finger, and specifies the operation content to be guided by the finger touch. An operation instruction by the user is transferred from the terminal device 30 to the work robot 40, and the work robot 40 executes a desired operation. Since the actual shape of the structure can be grasped by applying the geometric primitive to the three-dimensional measurement data of the structure, the operation of the structure by the work robot 40 can be executed with high accuracy. Further, the operation instruction transmitted to the work robot 40 is specified in relation to the geometric primitive fitted to the data obtained from the structure. Thereby, it becomes possible to simply show the operation content of the work robot 40 for the structure.

  As shown in FIG. 4, the work robot 40 includes a vehicle part 40a, a body part 40b, a head part 40c, and arm parts 40d and 40e as appearance elements. The arm portion 40d of the work robot 40 is configured by jointly connecting link portions 40d1 and 40d2 and a hand portion 40d3. The arm 40e of the work robot 40 is the same as the arm 40d. The vehicle unit 40a is provided with a set of wheels 40a1 that are individually rotationally controlled. The vehicle unit 40a incorporates a control mechanism that individually controls a set of wheels. A control device for controlling various operations performed by the work robot 40 is incorporated in the body portion 40b. Various sensing devices and the like are incorporated in the head 40c.

  For example, the work robot 40 is configured to autonomously determine a movement route to the destination and autonomously move along the determined movement route. The work robot 40 is configured to be able to detect its current posture based on the sensing result detected by the sensing device. The work robot 40 is configured to be able to change to a desired posture autonomously or in response to a command from a user. The operation of the structure by the work robot 40 is executed by the posture change of the work robot 40. The posture control of the work robot 40 is executed by making the operation amount of the actuator, the motor, etc. incorporated in the work robot 40 appropriate.

  Again, in this embodiment, an operation tag is virtually attached to each structure. The operation content of the work robot made for each structure is executed based on an operation tag set in advance for the structure to be operated. More preferably, the operation tag is position data (position information) indicating the position of the object to be operated, and structure data obtained by adapting the geometric primitive to the stereoscopic image obtained from the three-dimensional measurement of the object to be operated. (Geometric information) and operation content data (operation content information) indicating the operation content to be performed by the work robot on the operated body. A user can construct an appropriate operation environment model by virtually setting operation tags for the structure, and can reduce the burden of modeling the environment in which the work robot 40 is introduced. become.

  Virtually assigning an operation tag to a structure means setting an operation tag in association with the structure in the virtual space, and giving information directly to the structure Does not mean. Management of operation tags associated with a structure as information is executed by the system 100 (in this example, the environment modeling server 10).

  The information for specifying the structure (information for specifying the position, shape, etc. of the structure) is repeatedly added and stored with the operation content to be performed by the work robot 40 for the structure. Thus, an operation tag can be virtually attached to each structure. When causing the work robot 40 to perform an actual work, the system 100 preliminarily applies the structure to be operated by the work robot 40 based on data (position data, posture data, etc.) obtained by the work robot 40. Present the given operation tag to the user. The user is presented with the action content that the work robot can perform. The user causes the work robot 40 to execute a predetermined operation by operating the terminal device 30.

  Details of the operation of the system 100 will be described with reference to FIGS. As described above, the specific configuration of the system is arbitrary. The terminal device 30 may be handled as a simple input / output device, and the environment modeling server 10 may be caused to execute most of the arithmetic processing functions executed by the terminal device 30. Although the operation tag is set using the terminal device 30, the operation tag may be set using the work robot 40. In this case, it is possible to avoid the necessity of providing the terminal device 30 with a sensing mechanism such as a three-dimensional measuring unit or a position / orientation measuring unit.

  As illustrated in FIG. 5, the environment modeling server 10 includes a communication unit 11, a control unit 12, an operation tag database 13, an operation tag generation unit 14, and a fitting execution unit 15. The communication unit 11 is wirelessly connected to the wireless network 20. The control unit 12 is connected to the communication unit 11, the operation tag database 13, and the operation tag generation unit 14. The operation tag generation unit 14 is connected to the fitting execution unit 15.

  The communication unit 11 communicates with the terminal device 30 and the work robot 40 via the wireless network 20. The communication unit 11 is configured in accordance with a technical standard. The control unit 12 performs communication control, database access control, memory access control, sequence control, calculation control, data transfer control, data compression / decompression control, and the like. The operation tag database 13 is a unit for managing data and incorporates a database controller. The operation tag generator 14 generates operation tags stored in the operation tag database. The fitting execution unit 15 performs fitting of geometric primitives registered in advance on the three-dimensional data transferred from the terminal device 30 and the work robot 40.

  As illustrated in FIG. 6, the terminal device 30 includes a communication unit 31, a control unit 32, a display image generation unit 33 (a menu generation unit 33a, a virtual image generation unit 33b, and a composite image generation unit 33c), a display unit 36, and an instruction input. A unit 37, a 3D image acquisition unit 38, and a position and orientation information acquisition unit 39. The communication unit 31 is wirelessly connected to the wireless network 20. The control unit 32 is connected to the communication unit 31, the display image generation unit 33, the display unit 36, the instruction input unit 37, the 3D image acquisition unit 38, and the position / orientation information acquisition unit 39.

  The communication unit 31 communicates with the environment modeling server 10 and the work robot 40 via the wireless network 20. The communication unit 31 is configured in accordance with a technical standard. The control unit 32 performs communication control, memory access control, sequence control, calculation control, data transfer control, data compression / decompression control, and the like. The display image generation unit 33 generates an image to be generated by the display unit 36. The menu generation unit 33a generates display items such as menus. The virtual image generation unit 33b generates a virtual display that does not exist. The composite image generation unit 33c executes an image composition process. The display unit 36 is an image display unit configured from a liquid crystal display or the like. The instruction input unit 37 is configured by an instruction input position detection device or the like stored in a touch panel display. The 3D image acquisition unit 38 acquires the depth information of the image in addition to the two-dimensional image showing the surrounding environment. The position and orientation information acquisition unit 39 acquires position and orientation information of the terminal device 30.

  The specific configuration of the 3D image acquisition unit 38 is arbitrary. For example, images may be acquired by two cameras arranged at different positions, and depth information may be calculated based on two images acquired from different viewpoints. Depth information may be obtained based on the principle of TOF (Time Of Flight). Alternatively, the depth information may be acquired based on detection of reflected light obtained by projecting coded light. As illustrated in FIG. 7, the 3D image acquisition unit 38 includes a control unit 38a, an image acquisition unit 38b, a depth information acquisition unit 38c, and a depth calculation unit 38d. The control unit 38a is connected to the image acquisition unit 38b, the depth information acquisition unit 38c, and the depth calculation unit 38d. The depth information acquisition unit 38c is connected to the depth calculation unit 38d. The control unit 38a is connected to the control unit 32.

  The control unit 38a controls operations of the image acquisition unit 38b, the depth information acquisition unit 38c, and the depth calculation unit 38d. The image acquisition unit 38b is preferably configured by a camera. The camera includes a CMOS sensor (imaging device), a lens (optical system) that forms a subject image on the CMOS sensor, a mechanism (lens driving device) that displaces the lens, a functional circuit that processes output data of the CMOS sensor, and the like. Consists of The image data acquired by the image acquisition unit 38b is transmitted to the display unit 36 via the bus. The depth information acquisition unit 38c includes a pattern light projecting unit and a pattern light receiving unit. The depth calculation unit 38d calculates the depth information based on the time measurement between the pattern light reception times reflected from the pattern light projection time. Alternatively, the depth information may be calculated based on the stereo principle based on the pattern light projection position and the parallax of the camera.

  The specific configuration of the position and orientation information acquisition unit 39 is arbitrary. For example, the position and orientation information acquisition unit 39 calculates the current position based on the reception strength of signals transmitted from a plurality of transmission sources set in the introduction environment. In recent years, it is often used to calculate the current position based on GPS signals received from satellites. Therefore, the current position may be acquired using this technique. The posture information can be obtained from the output of a gyro sensor or the like built in the main body.

  As exemplarily shown in FIG. 8, the position / orientation information acquisition unit 39 includes a control unit 39a, a GPS signal reception unit 39b, an acceleration detection unit 39c, and a geomagnetism detection unit 39d. The controller 39a is connected to the GPS signal receiver 39b, the acceleration detector 39c, and the geomagnetism detector 39d. The control unit 39a is connected to the control unit 32.

  The control unit 39a controls the GPS signal reception unit 39b, the acceleration detection unit 39c, and the geomagnetism detection unit 39d, and performs arithmetic processing on data input from each unit. The GPS signal receiving unit 39b receives a GPS signal. The acceleration detector 39c detects the acceleration in the triaxial direction. The geomagnetism detector 39d detects geomagnetism.

  As illustrated in FIG. 9, the work robot 40 includes a communication unit 41, a control unit 42, a 3D image acquisition unit 43, a position and orientation information acquisition unit 44, and a drive control unit 45. The communication unit 31 is wirelessly connected to the wireless network 20. The control unit 42 is connected to the communication unit 41, the 3D image acquisition unit 43, the position / orientation information acquisition unit 44, and the drive control unit 45.

  The communication unit 31 communicates with the environment modeling server 10 and the terminal device 30 via the wireless network 20. The communication unit 41 is configured in accordance with a technical standard. The control unit 42 performs communication control, memory access control, sequence control, calculation control, data transfer control, data compression / decompression control, and the like. The 3D image acquisition unit 43 acquires the depth information of the image in addition to the two-dimensional image showing the surrounding environment. The position and orientation information acquisition unit 44 acquires position and orientation information of the work robot 40. The drive control unit 45 controls a drive mechanism built in the work robot 40.

  As illustrated in FIG. 10, the position / orientation information acquisition unit 44 includes a control unit 44a, a GPS signal reception unit 44b, an acceleration detection unit 44c, a geomagnetism detection unit 44d, and a joint axis angle detection unit 44e. The controller 44a is connected to the GPS signal receiver 44b, the acceleration detector 44c, the geomagnetism detector 44d, and the joint axis angle detector 44e. The control unit 44a is connected to the control unit 32. The joint axis angle detection unit 44e detects each rotation between links by using a rotary encoder or the like. As described above, the work robot 40 includes various sensing devices necessary for detecting the current posture of the work robot 40.

  The specific operation contents of the elements disclosed in FIGS. 5 to 10 will be further clarified from the following description.

  As illustrated in FIG. 11, the system 100 constructs an operation tag database, and then performs robot control using the operation tags stored in the operation tag database.

  With reference to FIG. 12, the procedure for arranging operation tags will be described. This will be described with reference to FIGS. 13 to 20 as appropriate.

  First, an object to be operated is displayed (S100). Specifically, as schematically shown in FIG. 3, the user holds the terminal device 30 in the introduction space of the work robot 40 and photographs a structure (for example, the handle 91 a of the chest 90 a). The user photographs the handle 91a by the terminal device 30 by setting the terminal device 30 to an appropriate position. The display unit 36 of the terminal device 30 displays a handle 91a as schematically shown in FIG.

  Next, a capture operation is executed (S101). Specifically, the user touches the camera icon displayed on the display unit 36 of the terminal device 30 with a finger, and instructs the terminal device 30 to execute a capture operation.

  In response to the input of the capture execution instruction, the terminal device 30 performs the same steps of (1) capturing an image of the operated object, (2) three-dimensional measurement of the operated object, and (3) acquiring the position and orientation data of the terminal device. Run at time. Note that the input of the capture operation execution instruction is transmitted to the control unit 32 via the instruction input unit 37. The control unit 32 instructs the 3D image acquisition unit 38 and the position / orientation information acquisition unit 39 to perform a data acquisition operation. A specific method of three-dimensional measurement is arbitrary. For example, three-dimensional measurement is performed by one or more cameras and a pattern light projector. In another example, three-dimensional measurement is performed by a stereo image measurement device using two or more cameras. In another example, three-dimensional measurement is performed by a laser range scanner.

  The image acquisition unit 38b holds the current acquired image according to the input of the capture execution instruction.

  The depth information acquisition unit 38c acquires depth information in the plane of the captured image in response to an input of a capture execution instruction. For example, depth information is acquired by an optical method. The depth information within the irradiation range of the pattern light is acquired by measuring the time from the emission of the pattern light to the detection of the return light that is reflected by the object and returns. The depth information acquisition method is arbitrary and should not be limited to this example.

  When acquiring depth information by an optical method, the following configuration can be employed. The depth information acquisition unit 38c is a sensing unit that measures the time from emission of pattern light to reception of return light. The depth calculation unit 38d is an arithmetic unit that calculates a distance from time information output from the sensing unit.

  The position / orientation information acquisition unit 39 acquires position / orientation information of the terminal device 30 in response to an input of a capture execution instruction. The GPS signal receiving unit 39b receives a GPS signal. The acceleration detector 39c detects the acceleration in the triaxial direction. The geomagnetism detector 39d detects geomagnetism. The control unit 39a calculates the current position of the terminal device 30 from the GPS signal. The control part 39a calculates the value which shows the attitude | position state from the perpendicular | vertical of the terminal device 30 from the acceleration of a triaxial direction. The control unit 39a calculates the attitude of the terminal device 30 in the horizontal plane from the detected geomagnetism value. The control unit 39a calculates the posture of the terminal from these vertical and horizontal postures.

  The (1) captured image, (2) three-dimensional measurement data of the object to be operated, and (3) the position and orientation data of the terminal device acquired in response to the input of the capture operation execution instruction are temporarily stored in a buffer or the like. The This buffer is provided in the control unit 32, for example.

  Next, an icon selection image is generated / displayed (S102). Specifically, the control unit 32 instructs the display image generation unit 33 to generate a display image obtained by adding the operation icon selection image to the capture image. In response to this, the display image generation unit 33 instructs the menu generation unit 33a to generate an icon selection image, and instructs the composite image generation unit 33c to combine the captured image and the icon selection image. The composite image generated by the composite image generation unit 33c is displayed by the display unit 36. Thereby, the terminal device 30 displays a screen schematically shown in FIG.

  It is assumed that the icons schematically shown in FIG. 18 are displayed on the icons indicating the actions A to D disclosed in FIG. The icon shown in FIG. 18A is an icon indicating an operation of pushing / pulling the handle along a straight line. The icon shown in FIG. 18B is an icon indicating an operation of pushing / pulling the handle along the curve. The icon shown in FIG. 18C is an icon indicating an operation of rotating the handle along a straight line and subsequently pushing / pulling the handle along the curve. The icon shown in FIG. 18D is an icon indicating an operation of pressing / releasing a button along a straight line. The icon disclosed in FIG. 18 indicates the operation content that the work robot should perform on the structure.

  The specific image generation mode of the display image generation unit 33 is arbitrary. For example, the menu generation unit 33a generates an icon selection image to be displayed in the display area R10 illustrated in FIG. The composite image generation unit 33c generates a composite image based on the image generated by the menu generation unit 33a and the captured image. In addition, in display area | region R10, R20 shown in FIG. 14, you may display the icon for showing / guiding another process to a user suitably. Alternatively, the display area R20 may be displayed as the entire screen, and the display area R10 may be transparently displayed.

  Next, an operation tag icon is selected (S103). Specifically, as schematically shown in FIG. 15, the user selectively moves an appropriate icon indicating the operation to be performed by the work robot 40 onto the structure displayed on the display unit 36. The structure displayed in FIG. 15 is a handle 91a of the chase 90a of FIG. Accordingly, the user selects the icon shown in FIG. 18A and moves it onto the handle 91a. In this way, operation information is associated with the captured structure. The selection of the icon by the user is realized by cooperative operation of the instruction input unit 37, the control unit 32, the display unit 36, and the like. Since the control technology of the touch panel sensor has been established, detailed operation description thereof is omitted.

  Next, fitting is executed (S104). Specifically, the terminal device 30 transmits data necessary for fitting to the environment modeling server 10. The environment modeling server 10 fits a geometric primitive to the three-dimensional measurement data of the operated object based on the data received from the terminal device 30.

  The terminal device 30 transmits data (capture image data, three-dimensional measurement data) acquired by the capture operation to the environment modeling server 10. Preferably, in order to shorten the fitting process of the environment modeling server 10, the terminal device 30 also transmits data indicating the position on the captured image where the operation icon is dropped to the environment modeling server 10.

  Based on the data received from the terminal device 30, the environment modeling server 10 fits geometric primitives to the three-dimensional measurement data of the operated object, and transmits the data obtained by the fitting to the terminal device 30. Data obtained by the fitting is data for designating the shape, position, and orientation of the geometric primitive. Based on the data supplied from the environment modeling server 10, the terminal device 30 synthesizes and displays an image in which geometric primitives are superimposed on the captured image, as schematically shown in FIG.

  The fitting execution unit 15 generates data indicating the three-dimensional shape of the structure using a geometric primitive pair (a set of a cylindrical primitive and a rectangular primitive). As described above, the above-described data (capture image data, three-dimensional measurement data, and data indicating the position on the capture image where the operation icon is dropped) are transferred from the terminal device 30 to the environment modeling server 10.

  The fitting execution unit 15 fits a geometric primitive to the depth information (three-dimensional measurement data) around the operation icon drop position based on the drop position of the operation icon on the captured image.

  The specific method of fitting is arbitrary. For example, the outline and surface shape of the structure can be obtained from the two-dimensional image and its depth information. A cylinder primitive is applied to the outline and surface shape of the structure to obtain specific data of the cylinder primitive. The specific data of the cylinder primitive includes data indicating the initial position and orientation of the cylinder primitive and data indicating the shape of the cylinder primitive. By such fitting, the position vector of the cylinder primitive (vector connecting the reference point to the center point of the cylinder primitive) and the attitude vector of the cylinder primitive (vector indicating the longitudinal direction / extension direction of the cylinder) are determined.

  Next, the surface shape of the surface to which the handle is attached is obtained from the two-dimensional image and its depth information. A rectangular primitive is applied to the surface to which the handle is attached, and specific data of the rectangular primitive is obtained. The specific data of the rectangle primitive includes data indicating the distance from the cylinder, data indicating the horizontal width of the rectangle, and data indicating the vertical width of the rectangle. In this way, data sufficient to specify the geometry of the object to be operated is acquired. In the case described above, the position and orientation of the rectangular primitive are grasped with reference to the position vector and orientation vector of the cylindrical primitive. The rectangular primitives are set so as to be arranged in parallel to the vertical direction.

  The specific mode of fitting is arbitrary. For example, a geometric primitive is applied to the surface shape of a structure extracted from three-dimensional measurement data, and a convergence point of the shape, position, and orientation of the geometric primitive is obtained using a predetermined algorithm. As a result, the shape, position, and orientation of the geometric primitive fitted to the structure can be calculated.

  Next, the fitting result is displayed (S105). Specifically, the above-described data obtained by fitting by the environment modeling server 10 is transferred to the terminal device 30 and processed by the terminal device 30. As a result, an image in which the geometric primitive is superimposed on the captured image is synthesized and displayed.

  Specifically, it is as follows. Data generated by the fitting execution unit 15 of the environment modeling server 10 is transferred to the display image generation unit 33 of the terminal device 30. The display image generation unit 33 causes the virtual image generation unit 33b to generate a virtual image in which a cylindrical primitive and a rectangular primitive are displayed based on the received data. The display image generation unit 33 causes the composite image generation unit 33c to synthesize an image obtained by adding a virtual image to the capture image. As a result, as shown in FIG. 16, an image in which a cylindrical primitive and a rectangular primitive are superimposed on a structure (handle 91a) is displayed.

  Next, it is determined whether or not the fitting is successful (S106). Specifically, the user determines whether or not the fitting state displayed on the terminal device 30 is appropriate, and inputs the determination result to the terminal device 30. As schematically shown in FIG. 16, an OK icon and an NG icon are displayed on the display unit 36 of the terminal device 30. The user inputs a fitting OK or NG determination result to the terminal device 30 by selecting one of the icons. In the case of fitting NG, the process returns to step S100, the structure to be captured is reset, and the same operation is executed.

  In the case of fitting OK, the operation tag is stored (S107). Specifically, the terminal device 30 transmits a signal instructing storage of the operation tag to the environment modeling server 10. In response to this signal, the operation tag generation unit 14 generates an operation tag having a predetermined data structure (described later with reference to FIG. 19). Next, the control unit 12 stores the generated operation tag in the operation tag database 13.

  As shown in FIG. 19A, the operation tag includes an ID, a reference position, an operation model, and a unique parameter. ID is an identification value for identifying each operation tag. The reference position indicates the reference position of the operation tag (the reference position is a representative point of the operation model such as the center of the handle, that is, the center of the cylindrical primitive, for example). The operation model corresponds to the operation icon schematically shown in FIG. 18, and indicates the classification of the operation content to be performed by the work robot 40 on the structure. The unique parameters are stored in a table different from that in FIG. In the unique parameter, a pointer for specifying the associated table is set.

  FIG. 19B shows the contents of the table indicated by the pointer PO1 shown in FIG. As shown in FIG. 19B, the unique parameter includes geometric primitive information and operation model information. The geometric primitive information includes cylindrical primitive information and rectangular primitive information. Specific data (position vector, posture vector, cylinder diameter, cylinder length) is associated with the cylinder primitive. The rectangular primitive is associated with the distance from the cylindrical primitive, the vertical width, and the horizontal width.

  The operation model is information that defines the operation content to be performed by the work robot 40 on the operated object. In the examples shown in FIGS. 13 to 17, the operation performed by the work robot 40 on the handle is an operation of pulling or returning the handle. The operation model information is composed of data necessary for specifying such an operation. Preferably, the operation content of the work robot 40 is defined by the trajectory and the operation parameters. The trajectory is defined by the initial position and the extraction vector. The operation parameter is defined by the maximum operation force. The operation operation by the work robot 40 is defined in the operation model. The operation content indicated by the operation model is determined by the type of operation model. Here, it is assumed that predetermined operation content is specified from the operation model type designation.

  The data shown in FIG. 19 can be schematically represented in FIG. The reference position of the operation tag corresponds to T1. The diameter cd and length dl of the cylindrical primitive, and the vertical width ra and horizontal width rb of the rectangular primitive are as shown in FIG. The interval r1 between the rectangular primitives from the cylindrical primitives is also as shown in FIG. The position vector of the cylinder primitive is specified by, for example, the center (geometric center) of the cylinder viewed from the reference position of the operation tag in the space. Note that the method for specifying the position and orientation of the cylindrical primitive is arbitrary, and should not be limited to this example. The initial position of the trajectory of the operation model is as indicated by the point SP, and the end point position is as indicated by the point EP. The initial position and end position of the trajectory are indicated by coordinate values viewed from the reference position. A vector L1 connecting the initial position SP and the end position EP indicates the operation direction of the operated object. The maximum operating force for the operation should be appropriately set according to the weight of the operated object.

  Next, the operation tag icon registered in the database is displayed (S108). As schematically illustrated in FIG. 17, the terminal device 30 stops displaying the geometric primitive and displays an operation tag icon indicating that an operation tag is assigned to the structure. Thereafter, the process returns to step S100 as appropriate, and the operation tag setting operation for other objects to be operated is repeated.

  After registering the operation tag in the operation tag database 13, the control unit 12 transfers data indicating the position of the operation tag to the terminal device 30. The display image generation unit 33 causes the virtual image generation unit 33b to generate an image in which the operation tag icon is arranged at a position specified from data indicating the position of the operation tag. The display image generation unit 33 causes the composite image generation unit 33c to perform a composite process on the captured image with the image on which the operation tag icon is displayed. The display unit 36 displays the composite image generated by the composite image generation unit 33c.

  In the above case, the geometric primitive is applied to the structure using the cylindrical primitive and the rectangular primitive. However, the type of the geometric primitive applied to the operation target is arbitrary.

  FIG. 21A shows a case where the operated body is a button. In this case, only rectangular primitives are used without using cylindrical primitives. As schematically shown in the figure, an operation action is defined by an operation start / end point, an operation vector, and the like.

  FIG. 21B shows a case where the operated body is a door. In this case, as in the above example, the cylinder primitive and the rectangle primitive are adapted to the door. Since the object to be operated is different, the posture of the cylindrical primitive is set to be different from the above example. Also, the operation is defined differently.

  With reference to the flowchart of FIG. 22, the property-change procedure of the operation tag set as described above will be described. This will be described with reference to FIGS. The operation tag property-change means changing a specific value or the like of the operation model specified from the operation tag. The specific values of the operation model are assigned appropriate default values when generating operation tags in order to save the time and effort to specify them. The content can be optimized.

  First, an object to be operated is displayed (S200). This operation is the same as S100 described above.

  Next, acquisition / transmission of position and orientation data is executed (S201). Specifically, the terminal device 30 acquires information indicating its current position and transmits it to the environment modeling server 10.

  Next, the operation tag is downloaded (S202). Specifically, the environment modeling server 10 specifies an operation tag located in the vicinity of the received position data, and transmits the specified operation tag to the terminal device 30. Preferably, the control unit 12 of the environment modeling server 10 causes the operation tag database 13 to output operation tags that exist within a predetermined range of position data received from the terminal device 30. The control unit 12 transmits the operation tag output from the operation tag database 13 to the terminal device 30 via the communication unit 11. As described above, data having the data structure shown in FIG. 19 may be simply described as “operation tag”.

  Next, a composite image is generated (S203). First, the terminal device 30 temporarily buffers the operation tag (ID, reference position, operation model, unique parameter (position / orientation parameter, shape parameter, operation model content of each primitive)) received from the environment modeling server 10. . The control unit 32 supplies position data indicating the position of the operation tag to the display image generation unit 33. The display image generation unit 33 instructs the virtual image generation unit 33b to generate a virtual image in which the operation tag icon is arranged at the position indicated by the position data. The display image generation unit 33 instructs the composite image generation unit 33c to generate a composite image based on the currently captured and displayed image and the virtual image generated by the virtual image generation unit 33b. The display unit 36 displays the synthesized image synthesized by the synthesized image generation unit 33c. In this way, as shown in FIG. 23, the operation tag icon is superimposed on the current view presented to the user by the terminal device 30.

  Next, it is determined whether or not the operation tag icon is pressed long (S204). The terminal device 30 detects whether or not the operation tag icon has been pressed by the user for a predetermined time or more by the operation of the touch panel by the user. For example, a time of about 2 seconds to 3 seconds is used as the threshold value. Specifically, based on the input from the instruction input unit 37, the control unit 32 determines whether or not the user's finger has been placed for a predetermined time or more at the location where the operation tag icon is displayed.

  If there is a long press of the operation tag icon, the property display of the operation tag is executed (S205). Specifically, when it is detected that the user's finger is placed on the operation tag icon for a predetermined time or longer, the terminal device 30 displays a property associated with the selected operation tag. In response to detection of the long press, the control unit 32 instructs the display image generation unit 33 to generate an image with the property-screen added. The display image generation unit 33 instructs the menu generation unit 33a to generate a property-screen, and causes the composite image generation unit 33c to generate an image with the property-screen added to the current captured image.

  The specific information displayed as the property is preferably the content of the operation model or information calculated therefrom. For example, as shown in FIG. 24, the opening width calculated from the unique parameters (trajectory: initial position; end point position) shown in FIG. 19 is displayed. Further, the unique parameter (maximum operating force) of the operation model shown in FIG. 19 is displayed.

  Next, a property information change instruction is input (S206). For example, when the terminal device 30 detects that the user's finger has touched the display position of the maximum operating force, the terminal device 30 presents a value change screen as shown in FIG. Specifically, the terminal device 30 lights up and displays a display frame of the maximum operating force and displays a keyboard for inputting a value. These display controls are executed, for example, when the control unit 32 gives an appropriate command to the display image generation unit 33. As an example, the display image generation unit 33 instructs the menu generation unit 33a to generate a keyboard display, and instructs the composite image generation unit 33c to incorporate the keyboard display into the current view. As a result, a parameter change can be input. The user touches the displayed keyboard to change the value of the maximum operating force. The changed value is temporarily held by the control unit 32. Depending on the keyboard input by the user, the screen of the display unit 36 is displayed as shown in FIGS.

  Next, preparation for reflecting the changed value is executed (S207). Specifically, the terminal device 30 transmits an operation tag ID, a change value, and the like to the environment modeling server 10. The environment modeling server 10 stores a new change value in the operation tag database 13 based on the received data from the terminal device 30. The actual rewriting process is performed after the update is confirmed. Therefore, the changed value is temporarily buffered in the free space of the database.

  Next, confirmation / execution of the change is performed (S208). Specifically, the environment modeling server 10 transmits a signal indicating completion of update preparation to the terminal device 30. The terminal device 30 displays the updated value input by the user to the user in response to the signal reception from the environment modeling server 10. For example, the terminal device 30 lights up and displays the frame shown in FIG. 26 in which the change has been made. If the updated value is correct, the user touches the part other than the display part of the operation property. This confirms the update. Specifically, the control unit 12 detects the finger touch detection in the part other than the display part of the operation property from the output of the instruction input unit 37, and notifies the environment modeling server 10 that the updated value is approved. Notice. In response to this notification, the environment modeling server 10 executes a process of actually rewriting the value. In this way, the property value (in this example, the maximum operating force shown in FIG. 19B) is updated. When there is an error in the update contents, the user repeats the change work again.

  By repeating the above-described operation, an operation tag is virtually attached to the structure arranged in the introduction space of the work robot 40 as schematically shown in FIG.

  With reference to the flowchart of FIG. 28, a procedure for operating the work robot 40 using the operation tags stored in the operation tag database 13 of the environment modeling server 10 will be described. This will be described with reference to FIGS. The images shown in FIGS. 29 to 33 are displayed on the terminal device 30, but are generated based on the image acquired by the work robot 40. The flowchart shown in FIG. 28 is based on the premise that the work robot 40 is arranged in the introduction space and the user is viewing the acquired image of the work robot 40 through the terminal device 30. Note that these assumptions are merely one aspect of the present invention, and the technical scope of the present invention is not allowed to be narrowly interpreted based on this disclosure.

  First, the display of the operated object is executed (S300). The video acquired by the work robot 40 is transferred to the terminal device 30 via the wireless network 20 and displayed by the terminal device 30. As schematically illustrated in FIG. 9, the work robot 40 is configured to be able to supply an image acquired by the built-in camera to the terminal device 30 via a communication line. The data transfer between the work robot 40 and the terminal device 30 may be executed via the environment modeling server 10.

  Next, acquisition / transmission of position and orientation data is executed (S301). The work robot 40 acquires position and orientation data indicating the current position of the work robot 40 itself, and transmits this to the environment modeling server 10. Based on the current position of the work robot 40, the environment modeling server 10 identifies operation tags that exist around it.

  Next, the operation tag is downloaded (S302). The environment modeling server 10 transmits the identified operation tag to the terminal device 30 via the wireless network 20.

  Next, a composite image is generated (S303). Based on the operation tag downloaded from the environment modeling server 10, the terminal device 30 generates and displays a view in which the operation tag icon is incorporated with respect to the current view transferred from the work robot 40, as shown in FIG. (See FIG. 29).

  Next, it is determined whether or not an operation tag icon has been selected (S304). Based on the output of the instruction input unit 37, the terminal device 30 determines whether or not the user's finger has touched the display portion of the operation tag icon.

  When the user's finger touches the display portion of the operation tag icon, an operation menu is displayed (S305). The terminal device 30 displays an operation menu in response to a finger touch by the user (see FIG. 30). The content of the operation menu displayed by the terminal device 30 corresponds to the content of the operation model associated with the operation tag selected by the user. The motion model assigned to the handle 91a is the model shown in FIG. 18A, and information corresponding to the model (gripping / releasing, opening / closing) is displayed.

  Next, the operation content is selected (S306). The user touches an area where desired operation content is presented from the displayed operation menu. For example, the user designates an operation icon to be grabbed in the displayed operation menu (see FIG. 30). In response to this, the terminal device 30 detects the operation content selected by the user.

  Next, an operation content is instructed (S307). The terminal device 30 transmits the instruction content input by the user to the work robot 40 via the wireless network 20. Data transfer from the terminal device 30 to the work robot 40 may be executed via the environment modeling server 10 as described above.

  Next, the operation content is executed (S308). The work robot 40 executes the operation content according to the received operation instruction. The work robot 40 calculates the target posture of the work robot 40 in response to a command to grab, and changes the current posture to the target posture. As a result, as shown in FIG. 31, the work robot 40 is in a state of gripping the handle 91a.

  The posture change control of the work robot 40 is executed by the control unit 42 controlling the drive control unit 45. The drive control unit 45 actually drives and controls the actuator and the like according to the control instruction transmitted from the control unit 42.

  Next, the terminal device 30 determines whether or not there is a next operation instruction (S309). When there is a next operation instruction from the user, the terminal device 30 repeats steps S306 to S308 again.

  In the next cycle, an operation instruction is selected (S306). When the state shown in FIG. 31 is displayed on the terminal device 30, the user selects an operation icon of opening / closing. As schematically shown in FIG. 33, the terminal device 30 presents a vector for defining an opening / closing operation to the user. In order to instruct how far the handle 91a is to be opened, the user traces the vector with a finger from the portion where the operation tag is displayed. At the position where the movement of the handle 91a stops, the user releases his / her finger from the touch panel. Thereby, the terminal device 30 can detect the end point position of the movement of the handle 91a.

  Next, the operation content is transferred from the terminal device 30 to the work robot 40. Next, the work robot 40 performs work according to the received instruction content. By repeating such a cycle, the operation of opening the drawer for the chest is performed in multiple stages. Moreover, the operation | work which opens a drawer continuously can also be performed by transferring the drag operation before a user lifts a finger | toe from a touch panel to a work robot one by one.

  In the above description, the three-dimensional shape of the structure is grasped by using the cylindrical primitive and the rectangular primitive, but the type of the geometric primitive is arbitrary. FIG. 34 shows an example of geometric primitives. The specific shape of the geometric primitive is arbitrary, and a geometric shape illustrated in FIGS. 34A to 34F can be adopted. FIG. 34A shows a point primitive. FIG. 34B shows a line primitive. FIG. 34C shows a rectangular primitive. FIG. 34D shows a cylindrical primitive. FIG. 34E shows a three-dimensional primitive. FIG. 34 (f) shows a two-sided primitive.

  As is clear from the above description, in this embodiment, operation tags are virtually assigned to each structure, and the operation content of the work robot performed on each structure is the same as the operation target structure. And executed based on the preset operation tag. The operation tag includes data indicating the position of the operated object, data obtained by applying a geometric primitive to a stereoscopic image acquired by three-dimensional measurement of the operated object, and an operation to be performed on the operated object. Contains data indicating content. The setting of the operation tag itself can be arbitrarily performed by the user who purchased the system, so that the burden of modeling the environment in which the work robot is introduced can be greatly reduced. In addition, since the geometric characteristics of the operation target can be detected by applying the geometric primitive, the same kind of operation can be realized for various structures having different sizes, shapes, places, and the like.

Embodiment 2
A second embodiment of the present invention will be described with reference to FIGS. In the present embodiment, in addition to the procedure of the first embodiment, a geometric feature (feature) existing around the structure to which the operation tag is attached is extracted, and data for spatially specifying the geometric feature is manipulated. Store in the tag. The operation tag includes data for spatially specifying a registered geometric feature (hereinafter sometimes referred to as feature specifying data). Thereafter, when the operation tag is superimposed and displayed on the structure, a geometric feature registered in the past is fitted to the currently acquired geometric feature. Since the relative relationship of the registered position data is stored in the operation tag, it is possible to display the information to be displayed now by fitting the current geometric feature to the geometric feature registered in the past. The position is fixed. For example, when the operation tag is displayed in a superimposed manner, the operation target and the operation tag are displayed depending on an error at the time of position data acquisition (position and orientation of the terminal device 30) and an error at the time of display (position and orientation of the terminal device or robot). Deviation occurs. However, by including the spatial position of the geometric feature around the structure in the operation tag given to the structure, the information is displayed with the position of the information to be displayed corrected. It becomes possible. Moreover, it may be located on a different place from the time of data recording, such as a structure (such as a half-open door) that is movable by an operation, but on the track. In such a case, fitting can be performed efficiently by expanding the search range of the geometric feature along the trajectory from the recorded tag position. This makes it possible to present a more accurate display to the user. Moreover, it becomes possible to absorb the fluctuation | variation of the position of a structure. Note that the type of feature to be extracted is arbitrary, and is not limited to a geometric feature, and may be the color of an object existing around the feature.

  As illustrated in FIG. 35, the environment modeling server 10 includes a peripheral feature extraction unit 16 and a feature fitting execution unit 17. As schematically shown in FIG. 36, the peripheral feature extraction unit 16 performs any kind and number of geometric features SL1 to SL4 between the handle 91a and the attachment surface of the handle 91a by image processing on the captured image. To extract. As schematically shown in FIG. 37, the peripheral feature extraction unit 16 stores the geometric feature position extracted in the captured image in the operation tag database. The geometric features SL1 and SL3 are specified in position, posture, diameter, and length by a cylindrical surface. In the geometric features SL2 and SL4, the center, the radius, the inclination, and the start point end point angle are specified by the arc edge. The operation tag is configured to be able to store pixel data of a characteristic image portion as a characteristic image. In this way, the spatial position of the extracted feature is stored in the operation tag at the past time point.

  Thereafter, when the operation feature is displayed on the structure, the peripheral feature extraction unit 16 similarly extracts the geometric features around the structure. The feature fitting execution unit 17 fits the past geometric feature to the geometric feature extracted this time. As a result, the display position of other position data stored in the operation tag (the reference position shown in FIG. 19A, the initial position shown in FIG. 19B, etc.) is more accurate than the actual operation target. Good positioning is possible. As a result, it is possible to present information to the user at an appropriate display position without depending on an error in position data registered in the operation tag, a position or orientation measurement error of the terminal or robot, or the like.

  As shown in FIG. 38, when fitting OK is determined in step S106, the feature is extracted as described above (S109). Next, as described above, the operation tag is stored including the data for specifying the spatial position of the extracted feature (S107).

  As shown in FIG. 39, when the information based on the position data included in the operation tag is presented to the terminal device 30, first, the surrounding geometric features are extracted (S201). Next, an operation tag to be displayed in the current view is searched (S202). Next, the geometric feature of the operation tag obtained by the search is fitted to the geometric feature obtained in S201 (S203). Next, the position and orientation of the geometric primitive of the operation tag is corrected from the recording data of the relative relationship between the position and orientation specified by the fitting and the geometric primitive of the operation tag, and information is displayed (S204). In this way, it is possible to further improve the information display position accuracy from the spatial position fitting of the old and new features. This makes it possible to present information at an appropriate position. The position data value in the operation tag in the database may be corrected based on the position obtained from the fitting. By repeating such correction work, it is possible to improve the accuracy of the position data in the operation tag.

  As schematically shown in FIG. 40, if the object to be operated is the one shown in FIG. 18D, the edge of the frame existing around it may be extracted as the feature point.

  Note that the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention. For example, the function of the terminal device 30 and the function of the work robot 40 may be partially shared. The specific shape of the terminal device 30 is arbitrary.

100 system

DESCRIPTION OF SYMBOLS 10 Environment modeling server 11 Communication part 12 Control part 13 Operation tag database 14 Operation tag production | generation part 15 Fitting execution part 16 Peripheral feature extraction part 17 Feature fitting execution part 20 Wireless network 30 Terminal device 31 Communication part 32 Control part 33 Display image generation part 36 Display unit 37 Instruction input unit 38 3D image acquisition unit 39 Position and orientation information acquisition unit

40 work robot 41 communication unit 42 control unit 43 image acquisition unit 44 position and orientation information acquisition unit 45 drive control unit 90 structure

Claims (13)

Applying geometric primitives to the stereoscopic image obtained by measurement of the operated object to generate geometric information indicating the geometric characteristics of the operated object,
Obtaining position information indicating a spatial position of the object to be operated which is a generation target of the geometric information;
An operation environment model construction system that stores operation information indicating operation contents to be performed on the object to be operated by a work robot in association with the generated geometric information and the acquired position information.   The operating environment model construction system according to claim 1, wherein the geometric characteristic of the object to be operated is expressed by adaptation of a plurality of the geometric primitives to the stereoscopic image.   The operation environment model construction system according to claim 1, wherein the geometric information includes information indicating shapes of the plurality of geometric primitives adapted to the stereoscopic image.   4. The operating environment model construction system according to claim 1, wherein the geometric information includes information indicating positions of a plurality of the geometric primitives adapted to the stereoscopic image. 5.   The operation environment model construction system according to any one of claims 1 to 4, wherein the operation information includes at least information for indicating a moving direction of the object to be operated.   The operation environment according to any one of claims 1 to 5, wherein the acquisition of the position information and the generation of the geometric information are executed in accordance with an assignment by a user of an operation content for the object to be operated. Model building system.   7. The operation information to be stored is operation information selected by a user from a plurality of operation information prepared in advance according to the type of operation content. The operating environment model construction system according to one item.   The operation environment model construction system according to claim 1, wherein an image in a state in which the geometric primitive is applied to the object to be operated is generated.   The operation environment model construction system according to any one of claims 1 to 8, wherein the contents of the operation information are configured to be changeable. Extracting a feature around the object to be operated, storing the spatial position of the feature in advance,
The feature is extracted again, the feature already registered with respect to the feature extracted this time is spatially fitted, and information on the relative position with respect to the pre-registered feature is determined according to the result of the fitting The operation environment model construction system according to claim 1, wherein the operation environment model construction system is displayed at a selected position. The operation environment model construction system according to any one of claims 1 to 10,
A terminal device that can be carried by the user;
A server communicably connected to the terminal device,
The terminal device executes the acquisition of the position information,
The server executes generation of the geometric information. An operation environment model construction system according to claim 11;
A work robot configured to be able to communicate with the operation environment model construction system;
A system comprising:
The work robot transmits position information for specifying the position of the operated body to the server,
The server transmits the operation information associated with the position information received from the work robot to the terminal device;
The terminal device requests the user to input a work instruction for the work robot based on the operation information transmitted from the server. The system generates geometric information indicating a geometric characteristic of the operated object by adapting a geometric primitive to a stereoscopic image obtained by measurement of the operated object,
The system acquires position information indicating a spatial position of the object to be operated, which is an object for generating the geometric information,
A method for constructing an operation environment model, wherein the system stores operation information indicating an operation content to be performed by a work robot in association with the generated geometric information and the acquired position information.

Images