JP7638136B2 - INFORMATION PROVIDING SERVER, INFORMATION PROVIDING SYSTEM, INFORMATION PROVIDING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an information providing server, an information providing system, an information providing method, and a program.

Conventionally, there is known technology for remotely controlling a robot to operate experimental equipment or devices using data sent from a monitor terminal or another terminal (see, for example, Patent Document 1). Conventionally, there is also known technology for displaying live images of classmates in an array in each person's evaluation area, or displaying the seat in the line of sight (see, for example, Patent Documents 2 and 3).

JP 2003-092749 A Japanese Patent Application Publication No. 10-171337 Patent No. 6007377

However, remote operation can sometimes create a sense of loneliness because it is a task performed by one person. Also, no consideration has been given to making the user feel as if other people are actually present during remote operation.

The aspects of the present invention have been made in consideration of these circumstances, and one of the objectives is to provide an information providing server, information providing system, information providing method, and program that can provide information that makes the user feel as if they are in the work space, even when operating remotely.

An information providing server, an information providing system, an information providing method, and a program according to the present invention employ the following configuration.
(1): An information providing server according to one embodiment of the present invention is an information providing server including a communication unit that communicates with one or more operating terminals via a network, an acquisition unit that acquires information relating to the behavior of an operator of the operating terminal, a management unit that manages the location of the operator in space, and an output control unit that arranges information based on information representing the operator and information relating to the behavior at the location of the operator in space managed by the management unit, and outputs the arranged information to the operating terminal.

(2): In the aspect of (1) above, the management unit generates an image in which the operator selects a placement location, transmits the generated image to the operation terminal, and manages the placement location of the operator based on the placement location selected using the image.

(3): In the above embodiment (1) or (2), the operation terminal includes a terminal that remotely controls one or more robot devices.

(4): In any one of the above aspects (1) to (3), the management unit acquires the position of the robot device operated by the operator in real space, and when outputting an image including the robot device placed at the acquired position to the operation terminal, superimposes an image representing the operator on the display area of the robot device.

(5): In the above aspects (1) to (3), the management unit manages the location of the operator by associating it with seat information that has been assigned in advance in the virtual space, and when an image showing the virtual space is output to the operation terminal, an image representing the operator is superimposed in association with the position of the image of the seat in the virtual space.

(6): In the aspect described in any one of (1) to (5) above, the output control unit acquires the voice of the operator, localizes the acquired voice as a sound image at the placement location, and outputs the voice.

(7): Another aspect of the present invention is an information provision system comprising an information provision server described in any one of (1) to (6) above, the one or more operation terminals, and one or more robot devices remotely operated by the one or more operation terminals.

(8): Another aspect of the present invention is an information provision method in which a computer communicates with one or more operation terminals via a network, acquires information related to the behavior of an operator of the operation terminal, manages the location of the operator in space, arranges information based on information representing the operator and information related to the behavior at the managed location of the operator in space, and outputs the arranged information to the operation terminal.

(9): Another aspect of the present invention is a program that causes a computer to communicate with one or more operation terminals via a network, acquire information related to the actions of an operator of the operation terminal, manage the location of the operator in space, arrange information based on information representing the operator and information related to the actions at the managed location of the operator in space, and output the arranged information to the operation terminal.

According to the above aspects (1) to (9), it is possible to provide information that makes the user feel as if they are in the work space, even during remote operation.

According to the above aspect (2), information based on information representing the operator and information related to the action can be placed in a position in space desired by the operator.

According to the above aspects (3) and (4), an image representing the operator is displayed at the position where the robot device 200 is displayed, thereby preventing the operator from feeling lonely.

According to the above aspect (5), even if the work location is in a virtual space, it is possible to grasp the surrounding situation and provide information that makes the user feel as if they are in the work space.

According to the above aspect (6), the operator can hear a sound that corresponds to the placement location, making the operator feel as if he or she is in a more realistic working space.

1 is a diagram illustrating an example of a configuration of an information providing system 1 according to an embodiment. FIG. 2 is a diagram illustrating an example of a configuration of an information providing server 100 according to the embodiment. FIG. 2 is a diagram for explaining the contents of user information 171. 10 is a diagram for explaining the contents of usage status information 172. FIG. FIG. 11 is a diagram for explaining the contents of intention estimation information 174. 10 is a diagram for explaining the contents of sensor information 175. FIG. FIG. 2 is a diagram illustrating an example of a configuration of a robot device 200 according to an embodiment. FIG. 2 is a diagram illustrating an example of a configuration of an operation terminal 300 according to the embodiment. FIG. 2 is a diagram for explaining remote control of the robot device 200 according to the embodiment. FIG. 13 is a diagram showing an example of an image IM10 for allowing a user to select a seat. FIG. 2 is a diagram showing an example of an image IM20 captured by the robot device 200. FIG. 13 is a diagram for explaining superimposing image information of a user on another robot device 200 (U4) existing in the vicinity. FIG. 13 is a diagram showing an example of an image IM40 from which images of a worker and a robot device 200 present in the vicinity have been deleted. 11 is a diagram showing an example in which information representing an operator and information based on information regarding an action are arranged in a virtual space. FIG. 2 is a sequence diagram for explaining a process executed by the information providing system 1 in the embodiment. FIG.

Below, with reference to the drawings, an embodiment of the information providing server, information providing system, information providing method, and program of the present invention will be described.

[System configuration]
FIG. 1 is a diagram showing an example of the configuration of an information providing system 1 according to an embodiment. The information providing system 1 shown in FIG. 1 includes, for example, an information providing server 100, a robot device 200, and an operation terminal 300. The information providing server 100 may include a robot manufacturer terminal 400 and a telecommunications company terminal 500 in addition to the above configuration. The information providing server 100 is communicatively connected to the robot device 200, the operation terminal 300, the robot manufacturer terminal 400, and the telecommunications company terminal 500 via a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a WAN (Wide Area Network), a LAN (Local Area Network), a provider device, a wireless base station, and the like. In the information providing system 1, the robot device 200 and the operation terminal 300 may each include one or more. In the example of FIG. 1, the robot devices 200A, 200B, and 200C are shown, but the number and types of the robot devices are not limited to this. 1 shows a plurality of operation terminals 300-1 to 300-n (n is a natural number equal to or greater than 2). In the following description, unless there is a need to distinguish between the robot device and the operation terminal, they will simply be referred to as the "robot device 200" and the "operation terminal 300." The information providing server 100 is an example of a "server."

The information providing server 100 manages one or more operation terminals 300 and one or more robotic devices 200 to be operated in association with each other. In this case, the information providing server 100 may associate one robotic device 200 with each operation terminal 300, may associate one robotic device 200 with multiple operation terminals 300, or may associate multiple robotic devices 200 with one operation terminal 300. The information providing server 100 acquires operation contents input from the operation terminal 300 and transmits operation control information corresponding to the operation contents to the robotic device 200, thereby remotely operating the robotic device 200. The information providing server 100 transmits information acquired by the robotic device 200 to the operation terminal 300. The information providing server 100 generates information to make the user feel as if he or she is in a working space (for example, around the robotic device 200 to be remotely operated) even during remote operation, and provides the generated information to the operation terminal 300.

The robot device 200 executes a predetermined operation based on control information transmitted from the information providing server 100 via the network NW. The robot device 200 includes a moving mechanism that can move by driving wheels, a cart, a crawler, a crane, or the like. In the example of FIG. 1, the robot device 200A is a robot with both arms that can move by at least a crane. The robot device 200B is a robot with both arms that can move by wheels provided at the bottom. The robot device 200C is a robot with both arms that can move by bipedal walking. The robot device 200 also includes an arm unit that performs tasks such as grasping, moving, and operating an object. The robot device 200 also includes a plurality of various sensors such as vibration, temperature, pressure, and tactile sensors, and transmits data detected by each sensor to the information providing server 100 at a predetermined cycle or at the timing when a request is received. The robot device 200 may also include a camera that captures the surroundings, a monitor that displays images, a speaker that outputs sound, a microphone that captures surrounding sounds, and the like.

The operation terminal 300 inputs the operation contents of the robot device 200 and notifies the operator of information acquired by the robot device 200. The operation terminal 300 can operate one or more robot devices 200. The operation terminal 300 may also operate some parts of the robot device 200 (e.g., the right arm or the left arm, etc.). The operation terminal 300 may also be provided with a visual device such as an HMD (head mounted display), an operation device, and an environmental sensor that detects the operator's movements. The operation terminal 300 is not limited to being provided within a company, but may be a terminal provided at the operator's home, or a terminal provided at a telework station installed in a station, department store, public facility, internet cafe, etc.

The robot manufacturer terminal 400 is a terminal used, for example, by the manufacturer or management company of the robot device 200. The robot manufacturer terminal 400, for example, updates the version of the software (program) installed in each robot device 200, acquires error information from the robot device 200, and performs remote operation or stop control of the robot device 200 based on the acquired error information.

The telecommunications company terminal 500 is, for example, a terminal used by a telecommunications carrier that manages communications on the network NW. The telecommunications company terminal 500 manages the amount of data communicated for remote operation of the information providing system 1, manages communication delays, etc., and performs maintenance of the communication environment within the system. The telecommunications company terminal 500 may also respond when a failure occurs on the network NW.

Next, the functions of the information providing server 100, the robot device 200, and the operation terminal 300 will be specifically described.
[Information server]
FIG. 2 is a diagram showing an example of the configuration of the information providing server 100 according to the embodiment. The information providing server 100 includes, for example, a communication unit 110, a communication control unit 120, an acquisition unit 130, a management unit 140, a control unit 150, and a storage unit 170. The communication control unit 120, the acquisition unit 130, the management unit 140, and the control unit 150 are each realized by a hardware processor such as a central processing unit (CPU) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as a large scale integration (LSI), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a graphics processing unit (GPU), or may be realized by cooperation between software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device equipped with a non-transient storage medium) such as a hard disk drive (HDD) or a flash memory equipped in the information providing server 100, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or a CD-ROM, and the storage medium may be installed in the HDD or flash memory equipped in the information providing server 100 by being attached to a drive device equipped in the information providing server 100. The information providing server 100 may be realized in a server device or a storage device incorporated in a cloud computing system. In this case, the functions of the information providing server 100 may be realized by a plurality of server devices and storage devices in the cloud computing system.

Under the control of the communication control unit 120, the communication unit 110 communicates with one or more robot devices 200, an operation terminal 300 that remotely operates one or more robot devices 200, and other external devices via the network NW. The communication control unit 120 controls the communication in the communication unit 110.

The communication control unit 120 controls the communication in the communication unit 110. For example, the communication control unit 120 receives the operation content of the robot device 200 input from the operation terminal 300, and transmits operation control information corresponding to the operation content to the robot device 200. The communication control unit 120 also receives information such as the detection results of sensors and camera images transmitted from the robot device 200, and transmits provided information based on the received information to the operation terminal.

The acquisition unit 130 acquires various information from the robot device 200 or the operation terminal 300. For example, the acquisition unit 130 acquires operation information for operating the robot device 200 from the operation terminal 300. The acquisition unit 130 also acquires information related to the operator's behavior from the operation terminal 300. The information related to the behavior includes, for example, information related to the operator's movements and information related to the operator's voice. The acquisition unit 130 also acquires information from sensors and cameras provided in the robot device 200. The acquisition unit 130 may also acquire information from an external device connected to the network NW.

The management unit 140 includes, for example, an authentication management unit 142, an information management unit 144, and a usage status management unit 146. The authentication management unit 142 manages the authentication and usage authority of the operator (system user) who operates the robot device 200. Furthermore, the authentication management unit 142 associates and manages the robot device 200 operated by the operation terminal 300 based on the authentication result.

The information management unit 144 manages information available to each user. For example, the information management unit 144 stores robot control information for controlling the robot and information (sensor information) acquired from a sensor provided in the robot device 200 in the memory unit 170, sets different access rights for each, and transmits information to the operation terminal 300 that has the corresponding authority to provide information when the set access rights are confirmed. This protects the confidentiality of the data. Therefore, for example, it is possible to prevent information from being stolen or tampered with by others who do not have access rights, and the security of the entire system can be improved. Furthermore, by managing the information provided according to the access rights, various users, etc. can use the information provision system 1 with peace of mind.

The usage status management unit 146 manages the usage status of the information provision system 1 for each user. For example, the usage status management unit 146 manages the spatial location for each user. The spatial location is, for example, a work space, and may be a real space or a virtual space. The following explanation focuses mainly on the real space. The usage status management unit 146 also manages the location of the operator relative to a seat that has been pre-assigned in the space, and manages the usage status of the information provided to the operation terminal 300, etc.

The control unit 150 includes, for example, a robot information control unit 151, a customized data control unit, an intention estimation unit 153, an operation control unit 154, an operation cancellation unit 155, and an output control unit 156. The robot information control unit 151 manages, for example, what operations the robot device 200 actually performed in response to the operation control information for the robot device 200 generated by the operation control unit 154. The robot information control unit 151 also causes the robot device 200 to perform basic operations based on the robot control information 173 stored in the storage unit 170. The basic operations are, for example, operations to move the robot device 200, change its posture, and pick up or place an object. The control information for these basic operations is stored in the robot control information 173.

The customization data control unit 152 generates customization data for causing the robot device 200 to execute specific operations (e.g., assembling an object, soldering, screwing, various cooking processes, etc.).

The intention estimation unit 153 estimates the intention of the operator from at least a part of the information of the operation contents input by the operation terminal 300. The intention of the operator includes, for example, not only the intention of the action (movement) that the operator wants the robot device 200 to perform, but also prediction of the action, etc., and assisting the driving of the robot device 200. For example, in remote operation, a discrepancy occurs between the feeling that the operator is instructing through communication, etc. and the movement of the robot device 200. Therefore, the intention estimation unit 153 does not directly execute the operation contents from the operator to the robot device 200, but estimates the intention of the operator based on the operation contents, and generates operation control information for the robot device 200 based on the estimation result, thereby more appropriately assisting the movement of the robot device 200 and realizing remote operation with less discomfort. The intention estimation unit 153, for example, refers to a learning model (intention estimation information 174) learned using teacher data, and estimates the intention corresponding to the operation contents. In addition, during learning, for example, the operation contents are input, and learning is performed using teacher data for outputting correct intention information set by the operator, etc. for the operation contents.

The intention estimation unit 153 may also update the intention estimation information 174 based on history information indicating the operation content input by the operator and the actual operation content of the robot device 200. This allows for more appropriate intention estimation.

Furthermore, the intention estimation unit 153 may estimate the intention of the operator by, for example, a GRASP Taxonomy method (see, for example, Reference 1). In the embodiment, the intention of the operator is estimated by classifying the operator state by classifying the posture, i.e., the gripping posture, of the operator or the robot device 200 by, for example, the GRASP Taxonomy method.
Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77.

The operation control unit 154 generates operation control information for operating the target robot device 200 based on the intention of the operator estimated by the intention estimation unit 153, transmits the generated operation control information to the target robot device 200, and controls the operation of the robot device 200. The operation control unit 154 may also acquire information on the operation of the robot device 200 obtained by the robot information control unit 151, the customized data control unit 152, etc. in addition to (or instead of) the intention of the operator, and generate operation control information based on the acquired information. For example, when multiple operators operate two arm units at the same time, the operation control unit 154 may perform synthesis processing such as addition or subtraction of forces based on sensor information such as tactile sense, and perform operation control for working in cooperation. This makes it possible to achieve well-balanced operations even when one robot device 200 is operated by multiple operators.

The action cancellation unit 155 selects an action that is different from the operator's intention from among the actions that the action control unit 154 causes the robot device 200 to execute, and cancels the execution of the selected action. In this case, the action cancellation unit 155 inquires of the operator whether or not the operation content input by the operator is the content to be executed by the robot device 200, and causes the action control unit 154 to generate action control information and update the intention estimation information 174 based on the inquiry result.

The output control unit 156 generates information to be provided to the operator operating the operation terminal 300 based on the audio and images acquired from the robot device 200. The information to be provided includes images and audio. The output control unit 156 also outputs sensor information and the like based on the result of the authentication process performed by the authentication management unit 142. The output control unit 156 also generates information to be provided for each user based on the contents managed by the usage status management unit 146.

The storage unit 170 may be realized by the various storage devices described above, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), a random access memory (RAM), or the like. For example, user information 171, usage status information 172, robot control information 173, intention estimation information 174, sensor information 175, programs, and other information are stored in the storage unit 170.

3 is a diagram for explaining the contents of the user information 171. The user information 171 is, for example, authentication information for authenticating a user when using a service provided by the information providing system 1, to which information such as name, authority, priority, and image information is associated. The authentication information includes, for example, a user ID and a password, which are identification information for identifying a user. The authentication information may also include biometric authentication information such as fingerprint information and iris information. The authority includes, for example, authority information such as access rights assigned to the user. Based on this authority information, sensor information acquired by the robot device 200 can be acquired. The information management unit 144 can manage who can access which data and which data cannot be accessed by referring to the information stored in the authority. The priority information stores, for example, the priority when multiple operators remotely operate the robot device 200 and the operation contents of each are the same or similar. The image information is image information representing a user. The image information includes, for example, a face image or a whole image of the user, an avatar image or an illustration image created to resemble the user so that the user can be identified. Additionally, the image information may include images of the user taken from 360 degrees, or images of only individual parts (e.g., arms and legs).

Figure 4 is a diagram for explaining the contents of the usage status information 172. In the usage status information 172, for example, a user ID is associated with a date and time, seat information, a robot ID which is identification information for identifying the robot device 200, and part information. The date and time may be, for example, the date and time of start of use, and may include information such as the duration of use. The seat information is information about the seat used by the user among the seats arranged in the space where the robot device 200 performs work. This seat may store information about the location (seat) where an actual worker other than the robot device 200 performs work. The part information is information about the parts in charge when the robot device is operated by multiple users (operators). The usage status information 172 is managed by the usage status management unit 146 and updated according to the situation.

Figure 5 is a diagram for explaining the contents of the intention estimation information 174. In the intention estimation information 174, for example, information regarding the estimated intention is associated with the operation content acquired from the operation terminal 300. This information may be managed for each operator, or the intention estimation information 174 of multiple operators may be integrated to store only information regarding frequently occurring intention estimations, or may be updated based on history information, etc.

Figure 6 is a diagram for explaining the contents of the sensor information 175. In the sensor information 175, the robot ID, which is identification information for identifying the robot device 200, is stored with the date and time, the operator ID, the operation parts, the sensor information, etc. The operator ID stores the user ID of the operator who performed the operation. The operation parts store the parts of the robot ID operated by the operator. This makes it possible to manage the operation history (sensor information) of each operator even when one robot device 200 is operated by multiple operators.

[Robot device]
7 is a diagram showing an example of the configuration of the robot device 200 according to the embodiment. The robot device 200 includes, for example, a communication unit 202, a sensor 204, a camera (an example of an imaging unit) 206, a microphone 208, a driving unit 210, a monitor 214, a speaker 216, a control device 240, and a storage unit 260. The driving unit 210 includes, for example, an arm unit 210A and a moving driving unit 210B. The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are each realized by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device having a non-transient storage medium) such as an HDD or flash memory provided in the robot device 200, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and the storage medium may be installed in the HDD or flash memory provided in the robot device 200 by being attached to a drive device provided in the robot device 200.

The communication unit 202 will be described later. Under the control of the communication control unit 242, the communication unit 202 communicates with the information providing server 100 and other external devices via the network NW. The communication unit 202 may also communicate with other robot devices 200, or with the operation terminal 300.

The sensor 204 includes a position sensor that detects the position of the robot device 200, a speed sensor that detects the speed, and a temperature sensor that detects the temperature of a specific position such as the periphery of the robot device 200 or the tip of the arm unit 210A. The position sensor receives information from, for example, a GPS (Global Positioning System) receiving device, and obtains position information (longitude and latitude information) based on the received information. The sensor 204 may also include a humidity sensor that detects the surrounding humidity, and a vibration sensor that detects vibrations of an object operated by the robot device 200. The sensor 204 may also include an object detection sensor that detects surrounding objects. Examples of surrounding objects include other robot devices 200, people, and obstacles.

The camera 206 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 206 captures an image of the surroundings of the robot device 200 at a predetermined timing, for example. The number of cameras 206 is not limited to one, and multiple cameras may be provided on the robot device 200. For example, the camera 206 is provided on the head of the robot device 200. The camera 206 may also be provided near the tip of the arm unit 210A. This allows the object to be worked on to be photographed at a close distance, making it easier to perform more detailed work.

Sounds around the robot device 200 are input to the microphone 208. The microphone 208 outputs information based on the input sound to the control device 240.

The arm unit 210A grasps a target object and performs a predetermined task on the target object. The arm unit 210A is, for example, a multi-joint robot arm, and includes, for example, an actuator, a gear, an artificial muscle, and the like. For example, the arm unit 210A includes a first arm unit in which one end of the multi-joint robot arm is connected near the right side of the robot device main body, and a second arm unit in which one end of the other multi-joint robot arm is connected near the left side of the robot device main body. The arm unit 210A may also include other arm units in addition to the first arm unit and the second arm unit. Hereinafter, when there is no need to distinguish between the respective arms, they will simply be referred to as "arm unit 210A". The other end of the arm unit is configured with a gripping unit capable of gripping a predetermined object. The arm unit 210A is driven based on the control of the control device 220. The arm unit is capable of movement equivalent to that of a human arm.

The moving drive unit 210B is a drive unit for moving the robot device 200 on the floor or ground. The moving drive unit 210B may be, for example, two legs, or may be provided with a moving mechanism such as wheels, a dolly, or a crawler. For example, when the moving drive unit 210B is a leg, the leg operates to make the robot device 200 walk based on the control of the control device 220. The moving drive unit 210B may also be structured to be movable along a ceiling or rail by driving a crane or the like. With these configurations, the robot device 200 can move in a desired direction. Note that the drive unit 210 may be provided with a mechanism for driving other joints such as the waist and head in addition to the arm unit 210A or the moving drive unit 210B. The drive unit 210 is driven based on the control of the drive control unit 246.

The monitor 214 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The monitor 214 displays the information output by the output control unit 248 as an image. A plurality of monitors 214 may be provided in the robot device 200. The monitor 214 may be provided, for example, in the head, abdomen, or back. The speaker 216 outputs the information output by the output control unit 248 as sound.

The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into the drive device.

The communication control unit 242 communicates with the information providing server 100, for example, by wireless communication via the communication unit 202, to transmit and receive information. The communication control unit 242 may also communicate with other robotic devices 200 via the communication unit 202.

The acquisition unit 244 acquires the detection results (sensor information) of the sensor 204 provided in the robot device 200. The acquisition unit 244 also acquires images captured by the camera 206. The acquisition unit 244 also acquires information based on sounds input from the microphone 208.

The drive control unit 246 operates the arm unit 210A and the movement drive unit 210B based on the motion control information acquired from the information providing server 100. The drive control unit 246 also drives other parts of the robot device 200 (e.g., the head, torso, waist, etc.) based on the motion control information to change the posture, etc. The drive control unit 246 may also drive the robot device 200 based on the basic motion information 262 and customized motion information 264 stored in the storage unit 260. The basic motion information 262 is drive control information for executing basic motions according to the type, etc. of the robot device 200. The customized motion information 264 is drive control information for causing the robot device 200 to execute a specific motion, for example, registered in advance for each operator.

The output control unit 248 causes the monitor 214 to output an image based on the information obtained from the information providing server 100. The output control unit 248 also causes the speaker 216 to output voice, warning sounds, etc. based on the information obtained from the information providing server 100.

The storage unit 260 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM. For example, the storage unit 260 stores basic operation information 262, customized operation information 264, programs, and other information.

[Operation terminal]
8 is a diagram showing an example of the configuration of the operation terminal 300 according to the embodiment. The operation terminal 300 includes, for example, a communication unit 310, a visual device 320, an operation device 330, an environmental sensor 340, a control device 360, and a storage unit 380. The visual device 320, the operation device 330, and the environmental sensor 340 are examples of an "HMI (Human Machine Interface)."

The communication unit 310 communicates with the information providing server 100 via the network NW under the control of the control device 360. The communication unit 310 may also communicate with other operation terminals 300 and robot devices 200.

The visual device 320 may be, for example, an HMD, or a configuration in which a sensor or the like is provided on a glasses-type device or display. The visual device 320 includes, for example, an image display unit 322, a gaze detection unit 324, a sensor 326, a microphone 327, a speaker 328, and a control unit 329. The visual device 320 outputs and displays the state image and sound of the robot device 200 received by the control unit 360 from the information providing server 100, and detects the movement of the operator's gaze, etc.

The image display unit 322 is, for example, an LCD or an organic EL display. The image display unit displays an image output by the output control unit 368 (described later) in response to the control of the control unit 329.

The gaze detection unit 324 detects the gaze of an operator using the visual device 320 under the control of the control unit 329, and outputs the detected gaze information (operator sensor value) to the control device 360. The gaze information is, for example, information including a gaze vector.

The sensor 326 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 326 detects the tilt and rotation of the head of the operator wearing the visual device 320, and outputs the detected head movement information (operator sensor value) to the control device 360.

The microphone 327 accepts voice input from the operator. The speaker 328 outputs the voice, warning sounds, etc. output by the output control unit 368.

The control unit 329 controls the execution of gaze detection by the gaze detection unit 324 based on control information from the control device 360, controls detection by the sensor 326, controls the display of images on the image display unit 322, and accepts information input from the operator via the visual device 320.

The operation device 330 includes, for example, a sensor (operator sensor) 332, a control unit 334, a feedback unit 336, and an input unit 338. The operation device 330 includes, for example, a tactile data glove that is worn on the operator's hand.

The sensor 332 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 332 includes a plurality of sensors. The sensor 332 tracks the movement of each finger using, for example, two sensors. The sensor 332 detects operator arm information (operator sensor value) which is information on the orientation and movement of each finger of the operator, the posture and position of the operator's arm, such as the movement of the hand, under the control of the control unit 334, for example. The operator arm information (operator sensor value) includes information covering the entire human arm, such as hand position and posture information, angle information of each finger, elbow position and posture information, and information tracking the movement of each part.

The control unit 334 outputs the operator's arm information detected by the sensor 332 to the control device 360. The control unit 334 also controls the feedback unit 336 based on the feedback information obtained from the control device 360.

The feedback unit 336 feeds back feedback information to the operator in response to the control of the control unit 334. In response to the feedback information, the feedback unit 336 feeds back sensations to the operator, for example, by means of vibration (not shown), air pressure (not shown), hand movement restriction (not shown), temperature sensation (not shown), hardness or softness sensation (not shown), vibration sensation (not shown), etc., attached to the arm unit 210A of the robot device 200.

The input unit 338 is, for example, an input device other than the tactile data glove, such as a keyboard, a mouse, a lever, a touch panel, a microphone, etc. The input unit 338 accepts input of operation details for the robot device 200 from each input device.

The environmental sensor 340 detects, for example, the movement of the operator. The environmental sensor 340 includes, for example, a camera (an example of an imaging unit) 342, a sensor 344, and an object position detection unit 346. The camera 342 captures an image including the operator. The camera 342 is, for example, an RGB camera. The camera 342 outputs the captured image to the object position detection unit 346. Note that in the environmental sensor 340, the positional relationship between the camera 342 and the sensor 344 is known.

Sensor 344 is, for example, a depth sensor. Sensor 344 outputs the detection result to object position detection unit 346. Note that camera 342 and sensor 344 may be distance sensors.

Based on the image captured by the camera 342 and the detection result detected by the sensor 344, the object position detection unit 346 detects the three-dimensional position, size, shape, etc. of the target object in the captured image using a known method. The object position detection unit 346 estimates the position of the object by performing image processing (edge detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, etc.) on the image captured by the camera 342 with reference to a pattern matching model stored in the object position detection unit 346. Note that, when multiple objects are detected from the captured image, the object position detection unit 346 detects the position of each object. The object position detection unit 346 transmits the detected object position information to the control device 360. Note that the data transmitted by the environmental sensor 340 may be, for example, a point cloud having position information.

The control device 360 includes, for example, a communication control unit 362, an acquisition unit 364, an operation content generation unit 366, and an output control unit 368. The communication control unit 362, the acquisition unit 364, the operation content generation unit 366, and the output control unit 368 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

The communication control unit 362 communicates with the information providing server 100, for example, by wireless communication via the communication unit 310, and transmits and receives information. The communication control unit 362 may also communicate with other operation terminals 300 via the communication unit 310. The communication control unit 362 also communicates with the visual device 320, the operation device 330, and the environmental sensor 340.

The acquisition unit 364 acquires the division information obtained from the visual device 320, the operation device 330, and the environmental sensor 340. The operation content generation unit 366 generates operation content for the robot device 200 based on the information acquired by the acquisition unit 364. For example, the operation content generation unit 366 generates operation content related to line of sight information and head direction based on information acquired from the visual device 320. The operation content generation unit 366 also generates operation content related to the movement of the arm unit 210A of the robot device 200 based on information acquired from the operation device 330. The operation content generation unit 366 also generates operation content related to the posture, movement direction, and movement amount of the robot device 200 based on information acquired from the environmental sensor 340. The information generated by the operation content generation unit 366 is transmitted to the information providing server 100 via the communication unit 310.

The output control unit 368 outputs the provided information acquired from the information providing server 100 to the visual device 320 and the operation device 330. For example, the output control unit 368 causes the image acquired from the information providing server 100 to be output to the image display unit 322 of the visual device 320. The output control unit 368 also causes the speaker 328 to output voice, warning sound, etc. based on the information acquired from the information providing server 100. The output control unit 368 also causes the operation device 330 to output feedback information for conveying tactile information acquired from the information providing server 100 to the operator.

The storage unit 380 is realized, for example, by a HDD, a flash memory, an EEPROM, a ROM, or a RAM. The storage unit 380 stores, for example, programs and other information.

[Regarding remote control of the robot device according to the embodiment]
Next, remote operation of the robot device 200 by the information provision system 1 according to the embodiment will be specifically described. Fig. 9 is a diagram for explaining remote operation of the robot device 200 according to the embodiment. Fig. 9 shows an example in which two operators U1 and U2 take turns operating one robot device 200. As shown in Fig. 9, each of the operators U1 and U2 is equipped with a visual device 320 and an operation device 330. The operation terminals 300-1 and 300-2 may be installed in different locations.

In addition, the operators U1 and U2 are provided with environmental sensors 340 that measure the movements of the operators and the surrounding environment. Note that a configuration similar to the environmental sensor 340 may be attached to the robot device 200, for example. The operation device 330 includes, for example, an operation device 330a worn on the left hand of the operator and an operation device 330b worn on the right hand. The operation contents input from each of the operation devices 330a and 330b are used to control the movements of the left hand LH and right hand RH of the robot device 200. The position of the operator's arms, the direction of the face, etc. are detected by the environmental sensor 340. Each detected data is transmitted to the information providing server 100 via a network.

The information providing server 100 estimates the operator's intention based on the operation content obtained from one of the operation terminals 300-1 and 300-2 that operates the robot device 200. Here, an example of information estimated by the intention estimation unit 153 will be specifically described. The intention estimation unit 153 estimates the operator's intention to operate based on the acquired operation content from the operator. For example, the intention estimation unit 153 classifies the posture of the arm including the grip of the robot device by classifying the posture of the operator's arm based on the operator sensor value of the operation terminal 300. In addition, the intention estimation unit 153 estimates the intention of the operation that the operator wants the robot device 200 to perform based on the classification result. The intention estimation unit 153 estimates, for example, the movement of hands and fingers at each time, the purpose of the work that the robot device 200 wants to perform, the work content, the movement of hands and fingers at each time, etc. as the operator's intention to operate. The purpose of the work is, for example, grasping an object, moving an object, etc. Examples of tasks include grasping and lifting an object, grasping and moving an object, etc.

The intention estimation unit 153 may also estimate the operator's intention to operate by, for example, inputting the operation content into a learned model or intention estimation information 174 stored in the storage unit 170. In the embodiment, the intention estimation is performed by classifying the grip posture, so that the operator's intention to operate can be estimated with high accuracy. Note that other methods may be used to classify the grip posture.

The intention estimation unit 153 may also perform an integrated estimation using the gaze and arm movement. In this case, the intention estimation unit 153 may input gaze information, hand movement information, and position information of the object OB on the table TB to a trained model to estimate the operator's intention to operate.

In the example of FIG. 9, the intention estimation unit 153 estimates the object to be grasped based on, for example, gaze information. Next, the intention estimation unit 153 estimates the posture of the operator's hand based on the estimated object to be grasped. Alternatively, the intention estimation unit 153 may first estimate the posture of the operator's hand based on the operation content, and then estimate the object to be grasped from the estimated posture of the operator's hand. Also, when operations of different parts of the body are assigned to each of the operators U1 and U2, the intention estimation unit 153 may estimate the intention based on the operation content of each.

For example, when an object OB is placed on the table TB shown in FIG. 9, the intention estimation unit 153 estimates that the object OB will be grasped based on the posture of the hand. The intention estimation unit 153 may also estimate in advance the future trajectory of the hand that the operator intends to grasp based on the operation content and state information of the robot device 200. The intention estimation unit 153 may also estimate the object to be grasped and the position of the object by using the detection results detected by the sensor, the results of image processing of the image captured by the environmental sensor 340, etc.

In addition, since the coordinate system is different between the environment in which the operator operates and the robot operating environment, for example, calibration between the operator's operating environment and the robot operating environment may be performed when the robot device 200 is started. In addition, when grasping, the information providing server 100 may determine the grasping position taking into account the error in the grasping position at the time of grasping based on the grasping force of the robot device and the frictional force between the object and the grasping part, etc.

When the robot device 200 is not remotely operated, its operation is controlled according to the control of the control device 240, and when it is remotely operated, its operation is controlled according to operation control information generated by the information providing server 100 based on the estimated intention, etc.

The control device 240 of the robot device 200 controls the driving unit 210 based on the operation control information from the information providing server 100. The robot device 200 also outputs information such as sensor information detected by the sensor 204, images captured by the camera 206, and audio output by the microphone 208 to the information providing server 100. The information providing server 100 generates information to be provided to the operation terminals 300-1 and 300-2 based on the acquired sensor information, images, audio, etc. In this case, the information providing server 100 may generate an image in which at least a part of the sensor information is superimposed on the image. The information providing server 100 generates information to be provided that also includes information to be transmitted by the sense of touch. In addition, since the robot device 200 can be operated by multiple operators, an image in which information for identifying the operator of each part of the robot device 200 is superimposed on the camera image may be generated. The operation terminal 300 displays the image output from the information providing server 100. This allows multiple operators to understand the status of the robots being operated by the other operators when multiple operators are operating the robot device 200.

When performing remote operation as described above, the operators may feel lonely because they are operating the operation terminals 300 in different locations. For this reason, the information providing server 100 generates information that makes the operator feel as if he or she is in the working space where the robot device 200 exists (creating an illusion or a simulated experience), and provides the generated image to the operation terminal 300.

For example, when a user instructs to remotely operate the robot device 200, the information providing server 100 transmits information (image) to the operation terminal 300 to allow the user to select a location in space where the robot device 200 will work. The location is determined, for example, based on seat information in the space. FIG. 10 is a diagram showing an example of an image IM10 for allowing the user to select a seat. Note that the display mode of the image IM10, such as the layout and display content, is not limited to the example of FIG. 10. The same applies to the display modes of the other images described below. The content of the image IM10 is generated, for example, by the output control unit 156.

Image IM10 includes, for example, a text display area A11 and a seat selection area A12. Text information that prompts the user to select a seat is displayed in text display area A11. In the example of FIG. 10, text information such as "Which seat would you like to work at?" is displayed. Note that the information providing server 100 may generate voice data equivalent to the text information and output the generated data from the operation terminal 300.

In the seat selection area A12, for example, information on the use status of seats in the work space where the robot device 200 operated by the user is operated is displayed. In the example of FIG. 10, six seats S1 to S6 are displayed corresponding to the positions of the seats in the real space. When the seats are set in the virtual space, the seat positions may be different depending on, for example, the normal business mode, the conference mode, etc. The seat arrangement may be, for example, a circular arrangement in the conference mode. The seat arrangement may be the same as the seat arrangement in the company, and the real world and the virtual world may be mixed by displaying "At work" on the seats of employees who are in the office. The information providing server 100 refers to the use status information 172 managed by the use status management unit 146, and displays the information on the seats in use in the seat selection area A12. In the example of FIG. 10, an image indicating that the seats S2 and S5 are in use and the seats S1, S3, S4, and S6 are vacant is displayed in the seat selection area A12. The information providing server 100 may also display information identifying the currently active user in the seat selection area A12. The example in FIG. 10 shows that user U4 is using seat S2, and user U3 is using seat S5. This allows the user to select the seat they desire based on the current usage situation. The following description will be given assuming that the user has selected seat S4.

When the seat is identified, the information providing server 100 updates the usage status information 172, generates motion control information for moving the robot device 200 to the position of seat S4, and transmits the generated motion control information to the robot device 200. The robot device 200 moves to the position S4 based on the received motion control information, and transmits images captured by the camera 206, etc. to the information providing server 100. The information providing server 100 transmits the images obtained from the robot device 200 to the operation terminal 300, and displays them on the visual device 320.

Figure 11 is a diagram showing an example of an image IM20 captured by the robot device 200. Image IM20 is a diagram of a virtual space viewed to the right from seat S4. In the example of Figure 11, user U3 and another robot device 200 (U4) operated by user U4 are displayed. This allows the user to identify workers or other robot devices present in the vicinity.

In addition, by looking directly at the image IM20, it is not possible to know who is operating the other robotic device 200 (U4). Furthermore, as remote work using other robotic devices 200 progresses in the future, there may be cases where there are only a few robotic devices 200 in the vicinity, which may create a sense of loneliness. Therefore, the information providing server 100 acquires an image representing the user U4 operating the other robotic device 200 (U4) from the image information stored in the user information 171, and generates an image by superimposing the acquired image on the other robotic device 200 (U4).

FIG. 12 is a diagram for explaining superimposing image information of a user on other robotic devices 200 (U4) existing in the vicinity. In the example of FIG. 12, an image IM30 is generated by superimposing an image IM (U4) of a user U4 operating another robotic device 200 (U4) on an image IM20. For example, the information providing server 100 detects the orientation of the other robotic device 200 (U4) included in an image captured by the camera 206 of the robotic device 200, and acquires image information captured at an angle corresponding to the detected angle from the user information 171. In addition, the acquired image is enlarged or reduced, and superimposed so that the other robotic device 200 (U4) is hidden according to the size of the other robotic device 200 (U4) included in the image IM20. In addition, the information providing server 100 partially moves the position of the arms and legs of the image of the user U4 according to the actions (movement of the arms and legs) of the other robotic devices 200 (U4) to display information on the actions of the operator. As described above, when other robotic devices 200 are included in the image provided to the operation terminal 300, an image representing the user operating the other robotic device 200 can be displayed superimposed while moving in accordance with the action, making the operator feel as if they are actually operating at that location, and even during remote operation, the operator can work while keeping track of the state of the surrounding workers without feeling alone. In addition, because information related to the action is displayed, it is possible to distinguish from the image whether the operator is busy or not.

In addition, the information providing server 100 acquires not only images but also sounds such as surrounding conversations as shown in FIG. 12, and provides the acquired sounds to the operation terminal 300. In this case, the information providing server 100 outputs the sound acquired from the operation terminal operating the other robot device 200 (U4) by localizing the sound image at the location of the other robot device 200 (U4). The voice of the worker U3 is input by the microphone 208 and output. This allows the operator to hear the sound corresponding to the location, making it feel as if he or she is in the work space more realistically. In addition, the work performed by the other robot device 200 (U4) and the workers in the real space can be recognized from the image, so the user can manage his or her own work and progress while directly checking the work status of the other workers.

In addition, when the line of sight (face direction) of the robot device changes, the information providing server 100 also provides the above-mentioned image, sound, and other information to other robot devices that are seated in the direction of the change. This makes it possible to display the operator operating the robot device 200 in the vicinity not only when the robot device turns left and right, but also when the robot device turns back.

In addition, when images and sounds of other workers are provided during work, the information may be perceived as annoying. Therefore, the information providing server 100 may provide an image from which the surrounding workers and robot devices have been deleted when requested by the operation terminal 300. FIG. 13 is a diagram showing an example of an image IM40 from which the images of the surrounding workers and robot devices 200 have been deleted. In the example of FIG. 13, an image IM40 from which the other robot devices 200 and workers displayed in the areas A41 and A42 have been deleted is displayed. In the deleted areas A41 and A42, a background image taken when the other robot devices 200 and workers are not present may be superimposed, or another mask image may be displayed. In addition, the information providing server 100 may delete only the other robot devices or only a specific user (worker) in response to a request from the user. In addition to the image, the corresponding sound may be deleted (mute function). This allows the user to switch the content of the surrounding situation provided depending on the work situation, etc., and work in his/her preferred environment.

In the embodiment, when seats are assigned in advance on the virtual section, the location of the operator may be managed in association with the assigned seat information, and an image showing the managed virtual space may be output to the operation terminal. FIG. 14 is a diagram showing an example of arranging information based on information representing the operator and information related to the action in the virtual space. As shown in FIG. 14, the information providing server 100 generates an image IM50 in which images IM (U3) and IM (U4) representing the operators U3 and U4 are superimposed on an image showing the seat positions in the virtual space at positions corresponding to the positions of the images showing the seats used by the operators. In addition, the information providing server 100 outputs information in which the voices of the operators are localized as sound images at the positions where the images IM (U3) and IM (U4) are superimposed from the operation terminal 300. This makes it possible to provide information that makes the operator feel as if he or she is in the working space even in the virtual space.

[Processing sequence]
Next, the process executed by the information providing system in the embodiment will be described with reference to a sequence diagram. Fig. 15 is a sequence diagram for explaining the process executed by the information providing system 1 in the embodiment. In the example of Fig. 15, the information providing server 100, the robot device 200, and the operation terminal 300 are used for the description.

The operation terminal 300 accesses the information providing server 100 to remotely operate the robot device 200 (step S100). The information providing server 100 performs authentication processing to determine whether or not the information providing system 1 can be used based on authentication information input from the operation terminal 300 (step S102). The following describes a case where use is permitted by authentication. If use is permitted, the information providing server 100 generates an image (e.g., image IM10) that allows the user to select a seat (step S104). The generated image IM10 is displayed on the operation terminal 300 (step S106). The information providing server 100 also acquires seat information selected by the operation terminal 300 (step S108) and manages the usage status (step S110). The information providing server 100 also generates motion control information to move to the selected seat and transmits the generated motion control information to the robot device 200 (step S112).

The robot device 200 moves to the selected seat position based on the operation control information from the information providing server 100 (step S114). The robot device 200 outputs the detection results of the sensor, images captured by the camera, etc. to the information providing server 100 (step S116).

The information providing server 100 acquires the information transmitted from the robot device 200, and generates information (provided information) to be provided to the operation terminal 300 based on the acquired information (step S118). The information providing server 100 then transmits the generated provided information to the operation terminal 300 (step S120).

The operation terminal 300 receives the provided information transmitted from the information providing server 100, and provides the information by outputting the received information from the visual device 320 and the operation device 330 (step S122). Next, the operation terminal 300 transmits the operation content related to the robot device 200 to the information providing server 100 (step S124).

The information providing server 100 performs intention estimation based on the operation contents obtained from each of the operation terminals 300 (step S126), generates operation control information based on the intention estimation result (step S128), and transmits the generated operation control information to the robot device 200 (step S130).

The robot device 200 performs an operation based on the operation control information (step S132), and transmits information obtained from sensors, etc. (sensor information) and camera images, etc. during or as a result of the operation, to the information providing server 100 (step S134).

The information providing server 100 stores the received information in the storage unit 170 (step S136), generates information to be provided for transmitting the acquired information to the operation terminal 300 (step S138), and transmits the generated information to be provided to the operation terminal 300 (step S140). The operation terminal 300 outputs the acquired information to provide the information to the operator (step S142). Thereafter, the processing of steps S124 to S142 is continued until the operation of the robot device 200 is completed.

<Modification>
A part or all of the processes of the information providing server 100, the robot device 200, and the operation terminal 300 in the embodiments may be realized by AI (Artificial Intelligence) technology. In addition, a part or all of the functional units or information stored in the functional units or storage units included in the information providing server 100, the robot device 200, and the operation terminal 300 may be included in other devices.

In addition, in the above embodiment, information representing the operator and information regarding their actions are arranged and output at locations arranged in space, but for example, when users are seated in a virtual space in a web conference mode or the like, the arrangement may be changed so that only the designated person is present in front of the user.

The information providing server 100 may also transmit operation history information, intent estimation information, and the like to the robot manufacturer terminal 400. This can be used for developing the robot manufacturer terminal 400 and products, updating programs, and the like. In this case, a discount or reduction service for system usage may be provided to companies that perform eco-driving of the robot device 200 (e.g., low-power driving, low-load driving, low-communication-volume driving), or to companies that provide a robot control method suited to the operator or telework characteristics. This can encourage users to provide information, which can be useful for improving services.

In addition, in the above embodiment, when multiple operators operate one robot device 200, an example was shown in which each operator's operating intention and work status were displayed. However, for example, the above-mentioned intention estimation results, work status, and information about the operator may be provided between multiple robot devices 200 that exist within a specified distance, or between robot devices performing collaborative work. In this way, by using the intention estimation function between robots, it becomes possible to share awareness more effectively than between humans. In addition, since it is possible to obtain the status of surrounding robot devices 200, group work, etc. can also be performed smoothly.

The embodiment described above includes an information providing server 100 that includes a communication unit 110 that communicates with one or more operation terminals 300 via a network NW, an acquisition unit 130 that acquires information related to the behavior of the operator of the operation terminal 300, a management unit 140 that manages the location of the operator in space, and an output control unit 156 that arranges information based on information representing the operator and information related to the operator's behavior at the location of the operator in space managed by the management unit 140 and outputs the arranged information to the operation terminal 300.By doing so, it is possible to provide information that makes the user feel as if they are in the work space even when operating remotely.

Specifically, according to the above-mentioned embodiment, for example, by coordinating the position (location) of the operator in the virtual space (workplace) and arranging information showing the state of the operator working in the virtual space based on that position, it is possible to output the state seen from the operator's own position in the virtual space (the state of people working in the same space) in the image seen by the operator (image of the visual device). Furthermore, according to the above-mentioned embodiment, when performing telework by remotely operating a robot device, it is possible to use various devices more generally and operate them in a team. Furthermore, while in the past, only limited tasks could be performed with limited dedicated robots even when remotely operated, according to this embodiment, anyone can easily perform telework using a highly versatile robot device.

According to the embodiment, robot information and sensor information (information that may include corporate secrets) can be managed separately, and a server capable of handling confidential information can be provided. Remote operators can access the same server and can share the control rights and sensor information of the same robot or different robots. For example, when operating the same robot device, multiple operators can operate different arms, allowing a senior and a junior to simultaneously operate the same arm of the same robot and add or subtract force while sharing sensor information such as touch. In addition, when operating different robots, telepathy is possible by sharing the intention estimation function and disclosing the intention of others. According to the embodiment, one operator can operate two robots. In this case, for example, if two robots are to carry a large desk, one robot is brought to the edge of the desk in advance and synchronized with the other robot. Also, when the other robot is brought to the other end and an operation to lift it is performed, the robots are lifted together by synchronization processing. Furthermore, in the embodiment, more accurate work is possible than when humans call out to each other to perform the operation. In addition, in the embodiment, when one operator has multiple robots perform the same work, the supported teleoperation function can absorb individual differences between robots and individual differences between work objects.

The above-described embodiment can be expressed as follows.
A server computer that communicates with a mobile object and a mobile terminal of a user of the mobile object,
A storage device storing a program;
a hardware processor;
The hardware processor executes the program stored in the storage device,
Communicating with one or more operation terminals via a network;
Acquire information regarding the behavior of an operator of the operation terminal;
Managing the location of the operator in space;
arranging information based on information representing the operator and information related to the action at a location in the space of the managed operator, and outputting the arranged information to the operation terminal;
An information providing server configured to:

The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1... Information providing system, 100... Information providing server, 110, 202, 310... Communication unit, 120... Communication control unit, 130... Acquisition unit, 140... Management unit, 150... Control unit, 170... Storage unit, 200... Robot device, 204... Sensor, 206... Camera, 208... Microphone, 210... Driving unit, 214... Monitor, 216... Speaker, 240... Control device, 260... Storage unit, 300... Operation terminal, 320... Visual device, 330... Operation device, 340... Environmental sensor, 360... Control device, 380... Storage unit

Claims (7)

A communication unit that communicates with a plurality of operation terminals via a network;
an acquisition unit that acquires information regarding an action of an operator of a first operation terminal among the plurality of operation terminals and information regarding an action of an operator of an operation terminal other than the first operation terminal;
a management unit that manages the location of the operator of the other operation terminal in a virtual space of an image captured by an imaging unit of a space in which one or more robot devices exist;
an output control unit that arranges information based on information representing the operator of the other operating terminal and information regarding the behavior at an arrangement location in a virtual space of the operator of the other operating terminal managed by the management unit, and outputs the arranged information to the first operating terminal;
the information about the actions of the operators of the other operation terminals includes operation content for remotely operating the one or more robot devices;
the information based on the information representing the operator of the other operating terminal and the information relating to the behavior includes information in which an image representing the operator of the other operating terminal is superimposed at a position where a robot device operated by the operator of the other operating terminal is present among the one or more robot devices present in the image captured by the imaging unit, and information in which a sound image is localized for a voice of the operator of the other operating terminal ,
the output control unit, in response to a request from the first operation terminal, generates an image by deleting at least one of an image representing an operator of the other operation terminal and an image representing the robot device from an image captured by the imaging unit, and provides the image to the first operation terminal.
Information server. the management unit generates an image that allows an operator of the first operation terminal to select a location, transmits the generated image to the first operation terminal, and manages the location of the operator of the first operation terminal based on the location selected using the image.
2. The information providing server according to claim 1. the plurality of operation terminals include a terminal for remotely operating one or more robot devices,
3. The information providing server according to claim 1 or 2. the output control unit acquires a voice of an operator of the other operation terminal , and outputs the acquired voice by localizing a sound image at a location of the operator of the other operation terminal .
4. The information providing server according to claim 1. An information providing server according to any one of claims 1 to 4 ;
The plurality of operation terminals;
one or more robot devices remotely operated by the plurality of operation terminals;
An information provision system comprising: The computer
Communicate with multiple operation terminals via a network,
acquiring information regarding an action of an operator of a first operation terminal among the plurality of operation terminals and information regarding an action of an operator of another operation terminal other than the first operation terminal;
managing locations of operators of the other operation terminals in a virtual space of an image captured by an imaging unit of a space in which one or more robot devices exist;
arranging information representing the operator of the other operation terminal and information related to the behavior at a location in a virtual space of the operator of the other operation terminal that is managed, and outputting the arranged information to the first operation terminal;
the information about the actions of the operators of the other operation terminals includes operation content for remotely operating the one or more robot devices;
the information based on the information representing the operator of the other operating terminal and the information relating to the behavior includes information in which an image representing the operator of the other operating terminal is superimposed at a position where a robot device operated by the operator of the other operating terminal is present among the one or more robot devices present in the image captured by the imaging unit, and information in which a sound image is localized for a voice of the operator of the other operating terminal ,
generating an image obtained by deleting at least one of an image representing the operator of the other operation terminal and an image representing the robot device from an image captured by the imaging unit in response to a request from the first operation terminal, and providing the image to the first operation terminal;
How to provide information. On the computer,
Communicating with multiple operation terminals via a network,
acquiring information on an action of an operator of a first operation terminal among the plurality of operation terminals and information on an action of an operator of another operation terminal other than the first operation terminal;
managing locations of operators of the other operation terminals in a virtual space of an image captured by an imaging unit of a space in which one or more robot devices exist;
arranging information representing the operator of the other operation terminal and information related to the behavior at a location in a virtual space of the operator of the other operation terminal being managed, and outputting the arranged information to the first operation terminal;
the information about the actions of the operators of the other operation terminals includes operation content for remotely operating the one or more robot devices;
the information based on the information representing the operator of the other operating terminal and the information relating to the behavior includes information in which an image representing the operator of the other operating terminal is superimposed at a position where a robot device operated by the operator of the other operating terminal is present among the one or more robot devices present in the image captured by the imaging unit, and information in which a voice of the operator of the other operating terminal is localized as a sound image,
generating an image obtained by deleting at least one of an image representing the operator of the other operation terminal and an image representing the robot device from an image captured by the imaging unit in response to a request from the first operation terminal, and providing the image to the first operation terminal;
program.

Images