JP2020005147A - Information processing apparatus, movable body, remote control system, information processing method, and program - Google Patents

Abstract

To make it possible to achieve smooth communication using a movable body.SOLUTION: An information processing apparatus according to one embodiment of the present invention is an information processing apparatus that adjusts the position of a robot 10 including a photographing device 21 that photographs a subject to acquire image data 200, acquires the image data 200 acquired by the photographing device 21, and calculates a target point indicating a movement destination of the robot 10 on the basis of the positions and directions of a plurality of people included in the image data 200. The information processing apparatus according to one embodiment of the present invention calculates the target point to move the robot 10 to the target point.SELECTED DRAWING: Figure 19

Description

  The present invention relates to an information processing device, a moving object, a remote control system, an information processing method, and a program.

  2. Description of the Related Art A remote control system for remotely controlling a telepresence robot (hereinafter, referred to as a robot) located at a remote base via a communication network using a display terminal located at another base is known. This remote control system allows an image photographed by a photographing device provided in the robot to be displayed on a display terminal, so that information on a base where the robot is located can be confirmed from a remote place.

  A method of activating communication using a robot is also known. Patent Literature 1 discloses a technique for detecting arrangement information (position, body orientation, and line-of-sight direction) between a person and a robot and controlling the operation of the robot based on the detected arrangement information.

  However, in the conventional method, when communicating with a plurality of people from a remote place using a robot (for example, a moving body), the moving body cannot be moved to an appropriate position based on a positional relationship between the plurality of people. However, there is a problem that it hinders smooth communication using a moving object.

  The information processing apparatus according to claim 1, wherein the information processing apparatus includes a photographing device that photographs a subject and acquires image data, and adjusts a position of a moving body, and acquires the image data acquired by the photographing device. Obtaining means, and calculating means for calculating a target point indicating a destination of the moving body based on the positions and orientations of the plurality of persons included in the obtained image data, and By calculating a target point, the moving body is moved to the target point.

  According to the present invention, smooth communication using a moving object can be realized.

It is a figure showing an example of the system configuration of the remote control system concerning an embodiment. FIG. 1 is a diagram illustrating an example of a schematic configuration of a robot according to an embodiment. It is a figure which shows the outline of a structure of the robot which concerns on the modification 1 of embodiment. It is a figure showing the outline of the composition of the robot concerning modification 2 of the embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the robot according to the embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of a display terminal according to the embodiment. FIG. 3 is a diagram illustrating an example of a hardware configuration of a management server according to the embodiment. It is a figure showing an example of the functional composition of the remote control system concerning a 1st embodiment. FIG. 4 is a diagram illustrating an example of a command table according to the first embodiment. FIG. 4 is a diagram illustrating an example of a condition table according to the first embodiment. FIG. 4 is a diagram illustrating an example of a user command table according to the first embodiment. (A) is a conceptual diagram showing an example of an authentication management DB, and (b) is a conceptual diagram showing an example of a terminal management DB. (A) is a conceptual diagram showing an example of a destination list management DB, and (b) is a conceptual diagram showing an example of a session management DB. FIG. 9 is a sequence diagram illustrating an example of a process in a preparation stage for starting transmission and reception of data between the robot and the display terminal. FIG. 5 is a diagram illustrating an example of a destination list screen displayed on the display terminal according to the first embodiment. FIG. 9 is a sequence diagram illustrating an example of processing from selection of a destination candidate to start of data transmission / reception. 5 is a flowchart for explaining an example of a robot control process based on a request command from a display terminal in the remote control system according to the first embodiment. FIG. 2A is a diagram illustrating a screen example of a display screen displayed on the robot according to the first embodiment, and FIG. 2B is a diagram illustrating a screen example of a display screen displayed on a display terminal according to the first embodiment. 5 is a flowchart illustrating an example of a target point calculation process in the robot according to the first embodiment. (A) is processing data schematically showing a human head image extracted by the image processing unit, and (b) is a human head image (a1 to a4) of the human head image calculated by the attention point calculation unit. Position and size values. (A) is coordinate information indicating the relative position specified by the attention point calculation unit, and (b) is processing data schematically indicating the relative position of the person specified by the attention point calculation unit. (A) is processing data schematically showing an image of a person's eyes and nose extracted by the image processing unit, and (b) is a direction of the person specified by the attention point calculation unit. (A) is information on the relative position and orientation specified by the attention point calculation unit, and (b) processing data schematically indicating the relative position and orientation of the person identified by the attention point calculation unit. (A) is a diagram for explaining an example of processing for estimating the position and orientation of a person located around the robot 10 when a beacon is installed in a base. (B) is information on the relative positions and orientations of the persons (h1 to h4) specified by the method shown in (a). It is a calculation result of calculating the intersection of a straight line extending in the direction of the direction of the person specified by the attention point calculation unit from the relative position of the person (h1 to h4) specified by the attention point calculation unit. (A) is the coordinate information indicating the position of the intersection summarized by the processing of the attention point calculation unit, and (b) is the processing data schematically indicating the position of the intersection summarized by the processing of the attention point calculation unit. is there. (A) is information on the target point specified by the target point calculation unit, and (b) is processing data schematically indicating the position of the target point specified by the target point calculation unit. 5 is a flowchart illustrating an example of a movement process in the robot according to the first embodiment. 5 is a flowchart illustrating an example of a presence notification process in the robot according to the first embodiment. (A) (b) It is a figure for explaining an example of the movement course of the robot concerning a 1st embodiment. 6A is a flowchart illustrating an example of a process until a target point calculation process is started in the robot according to the first embodiment. It is a figure showing an example of the functional composition of the remote control system concerning a 2nd embodiment. It is a sequence diagram for explaining an example of control processing of the robot in the remote control system according to the second embodiment. It is a figure showing an example of the system configuration of the remote control system concerning a 3rd embodiment. It is a figure showing an example of the functional composition of the remote control system concerning a 3rd embodiment. It is a sequence diagram for explaining an example of control processing of the robot in the remote control system according to the third embodiment.

  Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the description of the drawings, the same elements will be denoted by the same reference symbols, without redundant description.

● Embodiment ●
System Configuration FIG. 1 is a diagram illustrating an example of a system configuration of a remote control system according to the embodiment. The remote control system shown in FIG. 1 uses a display terminal 50 to remotely control a robot 10 located at a remote site, thereby managing or maintaining a device at the site, or the position of a person located at the site. It is a system that can check the traffic flow and the like. Further, the remote control system 1a realizes a remote conference by bidirectionally transmitting and receiving an image (video) shot at the base of the robot 10 and an image (video) shot at the base of the display terminal 50. It is a system that can do.

  The remote control system 1a includes a robot 10 (10A, 10B, 10C, hereinafter referred to as a robot 10 when there is no need to distinguish between them) located at each of a plurality of bases (base A, base B, base C). It is composed of a display terminal 50 and a management server 90. The robot 10, the display terminal 50, and the management server 90 are communicably connected via the communication network 9. The communication network 9 is constructed by, for example, a LAN (Local Area Network), a dedicated line, the Internet, and the like. The communication network 9 may include a place where not only wired communication but also wireless communication such as Wi-Fi (registered trademark) is performed.

  The robot 10 is a mobile object installed at each base (base A, base B, base C) and autonomously running by remote operation from the display terminal 50. The robot 10 uses the display terminal 50 by moving within the base while capturing an object around the robot 10 with the imaging device 21 described below and transmitting the image acquired by the imaging device 21 to the display terminal 50. The information (image) in the base is provided to the operator who operates the robot 10. The robot 10 is an example of a moving object.

  The display terminal 50 is a terminal device such as a PC (Personal Computer) that remotely controls the robot 10 installed at each base (base A, base B, base C). The display terminal 50 displays an image transmitted from the robot 10. The operator can remotely control the robot 10 while viewing the image displayed on the display terminal 50.

  The display terminal 50 may be any device that has a display unit that displays an image transmitted from the robot 10. For example, a wearable terminal such as a tablet terminal, a mobile phone, a smartphone, and a head-mounted display (HMD) may be used. A communication terminal having a wide-angle screen (cylindrical, spherical, semi-spherical, or the like), a PDA (Personal Digital Assistant), or the like may be used.

  The management server 90 is a server computer for managing communication between the robot 10 located at each site and the display terminal 50. The management server 90 is connected to the robot 10 and the display terminal 50 via the communication network 9. The management server 90 may be composed of a plurality of server computers, and any server computer may have a function.

  The base where the robot 10 is installed is, for example, an office, a warehouse, a factory, a school, or the like. The operator who operates the robot 10 using the display terminal 50 checks the image in the base transmitted from the robot 10 to confirm the position and the flow line of the person in the base, and the device installed in the base. Management and maintenance. The robot 10 and the display terminal 50 can also perform two-way communication (remote conference) by transmitting and receiving images captured by both.

  Although FIG. 1 illustrates the configuration in which one robot 10 is installed in each site, a plurality of robots 10 may be installed in one site. The display terminal 50 may be configured to communicate with each of the robots 10 arranged at a plurality of bases, or may be configured to communicate only with the robot 10 arranged at one base.

Configuration of Robot 10 Here, a specific configuration of the robot 10 shown in FIG. 1 will be described with reference to FIGS. FIG. 2 is a diagram illustrating an example of a schematic configuration of the robot according to the embodiment.

  The robot 10 shown in FIG. 2 includes a moving mechanism 17 for moving the robot 10, a housing 15 including a control device 30 for controlling the processing or operation of the robot, a display 306a for displaying an image transmitted from the display terminal 50, and a display. The apparatus includes a support member 13 for supporting the support member 306a, a rotating shaft 12 for rotating the support member 13, an imaging device 21 for photographing an object around the robot 10, a microphone 308a for collecting sounds around the robot, and a speaker 308b.

  The moving mechanism 17 is a unit that moves the robot 10, and includes wheels, a traveling motor, a traveling encoder, a steering motor, a steering encoder, and the like. Since the movement control of the robot 10 is an existing technology, a detailed description thereof will be omitted. However, the robot 10 receives a traveling instruction from an operator (display terminal 50), and the moving mechanism 17 The robot 10 is moved based on the instruction.

  Hereinafter, an example in which the moving mechanism 17 includes two wheels will be described. However, the moving mechanism 17 may be a bipedal footprint or a single wheel. Further, the shape of the robot 10 is not limited to the vehicle type as shown in FIG. 2 and may be, for example, a bipedal humanoid, a form imitating a living thing, a form imitating a specific character, or the like.

  The housing 15 is located at the body of the robot 10, and includes a power supply unit that supplies necessary power to the entire robot 10, a control device 30 that controls processing or operation of the robot 10, and the like.

  The photographing device 21 is a camera arranged in front of the robot 10 and for photographing a subject around the robot 10 and acquiring image data 200. The image capturing device 21 may be an image capturing device that captures a part of the periphery of the robot 10 to obtain a planar image, or may capture an image of the periphery of the robot 10 to obtain a panoramic (360 °) panoramic image. It may be a special photographing device. Further, the image of the image data 200 may be a moving image or a still image, or may be both a moving image and a still image. Further, the image of the image data 200 may be a video including sound data together with the image.

  The support member 13 is fixed to the housing 15 and supports the display 306a. The support member 13 changes the direction of the display 306a by being rotated by the rotation shaft 12. The support member 13 may be a pole fixed to the housing 15 or a pedestal fixed to the housing 15. Further, the display 306a displays, for example, an image captured and acquired by the display terminal 50 of the other party with which the remote conference is performed.

  The robot 10 may have a configuration in which an operation unit that enables an additional operation other than the movement of the robot 10 is installed in addition to the above configuration. The operating means is, for example, a hand for grasping an object.

<< Modification of Robot 10 Configuration >> Modification 1
Here, a first modification of the configuration of the robot 10 will be described with reference to FIG. In the robot 10a shown in FIG. 3, the display 306a is arranged on the movable arm 11 by changing the support member 13. The movable arm 11 is a movable member that can be deformed to turn the display 306a in a specific direction. Thereby, the robot 10a can change the direction of the display 306a by changing the movable arm 11 in addition to changing the direction of the display 306a by the rotation shaft 12.

〇Modification 2
Next, a second modification of the configuration of the robot 10 will be described with reference to FIG. The robot 10b shown in FIG. 4 has a configuration obtained by removing the moving mechanism 17 from the configuration shown in FIG. The robot 10b shown in FIG. 4 is used by being fixed on a desk or the like. The robot 10b is an example of a transmission terminal.

Hardware Configuration Next, the hardware configurations of the robot 10, the display terminal 50, and the management server 90 in the embodiment will be described with reference to FIGS. The hardware configurations shown in FIGS. 5 to 7 may have the same configuration in each embodiment, and components may be added or deleted as needed.

<< Hardware Configuration of Robot 10 >> First, the hardware configuration of the robot 10 will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of a hardware configuration of the robot according to the embodiment. Note that the hardware configuration illustrated in FIG. 5 may have components added or deleted as needed. The robot 10 includes a control device 30 that controls processing or operation of the robot 10. The control device 30 is provided inside the housing 15 of the robot 10 as described above. The control device 30 may be provided outside the housing 15 of the robot 10 or may be provided as a device separate from the robot 10. The control device 30 is an example of an information processing device.

  The control device 30 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an HDD (Hard Disk Drive) 304, a network interface (I / F) 305, an input / output I / O An F / F 306, a media I / F 307, a sound input / output I / F 308, an external I / F 309, and a bus line 310 are provided.

  The CPU 301 controls the entire robot 10. The CPU 301 is an arithmetic unit that realizes each function of the robot 10 by reading out a program or data stored in the ROM 302 or an HD (Hard Disk) 304 a onto the RAM 303 and executing the processing. The remote control system 1a realizes the information processing method according to the present invention by causing the CPU 301 to execute a part or all of the program according to the present invention, for example.

  The RAM 303 is a volatile memory used as a work area or the like of the CPU 301. The ROM 302 is a non-volatile memory that can retain programs or data even when the power is turned off. The HDD 304 controls reading or writing of various data to the HD 304 a under the control of the CPU 301. The HD 304a stores various data such as programs.

  The network I / F 305 is a communication interface that performs communication (connection) with the display terminal 50 via the communication network 9. The network I / F 305 is a communication interface such as a wired or wireless LAN (Local Area Network). The network I / F 305 includes 3G (3rd Generation), LTE (Long Term Evolution), 4G (4rd Generation), 5G (5rd Generation), Zigbee (registered trademark), BLE (Bluetooth (registered trademark) Low Energy), A communication interface for millimeter wave wireless communication may be provided.

  The input / output I / F 306 is an interface for inputting and outputting characters, numerical values, various instructions, and the like to and from various external devices. The input / output I / F 306 controls display of various information such as a cursor, a menu, a window, characters, and images on a display 306a such as an LCD (Liquid Crystal Display). The display 306a may be a touch panel display including an input unit. The input / output I / F 306 may be connected to input means such as a mouse and a keyboard in addition to the display 306a.

  The media I / F 307 controls reading or writing (storage) of data from or to a recording medium 307a such as a USB (Universal Serial Bus) memory, a memory card, an optical disk, or a flash memory. The sound input / output I / F 308 is a circuit that processes input / output of a sound signal between the microphone 308a and the speaker 308b under the control of the CPU 301. The external I / F 309 is an interface for connecting the control device 30 to another device.

  The bus line 310 is an address bus, a data bus, or the like for electrically connecting the above components, and transmits an address signal, a data signal, various control signals, and the like. The CPU 301, the ROM 302, the RAM 303, the HDD 304, the network I / F 305, the input / output I / F 306, the media I / F 307, the sound input / output I / F 308, and the external I / F 309 are interconnected via a bus line 310.

  Further, the control device 30 is provided with a moving motor 101, an actuator 102, an acceleration / direction sensor 103, a GPS receiving unit 104, a power supply unit 105, various sensors 106, a timer 107, and the above-described imaging via an external I / F 309. The device 21 is connected.

  The moving motor 101 rotates the moving mechanism 17 based on a command from the CPU 301 to move the robot 10 along the ground. The actuator 102 deforms the movable arm 11 based on a command from the CPU 301. The acceleration / azimuth sensor 103 is a sensor such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. A GPS (Global Positioning System) receiving unit 104 receives a GPS signal from a GPS satellite. The power supply unit 105 is a unit that supplies necessary power to the entire robot 10. The various sensors 106 are sensor devices that can detect information around the robot 10. The various sensors 106 are, for example, a barometer, a thermometer, a photometer, a human sensor, or an illuminometer. The timer 107 is a measuring device having a time measuring function. The timer 107 may be a soft timer by a computer.

<< Hardware Configuration of Display Terminal 50 >> Next, a hardware configuration of the display terminal 50 will be described with reference to FIG. FIG. 6 is a diagram illustrating an example of a hardware configuration of the display terminal according to the embodiment. The display terminal 50 includes a CPU 501, a ROM 502, a RAM 503, an EEPROM (Electrically Erasable Programmable Read-Only Memory) 504, an image sensor I / F 505, a CMOS (Complementary Metal Oxide Semiconductor) sensor 505a, an acceleration / azimuth sensor 506, a media I / F 507, A GPS receiver 508 is provided.

  The CPU 501 controls the operation of the entire display terminal 50. The CPU 501 is an arithmetic device that realizes each function of the display terminal 50 by reading out a program or data stored in the ROM 502 or the like onto the RAM 503 and executing a process. The remote control system 1a realizes the information processing method according to the present invention by causing the CPU 501 to execute a part or all of the program according to the present invention, for example.

  The ROM 502 stores a program used for driving the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area of the CPU 501. The EEPROM 504 reads or writes various data such as a display terminal program under the control of the CPU 501.

  The CMOS sensor 505a captures a subject (mainly a self-portrait) under the control of the CPU 501 to obtain image data. The image sensor I / F 505 is a circuit that controls driving of the CMOS sensor 505a. The acceleration / azimuth sensor 506 is various sensors such as an electronic magnetic compass, a gyro compass, and an acceleration sensor for detecting geomagnetism. The media I / F 507 controls reading or writing (storage) of data from or to a recording medium 507a such as a flash memory. The GPS receiving unit 508 receives a GPS signal from a GPS satellite.

  The display terminal 50 includes a long-distance communication circuit 510, an antenna 510a of the long-distance communication circuit 510, a sound input / output I / F 511, a microphone 511a, a speaker 511b, a display 512, an external device connection I / F 513, and a short-distance communication circuit 514. , A short-range communication circuit 514, an antenna 514a, a touch panel 515, a timer 516, and a line-of-sight detection device 517.

  The long-distance communication circuit 510 is a circuit that communicates with another device via the communication network 9. The microphone 511a is a type of a built-in sound collecting unit for inputting sound. The sound input / output I / F 511 is a circuit that processes input / output of a sound signal between the microphone 511a and the speaker 511b under the control of the CPU 501.

  The display 512 is a type of display means such as a liquid crystal display or an organic EL display that displays an image of a subject and various icons. The external device connection I / F 513 is an interface for connecting various external devices. The short-range communication circuit 514 is a communication circuit such as Wi-Fi, NFC, or Bluetooth. The touch panel 515 is a type of input unit that operates the display terminal 50 when the user presses the display 512. The timer 516 is a measuring device having a time measuring function. Timer 516 may be a soft timer by a computer. The gaze detection device 517 continuously detects the position of the gaze of the user as gaze information. The eye gaze detecting device 517 includes, for example, an image processing device that analyzes an image captured by the CMOS sensor 505a. The eye gaze detecting device 517 detects the direction of the eye gaze from the positional relationship between the eye and the iris, for example, with the reference point as the inner eye.

  The display terminal 50 includes a bus line 509. The bus line 509 is an address bus, a data bus, and the like for electrically connecting each component such as the CPU 501.

<< Hardware Configuration of Management Server 90 >> Subsequently, a hardware configuration of the management server 90 will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of a hardware configuration of the management server according to the embodiment. The management server 90 is constructed by a general computer. The management server 90 includes a CPU 901, a ROM 902, a RAM 903, an HDD 905, a media I / F 907, a display 908, a network I / F 909, a keyboard 911, a mouse 912, a CD-RW (Compact Disc-ReWritable) drive 914, a timer 915, and a bus line 910. It has. In addition, since the management server 90 functions as a server, the management server 90 may not include an input device such as the keyboard 911 and the mouse 912 or an output device such as the display 908.

  The CPU 901 controls the operation of the entire management server 90. The ROM 902 stores a program used for driving the CPU 901. The RAM 903 is used as a work area of the CPU 901. The HDD 905 controls reading or writing of various data from or to the HD 904 under the control of the CPU 901. The HD 904 stores various data such as programs. A media I / F 907 controls reading or writing (storage) of data from or to a recording medium 906 such as a flash memory.

  The display 908 displays various information such as a cursor, a menu, a window, characters, and an image. The network I / F 909 is an interface for performing data communication using the communication network 9. The keyboard 911 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 912 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 914 controls reading of various data from a CD-RW 913 as an example of a removable recording medium. The timer 915 is a measuring device having a time measuring function. Timer 915 may be a computer soft timer.

  The management server 90 includes a bus line 910. The bus line 910 is an address bus, a data bus, and the like for electrically connecting each component such as the CPU 901 shown in FIG.

● First embodiment ●
Next, a configuration of the remote control system according to the first embodiment will be described with reference to FIGS.

First, the functional configuration of the remote control system according to the first embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of a functional configuration of the remote control system according to the first embodiment.

<< Functional Configuration of Control Device 30 >> First, the functional configuration of the control device 30 that controls the processing or operation of the robot 10 will be described with reference to FIG. The functions realized by the control device 30 include a transmission / reception unit 31, an operation input reception unit 32, a display control unit 33, a determination unit 34, a shooting instruction unit 35, an image acquisition unit 36, a movement control unit 37, a direction adjustment unit 38, a position It includes a specifying unit 39, an image processing unit 41, a point of interest calculation unit 42, a target point calculation unit 43, a peripheral information detection unit 44, a presence notification unit 45, a storage / readout unit 46, and a storage unit 3000.

  The transmission / reception unit 31 has a function of transmitting / receiving various data or information to / from another device via the communication network 9. The transmission / reception unit 31 transmits the image data 200 acquired by the image acquisition unit 36 to the display terminal 50 via the communication network 9, for example. The transmission / reception unit 31 receives, for example, a request command (request information) transmitted from the display terminal 50 via the communication network 9. Further, the transmitting / receiving unit 31 receives, for example, image data 250 that is an image acquired by the display terminal 50 from the display terminal 50 via the communication network 9. In addition, the transmission / reception unit 31 transmits the image data 200 obtained by the imaging device 21 at the target point calculated by the target point calculation unit 43 to the display terminal 50 via the communication network 9. The transmission / reception unit 31 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the network I / F 305. The transmission / reception unit 31 is an example of a first transmission unit.

  The operation input receiving unit 32 has a function of receiving an operation input to an input unit such as the display 306a. The operation input receiving unit 32 is mainly realized by the processing of the CPU 301 and the input / output I / F 306 shown in FIG. The display control unit 33 has a function of displaying various screens on the display 306a. The display control unit 33 is realized mainly by the processing of the CPU 301 and the input / output I / F 306 shown in FIG.

  The determination unit 34 has a function of determining a process or an operation to be executed by the robot 10 in response to a request command transmitted from the display terminal 50. The determination unit 34 is mainly realized by the processing of the CPU 301 shown in FIG.

  The imaging instruction unit 35 has a function of instructing the imaging device 21 to perform imaging processing. The imaging instruction unit 35 transmits, for example, instruction information for instructing the imaging device 21 to perform imaging, to the imaging device 21. The imaging instruction unit 35 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309.

  The image acquisition unit 36 has a function of acquiring an image acquired by the imaging device 21. The image acquiring unit 36 acquires, for example, image data 200 which is an image acquired by photographing the subject by the photographing device 21 from the photographing device 21. The image acquisition unit 36 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309. The image acquisition unit 36 is an example of an acquisition unit.

  The movement control unit 37 has a function of controlling the movement of the robot 10 by driving the movement mechanism 17. The movement control unit 37 moves the robot 10 by controlling the driving of the movement mechanism 17 in response to a request command transmitted from the display terminal 50, for example. Further, the movement control unit 37 controls the driving of the moving mechanism 17 so as to move the robot 10 to the target point calculated by the target point calculation unit 43, for example. The movement control unit 37 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309. The movement control unit 37 is an example of a movement control unit.

  The direction adjusting unit 38 has a function of changing the direction of the robot 10. The direction adjusting unit 38 controls the drive of the moving mechanism 17 or the rotating shaft 12 according to the direction of the robot 10 to which the robot 10 is moved, for example, calculated by the target point calculating unit 43. To change. The direction adjustment unit 38 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309.

  The position specifying unit 39 has a function of specifying the current position of the robot 10. The position specifying unit 39 specifies the current position of the robot 10 based on a detection result of each direction (azimuth angle, magnetic north) detected by the acceleration / azimuth sensor 103 or the GPS receiving unit 104, for example. The detection result such as the direction of each direction is position information indicating the position and direction of the robot 10 at a predetermined time. The position specifying unit 39 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309. The position specifying unit 39 is an example of a position detecting unit.

  The image processing unit 41 has a function of performing processing on the image data 200 obtained by the image obtaining unit 36. The image processing unit 41 extracts, for example, an image of a person's head included in the image data 200 acquired by the image acquisition unit 36. The image processing unit 41 extracts, for example, the eyes and nose images of the person acquired by the image acquiring unit 36. The image processing unit 41 is realized mainly by the processing of the CPU 301 shown in FIG.

  The attention point calculation unit 42 has a function of specifying an attention point to which a plurality of people pays attention based on the positions and orientations of the plurality of people included in the image data processed by the image processing unit 41. The point of interest is a position where a plurality of people are paying attention at the base where the robot 10 is installed, for example, an intersection in a direction in which a plurality of people around the robot 10 are facing. The point of interest is, for example, the direction in which a plurality of participants at the base on the robot 10 side are facing in the remote conference, that is, the center position of the topic at a certain point in time. The attention point calculation unit 42 is realized mainly by the processing of the CPU 301 shown in FIG. The attention point calculation unit 42 is an example of a calculation unit. In addition, the attention point calculation unit 42 is an example of an estimation unit. Furthermore, the attention point calculation unit 42 is an example of an attention point identification unit.

  The target point calculation unit 43 has a function of specifying a target point to which the robot 10 moves based on the target point calculated by the target point calculation unit 42. The target point specified by the target point calculation unit 43 includes position information (coordinate information) indicating the target position G of the destination of the robot 10 and direction information indicating the direction of the robot 10 of the destination of the robot 10. The target point calculation unit 43 is realized mainly by the processing of the CPU 301 shown in FIG. The target point calculation unit 43 is an example of a calculation unit. The target point calculation unit 43 is an example of a target point specifying unit.

  The peripheral information detection unit 44 has a function of detecting information around the robot 10 using the image data 200 acquired by the imaging device 21 or the detection results detected by various sensors 106. The peripheral information detection unit 44 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309.

  The presence notification unit 45 has a function of notifying the presence of the robot 10 based on a predetermined condition. The presence notification unit 45 notifies the presence of the robot 10 when, for example, the distance between the current position detected by the position identification unit 39 and the target point calculated by the target point calculation unit 43 is equal to or greater than the threshold or longer than the threshold. Output notification information. The notification information is, for example, a sound signal, and the presence notification unit 45 outputs a sound signal from the speaker 308b. The notification information is, for example, a predetermined gesture of the robot 10, and the presence notification unit 45 drives the moving mechanism 17, the movable arm 11, or the rotating shaft 12. The threshold is stored in the storage unit 3000 in advance. The presence notification unit 45 is mainly realized by the processing of the CPU 301 illustrated in FIG. 5 and the external I / F 309. The presence notification unit 45 is an example of a presence notification unit.

  The storage / readout unit 46 has a function of storing various data in the storage unit 3000 or reading various data from the storage unit 3000. The storage / readout unit 46 is realized mainly by the processing of the CPU 301 shown in FIG. The storage unit 3000 is mainly realized by the ROM 302, the HD 304a, and the recording medium 307a illustrated in FIG.

  The storage unit 3000 stores image data 200 captured by the image acquisition unit 36 and image data 250 transmitted from the display terminal 50. Further, the storage unit 3000 stores a command table 3001 and a condition table 3002. The image data 200 stored in the storage unit 3000 may be deleted when a predetermined time has elapsed since the image data was acquired by the image acquisition unit 36, or the data transmitted to the display terminal 50 may be deleted. May be deleted. Similarly, the configuration may be such that the image data 250 stored in the storage unit 3000 is deleted when a predetermined time has elapsed since the image data 250 was transmitted from the display terminal 50.

〇 Command table 3001
Here, details of the data stored in the storage unit 3000 will be described. FIG. 9 is a diagram illustrating an example of the command table according to the first embodiment. The command table 3001 shown in FIG. 9 is for specifying a process or an operation to be executed by the robot 10 based on a request command transmitted from the display terminal 50. The command table 3001 stores the processing contents corresponding to each request command in association with each request command. Using the command table 3001, the determination unit 34 of the control device 30 specifies a process corresponding to the request command transmitted from the display terminal 50.

  For example, in the command table 3001, a process corresponding to the request command “FORWARD” is a process of moving the robot 10 forward by a certain amount. Note that the content of the processing corresponding to each command is not limited to this, and can be appropriately modified or changed by an operator who operates the robot 10 using the display terminal 50 or an administrator of the robot 10.

〇 Condition table 3002
FIG. 10 is a diagram illustrating an example of imaging parameters according to the first embodiment. A condition table 3002 shown in FIG. 10 shows conditions for starting a position adjustment process for a point of interest. The condition table 3002 stores condition information in association with each site indicated by the site name. The condition information includes an elapsed time since the previous position adjustment processing was performed, a distance to a person, and an adjustment cycle of the position adjustment processing. Note that the items included in the condition information are not limited thereto, and the condition of one item may be included for each site, or the condition of a plurality of items may be included for each site.

  The condition table 3002 stores the condition information for each site, so that the frequency and conditions for performing the position adjustment process can be changed for each site where the robot 10 is located. Although the condition table 3002 has described an example in which the condition information is associated with each base, the condition information may be stored in association with each robot 10.

<< Functional Configuration of Display Terminal 50 >> Next, a functional configuration of the display terminal 50 will be described with reference to FIG. The functions implemented by the display terminal 50 include a transmission / reception unit 51, an operation input reception unit 52, a display control unit 53, a determination unit 54, a request command generation unit 55, a line-of-sight detection unit 56, a storage / readout unit 57, and a storage unit 5000. Including. The display terminal 50 has installed a dedicated application program for remotely controlling the robot 10. The display terminal 50 realizes each function by the CPU 501 executing the installed application program, for example.

  The transmission / reception unit 51 has a function of transmitting / receiving various data or information to / from another device via the communication network 9. The transmission / reception unit 51 receives, for example, image data 200 acquired by the imaging device 21 included in the robot 10 from the robot 10 via the communication network 9. The transmitting / receiving unit 51 transmits, for example, image data 250 that is an image captured and acquired by the CMOS sensor 505a to the robot 10 via the communication network 9. The transmission / reception unit 51 is mainly realized by the processing of the CPU 501 illustrated in FIG. The transmitting / receiving unit 51 is an example of a receiving unit.

  The operation input receiving unit 52 has a function of receiving various selections or operation inputs to the display terminal 50. The operation input receiving unit 52 is mainly realized by the processing of the CPU 501 illustrated in FIG. Note that the touch panel 515 may be shared with the display 512. Further, the operation input receiving unit 52 may be realized by an input unit other than the touch panel.

  The display control unit 53 has a function of displaying various images on the display 512 of the display terminal 50. The display control unit 53 causes the display 512, which is an example of a display unit, to display the image data 200 received by the transmission / reception unit 51, for example. The display control unit 53 is mainly realized by the processing of the CPU 501 illustrated in FIG. The display control unit 53 is an example of a display control unit.

  The determination unit 54 has a function of determining a specific process requested to the robot 10. The determination unit 54 determines a specific process to request the robot 10 based on the operation input received by the operation input reception unit 52, for example. The determination unit 54 is mainly realized by the processing of the CPU 301 shown in FIG.

  The request command generation unit 55 has a function of generating a request command that is an execution request for the robot 10 to execute a specific process. The request command generation unit 55 is realized mainly by the processing of the CPU 501 shown in FIG.

  The gaze detection unit 56 has a function of detecting the gaze of the operator who operates the robot 10 using the display terminal 50. The line-of-sight detection unit 56 is mainly realized by the processing of the CPU 501 illustrated in FIG.

  The storage / readout unit 57 has a function of storing various data in the storage unit 5000 or reading various data from the storage unit 5000. The storage / readout unit 57 is realized mainly by the processing of the CPU 501 shown in FIG. The storage unit 5000 is mainly realized by the ROM 502, the EEPROM 504, and the recording medium 507 illustrated in FIG.

  The storage unit 5000 stores the image data 200 received by the transmission / reception unit 51 and the image data 250 captured and acquired by the CMOS sensor 505a. Further, the storage unit 5000 stores a user command table 5001. Note that the image data 200 stored in the storage unit 5000 may be deleted when a predetermined time has elapsed after being received by the transmission / reception unit 51, or may be received by the operation input reception unit 52. Alternatively, the configuration may be such that deletion is performed based on a deletion command from a user. Similarly, the image data 250 stored in the storage unit 5000 may be deleted when a predetermined time has elapsed since the image data 250 was acquired by the CMOS sensor 505a, or may be received by the operation input receiving unit 52. Alternatively, the configuration may be such that deletion is performed based on a deletion command from a user.

〇User command table 5001
FIG. 11 is a diagram illustrating an example of the user command table according to the first embodiment. The user command table 5001 shown in FIG. 11 is used to specify the content of a process or an operation requested to the robot 10 based on the operation input received by the operation input receiving unit 52. The user command table 5001 stores an input command corresponding to an operation input received by the operation input receiving unit 52 in association with the content of the corresponding command.

<< Functional Configuration of Management Server 90 >> Next, a functional configuration of the management server 90 will be described with reference to FIG. The functions realized by the management server 90 include a transmission / reception unit 91, an authentication unit 92, a determination unit 93, a creation unit 94, a storage / readout unit 95, and a storage unit 9000.

  The transmission / reception unit 91 has a function of transmitting / receiving various data or information to / from another device via the communication network 9. The transmission / reception unit 91 is mainly realized by the processing of the CPU 901 illustrated in FIG. 7 and the network I / F 909.

  The authentication unit 92 has a function of authenticating a login request source based on the login request received by the transmission / reception unit 91. The authentication unit 92 searches the authentication management DB 9001 of the storage unit 9000 using the terminal ID and the password included in the login request received by the transmission / reception unit 91 as search keys. The authentication unit 92 performs terminal authentication by determining whether the same set of terminal ID and password is managed in the authentication management DB 9001. The authentication unit 92 is realized mainly by the processing of the CPU 901 shown in FIG.

  The determination unit 93 has a function of determining whether the terminal ID of the display terminal 50 is managed in a session management table described later. The determining unit 93 is mainly realized by the processing of the CPU 901 shown in FIG.

  The creation unit 94 has a function of creating a session ID used for communication. The creating unit 94 is realized mainly by the processing of the CPU 901 shown in FIG.

  The storage / readout unit 95 has a function of storing various data in the storage unit 9000 or reading various data from the storage unit 9000. The storage / readout unit 95 is realized mainly by the processing of the CPU 901 shown in FIG. The storage unit 9000 is mainly realized by the ROM 902, the HD 904, or the recording medium 906 illustrated in FIG. The storage unit 9000 also includes destination list frame data (address list content information such as icons, “rA01”, and “robot 10A-1” shown in FIG. 15) on a destination list screen 500 shown in FIG. Is not remembered.

〇Authentication Management Table The storage unit 9000 has an authentication management DB 9001 configured by an authentication management table as shown in FIG. In this authentication management table, related information in which each password is associated with each terminal ID of all the display terminals 50 managed by the management server 90 is managed. For example, the authentication management table shown in FIG. 12A indicates that the terminal ID of the display terminal 50A is “o01” and the password is “aaaa”.

Terminal management table In the storage unit 9000, a terminal management DB 9002 composed of a terminal management table as shown in FIG. In the terminal management table, for each terminal ID of each terminal (robot 10 and display terminal 50), the terminal name of each terminal, the IP address of the terminal, operating state information indicating the current operating state of each terminal, and the terminal In the case of the robot 10, related information in which a base name indicating a base where the robot 10 is located is managed.

  For example, in the terminal management table shown in FIG. 12B, the display terminal 50 having the terminal ID “o01” has the terminal name “display terminal 50A” and the IP address of the display terminal 50 is “1.2.1”. ..3 "indicates that the operating state is" online (communicable) ". In addition, the robot 10 having the terminal ID “rA01” has the terminal name “robot 10A-1”, the IP address of the robot 10 is “1.3.2.3”, and the operation state is “online (communicable). ) "Indicates that the site name is" site A ".

{Destination List Management Table} Further, in the storage unit 9000, a destination list management DB 9003 configured by a destination list management table as shown in FIG. In this destination list management table, for each terminal ID of the display terminal 50 as a start terminal for requesting the start of communication for remote control of the robot 10, a destination candidate registered as a candidate for the robot 10 as a destination Related information associated with the terminal ID of the robot 10 is managed.

  For example, in the destination list management table shown in FIG. 13A, a destination candidate that can request a start of communication from a start terminal (display terminal 50A) whose terminal ID is “o01a” is a terminal ID “rA01”. The robot 10A-1 has a terminal ID of "rA02", the robot 10A-2 has a terminal ID of "rC01", and the like. Note that the terminal ID of the destination candidate robot 10 is updated by being added or deleted by a request for addition or deletion from the arbitrary start terminal (display terminal 50) to the management server 90.

{Session Management Table} Further, in the storage unit 9000, a session management DB 9004 including a session management table as shown in FIG. The session management table includes, for each session ID for identifying a session used when communicating between the robot 10 and the display terminal 50, the robot 10 using the session specified by the session ID, Related information associated with the terminal ID of the display terminal 50 is managed. For example, in the session management table shown in FIG. 13B, the terminal using the session executed using the session ID “se1” is the display terminal 50A with the terminal ID “o01” and the terminal ID is “rA01”. It is shown that the robot 10A-1 is the robot 10A-1 and the terminal ID is the robot 10C-1 of "rC01".

● Process or Operation of First Embodiment {Session Establishment Process} Next, the operation or process of the remote control system according to the first embodiment will be described with reference to FIGS. In the following description, the processing executed by the control device 30 included in the robot 10 will be described as the processing executed by the robot 10.

  First, a process of establishing a communication session between the robot 10 and the display terminal 50 will be described with reference to FIGS. FIG. 14 is a sequence diagram illustrating an example of a process in a preparation stage for starting transmission and reception of data between the robot and the display terminal. Here, a description will be given of a process of transmitting and receiving management information in a preparation stage before starting transmission and reception of data between the display terminal 50A as a start terminal and the robot 10A-1 as a destination terminal.

  First, in step S11, the transmission / reception unit 51 of the display terminal 50A transmits a login request to the management server 90 via the communication network 9. Specifically, when the user of the display terminal 50A turns on the power switch of the display terminal 50A, the power turns on. Then, the transmission / reception unit 51 of the display terminal 50A transmits a login request from the transmission / reception unit 51 to the management server 90 via the communication network 9 when the power is turned on. Thereby, the transmission / reception unit 91 of the management server 90 receives the login request transmitted from the display terminal 50A.

  This login request includes a terminal ID for identifying the start terminal as the display terminal 50A, and a password. The terminal ID and the password are data read from the storage unit 5000 by the storage / readout unit 57 and sent to the transmission / reception unit 51. Note that these terminal IDs and passwords are not limited to these, and terminal IDs and passwords input by the user through input means such as the touch panel 515 may be transmitted. Further, a terminal ID and a password read from a storage medium such as a SIM (Subscriber Identity Module Card) card or an SD card connected to the display terminal 50A may be transmitted.

  When a login request is transmitted from the display terminal 50A to the management server 90, the management server 90 on the receiving side can acquire the IP address of the display terminal 50A on the transmitting side. The start of the login request does not necessarily need to be triggered by turning on the power switch, and may be transmitted in response to an input to an input unit such as the touch panel 515 by the user.

  Next, in step S12, the authentication unit 92 of the management server 90 uses the terminal ID and the password included in the login request received via the transmission / reception unit 91 as a search key, as an authentication management table (FIG. 12). (A), and performs authentication by determining whether the same terminal ID and the same password are managed in the authentication management DB 9001. Hereinafter, a case will be described where the display unit 50A is determined by the authentication unit 92 to be a terminal having a valid use right.

  In step S13, when the same terminal ID and the same password are managed by the authentication unit 92, the storage / readout unit 95 determines that the log-in request is from a start terminal having valid use authority. Then, the destination list frame data is read from the storage unit 9000.

  In step S14, the transmission / reception unit 91 transmits the authentication result information indicating the authentication result obtained by the authentication unit 92 to the display terminal 50 that has made the login request via the communication network 9. Thereby, the transmitting / receiving unit 51 of the display terminal 50A receives the authentication result information. This authentication result information includes the destination list frame data read out in step S13. Then, in step S15, the storage / readout unit 57 of the display terminal 50A causes the storage unit 5000 to store the destination list frame data received in step S14.

  Next, in step S16, when the transmission / reception unit 51 of the display terminal 50A receives the authentication result information indicating the authentication result determined to be a terminal having a valid use right, the management unit 51 manages the information via the communication network 9. The server 90 requests the contents of the destination list. Thereby, the transmission / reception unit 91 of the management server 90 receives the request for the contents of the destination list. This request includes the terminal ID of the display terminal 50A.

  Next, in step S17, the storage / readout unit 95 of the management server 90 uses the terminal ID “o01” of the display terminal 50A received in step S16 as a search key and a destination list management table (FIG. 13A). The reference is searched, and the terminal IDs of all corresponding destination candidates are read. Further, in step S18, the storage / readout unit 95 searches the terminal management table (see FIG. 12B) using each terminal ID read out in step S17 as a search key, and finds the corresponding terminal candidate terminal name, operation Read the status information and the site name.

  Next, in step S19, the transmission / reception unit 91 of the management server 90 transmits the destination list content information to the display terminal 50A via the communication network 9. Thereby, the transmission / reception unit 51 of the display terminal 50A receives the destination list content information. The destination list content information includes the terminal ID of the destination candidate, the terminal name of the destination candidate, the operating state information, and the base name read out in steps S17 and S18.

  Next, in step S20, the display control unit 53 of the display terminal 50A is created using the destination list frame data stored in the storage unit 5000 in step S15 and the destination list content information received in step S19. The displayed destination list screen 500 is displayed on the display 512.

  On the display 512, a destination list screen 500 as shown in FIG. 15 is displayed. On the destination list screen 500, for each destination candidate, an icon indicating the operating status of the destination candidate terminal (robot 10), the terminal ID of the destination candidate terminal, the destination candidate destination name, and the destination candidate terminal are displayed. The name of the base to be used is displayed. The “terminal name” transmitted from the management server 9 in step S19 is displayed as a “destination name” on the destination list screen 500 shown in FIG.

  Next, processing from selection of a destination candidate on the display terminal 50 to start of transmission / reception of image data will be described with reference to FIG. FIG. 16 is a sequence diagram showing a process from selection of a destination candidate to start of transmission / reception of image data.

  First, in step S31, the operation input receiving unit 52 of the display terminal 50A receives selection of a destination candidate (here, the robot 10A-1) on the destination list screen shown in FIG. 15 from the user. Then, in step S32, the transmission / reception unit 51 of the display terminal 50A transmits a start request indicating that transmission / reception of image data or the like is to be started to the management server 90. Thereby, the transmission / reception unit 91 of the management server 90 receives the start request. The start request includes the terminal ID of the display terminal 50A and the terminal ID of the destination candidate terminal.

  Next, in step S33, the determination unit 93 of the management server 90 determines whether or not the terminal ID of the display terminal 50A received in step S32 is managed in the session management table (see FIG. 13B). I do. Here, the case where the terminal ID of the destination terminal (robot 10A-1) is not managed will be described below.

  In step S34, when the terminal ID of the terminal candidate terminal is not managed, the creating unit 94 of the management server 90 creates a new session ID. In step S35, the storage / readout unit 95 stores the session ID created in step S34 and the terminal ID of the display terminal 50A received in step S32 in the session management table (see FIG. 13B). And a new record in which the terminal ID of the destination candidate terminal is associated. Here, as shown in FIG. 12B, by adding a new record, the session ID “se3” and the terminal IDs “o01” and “rA01” are managed in association with each other.

  Next, in step S36, the transmitting and receiving unit 91 of the management server 90 transmits a session start instruction including the session ID created in step S34 to the display terminal 50A. Thereby, the transmitting / receiving unit 51 of the display terminal 50A receives the session start instruction.

  Next, in step S37, the storage / readout unit 95 of the management server 90 uses the terminal ID of the destination candidate terminal (robot 10A-1) sent in step S32 as a search key to create a terminal management table (FIG. b) to read out the corresponding IP address. Then, in step S38, the transmitting / receiving unit 91 of the management server 90 transmits a session start instruction including the session ID created in step S33 to the IP address read in step S37. Thereby, the transmission / reception unit 31 of the destination terminal (robot 10A-1) receives the session start instruction.

  As described above, the start terminal (display terminal 50A) and the destination terminal (robot 10A-1) respectively establish communication sessions with the management server 90 (steps S39-1 and S39-2).

<< Transmission and Display of Image Data >> Subsequently, using a session established with the management server 90, data transmitted from the robot 10 to the display terminal 50 and processing or operation of the robot 10 by the display terminal 50 are described. The control process will be described. FIG. 17 is a flowchart illustrating an example of a robot control process based on a request command from a display terminal in the remote control system according to the first embodiment.

  In step S51, the image acquisition unit 36 of the robot 10 acquires the image data 200 acquired by the imaging device 21. Here, the robot 10 starts photographing by the photographing device 21 with a photographing instruction from the photographing instruction unit 35 to the photographing device 21 as a trigger. The image acquisition unit 36 of the robot 10 acquires image data 200 that is an image acquired by the imaging device 21 from the imaging device 21.

  In steps S52-1, 52-2, the transmission / reception unit 31 of the robot 10 transmits the image data 200 acquired by the image acquisition unit 36 to the display terminal 50 using the established communication session with the management server 90. I do. Thereby, the transmission / reception unit 51 of the display terminal 50 receives the image data 200.

  In step S53, the display control unit 53 of the display terminal 50 displays the image data 200 received by the transmission / reception unit 51 on the display screen 600 of the display 512. Thus, the operator who remotely controls the robot 10 using the display terminal 50 can check the status of the base where the robot 10 is located while viewing the display screen 600 on which the image data 200 is displayed. Similarly, the display terminal 50 transmits, to the robot 10, image data 250 that is an image of a base on the display terminal 50 side. Then, the transmitting / receiving unit 31 of the robot 10 receives the image data 250, and the display control unit 33 displays the received image data 250 on the display screen 400 of the display 306a.

  Here, a display screen displayed on the robot 10 and the display terminal 50 will be described. FIG. 18A is a diagram illustrating an example of a screen example of a display screen displayed on the robot according to the first embodiment. In the robot 10, the display control unit 33 causes the display 306a to display a display screen 400 as shown in FIG. On the display screen 400, the image data 250 transmitted from the display terminal 50 is displayed. As shown in FIG. 18A, on the display screen 400, for example, the face of an operator who operates the robot 10 using the display terminal 50 is displayed. It is possible to confirm who is operating the robot 10.

  FIG. 18B is a diagram illustrating an example of a screen example of a display screen displayed on the display terminal according to the first embodiment. In the display terminal 50, the display control unit 53 causes the display 512 to display a display screen 600 as shown in FIG. On the display screen 600, the image data 200 transmitted from the robot 10 is displayed. As shown in FIG. 18B, for example, people around the robot 10 are displayed on the display screen 600. In addition, the display screen 600 displays a movement instruction key 601 and an input area 603a that are pressed when requesting horizontal movement (forward, backward, right rotation, left rotation) of the robot 10 beside the image of the image data 200. "Move" button 603b to be pressed when requesting movement of the robot 10 to the position input to the "Rotation" button pressed to change the direction of the robot 10 to the direction input to the input area 605a. 605b, a “stop” button 607 pressed to stop the robot 10, and a “♪” button 609 pressed to emit a sound from the speaker 308b of the robot 10. The operator who operates the robot 10 using the display terminal 50 controls the movement of the robot 10 by looking at the image data 200 displayed on the display screen 600 and performing an input operation on the movement instruction key 601 and the like. .

  In FIG. 18B, an example in which the movement of the robot 10 is operated by the movement instruction key 601 displayed on the display screen 600 has been described, but the movement operation of the robot 10 is performed by a game provided with a keyboard or a joystick. The configuration may be performed by a dedicated controller such as a pad.

  Referring back to FIG. 17, in step S54, the operation input receiving unit 52 of the display terminal 50 receives a specific operation input on the display screen 600 displayed on the display 512. For example, the operation input receiving unit 52 receives an operation input to any one of the movement instruction keys 601 included in the display screen 600, for example.

  In step S55, the request command generation unit 55 of the display terminal 50 generates a request command corresponding to a specific operation input received by the operation input reception unit 52. Specifically, when a specific operation input is received by operation input receiving unit 52, storage / read unit 57 of display terminal 50 reads user command table 5001 (see FIG. 11) stored in storage unit 5000. . The determination unit 54 of the display terminal 50 searches the user command table 5001 read by the storage / read unit 57 and specifies an input command corresponding to the operation input received by the operation input reception unit 52. Then, the determining unit 54 searches the user command table 5001 and extracts a command corresponding to the specified input command. Then, the request command generation unit 55 generates a request command including the extracted command. For example, when an input to the “↑ (forward)” key of the movement instruction key 601 is received by the operation input receiving unit 52, the determination unit 54 specifies “pressing the“ ↑ (forward) ”key” as an input command. . Then, the determining unit 54 extracts “FORWARD” as a command corresponding to the specified input command. Then, the request command generator 55 generates a request command including the extracted command “FORWARD”.

  Although the example in which the determination unit 54 extracts the process using the user command table 5001 in step S55 has been described, the display terminal 50 performs the predetermined process based on the event given to the input key such as the movement instruction key 605. May be extracted or executed.

  In steps S56-1 and 56-2, the transmission / reception unit 51 of the display terminal 50 transmits the request command generated by the request command generation unit 55 to the robot 10 using the established communication session with the management server 90. I do. Thereby, the transmission / reception unit 31 of the robot 10 receives the request command.

  In step S57, the storage / read unit 46 of the robot 10 reads the command table 3001 (see FIG. 9) stored in the storage unit 3000. In step S58, the determination unit 34 of the robot 10 searches the command table 3001 read by the storage / read unit 46, and extracts a process corresponding to the request command received by the transmission / reception unit 31. In step S59, the robot 10 executes the processing extracted in step S58.

  For example, when the request command received by the transmission / reception unit 31 is “FORWARDMOVE”, the determination unit 34 searches the command table 3001 and extracts a process corresponding to the request command “FORWARD”. In this case, the process to be extracted is “the robot 10 is advanced by a certain amount”. The determination unit 34 notifies the movement control unit 37 of the execution request of the extracted process. Then, the movement control unit 37 drives the movement mechanism 17 of the robot 10 based on the processing extracted by the determination unit 34. Note that the notification destination of the execution request differs depending on the process extracted by the determination unit 34. For example, when the request command received by the transmission / reception unit 31 is “SOUND”, the notification destination of the execution request is the presence notification unit 45.

  Thereby, the remote control system 1a displays the image data 200 acquired by the photographing device 21 included in the robot 10 on the display terminal 50, and provides the operator who remotely controls the robot 10 using the display terminal 50. Thus, information around the robot 10 can be more accurately grasped. Further, the remote control system 1a can move the robot 10 while confirming the situation around the robot 10 by performing an operation input by the operator on the display terminal 50 on which the image data 200 is displayed. The operability for the operator who operates the robot 10 using the display terminal 50 can be improved.

<< Position Adjustment Process of Robot 10 >> Target Point Calculation Process Next, the position adjustment process of the robot 10 will be described with reference to FIGS. Here, it is assumed that a remote conference is being held between the base where the robot 10 is installed and the base where the display terminal 50 is installed. At the base where the robot 10 is installed, when a plurality of people are participating in the conference, if the participants move or the topics are switched during the conference, the attention points to which the plurality of people (participants) pay attention change. Sometimes. In this case, even if the point of interest changes, the user of the display terminal 50 located in a remote place sees an image that continuously captures the position before the change in the point of interest. They are left behind and hinder smooth communication. Accordingly, the remote control system 1a moves the robot 10 based on the points of interest of a plurality of persons located around the robot 10 to achieve smooth communication with the user of the display terminal 50 located at a remote place. Can be.

  FIG. 19 is a flowchart illustrating an example of a target point calculation process in the robot according to the first embodiment. In step S71, the image acquisition unit 36 of the robot 10 acquires image data 200 that is an image acquired by the imaging device 21 (an example of an acquisition step). In step S72, the image processing unit 41 of the robot 10 extracts the image of the head of the person included in the image data 200 acquired by the image acquisition unit 36. Specifically, the image processing unit 41 collates the image data 200 obtained by the image obtaining unit 36 with a previously stored image pattern of the head of the person, and determines a similar image area in FIG. Is extracted as an image of the head of a person as shown in FIG. The processing data 800a shown in FIG. 20A includes images of the heads of four persons a1 to a4.

  In step S73, the attention point calculation unit 42 of the robot 10 calculates the size of the head image extracted by the image processing unit 41 and the position of the person's head image in the image data 200. Specifically, the attention point calculation unit 42 calculates the size of the image of the person's head and the position of the image of the person's head based on the processing data 800a processed by the image processing unit 41. The position of the image of the person's head is calculated, for example, assuming that the coordinates of the center of the image data 200 are (0, 0), the coordinates of the upper right are (100, 100), and the coordinates of the lower left are (-100, -100). Is done. For example, in the processing data 800a shown in FIG. 20A, the position and size of the image (a1 to a4) of the head of the person calculated by the attention point calculation unit 42 are the values shown in FIG. .

  In step S74, the attention point calculation unit 42 of the robot 10 specifies the relative position in space of the image of the head of the person included in the image data 200. For example, the attention point calculation unit 42 uses the coordinates (X, Y) in the image data 200 and the head size r calculated in step S73 to calculate the position of the image of the head of the person in the image data 200. Is converted to a relative position (x, y) in space. As an example of the method of converting to the relative position (x, y) in space, the following heuristics (1) and (2) are used.

(1) It is assumed that the smaller the head size r is, the farther it is (the difference in head size between people is ignored).
(2) Disregarding the perspective, it is assumed that if the value of the X coordinate is positive, it is on the right side of the robot 10 (if it is negative, it is on the left side of the robot 10), and if the Y coordinate is large, it is on the back.

  The calculation formula based on the above conditions (1) and (2) is as shown in the following (Formula 1). Note that k1 and k2 are predetermined coefficients. Here, k1 = 0.2 and k2 = 1000.

  The attention point calculation unit 42 specifies the relative position (x, y) in the space of the image of the head of the person using the above (Equation 1). FIG. 21A shows coordinate information (x, y) indicating the specified relative position. FIG. 21B is processing data 810 schematically showing the specified relative position. The processing data 810 shown in FIG. 21B is a visualization of the relative positions of the human head images a1 to a4 and the robot R based on the center position S (0,0) of the image data 200. is there.

  In step S75, the image processing unit 41 of the robot 10 extracts an image of a person's eyes and nose included in the image data 200 acquired in step 71. Specifically, the image processing unit 41 compares the image data 200 acquired by the image acquiring unit 36 with the image patterns of the eyes and the nose of a person stored in advance, and determines a similar image region in FIG. The image is extracted as an image of a person's eyes and nose as shown in a). The processing data 800b shown in FIG. 22A includes the eyes and nose images of the head images (a1 to a4) of the four persons extracted in step S72.

  In step S76, the attention point calculation unit 42 of the robot 10 specifies the direction of each person from the positions of the eyes and the nose extracted by the image processing unit 41. The identification of the direction of the person uses, for example, the following heuristics (1) and (2).

(1) If the number of eyes is two, it is assumed that the face is facing the front (the direction opposite to the direction of the robot 10), if the number is one, the face is left or right, and if the number is zero, the face is facing the rear (the same direction as the direction of the robot 10).
(2) If there is only one eye and there is a nose, it is assumed that the nose is facing (the direction is 0 ° in front and 360 ° clockwise).

  Based on the conditions (1) and (2), the direction of the person is specified as shown in FIG. For example, since a1 has one eye and the nose faces rightward, the direction (θ) of a1 is 90 °.

  Thereby, the attention point calculation unit 42 of the robot 10 can approximately obtain the relative position and the direction of the person corresponding to the image data 200. FIG. 23A shows information on the relative position and the direction specified by the attention point calculation unit 42. FIG. 23B is processing data 820 schematically illustrating the relative position (x, y) and direction (θ) of each person. As shown in FIG. 23A, the relative position (x, y) of a1 is (0, 60), and the direction (θ) is 90 °. The relative position (x, y) of b1 is (90, 76.7), and the direction (θ) is 180 °. The relative position (x, y) of c1 is (−20, 42), and the direction (θ) is 0 °. The relative position (x, y) of d1 is (70, 51), and the direction (θ) is 0 °.

  Thereby, the robot 10 can easily estimate the position and orientation of a person located around the robot 10. The method of estimating the position and orientation of a person located around the robot 10 is not limited to this, and may be a method of adopting a perspective method or using a stereo image captured by a stereo camera. . Further, a method of measuring a distance using a light beam such as a laser or a method of arranging a plurality of cameras in a base and calculating the position and orientation of a person from a plurality of images may be used.

  Here, another example of a method for estimating the position and orientation of a person located around the robot 10 will be described. The robot 10 may estimate (calculate) the position and orientation of a person located around the robot 10 using a beacon or the like installed at a base where the robot 10 is located. FIG. 24A is a diagram for explaining an example of a process of estimating the position and orientation of a person located around the robot 10 when the beacon is installed in the base.

  In the example shown in FIG. 24, a beacon for transmitting a predetermined radio wave is installed at each location in a base (for example, a closed space such as a room) 1100. Participants (h1 to h4) of a conference or the like located in the base 1100 hold a communication terminal such as a smartphone having a function of receiving a radio wave transmitted from a beacon and a gyro sensor. The participants (h1 to h4) of the conference or the like wear the communication terminal at a specific position such as a breast pocket or a waist belt, so that the orientation of the body and the orientation of the communication terminal are linked.

  Thereby, the communication terminal held by the participants (h1 to h4) of the conference or the like can acquire the position of the communication terminal (participant) from the intensity of the radio waves transmitted from the plurality of beacons. Further, the communication terminals held by the participants (h1 to h4) of the conference or the like can acquire the orientation of the communication terminal (participant) from the information on the gyro sensor. Then, the robot 10 located in the base 1100 receives (acquires) the information on the position and orientation of the participants (h1 to h4) transmitted from each of the communication terminals held by each of the participants (h1 to h4). can do.

  FIG. 24B shows information on the relative position (x, y) and direction (θ) of each person (h1 to h4) specified by the method shown in FIG. As shown in FIG. 24B, the relative position (x, y) of h1 is (5, 17), and the direction (θ) is 45 °. The relative position (x, y) of h2 is (18, 18), and the direction (θ) is 240 °. The relative position (x, y) of h3 is (5, 12), and the direction (θ) is 60 °. The relative position (x, y) of h4 is (15, 12), and the direction (θ) is 330 °.

  In the following description, the values shown in FIG. 24B are used as the values of the coordinate information (x, y) and the direction (θ) indicating the relative position calculated by the attention point calculation unit 42.

  Returning to FIG. 19, description of the target point calculation processing by the robot 10 will be continued. In the processing after step S77, the robot 10 calculates the points of interest of a plurality of persons around the robot 10. The point of interest is a position where a plurality of people are paying attention at the base where the robot 10 is installed, for example, a direction in which a plurality of people around the robot 10 are facing. The point of interest is, for example, the direction in which a plurality of participants at the base on the robot 10 side are facing in the remote conference, that is, the center position of the topic at a certain point in time.

  In step S <b> 77, the image processing unit 41 of the robot 10 uses the relative positions of the persons (h 1 to h 4) specified by the attention point calculation unit 42 to calculate the respective persons (h 1 to h 1) specified by the attention point calculation unit 42. The intersection of the straight line extending in the direction (θ) of h4) is calculated. The calculated result is shown in FIG. FIG. 25 shows combinations of points and coordinates (x, y) of intersections. Since the direction of the person's direction is one direction, there may be no intersection like the combination of h1 and h2 and the combination of h2 and h3.

  In step S78, the attention point calculation unit 42 of the robot 10 shifts the processing to step S79 when there are a plurality of intersections obtained from the largest number of directions among the calculated intersections. On the other hand, the attention point calculation unit 42 shifts the processing to step S80 when one of the calculated intersections is obtained from the largest number of directions.

  Specifically, first, the attention point calculation unit 42 puts together the coordinates having a short distance among the intersections calculated in step S77. The coordinates of the combined result are, for example, the average value of the coordinates of each intersection. FIG. 26A shows coordinate information indicating the positions of the intersections collected by the above processing. FIG. 26B is processing data 830 schematically showing the positions of intersections collected by the above processing. As shown in FIG. 26A, the intersection of the straight lines extending from h1, h3, and h4 is A (10.5, 18), and the intersection of the straight lines extending from h2 and h4 is B (14, 12). ).

  Then, the attention point calculation unit 42 extracts the intersections obtained from the largest number of directions from the points (intersections) obtained by grouping the intersections having short distances. In the case of FIG. 26A, the intersection A is an intersection obtained from three directions (h1, h2, h3), and the intersection B is an intersection obtained from two directions (h2, h4). Therefore, the attention point calculation unit 42 shifts the processing to step S80 because the number of intersections obtained from the most directions is one (intersection A).

  In step S80, the attention point calculation unit 42 specifies the intersection A, which is the intersection obtained from the most directions, as the attention point. On the other hand, in step S79, the attention point calculation unit 42 specifies the coordinates of the center position between the plurality of intersections obtained from the most directions as the attention point.

  In step 77, the point-of-interest calculation unit 42 performs the process of summarizing the coordinates of the intersections having a shorter distance among the calculated intersections, but focuses on the intersections obtained from the largest number of directions using the calculated intersections as they are. It may be specified as a point.

  Thereby, the remote control system 1a can calculate a point of interest to which a plurality of persons located around the robot 10 pay attention, using the image data 200 acquired by the imaging device 21 provided in the robot 10. For this reason, the remote control system 1a can grasp a place where a plurality of participants on the base side where the robot 10 is installed is paying attention, for example, in a remote conference using the robot 10 and the display terminal 50.

  Next, a process of specifying the moving destination of the robot 10 using the attention point calculated by the attention point calculation unit 42 will be described. In step S81, the target point calculation unit 43 of the robot 10 specifies a target point to which the robot 10 moves based on the position of the target point calculated by the target point calculation unit 42. The target point includes, for example, position information indicating the target position G (X, Y) of the destination of the robot 10 and direction information indicating the direction (θ) of the robot 10 at the destination of the robot 10. Specifically, as shown in FIG. 27B, the target point calculation unit 43 determines a position that is separated from the target point A by a distance p on a line connecting the target point A and the position of the robot 10 to the target position G. It is calculated as (X, Y). Here, the distance p is, for example, an average value of the distance between the position of each person (h1 to h4) and the point of interest A. The relative positions (x, y) calculated by the attention point calculation unit 42 are used as the positions of the persons (h1 to h4). Further, the target point calculation unit 43 calculates the direction of the point of interest A viewed from the current position of the robot 10 as the direction (θ) of the robot 10. The target point (target position G and direction) specified by the target point calculation unit 43 is, for example, a value illustrated in FIG. In this case, the target position G (X, Y) is (6, 11), and the direction (θ) of the robot 10 is 35 °.

  In addition, the direction of the robot 10 calculated by the target point calculation unit 43 is not limited to the direction of the attention point A, and may be an average direction of a plurality of persons calculated by the attention point calculation unit 42.

  Thereby, the remote control system 1a can calculate a target point to which the robot 10 moves based on a point of interest of a plurality of persons located around the robot 10. Therefore, for example, in a remote conference or the like using the robot 10 and the display terminal 50, the remote control system 1a controls the robot 10 based on a location where a plurality of participants are paying attention on the base side where the robot 10 is installed. The destination can be specified.

Next, a process of moving the robot 10 toward the target point specified by the target point calculation unit 43 will be described. FIG. 28 is a flowchart illustrating an example of a movement process in the robot according to the first embodiment.

  In step S101, the target point calculation unit 43 of the robot 10 calculates a moving path toward the calculated target position G (X, Y).

  In step S102, the robot 10 moves based on the calculated movement route. First, the direction adjusting unit 38 of the robot 10 changes the direction of the robot 10 by rotating the moving mechanism 17 with respect to the traveling direction on the moving path calculated by the target point calculating unit 43. Then, when the change of the direction in the traveling direction by the direction adjusting unit 38 is completed, the movement control unit 37 of the robot 10 moves the robot 10 by driving the movement mechanism 17. The movement control unit 37 issues (transmits) a movement command corresponding to the movement route calculated by the target point calculation unit 43 to the movement mechanism 17 as needed, thereby causing the robot 10 to move along the calculated movement route. Move.

  In step S103, when the surrounding information detection unit 44 of the robot 10 detects an obstacle while the robot 10 is moving, the process proceeds to step S104. Specifically, the peripheral information detection unit 44 detects an obstacle existing on the movement route of the robot 10 using the image data 200 acquired by the imaging device 21 and the detection results detected by the various sensors 106. The obstacle is, for example, a device or an object installed in the base, or a person located in the base. In step S104, the movement controller 37 of the robot 10 stops the robot 10 when an obstacle is detected by the peripheral information detector 44.

  Accordingly, when the robot 10 detects an obstacle while moving toward the target position G, the robot 10 stops in front of the detected obstacle, and thus can avoid contact with the obstacle. In this case, the robot 10 may resume the movement toward the target position G again, once stopped, when the peripheral information detection unit 44 detects that the obstacle has disappeared. Further, the robot 10 changes the movement route calculated by the target point calculation unit 43 when the obstacle detected by the peripheral information detection unit 44 does not disappear after a predetermined time has elapsed after the robot 10 has stopped temporarily. It may be.

  On the other hand, in step S103, when the peripheral information detection unit 44 does not detect an obstacle while the robot 10 is moving, the processing proceeds to step S105. In this case, the movement control unit 37 of the robot 10 continues the movement control of the robot 10 toward the target position G.

  In step S105, when the robot 10 reaches the target position G, the movement control unit 37 of the robot 10 shifts the processing to step S106. On the other hand, when the robot 10 has not reached the target position G, the movement control unit 37 repeats the processing from step S103 and continues the movement control of the robot 10 toward the target position G.

  In step S106, the movement control unit 37 of the robot 10 stops the robot 10 when the robot 10 reaches the target position G. In step S107, the direction adjusting unit 38 of the robot 10 changes the direction of the robot 10 to the direction calculated by the target point calculating unit 43 (see FIG. 27A). Specifically, the direction adjusting unit 38 changes the direction of the robot 10 based on the direction information indicating the direction (θ) calculated by the target point calculating unit 43 by rotating the moving mechanism 17 or the like. . Then, the transmission / reception unit 31 of the robot 10 transmits the image data 200 captured and acquired by the imaging device 21 at the moved target point to the display terminal 50 via the communication network 9. Thereby, the display terminal 50 can display the image data 200 obtained by photographing the periphery of the robot 10 at the target point to which the robot 10 moves.

  Therefore, the remote control system 1a can move the robot 10 to the target point calculated by the target point calculator 43. Therefore, the remote control system 1a can move the robot 10 based on the points of interest of a plurality of participants at the base where the robot 10 is located, and photograph the point of interest and the vicinity thereof on the display terminal 50 at a remote place. Since the displayed image can be displayed, smooth communication between the remote sites can be realized.

  When the robot 10 b does not have the moving mechanism 17 as in the robot 10 b illustrated in FIG. 4, the robot 10 b (transmission terminal) determines the direction (θ) of the person among the target points calculated by the target point calculation unit 43. A configuration may be used in which the direction of the robot 10b is changed to the change destination direction indicated by the direction information using only the direction information indicated.

〇Presence notification processing Next, a description will be given of processing in which the robot 10 notifies its surroundings of its own when the robot 10 is moving to a target point. FIG. 29 is a flowchart illustrating an example of a presence notification process in the robot according to the first embodiment.

  In step S121, the position specifying unit 39 of the robot 10 specifies the current position of the robot 10. Specifically, the position specifying unit 39 specifies the current position of the robot 10 based on the detection result of each direction (azimuth angle, magnetic north) detected by the acceleration / azimuth sensor 103 or the GPS receiving unit 104. I do. In step S122, the position specifying unit 39 of the robot 10 calculates the distance between the specified current position of the robot 10 and the attention point A calculated by the attention point calculation unit 42.

  In step S123, if the distance calculated by the position specifying unit 39 is equal to or larger than the predetermined threshold or longer than the predetermined threshold, the robot 10 shifts the processing to step S124. On the other hand, when the distance calculated by the position specifying unit 39 is shorter than the predetermined threshold or equal to or smaller than the predetermined threshold, the robot 10 repeats the processing from step S121.

  In step S124, the presence notification unit 45 of the robot 10 outputs notification information for notifying the presence of the robot 10. Specifically, the presence notification unit 45 outputs a sound signal from the speaker 308b as notification information. The presence notification unit 45 performs a predetermined gesture as the notification information by driving the moving mechanism 17, the movable arm 11, or the rotating shaft 12. The method of outputting the notification information by the presence notification unit 45 is not limited to this.

  Accordingly, when the robot 10 is far away from the point of interest, the robot 10 approaches the point of interest while notifying the surrounding people of the existence of the robot 10, so that the robot 10 suddenly appears to surprise the surrounding people. Can be prevented. Further, the robot 10 can prevent the progress of a conference or the like from being hindered by approaching the point of interest.

  Note that the presence notification unit 45 of the robot 10 may be configured to notify the presence of the robot when the distance between the current position of the robot 10 and the point of interest becomes equal to or smaller than a predetermined threshold or shorter than the predetermined threshold. .

  As described above, since the remote control system 1a automatically moves the robot 10 to a suitable position based on a point of interest of a plurality of persons (participants) located around the robot 10, In the remote communication performed by using the remote control 10, it is possible to reduce a sense of incongruity and annoyance felt by a participant, and to realize the remote communication through the robot 10 in a natural manner.

  Since the conventional fixed position video conferencing system is not moved from the position where it was initially placed, for example, when the center position (point of interest) of the topic changes, those who are participating from a remote place are left out of the conversation. I get the impression that Also, for example, a participant around the robot 10 may feel that a person who is participating from a remote place is not interested in the topic, and may exclude a person who is participating from a remote place from being a conversation target. There is. As a result, the sense of unity due to remote communication is impaired, which causes a feeling of alienation.

  On the other hand, the robot 10 follows a central position (point of interest) of the topic and changes the position and orientation of the robot 10 in order to change the position and orientation of the robot 10 (for example, the robot 10 using the display terminal 50). Can be obtained through the robot 10. On the other hand, a plurality of participants located around the robot 10 also have an impression that a participant in a remote place is participating in a conference and listening to a story. Thereby, the remote control system 1a can realize smooth communication by the remote conference using the robot 10.

<< Example of Moving Path of Robot 10 >> Here, the moving path of the robot 10 calculated by the target point calculating unit 43 will be described. The example shown in FIG. 30A is a method in which the robot 10 (R) does not head straight to the target position G, but finds between people and interrupts it. This method is effective to prevent the body distance (private space) from being affected by the robot 10 (R) coming too close to a person. In this case, the target point calculation unit 43 of the robot 10 (R) specifies a position having a certain distance or more from all combinations of positions of the participants (h1 to h4), and sets the robot to an intermediate distance between the positions. The moving route is calculated so as to move 10 (R).

  The example shown in FIG. 30 (b) is a method in which the robot 10 (R) does not head straight to the target position G but passes by the person closest to the current position of the robot 10 (R). The target point calculation unit 43 of the robot 10 (R) specifies the participant (for example, h3) closest to the current position of the robot 10 (R) specified by the position specifying unit 39, and determines from the position of the specified participant. A movement path is calculated so as to move the robot 10 (R) to a position separated by a certain distance. For example, the robot 10 (R) may move to the target position G by passing through a position separated by a certain distance from the position of h3, or may stop at a position separated by a certain distance from the position of h3. Good. In this case, as shown in FIG. 30B, the robot 10 (R) is not turned in the direction of the point of interest A, but is turned in the same direction as h3 by the robot 10 (R). May be changed.

<< Process Before Starting Target Point Calculation Process >> Here, the frequency or timing of performing the target point calculation process shown in FIG. 19 will be described. In a meeting or the like, in a case where the position of a participant around the robot 10 changes frequently, if the robot 10 moves each time, the participant may interfere with the meeting. Therefore, the remote control system 1a performs the target point calculation processing shown in FIG. 19 in the robot 10 when a predetermined condition is satisfied.

  The flowchart shown in FIG. 31A is an example in which the target point calculation processing is performed based on the elapsed time from the previous calculation of the target point. In step S151a, the determination unit 34 of the robot 10 uses the timer 107 to measure the time elapsed since the previous calculation of the target point. In step S152a, when the elapsed time from the previous calculation of the target point exceeds the set elapsed time condition, the determination unit 34 shifts the processing to step S153a. Specifically, the storage / read unit 46 of the robot 10 reads the condition table 3002 (see FIG. 10) stored in the storage unit 3000. The determination unit 34 searches the condition table 3002 read by the storage / read unit 46, and extracts the condition of the elapsed time corresponding to the base where the robot 10 is located. For example, when the robot 10 is located at the base A, the determination unit 34 extracts “3 min” as a condition of the elapsed time corresponding to the base A. Then, when the elapsed time from the previous calculation of the target point exceeds the extracted condition, the determination unit 34 shifts the processing to step S153a. On the other hand, in step S152a, if the elapsed time from the previous calculation of the target point does not exceed the set elapsed time condition (is within the set elapsed time), the determination unit 34 determines in step S151a. Repeat the process from.

  In step S153a, the robot 10 performs the target point calculation processing shown in FIG. 19 when a predetermined time has elapsed since the previous target point calculation. Thereby, the robot 10 performs the process of calculating the target point indicating the destination of the robot 10 based on the elapsed time since the previous target point calculation process is performed, and moves the robot 10 to the calculated target point. In addition, it is possible to prevent smooth communication from being hindered by the frequent movement of the robot 10.

  The flowchart shown in FIG. 31B is an example in which the target point calculation processing is performed based on the distance from a person (participant) located around the robot 10. In step S151b, the robot 10 measures the distance to a person (participant) located around the robot 10. Specifically, the position specifying unit 39 of the robot 10 determines the current position of the robot 10 based on the detection result of each direction (azimuth, magnetic north) detected by the acceleration / direction sensor 103 or the GPS receiving unit 104. Identify the location. Then, the position specifying unit 39 calculates the distance between the specified current position of the robot 10 and the person (participant) calculated by the attention point calculating unit 42.

  In step S152b, when the calculated distance to the person (participant) exceeds the set condition for the distance to the person, the determination unit 34 shifts the processing to step S153b. Specifically, the storage / read unit 46 of the robot 10 reads the condition table 3002 (see FIG. 10) stored in the storage unit 3000. The determination unit 34 searches the condition table 3002 read by the storage / read unit 46 and extracts the condition of the distance to the person corresponding to the base where the robot 10 is located. For example, when the robot 10 is located at the base A, the determination unit 34 extracts “8 m” as a condition for the distance to the person corresponding to the base A. Then, if at least one of the distances between the robot 10 and the person calculated by the position specifying unit 39 exceeds the extracted condition, the determining unit 34 shifts the processing to step S153b. On the other hand, in step S152b, the determination unit 34 determines that the calculated distance to the person (participant) does not exceed the condition of the distance to the set person (the robot 10 and all the participants have a predetermined distance). If it exists within the distance), the processing from step S151b is repeated.

  In step S153b, when the distance from the person (participant) is greater than or equal to the predetermined distance, the robot 10 performs the target point calculation process illustrated in FIG. Accordingly, the robot 10 performs the process of calculating the target point indicating the destination of the robot 10 only when the distance from a person (participant) located around the robot 10 is greater than or equal to a predetermined distance. In order to move the robot 10 to the set target point, it is possible to prevent the smooth communication from being hindered by the frequent movement of the robot 10.

  The flowchart shown in FIG. 31C is an example in which the target point calculation processing is performed based on the set adjustment cycle. In step S151c, the determination unit 34 of the robot 10 uses the timer 107 to measure the elapsed time from the previous calculation of the target point. In step S152c, the determination unit 34 shifts the processing to step S153c when the elapsed time from the previous calculation of the target point exceeds the condition of the set adjustment cycle. Specifically, the storage / read unit 46 of the robot 10 reads the condition table 3002 (see FIG. 10) stored in the storage unit 3000. The determination unit 34 searches the condition table 3002 read by the storage / read unit 46, and extracts the condition of the adjustment cycle corresponding to the base where the robot 10 is located. For example, when the robot 10 is located at the site A, the determination unit 34 extracts “5 min” as a condition of the adjustment cycle corresponding to the site A. Then, when the elapsed time since the previous calculation of the target point exceeds the extracted condition, the determining unit 34 shifts the processing to step S153c. On the other hand, in step S152c, the determination unit 34 determines that the elapsed time from the previous calculation of the target point does not exceed the condition of the set adjustment cycle (is within the range of the set adjustment cycle). The processing from step S151c is repeated.

  In step S153c, the robot 10 performs the target point calculation processing shown in FIG. 19 when a predetermined adjustment cycle has elapsed since the previous target point calculation. Accordingly, the robot 10 performs the process of calculating the target point indicating the destination of the robot 10 based on the set adjustment cycle of the target point, and frequently moves the robot 10 to the calculated target point. It is possible to prevent smooth communication from being hindered by the movement of 10.

  Therefore, as shown in FIGS. 31A to 31C, the remote control system 1a determines that the frequency or timing at which the target point calculation process is started based on a predetermined condition, so that the robot 10 frequently moves. Can prevent smooth communication from being hindered. The processing illustrated in FIGS. 31A to 31C and the conditions included in the condition table 3002 illustrated in FIG. 10 are examples, and are not limited thereto. Can be modified and changed by an operator or the like operating the robot 10 as needed.

  Further, the remote control system 1a may be configured to perform the target point calculation process when at least one person (participant) around the robot 10 moves. In this case, the robot 10 may change the position or direction of at least one of the plurality of persons included in the image data 200 acquired by the image acquisition unit 36, for example, by changing the attention point calculation unit 42 and the target point calculation unit 43. If so, the target point is calculated based on the changed position or orientation of the person. Specifically, the robot 10 sets the attention point calculation unit 42 and the target point calculation unit 43 to a fixed value of the position or orientation of a person who has not changed among a plurality of persons included in the image data 200, and A target point is calculated based on the amount of change in position or orientation.

[Effects of the First Embodiment] As described above, the remote control system 1a according to the first embodiment includes the robot 10 (an example of a moving object) including the photographing device 21 that photographs the subject and acquires the image data 200. ), A display terminal 50 that can communicate with the robot 10 via the communication network 9 and remotely controls the robot 10, and performs remote communication using image data transmitted and received between the robot 10 and the display terminal 50. Realize.

  Here, in the remote control system 1a according to the first embodiment, the control device 30 included in the robot 10 (an example of a moving object) is an example of the information processing device according to the present invention. The control device 30 serving as the information processing device includes an image acquisition unit 36 (an example of an acquisition unit) that acquires the image data 200 acquired by the imaging device 21, a position of a plurality of persons included in the acquired image data 200, A target point calculation unit 42 and a target point calculation unit 43 (an example of a calculation unit) for calculating a target point indicating a destination of the robot 10 based on the direction are provided, and the robot 10 is moved to the calculated target point. For this reason, the control device 30 as the information processing device according to the first embodiment moves the robot 10 according to the positions and orientations of a plurality of participants at one base, thereby joining the participants at the other base. Can activate the remote communication with the person.

  Further, the control device 30 included in the robot 10 (an example of a moving object) according to the first embodiment estimates the positions and orientations of a plurality of persons included in the acquired image data 200 (an example of an estimation unit, estimation Based on the specified position and orientation, a point of interest calculated by a plurality of persons included in the image data 200 (an example of a point of interest specifying unit) is specified. The control device 30 as an information processing device according to the first embodiment includes a target point calculation unit 43 (an example of a target point specifying unit) that specifies a target point indicating a destination of the robot 10. By moving the robot 10 based on a point of interest of a plurality of participants located around the, the remote communication with the participant at the other base can be activated. Kill.

● Second embodiment ●
Next, a remote control system according to a second embodiment will be described. Note that the same configurations and functions as those of the first embodiment are denoted by the same reference numerals, and description thereof will be omitted. The remote control system 1b according to the second embodiment has a configuration in which the position adjustment process of the robot 10 is performed by the display terminal 50 that displays the image data transmitted from the robot 10.

  The hardware configuration of the devices or terminals (robot 10, display terminal 50, and management server 90) constituting the remote control system 1b according to the second embodiment is the same as that of the remote control system 1a according to the first embodiment. This is the same as the device or terminal (robot 10, display terminal 50, and management server 90) that performs the operation, and thus the description is omitted.

First, the functional configuration of the remote control system according to the second embodiment will be described with reference to FIG. FIG. 32 is a diagram illustrating an example of a functional configuration of the remote control system according to the second embodiment. Note that the functions implemented by the management server 90 according to the second embodiment are the same as the functions implemented by the management server 90 according to the first embodiment (see FIG. 8), and a description thereof will be omitted.

<< Functional Configuration of Control Device 30 >> First, the functional configuration of the control device 30 that controls the processing or operation of the robot 10 according to the second embodiment will be described with reference to FIG. The control device 30 according to the second embodiment controls the image processing unit 41, the attention point calculation unit 42, and the target point calculation unit 43 from the functions (see FIG. 8) realized by the control device 30 according to the first embodiment. Implement the excluded functions. Specifically, the functions realized by the control device 30 according to the second embodiment include a transmission / reception unit 31, an operation input reception unit 32, a display control unit 33, a determination unit 34, a shooting instruction unit 35, and an image acquisition unit 36. , A movement control unit 37, a direction adjustment unit 38, a position identification unit 39, a peripheral information detection unit 44, a presence notification unit 45, a storage / readout unit 46, and a storage unit 3000. The respective functions realized by the control device 30 according to the second embodiment are the same as those shown in FIG.

<< Functional Configuration of Display Terminal 50 >> Next, a functional configuration of the display terminal 50 will be described with reference to FIG. The functions realized by the display terminal 50 according to the second embodiment include, in addition to the functions realized by the display terminal 50 according to the first embodiment (see FIG. 8), an image processing unit 61 and a point-of-interest calculation unit. 62 and a target point calculation unit 63. Specifically, the functions realized by the display terminal 50 include a transmission / reception unit 51, an operation input reception unit 52, a display control unit 53, a determination unit 54, a request command generation unit 55, a line-of-sight detection unit 56, a storage / readout unit 57. , An image processing unit 61, an attention point calculation unit 62, a target point calculation unit 63, and a storage unit 5000. The transmission / reception unit 51 is an example of an acquisition unit that receives (acquires) image data 200 that is an image acquired by the imaging device 21. Further, the transmission / reception unit 51 is an example of a second transmission unit that transmits, to the robot 10, a request command (request information) for moving the robot 10 to a target point calculated by a target point calculation unit 63 described later. .

  The image processing unit 61 has a function of performing processing on an image received (acquired) by the transmission / reception unit 51. The image processing unit 61 extracts, for example, an image of a person's head included in the image data 200 acquired by the transmission / reception unit 51. The image processing unit 61 extracts, for example, images of the eyes and nose of a person acquired by the transmission / reception unit 51. The image processing unit 61 is realized mainly by the processing of the CPU 501 shown in FIG.

  The attention point calculation unit 62 has a function of specifying an attention point to which a plurality of people pays attention based on the positions and orientations of the plurality of people included in the image data processed by the image processing unit 61. The functions realized by the attention point calculation unit 62 are the same as those of the attention point calculation unit 42 shown in FIG. The attention point calculation unit 62 is realized mainly by the processing of the CPU 501 shown in FIG. The attention point calculation unit 62 is an example of a calculation unit. In addition, the attention point calculation unit 62 is an example of an estimation unit. Further, the attention point calculation unit 62 is an example of an attention point identification unit.

  The target point calculation unit 63 has a function of specifying a target point to which the robot 10 moves based on the target point calculated by the target point calculation unit 62. The functions realized by the target point calculation unit 63 are the same as those of the target point calculation unit 43 shown in FIG. The target point calculation unit 63 is realized mainly by the processing of the CPU 501 shown in FIG. The target point calculation unit 63 is an example of a calculation unit. Further, the target point calculation unit 63 is an example of a target point specifying unit.

-Processing or operation of the second embodiment Next, processing or operation of the robot in the remote control system according to the second embodiment will be described with reference to FIG. In FIG. 33, the processing executed by the control device 30 included in the robot 10 will be described as the processing executed by the robot 10. FIG. 33 is a sequence diagram illustrating an example of a control process of the robot in the remote control system according to the second embodiment.

  In step S201, the image acquisition unit 36 of the robot 10 acquires image data 200 that is an image captured by the imaging device 21.

  In steps S202-1 and S202-2, the transmission / reception unit 31 of the robot 10 transmits the image data 200 acquired by the image acquisition unit 36 to the display terminal 50 using the established communication session with the management server 90. I do. Thereby, the transmission / reception unit 51 of the display terminal 50 receives (acquires) the image data 200 which is an image acquired by the imaging device 21 (an example of an acquisition step).

  In step S203, the display control unit 53 of the display terminal 50 displays the image data 200 received by the transmission / reception unit 51 on the display screen 600 of the display 512 (see FIG. 18B). Thus, the operator who operates the robot 10 using the display terminal 50 can check the status of the base where the robot 10 is located while viewing the display screen 600 on which the image data 200 is displayed.

  In step S204, the display terminal 50 performs a process of calculating a target point to which the robot 10 moves (an example of a calculation step). The target point calculation processing by the image processing unit 61, the attention point calculation unit 62, and the target point calculation unit 63 of the display terminal 50 is the same as the processing illustrated in FIG.

  In step S205, the request command generator 55 of the display terminal 50 generates a request command to be transmitted to the robot 10 based on the target point calculated by the target point calculator 63. In this case, the request command generated by the request command generation unit 55 is a “GO (X, Y)” command using the coordinate information (X, Y) of the target position G calculated by the target point calculation unit 63 as a variable. Including. The request command generated by the request command generator 55 includes a “TURN (θ)” command in which the direction (θ) of the robot 10 calculated by the target point calculator 63 is used as a variable. The request command generated by the request command generation unit 55 is an example of request information for moving the robot 10 to the target point calculated by the target point calculation unit 63.

  In steps S206-1 and S206-2, the transmission / reception unit 31 of the display terminal 50 uses the established communication session with the management server 90 to request the request command (“GO (X, Y), “TURN (θ)”) to the robot 10. Thereby, the transmission / reception unit 31 of the robot 10 receives the request command (“GO (X, Y)”, “TURN (θ)”).

  In step S207, the storage / readout unit 46 of the robot 10 reads out the command table 5001 stored in the storage unit 5000. In step S208, the determination unit 34 of the robot 10 searches the command table 3001 read by the storage / read unit 46, and extracts a process corresponding to the request command received by the transmission / reception unit 31. In this case, since the request commands received by the transmission / reception unit 31 are “GO (X, Y)” and “TURN (θ)”, the determination unit 34 searches the command table 3001 and determines the request command “GO ( X, Y) ”and“ TURN (θ) ”. The extracted processes are "moving the robot 10 to the position (X, Y)" and "rotating the robot 10 to the right θ °".

  In step S209, the robot 10 executes the processing extracted in step S208. Specifically, the determination unit 34 of the robot 10 notifies the movement control unit 37 of an execution request for the extracted processing. Then, the movement control unit 37 moves the robot 10 based on the processing extracted by the determination unit 34. The movement process of the robot 10 is the same as the process shown in FIG.

  Then, the transmission / reception unit 31 of the robot 10 transmits, to the display terminal 50, image data 200 captured by the imaging device 21 at the target point included in the request command transmitted from the display terminal 50. The display terminal 50 receives the image data 200 transmitted from the robot 10. Thereby, the display terminal 50 can display the image data 200 obtained by photographing the periphery of the robot 10 at the target point included in the request command.

[Effects of the Second Embodiment] As described above, the remote control system 1b according to the second embodiment performs the target point calculation processing of the robot 10 (an example of a moving object) through the communication network 9 through the robot 10 Is performed by the display terminal 50 that remotely controls the operation. Therefore, the remote control system 1b according to the second embodiment can reduce the processing load on the robot 10 and perform remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50. Can be activated.

  Here, in the remote control system 1b according to the second embodiment, the display terminal 50 is an example of the information processing device according to the present invention. The display terminal 50 as the information processing device includes a transmitting / receiving unit 51 (an example of an acquiring unit) that acquires the image data 200 acquired by the imaging device 21 and positions and orientations of a plurality of persons included in the acquired image data 200. , A target point calculation unit 62 and a target point calculation unit 63 (an example of a calculation unit) that calculate a target point indicating a destination of the robot 10 based on the target position, and move the robot 10 to the calculated target point. Therefore, the display terminal 50 as the information processing apparatus according to the second embodiment calculates the destination of the robot 10 based on the positions and orientations of a plurality of participants at one base, and sends the robot 10 to the calculated destination. By moving the, the remote communication with the participant at the other base can be activated.

● Third embodiment ●
Next, a remote control system according to a third embodiment will be described. The same configurations and the same functions as those in the first embodiment or the second embodiment are denoted by the same reference numerals, and description thereof will be omitted. The remote control system 1c according to the third embodiment is configured such that an image processing server 70 capable of communicating with the robot 10 and the display terminal 50 via the communication network 9 performs the position adjustment processing of the robot 10.

First, the outline of the configuration of the remote control system according to the third embodiment will be described with reference to FIG. FIG. 34 is a diagram illustrating an example of a system configuration of a remote control system according to the third embodiment. As shown in FIG. 34, in a remote control system 1c according to the third embodiment, an image processing server 70 is added to the configuration shown in FIG. The image processing server 70 is communicably connected to the robot 10, the display terminal 50, and the management server 90 via the communication network 9. The image processing server 70 transmits and receives image data to and from the robot 10 or the display terminal 50 using the communication session established by the management server 90.

  The image processing server 70 is a server computer, and includes a case where a plurality of server computers perform image processing in a distributed manner. The image processing server 70 performs image processing on the image data 200 transmitted from the robot 10 according to the moving state of the robot 10, and transmits the processed data to the display terminal 50. Note that the image processing server 70 may be configured by a plurality of server computers, and any server computer may have a function. The image processing server 70 is an example of an information processing device.

  Here, the image processing server 70 and the management server 90 constitute the server system 7. The image processing server 70 and the management server 90 may be configured by one server computer. Further, the display terminal 50 and the image processing server 70 constitute the display system 5. Further, the robot 10 (10A, 10B, 10C) and the image processing server 70 constitute a moving object control system (transmission control system) 3.

  Note that the hardware configuration of the image processing server 70 is the same as the hardware configuration of the management server 90 shown in FIG. Hereinafter, the image processing server 70 will be described as having the hardware configuration shown in FIG.

[Functional Configuration] FIG. 35 is a diagram illustrating an example of a functional configuration of the remote control system according to the third embodiment. The functions realized by the image processing server 70 include a transmission / reception unit 71, a determination unit 72, a data processing unit 73, an image processing unit 74, an attention point calculation unit 75, a target point calculation unit 76, a storage / readout unit 77, and a storage unit 7000. including.

  The transmission / reception unit 71 has a function of transmitting / receiving various data or information to / from another device (for example, the management server 90, the display terminal 50, or the robot 10) via the communication network 9. The transmission / reception unit 71 receives (acquires) image data 200 which is an image acquired by the imaging device 21, for example. Further, the transmission / reception unit 71 transmits the request data (request information) generated by the data processing unit 73 to the robot 10 via the communication network 9. The transmission / reception unit 71 is mainly realized by the processing of the CPU 901 illustrated in FIG. 7 and the network I / F 909. The transmission / reception unit 71 is an example of an acquisition unit. The transmitting / receiving unit 71 is an example of a third transmitting unit.

  The determining unit 72 has a function of determining the contents of processing for various data transmitted and received by the transmitting and receiving unit 71. The determination unit 72 is realized mainly by the processing of the CPU 901 shown in FIG.

  The data processing unit 73 has a function of processing various data transmitted and received by the transmission and reception unit 71. The data processing unit 73 generates request data (request information) for moving the robot 10 to the target point calculated by the target point calculation unit 76. The data processing unit 73 is realized mainly by the processing of the CPU 901 shown in FIG.

  The image processing unit 74 has a function of performing processing on an image received (acquired) by the transmission / reception unit 71. The image processing unit 74 extracts, for example, an image of a person's head included in the image data 200 acquired by the transmission / reception unit 71. The image processing unit 74 extracts, for example, images of the eyes and nose of the person acquired by the transmission / reception unit 71. The image processing unit 74 is realized mainly by the processing of the CPU 901 shown in FIG.

  The attention point calculation unit 75 has a function of specifying an attention point to which a plurality of people pays attention based on the positions and orientations of the plurality of people included in the image data processed by the image processing unit 74. The function realized by the attention point calculation unit 75 is the same as that of the attention point calculation unit 42 illustrated in FIG. The attention point calculation unit 75 is realized mainly by the processing of the CPU 901 shown in FIG. The attention point calculation unit 75 is an example of a calculation unit. In addition, the attention point calculation unit 75 is an example of an estimation unit. Furthermore, the attention point calculation unit 75 is an example of an attention point identification unit.

  The target point calculation unit 76 has a function of specifying a target point to which the robot 10 moves based on the target point calculated by the target point calculation unit 75. The functions realized by the target point calculation unit 76 are the same as those of the target point calculation unit 43 shown in FIG. The target point calculation unit 76 is realized mainly by the processing of the CPU 901 shown in FIG. The target point calculation unit 76 is an example of a calculation unit. Further, the target point calculation unit 76 is an example of a target point specifying unit.

  The storage / readout unit 77 has a function of storing various data in the storage unit 7000 or reading various data from the storage unit 7000. The storage / readout unit 77 is mainly realized by the processing of the CPU 901 shown in FIG. The storage unit 7000 is mainly realized by the ROM 902, the HD 904, or the recording medium 906 illustrated in FIG.

  The storage unit 7000 stores the image data 200 and the image data 250 received by the transmission / reception unit 71. Further, storage unit 7000 stores condition table 7001. Note that the condition table 7001 is the same as the condition table 3002 shown in FIG. Further, the image data 200 and the image data 250 stored in the storage unit 7000 may be deleted when a predetermined time has elapsed after being received by the transmission / reception unit 71, or the operation input reception unit 52 May be deleted based on a deletion command received from the user. Similarly, the image data 250 stored in the storage unit 5000 may be deleted when a predetermined time has elapsed since the image data 250 was acquired by the CMOS sensor 505a, or transmitted to the robot 10 or the display terminal 50. The deleted data may be deleted.

[Process or Operation of Third Embodiment] Next, the process or operation of the robot in the remote control system according to the third embodiment will be described with reference to FIG. In FIG. 36, the processing executed by the control device 30 included in the robot 10 will be described as the processing executed by the robot 10. FIG. 36 is a sequence diagram illustrating an example of a control process of the robot in the remote control system according to the third embodiment.

  In step S301, the image acquisition unit 36 of the robot 10 acquires image data 200 that is an image acquired by the imaging device 21.

  In steps S302-1, S302-2, and S302-3, the transmission / reception unit 31 of the robot 10 displays the image data 200 acquired by the image acquisition unit 36 using the established communication session with the management server 90. Send to terminal 50. The transmission / reception unit 71 of the image processing server 70 receives (acquires) the image data 200 transmitted from the robot 10 and acquired by the imaging device 21 (an example of an acquisition step). Thereby, the transmission / reception unit 51 of the display terminal 50 receives the image data 200.

  In step S303, the display control unit 53 of the display terminal 50 displays the image data 200 received by the transmission / reception unit 51 on the display screen 600 of the display 512 (see FIG. 18B). Thus, the operator who operates the robot 10 using the display terminal 50 can check the status of the base where the robot 10 is located while viewing the display screen 600 on which the image data 200 is displayed.

  In step S304, the image processing server 70 executes a process of calculating a target point to which the robot 10 moves (an example of a calculation step). The target point calculation processing by the image processing unit 74, the attention point calculation unit 75, and the target point calculation unit 76 of the image processing server 70 is the same as the processing illustrated in FIG.

  In step S305, the data processing unit 73 of the image processing server 70 generates request data (request information) to be transmitted to the robot 10 in order to move the robot 10 to the target point calculated by the target point calculation unit 76. In this case, the request data generated by the data processing unit 73 includes position information indicating the target position G (X, Y) calculated by the target point calculation unit 63, and the direction (θ) of the robot 10 at the destination of the robot 10. Is included. The request data generated by the data processing unit 73 corresponds to the request command generated by the request command generation unit 55 of the display terminal 50.

  In steps S306-1 and S306-2, the transmission / reception unit 71 of the image processing server 70 transmits the request data (request information) generated by the data processing unit 73 to the robot 10. Thereby, the transmission / reception unit 31 of the robot 10 receives the request data (request information).

  In step S307, the storage / read unit 46 of the robot 10 reads the command table 5001 stored in the storage unit 5000. In step S308, the determination unit 34 of the robot 10 searches the command table 3001 read by the storage / read unit 46, and corresponds to the request data (request command corresponding to the request data) received by the transmission / reception unit 31. Extract processing. In this case, since the request commands corresponding to the request data received by the transmission / reception unit 31 are “GO (X, Y)” and “TURN (θ)”, the determination unit 34 searches the command table 3001 and The processing corresponding to the request commands “GO (X, Y)” and “TURN (θ)” is extracted. The extracted processes are "moving the robot 10 to the position (X, Y)" and "rotating the robot 10 to the right θ °".

  In step S309, the robot 10 executes the processing extracted in step S308. Specifically, the determination unit 34 of the robot 10 notifies the movement control unit 37 of an execution request for the extracted processing. Then, the movement control unit 37 moves the robot 10 based on the processing extracted by the determination unit 34. The movement process of the robot 10 is the same as the process shown in FIG.

  Then, the transmission / reception unit 31 of the robot 10 transmits, to the display terminal 50, the image data 200 captured by the imaging device 21 at the target point included in the request data transmitted from the image processing server 70. The display terminal 50 receives the image data 200 transmitted from the robot 10. Accordingly, the display terminal 50 can display the image data 200 obtained by photographing the periphery of the robot 10 at the target point included in the request data generated by the image processing server 70.

[Effects of Third Embodiment] As described above, the remote control system 1c according to the third embodiment performs the target point calculation process of the robot 10 (an example of a moving object) through the communication network 9 through the robot 10 And an image processing server 70 that can communicate with the display terminal 50. Therefore, the remote control system 1c according to the third embodiment can reduce the processing load on the robot 10 and the display terminal 50, and can control the robot 10 between the base where the robot 10 is located and the display terminal 50. The used remote communication can be activated.

  Here, in the remote control system 1c according to the third embodiment, the image processing server 70 is an example of the information processing device according to the present invention. The image processing server 70 as the information processing apparatus includes a transmitting / receiving unit 71 (an example of an acquiring unit) that acquires the image data 200 acquired by the photographing device 21, a position of a plurality of persons included in the acquired image data 200 and A target point calculation unit 75 and a target point calculation unit 76 (an example of a calculation unit) that calculate a target point indicating a destination of the robot 10 (an example of a moving object) based on the orientation are provided. The robot 10 is moved. Therefore, the image processing server 70 as the information processing apparatus according to the third embodiment calculates the destination of the robot 10 based on the positions and orientations of a plurality of participants at one base, and moves the robot 10 to the calculated destination. By moving 10, the remote communication with the participant at the other base can be activated.

● Summary ●
As described above, the information processing apparatus according to the embodiment of the present invention adjusts the position of the robot 10 (an example of a moving object) including the image capturing device 21 that captures an image of a subject and acquires the image data 200. A device that obtains image data 200 obtained by the imaging device 21 and calculates a target point indicating a destination of the robot 10 based on the positions and orientations of a plurality of persons included in the image data 200 . Then, the information processing apparatus according to one embodiment of the present invention calculates the target point, and moves the robot 10 to the target point. Therefore, the information processing device according to the embodiment of the present invention can realize smooth communication using the robot 10.

  Further, the remote control system 1a according to the embodiment of the present invention includes a robot 10 (an example of a moving object) including an image capturing device 21 that captures an image of a subject and acquires image data 200, and a robot 10 and a communication network 9 via the communication network 9. And a display terminal 50 that can remotely control the robot 10. Then, the robot 10 is configured based on the image acquisition unit 36 (an example of an acquisition unit) that acquires the image data 200 acquired by the imaging device 21 and the positions and orientations of a plurality of persons included in the acquired image data 200. An attention point calculation unit 42 and a target point calculation unit 43 (an example of a calculation unit) that calculate a target point indicating a destination of the robot 10, and a movement control unit 37 (movement control) that moves the robot 10 to the calculated target point. And a transmission / reception unit 31 (an example of a first transmission unit) that transmits the image data 200 acquired by the imaging device 21 at the calculated target point to the display terminal 50. The display terminal 50 also includes a transmitting / receiving unit 51 (an example of a receiving unit) that receives the image data 200 transmitted from the robot 10 and a display that displays the received image data 200 on a display 512 (an example of a displaying unit). A control unit 53 (an example of a display control unit). Therefore, the remote control system 1a can activate the remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50.

  Further, the remote control system 1b according to one embodiment of the present invention includes a robot 10 (an example of a moving object) including an image capturing device 21 that captures an image of a subject and acquires image data 200, and the robot 10 and the communication network 9 via the communication network 9. And a display terminal 50 that can remotely control the robot 10. Then, the display terminal 50 is configured to transmit and receive the image data 200 acquired by the imaging device 21 based on the position and orientation of a plurality of persons included in the acquired image data 200. A target point calculating unit 62 and a target point calculating unit 63 (an example of a calculating unit) for calculating a target point indicating a destination of the robot 10, and a request command (request information) for moving the robot 10 to the calculated target point. And a transmission / reception unit 51 (an example of a second transmission unit) for transmitting the robot 10. Further, the robot 10 acquires the movement control unit 37 (an example of a movement control unit) that moves the robot 10 to the calculated target point, and acquires the target point included in the request command transmitted from the display terminal 50 by the imaging device 21. And a transmission / reception unit 31 (an example of a first transmission unit) for transmitting the image data 200 to the display terminal 50. Further, the display terminal 50 includes a transmission / reception unit 51 (an example of a receiving unit) that receives the image data 200 transmitted from the robot 10 and a display that displays the received image data 200 on a display 512 (an example of a display unit). A control unit 53 (an example of a display control unit). Therefore, the remote control system 1b can activate the remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50.

  Further, the remote control system 1c according to one embodiment of the present invention includes a robot 10 (an example of a moving object) including an image capturing device 21 that captures an image of a subject and acquires image data 200, and the robot 10 and the robot 10 via the communication network 9. And a display terminal 50 (an example of an information processing device) that can communicate with the robot 10 and that can communicate with each of the robot 10 and the display terminal 50 via the communication network 9. . The image processing server 70 is configured to transmit and receive the image data 200 acquired by the imaging device 21 based on the position and orientation of a plurality of persons included in the acquired image data 200. The target point calculation unit 75 and the target point calculation unit 76 (an example of a calculation unit) that calculate a target point indicating the destination of the robot 10, and request data (request data) for moving the robot 10 to the calculated target point And a transmission / reception unit 71 (an example of a third transmission unit) that transmits the information (an example of information) to the robot 10. In addition, the robot 10 is controlled by the movement control unit 37 (an example of a movement control unit) that moves the robot 10 to the calculated target point, and by the photographing device 21 at the target point included in the request data transmitted from the image processing server 70. A transmission / reception unit 31 (an example of a first transmission unit) that transmits the acquired image data 200 to the display terminal 50. Further, the display terminal 50 includes a transmission / reception unit 51 (an example of a receiving unit) that receives the image data 200 transmitted from the robot 10 and a display that displays the received image data 200 on a display 512 (an example of a display unit). A control unit 53 (an example of a display control unit). Therefore, the remote control system 1c can activate remote communication using the robot 10 between the base where the robot 10 is located and the display terminal 50.

● Supplement ●
The functions of each embodiment can be realized by a computer-executable program described in a legacy programming language such as assembler, C, C ++, C #, Java (registered trademark), or an object-oriented programming language. , EPROM (Erasable Programmable Read-Only Memory), flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disk, SD card, MO (Magneto-Optical disc) And the like, or can be distributed via a telecommunication line.

  Further, a part or all of the functions of each embodiment can be implemented on a programmable device (PD) such as an FPGA (Field Programmable Gate Array), or can be implemented as an ASIC. Circuit configuration data (bit stream data) to be downloaded to the PD in order to realize the function of PD on the PD, HDL (Hardware Description Language) for generating the circuit configuration data, VHDL (Very High Speed Integrated Circuits Hardware Description Language), It can be distributed on a recording medium as data described in Verilog-HDL or the like.

  Although the information processing apparatus, the moving object, the remote control system, the information processing method, and the program according to the embodiment of the present invention have been described above, the present invention is not limited to the above-described embodiment, and other embodiments are applicable. Additions, changes, or deletions of the forms can be made within a range that can be conceived by those skilled in the art, and any aspect is within the scope of the present invention as long as the functions and effects of the present invention are exhibited. is there.

1a, 1b, 1c Remote control system 10 Robot (an example of a moving object)
17 Moving mechanism 21 Imaging device 30 Control device (an example of an information processing device)
31 Transmission / Reception Unit (Example of First Transmission Means)
36 Image acquisition unit (an example of an acquisition unit)
37 Movement control unit (an example of movement control means)
39 Position specifying unit (an example of a position detecting unit)
41 Image processing unit 42 Attention point calculation unit (an example of a calculation unit, an example of an estimation unit, and an example of an attention point identification unit)
43 Target point calculating unit (an example of a calculating unit, an example of a target point specifying unit)
45 Presence notification unit (an example of a presence notification unit)
50 display terminal (an example of an information processing device)
51 Transmission / Reception Unit (Example of Acquisition Unit, Example of Reception Unit, Example of Second Transmission Unit)
53 display control unit (example of display control means)
61 Image processing unit 62 Attention point calculation unit (an example of a calculation unit, an example of an estimation unit, and an example of an attention point identification unit)
63 Target point calculating unit (an example of a calculating unit, an example of a target point specifying unit)
70 Image processing server (an example of an information processing device)
71 Transmission / Reception Unit (Example of Acquisition Unit, Example of Third Transmission Unit)
74 Image processing unit 75 Attention point calculation unit (an example of a calculation unit, an example of an estimation unit, and an example of an attention point identification unit)
76 Target point calculating unit (an example of a calculating unit, an example of a target point specifying unit)
90 Management server

JP 2012-161851 A

Claims (15)

An information processing device that adjusts the position of a moving object including a photographing device that photographs a subject and acquires image data,
Acquisition means for acquiring the image data acquired by the imaging device,
Based on the positions and orientations of a plurality of persons included in the acquired image data, a calculating unit that calculates a target point indicating a destination of the moving body,
An information processing apparatus for moving the moving body to the target point by calculating the target point by the calculating unit.   The information processing according to claim 1, wherein when the position or orientation of at least one of the plurality of persons changes, the calculation unit calculates the target point based on the changed position or orientation of the person. apparatus. The information processing apparatus according to claim 1 or 2,
The calculating means,
Estimating means for estimating the position and orientation of a plurality of persons included in the obtained image data,
Attention point identification means for identifying an attention point to which the plurality of persons pay attention, based on the estimated position and orientation,
Target point specifying means for specifying the target point based on the specified point of interest,
Information processing device having The target point includes position information indicating a target position of the destination,
The target point specifying means,
The information processing apparatus according to claim 3, wherein a position separated from the estimated point of interest by an average value of a distance between the point of interest and the plurality of persons is specified as the target position. The target point includes direction information indicating an orientation of the moving body at the destination,
The target point specifying means,
The information processing apparatus according to claim 3, wherein a direction of the estimated point of interest viewed from a current position of the moving object is specified as a direction of the moving object. " An information processing apparatus according to any one of claims 1 to 5,
A moving mechanism for moving the moving body to the calculated target point,
Mobile body equipped with. The mobile object according to claim 6, further comprising:
Position detecting means for detecting a current position of the moving body,
When the distance between the detected current position and the calculated target point is equal to or less than a threshold, or is smaller than the threshold, presence notification means for outputting notification information for notifying the presence of the moving object,
A moving body comprising: The notification information is a sound signal,
The moving body according to claim 7, wherein the presence notification unit outputs the sound signal when a distance between the detected current position and the calculated target point is equal to or larger than a threshold or longer than the threshold. A remote control system comprising: a moving body including a photographing device for photographing a subject to acquire image data; and a display terminal capable of communicating with the moving body via a communication network and remotely controlling the moving body. hand,
Acquisition means for acquiring the image data acquired by the imaging device,
Calculating means for calculating a target point indicating a destination of the moving body, based on the positions and orientations of the plurality of persons included in the obtained image data,
Movement control means for moving the moving body to the calculated target point,
Remote control system comprising: The remote control system according to claim 9,
The moving object is
Image data acquired by the imaging device at the calculated target point, the first transmission means for transmitting to the display terminal,
The display terminal,
Receiving means for receiving the image data transmitted by the first transmitting means;
A display control means for displaying the received image data on a display means. The remote control system according to claim 9,
The display terminal,
Requesting information for moving the moving body to the calculated target point, the second transmitting means for transmitting to the moving body,
The moving object is
First transmission means for transmitting image data acquired by the imaging device at the target point included in the request information transmitted by the second transmission means to the display terminal,
The display terminal further comprises:
Receiving means for receiving the image data transmitted by the first transmitting means;
A display control means for displaying the received image data on a display means. The remote control system according to claim 9, further comprising:
An information processing device capable of communicating with each of the moving object and the display terminal via the communication network,
The information processing device,
A third transmission unit that transmits request information for moving the moving body to the calculated target point to the moving body,
The moving object is
A first transmission unit for transmitting image data acquired by the imaging device at the target point included in the request information transmitted by the third transmission unit to the display terminal;
The display terminal,
Receiving means for receiving the image data transmitted by the first transmitting means;
A display control means for displaying the received image data on a display means. An information processing method executed by an information processing apparatus that adjusts a position of a moving object including an imaging device that captures an image of a subject and obtains image data,
An acquisition step of acquiring the image data acquired by the imaging device;
Based on the positions and orientations of the plurality of persons included in the obtained image data, a calculating step of calculating a target point indicating a destination of the moving body,
An information processing method for moving the moving body to the target point by calculating the target point in the calculating step.   A program for causing a computer to execute the method according to claim 13. An information processing device that adjusts the direction of a transmission terminal including a photographing device that photographs a subject and acquires image data,
Acquisition means for acquiring the image data acquired by the imaging device,
Based on the positions and orientations of a plurality of persons included in the obtained image data, and a calculating unit that calculates the direction of the destination of the transmission terminal,
An information processing apparatus for changing the direction of the transmission terminal to the direction of the change destination by calculating the direction of the change destination by the calculating means.