JP2019079091A - Moving body control program, moving body control method and information processing apparatus - Google Patents

Abstract

To improve operability in control of a moving body.SOLUTION: An information processing apparatus 10 includes a display unit 13, a detection unit 12, and control unit 11. The display unit 13 displays image information photographed by an imaging unit of a moving body 20 equipped with the imaging unit. The detection unit 12 detects a detection coordinate that is a coordinate on a screen of the display unit 13 at which an operator gazes. The control unit 11 receives the image information from the moving body 20, determines control information for controlling the moving body 20 according to the detection coordinate, and notifies the moving body 20 of the control information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a mobile control program, a mobile control method, and an information processing apparatus.

  In recent years, there is a technique of detecting a line of sight from a corneal curvature center and a pupil center obtained from the position of a reflection image on the surface of the cornea by irradiating near infrared rays or the like to the eyeballs of the user. By using this technology for detecting the sight line and determining the position on the display at which the user gazes, a technology for selecting an object on the display without the user operating the keyboard is realized.

  Moreover, in recent years, the technology of the flying body called drone (drone) has advanced, and the use is expected in distribution, aerial photography, surveying, pesticide spraying, and the like. The drone is wirelessly controlled by a controller using wireless communication.

JP, 2012-65781, A

  Heretofore, the drone has been controlled by the operator manually operating the controller. The drone's controller has two control sticks, a control stick for manipulating up and down and left and right rotational movements and a control stick for manipulating back and forth and left and right movements. The operator is required to operate the two control sticks with the left and right hands respectively. It is difficult for the controller to control the drone for those who are not familiar with the operation.

  In addition, in the operation of the mobile unit wirelessly controlled as well as the drone, the operator often operates the controller with both hands to control the position of the mobile unit. For this reason, it has been difficult for the operator to execute operations of functions (operation of an arm, operation of an angle of a camera, operation of agrochemical spraying, etc.) other than movement provided in a moving body.

For this reason, it is desirable to improve the operability in the operation of moving the moving object to be wirelessly controlled.
In one aspect, an object of the present invention is to provide a mobile control program, a mobile control method, and an information processing apparatus capable of improving operability in control of a mobile.

  In one aspect, a mobile control program is provided. The moving body control program receives the image information captured by the imaging unit from the moving body including the imaging unit, and on the display unit displaying the image information, the coordinates are on the screen of the display unit at which the operator is gazing The detection coordinates are detected, control information for controlling the moving body is determined according to the detection coordinates, and the control information is notified to the moving body.

  In one aspect, operability can be improved in the control of the moving object.

It is a figure showing an example of the mobile control system of a 1st embodiment. It is a figure showing an example of the control object control system of a 2nd embodiment. It is a figure showing an example of the hardware of an information processor. It is a figure showing an example of the hardware of a flight body. It is a figure which shows an example of a gaze detection coordinate and a detection area. It is a figure which shows another example of a detection area | region. It is a block diagram showing an example of a function which a flight control system has. It is a figure which shows an example of control management information. It is a flowchart which shows an example of a flying body control process. It is a flow chart which shows an example of movement control processing. It is a flowchart which shows an example of an image display process.

Hereinafter, the present embodiment will be described with reference to the drawings.
First Embodiment
FIG. 1 is a diagram showing an example of a mobile control system according to the first embodiment.

  The mobile control system according to the first embodiment includes an information processing apparatus 10 and a mobile 20. The moving object control system is a system in which the information processing apparatus 10 detects the coordinates of the line of sight of the operator and controls the movement of the moving object 20 based on the detected coordinates. The information processing apparatus 10 and the mobile unit 20 can communicate wirelessly.

  The information processing apparatus 10 is, for example, a computer having a control unit 11 and a storage unit. The information processing apparatus 10 includes a detection unit 12 and a display unit 13. The storage unit is not shown.

  The control unit 11 controls the information processing apparatus 10 and is, for example, a processor. The processor is a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like. The control unit 11 determines control information for controlling the movement of the mobile unit 20 based on the information detected by the detection unit 12 and notifies the mobile unit 20 of the control information.

  The control information includes the movement control instruction and the execution time of the movement control instruction. The movement control instruction is, for example, an instruction (command) for moving the moving body 20 in the up, down, left, and right directions. The execution time is, for example, a time to execute an instruction to move the mobile 20. The control unit 11 receives image information from the moving body 20 and controls the display unit 13 to display the image information. The storage unit is a storage device that stores various programs and information, and is, for example, a random access memory (RAM), a hard disk drive (HDD), a solid state drive (SSD), or the like.

  The detection unit 12 is a sensor that detects the line of sight of the operator who operates the information processing apparatus 10. The detection unit 12 acquires coordinates (detection coordinates) on the screen at which the operator gazes at the display unit 13 and a time at which the operator gazes (gaze time), and outputs the acquired time to the control unit 11. The detected coordinates can also be said to be coordinates of an intersection point of a straight line corresponding to the line of sight of the operator and a plane corresponding to the screen of the display unit 13.

The display unit 13 displays an image, and is, for example, a display. The display unit 13 displays an image according to the input image information according to the control of the control unit 11.
The moving body 20 is, for example, a flying body such as a drone flying at an altitude of several cm to several tens of meters from the ground. Although a drone is drawn as an example of the mobile unit 20 in FIG. 1, a mobile unit other than the drone may be used. A drone is a multicopter (rotorcraft equipped with three or more rotors) having a remotely controlled or autonomous control mechanism. Examples of mobiles other than drone include radio-controllable vehicles, ships, submersibles, robots and endoscopes.

The movable body 20 includes a control unit, an imaging unit, and a drive unit, but the illustration is omitted. The control unit controls the mobile unit 20 and is, for example, a processor.
The control unit controls the drive unit based on the control information received from the information processing apparatus 10. For example, when the control unit receives control information for executing an instruction to move upward for 3 seconds, the control unit controls the driving unit based on the control information. The control unit controls the imaging unit to capture an image, and transmits the captured image data to the information processing apparatus 10. The imaging unit is provided in front of the movable body 20 and picks up an image, and is, for example, a camera. The orientation of the imaging unit is fixed to face the front of the moving body 20. Then, the imaging unit images the scenery in front of the moving body 20. When the mobile unit 20 moves, the imaging unit images a landscape facing in the traveling direction of the mobile unit 20. The drive unit controls the rotary wings of the movable body 20 according to the control of the control unit.

  Here, a mobile control method for controlling the mobile 20 by the information processing apparatus 10 will be described. First, the operator visually observes the screen of the display unit 13 of the information processing apparatus 10. The screen displayed on the display unit 13 is assumed to be the first screen 13a. The first screen 13 a is one in which image information of an image obtained by capturing a trapezoidal object (for example, a mountain) by a camera provided in front of the moving body 20 is displayed on the display unit 13. The information processing apparatus 10 displays image information on the display unit 13 each time image information is received from the moving body 20. In the screen displayed on the display unit 13, the central portion is set as the reference position 13o, and a predetermined region including the central portion is set as the reference region 13p. The reference position 13o is merely an example, and another position may be used as the reference position 13o. The reference area 13p is merely an example, and areas of other shapes (for example, a rectangle, an ellipse, etc.) may be used as the shape of the reference area 13p.

  The detection unit 12 detects the coordinates on the first screen 13a at which the operator is gazing. The detection unit 12 detects that the coordinates detected by the operator detected on the first screen 13a are the detected coordinates 13j. The control unit 11 acquires the detected coordinates 13 j from the detection unit 12, determines that the detected coordinates 13 j are included in the reference area 13 p, and transmits control information including a movement control command for hovering to the moving body 20. The control unit 11 can include, in the control information, a movement control instruction (hovering command) that performs hovering for a predetermined time (for example, 10 seconds). The mobile unit 20 receives control information from the control unit 11 and performs hovering. Note that the movement control command for hovering is merely an example, and may be a command that maintains the current (time when the operator gazes at the reference area 13p) according to the moving body 20. For example, when the moving body 20 is a vehicle or a robot, a command to stop the moving body 20 may be used. When the moving body 20 is a ship or a submersible, it may be a command to move in a direction against the flow of the tide or a command to drop a weir in order to maintain the position at the time of watching the reference area 13p.

  Next, it is assumed that the operator moves the line of sight. The detection unit 12 detects that the coordinates at which the operator is gazing are detected coordinates 13k. The control unit 11 acquires the detected coordinate 13k from the detection unit 12, determines that the detected coordinate 13k is in the upper left direction with respect to the reference position 13o, and moves to the upper left movement control instruction (left advancing command and rising command) And transmit control information to the mobile unit 20. The mobile unit 20 receives control information from the control unit 11 and moves to the upper left.

  The moving body 20 moves to the upper left, and notifies the information processing apparatus 10 of the captured image information. The information processing apparatus 10 acquires image information from the moving body 20 and displays the second screen 13 b on the display unit 13.

  Thus, when the detected coordinates 13k are located in the first direction with respect to the reference position 13o, the information processing apparatus 10 instructs the information processing unit 10 to move the moving body 20 in the first direction with respect to the imaging direction by the imaging unit of the moving body 20. The mobile 20 is notified. The first direction is a direction relative to a component horizontal to the ground in the imaging direction. The first direction may be a direction from the reference position 13o to the detection coordinate 13k with respect to the coordinate system (X coordinate and Y coordinate) on the screen of the display unit 13.

  As described above, the information processing apparatus 10 detects the coordinates on the screen at which the operator is gazing, determines control information for controlling the moving body 20 according to the detected coordinates, and notifies the moving body 20 of the control information. As a result, the operator can move the moving body 20 by gazing the display unit 13 and moving the sight line, and can move the moving body without operating the controller manually.

Thus, the mobile control system can improve operability in the control of the mobile 20.
Second Embodiment
FIG. 2 is a diagram showing an example of a flying object control system according to the second embodiment. The flying object control system of the second embodiment includes an information processing device 100 and a flying object 200.

  The information processing apparatus 100 is a computer that detects the line of sight of the operator and transmits control information to the flying object 200. The information processing apparatus 100 includes an apparatus for detecting a line of sight, and transmits control information corresponding to the detected line of sight to the aircraft 200. The information processing apparatus 100 receives image information from the flying object 200 and displays the image information on a display unit such as a display.

  The information processing apparatus 100 can wirelessly communicate with the aircraft 200. Examples of standards for wireless communication include standards such as IEEE (The Institute of Electrical and Electronic Engineers) 802.15.1 (Bluetooth (registered trademark)) and IEEE 802.11 (Wireless Local Area Network (LAN)). The standard used for wireless communication between the information processing apparatus 100 and the aircraft 200 may be another standard such as LTE (Long Term Evolution) or Wi-Fi (registered trademark).

  The flying object 200 is a device that receives control information from the information processing device 100 and moves based on the control information. As an example of the flying object 200, a drone, which is a type of rotary wing aircraft, is illustrated. A drone is a multicopter (rotorcraft with three or more rotors) that has a remotely controlled or autonomous control mechanism. In addition, the drone has a camera and a communication function, and can capture and transmit image information. The drone has a remote control type control function that remotely controls the movement route and various processes, and an autonomous control function that executes a program that autonomously controls the movement route and various processes There is something that is provided. Here, although a drone provided with a remote-controlled control function is targeted as the flying object 200, an autonomous control function may be provided.

  Here, a case where the operator operates the flying object 200 using the information processing apparatus 100 will be described. First, the information processing apparatus 100 detects the line of sight of the operator. Specifically, the information processing apparatus 100 acquires coordinate information (line-of-sight detection coordinates) in which the line of sight is detected at predetermined time intervals (for example, 15 times per second) from the apparatus that detects the line of sight. The information processing apparatus 100 transmits control information for executing movement control to the flying object 200 based on the gaze detection coordinates. The information processing apparatus 100 also receives image information captured by the camera of the flying object 200 at predetermined time intervals and displays the image information on a display unit such as a display. The flying object 200 is provided with a camera that faces in the forward direction of the flying object 200 and captures a landscape in the forward direction. It is assumed that the flying object 200 transmits image information obtained by imaging a rectangular object from the front using the camera to the information processing apparatus 100. The information processing apparatus 100 receives image information from the flying object 200 and displays the image information on the display unit.

  Next, the information processing apparatus 100 detects that the line of sight of the operator has moved. The information processing apparatus 100 transmits control information for executing movement control to the flying object 200 based on the gaze detection coordinates. For example, when the information processing apparatus 100 detects that the line of sight has moved to the upper left, the control information to move to the upper left (with respect to the imaging direction of the flying object 200) ahead of the Send. The flying object 200 moves based on the control information received from the information processing device 100. The flying object 200 transmits, to the information processing apparatus 100, image information captured from the moved position (image information including an image of a rectangular object entering the field of view of the camera from the upper left front of the position before moving). The information processing apparatus 100 displays the image information received from the flying object 200 on the display unit.

  In this manner, the information processing apparatus 100 detects the line of sight of the operator, and transmits control information to the aircraft 200 based on the line-of-sight detection coordinates, thereby operating the aircraft 200. In addition, the operator can operate the flying object 200 while visually confirming the image displayed on the display unit of the information processing apparatus 100.

  FIG. 3 is a diagram illustrating an example of hardware of the information processing apparatus. The information processing apparatus 100 includes a processor 101, a RAM 102, an HDD 103, an image signal processing unit 104, an input signal processing unit 105, a medium reader 106, an NIC (Network Interface Card) 107, and an external interface 108. Each piece of hardware is connected to the bus of the information processing apparatus 100.

  The processor 101 is hardware that controls the information processing of the information processing apparatus 100. The processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU, a DSP, an ASIC, or an FPGA. The processor 101 may be a combination of two or more elements of a CPU, a DSP, an ASIC, an FPGA, and the like.

  The RAM 102 is a main storage device of the information processing apparatus 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data used for processing by the processor 101.

  The HDD 103 is an auxiliary storage device of the information processing apparatus 100. The HDD 103 magnetically writes data to and reads data from the built-in magnetic disk. The HDD 103 stores an OS program, an application program, and various data. The information processing apparatus 100 may include other types of auxiliary storage devices such as a flash memory and an SSD, and may include a plurality of auxiliary storage devices.

  The image signal processing unit 104 outputs an image to the display 121 connected to the information processing apparatus 100 in accordance with an instruction from the processor 101. As the display 121, a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like can be used.

  The input signal processing unit 105 acquires an input signal from the input device 122 connected to the information processing apparatus 100, and outputs the input signal to the processor 101. As the input device 122, for example, a pointing device such as a mouse or a touch panel, a keyboard, or the like can be used.

  The medium reader 106 is a device that reads a program or data recorded on the recording medium 123. As the recording medium 123, for example, a magnetic disk such as a flexible disk (FD: Flexible Disk) or HDD, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), a magneto-optical disk (MO: Magneto-Optical disk) It can be used. Also, as the recording medium 123, for example, a nonvolatile semiconductor memory such as a flash memory card can be used. The medium reader 106 stores programs and data read from the recording medium 123 in the RAM 102 or the HDD 103, for example, in accordance with an instruction from the processor 101.

The NIC 107 communicates with other devices via the network 130. The NIC 107 may be a wired communication interface or a wireless communication interface.
The external interface 108 acquires a signal from the line-of-sight detection device 140 connected to the information processing device 100 and outputs the signal to the processor 101. The line-of-sight detection device 140 detects coordinates (line-of-sight detection coordinates) on the display 121 at which the operator is gazing, and can detect time during which the operator is gazing at predetermined coordinates.

  FIG. 4 is a diagram showing an example of the hardware of the flying object. The flying object 200 includes a CPU 201, a memory 202, a camera 203, a communication processing unit 204a, an antenna 204b, a driving unit 205a, and a rotor 205b.

The CPU 201 implements control of the flying object 200. The CPU 201 can be replaced by a DSP, an ASIC, an FPGA or the like.
The memory 202 is, for example, a storage device such as a RAM, a read only memory (ROM), an HDD, or an SSD.

  The camera 203 is provided in front of the flying object 200, captures an image, and outputs image data to the CPU 201. The camera 203 may be capable of capturing both moving and still images, or may be capable of capturing only moving images.

  The communication processing unit 204a includes an antenna 204b, and transmits and receives information to and from the information processing apparatus 100. For example, the communication processing unit 204 a receives control information from the information processing apparatus 100 and outputs the control information to the CPU 201. Also, the communication processing unit 204 a transmits image information of an image captured by the camera 203 to the information processing apparatus 100. The communication processing unit 204a includes an RF (Radio Frequency) circuit necessary for communication, a signal processing circuit, and the like. The RF circuit executes processing such as transmission / reception of an RF signal by the antenna 204b, frequency conversion, AD (Analog to Digital) / DA (Digital to Analog) conversion processing, and the like. The signal processing circuit executes processing such as modulation and demodulation. The antenna 204 b is an antenna used to transmit and receive an RF signal.

  The driving unit 205a drives a motor connected to the rotation shaft of a rotor 205b which is a wing of a rotary wing aircraft, and rotates the rotor 205b. The driving unit 205a controls the rotation of the rotor 205b to control the horizontal position, the altitude, the attitude, the moving direction, and the like. Although the flying body 200 includes the plurality of rotors 205b, the illustration is omitted.

  In addition, the flying body 200 can be equipped with the sensor group which is not shown in figure. The sensor group is, for example, a collection of sensors including an air pressure sensor and an ultrasonic sensor. A barometric pressure sensor is used, for example, for measurement of altitude. The ultrasonic sensor is used, for example, for detection of an obstacle. The sensor group may include a GPS (Global Positioning System), an acceleration sensor, etc. that can be used to measure the current position.

  FIG. 5 is a diagram showing an example of gaze detection coordinates and a detection area. FIG. 5A is a diagram showing an example of gaze detection coordinates. The information processing device 100 detects the coordinates at which the operator gazes at the screen of the display 121 via the sight line detection device 140. Here, it is assumed that coordinates at which the operator gazes are gaze detection coordinates 350. In the screen, the center position is set as the origin 301, and the position of the gaze detection coordinate 350 is indicated by the X coordinate and the Y coordinate. The positive direction of the X axis is the horizontal right direction from the origin 301. The negative direction of the X axis is the horizontal left direction from the origin 301. The positive direction of the Y axis is vertically upward from the origin 301. The negative direction of the Y axis is vertically downward from the origin 301. The gaze detection coordinates 350 may be indicated by values such as dots and pixels.

  FIG. 5B is a diagram showing an example of the detection area. The detection area is an area obtained by dividing the screen of the display 121 into a plurality of sections. The detection area is the upper left instruction area 302, the upper instruction area 303, the upper right instruction area 304, the left horizontal instruction area 305, the center area 306, the right horizontal instruction area 307, the lower left instruction area 308, the downward instruction area 309 and the lower right instruction area 310. including. In addition, the area | region which divides a detection area is only an example, and it is also possible to divide a detection area in another area | region.

  The information processing apparatus 100 determines control information to be transmitted to the flying object 200 in accordance with the detection area in which the sight line detection coordinates 350 are included. For example, when the information processing apparatus 100 detects that the line-of-sight detection coordinate 350 is included in the upper left instruction area 302, the information processing apparatus 100 transmits control information for moving to the upper left to the aircraft 200.

  FIG. 6 is a diagram showing another example of the detection area. The detection area shown in FIG. 6 is an area obtained by dividing the screen of the display 121 into five-row, five-column, matrix-like sections. Each detection area can be identified by row and column. For example, the upper left detection area is identified as detection area A1, and the lower right detection area is identified as detection area E5. The detection area is not limited to m rows and m columns, and may be areas divided into m rows and n columns (m ≠ n) into matrix-like sections.

  Although FIG. 5 shows an example in which the visual axis detection device 140 detects in detail the position at which the operator gazes, in FIG. 6 the accuracy of detection of the position at which the visual axis detection device 140 looks at is An example of the low case is shown.

In FIG. 6, it is assumed that the visual axis detection device 140 detects that the position at which the operator gazes is included in any of the detection areas A1 to E5.
For example, when the position at which the operator gazes is included in the detection area A1, the visual axis detection device 140 outputs (−2, 2) as the visual axis detection coordinate 350 to the processor 101. In addition, when the position at which the operator gazes is included in the detection area E5, the line-of-sight detection device 140 outputs (2, -2) as the line-of-sight detection coordinate 350 to the processor 101.

  The information processing apparatus 100 treats the detection areas A1, A2, B1 and B2 as the upper left instruction area 302, treats the detection areas A3 and B3 as the upward instruction area 303, and instructs the detection areas A4, A5, B4 and B5 as the upper right It can be treated as the area 304. Further, the information processing apparatus 100 can treat the detection areas C1 and C2 as the left lateral indication area 305, treat the detection area C3 as the central area 306, and treat the detection areas C4 and C5 as the right lateral indication area 307. Further, the information processing apparatus 100 treats the detection areas D1, D2, E1, and E2 as the lower left instruction area 308, treats the detection areas D3, E3 as the descent instruction area 309, and lowers the detection areas D4, D5, E4, and E5. It can be treated as the instruction area 310.

  As described above, even when the accuracy of the visual axis detection device 140 is low, the information processing apparatus 100 can detect which detection area the operator is gazing at, and can determine control information according to the detection area. .

  FIG. 7 is a block diagram showing an example of the functions of the aircraft control system. The information processing apparatus 100 includes a control unit 150, a storage unit 151, and a display unit 152. The control unit 150 is realized by the processor 101. Specifically, the processor 101 executes the program stored in the RAM 102 to exert the function of the control unit 150. However, the control unit 150 may be realized by hard wired logic such as FPGA or ASIC.

  The storage unit 151 is realized using the storage area of the RAM 102 and the storage area of the HDD 103. The storage unit 151 stores control management information. The control management information is information used by the information processing apparatus 100 to control the flying object 200 based on the gaze detection coordinates. Control management information will be described later with reference to FIG.

  The display unit 152 is realized by the display 121. The display unit 152 displays information in accordance with the control of the control unit 150. The display unit 152 may be externally attached to the information processing apparatus 100 or may be integrated with the information processing apparatus 100.

  The control unit 150 transmits control information to the flying object 200 based on the gaze detection coordinates detected by the gaze detection device 140 and the control management information stored in the storage unit 151. The control unit 150 also displays the image information received from the flying object 200 on the display unit 152.

  The flying object 200 has a control unit 250. The control unit 250 is realized by the CPU 201. Specifically, the CPU 201 executes the program stored in the memory 202 to exert the function of the control unit 250. However, the control unit 250 may be realized by hard wired logic such as FPGA or ASIC.

  The control unit 250 captures an image with the camera 203, and transmits the captured image data to the information processing apparatus 100. The control unit 250 controls the drive unit 205 a based on the control information received from the information processing apparatus 100 to move the flying object 200.

  FIG. 8 is a diagram showing an example of control management information. The control management information 401 is information for managing control information according to the detection area. The control management information 401 is information stored in the storage unit 151 of the information processing apparatus 100.

Control management information 401 includes items of an area, an execution command, and an execution time (seconds). Control information includes execution command and execution time.
A detection area is registered in the item of the area. The area registered in the detection area is as described in FIG. 5 (B).

  In the item of the execution command, a command for controlling the movement of the aircraft 200 is registered. Here, the moving direction of each command will be described. A camera 203 is provided in front of the flying object 200. When the flying object 200 executes a left advance command, the flying object 200 moves in the left direction with respect to the direction in which the camera 203 is imaging. In addition, the flying object 200 moves in the right direction with respect to the direction in which the camera 203 is capturing an image, when the right advance command is executed. The flying object 200 moves upward with respect to the direction in which the camera 203 is imaging, when the lifting command is executed. The flying object 200 moves downward with respect to the direction in which the camera 203 is imaging when the lowering command is executed.

In the item of execution time, the time to execute the execution command is registered. Note that | X coordinate | indicates the absolute value of X coordinate. In addition, | Y coordinate | indicates the absolute value of Y coordinate.
For example, in the control management information 401, the area is “upper left instruction area”, the execution command is “left progress command”, the execution time is “| X coordinate | × 0.2”, the execution command is “rise command” and the execution time The information that “Y coordinate × 0.1” is registered.

  This indicates the control information to be transmitted to the flying object 200 when the information processing apparatus 100 detects that the gaze detection coordinates at which the operator gazes are included in the upper left instruction area 302. The information processing apparatus 100 transmits, to the flying object 200, control information for executing the left advancing command for “| X coordinate | × 0.2” seconds, and control information for executing the rising command for “Y coordinate × 0.1” seconds. Indicates to do.

  As described above, when the operator gazes at the upper left instruction area 302 on the screen, the information processing apparatus 100 transmits to the flying object 200 the control information in which the left advancing command and the raising command are combined. Can be moved to the upper left. Further, the information processing apparatus 100 sets the execution time according to the distance (X coordinate, Y coordinate) between the origin 301 and the position (visual axis detection coordinate) at which the operator gazes. Thus, the information processing apparatus 100 can control the moving distance of the aircraft 200 in proportion to the distance.

In the control management information 401, by providing an upper limit to the execution time of the descent command, it is possible to avoid contact with the ground due to rapid descent.
The hovering command is merely an example, and any other command may be used as long as it can maintain the position of the aircraft 200 at the present time (time when the operator gazes at the central area 306). The same applies to the following description.

  Next, a procedure of transmitting control information to the flying object 200 by the information processing apparatus 100 will be described. FIG. 9 is a flowchart showing an example of a flight control process. Hereinafter, the process shown in FIG. 9 will be described in order of step number. The airframe control process is a process in which the information processing apparatus 100 transmits control information to the airframe 200 and operates the airframe 200. The control unit 150 of the information processing device 100 acquires the gaze detection coordinates and the gaze time from the gaze detection device 140, and executes the flying body control processing. Note that the sight line detection device 140 outputs the sight line detection coordinates and the gaze time to the control unit 150 at predetermined time intervals (for example, fifteen times per second).

[Step S11] The control unit 150 acquires line-of-sight detection coordinates from the line-of-sight detection device 140.
[Step S12] The control unit 150 determines whether or not the sight line detection coordinates are included in the central area 306. The control unit 150 proceeds to step S13 when the line-of-sight detection coordinates are included in the central area 306, and proceeds to step S11 when not included.

  [Step S13] The control unit 150 transmits a takeoff command to the aircraft 200. Specifically, the control unit 150 transmits control information for executing a rising command for a predetermined time (for example, 5 seconds) to the flying object 200. Here, it is assumed that the flying object 200 receives control information and takes off.

  [Step S14] The control unit 150 transmits a hovering command to the flying object 200. The control unit 150 transmits a hovering command to the aircraft 200 so that the aircraft 200 continues to hover until transmitting the next control information.

[Step S15] The control unit 150 acquires line-of-sight detection coordinates from the line-of-sight detection device 140.
[Step S16] The control unit 150 determines whether a predetermined time (for example, one second) has elapsed. If the predetermined time has elapsed, the control unit 150 proceeds to step S17. If the predetermined time has not elapsed, the control unit 150 proceeds to step S15.

  The process of this step is a process in which the control unit 150 acquires line-of-sight detection coordinates for determining which detection area the operator is gazing at. For example, when the gaze detection apparatus 140 sends gaze detection coordinates to the control unit 150 at a frequency of 15 times per second, the control unit 150 acquires fifteen gaze detection coordinates in one second which is a predetermined time. Further, it is assumed that the time during which the operator gazes at one gaze detection coordinate is one fifteenth of a second (1/15 second).

  [Step S17] The control unit 150 determines whether the sight line detection coordinates are within the detection area. If the line of sight detection coordinates are within the detection area, the control unit 150 proceeds to step S18. If the sight line detection coordinates are not within the detection area, in other words, if the sight line detection coordinates are outside the screen, the control unit 150 proceeds to step S19.

  [Step S18] The control unit 150 executes movement control processing. The movement control process is a process of specifying control information to be transmitted to the flying object 200 based on the gaze detection coordinates and the gaze time. The movement control process will be described later with reference to FIG.

  [Step S19] The control unit 150 determines whether a predetermined time (for example, 15 seconds) has elapsed. If the predetermined time has elapsed, in other words, if the state in which the sight line detection coordinates are outside the detection area continues for a predetermined time, the control unit 150 proceeds to step S20. If the predetermined time has not elapsed, in other words, if the state in which the sight line detection coordinates are outside the detection area has not continued for a predetermined time, the control unit 150 proceeds to step S15.

[Step S20] The control unit 150 transmits a landing command to the aircraft 200, and ends the process.
Note that the control unit 150 of the information processing apparatus 100 may execute takeoff and landing by another method. For example, the information processing apparatus 100 may transmit the takeoff command to the aircraft 200 in response to the input of the takeoff command by the operator. Further, the information processing apparatus 100 may transmit the landing command to the aircraft 200 in response to the input of the landing command by the operator.

  FIG. 10 is a flowchart showing an example of the movement control process. Hereinafter, the process shown in FIG. 10 will be described in order of step number. The movement control process is a process of specifying control information to be transmitted to the flying object 200 based on the gaze detection coordinates and the gaze time. The movement control process is a process executed by the control unit 150 in step S18 of the flight vehicle operation process.

[Step S31] The control unit 150 specifies an area at which the operator gazes, based on the gaze detection coordinates acquired in Step S15.
For example, when the number of eye gaze detection coordinates acquired in step S15 is 15 and the number of eye gaze detection coordinates included in the upper left designation area 302 is the largest, the control unit 150 specifies the upper left designation area 302.

  [Step S32] The control unit 150 specifies control information from the control management information 401 based on the detection area. Specifically, the control unit 150 specifies an execution command corresponding to the area specified in step S31. In addition, the control unit 150 specifies the execution time corresponding to the execution command based on the gaze detection coordinates (X coordinate, Y coordinate).

  For example, when the upper left instruction area 302 is specified, and the line-of-sight detection coordinate is (-2, 2), the control unit 150 executes the left advancing command | -2 | × 0.2 = 0.4 (seconds). , Specify the control information for executing the rise command 2 × 0.1 = 0.2 (seconds).

[Step S33] The control unit 150 transmits the specified control information (execution command and execution time) to the aircraft 200.
[Step S34] The control unit 150 calculates a fixation time which is a time when the operator gazes at the detection area specified in Step S31.

  For example, in the case where the number of visual axis detection coordinates included in the upper left instruction area 302 is 10 in step S31, the time for which one visual axis detection coordinate is gazed is 1/15 second, so (1/15) × It can be calculated as 10 = 2/3 (seconds).

In addition, the calculation method of gaze time is only an example, and may be calculated by other methods, and the gaze time may be detected by a sensor.
[Step S35] The control unit 150 transmits, to the flying object 200, control information having the forward command and the execution time as the gaze time, and ends the processing. The forward command is a movement control command for moving the flying object 200 in the imaging direction (that is, the forward direction of the flying object 200) by the camera 203 included in the flying object 200.

  The control unit 150 transmits, to the flying object 200, control information for moving the flying object 200 in the direction in which the line of sight of the operator is detected, and sends, to the flying object 200, control information for advancing the flying object 200 according to the gaze time. . The longer the gaze time by the operator, the longer the execution time of the command for advancing the flying object 200, and hence the distance the flying object 200 travels forward becomes longer.

  Thus, the information processing apparatus 100 detects the line of sight of the operator with the line of sight detection apparatus 140 and controls the flying object 200. The operator can control the flying object 200 by moving the sight line on the screen of the display unit 152 without manually operating the controller.

  Next, the procedure of displaying the image information received from the flying object 200 by the information processing apparatus 100 will be described. FIG. 11 is a flowchart showing an example of the image display process. The process shown in FIG. 11 will be described below in order of step number.

  The image display process is a process of displaying the image information received from the flying object 200 by the control unit 150 of the information processing apparatus 100. The control unit 150 of the information processing apparatus 100 receives the image information from the flying object 200 and executes the image display process. The control unit 150 can execute the image display process independently of the flying object control process. In addition, the flying object 200 can transmit image information to the information processing apparatus 100 at predetermined time intervals.

[Step S41] The control unit 150 receives image information from the aircraft 200. The image information may be still image data or moving image data.
[Step S42] The control unit 150 displays the image information on the display unit 152, and the process goes to Step S41.

  In this manner, the control unit 150 displays the image information received from the flying object 200 on the display unit 152, so that the operator controls the flying object 200 while visually confirming the image information displayed on the display unit 152. Can.

  Thus, the flying object control system detects the line of sight of the operator in the information processing apparatus 100, specifies control information according to the coordinate information of the detected line of sight, and transmits the control information to the aircraft 200. Operation of the flying object 200 by the line of sight of

  In the first embodiment, the moving object 20 is illustrated as the control object, and in the second embodiment, the flying object 200 is illustrated as the control object, but other objects may be used as the control object. Good. For example, the mobile object to be controlled may be a wirelessly controllable vehicle, a ship, a submersible, a robot, an endoscope and the like. In this system, the movement of the moving body can be controlled by the line of sight, for example, the operator uses a hand to perform functions other than movement (operation of an arm provided to the moving body, camera angle operation, pesticide spreading operation, etc.) Operation can be easily performed.

  Note that the information processing of the first embodiment can be realized by causing the control unit 11 to execute a program. Also, the information processing of the second embodiment can be realized by causing the processor 101 to execute a program. The program can be recorded on a computer readable recording medium 123.

  For example, the program can be distributed by distributing the recording medium 123 in which the program is recorded. Alternatively, the program may be stored in another computer, and the program may be distributed via the network 130. For example, the computer may store (install) a program recorded in the recording medium 123 or a program received from another computer in a storage device such as the RAM 102 or the HDD 103, and read and execute the program from the storage device.

DESCRIPTION OF SYMBOLS 10 information processing apparatus 11 control part 12 detection part 13 display part 13a 1st screen 13b 2nd screen 13j, 13k detection coordinate 13o reference position 13p reference area 20 moving body

Claims (10)

Receiving image information captured by the imaging unit from a moving object including the imaging unit;
In the display unit on which the image information is displayed, detection coordinates which are coordinates on the screen of the display unit at which the operator is gazing are detected.
Control information for controlling the mobile unit is determined according to the detected coordinates, and the control information is notified to the mobile unit.
Mobile control program that causes a computer to execute processing. Determining a movement control command to move the moving body according to the direction from the reference position on the screen to the detected coordinates;
The mobile control program according to claim 1. The execution time of the movement control instruction is determined according to the distance between the reference position and the detected coordinates.
The mobile control program according to claim 2. Notifying the mobile unit of the control information including the movement control instruction and an execution time of the movement control instruction;
The mobile control program according to claim 2. In the determination of the movement control command, when the detected coordinates are positioned in a first direction with respect to the reference position, a command to move the moving body in a first direction with respect to an imaging direction by an imaging unit of the moving body As the movement control command,
The mobile control program according to claim 2. When the detected coordinates are included in a reference area including the reference position:
A command to maintain the position of the mobile at a time when the operator gazes at the reference area is determined as the movement control command.
Notifying the mobile body of control information including the movement control instruction and an execution time for causing the movement control instruction to be executed for a predetermined time;
The mobile control program according to claim 2. Determining a movement control command to move the moving body in the imaging direction by the imaging unit;
The execution time of the movement control command is determined according to the time at which the operator gazes.
The mobile control program according to claim 1. Each time the image information is received from the moving body, the image information is displayed;
Detecting the detected coordinates at predetermined time intervals;
The mobile control program according to claim 1. The computer is
Receiving image information captured by the imaging unit from a moving object including the imaging unit;
In the display unit on which the image information is displayed, detection coordinates which are coordinates on the screen of the display unit at which the operator is gazing are detected.
Control information for controlling the mobile unit is determined according to the detected coordinates, and the control information is notified to the mobile unit.
Mobile control method. A display unit that displays image information captured by the imaging unit of the moving object including the imaging unit;
A detection unit that detects detection coordinates that are coordinates on the screen of the display unit at which an operator is gazing;
A control unit that receives the image information from the mobile unit, determines control information for controlling the mobile unit according to the detected coordinates, and notifies the mobile unit of the control information;
An information processing apparatus comprising:

Images