JP7128409B2 - REMOTE CONTROL DEVICE, REMOTE CONTROL SYSTEM, REMOTE CONTROL METHOD AND REMOTE CONTROL PROGRAM - Google Patents

Description

The present invention relates to a remote control device, a remote control system, a remote control method and a remote control program.

Conventionally, there has been provided a line-of-sight control apparatus that includes communication means for receiving a line-of-sight detection signal from an imaging device having detection means for detecting a line of sight, and that controls a controlled object based on the line-of-sight detection signal transmitted from the imaging device. It has been proposed (for example, Patent Document 1).

A system for manipulating images using eyeball tracking has also been proposed (eg, Patent Document 2). The system places a plurality of hotspots associated with image parameter adjustments around a region of interest (ROI), and performs image adjustments associated with the hotspots when the user's gaze point is within the hotspot. I do.

JP-A-11-46318 Japanese Patent Publication No. 2009-530731

A controlled device such as a multi-copter, a multi-legged robot, a robot that runs on wheels, a crawler, etc., which is wirelessly or wiredly connected to the control device, is operated by the operator while viewing an image taken from the first-person viewpoint of the controlled device. It is sometimes convenient to be able to control by the driver's line of sight when the In other words, the controlled device can be operated even when the operator cannot use his/her hands. It is also possible to perform operations such as hands. However, there is a problem that the operation tends to be difficult because the operator simultaneously grasps the situation around the robot or the like and operates the robot or the like based on the line of sight.

An object of the present invention is to improve the operability of remote control by line of sight.

A remote control apparatus for a moving object according to the present invention is a control apparatus for remotely controlling a moving object to be controlled while a user views an image captured by an imaging device included in the moving object to be controlled. Acquiring gaze point information from a gaze detection device that detects the gaze of a user facing a display device that displays an image captured by an imaging device provided on a moving object and outputs the user's gaze point on the image, An operation detection unit that determines whether the point of gaze indicated by the acquired information is included in a predetermined area, and if the point of gaze indicated by the information is included in the predetermined area, a moving object to be controlled that is associated with the area and a signal output control unit for transmitting a signal for controlling the predetermined area has a waiting area for waiting the moving object to be controlled in the approximate center of the image, and inside the waiting area An advance area for advancing the moving object to be controlled is included at a position spaced apart from the center of the standby area by a predetermined distance.

In this way, the user can make the moving object to be controlled stand by by directing the user's line of sight to the waiting area arranged substantially in the center of the image. Therefore, the entire image can be recognized without causing an accident such as contacting the moving object to be controlled with an obstacle or the like. In addition, by moving the line of sight to the advance area arranged at a predetermined distance from the center in the standby area, the moving object to be controlled can be advanced. According to the area provided with such a positional relationship, the user can make the moving object to be controlled stand by only by moving the line of sight from the forward movement area to one of the surrounding directions. operability of remote control can be improved.

Further, the boundary of the predetermined area may be displayed as an image on the display device, or may be displayed on a sheet or panel superimposed on the display device. In this way, the user can clearly recognize the area for operation.

In addition, the forward region is defined by the distance l from the eyeball to the point of gaze on the image, the straight line connecting the eyeball and the point of gaze, and the straight line connecting the eyeball and the end of the central vision range on the image displayed on the display device. The radius r of the centrally visible range obtained when the angle between the Can be placed in range.
r = l tan θ

Further, the advance area may be arranged at a position r in the upward direction from the lowermost end in the downward direction from the center of the standby area. Also, θ may be 1 to 2 degrees. Specifically, by adopting such a size, it is possible to generally clearly recognize an object by central vision, and to suppress unintended control by the user due to microsaccades that occur in fixation.

Also, on the left and right of the stationary area, areas may be provided for horizontally rotating the moving object to be controlled to the left and right around the vertical axis of rotation. In this way, when objects of interest of the user exist on the left and right sides of the image, the user can remotely control the apparatus to change the direction of the device to be controlled by moving the line of sight in the direction in which these objects exist. can be operated. In addition, since it is difficult to grasp the situation on the left and right sides of the image, it is more likely that the device to be controlled will come into contact with an obstacle or the like if the device to be controlled is moved in parallel to the left or right. Therefore, the occurrence of accidents can be reduced by horizontally rotating the device to be controlled to the left and right.

Also, the moving object to be controlled is an unmanned flying object, and areas above and below the static area may be provided for raising and lowering the flying object. Note that the unmanned flying object may be any one of a multicopter, a drone, a helicopter, and an airship. In this way, when the user wants to move the unmanned flying object up and down, the user can remotely control the unmanned flying object by moving the line of sight up and down the stationary area, and can intuitively operate the unmanned flying object.

A remote control system according to the present invention includes the above-described remote control device, line-of-sight detection device, moving object to be controlled, and a controller for transmitting a signal for controlling the moving object to be controlled. With such a remote control system as well, the user can stop the device to be controlled simply by moving his or her line of sight from the forward movement area to any direction around the area. can improve sexuality.

The contents of the means for solving the above problems can be combined as much as possible without departing from the scope of the problems and technical ideas of the present invention. Further, the means may be realized as a method executed by a computer or as a program causing a computer to execute the means. The program may be recorded on a computer-readable recording medium and provided.

According to the present invention, it is possible to improve the operability of remote control by line of sight.

1 is a configuration diagram showing an example of a remote control system including a remote control device according to the present embodiment; FIG. 1 is a functional block diagram showing an example of a remote control system; FIG. It is a processing flow diagram showing an example of pre-processing performed by the control device of the remote control system. FIG. 4 is a processing flow diagram showing an example of processing performed by the multicopter; FIG. 4 is a processing flow diagram showing an example of processing performed by a controller; It is a processing flow diagram showing an example of processing performed by the display device. It is a processing flow diagram showing an example of processing performed by the line-of-sight detection device. It is a processing flow diagram showing an example of processing performed by a control device. FIG. 4 is a diagram showing an example of allocation of operation areas set on a display device; FIG. 4 is a diagram showing an example of a first-person viewpoint image of a multicopter for explaining the operation of the multicopter. FIG. 4 is a diagram showing an example of a first-person viewpoint image of a multicopter for explaining the operation of the multicopter; FIG. 4 is a diagram showing an example of a first-person viewpoint image of a multicopter for explaining the operation of the multicopter; FIG. 4 is a diagram for explaining the position and size of an operation area set on a display device; FIG. 4 is a diagram for explaining the position and size of an operation area set on a display device; It is a figure which shows the modification of arrangement|positioning of an operation area.

A remote control system according to an embodiment of the present invention will be described below with reference to the drawings.

<Device configuration>
FIG. 1 is a configuration diagram showing an example of a remote control system including a remote control device according to this embodiment. A remote control system 1 shown in FIG. The remote operation system 1 is a system for remotely operating a moving object to be operated based on the user's line of sight. The multicopter 6 is a rotorcraft having a plurality of rotor blades, and is a moving object to be operated in this embodiment. The multicopter 6 is also equipped with an imaging device such as a digital camera, and can display the captured image on the display device 3 via the controller 5, for example. The controller 5 is a transmitter for remotely controlling the operation of the multicopter 6 wirelessly. The display device 3 acquires imaging data from an imaging device provided in the multicopter 6 via, for example, the control device 2 and displays it. The line-of-sight detection device 4 includes, for example, a camera that captures infrared rays, and detects the line of sight based on the movement of the user's eyes. The control device 2 acquires from the line-of-sight detection device 4 information indicating the gaze point, which is the portion the user is gazing at (or fixating) on the display device 3, and controls the multicopter 6 according to the position of the gaze point. A signal for operation is output to the controller 5 .

<Functional configuration>
FIG. 2 is a functional block diagram showing an example of a remote control system. The remote control system 1 of FIG. 2 includes the control device 2, the display device 3, the line-of-sight detection device 4, the controller 5, and the multicopter 6 described above.

The control device 2 is a general computer such as a PC (Personal Computer), a smart phone, a tablet, or the like. The control device 2 also includes a communication interface (I/F) 21 , a storage device 22 , an input/output device 23 and a processor 24 , which are connected via a bus 25 . The communication I/F 21 includes a wireless module for communication using Wi-Fi (registered trademark), Bluetooth (registered trademark), and other RF (Radio Frequency) signals, a wired network card, a parallel port, and a serial port. etc., and communicates with other devices based on a predetermined protocol. The storage device 22 includes main storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory), and auxiliary storage devices (secondary storage devices) such as HDD (Hard-disk Drive), SSD (Solid State Drive), and flash memory. equipment). The main memory temporarily stores programs read by the processor 24 and secures a work area for the processor 24 . The auxiliary storage stores programs and the like executed by the processor 24 . The input/output device 23 is a user interface such as a keyboard, mouse, touch panel, or the like. Note that the input/output device 23 may also serve as the display device 3 . The processor 24 is an arithmetic processing device such as a CPU (Central Processing Unit), and performs each process described in this embodiment by executing a program. Also, in the example of FIG. 2, functional blocks are shown within the processor 24 . Specifically, the processor 24 includes an operation detection section 241 and a signal output control section 242 . The operation detection unit 241 acquires information about the movement of the user's line of sight from the line-of-sight detection device 4 via the communication I/F 21, and detects the user's operation instruction according to the position of the gaze point. The signal output control unit 242 transmits information for instructing the output of a signal for operating the multicopter 6 to the controller 5 via the input/output device 23 based on the operation instruction corresponding to the position of the gaze point. Send.

The display device 3 is a video display device that acquires and displays a video signal of an image (still image or moving image) output by a computer or the like. In FIG. 2, functional blocks are also shown within the display device 3 . Display device 3 includes communication I/F 31 and display unit 32 . The communication I/F 31 is an interface for receiving input of video signals, such as an HDMI (High-Definition Multimedia Interface, registered trademark) terminal, a DVI (Digital Visual Interface) terminal, a VGA terminal, and a wireless communication module. The display unit 32 is a liquid crystal display (LCD), a plasma display (PDP: Plasma Display Panel), an organic EL display, a cathode ray tube (CRT: Cathode Ray Tube), or the like.

The line-of-sight detection device 4 is a device that detects the movement of the user's line of sight and identifies the gaze point. In FIG. 2, functional blocks are also shown in the line-of-sight detection device 4 . The line-of-sight detection device 4 includes a communication I/F 41 , an infrared transmitting/receiving section 42 , and a line-of-sight detection section 43 . The communication I/F 41 is an interface such as a USB (Universal Serial Bus), and outputs gaze point information to the control device 2 . The infrared transmitting/receiving unit 42 irradiates the user facing the line-of-sight detection device 4 with infrared rays, and captures an image of the reflected light. The line-of-sight detection unit 43 detects the direction of the line of sight from the user's image based on the position of the pupil in the eye, and calculates the position of the gaze point on the display device.

The controller 5 is a transmitter that transmits radio signals for remotely controlling the multicopter 6 . In FIG. 2, functional blocks are also shown within the controller 5 . The controller 5 includes an operation input section 51 and a communication I/F 52 . The operation input unit 51 inputs, for example, the type of operation such as forward movement, backward movement, ascending, descending, horizontal movement to the left and right, horizontal rotation to the left and right, and the movement speed to the multicopter 6. It is an interface for receiving from Note that the horizontal movement is an operation of moving the multicopter 6 left and right with respect to the direction in which the multicopter 6 faces without changing the direction. Further, the horizontal rotation is an operation of changing the direction in which the multicopter 6 faces about the vertical rotation axis without changing the position of the multicopter 6 . That is, the multicopter 6 rotates horizontally on the spot to the left or right when viewed from above. In this embodiment, the types of these operations may be partially restricted, and the speed may be constant. Also, the communication I/F 52 includes, for example, an antenna, and transmits a radio signal for instructing a predetermined operation to the multicopter 6, and relays an image captured by the multicopter 6 to the control device 2. .

The multicopter 6 is a rotorcraft having a plurality of rotor blades. In FIG. 2, functional blocks are also shown within the multicopter 6 . The multicopter 6 includes a communication I/F 61 , a rotor 62 , a sensor 63 , an imaging device 64 and a control section 65 . The communication I/F 61 includes a communication module for receiving wireless signals from the controller 5 and a communication module for transmitting image data output by the imaging device 64 to the control device 2 and the like. The rotor 62 is a rotary blade controlled by a servomotor, and a plurality of rotors 62 are provided. The sensor 63 is a gyro sensor, a GPS (Global Positioning System) receiver, or the like, and acquires information for supporting control of the multicopter 6 . The imaging device 64 is, for example, a digital video camera, which converts visible light into image data with an imaging element and continuously outputs the image data. The control unit 65 receives a signal indicating an instruction such as a flight operation from the controller 5 via the communication I/F 61 and controls the rotation of the rotor 62 . At this time, the information output by the sensor 63 may be used to stabilize the flight.

<Remote operation processing>
FIG. 3 is a processing flow diagram showing an example of preprocessing performed by the remote control system. The system performs processing as shown in FIG. 3 when it is started. When each device is activated, for example, the line-of-sight detection device is calibrated (FIG. 3: S1). This step is calibrated to properly detect the user's line of sight. Also, the multicopter 6 and the controller 5 establish a connection by a predetermined wireless communication method (S2). Also, the line-of-sight detection device 4 and the control device 2 establish a connection by a predetermined wireless or wired communication method (S3). Also, a connection is established between the controller 5 and the control device 2 by a predetermined wireless or wired communication method (S4). It should be noted that the processing of S1 to S4 may be performed in any order, and may be performed in parallel.

FIG. 4 is a processing flow diagram showing an example of processing performed by the multicopter. The imaging device 64 of the multicopter 6 captures an image in front of the multicopter 6 (Fig. 4: S11). After that, the multicopter 6 transmits the image to the controller 5 via the communication I/F 61 (S12). Also, the multicopter 6 receives an operation signal from the controller 5 via the communication I/F 61 (S13). The control unit 65 of the multicopter 6 controls the rotation of the rotor 62 according to the operation signal, so that the multicopter 6 moves forward, ascends, descends, rotates left and right horizontally, hovers, and the like. Also, the control unit 65 of the multicopter 6 determines whether to continue the operation (S15). In this step, for example, when a command to end the operation is received from the user, when it is detected that the remaining battery level is lower than a predetermined standard, when a control signal from the controller cannot be received for a predetermined time or longer, strong wind or the aircraft When damage is detected, when an abnormal inclination of the aircraft is detected, for example, when obstacle sensors detect obstacles in all directions and it is determined that there is no course for the aircraft, it is determined to end the operation. When ending the operation, the robot may return to the spot or near the controller 5 and stop the operation after landing. If the operation is to be continued (S15: YES), return to S11 and repeat the process. On the other hand, if the operation is not to be continued (S15: NO), the process is terminated.

FIG. 5 is a processing flow diagram showing an example of processing performed by the controller. The controller 5 receives the video signal from the multicopter 6 via the communication I/F 52 (FIG. 5: S21). After that, the controller 5 transmits a video signal to the display device 3 via the communication I/F 52 (S22). That is, the controller 5 relays video signals between the multicopter 6 and the display device 3 . Further, the operation input unit 51 of the controller 5 receives data for instructing the operation of the multicopter 6 from the signal output control unit 242 of the control device 2 (S23). Then, the controller 5 transmits a signal instructing an operation to the multicopter 6 via the communication I/F 52 (S24). Also, the controller 5 determines whether to continue the operation (S26). In this step, it is determined to end the operation, for example, when an instruction to end the operation is received from the user. If the operation is to be continued (S25: YES), the process returns to S21 and repeats the process. On the other hand, if the operation is not to be continued (S25: NO), the process is terminated.

FIG. 6 is a processing flow diagram showing an example of processing performed by the display device. The display device 3 receives the video signal from the controller 5 via the communication I/F 31 (FIG. 6: S31). After that, the display unit 32 of the display device 3 displays the image (S32). Also, the display unit 32 determines whether to continue the operation (S33). For example, when an instruction to end the operation is received from the user, or when no video signal is received for a predetermined time or longer, it is determined that the operation is to be finished. If the operation is to be continued (S33: YES), the process returns to S31 and repeats the process. On the other hand, if the operation is not to be continued (S33: NO), the process is terminated.

FIG. 7 is a processing flow diagram showing an example of processing performed by the line-of-sight detection device. With the line-of-sight detection device 4 installed on the display device 3, the infrared transmitting/receiving unit 42 transmits infrared rays to the front of the display device 3 and receives the reflected light (FIG. 7: S41). After that, the line-of-sight detection unit 43 of the line-of-sight detection device 4 determines whether or not the user's line of sight has been detected from the reflected light (S42). In this step, the line-of-sight detection unit 43 of the line-of-sight detection device 4 identifies, for example, the position of the pupil in the user's eye from the image of the user, and determines that the line of sight has been detected if the position is identified. If the line of sight can be detected (S42: YES), the line of sight detection unit 43 calculates the gaze point based on, for example, the position of the pupil (S43). In this step, existing sight line detection technology can be used, and the position of the line of sight is calculated based on the position of the pupil with respect to some reference, such as infrared light reflected from the inner corner of the eye or the cornea. The gaze point is obtained as coordinates on the display device 3 or an image displayed on the display device 3, for example. The line-of-sight detection unit 43 also transmits information (point-of-regard data) indicating the position of the point-of-regard to the control device 2 via the communication I/F 41 (S44). After S44, or if the line of sight could not be detected in S42 (S42: NO), the line of sight detection device 4 determines whether to continue the operation (S45). In this step, it is determined to end the operation, for example, when an instruction to end the operation is received from the user. If the operation is to be continued (S45: YES), the process returns to S41 to repeat the process. On the other hand, if the operation is not to be continued (S45: NO), the process is terminated.

FIG. 8 is a processing flow diagram showing an example of processing performed by the control device. The operation detection unit 241 of the control device 2 acquires information indicating the position of the gaze point of the user on the display unit 32 of the display device 3 or on the image displayed on the display unit 32 from the line-of-sight detection device 4 (FIG. 8). : S51). Further, the operation detection unit 241 determines whether the acquired information indicating the position of the gaze point is within a predetermined range on the display unit 32 of the display device 3 (S52). In this embodiment, the multicopter 6 is instructed to fly according to the position of the gaze point on the display section 32 . That is, an operation area for operating the multicopter 6 by the user's gaze is provided on the display unit 32 (in other words, on the first-person viewpoint image of the multicopter 6).

FIG. 9 is a diagram showing an example of allocation of operation areas set on the display device. In the example of FIG. 9, on the display unit 32 of the display device 3, there are commands for instructing the multicopter 6 to go straight, ascend, descend, horizontally rotate to the left, and horizontally rotate to the right. Operation area is assigned. A circular area 321 arranged substantially in the center of the display unit 32 is a standby area (also called a neutral area) for keeping the multicopter 6 stationary (also called "hovering" or "standby") on the spot. A circular area 322 located inside the standby area 321 and at a predetermined distance from the center of the standby area 321 is an advance area 322 for instructing the multicopter 6 to advance. In the example of FIG. 9 , the advance area 322 is provided below the center of the standby area 321 . An area 323 located outside the standby area 321 and on the left side of the standby area 321 is a left rotation area for instructing the multicopter 6 to horizontally rotate leftward. In the example of FIG. 9, a left rotation area 323 is provided on the left side of the tangent line that is in contact with the left side of the standby area 321 that is a perfect circle. An area 324 located outside the standby area 321 and on the right side of the standby area 321 is a right rotation area for instructing the multicopter 6 to horizontally rotate to the right. In the example of FIG. 9, a right rotation area 324 is provided on the right side of a tangent line that is in contact with the right side of the standby area 321 as a boundary. An area 325 located outside the standby area 321 and above the standby area 321 is an ascending area for instructing the multicopter 6 to ascend. In the example of FIG. 9, a raised area 325 is provided so as to be surrounded by the standby area 321, a tangent line in contact with the left side of the standby area 321, a tangent line in contact with the right side of the standby area 321, and an upper edge of the display section 32. It is An area 326 located outside the standby area 321 and below the standby area 321 is a descending area for instructing the multicopter 6 to descend. In the example of FIG. 9, a descending area 326 is provided so as to be surrounded by the standby area 321, a tangent line in contact with the left side of the standby area 321, a tangent line in contact with the right side of the standby area 321, and the lower edge of the display section 32. It is The boundary of the operation area may be superimposed on the image displayed on the display device 3 by the control device 2, for example, or the boundary of the operation area may be displayed on a transparent sheet, panel, or the like. It may be pasted over the display section 32 of the display device 3 .

Therefore, when the user's gaze point acquired in S51 of FIG. 8 is included in the forward movement area 322, the left rotation area 323, the right rotation area 324, the rising area 325, the falling area 326, or the standby area 321, in S52, The operation detection unit 241 determines that the gaze point of the user exists within a predetermined range. If the gaze point is outside the display unit 32, it is determined in S52 that the gaze point is not within the predetermined range. At this time, if the multicopter 6 is flying, for example, the multicopter 6 is controlled to hover on the spot, return to the vicinity of the controller 5 and land and wait, or land and wait on the spot. You may make it the part 65 control autonomously.

When it is determined that the point of gaze of the user exists within the predetermined range (S52: YES), the signal output control section 242 of the control device 2 instructs the controller 5 to transmit a control signal according to the position of the point of gaze. An operation signal for instructing is generated (S53). Further, the signal output control section 242 outputs the generated operation signal to the controller 5 (S54). That is, when the user's gaze point exists within the forward movement area 322, the signal output control section 242 causes the controller 5 to transmit a signal for moving the multicopter 6 forward via the communication I/F 21. FIG. Further, when the user's gaze point exists within the left rotation area 323, the signal output control unit 242 transmits a signal for horizontally rotating the multicopter 6 to the left to the controller 5 via the communication I/F 21. Let Further, when the user's gaze point is within the right rotation area 324, the signal output control unit 242 transmits a signal for horizontally rotating the multicopter 6 to the right to the controller 5 via the communication I/F 21. Let Also, when the user's gaze point is within the rising region 325, the signal output control unit 242 causes the controller 5 to transmit a signal for moving the multicopter 6 vertically upward via the communication I/F 21. Also, when the user's gaze point exists within the descending region 326, the signal output control unit 242 causes the controller 5 to transmit a signal for moving the multicopter 6 vertically downward via the communication I/F 21.

After S54, or when it is determined in S52 that the point of gaze of the user does not exist within the predetermined range (S52: NO), the control device 2 determines whether to continue the operation (S55). In this step, for example, when an instruction to end the operation is received from the user, it is determined to end the operation. An area for instructing the end of the action may be further provided in the operation area of the display unit 32, and it may be determined that the action is finished when the user looks at the area for a predetermined time or longer. It is also assumed that the multicopter 6 is to land before the control device 2 finishes its operation. If the operation is to be continued (S55: YES), the process returns to S51 and repeats the process. On the other hand, if the operation is not to be continued (S55: NO), the process is terminated.

According to the above-described remote control processing, the user can set the line of sight near the center of the first-person viewpoint image of the multicopter 6 displayed on the display unit 32 (that is, near the center of the standby area 321), for example. , the multicopter 6 is made stationary, and the entire first-person viewpoint image can be recognized. In other words, the user can see the entire display section 32 in his or her field of vision while gazing at the vicinity of the center of the display section 32 . Further, when the object of interest of the user exists at the edge or outside of the first-person viewpoint image, the user moves the line of sight toward the direction of the object of interest, and the multicopter 6 moves in the direction of the object of interest. It can be remotely operated to change its orientation and position.

10 to 12 are diagrams showing examples of first-person viewpoint images of a multicopter for explaining the operation of the multicopter. For example, as shown in FIG. 10, when the target 7 to which the user wants the multicopter 6 to approach is present in the upper right of the first-person viewpoint image displayed on the display unit 32, the user is positioned at the upper right of the display unit 32. Focus on Goal 7. At this time, the operation detection unit 241 detects the gaze point within the right rotation area 324 . After that, the signal output control section 242 outputs a signal instructing horizontal rotation to the right to the multicopter 6 via the controller 5 . When the multicopter 6 horizontally rotates to the right, the target 7 described above moves to the left in the first-person viewpoint image displayed on the display unit 32, as shown in FIG. 11, for example. Then, when the target 7 moves into the rising region 325, the user's gaze point also moves into the rising region 325 accordingly. Then, the operation detection unit 241 detects the gaze point within the rising area 325 . After that, the signal output control section 242 outputs a signal instructing the multicopter 6 to ascend via the controller 5 . When the multicopter 6 moves vertically upward, the target 7 described above moves downward in the first-person viewpoint image displayed on the display unit 32, as shown in FIG. 12, for example. When the target 7 moves to the waiting area 321, the user's gaze point also moves into the waiting area 321 accordingly. Then, the operation detection unit 241 detects the gaze point of the user within the standby area 321 . In this case, the signal output control unit 242 does not instruct the multicopter 6 to operate, and causes it to hover on the spot. Also, when the user wants to bring the multicopter 6 closer to the target 7, the user gazes at the forward movement area 322 to cause the operation detection section 241 to detect the point of gaze, and the signal output control section 242 instructs the controller 5 to advance. can output a signal for

As described above, according to the arrangement of the operation areas and the association of operation instructions to the multicopter 6 with each area according to the present embodiment, the user can remotely operate the multicopter 6 intuitively. In addition, by limiting the horizontal movement to only forward movement, it is possible to limit backward movement and left-right parallel movement, which are difficult for the user to perceive danger from the first-person image of the multicopter 6, and to suppress the occurrence of accidents.

<Operation area>
13 and 14 are diagrams for explaining the position and size of the operation area set on the display device. In this embodiment, the operation area is set based on the human visual field and eye movement. FIG. 13 shows the relationship between the user's viewpoint (eyeball) and the gaze point on the display unit 32 . In general, in order for a human to recognize the shape of an object with high resolution, it is necessary to use central vision such that a retinal image is projected at the center of the retina with a visual angle of about 1 to 2 degrees. In this embodiment, the size of the operation area (for example, the forward movement area 322) is at least about the size of the range that can be captured by central vision.

In addition, when a human keeps (fixes) an object with central vision, an eye movement called a microsaccade occurs. Microsaccades are involuntary movements of the order of magnitude of 1-2 degrees of gaze. Although there are variations, the average speed is about 10 degrees/second and the frequency is about 1 to 3 Hz. If the line-of-sight detection device 4 can detect changes in the line-of-sight due to microsaccades, the user may unknowingly make an erroneous operation due to the influence of the microsaccades, depending on the layout or size of the operation area. Therefore, in the present embodiment, the operation area is set with an arrangement and size that does not deviate from the area where the user is gazing even with a microsaccade.

The radius r of the circular advance region 322 shown in FIG. is the angle .theta. That is, r is the radius of the centrally visible range. θ is also the angle at which the line of sight moves due to the microsaccade.
r=l tan θ (1)
For example, if l is 550 mm and θ is 2 degrees, r is 19.2 mm.

FIG. 14 is a diagram showing the relationship between the operation area shown in FIG. 9 and the radius r described above. The advance area 322 is arranged apart from the center of the standby area 321 by a predetermined distance or more. For example, when the user fixes the point of gaze at the center of the standby area 321, the forward area 322 is positioned at the center of the standby area 321 so that the detected point of gaze does not enter the forward area 322 due to the influence of microsaccades. On the other hand, it is arranged so as not to enter the circle of radius r. In other words, the forward movement area 322 is provided outside the centrally visible range centering on the center of the standby area 321 . Therefore, the distance between the center of the standby area 321 and the center of the forward area 322 is set to 2r or more. In other words, the distance between the center of the standby area 321 and the advance area 322 is r or more. In the example of FIG. 14, the distance between the center of the standby area 321 and the center of the forward area 322 is 3r. In addition, the distance between the forward region 322 and the descending region 326 is also separated by a length r or more with the standby region 321 interposed therebetween so that the movement of the gaze point beyond the boundary is not detected due to the influence of microsaccades, for example. .

With the arrangement of the operation regions as described above, unconscious erroneous operations (in other words, erroneous detection by the line-of-sight detection device 4) due to the influence of microsaccades can be suppressed.

<Example>
Setting the distance l between the user and the display unit 32 of the display device 3 to 550 mm, the radius r of the advance region 322 to 19.2 mm, and the radius 5r of the standby region 321 to 96.0 mm, the operation regions shown in the above embodiments are set. did. The subjects were 12 adults in their twenties who were using the system for the first time. As a result of having the subject remotely operate the multicopter using this system, one of the 12 people crashed the multicopter in the first flight, but in the second flight, the predetermined section was completed without crashing. was able to fly. In addition, other 11 people were able to fly the specified section without crashing the multicopter.

<Modified example of operation area>
FIG. 15 is a diagram showing a modification of the layout of the operation regions. In this modification, the shapes of the standby area 321 and the advance area 322 are ellipses. This shape approximates the human visual field in binocular vision, and the long axis is in the horizontal direction of the display unit 32 of the display device 3 and the short axis is in the vertical direction. Also, the ratio between the single axis and the long axis can be determined as appropriate, for example, by adjusting to the field of view.

Also in this modification, it is desirable to set the position, size, and shape of the operation area so that the movement of the gaze point beyond the boundary of the operation area is not detected due to the influence of microsaccades. For example, the length r of the minor axis of the advancing region 322 is a value obtained using θ, which is the radius of the central vision range or the amplitude of the line of sight due to the microsaccade, and the distance l from the eyeball to the point of gaze. good too. However, when the size is determined in this way, depending on the size of the display unit 32, it may not be possible to arrange the operation areas in a well-balanced manner. Therefore, the distance between the center of the waiting area 321 and the center of the advancing area 322 may be set to 2r instead of 3r, and in particular, the distance from the lower end of the advancing area 322 to the lower end of the waiting area may be set to r or less. Also, the standby area 321 and the advance area 322 defined in this way may be reduced as a whole.

An operation area according to such a modified example can also suppress unconscious erroneous operations due to the influence of microsaccades.

<Others>
A moving object to be remotely controlled by this system is not limited to a multicopter. For example, drones (unmanned aerial vehicles), helicopters, airships, multi-legged robots, or robots that run on wheels or crawlers may be controlled. If the object to be controlled is a device that moves on the ground, it is not necessary to provide the ascending area and descending area, or instead of the ascending area and descending area, an area for instructing other operations may be assigned. good. In addition, the device to be controlled is not limited to one that is wirelessly controlled, and may be one that receives power through a wire. Also, the device to be controlled may further include an end effector such as a robot hand, a microphone, a speaker, and other devices. These devices may also be operated by sight, or may be operated manually by the user using a controller. In addition to controlling the movement of objects such as devices, it may be used to operate remote computers, home electric appliances, factory machines, medical equipment, and the like. Since the system described above can be operated remotely, it is suitable for operating a device that can be controlled wirelessly, and is particularly suitable for operating an unmanned flying object capable of ascending and descending.

Further, the position, size, shape, and number of divisions of the operation area are not limited to the examples shown in FIG. 9 and the like. With a tangent line in contact with the upper side of the standby area 321 as a boundary, a rising area 325 is provided above the tangent line, and with a tangent line in contact with the lower side of the standby area 321 as a boundary, a descending area 326 is provided below the tangent line. good too. In this case, for example, the left rotation area 323 is surrounded by the standby area 321 , a tangent line in contact with the upper side of the standby area 321 , a tangent line in contact with the lower side of the standby area 321 , and the left edge of the display unit 32 . is provided, and a right rotation area 324 is provided so as to be surrounded by the standby area 321 , a tangent line in contact with the upper side of the standby area 321 , a tangent line in contact with the lower side of the standby area 321 , and the right edge of the display unit 32 . The boundary between the counterclockwise rotation region 323 and the ascending region 325, the boundary between the ascending region 325 and the clockwise rotation region 324, the boundary between the clockwise rotation region 324 and the descending region 326, and the boundary between the descending region 326 and the counterclockwise rotation region 323 are , respectively, are not limited to the tangent lines of the waiting area 321 . For example, these boundaries may be defined by lines drawn radially outward from the center of the display section 32 .

The shape of the standby area 321 and the advance area 322 is not limited to a perfect circle or an ellipse, and may be polygonal or the like. Also, the number of divisions of the operation area may be increased so that take-off and landing, change of motion speed, and other motions can be controlled by the movement of the line of sight. Further, in the operation areas shown in the embodiment or modification, for example, leftward horizontal rotation, rightward horizontal rotation, upward movement, and downward movement are performed as the distance from the center of the display unit 32 increases in each operation area. Alternatively, the operation speed may be increased as it approaches the left, right, top, and bottom edges of the display unit 32 .

Also, the distance l between the display device 3 and the user is not limited to the example described above. Alternatively, the distance between the display device 3 and the user may be measured by an existing distance measuring sensor or the like, and the size of the operation area may be changed according to the change in distance.

An area 324 located outside the standby area 321 and on the right side of the standby area 321 is a right rotation area for instructing the multicopter 6 to horizontally rotate to the right. In the example of FIG. 9, a right rotation area 324 is provided on the right side of a tangent line that is in contact with the right side of the standby area 321 as a boundary. An area 325 located outside the standby area 321 and above the standby area 321 is an ascending area for instructing the multicopter 6 to ascend. In the example of FIG. 9, a raised area 325 is provided so as to be surrounded by the standby area 321, a tangent line in contact with the left side of the standby area 321, a tangent line in contact with the right side of the standby area 321, and an upper edge of the display section 32. It is An area 326 located outside the standby area 321 and below the standby area 321 is a descending area for instructing the multicopter 6 to descend. In the example of FIG. 9, a descending area 326 is provided so as to be surrounded by the standby area 321, a tangent line in contact with the left side of the standby area 321, a tangent line in contact with the right side of the standby area 321, and the lower edge of the display section 32. It is The boundary of the operation area may be superimposed on the image displayed on the display device 3 by the control device 2, for example, or the boundary of the operation area may be displayed on a transparent sheet, panel, or the like. It may be pasted over the display section 32 of the display device 3 .

The device configuration shown in FIG. 2 is an example, and a plurality of devices may share the same processing units. For example, instead of the controller 5 , the control device 2 may relay the image of the imaging device 64 . Also, an FPV (First Person View) system that operates in cooperation with the imaging device 64 of the multicopter 6 may be provided separately from the control device 2 . All or part of the control device 2, the display device 3, the line-of-sight detection device 4, and the controller 5 may be integrated.

Moreover, the line-of-sight detection device 4 is not limited to one that detects the position of the user's pupil using infrared rays. For example, the direction of the line of sight may be specified by detecting the position of the iris using an image obtained by capturing visible light, or a user contact type method using a predetermined potential difference may be used. good.

The configuration described above is an example, and the present invention is not limited to the illustrated configuration. For example, the matters described above can be appropriately combined without departing from the subject and technical idea of the present invention. Further, the steps of the processing described above may be implemented as a method executed by a computer using software, a method executed by a device using hardware such as a PLD (Programmable Logic Device), or a program executed by a computer. The program may be recorded on a computer-readable recording medium and provided. A computer-readable recording medium is a recording medium that can store information by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Examples of such recording media that can be removed from the computer include optical discs, magneto-optical discs, flexible discs, magnetic tapes, memory cards, and the like. In addition, HDDs (Hard Disk Drives) and SSDs (Solid Drives) are used as recording media fixed to computers.
State Drive), ROM (Read Only Memory), and the like.

1: remote control system 2: control device 21: communication I/F
22: storage device 23: input/output device 24: processor 241: operation detection unit 242: signal output control unit 25: bus 3: display device 31: communication I/F
32: Display unit 321: Standby area 322: Advance area 323: Left rotation area 324: Right rotation area 325: Rising area 326: Falling area 4: Line of sight detection device 41: Communication I/F
42: Infrared transmission/reception unit 43: Line of sight detection unit 5: Controller 51: Operation input unit 52: Communication I/F
6: Multicopter 61: Communication I/F
62: rotor 63: sensor 64: imaging device 65: control unit

Claims (11)

A control device for remotely controlling a moving object to be controlled while a user views an image captured by an imaging device provided in the moving object to be controlled,
from a line-of-sight detection device that detects a line of sight of the user facing a display device that displays an image picked up by the imaging device included in the moving object to be controlled, and that outputs the gaze point of the user on the image; an operation detection unit that acquires gaze point information and determines whether the gaze point indicated by the acquired information is included in a predetermined area;
a signal output control unit for transmitting a signal for controlling the moving object to be controlled, which is associated with the predetermined area when the point of gaze indicated by the information is included in the predetermined area;
with
The predetermined area has a standby area for waiting the moving object to be controlled in approximately the center of the image, and is inside the standby area and separated from the center of the standby area by a predetermined distance. A remote control device for a moving object, wherein the position includes an advance area for advancing the moving object to be controlled. 2. The remote control device for a moving object according to claim 1, wherein the boundary of said predetermined area is displayed as an image on said display device, or displayed on a sheet or panel superimposed on said display device. The forward region is defined by a distance l from the eyeball to the point of gaze on the image, and a straight line connecting the eyeball and the point of gaze, and a line connecting the eyeball and the end of the central vision range on the image displayed on the display device. The radius r of the centrally visible range obtained when the angle with the straight line is θ is obtained by the following formula (1),
r=l tan θ (1)
3. The remote control device for a moving object according to claim 1, wherein the remote control device for a moving object is arranged in a range of length r in the vertical direction centered on a position 3r vertically downward from the center of the standby area. 4. The remote control device for a moving object according to claim 3, wherein the forward movement area is arranged at a position r in the upward direction from the lowermost end in the downward direction from the center of the standby area. 5. The remote control device for a moving object according to claim 3, wherein said θ is 1 to 2 degrees. 6. The remote control of the moving object according to any one of claims 1 to 5, wherein areas for horizontally rotating the moving object to be controlled to the left and right around a vertical rotation axis are provided on the left and right sides of the standby area. Control device. the moving object to be controlled is an unmanned flying object;
The remote control device for a moving object according to any one of claims 1 to 6, further comprising areas for raising and lowering the unmanned flying object above and below the standby area. The remote control device for a moving object according to claim 7, wherein the unmanned flying object is any one of a multicopter, a drone, a helicopter, and an airship. A remote control device for a moving object according to any one of claims 1 to 7;
the line-of-sight detection device;
the moving object to be controlled;
a controller that transmits a signal for controlling the moving object to be controlled;
A remote control system with A control method for remotely controlling a moving object to be controlled while a user views an image captured by an imaging device included in the moving object to be controlled,
from a line-of-sight detection device that detects a line of sight of the user facing a display device that displays an image picked up by the imaging device included in the moving object to be controlled, and that outputs the gaze point of the user on the image; a step of acquiring information about the gaze point and determining whether the gaze point indicated by the acquired information is included in a predetermined area;
if the point of gaze indicated by the information is included in the predetermined area, transmitting a signal for controlling the moving object to be controlled, which is associated with the area;
is executed by the remote controller and
The predetermined area has a standby area for waiting the moving object to be controlled in approximately the center of the image, and is inside the standby area and separated from the center of the standby area by a predetermined distance. A moving object remote control method, wherein the position includes an advancing area for advancing the moving object to be controlled. A control program for remotely controlling a moving object to be controlled while a user views an image captured by an imaging device included in the moving object to be controlled,
from a line-of-sight detection device that detects a line of sight of the user facing a display device that displays an image picked up by the imaging device included in the moving object to be controlled, and that outputs the gaze point of the user on the image; a step of acquiring information about the gaze point and determining whether the gaze point indicated by the acquired information is included in a predetermined area;
if the point of gaze indicated by the information is included in the predetermined area, transmitting a signal for controlling the moving object to be controlled, which is associated with the area;
is executed by the remote control device,
The predetermined area has a standby area for waiting the moving object to be controlled in approximately the center of the image, and is inside the standby area and separated from the center of the standby area by a predetermined distance. A remote control program for a moving object, wherein the position includes an advance area for advancing the moving object to be controlled.

Images