JP2023125011A - Remote control system and remote control program - Google Patents

Abstract

To provide a remote control system for realizing remote control without requiring position information of a moving object.SOLUTION: A remote control system comprises: a moving object 300; and an operation terminal 100 for remotely controlling the moving object 300 by wireless communication. The moving object 300 has an imaging unit that captures an image around the moving object 300 in an omnidirectional area and the operation terminal 100 has a control unit 102 which causes a display unit to display a first image corresponding to a viewing direction input from an input unit in at least a part of the omnidirectional area imaged by the imaging unit and causes the display unit to display a direction of the first image in the omnidirectional area, a first mark for indicating a traveling direction of the moving object 300, and a second mark for indicating a movement target direction of the moving object 300 while superimposing on the first image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a remote control system and a remote control program.

In a remote control system, a user (remote operator) remotely controls an unmanned mobile object. In a remote control system, for example, a user can view images of a remote location by remotely controlling an unmanned moving body equipped with a camera. In particular, in a remote control system in which the camera can be freely moved to capture images, the user can view the situation at a remote location as if he or she were actually there. Therefore, such a remote control system is used for telepresence, for example.

In a remote control system, a user operates a moving object relying only on images captured by a camera mounted on the moving object. For this reason, in a remote control system, it is difficult for the user to move the moving object to the location intended by the user, unlike when the user is actually at the site and operates the moving object while directly observing the movement of the moving object.

For example, when a user attempts to make a moving object turn to the left while looking at an image captured by a camera, the remote control system will not be able to determine how far to turn the object because there is a slight delay in the image. Sometimes it's hard to understand. Furthermore, when the user performs an operation to cause the moving object to move straight, in the remote control system, if the orientation of the image and the orientation of the moving object do not match, the moving object may move diagonally. In this way, in a remote control system, the user is unable to monitor the movement of a remote moving object in real time due to the relationship between the orientation of the image the user is viewing and the orientation of the moving object, or due to image delay or operation delay. I can't feel it.

Conventionally, various technological developments have been made to facilitate remote control. For example, in the technique disclosed in Patent Document 1, the moving position of the moving object is specified on the screen and the moving object is moved. This technology determines the moving position by converting from the image coordinate system to the world coordinate system in order to match the position information on the screen with the position information of the moving object.

Furthermore, the technique disclosed in Patent Document 2 displays a robot to be moved on an overhead display, and uses a point-and-click device to specify a moving position and move the moving object.

Japanese Patent Application Publication No. 2011-170844 Special Publication No. 2003-532218

The technique disclosed in Patent Document 1 requires acquiring position information of a moving body in a world coordinate system. Therefore, this technique has a problem in that unless the acquired position accuracy is high, the moving object cannot be accurately moved to the intended position.

The technique disclosed in Patent Document 2 requires that the coordinates on the screen be associated with the actual coordinates where the robot exists. Therefore, in this technique as well, it is necessary to acquire the position information of the robot. Therefore, similar to Patent Document 1, this technique has a problem in that unless the acquired positional accuracy is high, the robot cannot be accurately moved to the intended position.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a remote control system and a remote control program for realizing remote control of a moving body without requiring position information of the moving body.

The above object of the present invention is achieved by the following means.

(1) A moving object,
an operation terminal for remotely controlling the mobile object via wireless communication;
A remote control system having
The mobile body has an imaging unit that captures images around the mobile body in all directions,
The operation terminal is
an input section;
A display section;
Displaying on the display unit a first image that is at least a part of the omnidirectional area imaged by the imaging unit and that corresponds to the viewing direction input from the input unit;
A first mark indicating the direction of the first image in the omnidirectional area, a traveling direction of the moving body, and a second mark indicating a movement target direction of the moving body are superimposed on the first image and displayed on the display unit. A control unit to display,
A remote control system with

(2) The operation terminal has a sensor that detects acceleration and angular velocity,
The remote control system according to (1) above, wherein the control unit controls movement of the mobile body based on the output from the sensor.

(3) The control unit controls the movement of the moving body based on the input for moving the moving body from the input unit and the acceleration and angular velocity detected by the sensor, as described in (2) above. The remote control system described in .

(4) The control unit moves the line of sight based on a first input input from the input unit and an acceleration and an angular velocity detected by the sensor after the first input, (3) above. The remote control system described in .

(5) The remote control unit according to (3) or (4) above, wherein the control unit moves the second mark to the center of the screen of the display unit based on a second input input from the input unit. system.

(6) The remote control unit according to any one of (3) to (5) above, wherein the control unit controls the moving speed of the moving body based on a third input input from the input unit. system.

(7) The remote control unit according to any one of (3) to (6) above, wherein the control unit reverses the moving direction of the moving body based on a fourth input input from the input unit. system.

(8) The controller according to any one of (3) to (7) above zooms in or zooms out the first image based on a fifth input input from the input unit. operation system.

(9) The remote control system according to any one of (1) to (8) above, wherein the first image is cut out on the movable body side and transmitted from the movable body to the operating terminal.

(10) In addition to the first image, the control unit displays, on the display unit, a second image including an area in the movement target direction of the moving object and an area different from the first image. The remote control system according to any one of (9).

(11) The remote control system according to (10), wherein the control unit detects a person and/or an object in the area shown in the second image.

(12) The remote control system according to (10) or (11), wherein the second image is cut out on the mobile body side and transmitted to the operating terminal.

(13) A remote control program for remotely controlling a mobile body having an imaging unit that captures images in all directions from an operation terminal connected to the mobile body by wireless communication,
displaying a first image of at least a part of the omnidirectional area imaged by the imaging unit and corresponding to the input line-of-sight direction;
Displaying a direction of the first image in the omnidirectional area, a first mark indicating the traveling direction of the moving body, and a second mark indicating the target direction of movement of the moving body, superimposed on the first image ( b) and
A remote control program that allows a computer to execute

(14) The operation terminal has a sensor that detects acceleration and angular velocity,
The remote control program according to (13) above, further comprising the step (c) of controlling movement of the mobile body based on the output from the sensor.

(15) The step (c) according to the above (14), wherein the movement of the moving body is controlled based on the input for moving the moving body and the acceleration and angular velocity detected by the sensor. Remote control program.

(16) The remote control according to (15), wherein the step (c) moves the line of sight based on a first input and an acceleration and an angular velocity detected by the sensor after the first input. program.

(17) The step (c) is one of any one of (14) to (16) above, wherein the second mark is moved to the center of the screen where the first image is displayed, based on a second input. The remote control program described in .

(18) The remote control program according to any one of (14) to (17), wherein the step (c) controls the moving speed of the moving object based on the third input.

(19) The remote control program according to any one of (14) to (18), wherein the step (c) reverses the moving direction of the moving body based on the fourth input.

(20) The remote control program according to any one of (14) to (19), wherein the step (c) zooms in or zooms out the first image based on a fifth input.

(21) The remote control program according to any one of (13) to (20) above, comprising the step (d) of receiving the first image cut out by the moving body from the moving body.

(22) The step (e) of displaying, in addition to the first image, a second image including an area in the movement target direction of the moving body and an area different from the first image, (13) to ( 19) The remote control program according to any one of 19).

(23) The remote control program according to (22) above, further comprising the step (f) of detecting a person and/or an object in the area shown in the second image.

(24) The remote control program according to (22) or (23) above, comprising the step (g) of receiving the second image cut out by the moving body from the moving body.

The mobile body of the remote control system has an imaging unit that captures images in all directions. This allows the remote operator to freely view and operate the moving object in any desired direction when moving the moving object. Therefore, the remote operator can operate the mobile object while looking in the desired direction. Therefore, remote control of a moving object can be realized without requiring position information of the moving object.

FIG. 1 is a schematic diagram for explaining a remote control system according to an embodiment. FIG. 2 is a block diagram for explaining the configuration of an operating terminal. FIG. 3 is a schematic diagram for explaining the rotation of the operating terminal itself when the operating terminal is used in a horizontal position. FIG. 2 is a schematic diagram for explaining the rotation of the operating terminal itself when the operating terminal is used in a vertical position. FIG. 3 is an explanatory diagram for explaining example 1 of a display image displayed on an operation display unit. FIG. 7 is an explanatory diagram for explaining example 2 of a display image displayed on the operation display section. FIG. 7 is an explanatory diagram for explaining display image example 3 displayed on the operation display unit. FIG. 2 is an explanatory diagram for explaining the vertical position of a reticle within a screen. FIG. 2 is an explanatory diagram for explaining the vertical position of a reticle within a screen. FIG. 2 is an explanatory diagram for explaining example 1 of a captured image captured by an omnidirectional camera. FIG. 7 is an explanatory diagram for explaining example 2 of a captured image captured by an omnidirectional camera. FIG. 7 is an explanatory diagram for explaining example 3 of a captured image captured by an omnidirectional camera. FIG. 6 is an explanatory diagram for explaining an operation for changing the viewing direction L; FIG. 6 is an explanatory diagram for explaining an operation for changing the reticle direction R; FIG. 3 is an explanatory diagram for explaining operations for accelerating and decelerating a moving body. FIG. 7 is an explanatory diagram for explaining other acceleration/deceleration operations of a moving body. FIG. 3 is an explanatory diagram for explaining an operation for reversing the moving direction of a moving body. FIG. 3 is an explanatory diagram for explaining an omnidirectional image when the moving direction of a moving object is reversed. FIG. 3 is an explanatory diagram for explaining a zoom operation. FIG. 7 is an explanatory diagram for explaining another zoom operation. FIG. 3 is an explanatory diagram for explaining collision detection. It is a vector diagram regarding movement control. FIG. 3 is a vector diagram for explaining a case where a moving body is tilted. FIG. 3 is a vector diagram for explaining a case where the direction of a moving body changes. FIG. 3 is a vector diagram for explaining a case where the direction of a moving body changes. It is a flow chart showing a rotation operation procedure. 7 is a flowchart showing a movement control procedure in the line-of-sight direction L. FIG. 7 is a flowchart showing a movement control procedure in the reticle direction R. FIG.

Embodiments of the present invention will be described in detail below with reference to the drawings. In addition, in the description of the drawings, the same elements are given the same reference numerals, and redundant description will be omitted. Furthermore, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

FIG. 1 is a schematic diagram for explaining a remote control system according to an embodiment.

The remote control system 1 according to the first embodiment includes an operating terminal 100 and a mobile object 300, as shown in FIG. The operating terminal 100 and the mobile object 300 are connected through a network 200. The operation terminal 100 is connected to the mobile body 300 by wireless communication via the network 200.

The details of each part will be explained below.

FIG. 2 is a block diagram for explaining the configuration of the operating terminal 100.

As shown in FIG. 2, the operation terminal 100 includes a communication section 101, a control section 102, a storage section 103, a sensor 104, and an operation display section 105.

Communication unit 101 wirelessly communicates with mobile object 300 via network 200 . The communication unit 101 has an interface based on a wireless communication standard such as 5G or IEEE802.11 (for example, WiFi) corresponding to a wireless base station or an access point (both not shown) of the network 200. The communication unit 101 may have a plurality of interfaces corresponding to a plurality of wireless communication standards. The communication unit 101 also includes antennas (not shown) corresponding to wireless communication stations and access points to which it may connect. Note that the wireless communication standard used is not limited to 5G or IEEE802.11 as long as communication with the communication unit 101 of the operating terminal 100 is possible.

The communication unit 101 receives image data from the moving body 300, the current moving direction and the current moving speed of the moving body 300, and outputs the received information to the control unit 102. Furthermore, the communication unit 101 transmits a control signal for controlling the mobile body 300 output from the control unit 102 to the mobile body 300. The control signal includes information for moving the mobile body 300.

Control unit 102 is a computer. The control unit 102 controls the movement of the mobile object 300 by executing a program for remote control, which will be described later.

The control unit 102 includes, like a well-known computer, a CPU (Central Processing Unit) serving as an arithmetic element, a ROM (Read Only Memory) serving as the storage unit 103, a RAM (Random Access Memory), and the like.

Furthermore, the control unit 102 includes a storage unit 103 in addition to the ROM and RAM. The storage unit 103 is, for example, a nonvolatile storage device such as an embedded MultiMediaCard (eMMC), a solid state drive (SSD), or a hard disk drive (HDD). Furthermore, the storage unit 103 may be a portable storage medium such as a memory card.

The sensor 104 is, for example, an acceleration sensor and an angular velocity sensor (gyro sensor). The sensor 104 outputs the detected acceleration and angular velocity values to the control unit 102. The control unit 102 obtains the rotation direction and rotation amount (rotation angle) of the operating terminal 100 itself from the detected acceleration and angular velocity values.

FIG. 3 is a schematic diagram for explaining the rotation of the operating terminal 100 itself when the operating terminal 100 is used in a horizontal position. FIG. 4 is a schematic diagram for explaining the rotation of the operating terminal 100 itself when the operating terminal 100 is used in a vertical position. Here, a case where a smartphone is used as the operation terminal 100 will be described as an example.

When the smartphone (operation terminal 100) is used in the horizontal position, the rotation of the operation terminal 100 itself is rotation about the rotation axis AX passing through the center of the long side of the smartphone, as shown in FIG. On the other hand, when the smartphone (operation terminal 100) is used in a vertical position, the rotation of the operation terminal 100 itself is rotation about the rotation axis AX passing through the center of the short side of the smartphone, as shown in FIG. The rotation axes AX are all perpendicular lines (lines perpendicular to the horizontal line).

The operation display section 105 is an input section and a display section. The operation display unit 105 is, for example, a touch panel display, and displays various information and receives various inputs from the user. An image captured by an omnidirectional camera 320 mounted on the moving object 300 is displayed on the operation display section 105. The displayed image is an image of a part of the omnidirectional image. Further, the operation display unit 105 receives input of an instruction for controlling the mobile body 300 by a remote operator (user) touching the operation display unit 105 .

The operation terminal 100 having the functions of each part as described above is, for example, a smartphone, a tablet computer, or the like. The operation terminal 100 may be another personal computer (PC), a dedicated controller, or the like.

The network 200 is a network 200 using a 5G communication line, a wireless local area network (LAN), or the like. The network 200 includes a wireless base station and an access point to which the operation terminal 100 and the mobile object 300 are connected via wireless communication. These networks 200 may further be connected to the Internet.

Note that in this embodiment, the operation terminal 100 and the mobile object 300 are connected by wireless communication via the network 200, but the operation terminal 100 and the mobile object 300 are not directly connected by wireless communication. Good too.

The mobile body 300 includes a mobile body main body 310, an omnidirectional camera 320, and a mobile body control terminal 330. The mobile object 300 is, for example, a robot car that travels on the ground by remote control (also referred to as radio control or the like). Furthermore, the mobile object 300 may be, for example, an aerial drone, an underwater or waterborne drone, or the like. In this embodiment, a robot car will be described as an example of the moving object 300.

In this embodiment, the mobile body 310 has a drive mechanism including a motor, wheels, etc., a steering mechanism, and the like. The mobile body 310 is also equipped with an acceleration sensor and a gyro sensor (both not shown). The acceleration sensor and the gyro sensor detect the moving direction and moving speed of the moving body 300. Note that the acceleration sensor and the gyro sensor may be included in the mobile control terminal 330 and the omnidirectional camera 320.

The omnidirectional camera 320 is an imaging unit. The omnidirectional camera 320 is also called a 360 degree camera. The omnidirectional camera 320 captures images globally. The image captured by the omnidirectional camera 320 is a moving image. Note that in a robot car, other moving objects 300 that travel on the ground, water drones, and the like, an omnidirectional camera 320 that images only a hemispherical portion above the moving object main body 310 may be used.

Furthermore, the mobile object control terminal 330 receives a control signal from the operation terminal 100 and controls the mobile object 300 according to instructions included in the control signal. Mobile object control terminal 330 is a computer for controlling a mobile object. The control signal includes information such as the line-of-sight direction input to the operation terminal 100, the direction for moving the mobile object 300, and the instructed speed. Mobile control terminal 330 controls omnidirectional camera 320 based on the control signal. The mobile body control terminal 330 controls the drive mechanism and steering mechanism of the mobile body main body 310 based on the control signal, and moves the mobile body 300 according to the direction and speed input from the operating terminal 100.

Furthermore, the mobile control terminal 330 wirelessly communicates with the operation terminal 100 via the network 200. The mobile control terminal 330 has an interface based on a wireless communication standard such as 5G or the IEEE 802.11 standard (for example, WiFi), corresponding to a wireless base station or an access point (both not shown) of the network 200. It may have a plurality of interfaces corresponding to a plurality of wireless communication standards. The mobile control terminal 330 also has an antenna (not shown) corresponding to a wireless communication station or access point to which it may connect. Note that the wireless communication standard used is not limited to 5G or IEEE802.11 as long as communication with the communication unit 101 of the operating terminal 100 is possible.

The mobile control terminal 330 cuts out a part of the omnidirectional area imaged by the omnidirectional camera 320. Mobile object control terminal 330 transmits the image of the cut out area to operation terminal 100 as image data. Mobile object control terminal 330 transmits the moving direction and moving speed of mobile object 300 to operation terminal 100.

Next, an example of operation will be explained. Here, an example will be described in which a smartphone is used as the operation terminal 100.

From the mobile object 300, images captured by the omnidirectional camera 320 are transmitted to the smartphone in real time. The remote operator can view the image captured by the omnidirectional camera 320 mounted on the remote mobile object 300 through the screen of the smartphone. Then, the remote operator can control the movement of the mobile object 300 by operating the smartphone.

Mobile object 300 is a robot car. The robot car can only move in horizontal directions, front, back, left, and right. Furthermore, in a robot car, the operation of changing the direction of travel differs depending on the configuration of the drive system such as wheels. As a configuration for changing the traveling direction, for example, a configuration that gradually turns left (or right) by changing the angle of the front wheels like a normal vehicle can be adopted. Further, as a configuration for changing the traveling direction, a configuration such as a mecanum wheel that rotates on the spot to direct the vehicle body in the traveling direction and then moves forward may be adopted. Moreover, as a structure for changing the traveling direction, a structure that allows the vehicle to move straight in an oblique direction, such as a mecanum wheel, may be adopted.

Here, unless otherwise specified, a moving object 300 that changes its traveling direction in the same manner as a normal vehicle will be described as an example.

FIG. 5 is an explanatory diagram for explaining display image example 1 displayed on the operation display section 105.

As shown in FIG. 5, the operation display unit 105 displays an image of a part of the omnidirectional area imaged by the omnidirectional camera 320. This image is the first image. The omnidirectional area image is a spherical image viewed from the inside of the sphere (described later). Therefore, some of the images in the omnidirectional area are also curved. Therefore, the operating terminal 100 corrects the curved image of the selected area so that it becomes flat. The corrected image is displayed on the operation display section 105 of the operation terminal 100.

The operation display unit 105 displays an image of a partial area of the image from the omnidirectional camera 320, as well as a forward mark, a reticle, a direction bar, and a speedometer.

In display screen example 1 shown in FIG. 5, the center of the screen is the viewing direction.

The forward mark is a first mark indicating the current direction (forward direction) of the moving body 300. In display screen example 1, the forward mark is displayed at the center of the screen. The forward mark moves on the direction bar. The front mark indicates which direction and how much the moving body 300 faces with respect to the viewing direction (the center of the screen).

The reticle is a second mark indicating the moving target direction. In display screen example 1, a reticle is displayed at the center of the screen, and a forward mark is also displayed pointing toward the center of the screen. The moving body 300 moves forward in the direction pointed by the reticle, that is, in the same direction as the viewing direction. The operating method for moving the moving body 300 forward and accelerating/decelerating the moving body 300 will be described later.

The direction bar is a guide for displaying the direction in an easy-to-understand manner.

The speedometer indicates the current speed of the moving object 300. Note that although the speedometer is displayed using a graphical figure, the actual speed may be displayed numerically instead of or in addition to this.

FIG. 6 is an explanatory diagram for explaining display image example 2 displayed on the operation display section 105.

Display image example 2 shown in FIG. 6 shows a case where the viewing direction of the moving body 300 is changed. When you change the line of sight, the position of the reticle moves within the screen. For example, if you shift your line of sight slightly to the upper right, the image displayed on the screen will move in that direction. On the other hand, as shown in FIG. 6, the reticle moves slightly to the lower left from the center of the screen, which is the viewing direction. Further, as shown in FIG. 6, since the front mark is on the right side of the reticle, it can be seen that the moving body 300 is currently facing to the right with respect to the instructed movement direction. Then, by inputting a forward instruction, the moving body 300 moves in the direction indicated by the reticle while turning to the left.

The image displayed on the operation display unit 105 is part of an omnidirectional (360 degree) image. When the front mark is located at the center of the direction bar (in the case of FIG. 5), the moving body 300 is moving in the same direction as the viewing direction. Furthermore, when the forward mark is located on the right side of the direction bar (in the case of FIG. 6), the moving body 300 is moving to the right of the line of sight.

FIG. 7 is an explanatory diagram for explaining display image example 3 displayed on the operation display section 105.

Display image example 3 shown in FIG. 7 shows a case where the moving target does not fit on the screen of the operation display unit 105. Such a screen occurs, for example, when attempting to change the moving direction beyond the range of the image displayed on the screen. In such a case, the reticle is only half displayed at the edge of the screen, as shown in FIG. Note that the display method when attempting to change the moving direction beyond the range of the image displayed on the screen is not limited to the method shown in FIG. 7. For example, changing the color of the reticle or disappearing the reticle may indicate that the direction of movement has changed beyond the range of the image displayed on the screen. Furthermore, a warning may be displayed to the effect that the moving direction is about to be changed beyond the range of the image displayed on the screen.

Further, the forward mark may not fit on the screen of the operation display unit 105 in some cases. In such cases, the forward mark will be displayed at the end of the direction bar. Further, in such a case, the forward mark may be displayed in a different color. Further, in such a case, a warning may be displayed in addition to displaying the forward mark.

The vertical position of the reticle within the screen will be explained. 8 and 9 are explanatory diagrams for explaining the vertical position of the reticle within the screen.

When the moving object 300 is running on flat ground and the line of sight direction L (described later) is the same as the direction directly in front of the moving object 300 (progressing direction), the reticle is located at the center of the screen in the vertical direction.

On the other hand, for example, if the moving body 300 is climbing a slope and the line of sight direction L is the same as the front (progressing direction) of the moving body 300, the line of sight direction L is upward, as shown in FIG. The vertical direction the reticle points is the same as the level line (a line parallel to the ground). Therefore, the reticle is located below the vertical center of the screen.

Conversely, when the moving body 300 is going down a slope and the line of sight direction L is the same as the direction directly in front of the moving body 300 (progressing direction), the line of sight direction L is downward, as shown in FIG. The vertical direction the reticle points is the same as the level line (a line parallel to the ground). Therefore, the reticle is located above the vertical center of the screen.

However, for aerial drones and underwater drones that move three-dimensionally, the restriction of fixing the reticle in a direction parallel to the level ground (level line) may be eliminated. For aerial and underwater drones that move three-dimensionally, if the reticle's position on the screen is above the vertical center, it is processed as an instruction to ascend, and if it is below the center, it is processed as an instruction to descend. It's okay.

Next, an image captured by the omnidirectional camera 320 will be described.

FIG. 10 is an explanatory diagram for explaining example 1 of the captured image captured by the omnidirectional camera 320. FIG. 11 is an explanatory diagram for explaining example 2 of the captured image captured by the omnidirectional camera 320.

Captured image examples 1 and 2 shown in FIGS. 10 and 11 show images captured by the omnidirectional camera 320 as virtual spherical surfaces viewed from the inside (center point of the sphere). As shown in FIGS. 10 and 11, when viewed from the center point of the sphere, the direction L (Line Of Sight) indicates the line of sight direction of the remote operator, and the direction F (Forward) indicates the direction of movement of the moving object 300. (forward), and the direction R (Reticle) indicates the direction in which the moving body 300 is desired to be moved, that is, the movement target direction indicated by the reticle. Hereinafter, the respective directions will be referred to as the line-of-sight direction L, the traveling direction F, and the reticle direction R. Information on these directions is stored in the storage unit 103 in the control unit 102 as internal parameters.

In captured image example 1 shown in FIG. 10, area A1 is an area displayed as display screen example 1 shown in FIG. Area A1 is cut out into a predetermined range on the mobile body 300 side and transmitted to the operation terminal 100.

In FIG. 10, the viewing direction L, the traveling direction F, and the reticle direction R are all the same direction. In this embodiment, the initial state is a state in which the viewing direction L, the traveling direction F, and the reticle direction R are all in the same direction.

On the other hand, in the captured image example 2 shown in FIG. 11, area A2 is an area displayed as the display screen example 2 shown in FIG. Similar to area A1, this area A2 is cut out into a predetermined range on the mobile body 300 side and transmitted to the operation terminal 100.

In FIG. 11, the viewing direction L indicates the direction changed by the remote operator, the traveling direction F indicates the direction in which the moving object 300 is traveling at that time, and the reticle direction R indicates the direction of the moving target at that time. ing.

Note that in the images of the omnidirectional area (FIGS. 10 to 12), the vertical position of the reticle direction R is always parallel to the flat ground, so it is on the plane H.

As shown in FIGS. 10 and 11, the image of the omnidirectional area is entirely spherical. Therefore, the image displayed on the operation display section 105 will be a curved image if left as is. Therefore, in the present embodiment, when displaying an image on the operation display unit 105, the areas A1 and A2 within a predetermined range from the center O are flattened, with the point where the straight line in the line of sight direction L intersects with the spherical surface as the center O. Display. The flattening is executed by the control unit 102 of the operation terminal 100.

The ranges of areas A1 and A2 (a predetermined range from center O) are the same in size, but differ in direction. The ranges of areas A1 and A2 are determined in advance according to the screen size and shape of the operation display unit 105, the resolution, and the permissible level of image distortion correction.

A straight line in the traveling direction F is always on a plane H parallel to the flat ground.

The direction bar displayed on the operation display unit 105 indicates the area A1 range in the meridian direction of the spherical surface of the captured image example 1.

The forward mark displayed on the operation display section 105 is displayed at a position where the point where the straight line in the traveling direction F intersects with the spherical surface is moved along the meridian and intersects with the direction bar in the area A1.

A straight line in the reticle direction R is always on a plane H parallel to the flat ground. A point P where a straight line in the reticle direction R intersects with the spherical surface indicates the position of the reticle, that is, the direction of the moving target. The display of these front marks and reticles is the same in area A2.

Here, a case will be described in which the forward mark and reticle do not fit within the line-of-sight image displayed on the operation display unit 105.

FIG. 12 is an explanatory diagram for explaining example 3 of the captured image captured by the omnidirectional camera 320. Captured image example 3 shows a case where the front mark and reticle do not fit within the screen (image) in the viewing direction. Note that in the third captured image example as well, the definitions of the line-of-sight direction L, the traveling direction F, and the reticle direction R are the same as in the captured image examples 1 and 2.

When the traveling direction F of the moving object 300 is outside the range of the area A3 cut out as the image in the line-of-sight direction, if the point where the straight line in the traveling direction F intersects with the spherical surface is moved along the meridian, it will not intersect with the direction bar. In such a state, the forward mark displayed on the operation display section 105 is displayed at the end of the direction bar within the area A3. Therefore, if the point where the straight line in the traveling direction F intersects with the spherical surface is moved along the meridian and the point is to the right of area A3, the forward mark will be at the right end of the direction bar, and the point will be further to the right than area A3. If you are off to the left, the forward mark will be displayed at the left end of the direction bar.

The reticle, like the forward mark, may also fall out of the image. When the moving target point P is outside the range of area A3, the reticle is displayed at a position where the left end of area A3 intersects surface H, as shown in FIG.

Vector coordinates are used to specify the viewing direction L, the traveling direction F, and the reticle direction R. For example, by fixing the coordinate system to the omnidirectional camera 320 and fixing the direction directly in front of the moving body 300 to the coordinate axis x, the traveling direction F can be expressed only by the inclination of the moving body 300 with respect to the flat ground, and the calculation is Can be omitted. The line-of-sight direction L and the reticle direction R are calculated by measuring the amount of rotation of the moving body 300 at any time and reflecting the measured amount of rotation on the fixed vector coordinates of the traveling direction F.

As a method for measuring the amount of rotation, a change in the orientation of the moving body 300 is directly measured using a gyro sensor, an acceleration sensor, or the like mounted on the moving body 300. In addition, the rotation amount can be measured by detecting an object using a TOF (Time of Flight) sensor 104 used in LiDAR (Light Detection and Ranging), or by image recognition AI, and calculating from the detected information. You may.

As described above, the positions of the front mark and the reticle on the image of a part of the area cut out from the image of the omnidirectional area captured by the omnidirectional camera 320, and the physics related to the image captured by the omnidirectional camera 320 are explained. The line-of-sight direction L, the traveling direction F, and the reticle direction R are related to each other.

Next, an operation for changing the viewing direction L will be explained.

FIG. 13 is an explanatory diagram for explaining an operation for changing the viewing direction L.

As described above, in the initial state, the viewing direction L and the traveling direction F match. That is, in the initial state, the viewing direction L points in the same direction as the traveling direction F of the moving body 300 on a plane H parallel to the flat ground.

The viewing direction L is changed from this initial state by rotating the operating terminal 100 while performing the first input. The first input is performed, for example, by touching the screen of the operation display unit 105 (smartphone screen, hereinafter also simply referred to as "screen") with one finger, as shown in FIG. 13 .

First, when the screen is touched with one finger as a first input, the control unit 102 executes screen lock. Here, the screen lock is a state in which the viewing direction L can be changed according to the movement of the operating terminal 100 (smartphone). Operating terminal 100 is rotated in this state. The rotation has already been explained.

The operation terminal 100 is equipped with an acceleration sensor and a gyro sensor. The acceleration sensor and the gyro sensor output the rotation direction and rotation amount of the operation terminal 100 to the control unit 102. The control unit 102 measures the amount of rotation from the moment the screen is touched with one finger. Then, the control unit 102 transmits the measured amount of rotation to the moving body 300 as the amount of rotation in the line-of-sight direction L. The moving body 300 moves the center O on the spherical surface that is the omnidirectional image shown in FIGS. 10 and 11. In conjunction with the movement of the center O, for example, area A1 moves to area A2 (more precisely, the cutting position of the image changes). The control unit 102 receives the area A2 updated by this movement from the mobile object 300. The control unit 102 flattens the received image of the area A2 and displays it on the operation display unit 105 in real time.

Returning to FIG. 13, explanation will be given. When the smartphone is slid (rotated) to the right from a state in which the screen is touched with one finger, an image on the right side with respect to the traveling direction of the mobile object 300 is displayed. As a result, the image displayed on the operation display section 105 shifts from the state before sliding (rotation) to the image after sliding (rotation). At this time, the traveling direction F and the reticle direction R do not change. Therefore, the front mark and reticle move to the left in the image on the operation display unit 105.

Note that portions that are not included in the image captured by the omnidirectional camera 320 are displayed as blanks on the operation display section 105. Specifically, for example, in a robot car or the like, the area below the mobile body 310 is not imaged. Therefore, when the operating terminal 100 is tilted downward, the center O moves in the direction in which the portion that is not captured as an omnidirectional image is viewed. In such a case, since no images have been acquired in the area below the mobile body 310, part of the image displayed on the operation display unit 105 is displayed as blank.

Next, an operation for changing the reticle direction R will be explained.

FIG. 14 is an explanatory diagram for explaining an operation for changing the reticle direction R.

As described above, in the initial state, the reticle direction R and the traveling direction F match. That is, in the initial state, the reticle direction R faces the same direction as the traveling direction F of the moving body 300.

The reticle direction R is changed from this initial state by rotating the operating terminal 100 while performing the second input. The second input is performed, for example, by touching the screen with two fingers, as shown in FIG.

First, when the screen is touched with two fingers as a second input, the control unit 102 executes screen lock. Here, the screen lock is a state in which the reticle direction R can be changed in accordance with the movement of the operating terminal 100 itself. At this time, simultaneously with the screen lock, the control unit 102 sets the horizontal direction of the reticle direction R to the line-of-sight direction L so that they match. In this state, the control unit 102 also allows the reticle direction R to be moved while always pointing in the same horizontal direction as the line-of-sight direction L. The vertical direction of the reticle direction R is always parallel to the flat ground.

When the operation terminal 100 is rotated in this state, the line-of-sight direction L changes similarly to the above-described operation of the line-of-sight direction L, and the range of the area A1 changes. On the other hand, the horizontal direction of the reticle direction R always points to the center and moves relative to the image in area A1. The vertical direction of the reticle direction R points to a direction parallel to the flat ground. At this time, the front mark moves according to the amount of rotation of the operating terminal 100.

For example, as shown in FIG. 14, when the operating terminal 100 is slid (rotated) to the right while the screen is touched with two fingers, the image displayed on the operating display unit 105 is moves to the right. At this time, the reticle is always located at the horizontal center of the screen. In other words, the image being moved to the operation display section 105 moves (the image flows) as the operation terminal 100 slides (rotates), but the reticle does not move from the center of the screen. The vertical position of the reticle is coincident with the viewing direction L and depends on the position of the image of the area currently being viewed. This causes the reticle to move in the direction of the moved image. At this time, the forward mark is displayed on the left side of the screen.

When the moving body 300 moves forward, it moves in the direction indicated by the reticle direction R, that is, curves to the right. Along with this movement, the front mark moves toward the center on the screen so as to approach the reticle.

Such an operation for changing the reticle direction R can be performed even while the moving body 300 is moving. Therefore, the remote operator can move the moving body 300 forward while turning left and right, just by changing the direction of the operating terminal 100, just as if he were operating the steering wheel of a vehicle. Moreover, by this operation, the remote operator can easily perform delicate operations on the direction of travel by simply moving the operating terminal 100 so that the desired direction is captured in the center of the screen.

The control unit 102 releases the interlock between the screen lock and the reticle at the moment when the finger is removed from the screen of the operation display unit 105. When the link between the screen lock and the reticle is released, the image from the camera changes as the moving object 300 moves, but the line-of-sight direction L and the reticle direction R do not change. Then, when the screen is next touched with two fingers, the control unit 102 maintains the image of the orientation of the operating terminal 100 at the moment of touch, the line of sight direction L and the reticle direction R at the moment when the fingers were previously removed. Display the location of and lock the screen.

Next, operations for accelerating and decelerating the moving body 300 will be explained.

Acceleration/deceleration of the moving body 300 is performed by the third input.

FIG. 15 is an explanatory diagram for explaining operations for accelerating and decelerating the moving body 300.

The third input is performed, for example, by touching and sliding the screen with one finger (drag).

As shown in FIG. 15, when the moving body 300 is dragged upward on the screen, the control unit 102 transmits an instruction (control signal) to accelerate the moving body 300. Note that when the moving body 300 is stopped and dragged upward on the screen, the moving body 300 starts (moves forward).

Conversely, when dragged downward on the screen, the control unit 102 transmits an instruction (control signal) to decelerate the moving body 300. Note that when the moving object 300 is moving at a low speed and is dragged downward on the screen, the moving object 300 stops.

Acceleration/deceleration operations can be performed in various ways other than using one finger, as described above. Other acceleration/deceleration operations are performed using, for example, two fingers as the third input.

FIG. 16 is an explanatory diagram for explaining other acceleration/deceleration operations of the moving body 300.

Other acceleration/deceleration operations are performed by moving (dragging) one finger up or down the screen while the screen is touched with two fingers, as shown in Figure 16. . When dragged upward on the screen, the control unit 102 transmits an instruction (control signal) to accelerate the moving body 300. Note that when the moving object 300 is dragged while it is stopped, the moving object 300 starts (moves forward). Conversely, when dragged downward on the screen, the control unit 102 transmits an instruction (control signal) to decelerate the moving body 300. Note that when the moving object 300 is moving at a low speed and is dragged downward on the screen, the moving object 300 stops.

Acceleration/deceleration operations are performed by various methods other than these. Although not shown, the speed may be specified, for example, by the amount of dragging (the amount of movement from the position where the finger first touched). Also, for example, while holding the smartphone, tilt the top side away from the remote operator to perform forward and acceleration operations, while holding the smartphone and tilt the top side towards the remote operator to decelerate. and stop operations may be performed. In addition, for example, icons such as forward, stop, backward, acceleration, and deceleration may be displayed on the screen, and when these icons are touched, the corresponding operation may be executed.

Next, an operation for reversing the moving direction of the moving body 300 will be described.

Reversal of the moving direction of the moving body 300 is performed by the fourth input.

FIG. 17 is an explanatory diagram for explaining an operation for reversing the moving direction of the moving body 300. FIG. 18 is an explanatory diagram for explaining an omnidirectional image when the moving direction of the moving body 300 is reversed.

For example, as shown in FIG. 17, the fourth input is performed by touching the screen with one finger and moving the finger upward or downward at high speed to move the finger away from the screen (flick). be exposed.

When flicked, the control unit 102 transmits a control signal for reversing the traveling direction of the moving body 300 by 180 degrees. At this time, the displayed image is changed as follows. As shown in FIG. 18, the omnidirectional image after inversion includes the center point C of the spherical surface, and if the direction inverted 180 degrees with respect to the plane S that intersects perpendicularly to the direction of movement F is the direction of movement F', This traveling direction F' becomes the new front of the moving body 300. In other words, when the moving body 300 has a front and rear distinction, the moving body 300 after reversal becomes forward with the rear facing forward and moves backward.

Then, the control unit 102 flattens the area A' in which the area A is mirror-symmetrical with respect to the surface S, and displays it on the operation display unit 105. At this time, the center O moves to the center O', and the viewing direction L points in the direction of the viewing direction L'. Further, the displayed reticle direction R does not change until a movement operation of the reticle direction R is performed by the remote operator. The control unit 102 recalculates the positional relationship between the reticle direction R and the area A' and displays the reticle at a new position.

Further, when the moving body 300 is moved and is flicked, the control unit 102 decelerates the moving body 300 and stops the moving body 300. Note that if the reticle is flicked while it is within the area A, the position of the reticle will be outside the new area A'. In FIG. 18, for example, if the reticle is located at the lower left in area A, after the reversal operation is performed by flicking, the reticle moves to the right end (lower side) of area A'. When moving forward in this state, the moving body 300 proceeds in a U-turn to the right. Normally, after being flicked, the position of the reticle (that is, the direction of the moving target) is reset, and then the moving body 300 is moved forward.

Next, the zoom operation will be explained.

A zoom operation is performed by the fifth input.

FIG. 19 is an explanatory diagram for explaining the zoom operation.

The fifth input is, for example, as shown in FIG. 19, the action of widening the distance between fingers that are narrowed while touching the screen (pinch out), or the action of widening the distance between fingers that are widened while touching the screen. This is done by a pinch-in action.

When pinched out, the control unit 102 sets the pinched-out position in the line-of-sight direction L, and makes the horizontal direction of the reticle direction R coincide with the line-of-sight direction L. The vertical direction of the reticle direction R is determined by the line-of-sight direction L, that is, the positional relationship with the area A. Then, the control unit 102 enlarges the image displayed on the screen of the operation display unit 105 (zooms in) according to the amount by which the fingers are spread by the pinch-out operation. Further, the control unit 102 causes the storage unit 103 to store an enlarged image at the moment when the pinch-out finger is released. Thereafter, the control unit 102 returns the image displayed on the screen of the operation display unit 105 to the image at the enlargement magnification before being pinched out. Subsequently, the control unit 102 moves the moving object 300 forward until the image received from the moving object 300 becomes the same size as the image with the magnification magnification at the time of pinch-out stored earlier.

On the other hand, when pinched in, the control unit 102 sets the pinched-in position in the line-of-sight direction L, and makes the horizontal direction of the reticle direction R coincide with the line-of-sight direction L. The vertical direction of the reticle direction R is determined by the line-of-sight direction L, that is, the positional relationship with the area A. Then, the control unit 102 reduces the image displayed on the screen of the operation display unit 105 (zooms out) according to the amount by which the finger is pinched by the pinch-in operation. Further, the control unit 102 causes the storage unit 103 to store the reduced image at the moment when the pinch-in finger is released. Thereafter, the control unit 102 returns the image displayed on the screen of the operation display unit 105 to the image at the enlargement magnification before being pinched in. Subsequently, the control unit 102 causes the moving object 300 to move backward until the image received from the moving object 300 becomes the same size as the image at the magnification magnification at the pinch-in time that was stored earlier.

As described above, the zoom operation is performed by various methods other than the pinch operation. Other zoom operations are performed using one finger, for example as a fifth input.

FIG. 20 is an explanatory diagram for explaining another zoom operation.

A fifth input for another zoom operation is, for example, a long press on the screen with one finger, as shown in FIG.

When pressed for a long time, the control unit 102 sets the long pressed position in the line-of-sight direction L, and makes the horizontal direction of the reticle direction R coincide with the line-of-sight direction L. The vertical direction of the reticle direction R is determined by the line-of-sight direction L, that is, the positional relationship with the area A. Then, the control unit 102 moves the moving body 300 forward while the finger is pressed for a long time, and stops the moving body 300 when the long-pressing finger is released from the screen. Therefore, the moving body 300 will move in that direction while the button is pressed for a long time. Therefore, to the remote operator, the object appearing in the image appears to be enlarged.

Among the zoom operations described above, pinch-out or long-press operations are used, for example, when the moving object 300 is far away and difficult to see. For example, if the letters written on a far wall are small and difficult to read, the remote operator can perform a zoom-in operation to not only enlarge the image but also to move the mobile object 300 closer to the wall. I can do it. This makes it easier for the remote operator to read characters.

On the other hand, the zoom-out operation is used, for example, when there is an object near the moving body 300 and the entire object cannot be seen. For example, if there is an object that is too close to the moving object 300 to be fully visible, the remote operator can perform a zoom-out operation to not only reduce the image but also move the moving object 300 away from the object. Can be done. This allows the remote operator to see the entire object.

During such a zoom operation, for example, the color of the reticle may be changed. For example, the color may be changed to green when zooming in and red when zooming out. Of course, any color may be used. Further, characters may be displayed while zooming in, zooming out, etc.

Note that while moving by zooming, the reticle direction R does not change if no operation is performed, but as the moving body 300 moves, the direction in which the object to be captured by the omnidirectional camera 320 is viewed also changes. Therefore, the image enlarged when zooming in or the image reduced when zooming out does not completely match the image after movement. Furthermore, the amount of movement (amount of forward or backward movement) when zooming in or zooming out is an approximate value calculated by triangulation.

Next, collision detection will be explained.

FIG. 21 is an explanatory diagram for explaining collision detection.

As shown in FIG. 21, collision detection is performed by displaying on the operation display unit 105, in addition to displaying area A, an image of area B centered on the reticle direction R in which the moving object 300 is about to proceed. be exposed. Region B is a predetermined range centered on the reticle direction R. Region B includes a different range from region A. This area B is displayed on the operation display section 105 as a second image.

The moving body 300 transmits a region B cut out from a predetermined range centered on the reticle direction R to the operating terminal 100. The control unit 102 flattens the received area B and displays it on the operation display unit 105.

The remote operator can view an image of the direction in which the moving body 300 will move from now on. Therefore, the remote operator can operate the moving body 300 while checking safety, such as whether there are any people and/or objects in the direction in which the moving body 300 is moving, or whether people and/or objects are moving.

Furthermore, an AI model for detecting people and objects may be installed in the control unit 102 in advance to detect people and/or objects from the image of area B. Thereby, the control unit 102 stops the moving body 300 when a person or object is detected in the reticle direction R by the AI model.

Furthermore, a TOF sensor used in LiDAR or the like may be installed, and such a sensor may be notified of information on the reticle direction R to detect objects in that direction. Thereby, the control unit 102 stops the moving body 300 when a person or object is detected in the reticle direction R by the sensor.

In this way, by setting not only area A but also area B, which is narrower than area A in the direction in which the remote operator will move from now on, and displaying it on the operation display section 105, the remote operator can It becomes easier to detect people and objects. In addition, by setting such a region B, when detecting people or objects using AI or TOF sensors, it is possible to detect people or objects from a small amount of data with a narrowed detection range. . Therefore, even with a small processing capacity, highly accurate detection and the resulting collision avoidance operation can be performed at high speed.

Note that although the area B mentioned above has a narrower range than the area A, the range is not limited to this, and the area B may be the same size as the area A or may be wider than the area A. Further, the display of this area B (second image) may not be displayed.

Next, details of movement control will be explained.

FIG. 22 is a vector diagram regarding movement control. FIG. 22 shows a three-dimensional coordinate system.

Vectors related to movement control are the line-of-sight direction L, the traveling direction F, and the reticle direction R, as described above. Each of these directions can be represented by a vector in a polar coordinate system, as shown in FIG. Here, only the direction of the vector is needed. Therefore, each direction can be represented by a vector P(Θ, φ) that is a function of argument angles Θ and φ, with the vector length r as a constant.

In the initial state, the line-of-sight direction L, the traveling direction F, and the reticle direction R are the same, and with Θ=C0 and φ=0, the line-of-sight direction L, the traveling direction F, and the reticle direction R are all based on the vector P0 (C0 ,0). Here, C0 is the initial value of the inclination of the moving body 300 with respect to the flat ground. The inclination of the moving body 300 can be acquired by an acceleration sensor and a gyro sensor mounted on the moving body 300. Note that this coordinate system is a coordinate fixed to the omnidirectional camera 320 mounted on the moving body 300, and the point directly in front of the moving body 300 is fixed to the coordinate axis x. The traveling direction F is always φ=0, and Θ is the inclination of the moving body 300 with respect to the flat ground.

FIG. 23 is a vector diagram for explaining a case where the moving body 300 is tilted.

For example, if the moving body 300 is on a slope with an inclination angle α and the declination angle Θ of the line of sight direction L is ΘL, the line of sight direction L has an elevation angle of 90° with respect to the flat ground (G (horizontal plane) in FIG. 23). This means that you are looking in the direction of ΘL+α. In this case, even if the moving body 300 moves on a slope and the inclination angle α changes, the declination angle Θ does not change unless a movement operation in the line-of-sight direction L or the reticle direction R is performed. Therefore, even if the inclination angle α changes, the coordinate system on the spherical surface of the image captured by the omnidirectional camera 320 does not change, but the image captured there changes. In other words, as can be seen by comparing FIGS. 22 and 23, this means that the vector P (Θ, φ) in the coordinate system of the moving body 300 does not change even if the moving body 300 is tilted. However, the reticle direction R always points in a direction parallel to the flat ground G. Therefore, the reticle direction R is corrected by the deflection angle Θ corresponding to the vertical component of the inclination due to the movement of the moving body 300. Correction of the reticle direction R is performed by the mobile object control terminal 330.

On the other hand, the value of the declination angle φ is updated one by one in accordance with the rotation (change in azimuth) of the moving body 300. 24 and 25 are vector diagrams for explaining a case where the direction of the moving body 300 changes. 24 and 25 show xy two-dimensional coordinates of the three-dimensional coordinate system shown in FIG. 22.

For example, if the moving body 300 rotates by an azimuth displacement ω=β as shown in FIG. 25 from a state where the declination angle φ1 is as shown in FIG. 24, the declination angle φ2 at that time is as follows ( 1) It is expressed by the formula.

φ2=φ1-β (1)
Here, the azimuth displacement ω is obtained by an acceleration sensor and a gyro sensor mounted on the moving body 300.

Therefore, if the line of sight direction L or the reticle direction R is not changed, the image moves up and down with respect to the movement of the moving object 300 in the vertical direction, but in the horizontal direction, the image is displayed in an almost constant direction. will be done.

Next, rotation control of the moving body 300 will be explained.

Vector coordinates in the line-of-sight direction L are represented by L (ΘL, φL), and vector coordinates in the reticle direction R are represented by R (ΘR, φR). Further, the azimuth displacement of the moving body 300 is assumed to be ω. These values are managed by mobile control terminal 330 (computer). In the initial state, ΘL=0, φL=0, ΘR=0, φR=0. ω is a value (displacement from the previous acquisition time) acquired from the acceleration sensor and gyro sensor of the moving body 300, and is updated at a constant cycle. Additionally, φL and φR are also updated at the same time as this update. Therefore, φL and φR to be updated become the following equations (2) and (3).

φL=φL−ω (2)
φR=φR−ω (3)
Further, if φR is not 0, the moving body 300 is bent (or rotated) so that φR approaches 0. This is an operation that causes the moving body 300 to bend in the reticle direction R. How fast and how φR approaches 0 depends on the mechanical control performance of the moving body 300.

FIG. 26 is a flowchart showing the rotation operation procedure. The control unit 102 executes a program created according to this rotation control procedure.

First, the control unit 102 sets ΘL=0, φL=0, ΘR=0, φR=0, and resets ω (S101).

Next, the control unit 102 obtains ω and calculates φL=φL−ω and φR=φR−ω (S102).

Subsequently, the control unit 102 determines whether φR>0 (S103). If φR>0 (S103: YES), then the control unit 102 transmits to the mobile body 300 a control signal for causing the mobile body 300 to turn left (S104).

The control signal transmitted from the operation terminal 100 is received by the mobile object control terminal 330, and is directly interpreted there to control the mobile object 300. At this time, the control signal may include instructions other than rotation. For example, the control signal includes instructions such as acceleration and deceleration. The mobile control terminal 330 executes these instructions (the same applies to the following processing).

After that, the control unit 102 determines whether or not the process is finished (S107). If the process has not ended (S107: NO), the control unit 102 returns to S101 and continues the process. As a result, the operation of the mobile body 300 is continued. On the other hand, if the process is to end (S107: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

In step S103, if φR>0 is not satisfied (S103: NO), it is determined whether φR<0 or not (S105). If φR<0 (S105: YES), then the control unit 102 transmits a control signal to the mobile body 300 for causing the mobile body 300 to turn to the right (S106).

After that, the control unit 102 determines whether or not the process is finished (S107). Furthermore, in the step S105, if φR<0 is not satisfied (S105: YES), the control unit 102 also determines whether the process is finished (S107). If the process has not ended (S107: NO), the control unit 102 returns to S101 and continues the process. As a result, the operation of the mobile body 300 is continued. On the other hand, if the process is to end (S107: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

Next, movement control in the viewing direction L will be explained.

As already explained, when the orientation of the operating terminal 100 (smartphone) is changed, the acceleration sensor and gyro sensor within the operating terminal 100 detect in which direction and by how much the operating terminal 100 has been rotated. Since the amount can be expressed by the declination angle Θ and φ, the amount is directly reflected in the amount of rotation in the viewpoint direction of the omnidirectional camera 320.

Specifically, the displacement angles Θd and φd from the moment of touch with one finger are transmitted from the operation terminal 100 to the mobile body 300. After a certain period of time, the displacement angles Θd and φd from the time of the previous transmission are transmitted to the moving body 300. After that, this operation is repeated at short intervals until one finger is released.

The mobile object control terminal 330 receives the displacement angles Θd and φd, and calculates and updates the coordinates L (ΘL, φL) in the line-of-sight direction L using equations (4) and (5) below.

ΘL=ΘL+Θd (4)
φL=φL+φd (5)
Then, the mobile object control terminal 330 cuts out a region A within a predetermined range centered on the coordinates L (ΘL, φL) (that is, the line-of-sight direction L) and transmits it to the operation terminal 100. The control unit 102 flattens area A and displays it.

FIG. 27 is a flowchart showing a movement control procedure in the line-of-sight direction L. The control unit 102 executes a program created in accordance with the movement control procedure in the line-of-sight direction L.

First, the control unit 102 determines whether the operation display unit 105 is touched with one finger (S201). When the operation display unit 105 is touched with one finger (S201: YES), the control unit 102 recognizes that the operation display unit 105 is touched with one finger, and calculates the declination angles Θ and φ. Reset (S202).

Subsequently, the control unit 102 determines whether or not the operation display unit 105 is still being touched with one finger (S203). If the operation display unit 105 is still touched with one finger (S203: YES), then the control unit 102 acquires Θd and φd and transmits them to the mobile object 300 (S204).

Upon receiving Θd and φd, the mobile body 300 side sets ΘL=ΘL+Θd, φL=φL+φd, and transmits a new area A to the operating terminal 100 (S205). This step S205 is a process performed by the mobile control terminal 330.

The control unit 102 that has received the new area A flattens the received area A and displays it on the operation display unit 105 (S206).

After that, the control unit 102 determines whether or not the process is finished (S207). Also, in the step S201, if the operation display unit 105 is not touched with one finger (S201: NO), and in the step S203, if the finger is removed from the operation display unit 105 (S203: NO), the control The unit 102 determines whether the processing is finished (S207). If the process has not ended (S207: NO), the control unit 102 returns to S201 and continues the process. As a result, the operation of the mobile body 300 is continued. On the other hand, if the process is to end (S207: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

Note that the rotation control of the mobile body 300 and the movement control in the viewing direction L are performed simultaneously. The moving body 300 also rotates during the operation of changing the viewing direction L. Therefore, the image captured by the omnidirectional camera 320 may change. Furthermore, there is a delay time before the image captured by the omnidirectional camera 320 is displayed on the operation display section 105. Taking these into consideration, it is preferable that the coordinate update process be performed at a low frequency that does not give a sense of discomfort to the remote operator.

Next, movement control in the reticle direction R will be explained.

Similar to the line-of-sight direction L, the reticle direction R is represented by the declination angles Θ and φ, and the amount of rotation of the operating terminal 100 is reflected in the amount of rotation in the reticle direction R.

Specifically, first, the operation terminal 100 transmits to the moving body 300 that the operation display unit 105 has been touched with two fingers. The mobile object control terminal 330 matches the coordinates of the line-of-sight direction L and the reticle direction R. That is, the mobile control terminal 330 sets each coordinate as shown in equations (6) and (7) below.

ΘR=ΘL (6)
φR=φL (7)
Thereafter, the control unit 102 continues to calculate the displacement angles Θd and φd and transmit them to the mobile object 300 as the terminal device rotates. When the two fingers are removed from the operation display unit 105, the operation terminal 100 transmits this to the mobile object 300. The mobile object control terminal 330 calculates the coordinates L (ΘL, φL) and the coordinates R (ΘR, φR) according to the following equations (8) to (11) and updates them at the same time.

ΘL=ΘL+Θd (8)
ΘR=C (9)
φL=φL+φd (10)
φR=φL (11)
In the formula, C is the amount of rotation required to align the reticle direction R with a plane H parallel to the flat ground. The amount of rotation C indicates in which direction and how much the moving body 300 is tilted from a plane parallel to the flat ground. The direction and how much the moving body 300 is tilted from a plane parallel to the flat ground is measured by, for example, a gyro sensor of the moving body 300.

As a result, the direction in which the line-of-sight direction L is projected onto the plane H parallel to the flat ground becomes the reticle direction R. Note that if the moving body 300 is parallel to the flat ground, C=90°.

Then, the mobile object control terminal 330 cuts out a region A within a predetermined range centered on the coordinates L (ΘL, φL) (that is, the line-of-sight direction L) and transmits it to the operation terminal 100. The control unit 102 flattens area A and displays it. Further, the control unit 102 displays the reticle at the center of the screen.

FIG. 28 is a flowchart showing a movement control procedure in the reticle direction R. The control unit 102 executes a program created in accordance with this reticle direction R movement control procedure.

First, the control unit 102 determines whether or not the operation display unit 105 is touched with two fingers (S301). When the operation display unit 105 is touched with two fingers (S201: YES), the control unit 102 subsequently recognizes that the operation display unit 105 is touched with two fingers, and sets ΘR=ΘL and φR= φL is transmitted to the mobile body 300 (S302).

Subsequently, the control unit 102 determines whether or not the operation display unit 105 is still being touched with two fingers (S303). If the operation display unit 105 is still being touched with two fingers (S303: YES), then the control unit 102 acquires Θd and φd and transmits them to the mobile object 300 (S304).

The mobile object 300 that has received Θd and φd calculates each value as ΘL=ΘL+Θd, φL=φL+φd, ΘR=C, φR=φL, and transmits a new area A to the operation terminal 100 (S305). This step S305 is a process executed in the mobile control terminal 330.

The control unit 102 that has received the new area A flattens the received area A and displays it on the operation display unit 105, and also displays the reticle at the center of the screen (S306).

After that, the control unit 102 determines whether or not the process is finished (S307). In addition, in the step S301, if the operation display unit 105 is not touched with two fingers (S301: NO), and in the step S303, if the fingers are removed from the operation display unit 105 (S303: NO), the control The unit 102 determines whether the process is finished (S307). If the process has not ended (S307: NO), the control unit 102 returns to S301 and continues the process. As a result, the operation of the mobile body 300 is continued. On the other hand, if the process is to end (S307: YES), the control unit 102 ends the process. Note that the end of the process is, for example, an end instruction from the user.

According to this embodiment described above, the following effects are achieved.

In the remote control system of this embodiment, the mobile object 300 has an omnidirectional camera 320 that captures images in all directions. Thereby, the remote operator can freely view and operate the desired direction when moving the moving body 300. Therefore, in this embodiment, position information of the mobile object 300 is not required. Thereby, the remote operator can operate the moving body 300 while looking in the desired direction. Furthermore, in this embodiment, when the remote operator wants to look in the desired direction, it is not necessary to physically change the imaging direction, such as changing the direction of the moving object 300 or changing the shooting direction of the camera. do not. Therefore, in this embodiment, no operation delay occurs for changing the photographing direction.

Furthermore, in this embodiment, even when it is desired to move along a complicated route, there is no need to use a point-and-click device to specify multiple points as the route.

Furthermore, in the present embodiment, an area to be displayed on the operating terminal 100 is cut out from an image captured in all directions on the moving body 300 side, and is transmitted to the operating terminal 100. Therefore, since the amount of data to be transmitted is reduced, data delay associated with wireless communication can also be reduced.

Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above.

The above-described embodiment has been described on the assumption that a smartphone is used as the operation terminal 100, but the operation terminal 100 in which the operation unit (operation controller) equipped with an acceleration sensor and a gyro sensor and the display are separated is used. It's okay.

Further, in the above-described embodiment, the operation was realized only by a touch operation on the screen of the operation display unit 105 and rotation of the operation terminal 100 itself. However, a more precise operation may be realized by providing an operation unit such as a game controller as the operation terminal 100 and accepting a plurality of button operations. Further, the operation terminal 100 may be realized by being separated into an operation section that can perform operations equivalent to touch operations on a smartphone, and XR glasses equipped with an acceleration sensor and a gyro sensor. That is, instead of the remote operator moving the smartphone by hand, the remote operator wearing XR glasses may be able to specify the viewing direction or reticle direction by moving his head. Note that XR glasses are high-performance glasses that realize VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and the like.

In addition, the imaging unit obtains an omnidirectional image by attaching wide-angle cameras to the front, rear, left, and right sides of the moving body 300 instead of the omnidirectional camera 320, capturing omnidirectional images, and combining the images. It's okay.

Furthermore, the input operations such as touch, drag, and flick in the embodiments described above may be, for example, gesture operations that do not involve touching the screen. Conversely, these input operations may be performed by displaying buttons, icons, etc. corresponding to these input operations on the screen and touching them.

Furthermore, regarding the movement of the mobile object 300, a mode in which the direction of movement changes (turns) by changing the angle of the front wheels like a normal vehicle is shown, but a mode in which the direction of movement changes by using mecanum wheels is also shown. may also be applied. When the vehicle body is rotated on the spot to point in the traveling direction and then moved forward like a mecanum wheel, when the reticle direction R is changed, the moving body 300 rotates on the spot and moves in the traveling direction. F can be made to coincide with the reticle direction R immediately. Further, in the case of a structure such as a mecanum wheel that moves straight in an oblique direction, the moving direction F can be considered to always match the reticle direction R because it can be moved in any direction. Furthermore, in this case, since the mobile object 300 can also move in the lateral direction, a touch input operation of the smartphone indicating movement in the lateral direction may be added. For example, the moving body 300 may be configured to move horizontally to the left (right) by dragging a finger in the left (right) direction on the smartphone.

Further, in the above-described embodiment, a mode has been described in which an image of a part of the omnidirectional area is cut out on the mobile body 300 side and the image data is transmitted to the operation terminal 100, but the present invention is not limited to this. For example, the moving body 300 may transmit image data of all omnidirectional areas to the operation terminal 100, and the operation terminal 100 may cut out a part of the area. In this case, the moving body 300 does not need to cut out the image. In such a case, for example, the moving body 300 may transmit the moving speed and direction to the operating terminal 100, and the omnidirectional camera 320 may directly transmit image data of an omnidirectional area to the operating terminal 100.

Moreover, the program according to the present invention can also be realized by a dedicated hardware circuit. In addition, this program may be provided on a computer-readable recording medium such as a USB (Universal Serial Bus) memory, a DVD (Digital Versatile Disc) or a ROM (Read Only Memory), or may be provided over a network such as the Internet regardless of the recording medium. It is also possible to provide the information online via 200. When provided online, this program is recorded on a recording medium such as a magnetic disk in a computer connected to the network 200.

Further, the present invention can be modified in various ways based on the configuration described in the claims, and these are also within the scope of the present invention.

1 remote control system,
100 operation terminal,
101 Communications Department,
102 control unit,
103 storage unit,
104 sensor,
105 operation display section,
200 network,
300 mobile object,
310 mobile body,
320 omnidirectional camera,
330 Mobile control terminal.

Claims (24)

A moving object,
an operation terminal for remotely controlling the mobile object via wireless communication;
A remote control system having
The mobile body has an imaging unit that captures images around the mobile body in all directions,
The operation terminal is
an input section;
A display section;
Displaying on the display unit a first image that is at least a part of the omnidirectional area imaged by the imaging unit and that corresponds to the viewing direction input from the input unit; a control unit for superimposing a direction of the first image, a first mark indicating a traveling direction of the moving body, and a second mark indicating a movement target direction of the moving body on the first image and displaying the same on the display unit; ,
A remote control system with The operation terminal has a sensor that detects acceleration and angular velocity,
The remote control system according to claim 1, wherein the control unit controls movement of the mobile body based on the output from the sensor. The remote controller according to claim 2, wherein the control unit controls movement of the moving body based on an input for moving the moving body from the input unit and acceleration and angular velocity detected by the sensor. operation system. The remote controller according to claim 3, wherein the control unit moves the viewing direction based on a first input input from the input unit and an acceleration and an angular velocity detected by the sensor after the first input. operation system. The remote control system according to claim 3 or 4, wherein the control unit moves the second mark to the center of the screen of the display unit based on a second input input from the input unit. The remote control system according to any one of claims 3 to 5, wherein the control unit controls the moving speed of the moving body based on a third input input from the input unit. The remote control system according to any one of claims 3 to 6, wherein the control unit reverses the moving direction of the moving body based on a fourth input input from the input unit. The remote control system according to any one of claims 3 to 7, wherein the control unit zooms in or zooms out the first image based on a fifth input input from the input unit. The remote control system according to any one of claims 1 to 8, wherein the first image is cut out on the moving body side and transmitted from the moving body to the operating terminal. Any one of claims 1 to 9, wherein the control unit displays, in addition to the first image, a second image on the display unit that includes an area in a movement target direction of the moving body and an area different from the first image. Remote control system described in one. The remote control system according to claim 10, wherein the control unit detects a person and/or an object in the area shown in the second image. The remote control system according to claim 10 or 11, wherein the second image is cut out on the moving object side and transmitted to the operating terminal. A remote control program for remotely controlling a mobile body having an imaging unit that captures images in all directions from an operation terminal connected by wireless communication, the program comprising:
displaying a first image of at least a part of the omnidirectional area imaged by the imaging unit and corresponding to the input line-of-sight direction;
Displaying a direction of the first image in the omnidirectional area, a first mark indicating the traveling direction of the moving body, and a second mark indicating the target direction of movement of the moving body, superimposed on the first image ( b) and
A remote control program that allows a computer to execute The operation terminal has a sensor that detects acceleration and angular velocity,
The remote control program according to claim 13, further comprising the step (c) of controlling movement of the mobile body based on the output from the sensor. 15. The remote control program according to claim 14, wherein said step (c) controls movement of said moving body based on an input for moving said moving body and acceleration and angular velocity detected by said sensor. 16. The remote control program according to claim 15, wherein the step (c) moves the line of sight based on a first input and an acceleration and an angular velocity detected by the sensor after the first input. The remote control program according to any one of claims 14 to 16, wherein the step (c) moves the second mark to the center of the screen where the first image is displayed based on a second input. . The remote control program according to any one of claims 14 to 17, wherein the step (c) controls the moving speed of the moving body based on a third input. The remote control program according to any one of claims 14 to 18, wherein in the step (c), the moving direction of the moving body is reversed based on a fourth input. The remote control program according to any one of claims 14 to 19, wherein said step (c) zooms in or zooms out said first image based on a fifth input. The remote control program according to any one of claims 13 to 20, further comprising a step (d) of receiving the first image cut out by the moving body from the moving body. 20. The method according to claim 13, further comprising a step (e) of displaying, in addition to the first image, a second image including an area in a movement target direction of the moving body and an area different from the first image. The remote control program described in . The remote control program according to claim 22, further comprising the step (f) of detecting a person and/or object in the area shown in the second image. The remote control program according to claim 22 or 23, further comprising a step (g) of receiving the second image cut out by the moving body from the moving body.