JP6873515B1 - Remote control system and its control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manage a moving body in a distant place with higher accuracy.
SOLUTION: The mobile body includes a moving body and a control device for remotely operating the moving body, the moving body has a photographing means, and the control device is information about the inner world and / or the outer world of the moving body. It has an information acquisition unit for acquiring a certain mobile body information and a display unit, and the mobile body information includes an image captured by the mobile body and current position information of the mobile body, and the display unit. The image and the map image are displayed on the map image, and a figure showing the current position of the moving body and a figure showing the position where the moving body has taken the image are displayed on the map image. This is solved by a remote control system characterized by the above and its control device.
[Selection diagram] Fig. 5

Description

The present invention relates to a remote control technique for a mobile body.

The following Patent Document 1 discloses a landing operation device for an unmanned aerial vehicle.

Japanese Unexamined Patent Publication No. 3-5297

The problem to be solved by the present invention is to manage a distant moving body with higher accuracy.

In order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely operating the moving body, the moving body has a photographing means, and the control device is the moving body. The mobile body information includes an information acquisition unit for acquiring mobile body information which is information about the inner world and / or the outer world of the body, and a display unit, and the mobile body information includes an image taken by the mobile body and the moving body. The current position information is included, the image and the map image are displayed on the display unit, and the figure showing the current position of the moving body and the moving body photograph the image on the map image. It is characterized in that a figure indicating a position and is displayed.

Further, it is preferable that a figure indicating the direction in which the moving body has taken the image is further displayed on the map image.

Further, it is preferable that the delay time of the image displayed on the display unit is further displayed on the display unit or the map image.

Further, in order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely operating the moving body, the moving body has a photographing means, and the control device includes a moving body. The mobile body information includes an information acquisition unit for acquiring mobile body information which is information about the inner world and / or the outer world of the mobile body, and a display unit, and the mobile body information includes the traveling direction photographed by the mobile body. The image and the current position information of the moving body are included, the image is displayed on the display unit, and a figure representing the current position of the moving body is superimposed on the image. ..

At this time, it is preferable that the moving body is an unmanned aerial vehicle and the current position information includes the flight altitude of the moving body.

Further, in order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely operating the moving body, and the control device includes the inner world and / or the outer world of the moving body. The mobile body information includes an information acquisition unit for acquiring mobile body information, a display unit, and three-dimensional map data, and the mobile body information includes the current position information of the mobile body, and the display unit. The map data is displayed on the map data, and a figure indicating the current position of the moving body is superimposed on the map data.

Further, in order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely operating the moving body, and the control device includes an inner world and / or an outer world of the moving body. The mobile body information includes an information acquisition unit for acquiring mobile body information, a display unit, and three-dimensional map data, and the mobile body information includes the current position information of the mobile body, and the display unit. Is characterized in that the state of the feature when the moving body is viewed in a specific direction from the current position on the map data is displayed.

Further, in order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely operating the moving body, and the control device includes an inner world and / or an outer world of the moving body. It has an information acquisition unit for acquiring moving body information, which is information about the moving body, a display unit, and three-dimensional map data. The moving body information includes the current position information of the moving body, and the control device. Is a third-party viewpoint display in which a figure showing the current position of the moving body is superimposed on the map data, and a feature when a specific direction is viewed from the current position of the moving body on the map data. It is characterized in that the first-person viewpoint display for displaying the state of the above can be switched to each other or can be displayed on the display unit at the same time.

At this time, it is preferable that the moving body is an unmanned aerial vehicle and the current position information includes the flight altitude of the moving body.

Further, in order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely controlling the moving body, the moving body has a photographing means, and the control device includes a moving body. The moving body has a display unit for displaying moving body information which is information about the inner world and / or the outer world of the moving body, and the moving body performs edge detection on a captured image or video, and the edge information of the image or video, It is characterized in that contour information or segment information can be transmitted to the control device.

Further, in order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely controlling the moving body, the moving body has a photographing means, and the control device includes a moving body. It has a display unit that displays moving body information that is information about the inner world and / or the outer world of the moving body, and a line-of-sight sensor that detects the line-of-sight direction of the operator. The moving body is detected by the line-of-sight sensor. Only the image of the predetermined range centered on the line-of-sight direction can be transmitted to the control device, or the resolution of the image other than the predetermined range centered on the line-of-sight sensor detected by the line-of-sight sensor is lowered to the control device. It is characterized in that it can be transmitted.

Further, the remote control system of the present invention may be in a place where the moving body cannot be seen from the position of the control device.

As described above, according to the remote control system of the present invention and the control device thereof, it is possible to manage a distant moving body with higher accuracy.

It is a schematic diagram which shows an example of the control device which concerns on embodiment. It is a block diagram which shows the functional structure of a control device. It is a block diagram which shows the functional structure of a multicopter. It is a partially enlarged view which shows an example of a camera which takes a picture of a multicopter from a third-party viewpoint. It is a schematic diagram which shows the example of the map data displayed on the monitor of a control device. This is an example of displaying an icon indicating the current position of the multicopter on the three-dimensional map data. This is an example of displaying the state of a feature when looking in a specific direction from the current position of the multicopter on three-dimensional map data. This is an example of superimposing a stereoscopic image showing the current position of the multicopter on the image taken by the multicopter. It is a graph which shows the relationship between the operation amount of a steering wheel, and the actual steering angle of a multicopter. It is a schematic diagram which shows the state in the facility of an operation center. It is a schematic diagram of a control device in which a pair of operators cooperates to manage a multicopter. It is a block diagram which shows the example of the remote monitoring system using a management device. It is a schematic diagram which shows the modification of the control device. It is a block diagram which shows the functional structure of the control device which concerns on a modification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below is an example of a remote control system S that monitors and controls a moving body, a multicopter 10, from a distance by a control device 50. The multicopter 10 of this embodiment is a small UAV (Unmanned aerial vehicle) that flies with a plurality of horizontal rotors. The multicopter 10 transmits the information acquired by the device mounted on the multicopter 10 to the control device 50, and the control device 50 remotely controls the operation of the multicopter 10 itself and the device mounted on the multicopter 10. The combination of the multicopter 10 and the control device 50 is not limited to one-to-one. One control device 50 manages a plurality of multicopters 10, and conversely, a plurality of control devices 50 manage one multicopter. It is also possible to manage 10.

[Control device]
Hereinafter, the configuration of the control device 50 will be described. FIG. 1 is a schematic view showing an example of the control device 50. FIG. 2 is a block diagram showing a functional configuration of the control device 50. FIG. 3 is a block diagram showing a functional configuration of the multicopter 10.

(Monitor system)
The cockpit 59 of the control device 50 is surrounded by a plurality of monitors 70 (display units). The monitor 70 of the present embodiment includes a front monitor 71 which is a monitor arranged in front of the driver's seat 59, side monitors 72 arranged on the left and right sides of the driver's seat 59, and a lower monitor 73 arranged on the floor surface of the control device 50. It is composed of an upper monitor 74 arranged on the ceiling surface of the control device 50 and a rear monitor 75 arranged behind the driver's seat 59.

The multicopter 10 of this embodiment is equipped with a plurality of cameras 41 (photographing means). The multicopter 10 captures front-back, left-right, and top-bottom images of the aircraft by these cameras 41. The image display program 61 of the control device 50 displays these images received from the multicopter 10 on the monitor 70 corresponding to the photographing direction. Specifically, the front monitor 71 displays the image in front of the multicopter 10 in a wide range, the side monitor 72 displays the left and right images of the multicopter 10, and the lower monitor 73 displays the image directly below or in front of the multicopter 10. The image diagonally below is displayed on the upper monitor 74, the image directly above or diagonally above the multicopter 10 is displayed on the upper monitor 74, and the image behind the multicopter 10 is displayed on the rear monitor 75. As a result, the operator can grasp the surrounding environment of the multicopter 10 as if he / she is on the multicopter 10.

Although the lower monitor 73 of this embodiment is installed on the floor surface of the control device 50, the position of the lower monitor 73 is not limited to the floor surface, and is near the operator's feet (the operator drops his / her line of sight downward). It suffices if it is placed at the viewing position). The lower monitor 73 may be arranged with the screen facing diagonally upward under the front monitor 71 so that the screen is not hidden by the operator's feet when viewing the lower monitor 73, for example. Similarly, the upper monitor 74 of the present embodiment is installed on the ceiling surface of the control device 50, but the position of the upper monitor 74 is not limited to the ceiling surface, but is arranged above the operator's head (the position where the operator looks up). I just need to be there. The upper monitor 74 may be arranged on the front monitor 71 with the screen facing diagonally downward so that, for example, the operator can check without turning his / her line of sight directly upward.

The image display program 61 of the present embodiment not only displays the images captured by each camera 41 on different monitors 70, but also displays these images side by side or superimposed on one monitor 70, or vice versa. An image having one wide angle of view can be divided and displayed on a plurality of monitors 70. For example, the images of the top, bottom, left, right, and rear of the multicopter 10 are superimposed on the edges of the front image and displayed simultaneously on the front monitor 71, or the images of the spherical camera and the hemispherical camera are flattened and divided, respectively. It can be displayed on the monitor 70 corresponding to the shooting direction of.

The number of cameras 41 mounted on the multicopter 10 and the shooting direction are arbitrary, and may be appropriately changed according to the intended use and purpose. The multicopter 10 may, for example, take a picture of only the front of the multicopter 10 or diagonally below the multicopter 10 with one camera 41, or may take a picture of a desired direction by operating one camera 41 with the camera gimbal 49. Good. Further, the camera 41 may not be mounted on the multicopter 10, and only the output value of another sensor device or the processed value thereof may be transmitted to the control device 50.

Further, the camera 41 mounted on the multicopter 10 is not limited to the FPV (First Person View) method. For example, as shown in FIG. 4, the camera 41 is mounted on the tip of the camera arm 42 extending from the body of the multicopter 10. However, it is also conceivable to photograph the multicopter 10 itself from a third-person perspective. The camera arm 42 can freely change the shooting position of the camera 41 and the shooting direction thereof like a movable fiberscope. As a result, the upper surface and the lower surface of the body portion 11 of the multicopter 10 and the surrounding environment can be widely confirmed. Further, the camera arm 42 can be extended from the body portion 11 of the multicopter 10 or retracted into the body portion 10 by a drive mechanism (not shown). By storing the camera arm 42 in the body 11 when not in use, for example, it is possible to prevent the camera arm 42 from breaking when the multicopter 10 crashes, and the camera arm 42 in an emergency Increased reliability.

In addition, for example, a photograph of the multicopter 10 photographed by a fixed camera installed on the ground or an image of the multicopter 10 photographed by another UAV may be transmitted to the control device 50. It is extremely useful to photograph the state of the actual machine from the outside (make it visible) in the non-visual control of the moving body. By displaying such an image on the monitor 70, the operator can more objectively grasp the state / situation of the multicopter 10.

The control device 50 has a communication device 52 (information acquisition unit) that receives the aircraft information of the multicopter 10 or the mobile body information i that is information about the environment in which the multicopter 10 is placed. The aircraft information of the multicopter 10 referred to here is information on the inner world of the multicopter 10 (the attitude of the aircraft, the orientation of the heading, the remaining battery level, etc.), and the information on the environment in which the multicopter 10 is placed is. Information about the outside world (such as latitude and longitude and surrounding objects) of the multicopter 10. The definitions of "inner world" and "outer world" here are for convenience only, and individual sensor values and acquired information may be classified into either of them. Any information about the state or situation of the multicopter 10 is a candidate for the mobile information i. The moving body information i of the present embodiment includes an image or video taken by the multicopter 10, an output value of the sensor device of the multicopter 10 or a processed value thereof, information indicating a positional relationship between the multicopter 10 and its peripheral objects, and information indicating the positional relationship between the multicopter 10 and its peripheral objects. , Information on the wind direction and speed at the current location of the multicopter 10.

More specifically, the multicopter 10 of this embodiment is acquired by the image taken by the camera 41, the point cloud data of the peripheral object acquired by LiDAR37 (Light Detection And Ranging), its processed value, the flight control sensor group S, and the like. Data indicating the direction and attitude of the aircraft, the flight position (latitude and pressure altitude), the remaining battery level, the wind direction and the wind speed at the current location acquired by the wind direction and wind speed sensor 36 are transmitted to the control device 50.

The mobile information i does not necessarily have to be acquired and transmitted only by the multicopter 10. The moving object information i includes, for example, information on the positional relationship between the multicopter 10 and its surrounding objects possessed by the UTM (UAV Traffic Management) system, weather information observed by other systems, and images taken by other UAVs. A photographed image of the multicopter 10 from a third-party perspective may be included.

The video display program 61 of the present embodiment displays a numerical value indicating the distance to the peripheral object closest to the multicopter 10 among the peripheral objects of the multicopter 10 on the image (see, for example, the example of FIG. 6). .. When the distance to the peripheral object approaches the danger zone, the video display program 61 calls the operator's attention by displaying the vicinity of the peripheral object in red or blinking the numerical value.

(Measures against communication delay)
FIG. 5 is a schematic view showing a display example of the map image M displayed on the monitor 70 of the control device 50. FIG. 6 is an example in which an icon (graphic) indicating the current position of the multicopter 10 is superimposed and displayed on the three-dimensional map data N acquired in advance. FIG. 7 is an example of displaying the state of the feature when the multicopter 10 is viewed in a specific direction from the current position on the three-dimensional map data N. FIG. 8 is an example in which a stereoscopic image showing the current position of the multicopter 10 is superimposed and displayed on the image captured by the multicopter 10.

The shooting control device 40 of the multicopter 10 operates the camera gimbal 49, starts / stops shooting by the camera 41, or starts / stops shooting by the camera 41 according to an instruction from the control device 50 or another system or according to pre-programmed conditions. The video data taken in 41 is processed.

The amount of video data captured by the multicopter 10 by the camera 41 is significantly larger than that of telemetry data indicating the current location of the multicopter 10, for example, and the delay during transfer tends to be large. That is, there is a discrepancy between the surrounding environment of the multicopter 10 grasped from the image displayed on the monitor 70 and the actual surrounding environment of the multicopter 10 at the time of confirming the image. Therefore, when the operator manually operates the multicopter 10 while watching the image, helicopter needs to be aware of how much the image he is viewing is displayed with a delay.

The photographing control device 40 of the present embodiment performs edge detection on the image captured by the camera 41 by, for example, a high-pass filter, transmits only the extracted edge information to the control device 50, and further performs object detection and segmentation to perform object detection. Only the contour information and the segment information can be transmitted to the control device 50. As a result, the amount of data transmitted from the multicopter 10 to the control device 50 is suppressed, and the time lag until the information from the camera 41 is displayed on the monitor 70 is reduced. Further, when the shooting control device 40 of the present embodiment has a distance to the multicopter 10 closer than a predetermined threshold value among the peripheral objects detected from the image, only the image information related to the peripheral object is obtained. Can be sent in preference to. As a result, the peripheral objects to be watched are promptly displayed on the monitor 70, and the operator can have a longer grace period to deal with them.

As shown in FIG. 5, in the map image M, the icon 10a of the current position of the multicopter 10 identified from the telemetry data of the multicopter 10 and the image displayed on the monitor 70 from which position to which direction. An icon 10b indicating whether the image was taken toward the subject is displayed. In addition, a delay time (300 msec in the example of this embodiment) is also displayed in the vicinity of the icon 10b. The shooting point and delay time of the icon 10b can be specified, for example, by combining the time stamp or latitude and longitude information of the previously received NMEA sentence (GPS data) with the time stamp of the RTP header (video data). Information on the shooting direction may be included in the video data or the telemetry data.

Further, as shown in FIG. 6, the video display program 61 of the present embodiment can display the icon 10a indicating the current position of the multicopter 10 on the three-dimensional map data N by superimposing the icon 10a. In the example of FIG. 6, the operator can check the flight status of the multicopter 10 from all directions from a third-party viewpoint. For example, in applications such as transporting goods between distribution centers, where the same air route is repeatedly flown to the multicopter 10, the topography, features, and altitude of the air route are acquired together with the latitude and longitude information, and the features are also acquired. It is possible to prepare the three-dimensional map data N of the air route in advance by superimposing the external photographs of the air routes. If the accuracy is sufficient, commercially available three-dimensional map data may be used. By virtually reproducing the flight status on the control device 50 side based on the current position information (latitude and pressure altitude) of the multicopter 10, it is possible to grasp the flight status of the multicopter 10 with a small amount of data communication. it can. This also makes it possible to reduce the number of peripheral monitoring devices mounted on the multicopter 10.

In addition to this, the image display program 61 of this embodiment shows the state of the feature when viewed in a specific direction from the current position (latitude and pressure altitude) of the multicopter 10 in the map data N, as shown in FIG. It can also be reproduced virtually. The "specific direction" referred to here may be arbitrarily set and changed by the operator. As a result, the pilot can check the surrounding environment of the multicopter 10 from the first-person viewpoint as if he / she is on the multicopter 10. The map data N may be lightweight data showing only the outline of the feature as in the examples of FIGS. 6 and 7, and a more realistic view may be reproduced by using a photographic image. Alternatively, these may be switchable.

In this embodiment, the operator can arbitrarily switch between the display of the third-person viewpoint and the display of the first-person viewpoint by the video display program 61, and can also display these displays on the monitor 70 at the same time.

Further, as shown in FIG. 8, the image display program 61 of the present embodiment displays an image of the traveling direction taken by the multicopter 10 on the monitor 70, and the multicopter specified from the telemetry data of the multicopter 10 there. It is also possible to superimpose and display a stereoscopic image showing the current positions of 10. When the image of the traveling direction taken by the multicopter 10 is displayed on the monitor 70, the image showing the current position of the multicopter 10 as seen from the shooting position of the image is three-dimensionally superimposed and displayed for maneuvering. The person can grasp the current position of the multicopter 10 without misunderstanding even from the delayed image.

For example, when the multicopter 10 is shooting an image in the traveling direction and the delay of the image data is not so large, the above-mentioned AR (Augmented Reality) method (the current position of the multicopter 10 in the image in the traveling direction with a time lag). The delay is dealt with by (a method of superimposing stereoscopic images showing), and when there is no image in the direction of travel, when the delay is large, or at night, the above-mentioned VR (Virtual Reality) method (using map data acquired in advance) is used. The method of virtually reproducing the flight environment of the multicopter 10) may be switched. The type of information displayed on the monitor 70 by these AR-like method or VR-like method is not particularly limited, and various sensor information of the multicopter 10 and various sensor information prepared in advance according to the use of the multicopter 10 and the information to be watched at each time. All you have to do is to additionally display the saved information.

(Steering system)
The control device 50 has a handle 53, which is a control stick, and an elevating lever 54. The control device 50 of the present embodiment controls the pitch and roll of the multicopter 10 by tilting the handle 53, and controls the yaw of the multicopter 10 by twisting the handle 53. The operation of the handle 53 and the elevating lever 54 by the operator is transmitted to the multicopter 10 by the control signal transmission program 63. The interface between the control device 50 and the operator is not limited to the handle 53 and the elevating lever 54 of this embodiment, and if the throttle, aileron, rudder, and elevator rudders can be steered to the multicopter 10, for example, for a radio control. It may be something like a radio or a controller for a home video game console. The control device 50 is also provided with a touch panel, a keyboard, a mouse, and the like, which are general input devices, and commands can be sent to the multicopter 10 from these. ..

Further, the control device 50 of the present embodiment has a resistance control program 62 that controls the amount of resistance when the operator operates the handle 53 and the elevating lever 54. The resistance control program 62 changes the amount of resistance when operating the handle 53 and the elevating lever 54 in each direction according to the weight of the multicopter 10 and the wind direction and speed at the current location of the multicopter 10. For example, when a strong wind is blowing from the front right of the multicopter 10 toward the multicopter 10, the operation of moving the operator to the front right is heavy, and the operation of moving the operator to the rear left is light. As a result, the operator P can intuitively grasp the properties of the multicopter 10 and the flight environment from the resistance amount of the handle 53 and the elevating lever 54.

FIG. 9 is a graph showing the relationship between the operating amount (steering amount) of the steering wheel 53 and the actual steering angle of the multicopter 10. When the image captured by the camera 41 of the multicopter 10 is delayed and displayed on the monitor 70, there is a time lag until the response is obtained even if the operator operates the handle 53 or the like. Especially inexperienced pilots may steer more deeply than necessary due to delayed response. The control signal transmission program 63 of the control device 50 desensitizes the vicinity of the rudder neutral from the original relationship between the steering amount and the actual rudder angle (P in FIG. 9) as the delay of the video data increases (EXP in FIG. 9). ), And narrow the range of the actual rudder angle (DR in FIG. 9). This reduces excessive steering due to video delay.

In addition, the control signal transmission program 63 may narrow only the range of the actual steering angle according to the delay amount of the video data while maintaining the direct proportional relationship between the steering amount and the actual steering angle. That is, only the upper limit of the flight speed may be changed according to the amount of delay of the video data. The steering amount and actual steering angle adjustment function by the control signal transmission program 63 is only a function at the time of manual steering, and may be automatically disabled at the time of autopilot.

(Control sharing function)
FIG. 10 is a schematic view showing a state of the operation center C in which a large number of control devices 50 are gathered at one base. FIG. 11 is a schematic view of a control device 50 in which a pair of two operators cooperate to manage the multicopter 10.

The control device 50 has a control switching program 66 which is a means (control switching means) for switching the control subject of the multicopter 10 and the equipment mounted on the multicopter 10 to another control device 50. The control switching program 66 is a function that entrusts the control of the multicopter 10 to an operator who can deal with an abnormality in the operation of the multicopter 10 or when a difficult maneuvering operation is required. .. Alternatively, this can be used to give the user control (control right) of the multicopter 10 and the equipment mounted on the multicopter 10, or simply to take over the work to another pilot.

The "control subject" here means, for example, the control device 50 or other control device 50 to which the authority to operate the multicopter 10, the authority to operate the equipment mounted on the multicopter 10, the authority to acquire and display the moving body information i, etc. belong. It means a device. The control switching program 66 can switch these authorities collectively or partially to another control device 50 or other device. The control switching program 66 can be manually executed, but can also be automatically executed as described below.

For example, when a lower-level pilot who is training and a higher-level pilot who is instructing the trainer use a set of control devices 50 as shown in FIG. 11 to control the multicopter 10, the monitors 70 of both control devices 50 are used. The moving body information i of the multicopter 10 is displayed in the above, and when some maneuvering operation is performed by the senior pilot, the control subject is forcibly switched to the maneuvering device 50 of the senior pilot by the control switching program 66. Is possible. The control switching program 66 used at this time may be that of the control device 50 of the senior operator, that of the control device 50 of the lower level operator, or that the control switching program 66 of both of these control devices 50 is performed in cooperation with each other. You may. Alternatively, as will be described later, the control switching program 66 included in the management device 80, which is a higher-level device of the control device 50, may be used. Similarly, for example, when a local personnel is assigned to support the operation of the multicopter 10, the operation from the control device 50 is invalidated while the local personnel directly operate the multicopter 10 using a radio or the like. (The control body of the multicopter 10 is switched from the control device 50 to a device such as a local radio). The implementation form of the control switching means is not limited to software (program), and may be configured by hardware.

Further, when the control switching program 66 is automatically executed and there are a plurality of candidates for the control device 50 of the switching destination, the control switching program 66 automatically selects the switching destination according to the content of the moving body information i. More specifically, the control switching program 66 classifies the proficiency level and the ex officio of the operator in advance, and determines the type of abnormality determined from the moving object information i, the required determination and the difficulty level of the operation, or It automatically determines which control device 50, that is, which operator to switch the control subject to, according to the degree of urgency or the like.

For example, when a mechanical abnormality is detected from the multicopter 10, it is necessary to immediately secure safety in preparation for a crash, so it is necessary to switch the control subject to an operator having advanced maneuvering skills and decision-making power. Also, for example, when a large number of multicopters 10 for transportation and delivery are congested on the flight course, basically only paying attention to the intervals and flight speeds of these multicopters 10, so an inexperienced pilot But it can be dealt with. However, if there is concern about battery exhaustion, it is necessary to switch to a pilot who can make decisions about route changes and crash landings. Also, for example, if the propeller is partially damaged, if the lift is sufficient, it is basically just watching, but if the operation of the multicopter 10 becomes difficult, switch to a pilot who can judge an emergency landing. There is a need. If a part of the propeller is damaged and the lift is insufficient, the aircraft of the multicopter 10 will gradually descend, but at this time, the rotation range of the remaining propeller is close to the limit, and skillful maneuvering skills are required. Required. Further, for example, when an abnormality occurs in the operation of the GPS receiver 32, the multicopter 10 may malfunction depending on the content of the abnormality. Even an inexperienced pilot can deal with the decrease in the number of satellites captured by the GPS receiver 32, but if the measurement accuracy is significantly reduced and the autonomous operation is hindered, the operator can be manually operated. You need to switch. Further, for example, when an abnormality occurs in the communication between the control device 50 and the multicopter 10, the multicopter 10 can be steered even if the image from the multicopter 10 cannot be displayed, and other telemetry data is received. It is not so urgent if it is possible. However, if it is not a temporary communication failure due to the flight environment but a failure of the communication equipment, the urgency is high. In addition, if the place where the abnormality occurred is a densely populated area or if the aircraft was flying at low altitude at the time of failure, more advanced judgment and higher maneuvering skills are required. The control switching program 66 can automatically switch a part or all of the control main body of the multicopter 10 to the operator (control device 50) who can deal with such a situation / condition.

The control switching program 66 of the present embodiment not only switches the control subject such as the multicopter 10 to another control device 50 or other device, but also uses it for communication with the multicopter 10 according to the content of the mobile information i. You can also switch lines automatically. For example, it is conceivable to use a best effort type communication line during normal flight of the multicopter 10 and switch to a guarantee type communication line with guaranteed Quality of Service (QoS) in an emergency. The communication line may be switched by a program different from the control switching program 66.

As described above, the target for switching the control subject by the control switching program 66 is not limited to the other control device 50, and the control subject can be switched to another device other than the control device 50. For example, a user of a service using the remote control system S, that is, a general personal computer or tablet terminal used by the user, is given control of the camera 41 or the camera gimbal 49, and the user is made to take a desired image / video. And so on. Further, for example, a camera 41 for the operator to operate the multicopter 10 and a camera for the user to capture a desired image / video are separately mounted, and the communication path of the camera for the user is set as the control device 50. By providing a communication path with the multicopter 10 separately and making the communication path and protocol of the camera for the user more accessible (for example, enabling access with a web browser), the user can control the control device 50. It is possible to easily access the camera 41 or the like even if the user is away from the camera 41. In addition, by sharing the image taken by the user's camera with the operator and further sharing the current position information of the multicopter 10 with the user, the operator and the user can perform the shooting work in smooth cooperation. You can also. The device that can give control to the user is not limited to the camera, and the user can be given control of any device according to the purpose.

As shown in FIG. 10, a large number of control devices 50 are installed in the operation center C. In this case, for example, as shown in FIG. 12, the control switching program 66 may be mounted on the management device 80, which is a higher-level device of the control device 50, and the control device 80 may centrally switch the control subject. In the example of FIG. 12, the user (other device 90) of the service using the remote control system S is given control of the camera 41, and the image captured by the camera 41 is displayed on the monitor 91 of the other device 90. Has been done. Even in this case, the essential functions and roles of the control switching program 66 remain unchanged. The difference is simply whether it is mounted on the individual control device 50 or independent of the control device 50. Even if the control switching programs 66 are mounted on the individual control devices 50, they may cooperate with each other, or the control switching program 66 of one control device 50 may be used as the master, and the control switching programs 66 of the other control devices 50 may be used as masters. By making the slave a master-slave relationship, it is possible to do the same thing as the management device 80 does.

In addition, the operation center C comprehensively covers information on a plurality of or all multicopters 10 under the jurisdiction of the operation center C, and a part or the whole of the airspace under the jurisdiction of the operation center C by separating colors and layers. It has a central monitoring screen 79 for comprehensive display. In Operation Center C, in some cases, it is possible to take over the drone of another company that is flying dangerously and avoid the danger.

Further, for example, it is conceivable to display the captured image and telemetry data of the multicopter 10 in which a serious problem has occurred on the central monitoring screen 79, and to deal with the problem by a team of managers or a plurality of pilots. .. In addition, for example, information on a plurality of multicopters 10 for which the situation should be watched is displayed side by side on the central monitoring screen 79, or these information are switched and displayed at predetermined seconds, and the situation is changed by a plurality of people. It is also possible to monitor.

(Camera fixing function)
The multicopter 10 of the present embodiment has a camera gimbal 49 (posture stabilizing device) that absorbs a change in the posture of the multicopter 10 and maintains the posture of the camera 41. Then, the control device 50 has a camera fixing button 531 (camera fixing means) that stops the posture stabilizing operation of the camera gimbal 49 and fixes the direction of the camera 41. By intentionally fixing the orientation of the camera 41, it is possible to actually grasp the actual tilt and vibration of the aircraft, and for example, the multicopter 10 can be landed more safely.

The multicopter 10 of the present embodiment has a plurality of cameras 41 oriented in multiple directions. For example, when the multicopter 10 includes only one camera 41, the camera fixing button 531 presses the camera 41. The orientation of the camera 41 may be fixed after automatically pointing forward or downward of the multicopter 10.

[Multicopter]
Hereinafter, the functional configuration of the multicopter 10 will be described with reference to FIG. The multicopter 10 of this embodiment is stored in a so-called drone port, which is a dedicated takeoff and landing facility located all over the country, and automatically departs from the drone port according to an instruction from the control device 50 or another system. In addition, such a drone port is not indispensable. For example, personnel are assigned to the site to prepare for takeoff and landing of the multicopter 10, and personnel assigned to the site cooperate with the operator to provide support by manual operation. You may go.

The multicopter 10 of this embodiment mainly includes a flight controller FC which is a control unit, a rotor R which is a plurality of horizontal rotors, a communication device 25 which communicates with an operator (control device 50), and a plurality of cameras 41. It is composed of a camera gimbal 49 that supports all or a part of these, and a battery (not shown) that supplies power to them.

The flight controller FC has a control device 20 which is a microcontroller. The control device 20 has a CPU 21 which is a central processing unit, and a memory 22 including a storage device such as a RAM, a ROM, and a flash memory. The control device 20 is not limited to a single microcontroller, and may be a combination with a so-called companion computer. In addition, it is also conceivable to configure the control device 20 with, for example, an FPGA (field-programmable gate array) or an ASIC (Application Specific Integrated Circuit).

The flight controller FC further has a flight control sensor group S including an IMU 31 (Inertial Measurement Unit), a GPS receiver 32, a pressure sensor 34, and an electronic compass 35, which are connected to the control device 20. Has been done.

The IMU 31 is a sensor that detects the inclination of the multicopter 10, and is mainly composed of a 3-axis acceleration sensor and a 3-axis angular velocity sensor. The barometric pressure sensor 34 is an altitude sensor that obtains the altitude above sea level (elevation) of the multicopter 10 from the detected barometric pressure value. The electronic compass 35 is a sensor that detects the azimuth angle of the heading (nose) of the flight controller FC, and is mainly composed of a three-axis geomagnetic sensor. The GPS receiver 32 is, to be exact, a receiver of a navigation satellite system (NSS). The GPS receiver 32 acquires the current latitude and longitude value from the Global Navigation Satellite System (GNSS) or the Regional Navigational Satellite System (RNSS).

The flight control sensor group S acquires the position information of the aircraft including the latitude and longitude, altitude, and azimuth of the nose in addition to the tilt and rotation of the aircraft by the flight control sensor group S.

The control device 20 has a flight control program FS, which is a program for controlling the attitude and basic flight operations of the multicopter 10 during flight. The flight control program FS adjusts the rotation speed of each rotor R based on the information acquired from the flight control sensor group S, and flies the multicopter 10 while correcting the disturbance of the attitude and position of the aircraft.

The control device 20 further has an autonomous flight program AP, which is a program for autonomously flying the multicopter 10. Then, in the memory 22 of the control device 20, a flight plan FP, which is a parameter in which the latitude and longitude of the course on which the multicopter 10 is flown, the altitude and speed during flight, and the like are specified, is registered. The autonomous flight program AP autonomously flies the multicopter 10 according to the flight plan FP, with instructions from the control device 50 and other systems, a predetermined time, and the like as starting conditions.

In addition, the multicopter 10 of the present embodiment includes a wind direction and wind speed sensor 36 that acquires the wind direction and speed at the current location, and a LiDAR 37 that acquires the positional relationship and distance between the multicopter 10 and its surrounding objects. In the multicopter 10 of this embodiment, the point cloud data acquired by the LiDAR 37 is mainly used in the internal processing of the flight controller FC, and if necessary, the processed values such as the distance to the surrounding objects are sent to the control device 50 together with the video data. Will be sent. In addition, for example, it is conceivable that the point cloud data acquired by LiDAR 37 is appropriately thinned out and transferred to the control device 50 in real time. As a result, the operator can obtain information on surrounding objects more quickly than waiting for the transfer of the image captured by the camera 41.

In addition, it is also significant to separately mount a sound collector such as a microphone array to collect the environmental sound around the multicopter 10 in order to grasp the flight state and the flight environment of the multicopter 10. In addition to this, by providing the control device 50 with a microphone and the multicopter 10 with a speaker, the operator can talk with a person in the vicinity via the multicopter 10. Further, the camera 41 mounted on the multicopter 10 is not limited to a general visible light camera, and may be, for example, a night-vision camera (high-sensitivity camera or infrared camera), a depth camera, or the like. In addition, the devices and equipment mounted on the multicopter 10 are not particularly limited, and for example, a parachute, a flashlight, a warning light, an alarm buzzer, or the like may be mounted.

The communication device 25 and the control device 50 of the multicopter 10 are not limited in their specific communication method, protocol, and line used as long as they can transmit and receive control signals and data. For example, mobile communication networks such as 3G, LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), and 5G (hereinafter, mobile communication networks using such terrestrial base stations are referred to as "mobile phone communication networks". By connecting the control device 50 and the multicopter 10 with (), the operator (control device 50) can access the multicopter 10 from anywhere within the service area of the mobile telephone communication network. It becomes. Alternatively, it is also conceivable to appropriately install an access point on the flight path of the multicopter 10 and connect the access point and the control device 50 by an optical line. The connection method between the access point and the multicopter 10 is not particularly limited, and for example, Wi-Fi (Wireless Fidelity) or frequency hopping type PCM (pulse code modulation) signal may be used, and mobile phone communication is performed from the access point. You may connect to the multicopter 10 using a net. Further, the communication path between the multicopter 10 and the control device 50 may be made redundant by using a plurality of communication methods in combination. In addition, for example, it is conceivable to separately prepare another UAV equipped with a repeater and connect the multicopter 10 to the control device 50 or the access point via the UAV.

Here, when the communication delay from the multicopter 10 to the control device 50 becomes a problem, for example, at least a part of the communication paths connecting them is switched between the analog system and the digital system. It is possible to make it possible. Analog communication is generally inferior in data reproducibility to digital communication, but analog communication has the advantage of less delay than digital communication. The communication method may be switched manually or automatically on the control device 50 side, or automatically on the multicopter 10 side. As the conditions for automatically switching, for example, the amount of data to be transmitted / received, the amount of delay, the communication failure, and other arbitrary conditions can be considered.

Further, when the moving range of the multicopter 10 extends beyond the service area of the mobile phone communication network such as offshore or mountainous areas, for example, satellite communication modules such as Iridium (registered trademark) and Inmarsat (registered trademark) are multi-multirotored. By mounting it on the copter 10, communication with the multicopter 10 can be maintained even outside the service area of the mobile phone communication network. Here, by making it possible to switch the means for connecting to the multicopter 10 between the satellite communication line and another communication line, the communication path with the multicopter 10 can be made redundant, and the multicopter 10 and the multicopter 10 can be made redundant. The movable range of the multicopter 10 can be expanded while maintaining the communication quality of the multicopter 10 as much as possible. For example, when the multicopter 10 and the control device 50 are connected by a 5G line, all the information including the image captured by the camera 41 is transmitted to the control device 50, and after switching to the satellite communication line, the latitude and longitude and the pressure altitude It is conceivable to transmit only the telemetry data of the above, or to transmit only the edge information, contour information, and segment information of the object as the video data to be transmitted after switching to the satellite communication line. The communication line may be switched manually or automatically on the control device 50 side, or automatically on the multicopter 10 side. As the conditions for automatically switching, for example, the current position (latitude and pressure altitude), the radio field strength of another communication line such as a mobile phone communication network, and other arbitrary conditions can be considered. In addition, it is conceivable to always use both satellite communication lines and other communication lines for communication.

[Modification example of control device]
FIG. 13 is a schematic view showing a control device 50b which is a modification of the control device 50. FIG. 14 is a block diagram showing a functional configuration of the control device 50b. Hereinafter, the control device 50b will be described with reference to FIGS. 13 and 14.

In the control device 50b, the operator enters the spherical monitor 76, which is a spherical monitor, and controls the multicopter 10. In this modification, the multicopter 10 includes a hemispherical camera and a spherical camera as its camera 41, and the image captured by the camera 41 is displayed as it is on the spherical monitor 76. By adopting the spherical camera and the spherical monitor 76 in this modification, the gaps and overlaps of the images when shooting in all directions with a plurality of cameras, the load of converting the flat image and the spherical image are reduced, and the video data is made efficient. It is possible to acquire, transfer, and display. The spherical monitor 76 is not limited to the global shape as in the present embodiment, and may be a hemispherical shape.

The control device 50b further has a line-of-sight sensor 55 capable of detecting the line-of-sight direction of the operator. The image display program 61 enlarges and displays the image in the line-of-sight direction of the operator detected by the line-of-sight sensor 55. Further, the output value of the line-of-sight sensor 55 is also transferred to the multicopter 10. The photographing control device 40 of the multicopter 10 receives this information, and transmits the image data in the direction of the operator's line of sight with high resolution and the image data with low resolution in the other directions to the control device 50b. As a result, the amount of communication of the video data transmitted from the multicopter 10 is suppressed, and the delay of the video displayed on the monitor 76 is reduced. In addition, only the image in the direction of the pilot's line of sight and its surroundings may be transmitted, and transmission may be omitted for images in other ranges. The direction of the operator's line of sight is not limited to the direction of the eyeball, and may be the direction of the face.

(Speaker system)
Further, the control device 50b has a plurality of speakers 56 arranged around the operator. The speaker 56 of this embodiment is arranged so as to surround the front, rear, left, and right of the operator in the circumferential direction.

The control device 50b has a speaker control program 64 (speaker control unit) that controls the operation of the speaker 56. The speaker control program 64 notifies the operator of the positional relationship and distance between the multicopter 10 and its peripheral objects by emitting a sound from the speaker 56 to the operator based on the content of the moving body information i. For example, when there is an obstacle on the right front side of the multicopter 10, a sound is emitted from the speaker 56 on the right front side of the operator, and the volume is increased as the obstacle approaches. This allows the operator to visually recognize the position of the obstacle and its distance.

The arrangement of the speakers 56 is not limited to that of the present embodiment. For example, three or four speakers 56 including the left and right speakers 56 are arranged around the operator, and the sound source direction is simulated by phase control. Is also possible. Further, for example, it is conceivable to change the type of sound according to the type of peripheral objects (structures such as buildings, other UAVs, people). The use of the speaker 56 is not limited to that of the present embodiment, and for example, it is possible to express the wind direction and the wind speed at the current location of the multicopter 10 by the sound emitted from the speaker 56.

(Blower system)
Further, the control device 50b includes a blower device 57 that is arranged around the operator and blows wind to the operator, and a blower control program 65 (blower control unit) that controls the wind direction and volume by the blower device 57. doing. The blow control program 65 changes the wind direction and volume of the wind blown to the operator according to the wind direction and speed at the current location of the multicopter 10. For example, when the wind is blowing from the right front of the multicopter 10 toward the multicopter 10, the air volume corresponding to the wind speed is blown to the operator from the blower 57 on the right front of the operator. As a result, the operator can operate the multicopter 10 as if he were at the site while being far away.

Although the embodiments of the present invention have been described above, the scope of the present invention is not limited to this, and various modifications can be made without departing from the gist of the invention. For example, the moving object of the present invention is not limited to an unmanned aerial vehicle, and may be a so-called passenger drone that transports passengers or a manned aircraft on which a pilot is on board. Further, the present invention is not limited to an aircraft, and may be, for example, a vehicle or a ship as long as it is a maneuverable mobile body.

S: Remote control system, C: Operation center, 10: Multicopter (moving body, unmanned aircraft), 11: Body, R: Rotor, FC: Flight controller, FS: Flight control program, AP: Autonomous flight program, FP : Flight plan, 25: Communication device, S: Flight control sensor group, 31: IMU, 32: GPS receiver, 34: Pressure sensor, 35: Electronic compass, 36: Wind direction and speed sensor, 37: LiDAR, 40: Shooting control Device, 41: Camera (shooting means), 42: Camera arm, 49: Camera gimbal (posture stabilizer), 50: Control device, 51: Control unit, 511: CPU, 512: Memory, 52: Communication device ( Information acquisition unit), 53: Handle, 531: Camera fixing button (camera fixing means), 54: Lifting lever, 55: Line-of-sight sensor, 56: Speaker, 57: Blower, 59: Driver's seat, 61: Video display program, 62: Resistance control program (resistance control unit), 63: Control signal transmission program, 64: Speaker control program (speaker control unit), 65: Blower control program, 66: Control switching program (control switching means), 70: Monitor ( Display), 71: Front monitor, 72: Side monitor, 73: Lower monitor, 74: Upper monitor, 75: Rear monitor, 76: Spherical monitor, 79: Central monitoring screen, 80: Management device, 90: Other Device, i: moving object information, M: map image, N: map data

Claims (4)

With a mobile body
A control device for remotely controlling the moving body is provided.
The moving body has a means of photographing and
The control device
An information acquisition unit that acquires mobile information, which is information about the inner world and / or the outer world of the mobile body, and
Has a display and
The moving body information includes an image which is a moving image taken by the moving body and current position information of the moving body.
The image of the moving body being photographed and a map image are displayed on the display unit.
The map image is characterized in that a figure showing the current position of the moving body and a figure showing the position where the moving body has captured the image displayed on the display unit are displayed. Remote control system. The remote control system according to claim 1, further comprising displaying a figure indicating the direction in which the moving body has taken the image on the map image. The remote according to claim 1 or 2, wherein a delay time, which is a time lag between the shooting of the image and the display on the display unit, is further displayed on the display unit or the map image. Maneuvering system. The control device according to any one of claims 1 to 3.

Images