JP2021179956A - Remote operation system and operation device thereof - Google Patents

Abstract

To manage a remote mobile body more accurately.SOLUTION: A remote operation system includes: a mobile body; and an operation device for remotely operating the mobile body, the mobile body having imaging means, the operation device having a display unit for displaying mobile body information as information on an inner world and/or an outer world of the mobile body, the mobile body detecting an edge for a taken image or picture and being capable of sending edge information, outline information, and segment information of an image or a picture to the operation device. There is also disclosed an operation device of the remote operation system.SELECTED DRAWING: Figure 1

Description

The present invention relates to a remote control technique for a mobile body.

The following Patent Document 1 discloses a landing operation device for an unmanned aerial vehicle.

Japanese Unexamined Patent Publication No. 3-5297

The problem to be solved by the present invention is to manage a distant moving object with higher accuracy.

In order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely controlling the moving body, the moving body has a photographing means, and the control device is the moving body. It has a display unit that displays moving body information that is information about the inner world and / or the outer world of the body, and the moving body performs edge detection on a captured image or video, and edge information and contour information of the image or video. Alternatively, the segment information can be transmitted to the control device.

Further, in the remote control system of the present invention, the moving body may be in a place invisible from the position of the control device.

As described above, according to the remote control system of the present invention and the control device thereof, it is possible to manage a distant moving object with higher accuracy.

It is a schematic diagram which shows an example of the control device which concerns on embodiment. It is a block diagram which shows the functional composition of a control device. It is a block diagram which shows the functional structure of a multicopter. It is a partially enlarged view which shows the example of the camera which takes a picture of a multicopter from a third-party viewpoint. It is a schematic diagram which shows the example of the map data displayed on the monitor of a control device. This is an example of displaying an icon indicating the current position of the multicopter on the three-dimensional map data. This is an example of displaying the state of a feature when looking in a specific direction from the current position of the multicopter on three-dimensional map data. This is an example of superimposing a stereoscopic image showing the current position of the multicopter on the image taken by the multicopter. It is a graph which shows the relationship between the operation amount of a steering wheel, and the actual steering angle of a multicopter. It is a schematic diagram which shows the state in the facility of an operation center. It is a schematic diagram of a control device in which a pair of operators cooperates to manage a multicopter. It is a block diagram which shows the example of the remote monitoring system using a management device. It is a schematic diagram which shows the modification of the control device. It is a block diagram which shows the functional structure of the control device which concerns on a modification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below is an example of a remote control system S that monitors and controls a mobile multicopter 10 from a distance by a control device 50. The multicopter 10 of this embodiment is a small UAV (Unmanned aerial vehicle) that flies with a plurality of horizontally rotating blades. The multicopter 10 transmits the information acquired by the device mounted on the multicopter 10 to the control device 50, and the control device 50 remotely controls the operation of the multicopter 10 itself and the device mounted on the multicopter 10. The combination of the multicopter 10 and the control device 50 is not limited to one-to-one. One control device 50 manages a plurality of multicopters 10, and conversely, a plurality of control devices 50 manage one multicopter. It is also possible to manage 10.

[Control device]
Hereinafter, the configuration of the control device 50 will be described. FIG. 1 is a schematic diagram showing an example of the control device 50. FIG. 2 is a block diagram showing a functional configuration of the control device 50. FIG. 3 is a block diagram showing a functional configuration of the multicopter 10.

(Monitor system)
The cockpit 59 of the control device 50 is surrounded by a plurality of monitors 70 (display units). The monitor 70 of the present embodiment includes a front monitor 71 which is a monitor arranged in front of the driver's seat 59, side monitors 72 arranged on the left and right of the driver's seat 59, and a lower monitor 73 arranged on the floor of the control device 50. It is composed of an upper monitor 74 arranged on the ceiling surface of the control device 50 and a rear monitor 75 arranged behind the driver's seat 59.

The multicopter 10 of this embodiment is equipped with a plurality of cameras 41 (shooting means). The multicopter 10 captures front-back, left-right, and top-bottom images of the machine body by these cameras 41. The image display program 61 of the control device 50 displays these images received from the multicopter 10 on the monitor 70 corresponding to the shooting direction. Specifically, the front monitor 71 displays the image in front of the multicopter 10 in a wide range, the side monitor 72 displays the left and right images of the multicopter 10, and the lower monitor 73 displays the image directly below or in front of the multicopter 10. The image diagonally below is displayed on the upper monitor 74, the image directly above or diagonally above the multicopter 10 is displayed on the upper monitor 74, and the image behind the multicopter 10 is displayed on the rear monitor 75. As a result, the operator can grasp the surrounding environment of the multicopter 10 as if he / she is on the multicopter 10.

Although the lower monitor 73 of this embodiment is installed on the floor surface of the control device 50, the position of the lower monitor 73 is not limited to the floor surface, and is near the operator's feet (the operator drops his / her line of sight down). It suffices if it is placed at the viewing position). The lower monitor 73 may be arranged with the screen facing diagonally upward under the front monitor 71 so that the screen is not hidden by the operator's feet when viewing the lower monitor 73, for example. Similarly, the upper monitor 74 of this embodiment is installed on the ceiling surface of the control device 50, but the position of the upper monitor 74 is not limited to the ceiling surface, and is arranged above the operator's head (the position where the operator looks up). You just have to. The upper monitor 74 may be arranged on the front monitor 71 with the screen facing diagonally downward so that the operator can check the monitor without turning the line of sight directly upward, for example.

The image display program 61 of the present embodiment not only displays the images taken by each camera 41 on different monitors 70, but also displays these images side by side or on top of each other on one monitor 70, or vice versa. A wide angle of view image can be divided and displayed on a plurality of monitors 70. For example, the images of the top, bottom, left, right, and rear of the multicopter 10 are superimposed on the edges of the front image and displayed simultaneously on the front monitor 71, or the images of the spherical camera and the hemispherical camera are flattened and divided, respectively. It can be displayed on the monitor 70 corresponding to the shooting direction of.

The number of cameras 41 mounted on the multicopter 10 and the shooting direction are arbitrary, and may be appropriately changed according to the intended use and purpose. The multicopter 10 may, for example, take a picture of only the front of the multicopter 10 or diagonally below the multicopter 10 with one camera 41, or may take a picture of a desired direction by operating one camera 41 with the camera gimbal 49. good. Further, the camera 41 may not be mounted on the multicopter 10, and only the output value of another sensor device or the processed value thereof may be transmitted to the control device 50.

Further, the camera 41 mounted on the multicopter 10 is not limited to the FPV (First Person View) method. For example, as shown in FIG. 4, the camera 41 is mounted on the tip of the camera arm 42 extending from the body of the multicopter 10. However, it is also conceivable to photograph the multicopter 10 itself from a third-person perspective. The camera arm 42 can freely change the shooting position and the shooting direction of the camera 41 like a movable fiberscope. As a result, the upper surface and the lower surface of the body portion 11 of the multicopter 10 and the surrounding environment can be widely confirmed. Further, the camera arm 42 can be extended from the body portion 11 of the multicopter 10 or retracted into the body portion 10 by a drive mechanism (not shown). By storing the camera arm 42 in the body 11 when not in use, for example, it is possible to prevent the camera arm 42 from breaking when the multicopter 10 crashes, and the camera arm 42 in an emergency can be prevented from breaking. Increased reliability.

In addition, for example, the image of the multicopter 10 taken by a fixed camera installed on the ground or the image of the multicopter 10 taken by another UAV may be transmitted to the control device 50. Taking a picture of the actual machine from the outside (making it visible) is extremely useful in the out-of-sight maneuvering of a moving body. By displaying such an image on the monitor 70, the operator can more objectively grasp the state / situation of the multicopter 10.

The control device 50 has a communication device 52 (information acquisition unit) that receives the aircraft information of the multicopter 10 or the mobile body information i that is information about the environment in which the multicopter 10 is placed. The aircraft information of the multicopter 10 referred to here is information on the inner world of the multicopter 10 (the attitude of the aircraft, the orientation of the heading, the remaining battery level, etc.), and the information on the environment in which the multicopter 10 is placed is. Information about the outside world (longitudinal latitude, peripheral objects, etc.) of the multicopter 10. The definitions of "inner world" and "outer world" here are for convenience, and it does not matter which of the individual sensor values and acquired information is classified. Any information about the state or situation of the multicopter 10 is a candidate for the mobile information i. The moving body information i of the present embodiment includes an image or video image taken by the multicopter 10, an output value or a processed value of the sensor device of the multicopter 10, information indicating a positional relationship between the multicopter 10 and its peripheral objects, and information indicating the positional relationship between the multicopter 10 and its peripheral objects. , Information on the wind direction and speed at the current location of the multicopter 10.

More specifically, the multicopter 10 of this embodiment is acquired by the image taken by the camera 41, the point cloud data of the peripheral object acquired by LiDAR37 (Light Detection And Ranging), its processed value, the flight control sensor group S, and the like. Data indicating the direction and attitude of the aircraft, the flight position (latitude and pressure altitude), the remaining battery level, the wind direction and the wind speed at the current location acquired by the wind direction and wind speed sensor 36 are transmitted to the control device 50.

The mobile information i does not necessarily have to be acquired and transmitted only by the multicopter 10. The moving object information i includes, for example, information on the positional relationship between the multicopter 10 and its surrounding objects possessed by the UTM (UAV Traffic Management) system, weather information observed by other systems, and photographs taken by other UAVs. The photographed image of the multicopter 10 from the viewpoint of a third party may be included.

The video display program 61 of the present embodiment displays on the video a numerical value indicating the distance between the peripheral objects of the multicopter 10 and the peripheral objects closest to the multicopter 10 (see, for example, the example of FIG. 6). .. The video display program 61 calls the operator's attention by displaying the vicinity of the peripheral object in red or blinking the numerical value when the distance to the peripheral object approaches the dangerous area.

(Measures against communication delay)
FIG. 5 is a schematic diagram showing a display example of the map image M displayed on the monitor 70 of the control device 50. FIG. 6 is an example in which an icon (graphic) indicating the current position of the multicopter 10 is superimposed and displayed on the three-dimensional map data N acquired in advance. FIG. 7 is an example of displaying the state of the feature when the multicopter 10 is viewed in a specific direction from the current position on the three-dimensional map data N. FIG. 8 is an example in which a stereoscopic image showing the current position of the multicopter 10 is superimposed and displayed on the image taken by the multicopter 10.

The shooting control device 40 of the multicopter 10 operates the camera gimbal 49, starts / stops shooting by the camera 41, or starts / stops shooting by the camera, according to an instruction from the control device 50 or another system, or according to pre-programmed conditions. The video data taken by 41 is processed.

The amount of video data captured by the multicopter 10 by the camera 41 is significantly larger than that of telemetry data indicating the current location of the multicopter 10, for example, and the delay during transfer tends to be large. That is, there is a discrepancy between the surrounding environment of the multicopter 10 grasped from the image displayed on the monitor 70 and the actual surrounding environment of the multicopter 10 at the time of confirming the image. Therefore, when the operator manually controls the multicopter 10 while watching the image, helicopter needs to be aware of how much the image he is looking at is displayed with a delay.

The photographing control device 40 of the present embodiment performs edge detection on the image captured by the camera 41 by, for example, a high-pass filter, transmits only the extracted edge information to the control device 50, and further performs object detection and segmentation to perform object detection. Only contour information and segment information can be transmitted to the control device 50. As a result, the amount of data transmitted from the multicopter 10 to the control device 50 is suppressed, and the time lag until the information from the camera 41 is displayed on the monitor 70 is reduced. Further, when the shooting control device 40 of the present embodiment has a distance to the multicopter 10 closer than a predetermined threshold value among the peripheral objects detected from the image, only the image information related to the peripheral object is obtained. Can be sent in preference to. As a result, the peripheral objects to be watched are promptly displayed on the monitor 70, and the operator can have a longer grace period to deal with them.

As shown in FIG. 5, in the map image M, the icon 10a of the current position of the multicopter 10 identified from the telemetry data of the multicopter 10 and the image displayed on the monitor 70 from which position to which direction. An icon 10b indicating whether the image was taken toward the subject is displayed. Further, a delay time (300 msec in the example of this embodiment) is also displayed in the vicinity of the icon 10b. The shooting point and delay time of the icon 10b can be specified, for example, by combining the time stamp and latitude and longitude information of the previously received NMEA sentence (GPS data) and the time stamp of the RTP header (video data). Information regarding the shooting direction may be included in the video data or may be included in the telemetry data.

Further, as shown in FIG. 6, the video display program 61 of the present embodiment can display the icon 10a indicating the current position of the multicopter 10 on the three-dimensional map data N by superimposing the icon 10a. In the example of FIG. 6, the operator can confirm the flight status of the multicopter 10 from all directions from a third-party viewpoint. For example, in applications such as the transportation of goods between distribution centers, where the same air route is repeatedly flown to the multicopter 10, the topography, features, and altitude of the air route are acquired together with the latitude and longitude information, and the features are also acquired. It is possible to prepare the three-dimensional map data N of the air route in advance by superimposing the external photographs of the air routes. If the accuracy is sufficient, commercially available 3D map data may be used. By virtually reproducing the flight status on the control device 50 side based on the current position information (latitude and pressure altitude) of the multicopter 10, it is possible to grasp the flight status of the multicopter 10 with a small amount of data communication. can. This also makes it possible to reduce the number of peripheral monitoring devices mounted on the multicopter 10.

In addition to this, the image display program 61 of this embodiment shows the state of the feature when a specific direction is viewed from the current position (latitude and pressure altitude) of the multicopter 10 in the map data N, as shown in FIG. It can also be reproduced virtually. The "specific direction" referred to here may be arbitrarily set and changed by the operator. As a result, the operator can check the surrounding environment of the multicopter 10 from the first-person viewpoint as if he / she is on the multicopter 10. The map data N may be lightweight data showing only the outline of the feature as in the examples of FIGS. 6 and 7, and a more realistic view may be reproduced by using a photographic image. Alternatively, these may be switchable.

In this embodiment, the operator can arbitrarily switch between the display of the third-party viewpoint and the display of the first-person viewpoint by the video display program 61, and can also display these displays on the monitor 70 at the same time.

Further, as shown in FIG. 8, the image display program 61 of the present embodiment displays an image in the traveling direction taken by the multicopter 10 on the monitor 70, and the multicopter specified from the telemetry data of the multicopter 10 there. It is also possible to superimpose and display a stereoscopic image showing the current position of 10. When the image of the traveling direction taken by the multicopter 10 is displayed on the monitor 70, the image showing the current position of the multicopter 10 as seen from the shooting position of the image is three-dimensionally superimposed and displayed for maneuvering. The person can grasp the current position of the multicopter 10 without misunderstanding even from the delayed image.

For example, when the multicopter 10 is shooting an image in the traveling direction and the delay of the video data is not so large, the above-mentioned AR (Augmented Reality) method (the current position of the multicopter 10 in the image in the traveling direction with a time lag). The above-mentioned VR (Virtual Reality) method (using the map data acquired in advance) is used when there is no image in the direction of travel, when the delay is large, or at night. You may switch to the method of virtually reproducing the flight environment of the multicopter 10). The type of information displayed on the monitor 70 by these AR-like methods or VR-like methods is not particularly limited, and various sensor information of the multicopter 10 and various sensor information prepared in advance according to the use of the multicopter 10 and the information to be watched at each time. You can additionally display the information you have saved.

(Steering system)
The control device 50 has a handle 53 and an elevating lever 54, which are control sticks. The control device 50 of the present embodiment controls the pitch and roll of the multicopter 10 by tilting the handle 53, and controls the yaw of the multicopter 10 by twisting the handle 53. The operation of the handle 53 and the elevating lever 54 by the operator is transmitted to the multicopter 10 by the control signal transmission program 63. The interface between the control device 50 and the operator is not limited to the handle 53 and the elevating lever 54 of this embodiment, and if the throttle, aileron, rudder, and elevator rudders can be steered to the multicopter 10, for example, for a radio control. It may be something like a radio or a controller for a home-use game machine. The control device 50 is also provided with a touch panel, a keyboard, a mouse, and the like, which are general input devices, and commands can be sent to the multicopter 10 from these. ..

Further, the control device 50 of the present embodiment has a resistance control program 62 that controls the amount of resistance when the operator operates the handle 53 and the elevating lever 54. The resistance control program 62 changes the amount of resistance when operating the handle 53 and the elevating lever 54 in each direction according to the weight of the multicopter 10 and the wind direction and speed at the current location of the multicopter 10. For example, when a strong wind is blowing from the front right of the multicopter 10 toward the multicopter 10, the operation of moving the operator to the front right is heavy, and the operation of moving the operator to the rear left is light. As a result, the operator P can intuitively grasp the properties of the multicopter 10 and the flight environment from the resistance amount of the handle 53 and the elevating lever 54.

FIG. 9 is a graph showing the relationship between the operation amount (steering amount) of the steering wheel 53 and the actual steering angle of the multicopter 10. When the image taken by the camera 41 of the multicopter 10 is delayed and displayed on the monitor 70, there is a time lag until the response is obtained even if the operator operates the handle 53 or the like. Especially inexperienced operators may steer more deeply than necessary due to delayed response. The control signal transmission program 63 of the control device 50 desensitizes the vicinity of the rudder neutral due to the original relationship between the steering amount and the actual rudder angle (P in FIG. 9) as the delay of the video data increases (EXP in FIG. 9). ), And narrow the range of the actual steering angle (DR in FIG. 9). This reduces excessive steering due to video delay.

In addition, the control signal transmission program 63 may narrow only the range of the actual steering angle according to the delay amount of the video data while maintaining the direct proportional relationship between the steering amount and the actual steering angle. That is, only the upper limit of the flight speed may be changed according to the delay amount of the video data. The function of adjusting the steering amount and the actual steering angle by the control signal transmission program 63 is only a function at the time of manual steering, and may be automatically disabled at the time of autopilot.

(Control sharing function)
FIG. 10 is a schematic diagram showing a state of the operation center C in which a large number of control devices 50 are gathered at one base. FIG. 11 is a schematic diagram of a control device 50 in which a pair of two operators cooperate to manage a multicopter 10.

The control device 50 has a control switching program 66 which is a means (control switching means) for switching the control main body of the multicopter 10 and the equipment mounted on the multicopter 10 to another control device 50. The control switching program 66 is a function that entrusts the control of the multicopter 10 to an operator who can deal with an abnormality in the operation of the multicopter 10 or when a difficult maneuvering operation is required. .. Alternatively, this can be used to give the user control (control right) of the multicopter 10 and the equipment mounted on the multicopter 10, or simply to take over the work to another operator.

The "control subject" here means, for example, the control device 50 or other control device 50 to which the authority to operate the multicopter 10, the authority to operate the equipment mounted on the multicopter 10, the authority to acquire and display the mobile body information i, etc. belong. It means a device. The control switching program 66 can switch these privileges collectively or partially to another control device 50 or other device. The control switching program 66 can be executed manually, but it can also be automatically executed as described below.

For example, when a lower-level operator under training and a higher-level operator who is instructing the training are paired to operate the multicopter 10 using a set of control devices 50 as shown in FIG. 11, the monitors 70 of both control devices 50 are used. The moving body information i of the multicopter 10 is displayed in the above, and when some control operation is performed by the senior operator, the control subject is forcibly switched to the control device 50 of the senior operator by the control switching program 66. Is possible. The control switching program 66 used at this time may be that of the control device 50 of the senior operator, that of the control device 50 of the lower level operator, or that the control switching program 66 of both of these control devices 50 is performed in cooperation with each other. You may. Alternatively, as will be described later, the control switching program 66 included in the management device 80, which is a higher-level device of the control device 50, may be used. Similarly, for example, when a local personnel is assigned to support the operation of the multicopter 10, the operation from the control device 50 is invalidated while the local personnel directly operate the multicopter 10 using a radio or the like. (The control body of the multicopter 10 is switched from the control device 50 to a device such as a local radio). The mounting form of the control switching means is not limited to software (program), and may be configured by hardware.

If there are a plurality of candidates for the control device 50 of the switching destination when the control switching program 66 is automatically executed, the control switching program 66 automatically selects the switching destination according to the content of the mobile information i. More specifically, the control switching program 66 classifies the skill level and the ex officio of the operator in advance, and the type of abnormality determined from the mobile information i, the required determination and the difficulty of the operation, or the difficulty level of the operation. It automatically determines which control device 50, that is, which operator the control subject is switched to, according to the degree of urgency or the like.

For example, when a mechanical abnormality is detected from the multicopter 10, it is necessary to immediately secure safety in case of a crash, so it is necessary to switch the control subject to an operator having advanced maneuvering skills and decision-making power. Also, for example, when a large number of multicopters 10 for transportation and delivery are congested on the flight course, basically only pay attention to the intervals and flight speeds of these multicopters 10, so an inexperienced operator. But it can be dealt with. However, if there is concern about battery exhaustion, it is necessary to switch to a driver who can make decisions about route changes and emergency landings. Also, for example, if the propeller is partially damaged, if the lift is sufficient, it is basically just watching, but if the operation of the multicopter 10 becomes difficult, switch to a driver who can judge an emergency landing. There is a need. If a part of the propeller is damaged and the lift is insufficient, the aircraft of the multicopter 10 will gradually descend, but at this time, the rotation range of the remaining propellers is close to the limit, and skillful maneuvering skills are required. Required. Further, for example, when an abnormality occurs in the operation of the GPS receiver 32, the multicopter 10 may malfunction depending on the content of the abnormality. Even an inexperienced operator can deal with the decrease in the number of satellites captured by the GPS receiver 32, but if there is a problem with autonomous operation due to a significant decrease in measurement accuracy, the operator can manually operate. You need to switch. Further, for example, when an abnormality occurs in the communication between the control device 50 and the multicopter 10, the multicopter 10 can be steered even if the image from the multicopter 10 cannot be displayed, and other telemetry data is received. It is not so urgent if it is possible. However, if it is not a temporary communication failure caused by the flight environment but a failure of the communication equipment, the urgency is high. In addition, if the place where the abnormality occurred is a densely populated area or if the aircraft was flying at low altitude at the time of failure, higher judgment and higher maneuvering skills are required. The control switching program 66 can automatically switch a part or all of the control main body of the multicopter 10 to the operator (control device 50) capable of coping with such a situation / condition.

The control switching program 66 of the present embodiment not only switches the control subject such as the multicopter 10 to another control device 50 or other device, but also uses it for communication with the multicopter 10 according to the content of the moving body information i. You can also switch lines automatically. For example, it is conceivable to use a best effort type communication line during normal flight of the multicopter 10 and switch to a guarantee type communication line with guaranteed QoS (Quality of Service) in an emergency. The communication line may be switched by a program different from the control switching program 66.

As described above, the target for switching the control subject by the control switching program 66 is not limited to the other control device 50, and the control subject can be switched to another device other than the control device 50. For example, a user of a service using the remote control system S, that is, a general personal computer or tablet terminal used by the user, is given control of the camera 41 or the camera gimbal 49, and the user is made to shoot a desired image / video. And so on. Further, for example, a camera 41 for the operator to operate the multicopter 10 and a camera for the user to capture a desired image / video are separately mounted, and the communication path of the camera for the user is set to the control device 50. By providing it separately from the communication path with the multicopter 10 and making the communication path and protocol of the camera for the user more accessible (for example, enabling access with a web browser), the user can control the control device 50. It is possible to easily access the camera 41 or the like even if the user is away from the camera 41. In addition, by sharing the image taken by the user's camera with the operator and sharing the current position information of the multicopter 10 with the user, the operator and the user can smoothly cooperate with each other to perform the shooting work. You can also. The device that can give control to the user is not limited to the camera, and the user can be given control of any device according to the purpose.

As shown in FIG. 10, a large number of control devices 50 are installed in the operation center C. In this case, for example, as shown in FIG. 12, the control switching program 66 may be mounted on the management device 80, which is a higher-level device of the control device 50, and the control device 80 may centrally switch the control main body. In the example of FIG. 12, the user (other device 90) of the service using the remote control system S is given control of the camera 41, and the image captured by the camera 41 is displayed on the monitor 91 of the other device 90. Has been done. Even in this case, the essential functions and roles of the control switching program 66 remain unchanged. The difference is simply whether it is mounted on the individual control device 50 or independent of the control device 50. Even if the control switching program 66 is mounted on each control device 50, they may be linked with each other, or the control switching program 66 of one control device 50 may be used as a master, and the control switching program 66 of another control device 50 may be used as a master. By making the slave a master-slave relationship, it is possible to do the same thing as the management device 80 does.

In addition, the operation center C comprehensively covers information on a plurality of or all multicopters 10 under the jurisdiction of the operation center C, and a part or the whole of the airspace under the jurisdiction of the operation center C by separating colors and layers. It has a central monitoring screen 79 for comprehensive display. In Operation Center C, in some cases, it is possible to take over the drone of another company that is flying dangerously and avoid the danger.

Further, for example, it is conceivable to display the captured image and telemetry data of the multicopter 10 in which a serious problem has occurred on the central monitoring screen 79, and to deal with the problem by a team of managers or a plurality of operators. .. In addition, for example, information on a plurality of multicopters 10 for which the situation should be watched is displayed side by side on the central monitoring screen 79, or these information are switched and displayed at predetermined seconds, and the situation is changed by a plurality of people. It is also possible to monitor.

(Camera fixing function)
The multicopter 10 of the present embodiment has a camera gimbal 49 (posture stabilizing device) that absorbs a change in the posture of the multicopter 10 and maintains the posture of the camera 41. Then, the control device 50 has a camera fixing button 531 (camera fixing means) for stopping the posture stabilizing operation of the camera gimbal 49 and fixing the direction of the camera 41. By intentionally fixing the orientation of the camera 41, it is possible to actually grasp the actual tilt and vibration of the aircraft, and for example, the multicopter 10 can be landed more safely.

The multicopter 10 of the present embodiment has a plurality of cameras 41 oriented in multiple directions. For example, when the multicopter 10 includes only one camera 41, the camera fixing button 531 presses the camera 41. The orientation of the camera 41 may be fixed after automatically pointing forward or downward of the multicopter 10.

[Multicopter]
Hereinafter, the functional configuration of the multicopter 10 will be described with reference to FIG. The multicopter 10 of this embodiment is stored in a so-called drone port, which is a dedicated takeoff and landing facility located all over the country, and automatically departs from the drone port according to an instruction from the control device 50 or another system. It should be noted that such a drone port is not indispensable. For example, personnel are assigned to the site to prepare for takeoff and landing of the multicopter 10, and personnel assigned to the site cooperate with the operator to provide support by manual operation. You may go.

The multicopter 10 of the present embodiment mainly includes a flight controller FC which is a control unit, a rotor R which is a plurality of horizontal rotors, a communication device 25 which communicates with an operator (control device 50), and a plurality of cameras 41. It is composed of a camera gimbal 49 that supports all or part of these, and a battery (not shown) that supplies power to them.

The flight controller FC has a control device 20 which is a microcontroller. The control device 20 has a CPU 21 which is a central processing device, and a memory 22 including a storage device such as a RAM, a ROM, and a flash memory. The control device 20 is not limited to a single microcontroller, but may be a combination with a so-called companion computer. In addition, it is also conceivable to configure the control device 20 with, for example, an FPGA (field-programmable gate array) or an ASIC (Application Specific Integrated Circuit).

The flight controller FC further has a flight control sensor group S including an IMU 31 (Inertial Measurement Unit), a GPS receiver 32, a pressure sensor 34, and an electronic compass 35, which are connected to the control device 20. Has been done.

The IMU 31 is a sensor that detects the inclination of the multicopter 10, and is mainly composed of a 3-axis acceleration sensor and a 3-axis angular velocity sensor. The barometric pressure sensor 34 is an altitude sensor that obtains the altitude above sea level (elevation) of the multicopter 10 from the detected barometric pressure value. The electronic compass 35 is a sensor that detects the azimuth angle of the heading (nose) of the flight controller FC, and is mainly composed of a three-axis geomagnetic sensor. The GPS receiver 32 is, to be exact, a receiver of a navigation satellite system (NSS). The GPS receiver 32 acquires the current latitude and longitude value from the Global Navigation Satellite System (GNSS) or the Regional Navigational Satellite System (RNSS).

The flight control sensor group S acquires the position information of the aircraft including the latitude and longitude, altitude, and the azimuth angle of the nose in addition to the tilt and rotation of the aircraft by the flight control sensor group S.

The control device 20 has a flight control program FS, which is a program for controlling the attitude and basic flight operations of the multicopter 10 during flight. The flight control program FS adjusts the rotation speed of each rotor R based on the information acquired from the flight control sensor group S, and flies the multicopter 10 while correcting the disturbance of the attitude and position of the aircraft.

The control device 20 further has an autonomous flight program AP, which is a program for autonomously flying the multicopter 10. Then, in the memory 22 of the control device 20, a flight plan FP, which is a designated parameter such as the latitude and longitude of the course on which the multicopter 10 is flown, the altitude and the speed during flight, and the like are registered. The autonomous flight program AP autonomously flies the multicopter 10 according to the flight plan FP, with instructions from the control device 50 and other systems, a predetermined time, and the like as starting conditions.

In addition, the multicopter 10 of the present embodiment includes a wind direction wind speed sensor 36 that acquires a wind direction and a wind speed at a current location, and a LiDAR 37 that acquires a positional relationship and a distance between the multicopter 10 and its peripheral objects. In the multicopter 10 of this embodiment, the point cloud data acquired by the LiDAR 37 is mainly used in the internal processing of the flight controller FC, and if necessary, the processed values such as the distance to the surrounding objects are sent to the control device 50 together with the video data. Will be sent. In addition, for example, it is conceivable to appropriately thin out the point cloud data acquired by LiDAR 37 and transfer it to the control device 50 in real time. As a result, the operator can obtain information on surrounding objects more quickly than waiting for the transfer of the image captured by the camera 41.

In addition, it is also significant to separately mount a sound collector such as a microphone array to collect environmental sounds around the multicopter 10 in order to grasp the flight state and flight environment of the multicopter 10. In addition to this, by equipping the control device 50 with a microphone and the multicopter 10 with a speaker, the operator can talk with a person in the vicinity via the multicopter 10. Further, the camera 41 mounted on the multicopter 10 is not limited to a general visible light camera, and may be, for example, a night-vision camera (high-sensitivity camera or infrared camera), a depth camera, or the like. In addition, the devices and equipment mounted on the multicopter 10 are not particularly limited, and for example, a parachute, a flashlight, a warning light, an alarm buzzer, or the like may be mounted.

The communication device 25 and the control device 50 of the multicopter 10 are not limited as long as they can transmit and receive control signals and data, regardless of their specific communication method, protocol, and line used. For example, mobile communication networks such as 3G, LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), and 5G (hereinafter, mobile communication networks using such terrestrial base stations are referred to as "mobile phone communication networks". By connecting the control device 50 and the multicopter 10 with (), the operator (control device 50) can access the multicopter 10 from anywhere within the service area of the mobile telephone communication network. Will be. Alternatively, it is conceivable to appropriately install an access point on the flight path of the multicopter 10 and connect the access point and the control device 50 by an optical line. The connection method between the access point and the multicopter 10 is not particularly limited, and for example, Wi-Fi (Wireless Fidelity) or frequency hopping type PCM (pulse code modulation) signal may be used, and mobile phone communication is performed from the access point. You may connect to the multicopter 10 using a net. Further, the communication path between the multicopter 10 and the control device 50 may be made redundant by using a plurality of communication methods in combination. In addition, for example, it is conceivable to separately prepare another UAV equipped with a repeater and connect the multicopter 10 to the control device 50 or an access point via the UAV.

Here, when the communication delay from the multicopter 10 to the control device 50 becomes a problem, for example, among the communication paths connecting these, at least a part of the communication methods is switched between the analog method and the digital method. It is possible to make it possible. Analog communication is generally inferior in data reproducibility to digital communication, but analog communication has the advantage of less delay than digital communication. The communication method may be switched manually or automatically on the control device 50 side, or automatically on the multicopter 10 side. As a condition for automatically switching, for example, the amount of data to be transmitted / received, the amount of delay, a communication failure, or any other condition can be considered.

If the range of movement of the multicopter 10 extends beyond the service area of the mobile phone communication network, such as offshore or mountainous areas, multi-satellite communication modules such as Iridium (registered trademark) and Inmarsat (registered trademark) are used. By mounting it on the copter 10, it is possible to maintain communication with the multicopter 10 even outside the service area of the mobile phone communication network. Here, by making it possible to switch the means for connecting to the multicopter 10 between the satellite communication line and another communication line, the communication path with the multicopter 10 can be made redundant, and the multicopter 10 and the multicopter 10 can be made redundant. The movable range of the multicopter 10 can be expanded while maintaining the communication quality of the multicopter 10 as much as possible. For example, when the multicopter 10 and the control device 50 are connected by a 5G line, all the information including the image captured by the camera 41 is transmitted to the control device 50, and after switching to the satellite communication line, the latitude and longitude and the pressure altitude It is conceivable to transmit only the telemetry data of the above, or to transmit the video data after switching to the satellite communication line by limiting the transmission to only the edge information, contour information, and segment information of the object. The communication line may be switched manually or automatically on the control device 50 side, or automatically on the multicopter 10 side. As the conditions for automatically switching, for example, the current position (latitude and pressure altitude), the radio field strength of another communication line such as a mobile phone communication network, and other arbitrary conditions can be considered. In addition, it is conceivable to always use both satellite communication lines and other communication lines for communication.

[Modification example of control device]
FIG. 13 is a schematic view showing a control device 50b which is a modification of the control device 50. FIG. 14 is a block diagram showing a functional configuration of the control device 50b. Hereinafter, the control device 50b will be described with reference to FIGS. 13 and 14.

In the control device 50b, the operator enters the spherical monitor 76, which is a spherical monitor, and controls the multicopter 10. In this modification, the multicopter 10 includes a hemispherical camera and an omnidirectional camera as its camera 41, and the image captured by the camera 41 is displayed as it is on the spherical monitor 76. By adopting the celestial sphere camera and the spherical monitor 76 in this modification, the gaps and overlaps of the images when shooting in all directions with multiple cameras, the load of converting the flat image and the spherical image are reduced, and the video data is made efficient. It is possible to acquire, transfer, and display. The spherical monitor 76 is not limited to the global shape as in the present embodiment, and may be a hemispherical shape.

The control device 50b further has a line-of-sight sensor 55 capable of detecting the line-of-sight direction of the operator. The image display program 61 enlarges and displays an image in the line-of-sight direction of the operator detected by the line-of-sight sensor 55. Further, the output value of the line-of-sight sensor 55 is also transferred to the multicopter 10. The photographing control device 40 of the multicopter 10 receives this information, and transmits the image data in the line-of-sight direction of the operator with high resolution and the image data with low resolution in the other directions to the control device 50b. As a result, the amount of communication of the video data transmitted from the multicopter 10 is suppressed, and the delay of the video displayed on the monitor 76 is reduced. In addition, only the image in the direction of the operator's line of sight and its surroundings may be transmitted, and transmission may be omitted for images in other ranges. The direction of the operator's line of sight is not limited to the direction of the eyeball, and may be the direction of the face.

(Speaker system)
Further, the control device 50b has a plurality of speakers 56 arranged around the operator. The speaker 56 of this embodiment is arranged so as to surround the front, back, left, and right of the operator in the circumferential direction.

The control device 50b has a speaker control program 64 (speaker control unit) that controls the operation of the speaker 56. The speaker control program 64 notifies the operator of the positional relationship and distance between the multicopter 10 and its peripheral objects by emitting a sound from the speaker 56 to the operator based on the content of the moving object information i. For example, when there is an obstacle on the right front side of the multicopter 10, a sound is emitted from the speaker 56 on the right front side of the operator, and the volume is increased as the obstacle approaches. As a result, the operator can recognize the position of the obstacle and its distance without visual recognition.

The arrangement of the speakers 56 is not limited to that of the present embodiment. For example, three or four speakers 56 including the left and right speakers 56 are arranged around the operator, and the sound source direction is simulated by phase control. Is also possible. Further, for example, it is conceivable to change the type of sound according to the type of peripheral objects (structures such as buildings, other UAVs, people). The use of the speaker 56 is not limited to that of the present embodiment, and for example, it is possible to express the wind direction and the wind speed at the current location of the multicopter 10 by the sound emitted from the speaker 56.

(Blower system)
Further, the control device 50b includes a blower device 57 that is arranged around the operator and blows wind to the operator, and a blower control program 65 (blower control unit) that controls the wind direction and volume by the blower device 57. doing. The blow control program 65 changes the wind direction and volume of the wind blown to the operator according to the wind direction and speed at the current location of the multicopter 10. For example, when the wind is blowing from the right front of the multicopter 10 toward the multicopter 10, the wind of the air volume corresponding to the wind speed is blown to the operator from the blower device 57 on the right front of the operator. As a result, the operator can operate the multicopter 10 as if he were at the site while being far away.

Although the embodiments of the present invention have been described above, the scope of the present invention is not limited thereto, and various modifications can be made without departing from the gist of the invention. For example, the moving object of the present invention is not limited to an unmanned aerial vehicle, and may be a so-called passenger drone that transports passengers or a manned aircraft on which a pilot is boarded. Further, the present invention is not limited to an aircraft, and may be, for example, a vehicle or a ship as long as it is a maneuverable mobile body.

S: Remote control system, C: Operation center, 10: Multicopter (mobile body, unmanned aircraft), 11: Body, R: Rotor, FC: Flight controller, FS: Flight control program, AP: Autonomous flight program, FP : Flight plan, 25: Communication device, S: Flight control sensor group, 31: IMU, 32: GPS receiver, 34: Pressure sensor, 35: Electronic compass, 36: Wind direction and speed sensor, 37: LiDAR, 40: Shooting control Device, 41: camera (shooting means), 42: camera arm, 49: camera gimbal (posture stabilization device), 50: control device, 51: control unit, 511: CPU, 512: memory, 52: communication device ( Information acquisition unit), 53: Handle, 531: Camera fixing button (camera fixing means), 54: Elevating lever, 55: Line-of-sight sensor, 56: Speaker, 57: Blower, 59: Driver's seat, 61: Video display program, 62: Resistance control program (resistance control unit), 63: control signal transmission program, 64: speaker control program (speaker control unit), 65: blower control program, 66: control switching program (control switching means), 70: monitor ( Display unit), 71: Front monitor, 72: Side monitor, 73: Lower monitor, 74: Upper monitor, 75: Rear monitor, 76: Spherical monitor, 79: Central monitoring screen, 80: Management device, 90: Other Device, i: moving object information, M: map image, N: map data

Claims (3)

With a mobile body
It is equipped with a control device for remotely controlling the moving body.
The moving body has a means of photographing, and the moving body has a means of photographing.
The control device has a display unit that displays moving body information that is information about the inner world and / or the outer world of the moving body.
The remote control system is characterized in that the moving body can perform edge detection on a captured image or video and transmit edge information, contour information or segment information of the image or video to the control device. The remote control system according to claim 1, wherein the moving body is located in a place invisible from the position of the control device. The control device according to claim 1 or 2.

Images