JP6785020B1 - Remote control system and its control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To manage a moving body in a distant place with higher accuracy. SOLUTION: The remote control system is provided with a moving body 10 and a control device for remotely controlling the moving body, and the moving body has a photographing means 41 capable of photographing its own machine, and its control. apparatus. At this time, the shooting means preferably includes a camera and an arm 42 that supports the camera and can change the shooting direction of the camera, and the arm preferably has a shooting position and a shooting direction of the camera. Is more preferable to be able to change. [Selection diagram] Fig. 4

Description

The present invention relates to a remote control technique for a mobile body.

The following Patent Document 1 discloses a landing operation device for an unmanned aerial vehicle.

Japanese Unexamined Patent Publication No. 3-5297

The problem to be solved by the present invention is to manage a distant moving body with higher accuracy.

In order to solve the above problems, the remote control system of the present invention includes a moving body and a control device for remotely controlling the moving body, the moving body has a photographing means, and the photographing means is outside the machine. It has a flexible arm that extends, and the photographing means is characterized in that the photographing position and the photographing direction can be changed by bending the arm .

At this time, it is preferable that the photographing means can photograph the upper surface and the lower surface of the body portion of the moving body.

At this time, it is preferable that the arm can be housed in the body of the moving body.

Further, in the remote control system of the present invention, the moving body may be an unmanned aerial vehicle.

Further, in the remote control system of the present invention, the moving body may be in a place invisible from the position of the control device.

As described above, according to the remote control system of the present invention and the control device thereof, it is possible to manage a distant moving body with higher accuracy.

It is a schematic diagram which shows an example of the control device which concerns on embodiment. It is a block diagram which shows the functional structure of a control device. It is a block diagram which shows the functional structure of a multicopter. It is a partially enlarged view which shows the example of the camera which takes a picture of a multicopter from a third party viewpoint. It is a schematic diagram which shows the example of the map data displayed on the monitor of a control device. This is an example of displaying an icon indicating the current position of the multicopter on the three-dimensional terrain data. This is an example of superimposing a stereoscopic image showing the current position of the multicopter on the image taken by the multicopter. It is a graph which shows the relationship between the operation amount of a steering wheel, and the actual steering angle of a multicopter. It is a schematic diagram which shows the state in the facility of an operation center. It is a schematic diagram of a control device in which a pair of operators cooperates to manage a multicopter. It is a block diagram which shows the example of the remote monitoring system using a management device. It is a schematic diagram which shows the modification of the control device. It is a block diagram which shows the functional structure of the control device which concerns on a modification.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below is an example of a remote control system S that monitors and controls a moving body, a multicopter 10, from a distance by a control device 50. The multicopter 10 of this embodiment is a small UAV (Unmanned aerial vehicle) that flies with a plurality of horizontal rotors. The multicopter 10 transmits the information acquired by the device mounted on the multicopter 10 to the control device 50, and the control device 50 remotely controls the operation of the multicopter 10 itself and the device mounted on the multicopter 10. The combination of the multicopter 10 and the control device 50 is not limited to one-to-one. One control device 50 manages a plurality of multicopters 10, and conversely, a plurality of control devices 50 manage one multicopter. It is also possible to manage 10.

[Control device]
Hereinafter, the configuration of the control device 50 will be described. FIG. 1 is a schematic view showing an example of the control device 50. FIG. 2 is a block diagram showing a functional configuration of the control device 50. FIG. 3 is a block diagram showing a functional configuration of the multicopter 10.

(Monitor system)
The cockpit 59 of the control device 50 is surrounded by a plurality of monitors 70 (display units). The monitor 70 of the present embodiment includes a front monitor 71 which is a monitor arranged in front of the driver's seat 59, side monitors 72 arranged on the left and right sides of the driver's seat 59, and a lower monitor 73 arranged on the floor surface of the control device 50. It is composed of an upper monitor 74 arranged on the ceiling surface of the control device 50 and a rear monitor 75 arranged behind the driver's seat 59.

The multicopter 10 of this embodiment is equipped with a plurality of cameras 41 (photographing means). The multicopter 10 captures front-back, left-right, and top-bottom images of the aircraft by these cameras 41. The image display program 61 of the control device 50 displays these images received from the multicopter 10 on the monitor 70 corresponding to the photographing direction. Specifically, the front monitor 71 displays the image in front of the multicopter 10 in a wide range, the side monitor 72 displays the left and right images of the multicopter 10, and the lower monitor 73 displays the image directly below or in front of the multicopter 10. The image diagonally below is displayed on the upper monitor 74, the image directly above the multicopter 10 or diagonally above the front, and the image behind the multicopter 10 is displayed on the rear monitor 75. As a result, the operator can grasp the surrounding environment of the multicopter 10 as if he / she is on the multicopter 10.

Although the lower monitor 73 of this embodiment is installed on the floor surface of the control device 50, the position of the lower monitor 73 is not limited to the floor surface, and is near the operator's feet (the operator drops his / her line of sight downward). It suffices if it is placed at the viewing position). The lower monitor 73 may be arranged with the screen facing diagonally upward under the front monitor 71 so that the screen is not hidden by the operator's feet when viewing the lower monitor 73, for example. Similarly, the upper monitor 74 of this embodiment is installed on the ceiling surface of the control device 50, but the position of the upper monitor 74 is not limited to the ceiling surface, and is arranged above the operator's head (the position where the operator looks up). You just have to. The upper monitor 74 may be arranged on the front monitor 71 with the screen facing diagonally downward so that, for example, the operator can check without turning his / her line of sight directly upward.

The image display program 61 of the present embodiment not only displays the images taken by each camera 41 on different monitors 70, but also displays these images side by side or superimposed on one monitor 70, or vice versa. An image having one wide angle of view can be divided and displayed on a plurality of monitors 70. For example, the images of the top, bottom, left, right, and rear of the multicopter 10 are superimposed on the edges of the front image and displayed simultaneously on the front monitor 71, or the images of the spherical camera and the hemispherical camera are flattened and divided, respectively. It can be displayed on the monitor 70 corresponding to the shooting direction of.

The number of cameras 41 mounted on the multicopter 10 and the shooting direction are arbitrary, and may be appropriately changed according to the intended use and purpose. The multicopter 10 may, for example, photograph only the front of the multicopter 10 or diagonally below the multicopter 10 with one camera 41, or may operate the camera 41 with the camera gimbal 49 to photograph a desired direction. Good. Further, the camera 41 may not be mounted on the multicopter 10, and only the output value of another sensor device or the processed value thereof may be transmitted to the control device 50.

Further, the camera 41 mounted on the multicopter 10 is not limited to the FPV (First Person View) method. For example, as shown in FIG. 4, the camera 41 is mounted on the tip of the camera arm 42 extending from the body of the multicopter 10. However, it is also conceivable to photograph the multicopter 10 itself from a third-person perspective. The camera arm 42 can freely change the shooting position of the camera 41 and the shooting direction thereof like a movable fiberscope. As a result, the upper surface and the lower surface of the body portion 11 of the multicopter 10 and the surrounding environment can be widely confirmed. Further, the camera arm 42 can be extended from the body portion 11 of the multicopter 10 or retracted into the body portion 10 by a drive mechanism (not shown). By storing the camera arm 42 in the body 11 when not in use, for example, it is possible to prevent the camera arm 42 from breaking when the multicopter 10 crashes, and the camera arm 42 in an emergency Increased reliability.

In addition, for example, a photograph of the multicopter 10 photographed by a fixed camera installed on the ground or an image of the multicopter 10 photographed by another UAV may be transmitted to the control device 50. It is extremely useful to photograph the state of the actual machine from the outside (make it visible) in the non-visual control of the moving body. By displaying such an image on the monitor 70, the operator can more objectively grasp the state / situation of the multicopter 10.

The control device 50 has a communication device 52 (information acquisition unit) that receives the aircraft information of the multicopter 10 or the mobile body information i that is information about the environment in which the multicopter 10 is placed. The aircraft information of the multicopter 10 referred to here is information on the inner world of the multicopter 10 (the attitude of the aircraft, the orientation of the heading, the remaining battery level, etc.), and the information on the environment in which the multicopter 10 is placed is. Information about the outside world (such as latitude and longitude and surrounding objects) of the multicopter 10. The definitions of "inner world" and "outer world" here are for convenience, and individual sensor values and acquired information may be classified into either. Any information about the state or situation of the multicopter 10 is a candidate for the mobile information i. The moving body information i of this embodiment includes an image or video image taken by the multicopter 10, an output value of the sensor device of the multicopter 10 or a processed value thereof, information indicating a positional relationship between the multicopter 10 and its peripheral objects, and , Information on the wind direction and speed at the current location of the multicopter 10.

More specifically, the multicopter 10 of this embodiment is acquired by the image taken by the camera 41, the point group data of the peripheral objects acquired by LiDAR37 (Light Detection And Ranging), its processed values, the flight control sensor group S, and the like. Data indicating the direction and attitude of the aircraft, the flight position (latitude and pressure altitude), the remaining battery level, the wind direction and the wind speed at the current location acquired by the wind direction and wind speed sensor 36 are transmitted to the control device 50.

The mobile information i does not necessarily have to be acquired and transmitted only by the multicopter 10. The moving object information i includes, for example, information on the positional relationship between the multicopter 10 and its surrounding objects possessed by the UTM (UAV Traffic Management) system, weather information observed by other systems, and images taken by other UAVs. A photographed image of the multicopter 10 from a third-party perspective may be included.

The video display program 61 of the present embodiment displays a numerical value indicating the distance to the peripheral object closest to the multicopter 10 among the peripheral objects of the multicopter 10 on the image (see, for example, the example of FIG. 6). .. When the distance to the peripheral object approaches the dangerous area, the video display program 61 calls the operator's attention by displaying the vicinity of the peripheral object in red or blinking the numerical value.

(Measures against communication delay)
FIG. 5 is a schematic view showing a display example of the map data M displayed on the monitor 70 of the control device 50. FIG. 6 is an example in which an icon indicating the current position of the multicopter 10 is superimposed and displayed on the three-dimensional terrain data N acquired in advance. FIG. 7 is an example in which a stereoscopic image showing the current position of the multicopter 10 is superimposed and displayed on the image captured by the multicopter 10.

The shooting control device 40 of the multicopter 10 operates the camera gimbal 49, starts / stops shooting by the camera 41, or starts / stops shooting by the camera 41 according to an instruction from the control device 50 or another system or according to pre-programmed conditions. The video data taken in 41 is processed.

The amount of video data captured by the multicopter 10 by the camera 41 is significantly larger than that of telemetry data indicating the current location of the multicopter 10, for example, and the delay during transfer tends to be large. That is, there is a discrepancy between the surrounding environment of the multicopter 10 grasped from the image displayed on the monitor 70 and the actual surrounding environment of the multicopter 10 at the time of confirming the image. Therefore, when the operator manually operates the multicopter 10 while watching the image, helicopter needs to be aware of how much the image he is viewing is displayed with a delay.

The photographing control device 40 of the present embodiment performs edge detection on the image captured by the camera 41 by, for example, a high-pass filter, transmits only the extracted edge information to the control device 50, and further performs object detection and segmentation to perform object detection. Only the contour information and the segment information can be transmitted to the control device 50. As a result, the amount of data transmitted from the multicopter 10 to the control device 50 is suppressed, and the time lag until the information from the camera 41 is displayed on the monitor 70 is reduced. Further, when the shooting control device 40 of the present embodiment has a distance to the multicopter 10 closer than a predetermined threshold value among the peripheral objects detected from the image, only the image information related to the peripheral object is obtained. Can be sent in preference to. As a result, the peripheral objects to be watched are promptly displayed on the monitor 70, and the operator can have a longer grace period to deal with them.

As shown in FIG. 5, the map data M includes the icon 10a of the current position of the multicopter 10 identified from the telemetry data of the multicopter 10, and the image displayed on the monitor 70 from which position to which direction. An icon 10b indicating whether the image was taken toward the subject is displayed. In addition, a delay time (300 msec in the example of this embodiment) is also displayed in the vicinity of the icon 10b. The shooting point and delay time of the icon 10b can be specified, for example, by combining the time stamp and the longitude / latitude information of the NMEA sentence (GPS data) received earlier and the time stamp of the RTP header (video data). Information on the shooting direction may be included in the video data or the telemetry data.

Further, as shown in FIG. 6, the video display program 61 of the present embodiment can display the icon 10a indicating the current position of the multicopter 10 on the three-dimensional terrain data N. For example, in applications such as transporting goods between distribution centers, where the same air route is repeatedly flown to the multicopter 10, the topography and altitude of the air route are acquired together with the latitude and longitude information, and an external photograph of the feature is taken. The three-dimensional topographical data N of the air route can be prepared in advance by overlapping the above. Then, by virtually reproducing the flight status on the terrain data N on the control device 50 side from the current position information (latitude and pressure altitude) of the multicopter 10, the flight status of the multicopter 10 with a small amount of data communication. Can be freely confirmed from all perspectives. This also makes it possible to reduce the number of devices such as cameras mounted on the multicopter 10.

Further, as shown in FIG. 7, the image display program 61 of the present embodiment displays an image in the traveling direction taken by the multicopter 10 on the monitor 70, and the multicopter 10 identified from the telemetry data of the multicopter 10 there. It is also possible to superimpose a stereoscopic image showing the current position of. When the image of the traveling direction taken by the multicopter 10 is displayed on the monitor 70, the image showing the current position of the multicopter 10 as seen from the shooting position of the image is three-dimensionally superimposed and displayed for maneuvering. The person can grasp the current position of the multicopter 10 without misunderstanding even from the delayed image.

For example, when the multicopter 10 is shooting an image in the traveling direction and the delay of the image data is not so large, the above-mentioned AR (Augmented Reality) method (a stereoscopic image of the multicopter 10 in the image in the traveling direction with a time lag). When there is no image in the direction of travel, when the delay is large, or at night, the above-mentioned VR (Virtual Reality) method (multicopter 10 on the terrain data acquired in advance) is used to deal with the delay. You may switch to (a method of virtually reproducing the flight situation). The type of information displayed on the monitor 70 by these AR-like method or VR-like method is not particularly limited, and various sensor information of the multicopter 10 and various sensor information prepared in advance according to the use of the multicopter 10 and the information to be watched at each time. You can additionally display the saved information.

(Steering system)
The control device 50 has a handle 53, which is a control stick, and an elevating lever 54. The control device 50 of the present embodiment controls the pitch and roll of the multicopter 10 by tilting the handle 53, and controls the yaw of the multicopter 10 by twisting the handle 53. The operation of the handle 53 and the elevating lever 54 by the operator is transmitted to the multicopter 10 by the control signal transmission program 63. The interface between the control device 50 and the operator is not limited to the handle 53 and the elevating lever 54 of this embodiment, and if the throttle, aileron, rudder, and elevator can be steered to the multicopter 10, for example, for a radio control. It may be something like a radio or a controller for a home video game console. The control device 50 is also provided with a touch panel, keyboard, mouse, and the like, which are general input devices, and commands can be sent to the multicopter 10 from these. ..

Further, the control device 50 of the present embodiment has a resistance control program 62 that controls the amount of resistance when the operator operates the handle 53 and the elevating lever 54. The resistance control program 62 changes the amount of resistance when operating the handle 53 and the elevating lever 54 in each direction according to the weight of the multicopter 10 and the wind direction and speed at the current location of the multicopter 10. For example, when a strong wind is blowing from the front right of the multicopter 10 toward the multicopter 10, the operation of moving the operator to the front right is heavy, and the operation of moving the operator to the rear left is light. As a result, the operator P can intuitively grasp the properties of the multicopter 10 and the flight environment from the resistance amount of the handle 53 and the elevating lever 54.

FIG. 8 is a graph showing the relationship between the operating amount (steering amount) of the steering wheel 53 and the actual steering angle of the multicopter 10. When the image captured by the camera 41 of the multicopter 10 is delayed and displayed on the monitor 70, there is a time lag until the response is obtained even if the operator operates the steering wheel 53 or the like. Especially inexperienced pilots may steer more deeply than necessary due to delayed response. The control signal transmission program 63 of the control device 50 desensitizes the vicinity of the rudder neutral from the original relationship between the steering amount and the actual rudder angle (P in FIG. 8) as the delay of the video data increases (EXP in FIG. 8). ), And narrow the range of the actual steering angle (DR in FIG. 8). This reduces excessive steering due to video delay.

In addition, the control signal transmission program 63 may narrow only the range of the actual steering angle according to the delay amount of the video data while maintaining the direct proportional relationship between the steering amount and the actual steering angle. That is, only the upper limit of the flight speed may be changed according to the amount of delay of the video data. The steering amount and actual steering angle adjustment function by the control signal transmission program 63 is only a function at the time of manual operation, and may be automatically disabled at the time of autopilot.

(Control sharing function)
FIG. 9 is a schematic view showing a state of the operation center C in which a large number of control devices 50 are gathered at one base. FIG. 10 is a schematic view of a control device 50 in which a pair of two operators cooperate to manage the multicopter 10.

The control device 50 has a control switching program 66 which is a means (control switching means) for switching the control subject of the multicopter 10 and the equipment mounted on the multicopter 10 to another control device 50. The control switching program 66 is a function that entrusts the control of the multicopter 10 to an operator who can deal with an abnormality in the operation of the multicopter 10 or when a difficult maneuvering operation is required. .. Alternatively, this can be used to give the user control (control right) of the multicopter 10 and the equipment mounted on the multicopter 10, or simply to take over the work to another operator.

The "control body" here means, for example, the control device 50 or other control device 50 to which the authority to operate the multicopter 10, the authority to operate the equipment mounted on the multicopter 10, the authority to acquire and display the moving body information i, etc. belong. It means a device. The control switching program 66 can switch these authorities collectively or partially to another control device 50 or other device. The control switching program 66 can be manually executed, but can also be automatically executed as described below.

For example, when a lower-level pilot who is training and a higher-level operator who is instructing the trainer use a set of control devices 50 as shown in FIG. 10 to control the multicopter 10, the monitors 70 of both control devices 50 are used. The moving body information i of the multicopter 10 is displayed in the above, and when some maneuvering operation is performed by the senior pilot, the control subject is forcibly switched to the maneuvering device 50 of the senior pilot by the control switching program 66. Can be considered. The control switching program 66 used at this time may be that of the control device 50 of the senior operator or that of the control device 50 of the lower level operator, or the control switching program 66 of both of these control devices 50 is performed in cooperation with each other. You may. Alternatively, as will be described later, the control switching program 66 included in the management device 80, which is a higher-level device of the control device 50, may be used. Similarly, for example, when a local personnel is assigned to support the operation of the multicopter 10, the operation from the control device 50 is invalidated while the local personnel directly operate the multicopter 10 using a radio or the like. (The control subject of the multicopter 10 may be switched from the control device 50 to a device such as a local radio). The implementation form of the control switching means is not limited to software (program), and may be configured by hardware.

Further, when the control switching program 66 is automatically executed and there are a plurality of candidates for the control device 50 of the switching destination, the control switching program 66 automatically selects the switching destination according to the content of the moving body information i. More specifically, the control switching program 66 classifies the proficiency level and the ex officio of the operator in advance, and the type of abnormality determined from the moving body information i, the required determination and the difficulty level of the operation, or Which control device 50, that is, which operator to switch the control subject to, is automatically determined according to the degree of urgency or the like.

For example, when a mechanical abnormality is detected from the multicopter 10, it is necessary to immediately secure safety in preparation for a crash, so it is necessary to switch the control subject to an operator having advanced maneuvering skills and decision-making power. Also, for example, when a large number of multicopters 10 for transportation and delivery are congested on the flight course, basically only paying attention to the intervals and flight speeds of these multicopters 10, so an inexperienced operator. But it can be dealt with. However, if there is concern about battery exhaustion, it is necessary to switch to a pilot who can make decisions about route changes and crash landings. Also, for example, if the propeller is partially damaged, if the lift is sufficient, it is basically just watching, but if the operation of the multicopter 10 becomes difficult, switch to a driver who can judge an emergency landing. There is a need. If a part of the propeller is damaged and the lift is insufficient, the aircraft of the multicopter 10 will gradually descend, but at this time, the rotation range of the remaining propeller is near the limit, and skillful maneuvering skills are required. Required. Further, for example, when an abnormality occurs in the operation of the GPS receiver 32, the multicopter 10 may malfunction depending on the content of the abnormality. Even an inexperienced pilot can deal with the decrease in the number of satellites captured by the GPS receiver 32, but if the measurement accuracy is significantly reduced and the autonomous operation is hindered, the operator can be manually operated. You need to switch. Further, for example, when an abnormality occurs in the communication between the control device 50 and the multicopter 10, the multicopter 10 can be steered even if the image from the multicopter 10 cannot be displayed, and other telemetry data is received. If it is possible, it is not so urgent. However, if it is not a temporary communication failure due to the flight environment but a failure of the communication equipment, the urgency is high. In addition, if the place where the abnormality occurred is a densely populated area or if the aircraft was flying at low altitude at the time of failure, more advanced judgment and higher maneuvering skills are required. The control switching program 66 can automatically switch a part or all of the control main body of the multicopter 10 to the operator (control device 50) who can deal with such a situation / condition.

The control switching program 66 of the present embodiment not only switches the control subject such as the multicopter 10 to another control device 50 or other device, but also uses it for communication with the multicopter 10 according to the content of the mobile information i. You can also switch lines automatically. For example, it is conceivable to use a best effort type communication line during normal flight of the multicopter 10 and switch to a guarantee type communication line with guaranteed Quality of Service (QoS) in an emergency. The communication line may be switched by a program different from the control switching program 66.

As described above, the target for switching the control subject by the control switching program 66 is not limited to the other control device 50, and the control subject can be switched to a device other than the control device 50. For example, a user of a service using the remote control system S, that is, a general personal computer or tablet terminal used by the user, is given control of the camera 41 or the camera gimbal 49, and the user is made to take a desired image / video. And so on. Further, for example, a camera 41 for the operator to operate the multicopter 10 and a camera for the user to capture a desired image / video are separately mounted, and the communication path of the camera for the user is set as the control device 50. By providing a communication path with the multicopter 10 separately and making the communication path and protocol of the camera for the user more accessible (for example, enabling access with a web browser), the user can control the control device 50. It is possible to easily access the camera 41 and the like even if the user is away from the camera 41. In addition, by sharing the image taken by the user's camera with the operator and further sharing the current position information of the multicopter 10 with the user, the operator and the user can perform the shooting work in smooth cooperation. You can also. The device that can give control to the user is not limited to the camera, and the user can be given control of any device according to the purpose.

As shown in FIG. 9, a large number of control devices 50 are installed in the operation center C. In this case, for example, as shown in FIG. 11, the control switching program 66 may be mounted on the management device 80, which is a higher-level device of the control device 50, and the control device 80 may centrally switch the control subject. In the example of FIG. 11, the user (other device 90) of the service using the remote control system S is given control of the camera 41, and the image captured by the camera 41 is displayed on the monitor 91 of the other device 90. Has been done. Even in this case, the essential functions and roles of the control switching program 66 remain unchanged. It is simply the difference between being mounted on the individual controls 50 and independent of the controls 50. Even if the control switching program 66 is implemented in each control device 50, they may cooperate with each other, or the control switching program 66 of one control device 50 may be used as a master, and the control switching program 66 of another control device 50 may be used as a master. By making the slave a master-slave relationship, it is possible to do the same thing as the management device 80 does.

In addition, the operation center C covers information on a plurality of or all multicopters 10 under the jurisdiction of the operation center C, and a part or the whole of the airspace under the jurisdiction of the operation center C by separating colors and layers. It has a central monitoring screen 79 for comprehensive display. In Operation Center C, in some cases, it is possible to take over the drone of another company that is flying dangerously and avoid the danger.

Further, for example, it is conceivable to display the captured video and telemetry data of the multicopter 10 in which a serious problem has occurred on the central monitoring screen 79, and to deal with the problem by a team of managers or a plurality of operators. .. In addition, for example, information on a plurality of multicopters 10 for which the situation should be watched is displayed side by side on the central monitoring screen 79, or these information are switched and displayed at predetermined seconds, and the situation is changed by a plurality of people. It is also possible to monitor.

(Camera fixing function)
The multicopter 10 of the present embodiment has a camera gimbal 49 (posture stabilizing device) that absorbs a change in the posture of the multicopter 10 and maintains the posture of the camera 41. Then, the control device 50 has a camera fixing button 531 (camera fixing means) for stopping the posture stabilizing operation of the camera gimbal 49 and fixing the direction of the camera 41. By intentionally fixing the orientation of the camera 41, it is possible to actually grasp the actual tilt and vibration of the airframe, and for example, the multicopter 10 can be landed more safely.

The multicopter 10 of this embodiment has a plurality of cameras 41 oriented in multiple directions. For example, when the multicopter 10 includes only one camera 41, the camera fixing button 531 presses the camera 41. The orientation of the camera 41 may be fixed after automatically pointing forward or downward of the multicopter 10.

[Multicopter]
Hereinafter, the functional configuration of the multicopter 10 will be described with reference to FIG. The multicopter 10 of this embodiment is stored in a so-called drone port, which is a dedicated takeoff and landing facility located all over the country, and automatically departs from the drone port according to an instruction from the control device 50 or another system. It should be noted that such a drone port is not indispensable. For example, personnel are assigned to the site to prepare for takeoff and landing of the multicopter 10, and personnel assigned to the site cooperate with the operator to provide support by manual operation. You may go.

The multicopter 10 of this embodiment mainly includes a flight controller FC as a control unit, a rotor R as a plurality of horizontal rotors, a communication device 25 for communicating with an operator (control device 50), and a plurality of cameras 41. It is composed of a camera gimbal 49 that supports all or a part of these, and a battery (not shown) that supplies power to them.

The flight controller FC has a control device 20 which is a microcontroller. The control device 20 has a CPU 21 which is a central processing unit and a memory 22 including a storage device such as a RAM, a ROM, and a flash memory. The control device 20 is not limited to a single microcontroller, and may be a combination with a so-called companion computer. In addition, it is also conceivable to configure the control device 20 with, for example, an FPGA (field-programmable gate array) or an ASIC (Application Specific Integrated Circuit).

The flight controller FC further has a flight control sensor group S including an IMU 31 (Inertial Measurement Unit), a GPS receiver 32, a pressure sensor 34, and an electronic compass 35, which are connected to the control device 20. Has been done.

The IMU 31 is a sensor that detects the inclination of the multicopter 10, and is mainly composed of a 3-axis acceleration sensor and a 3-axis angular velocity sensor. The barometric pressure sensor 34 is an altitude sensor that obtains the altitude above sea level (elevation) of the multicopter 10 from the detected barometric pressure value. The electronic compass 35 is a sensor that detects the azimuth angle of the heading (nose) of the flight controller FC, and is mainly composed of a three-axis geomagnetic sensor. The GPS receiver 32 is, to be exact, a receiver of a navigation satellite system (NSS). The GPS receiver 32 acquires the current latitude and longitude value from the Global Navigation Satellite System (GNSS) or the Regional Navigational Satellite System (RNSS).

The flight control sensor group S acquires the position information of the aircraft including the latitude and longitude, altitude, and azimuth of the nose in addition to the tilt and rotation of the aircraft by the flight control sensor group S.

The control device 20 has a flight control program FS which is a program for controlling the attitude and basic flight movements of the multicopter 10 during flight. The flight control program FS adjusts the rotation speed of each rotor R based on the information acquired from the flight control sensor group S, and flies the multicopter 10 while correcting the disturbance of the attitude and position of the aircraft.

The control device 20 further has an autonomous flight program AP, which is a program for autonomously flying the multicopter 10. Then, in the memory 22 of the control device 20, a flight plan FP, which is a designated parameter such as the latitude and longitude of the course on which the multicopter 10 is flown, the altitude during flight, and the speed, is registered. The autonomous flight program AP autonomously flies the multicopter 10 according to the flight plan FP, with instructions from the control device 50 and other systems, a predetermined time, and the like as starting conditions.

In addition, the multicopter 10 of the present embodiment includes a wind direction and wind speed sensor 36 that acquires the wind direction and speed at the current location, and a LiDAR 37 that acquires the positional relationship and distance between the multicopter 10 and its surrounding objects. In the multicopter 10 of this embodiment, the point cloud data acquired by the LiDAR 37 is mainly used in the internal processing of the flight controller FC, and if necessary, the processed values such as the distance to the surrounding objects are sent to the control device 50 together with the video data. Will be sent. In addition, for example, it is conceivable to appropriately thin out the point cloud data acquired by LiDAR 37 and transfer it to the control device 50 in real time. As a result, the operator can obtain information on surrounding objects more quickly than waiting for the transfer of the image captured by the camera 41.

In addition, it is also significant to separately mount a sound collector such as a microphone array to collect the environmental sound around the multicopter 10 in order to grasp the flight state and the flight environment of the multicopter 10. In addition to this, by providing the control device 50 with a microphone and the multicopter 10 with a speaker, the operator can talk with a person in the vicinity via the multicopter 10. Further, the camera 41 mounted on the multicopter 10 is not limited to a general visible light camera, and may be, for example, a night-vision camera (high-sensitivity camera or infrared camera), a depth camera, or the like. In addition, the devices and equipment mounted on the multicopter 10 are not particularly limited, and for example, a parachute, a flashlight, a warning light, an alarm buzzer, or the like may be mounted.

The communication device 25 and the control device 50 of the multicopter 10 are not limited in their specific communication method, protocol, and line used as long as they can transmit and receive control signals and data. For example, mobile communication networks such as 3G, LTE (Long Term Evolution), WiMAX (Worldwide Interoperability for Microwave Access), and 5G (hereinafter, mobile communication networks using such terrestrial base stations are referred to as "mobile phone communication networks". By connecting the control device 50 and the multicopter 10 with (), the operator (control device 50) can access the multicopter 10 from anywhere within the service area of the mobile telephone communication network. Will be. Alternatively, it is also conceivable to appropriately install an access point on the flight path of the multicopter 10 and connect the access point and the control device 50 by an optical line. The connection method between the access point and the multicopter 10 is not particularly limited, and for example, Wi-Fi (Wireless Fidelity) or frequency hopping type PCM (pulse code modulation) signal may be used, and mobile phone communication is performed from the access point. You may connect to the multicopter 10 using a net. Further, the communication path between the multicopter 10 and the control device 50 may be made redundant by using a plurality of communication methods in combination. In addition, for example, another UAV equipped with a repeater may be separately prepared, and the multicopter 10 may be connected to the control device 50 or an access point via the UAV.

Here, when the communication delay from the multicopter 10 to the control device 50 becomes a problem, for example, at least a part of the communication paths connecting them is switched between the analog system and the digital system. It is possible to make it possible. Analog communication is generally inferior in data reproducibility to digital communication, but analog communication has the advantage of less delay than digital communication. The communication method may be switched manually or automatically on the control device 50 side, or automatically on the multicopter 10 side. As the conditions for automatically switching, for example, the amount of data to be transmitted / received, the amount of delay, the communication failure, and other arbitrary conditions can be considered.

In addition, when the range of movement of the multicopter 10 extends beyond the service area of the mobile phone communication network, such as offshore or mountainous areas, satellite communication modules such as Iridium (registered trademark) and Inmarsat (registered trademark) are multi-multirotored. By mounting it on the copter 10, communication with the multicopter 10 can be maintained even outside the service area of the mobile phone communication network. Here, by making it possible to switch the means for connecting to the multicopter 10 between the satellite communication line and another communication line, the communication path with the multicopter 10 can be made redundant, and the multicopter 10 and the multicopter 10 can be made redundant. The movable range of the multicopter 10 can be expanded while maintaining the communication quality of the multicopter 10 as much as possible. For example, when the multicopter 10 and the control device 50 are connected by a 5G line, all the information including the image captured by the camera 41 is transmitted to the control device 50, and after switching to the satellite communication line, the latitude and longitude and the pressure altitude It is conceivable to transmit only the telemetry data of the above, or to transmit only the edge information, contour information, and segment information of the object after switching to the satellite communication line. The communication line may be switched manually or automatically on the control device 50 side, or automatically on the multicopter 10 side. As the conditions for automatically switching, for example, the current position (latitude and pressure altitude), the radio field strength of another communication line such as a mobile phone communication network, and other arbitrary conditions can be considered. In addition, it is conceivable to use both satellite communication lines and other communication lines at all times for communication.

[Modification example of control device]
FIG. 12 is a schematic view showing a control device 50b which is a modification of the control device 50. FIG. 13 is a block diagram showing a functional configuration of the control device 50b. Hereinafter, the control device 50b will be described with reference to FIGS. 12 and 13.

In the control device 50b, the operator enters the spherical monitor 76, which is a spherical monitor, and controls the multicopter 10. In this modification, the multicopter 10 includes a hemispherical camera and a spherical camera as its camera 41, and the image captured by the camera 41 is displayed as it is on the spherical monitor 76. By adopting the spherical camera and the spherical monitor 76 in this modification, the gaps and overlaps of the images when shooting in all directions with a plurality of cameras, the load of converting the flat image and the spherical image are reduced, and the video data is made efficient. It is possible to acquire, transfer, and display. The spherical monitor 76 is not limited to the global shape as in the present embodiment, and may be a hemispherical shape.

The control device 50b further has a line-of-sight sensor 55 capable of detecting the line-of-sight direction of the operator. The image display program 61 enlarges and displays the image in the line-of-sight direction of the operator detected by the line-of-sight sensor 55. Further, the output value of the line-of-sight sensor 55 is also transferred to the multicopter 10. The photographing control device 40 of the multicopter 10 receives this information, and transmits the image data in the line-of-sight direction of the operator with high resolution and the image data with low resolution in the other directions to the control device 50b. As a result, the amount of communication of the video data transmitted from the multicopter 10 is suppressed, and the delay of the video displayed on the monitor 76 is reduced. In addition, only the image in the direction of the pilot's line of sight and its surroundings may be transmitted, and transmission may be omitted for images in other ranges. The direction of the operator's line of sight is not limited to the direction of the eyeball, and may be the direction of the face.

(Speaker system)
Further, the control device 50b has a plurality of speakers 56 arranged around the operator. The speaker 56 of this embodiment is arranged so as to surround the front, rear, left, and right of the operator in the circumferential direction.

The control device 50b has a speaker control program 64 (speaker control unit) that controls the operation of the speaker 56. The speaker control program 64 notifies the operator of the positional relationship and distance between the multicopter 10 and its peripheral objects by emitting a sound from the speaker 56 to the operator based on the content of the moving body information i. For example, when there is an obstacle on the right front side of the multicopter 10, a sound is emitted from the speaker 56 on the right front side of the operator, and the volume is increased as the obstacle approaches. This allows the operator to visually recognize the position of the obstacle and its distance.

The arrangement of the speakers 56 is not limited to that of the present embodiment. For example, three or four speakers 56 including the left and right speakers 56 are arranged around the operator, and the sound source direction is simulated by phase control. Is also possible. Further, for example, it is conceivable to change the type of sound according to the type of peripheral objects (structures such as buildings, other UAVs, people). The use of the speaker 56 is not limited to that of the present embodiment, and for example, it is possible to express the wind direction and the wind speed at the current location of the multicopter 10 by the sound emitted from the speaker 56.

(Blower system)
Further, the control device 50b includes a blower 57 that is arranged around the operator and blows wind to the operator, and a blower control program 65 (blower control unit) that controls the wind direction and volume by the blower 57. doing. The blow control program 65 changes the wind direction and volume of the wind blown to the operator according to the wind direction and speed at the current location of the multicopter 10. For example, when the wind is blowing from the right front of the multicopter 10 toward the multicopter 10, the air volume of the wind corresponding to the wind speed is blown to the operator from the blower 57 on the right front of the operator. As a result, the operator can operate the multicopter 10 as if he were in the field while being far away.

Although the embodiments of the present invention have been described above, the scope of the present invention is not limited to this, and various modifications can be made without departing from the gist of the invention. For example, the moving object of the present invention is not limited to an unmanned aerial vehicle, and may be a so-called passenger drone that transports passengers or a manned aircraft on which a pilot is on board. Further, the present invention is not limited to an aircraft, and may be, for example, a vehicle or a ship as long as it is a maneuverable mobile body.

S: Remote control system, C: Operation center, 10: Multicopter (moving object, unmanned aircraft), 11: Body, R: Rotor, FC: Flight controller, FS: Flight control program, AP: Autonomous flight program, FP : Flight plan, 25: Communication device, S: Flight control sensor group, 31: IMU, 32: GPS receiver, 34: Pressure sensor, 35: Electronic compass, 36: Wind direction and speed sensor, 37: LiDAR, 40: Imaging control Device, 41: Camera (shooting means), 42: Camera arm, 49: Camera gimbal (posture stabilizer), 50: Control device, 51: Control unit, 511: CPU, 512: Memory, 52: Communication device ( Information acquisition unit), 53: Handle, 531: Camera fixing button (camera fixing means), 54: Lifting lever, 55: Line-of-sight sensor, 56: Speaker, 57: Blower, 59: Driver's seat, 61: Video display program, 62: Resistance control program (resistance control unit), 63: Control signal transmission program, 64: Speaker control program (speaker control unit), 65: Blower control program, 66: Control switching program (control switching means), 70: Monitor ( Display), 71: Front monitor, 72: Side monitor, 73: Lower monitor, 74: Upper monitor, 75: Rear monitor, 76: Spherical monitor, 79: Central monitoring screen, 80: Management device, 90: Other Device, i: moving object information, M: map data, N: topographical data

Claims (6)

With a mobile body
A control device for remotely controlling the moving body is provided.
The moving body has a means of photographing and
The imaging means has a flexible arm that extends out of the machine.
The photographing means is a remote control system capable of changing the photographing position and the photographing direction by bending the arm . The remote control system according to claim 1 , wherein the photographing means can photograph the upper surface and the lower surface of the body portion of the moving body. The remote control system according to claim 1 or 2 , wherein the arm can be housed in a body of the moving body. The remote control system according to any one of claims 1 to 3 , wherein the moving body is an unmanned aerial vehicle. The remote control system according to any one of claims 1 to 4 , wherein the moving body is located in a place that cannot be seen from the position of the control device. The control device according to any one of claims 1 to 5 .

Images