WO2024094378A1

WO2024094378A1 - - a system for real-time remote viewing of a site

Info

Publication number: WO2024094378A1
Application number: PCT/EP2023/077358
Authority: WO
Inventors: Jiang Zhou
Original assignee: Jiang Zhou
Priority date: 2022-11-02
Filing date: 2023-10-03
Publication date: 2024-05-10
Also published as: IE20220175A2

Abstract

A system (10) for real-time remote viewing of a site comprises micro-drone (11) mounted with a camera (12), an observer with an observer's personal computing device (13), a system server (15), and a remote viewer with a remote viewer's personal computing device (14). The system may require a relay box 20, if the micro-drone (11) does not have Bluetooth® capability but has Wi-Fi and is configured to a fixed IP address. The micro-drone (11) is a low-speed, palm-size micro-drone, with a gross weight of < 250g and flown by either the remote viewer or the observer within Visual Line of Sight (VLOS) of the observer, with control of the micro-drone (11) by the remote viewer being limited to three commands, being go-forward, yaw-left, and yaw-right, with the observer being able to take-over control of the micro-drone (11) at any time during the flight.

Description

- A system for real-time remote viewing of a site

This invention relates to a system for real-time remote viewing of a site and, in particular, to a system employing drones at the site.

Real-time remote viewing can be accomplished using teleconferencing applications such as WhatsApp^® and Skype^®, but these solutions are unwieldy, where a remote viewer has a requirement for particular viewing angles and locations at the site, as he is required to verbally instruct an observer at the site to manually move around his camera to each location.

Systems for the use of unmanned aerial vehicles (UAVs), such as drones, have been developed for military purposes. Such systems rely on proprietary, secure, high-speed networks for remote manipulation of a drone. These drones are launched and controlled by a user, such as an infantry soldier, who tracks the drone on a console, but is not necessarily in visual contact with the drone.

Commercial systems, which use drones, have been developed, where the drone is controlled remotely by a user.

US Patent No. 10,382,539 B1 relates to methods and apparatus for data control and transfer with an unmanned aerial vehicle, such as a drone. A system is described which has a drone session server to collect drone session information. A drone user machine is in a client relationship with the drone session server. A drone control machine is in a client relationship with the drone session server and is connected through a network with the drone user machine, and in a peer-to peer relationship with the drone user machine. The drone control machine is configured to relay video data from a drone to the drone user machine through a drone console and the drone control machine and the network via a peer-to-peer connection, wherein the drone console is within visual presence of the drone. The drone control machine sends autopilot commands to the drone console to lift off the drone and direct the drone into a three-dimensional geographical fence. The drone control machine evaluates user commands collected by the drone user machine, which drone user machine is operated by a user that is not within visual presence of the drone. These user commands are relayed to the drone control machine via the peer-to-peer connection to produce enforced limits commands through the drone console to maintain the drone within the three-dimensional geographical fence. The drone control machine sends autopilot commands to the drone to transport the drone from the three-dimensional geographical fence to a land site to complete a drone session.

Typically, the drone control console is used by a drone operator to fly a drone that is in the visual presence of the operator. However, in this case the operator is at the drone operator machine, which is not in the visual presence of the drone. In fact, the drone user machine may be used to pilot a drone thousands of miles away. Thus, it follows that the user must be a trained pilot in order to satisfy the flight regulations for the country in which the drone is being flown.

International Publication No. WO 2021/226316 A1 describes a long distance trans-continental remote drone piloting system. The remote drone piloting system includes a pilot endpoint system comprising a pilot endpoint and a controller connected to the pilot endpoint. The system includes a control endpoint system including a control endpoint, a signal adaptor connected to the control endpoint, and a transmitter connected to the signal adaptor. A drone is arranged to communicate drone video data to the control endpoint system. A remote bridge including a server is arranged to connect the pilot endpoint and the control endpoint using an internet protocol when the pilot endpoint and the control endpoint are located at different geographic locations such that the drone operating data and drone video data are communicated amongst the pilot endpoint, control endpoint, and drone in real-time.

In some examples of the system the pilot controller is a handheld controller that includes controls for piloting the drone. In operation, the pilot controller allows the remote pilot utilizing the pilot system to execute control of the drone in real-time by providing commands through the connection to the control endpoint system. Such commands operate to allow control of the flight of the drone, use of a camera on the drone and use of any other onboard drone systems or functionalities. A peer-to-peer audio connection allows the drone pilot and the remote pilot to communicate directly.

The drone pilot, who controls the drone through the pilot endpoint and the remote pilot, who controls the drone through the control endpoint must be trained to a level that will satisfy the flight regulations for the country in which the drone is being flown.

It is an object of the present invention to overcome the disadvantages of the real-time remote viewing systems hereinbefore described.

Thus, the invention provides a system for real-time remote viewing of a site, the system comprising a micro-drone, a camera mounted on the micro-drone, an observer’s personal computing device, with a system client software package installed thereon, with both the observer’s personal computing device and the micro-drone being locatable at the site, in use, and in communication with each other by a wireless connection, a remote viewer’s personal computing device, a system server, which system server manages the operation of the system, processes data collected by the micro-drone during a flight, and communicates the processed data to the observer’s personal computing device and the remote viewer’s personal computing device, a Web Real-Time Communication (WebRTC) connection between the observer’s personal computing device and the system server for audio, video and data transmission, a WebRTC connection between the remote viewer’s personal computing device and the system server, for audio, video and data transmission,

characterised in that, in use, the micro-drone is a low-speed, palm-size micro-drone, with a gross weight of < 250g and flown by either the remote viewer or the observer within Visual Line of Sight (VLOS) of the observer, with control of the micro-drone by the remote viewer being limited to three commands being go-forward, yaw-left, and yaw-right, with the observer being able to take-over control of the micro-drone at any time during the flight, and

wherein the system has a WebRTC connection between the remote viewer’s personal computing device and the observer’s personal computing device for direct audio, video and data transmission.

The word “device” in the phrases “observer’s personal computing device” and remote viewer’s computing device”, in this context, means a laptop, tablet, smartphone or the like.

The term server herein encompasses one or more physical or virtual, cloud-based servers, or a combination of both physical and virtual servers.

An advantage of the system for real-time remote viewing of a site in accordance with the invention is that, as the micro-drone is being flown by either the remote viewer or the observer within Visual Line of Sight (VLOS) of the observer, neither the observer nor the remote viewer is required to be a trained drone pilot. The remote viewer can be located anywhere in the world as long as he can connect to the Internet.

The micro-drone is flown, in accordance with the system in a low or no risk category to third parties. Thus, for example, operation of the micro-drone falls within EU regulation Open category A1 C0 (EU Commission Implementing Regulation (EU)2019/947) and can be used by anyone from the public without professional training. Open category, A1 subcategory and drone standard, C0, in accordance with the EU Regulation, requires that the maximum take-off mass of the unmanned aircraft must be less than 25kg, the unmanned aircraft must be operated within VLOS, and the unmanned aircraft must not be flown further than 120 metres from the closest point of the surface of the earth (Open category); the operations flown over people can only be conducted with unmanned aircraft that present a very low risk of harm or injury to other people due to their low weight (less than 250g), their type of construction, or because they are a toy. However, flight over open-air assemblies of people is not permitted (Subcategory A1); and drone must be less than 250g maximum take-off mass, have a maximum speed of 19m/s and are unable to be flown more than 120m from the controlling device (C0 standard).

A further advantage of the system is that the remote viewer and the observer can communicate with each other directly in real-time by the WebRTC connection. Such communication can include the transmitting of micro-drone control commands from the remote viewer’s personal computing device to the observer’s personal computing device.

However, the server can intervene in a remote viewing session by sending top priority commands such as force-landing, close-camera, etc. to the observer’s personal computing device, which will suppress the commands from the remote viewer’s personal computing device and stop illicit use of remote viewing.

Preferably, the maximum speed of the micro-drone is limited to within a range of 80 to 120cm/s, and the micro-drone is required to be flown within a radial distance of between 2 to 10m from the location of the observer’s personal computer device.

An advantage of the micro-drone specification is that it falls well within EU regulation Open category A1 C0.

Further, preferably, a sensor is mounted on the micro-drone and the data collected by the micro-drone during a flight includes data from the camera and data from the sensor.

An advantage of this aspect of the invention is that the inclusion of a sensor means that a combination of video data from the micro-drone camera with data collected by a sensor enhances the data being processed by the system server, resulting in additional information for use by the remote observer.

Suitably, the wireless connection between the observer’s personal computing device and the micro-drone has a Bluetooth^® connection.

An advantage of this aspect of the invention is that, where the micro-drone has Bluetooth^®, pairing the micro-drone with the observer’s personal computing device is straightforward. In practice the observer’s personal computing device first sets up a Bluetooth^® connection with the micro-drone and sends the micro-drone the Wi-Fi credentials, which are used by the micro-drone, to join a Wi-Fi network in which the observer’s personal computing device and the micro-drone exchange IP addresses and set up Wi-Fi connections for video and data transmission.

Preferably, where the micro-drone does not have Bluetooth^®capability, the system has a relay box, locatable in proximity to the observer, which relay box is a compact computing device with at least two Wi-Fi interfaces and a Bluetooth^® interface, and which relay box enables a wireless connection between the micro-drone and the relay box and a Wi-Fi connection and a Bluetooth^®connection between the relay box and the observer’s personal computing device.

An advantage of this aspect of the invention is that communication between the micro-drone and the observer’s personal computing device can be achieved by pairing the observer’s personal computing device with the relay box by way of a Bluetooth^®connection. The observer’s personal computer device then sends to the relay box Wi-Fi credentials which are used by a Wi-Fi interface of the relay box to join a Wi-Fi network where the observer’s personal computing device and the relay box exchange IP addresses and set up Wi-Fi connections.

Further, preferably, where there is no Internet availability through Wi-Fi at the site, the relay box includes a USB interface through which the relay box is connectable to the observer’s personal computing device by a USB cable.

An advantage of this aspect of the invention is that no Wi-Fi is required for Internet connection. The relay box connects to the observer’s personal computing device through the USB cable and sets up a Wi-Fi connection with the micro-drone and the Internet is accessible through a 4G/5G network connection on the observer’s personal computing device.

Further, preferably, the system server processes the data collected by the micro-drone by employing machine learning models, pretrained with large amounts of high-quality measurements, to rectify and enhance the data captured before it is transmitted to the observer’s personal computing device and the remote viewer’s personal computing device.

An advantage of this aspect of the invention is that the raw data from the micro-drone, which is of low quality can be enhanced using the machine learning modules. Thus, data collected by a basic camera and sensor mounted on the micro-drone can still provide accurate high-grade data to both the remote viewer’s personal computer and the observer’s personal computer device to provide real-time remote viewing of a quality that one would not expect from basic low-cost hardware.

Further, preferably, where a number of separate micro-drone flights using the same camera and/or sensor or a different camera/ and/ or sensor, per flight, is required, the system server spatially and temporally co-locates the enhanced data for each flight, for further analysis by the system server.

An advantage of this aspect of the invention is that, for example, when creating a map of a large area, smaller maps generated from separate micro-drone flights can be stitched together using computer vision techniques. This analysis may be accomplished by real-time, on-line processing by the system server or by off-line processing. The analysis can also be sent to a third party’s server for further processing.

Suitably, where the micro-drone does not have a Global Positioning System (GPS) capability, the position of the micro-drone may be calculated from the GPS coordinates of the observer’s personal computing device.

The advantage of using the GPS coordinates of the observer’s personal computing device is that a basic micro-drone without GPS can be used by the system.

Preferably, as the observer moves around the site the location of the observer’s personal computing device, is communicated to the system server for geofencing and a warning is communicated to the observer’s personal computing device and the remote user’s computing device if the observer enters a restricted area or the micro-drone enters into restricted airspace.

An advantage of this aspect of the invention is that the location of the micro-drone can be inferred from the GPS coordinates of the observer’s personal computing device.

Suitably, when more than one micro-drone is in flight at the same time, the micro-drones are programmed to fly autonomously and the system client software installed on the observer’s personal computing device monitors the flight of each micro-drone and implements drone swarming and collision avoidance measures, including sending a message to the observer’s personal computing device alerting the observer to the flight of a particular micro-drone.

An advantage of this aspect of the invention is that, in such a situation, as with more micro-drones in the air at the same time, it would be next to impossible for the observer to keep track of each micro-drone visually. So, monitoring of the micro-drones by the system client software, including alerting the observer to a potential hazard, makes such multi drone operations possible.

Preferably, the system provides Representational state transfer (REST) Application Programming Interfaces (APIs) and Webhooks to third parties for system integrations.

An advantage of this aspect of the invention is that it facilitates the integration of a customer’s in-house system with the system for real-time remote viewing in accordance with the invention. This integration is feasible as no trained drone pilots are required for operation of the system.

Further, preferably, the APIs are authorized with JavaScript Object Notation (JSON) Web Tokens RFC 7519.

An advantage of the use of JSON Web Tokens is that their use is a simple but secure method for authorising use of the system by a remote viewer through his company’s in-house system.

Each flight of the micro-drone is also authorized by a token containing information about the Time to Live of the remote viewing session, the session id, and the micro-drone registration number, such that unauthorized flights can be prevented, and inappropriate use of remote viewing can be traced.

The invention will be further illustrated by the following description of an embodiment thereof, given by way of example only with reference to the accompanying drawing in which:

Fig. 1

is schematic representation of a system for real-time remote viewing of a site in accordance with the invention;

Referring to , there is illustrated generally at 10, a system for real-time remote viewing of a site in accordance with the invention. The system 10 comprises a micro-drone 11, a camera 12 mounted on the micro-drone 11, an observer’s personal computing device 13, operable by an observer, with both the observer’s personal computing device 13 and the micro-drone 11 being locatable at the site. The system 10 also includes a remote viewer’s personal computing device 14, operable by a remote viewer, and a system server 15, which system server 15 manages the operation of the system 10, processes data collected by the micro-drone 11 during a flight, and communicates the processed data to the observer’s personal computing device 13 and the remote viewer’s personal computing device14.

The system 10 has a Web Real-Time Communication (WebRTC) connection, indicated by dashed line 16, between the observer’s personal computing device 13 and the system server 15 for audio, video and data transmission, and a WebRTC connection, indicated by dashed line 17, between the remote viewer’s personal computing device 14 and the system server 15, for audio, video and data transmission. The system 10 also has a WebRTC connection, indicated by dashed line 18, between the remote viewer’s personal computing device 14 and observer’s personal computing device 13 for audio, video and data transmission directly.

The micro-drone 11 is a low-speed, palm-size micro-drone, with a gross weight of < 250g and flown by either the remote viewer or the observer within Visual Line of Sight (VLOS) of the observer, with control of the micro-drone 11 by the remote viewer being limited to three commands, being go-forward, yaw-left, and yaw-right, with the observer being able to take-over control of the micro-drone 11 at any time during the flight. The micro-drone used is a Tello manufactured by DJI, DJI Headquarters Building, No.55 Xianyuan Road, Nanshan District, Shenzhen, China. The Tello micro-drone has a gross weight of 85g and is configured so that the maximum speed of the micro-drone 11 is limited to within a range of 80 to 120cm/s, and the micro-drone 11 is required to be flown within a radial distance of between 2 to 10m from the location of the observer’s personal computer device 13.

A sensor 19 is mounted on the micro-drone 11 and the data collected by the micro-drone 11 during a flight includes data from the camera 12 and data from the sensor 19, if required.

The micro-drone 11 does not have Bluetooth^®capability so the system 10 has a relay box 20, locatable in proximity to the observer, which relay box 20 is a compact computing device with at least two Wi-Fi interfaces 21, 22 and a Bluetooth^® interface 23, and which relay box 20 enables a wireless connection, indicated by dashed line 24, between the micro-drone 11 and the relay box 20 and wireless connections, indicated by dashed line 25, between the relay box 20 and the observer’s personal computing device 13.

It will be understood that in another embodiment of the invention, where the micro-drone 11 has Bluetooth^®capability, the relay box 20 is not required as communication between the observer’s personal computing device 13 and the micro-drone 11 can be by a direct Wi-Fi connection, indicated by dashed line 26.

The system server 15 processes the data collected by the micro-drone 11 by employing machine learning models, pretrained with large amounts of high-quality measurements, to rectify and enhance the data captured before it is transmitted to the observer’s personal computing device 13 and the remote viewer’s personal computing device 14.

Where a number of separate micro-drone flights using the same camera 12 and/or sensor 19 or a different camera or sensor, per flight, is required, the system server 15 spatially and temporally co-locates the enhanced data for each flight, for further analysis by the system server 15.

The micro-drone 11 does not have Global Positioning System (GPS) capability. However, the position of the micro-drone 11 may be calculated from the GPS coordinates of the observer’s personal computing device 13. In use, as the observer moves around the site the location of the observer’s personal computing device 13 is communicated to the system server 15 for geofencing and a warning is communicated to the observer’s personal computing device 13 and the remote user’s computing device 14 if the observer enters a restricted area or the micro-drone 11 enters into restricted airspace.

The following is a description of the use of the system in accordance with the invention, and the reference numerals of technical features in are used, where appropriate.

Micro-drone Connection Setup

The micro-drone 11 is positioned alongside the observer. The observer connects to the micro-drone 11 through Bluetooth^® and Wi-Fi using a client software installed on the observer’s personal computing device 13. The software sets up the wireless connection to the micro-drone in any Wi-Fi network with the following steps:

If a saved IP address of the micro-drone is found, go to 6.
Turn on the Bluetooth^® on the observer’s device and start scanning nearby Bluetooth^®-enabled devices.
Find the platform micro-drone and set up the Bluetooth^® connection to the micro-drone.
Input Wi-Fi credentials and send them to the micro-drone via the Bluetooth^® connection.
The micro-drone joins the Wi-Fi network with received credentials and sends its IP address to the observer’s device via the Bluetooth^® connection.
The observer’s device sets up a socket connection to the micro-drone with the micro-drone IP address and sends the observer’s device IP address to the micro-drone via the Wi-Fi connection.
The micro-drone sets up a socket connection to the observer’s device with the received IP address and sends the micro-drone IP address to the observer’s device again via the Wi-Fi connection.
If the IP received from the Wi-Fi connection on the observer’s device is the same as the saved IP or the IP received from the Bluetooth^® connection, the wireless connection between the observer endpoint and the micro-drone is set up, and the micro-drone IP address is saved on the observer’s device.
If the observer’s device cannot set up a socket connection to the micro-drone or the received IP address from the Wi-Fi connection does not match the IP received from the Bluetooth^® connection, go to 2.

The micro-drone 11 may not have Bluetooth^®connection but only Wi-Fi on board and is configured to a fixed IP address (as is the case with the embodiment of ). In this case, a relay box 20 is introduced. The relay box 20 is a small battery-powered single-board computer with two Wi-Fi interfaces 21, 22 and one Bluetooth^® interface 23. One Wi-Fi interface 21 connects to the micro-drone Wi-Fi with a given SSID and password. An IP address ip-drone is assigned to the relay box 20 while the micro-drone 11 has a fixed IP. Another Wi-Fi interface 22 together with the Bluetooth^® interface 23 is used to join the same Wi-Fi network that the observer’s personal computing device 13 is using, and obtains an IP address ip-observer. The second Wi-Fi connection 22 is set up with the same steps as setting up a micro-drone 11 connection to the observer’s personal computing device 13 directly, except by replacing the micro-drone 11 with the relay box 20 in the steps. With the ip-drone and ip-observer addresses, the control commands, drone videos, and sensor data can be relayed between the micro-drone 11 and the observer’s personal computing device 13.

Alternatively, where there is no Internet availability through Wi-Fi at the site, the relay box 20 includes a USB interface through which the relay box is connectable to the observer’s personal computing device 13 by a USB cable. The relay box 20 connects to the observer’s personal computing device 13 through the USB cable and sets up a Wi-Fi connection with the micro-drone 11 and the Internet is accessible through a 4G/5G network connection on the observer’s personal computing device 13.

Remote Viewing

The remote viewer connects to the system server 15 through the WebRTC connection 17 from the remote viewer’s personal computing device 14, using the Internet. The observer connects to the system server 15 through the WebRTC connection 16 from through a platform client software installed on the observer’s personal computing device 13, using the Internet. The remote viewer or viewers, as the case may be, and the observer are located at different geographic locations. With WebRTC connections 16 and 17, the remote viewer(s) and the observer can have a two-way videoconference in sound and vision. The remote viewer(s) can also communicate with each other in the videoconferencing. The system server authenticates, relays, and processes data between the remote viewer(s) and the observer.

Alternatively with the system server’s authentication, the remote viewer(s) and the observer can communicate directly with WebRTC connection 18. Communication among the remote viewer(s) can also be set up directly using WebRTC without being bridged by the system server.

Once the connection 26 between the micro-drone 11 and the observer’s personal computing device 13 is set up, the video captured by the micro-drone camera 12 is streamed to the observer’s personal computing device 13. If the relay box 20 is used, the video is streamed to the relay box 20 via the Wi-Fi connection 24 and then forwarded to the observer’s personal computing device 13 by the relay box 20 via the Wi-Fi connection 25. The observer’s personal computing device 13 receives the micro-drone video and forwards it to the remote viewers’ personal computing device 14 via WebRTC connections 16 and 17, or via WebRTC connection 18, directly. The micro-drone 11 video received by the remote viewers’ personal computing device 14 is displayed on a system web interface. Micro-drone control buttons are overlayed on the displayed video. The remote viewers can then control the micro-drone 11 flight using these web buttons, and each remote viewer, one at a time, is given sole ownership over these buttons. No physical control console such as a joystick is needed. The control commands executed from those web buttons are transmitted to the observer’s personal computing device 13 via the WebRTC connections 16 and 17, or 18 directly, then forwarded to the micro-drone via the Wi-Fi connections 24 and 25 or 26 directly. The system’s client software installed on the observer’s personal computing device 13 also displays the micro-drone video on the software interface to ease the flight monitoring for the observer while the observer is video conferencing with the remote viewer(s) at the same time. The client software can take flight control inputs from a keyboard, a computer mouse, a joystick, a touch screen, voice recognition, gesture recognition, etc., and these control commands executed by the observer are forwarded to the micro-drone 11 via the Wi-Fi connections 24 and 25 or 26 directly. The observer and the micro-drone 11 are at the same site, and the observer visually monitors the micro-drone 11 flight. The observer can interfere with the remote control from the remote viewer(s), by executing control commands from the client software, when a potential hazard is recognized by the observer. These control commands from the observer have higher priorities than those control commands from the remote viewer(s). In other words, when both commands from the observer and the remote viewers present, the commands from the observer will be forwarded to the drone.

The micro-drone 11 flight status and captured sensor data are also sent to the observer’s personal computing device 13 via the Wi-Fi connections 24 and 25 or 26, directly. Some of the flight status is displayed by the client software by overlaying the parameters on the drone video. The micro-drone 11 flight status and the sensor data are transmitted to the system server 15 together with the drone video via the WebRTC connection 16 for processing using computer vision and machine learning algorithms running on the system server 15. The processed results can be sent to both the observer and the remote viewer(s) via the WebRTC connections 16 and 17 and be displayed on the web interface and the client software interface.

Small, Light, Low-speed Micro-drone

Both the remote viewer(s) and the observer can be anyone with little experience in drone flying. This requires the Micro-drone to be small, light, and low-speed from safety concerns, therefore the platform deploys micro-drones in the service, which are palm-size with a weight of less than 250g. This weight includes the camera and all the other sensors mounted on the micro-drone 11. The speed of the micro-drone is fixed under 100 cm/s which is a very low speed.

“Stop, Look, Fly” Scheme

Network delays can raise the safety concerns for the remote viewer, therefore a “Stop, Look, Fly” scheme is employed. The implementation of the scheme limits the flight control for the remote viewer(s) to only three commands, go-forward, yaw-left, and yaw-right, which are executed mutually exclusively. No other commands, such as go-backward, speed-change, roll, follow-me, etc., are available. A remote viewer can watch the area in front of the micro-drone 11 and execute the go-forward command when there is no potential hazard. To change the direction of the micro-drone 11, the remote viewer waits until the micro-drone 11 stops and hovers in the air, then executes a yaw-left or yaw-right command, and checks around. When there is no risk in one direction the remote viewer can give the go-forward command and continue flying in the new direction. Such a procedure repeats during the entire remote piloting of the micro-drone 11.

Visual Line of Sight Compliance

The micro-drone 11 must fly within Visual Line of Sight (VLOS) of an operator. To comply with this provision for remote viewing, an observer situates alongside the micro-drone to maintain visual contact with the micro-drone 11 during a flight. The observer can be anyone above 16 years old and it is not necessary for him to be a trained pilot.

The distance between the observer and the micro-drone 11 is measured based on the strength of the Bluetooth^® and Wi-Fi signals, in the absence of (GPS) capability on the micro-drone 11. The micro-drone 11 is flown around the observer within a pre-set radius. The radius should be less than 10 meters due to the Bluetooth^®effective range for most Bluetooth^®devices in practice.

Hazard Monitoring

It is the observer’s responsibility to take evasive action in time to avoid hazards such as running out of battery, colliding with objects, etc., therefore there are more micro-drone 11 control commands provided in the client software for the observer. Observing the surroundings allows the observer to decide when and where to execute take-off or land commands. Besides the go-forward and yaw commands, the observer can also control the micro-drone using go-backward and roll commands. These commands can overwrite the remote viewer’s commands to avoid collisions. The observer can adjust the hovering height of the micro-drone with go-up and go-down commands and can also execute some AI functions such as follow-me from the client software to ease remote viewing.

Geofencing

Where there is no (GPS) capability on the micro-drone, the geofencing can be achieved by checking the observer’s location to avoid restricted areas and comply with altitude limitations. This is because the micro-drone 11 is within a very short distance of the observer. The observer’s location can be the GPS location from the observer’s device or be inferred based on the IP address of the device or the cell tower that the observer’s device connects to, etc. The observer’s location is not fixed because the observer can keep moving during remote viewing. The observer’s location is sent to a look-up table on the system server 15 through the connection 16 for geofencing. If a location is forbidden, a warning will be displayed on the client software installed on the observer’s personal computing device 13.

Privacy and Data Protection

Although it is the observer’s responsibility to respect the data protection and privacy of others, the system provides some mechanism to minimize the risks to privacy and to the protection of personal data. All data and videos transmitted on the system through the Internet are encrypted. No streamed video and data from the videoconferencing are saved. The non-permitted faces accidentally captured in the micro-drone 11 videos are blurred locally by the client software on the observer’s personal computing device 13 before being sent to the system server 15 for processing.

Moreover, each remote viewing session has a booking of the viewing time on the platform. The viewing time together with the observer’s account information is used for generating a time-based token. The platform authorizes each micro-drone 11 flight with this time-based token to prevent illicit use of remote viewing. Because the observer is within a very short distance of the micro-drone 11 for flight supervision, with the viewing time and the observer’s location having been recorded, an event of privacy and data protection violation can be easily traced back.

Platform Integration

The remote viewing scenarios of the system are agnostic. To make it versatile, the system provides REST APIs and Webhooks to third parties for system integrations. The APIs are authorized with JSON Web Tokens RFC 7519. With these APIs a third party can make remote viewing through the platform but have their own viewing session management.

Data Analysis

The remote viewing can be complemented with inspection or aerial survey. Due to the size, weight and battery limitations of a micro-drone, the inspection or the survey may require several rounds of remote viewing by a micro-drone 11 with different sensors in each flight. All these sensor data are transmitted to the system server 15 via connections 24, 25 and 26 or 26 and 16, bridged by the observer’s personal computing device 13. The micro-drone 11 onboard sensor(s) 19 are generally small low-cost sensors with low-quality measurements. On the system server 15, machine learning models pretrained with large amounts of high-quality measurements are used to rectify and enhance those low-quality sensor data captured and output measurements of much better quality. Moreover, these enhanced data are spatially collocated, and temporally collocated on the system server 15 for data analysis. For example, mapping of large areas can be achieved by stitching small maps generated from several flights of the micro-drone 11 using computer vision techniques. The analysis can be either real-time on-line processing or off-line processing. The results are transmitted to the remote viewers’ web interface via connection 17 and the observer’s client software via connection 16 for display. The results can also be sent to third parties’ servers for further processing.

Claims

A system for real-time remote viewing of a site, the system comprising
a micro-drone,
a camera mounted on the micro-drone,
an observer’s personal computing device, with a system client software package installed thereon, with both the observer’s personal computing device and the micro-drone being locatable at the site, in use, and in communication with each other by a wireless connection,
a remote viewer’s personal computing device,
a system server, which system server manages the operation of the system, processes data collected by the micro-drone during a flight, and communicates the processed data to the observer’s personal computing device and the remote viewer’s personal computing device,
a Web Real-Time Communication (WebRTC) connection between the observer’s personal computing device and the system server, for audio, video and data transmission,
a WebRTC connection between the remote viewer’s personal computing device and the system server, for audio, video and data transmission,
characterised in that, in use,
the micro-drone is a low-speed, palm-size micro-drone, with a gross weight of < 250g and flown by either the remote viewer or the observer within Visual Line of Sight (VLOS) of the observer, with control of the micro-drone by the remote viewer being limited to three commands being go-forward, yaw-left, and yaw-right, with the observer being able to take-over control of the micro-drone at any time during the flight, and
wherein the system has a WebRTC connection between the remote viewer’s personal computing device and the observer’s personal computing device for direct audio, video and data transmission.
A system for real-time remote viewing of a site according to Claim 1, wherein the maximum speed of the micro-drone is limited to within a range of 80 to 120cm/s, and the micro-drone is required to be flown within a radial distance of between 2 to 10m from the location of the observer’s personal computer device.
A system for real-time remote viewing of a site according to Claim 1 or 2, wherein a sensor is mounted on the micro-drone and the data collected by the micro-drone during a flight includes data from the camera and data from the sensor.
A system for real-time remote viewing of a site according to any one of Claims 1 to 3, wherein the wireless connection between the observer’s personal computing device and the micro-drone has a Bluetooth^® connection.
A system for real-time remote viewing of a site according to any one of Claims 1 to 3, wherein, where the micro-drone does not have Bluetooth^®capability, the system has a relay box, locatable in proximity to the observer, which relay box is a compact computing device with at least two Wi-Fi interfaces and a Bluetooth^® interface, and which relay box enables a wireless connection between the micro-drone and the relay box and a Wi-Fi connection and a Bluetooth^®connection between the relay box and the observer’s personal computing device.
A system for real-time remote viewing of a site according to Claim 5, wherein, where there is no Internet availability through Wi-Fi at the site, the relay box includes a USB interface through which the relay box is connectable to the observer’s personal computing device by a USB cable.
A system for real-time remote viewing of a site according to any one of Claims 3 to 6, wherein the system server processes the data collected by the micro-drone by employing machine learning models, pretrained with large amounts of high-quality measurements, to rectify and enhance the data captured before it is transmitted to the observer’s personal computing device and the remote viewer’s personal computing device.
A system for real-time remote viewing of a site according to Claim 7, wherein, where a number of separate micro-drone flights using the same camera and/or sensor or a different camera and/or sensor, per flight, is required, the system server spatially and temporally co-locates the enhanced data for each flight, for further analysis by the system server.
A system for real-time remote viewing of a site according to any preceding claim, wherein, where the micro-drone does not have a Global Positioning System (GPS) capability, the position of the micro-drone may be calculated from the GPS coordinates of the observer’s personal computing device.
A system for real-time remote viewing of a site according to any preceding claim, wherein, as the observer moves around the site the location of the observer’s personal computing device is communicated to the system server for geofencing and a warning is communicated to the observer’s personal computing device and the remote user’s computing device if the observer enters a restricted area or the micro-drone enters into restricted airspace.
A system for real-time remote viewing of a site according to any preceding claim, wherein, when more than one micro-drone is in flight at the same time, the micro-drones are programmed to fly autonomously and the system client software installed on the observer’s personal computing device monitors the flight of each micro-drone and implements drone swarming and collision avoidance measures, including sending a message to the observer’s personal computing device alerting the observer to the flight of a particular micro-drone.
A system for real-time remote viewing of a site according to any preceding claim, wherein the system provides Representational state transfer (REST) Application Programming Interfaces (APIs) and Webhooks to third parties for system integrations.
A system for real-time remote viewing of a site according to Claim 12, wherein the APIs are authorized with JavaScript Object Notation (JSON) Web Tokens RFC 7519.