WO2024091926A1

WO2024091926A1 - Enhanced functions of an unmanned aerial vehicle including audio synchronization and geofencing

Info

Publication number: WO2024091926A1
Application number: PCT/US2023/077620
Authority: WO
Inventors: Noah CALLAWAY; Matthew ERHART; Matthew Rabinowitz; Jonathan SHEENA; Gary S. GREENBAUM
Original assignee: Natureeye, Inc.
Priority date: 2022-10-24
Filing date: 2023-10-24
Publication date: 2024-05-02

Abstract

Functions of an unmanned aerial vehicle (UAV) can be location dependent. In some embodiments, a digital audio communications channel is synchronized and multiplexed with a high-definition video stream, captured from a user-controlled remote UAV. The accompanying audio stream is used to provide supplemental and enriching information to an end user who is controlling the actions of a remote UAV. This system can be used to provide a virtualized tourism experience at locations of high interest such as historic settlements, wildlife conservancies, and iconic destinations. In some embodiments, a virtual three-dimensional geometry, with a boundary surface commonly referred to as a geofence, and an accompanying control system restrict the flight path of a remote piloted UAV to the interior geometry.

Description

ENHANCED FUNCTIONS OF AN UNMANNED AERIAL VEHICLE INCLUDING AUDIO SYNCHRONIZATION AND GEOFENCING

BACKGROUND

[0001] Quadcopter UAVs, known as drones, are often equipped with high-definition video cameras with the capability to transmit captured digital video in real-time to a remote controller device. It has also been described in PCT/US2021/032011 a system and method for transmitting a real-time video stream over a wide-area-network (WAN) to a remote end user who can manipulate the UAV controls that include both the movement of the drone as well as the direction and focus on the attached imaging devices. See International Application No. PCT/US2021/032011, filed May 12, 2021, the entire contents of which are hereby incorporated by reference.

[0002] Quadcopter UAVs are increasingly popular for commercial and private use. As the frequency and quantity of drone activity increases, airspace congestion grows exponentially, leading to various adverse consequences such as a higher risk for in-air collision and collateral on-ground damage to property and persons and increased noise disturbance for humans and wildlife. To mitigate these adverse consequences, national aviation authorities have instituted laws governing the allowable drone airspace. These restrictions are mostly focused in and around areas of high aircraft traffic, in noise sensitive regions such as national parks, and above large assemblies of people, i.e., public stadiums. With most drones now equipped with imaging devices, such as a video camera, aviation authorities have also limited drone airspace in or around areas of national interest such as military bases, government buildings, utility plants, nuclear power stations and dams. Moreover, legal challenges involving encroachment on privacy and copyright infringement are further advancing drone airspace restrictions.

SUMMARY

[0003] This Summary introduces a selection of concepts in simplified form that are described further below in the Detailed Description. This Summary neither identifies key or essential features, nor limits the scope, of the claimed subject matter.

[0004] Various enhanced functions of an unmanned aerial vehicle (UAV) can be provided, such as functions dependent on the location of the UAV, synchronization of audio data from an additional source with a real-time video stream from the UAV, or taking over remote control of the movement of the UAV by another individual or system. [0005] In some embodiments, a digital audio communications channel is synchronized and multiplexed with a video stream captured by a user-controlled remote unmanned aircraft vehicle and transmitted by the UAV to provide a virtualized travel experience. The audio data provided by the digital audio communications channel can be provided by capturing an individual’s live narration, or by a computing device generating the audio data, or by a computing device selecting from recorded audio data based on the location of the UAV or based on information about objects within the real-time video stream from the UAV. The accompanying audio stream is used to provide supplemental and enriching information to an end user who is controlling the actions of a remote UAV. This system can be used to provide a virtualized tourism experience at locations of high interest such as historic settlements, wildlife conservancies, and iconic destinations. An individual narrating the real-time video stream from the UAV may take over control of the UAV from the end user. [0006] In some embodiments, a three-dimensional spatial region defines the allowable airspace for navigating unmanned aerial vehicles. The location of the UAV can be an input to the control system for the UAV and can be used to maintain the UAV within the allowable airspace. In some embodiments, a virtual three-dimensional geometry, with a boundary surface commonly referred to as a geofence, and an accompanying control system, restrict the flight path of a remote piloted unmanned aerial vehicle (“UAV”) to the interior geometry of the geofence. The control system can take over control of the UAV from the end user to ensure the UAV stays within the interior geometry of the geofence.

[0007] Accordingly, in one aspect, a computing device is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. The computing device for use in such a system comprises a processing system including processing circuitry and a memory storing computer program instructions. The computer program instructions configuring the computing device to track spatial coordinates for a geographic location of the unmanned aerial vehicle, and generate information based on the geographic location of the unmanned aerial vehicle location to be transmitted to the end user computing device. [0008] In one aspect, a method is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits real-time imaging data as a video transmission over the wide-area network. The method for use in such a system includes tracking spatial coordinates for a geographic location of the unmanned aerial vehicle, and generating information based on the geographic location of the unmanned aerial vehicle location to be transmitted to the end user computing device.

[0009] In one aspect, a system includes an unmanned aerial vehicle, a remote controller device, an end user computing device, and an audio guide computing device. The unmanned aerial vehicle and the remote controller communicate through a radio network. The remote controller device, the end user computing device, and the audio guide computing device communicate through a wide-area network. The unmanned aerial vehicle includes an imaging sensor and a radio. The radio is connected to transmit real time imaging data from the imaging sensors over the radio network and to receive control data from the radio network. The remote controller device includes a radio tuned to receive the real-time imaging data from the unmanned aerial vehicle over the radio network. The remote controller device is configured to transmit the control data to the unmanned aerial vehicle over the radio network, and to transmit a video transmission based on the received real-time image data over the wide-area network. The end user computing device includes one or more presentation devices. The end user computing device is configured to transmit instructions to the remote controller device to generate the control data for transmission to the unmanned aerial vehicle. The audio guide computing device transmits supplemental audio data. The end user computing device receives the video transmission and the supplemental audio data synchronized and multiplexed over a communications channel of the wide-area network, and presents the received video transmission and supplemental audio data through the one or more presentation devices.

[0010] In one aspect, a method comprises transmitting over a radio network, by an unmanned aerial vehicle, real time imaging data from imaging sensors of the unmanned aerial vehicle; receiving over the radio network, by the unmanned aerial vehicle, control data from the radio network; receiving from the radio network, by a remote controller device, the real time imaging data; transmitting over the radio network, by the remote controller device, the control data; transmitting over a wide-area network, by an end user computing device, instructions to the remote controller device to generate the control data for the unmanned aerial vehicle; providing, by an audio guide computing device, supplemental audio data; synchronizing and multiplexing on a communications channel of the wide-area network a video transmission based on the real-time imaging data and the supplemental audio data for transmission over the communications channel to the end user computing device; and presenting the video transmission and the supplemental audio data through one or more presentation devices of the end user computing device.

[0011] In one aspect, an audio guide computing device is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. For such a system, the audio guide computing device comprises a processing system including processing circuitry and a memory storing computer program instructions. The computer program instructions configuring the audio guide computing device to provide supplemental audio data, and to synchronize and multiplex the supplemental audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.

[0012] In one aspect, a method is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. The method for use in such a system includes providing supplemental audio data, and synchronizing and multiplexing the supplemental audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device. [0013] In one aspect, a system includes an unmanned aerial vehicle, a remote controller device, an end user computing device and a geofencing computing device. The unmanned aerial vehicle and the remote controller device communicate over a radio network. The remote controller device, end user computing device, and the geofencing computing device communicate over a wide area network. The unmanned aerial vehicle includes an imaging sensor and a radio. The radio is connected to transmit real time imaging data from the imaging sensors over a radio network and to receive control data from the radio network. The remote controller device includes a radio tuned to receive the real-time imaging data from the unmanned aerial vehicle over the radio network. The remote controller device transmits the control data to the unmanned aerial vehicle over the radio network, and transmits the received a video transmission based on the real-time image data over the wide-area network. The end user computing device includes one or more presentation devices. The end user computing device is configured to transmit instructions to the remote controller device to generate the control data for transmission to the unmanned aerial vehicle. The geofencing computing device creates a representation of a 3 -dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence, and tracks spatial coordinates for a geographic location of the unmanned aerial vehicle. The geofencing computing device determines a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

[0014] In one aspect, a method includes: transmitting over a radio network, by an unmanned aerial vehicle, real time imaging data from imaging sensors of the unmanned aerial vehicle; receiving over the radio network, by the unmanned aerial vehicle, control data from the radio network; receiving from the radio network, by a remote controller device, the real time imaging data; transmitting over the radio network, by the remote controller device, the control data; transmitting over a wide-area network, by an end user computing device, instructions to the remote controller device to generate the control data for the unmanned aerial vehicle; creating a representation of a 3 -dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence; tracking spatial coordinates for a geographic location of the unmanned aerial vehicle; and determining a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence. [0015] In one aspect, a geofencing computing device is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. The geofencing computing device includes a processing system including processing circuitry and a memory storing computer program instructions. The computer program instructions configure the geofencing computing device to create a representation of a 3 -dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence. The geofencing computing device further tracks spatial coordinates for a geographic location of the unmanned aerial vehicle. The geofencing computing device further determines a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence. [0016] In one aspect, a method is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. The method includes creating a representation of a 3- dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence. The method includes tracking spatial coordinates for a geographic location of the unmanned aerial vehicle. The method includes determining a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

[0017] In one aspect, a geofencing computing device is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits real-time imaging data as a video transmission over the wide-area network. The geofencing computing device includes a processing system including processing circuitry and a memory storing computer program instructions. The computer program instructions configure the geofencing computing device to create a representation of a 3 -dimensional spatial region defining an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence. The minimum above the ground level for each point within the interior of the geofence is determined by a viewshed analysis based on three dimensional coordinates for the geographic location of the remote controller device and an elevation map of the geographic terrain of the geofence region. The geofencing computing device tracks three-dimensional spatial coordinates for a geographic location of the unmanned aerial vehicle. The geofencing computing device determines a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

[0018] In one aspect, a method is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits real-time imaging data as a video transmission over the wide-area network. The method includes creating a representation of a 3 -dimensional spatial region defining an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence. A minimum above the ground level for each point within the interior of the geofence is determined by a viewshed analysis based on 1) three dimensional coordinates for the geographic location of the remote controller device and 2) an elevation map of the geographic terrain of the geofence region. The method includes tracking three-dimensional spatial coordinates for a geographic location of the unmanned aerial vehicle. The method includes determining a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

[0019] In one aspect, a computing device is for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. The computing device includes a processing system including processing circuitry and a memory storing computer program instructions. The computer program instructions configure the computing device to generate text describing content of the video transmission based on a large language model. The computing device generates audio data corresponding to the generated text using a text-to-speech engine. The supplemental audio data is synchronized and multiplexed with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.

[0020] In one aspect, a method for use in a system with a remote controlled unmanned aerial vehicle. The unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device. The remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle. The remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network. The method includes generating text describing content of the video transmission using a large language model. The method includes generating audio data corresponding to the generated text using a text-to-speech engine. The method includes synchronizing and multiplexing the audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.

[0021] Any of the foregoing aspects can include one or more of the following features. The unmanned aerial vehicle is a quadcopter drone. A low orbit satellite provides a direct radio link from the remote controller device to the unmanned aerial vehicle. A cellular tower provides a direct radio link between the remote controller device and the unmanned aerial vehicle. The remote controller device is configured to function as a network bridge between the radio network and a wide-area network.

[0022] Any of the foregoing aspects can include one or more of the following features. One or more additional audio guide computing devices are connected to the wide-area network, wherein each audio guide computing device provides respective supplemental audio data to be synchronized and multiplexed with the video transmission from the unmanned aerial vehicle. The supplemental audio data comprises live audio data of a narration by an audio guide captured though a microphone for the audio guide computing device of the audio guide. The respective supplemental audio data transmitted by the one or more additional audio guide computing devices comprises data from a file local on the additional audio guide computing device. Transmission of the supplemental audio data is triggered by the geographical location of the unmanned aerial vehicle. The remote controller device incorporates the audio guide computing device.

[0023] Any of the foregoing aspects can include one or more of the following features. An artificial intelligence system is configured to identify an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data. The artificial intelligence system is configured to assume control of the unmanned aerial vehicle to follow the object having the identified image signature. The artificial intelligence system is configured to assume control of the unmanned aerial vehicle and the imaging sensors to focus on the object having the identified image signature. The artificial intelligence system is configured to insert a graphic overlay on the video transmission based on the object having the identified image signature. The artificial intelligence system is configured to trigger transmission of audio information associated with the object having the identified image signature.

[0024] Any of the foregoing aspects can include one or more of the following features. The artificial intelligence system or the audio guide computing device can include a large language model configured to generate text describing content of the video transmission. A text-to-speech engine generates audio corresponding to the generated text, wherein the supplemental audio data includes the generated audio.

[0025] Any of the foregoing aspects can include one or more of the following features. A notification is generated to the end user computing device based on the determined spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

[0026] In any of the foregoing, a minimum above the ground level for each point within the interior of the geofence can be determined by a viewshed analysis based on 1) three dimensional coordinates for the geographic location of the remote controller device and 2) an elevation map of the geographic terrain of the geofence region.

[0027] Any of the foregoing aspects can include one or more of the following features. Determining the spatial relationship includes determining whether the geographical location of the unmanned aerial vehicle is either in the exterior space or in the proximity of the geofence. Generating the notification includes storing a minimum threshold time and a minimum threshold distance and evoking the notification based on either of (i) a distance of the unmanned aerial vehicle to the geofence is less than or equal to the minimum threshold distance or (ii) a predicted time of the unmanned aerial vehicle to reach the geofence is less than or equal to the minimum threshold time. The minimum threshold time and minimum threshold distance for evoking a notification are dynamically computed values based on kinematics of the unmanned aerial vehicle. The minimum threshold distance is based on a maximum deceleration of the unmanned aerial vehicle in a direction normal to the geofence. The minimum threshold time is based on a maximum deceleration of the unmanned aerial vehicle in a direction normal to the geofence.

[0028] Any of the foregoing aspects can include one or more of the following features. The notification comprises a graphic representation overlaid on the video transmission sent from the remote controller device to the end user computing device.

[0029] Any of the foregoing aspects can include one or more of the following features. The remote controller device is caused to send control data to the unmanned aerial vehicle to decelerate in a direction away from the geofence to avoid navigating into the exterior space. The remote controller is so caused after evoking a notification.

[0030] Any of the foregoing aspects may be embodied as a computer system, as any individual component of such a computer system, as a process performed by such a computer system or any individual component of such a computer system, or as an article of manufacture including computer storage in which computer program code is stored and which, when processed by the processing system(s) of one or more computers, configures the processing system(s) of the one or more computers to provide such a computer system or individual component of such a computer system, or to implement such a method.

[0031] The following Detailed Description references the accompanying drawings which form a part of this application, and which show, by way of illustration, specific example implementations. Other implementations may be made without departing from the scope of the disclosure.

BRIEF DESCRIPTION OF THE FIGURES

[0032] Reference will now be made to the accompanying drawings, which are not necessarily drawn to scale. Some embodiments may include fewer or more components than those shown in the figures. [0033] Figure 1 is a diagram illustrating a system for an audio and video narrated virtual tour of a remote destination navigated by an end user-controlled drone.

[0034] Figure 2 illustrates a block diagram of example circuitry embodying a computing device that may perform various operations in accordance with some example embodiments described herein.

[0035] Figure 3 is a diagram illustrating a system for an audio and video narrated virtual tour of a remote destination navigated by an end user-controlled drone.

[0036] Figure 4 illustrates the geometric interpretation of a voxel, or rectangular prism used for creating the interior region.

[0037] Figure 5 represents geometry of multiple voxels used to define an example of allowable airspace for UAV navigation above a terrain.

[0038] Figure 6 illustrates an example graphic for an overlay on a UAV when it approaches a boundary voxel side.

[0039] Figure 7 illustrates a color-coded terrain map illustrating an example geofence.

[0040] Figure 8 is a graph of example output results of the addViewshedPath function where the orange line is the minimum AGL generated for this radial direction.

DETAILED DESCRIPTION

[0041] Various enhanced functions of an unmanned aerial vehicle (UAV) can be provided, such as functions dependent on the location of the UAV, synchronization of audio data from an additional source with a real-time video stream from the UAV, or taking over remote control of the movement of the UAV by another individual or system.

[0042] Audio synchronization will first be addressed.

[0043] Quadcopter UAVs, known as drones, can be equipped with video cameras with the capability to transmit captured digital video in real-time to a remote controller device. As described in PCT/US2021/032011, hereby incorporated by reference, a real-time video stream can be transmitted from a UAV over a wide-area-network (WAN) to a remote end user. The video cameras can be high-definition video cameras. The remote end user can manipulate the UAV controls that include both the movement of the drone as well as the direction and focus on the attached imaging devices. A digital audio communications channel is synchronized and multiplexed with the video stream which was captured from a user-controlled remote unmanned aircraft vehicle. The accompanying audio stream is used to provide supplemental and enriching information to an end user who is controlling the actions of a remote UAV. This system can be used to provide a virtualized tourism experience at locations of high interest such as historic settlements, wildlife conservancies, and iconic destinations.

[0044] With an accompanying audio guide(s) providing real-time commentary, the end user can be directed to specific areas of interest, be presented supplementary information about the location in which the drone is flying and be engaged in an informative dialogue with one or more audio guide experts who also have access to the UAV’s video transmission. Furthermore, an accompanying audio guide can temporarily assume control of the UAVs movement and its imaging devices to efficiently direct the end user’s attention to an area of interest. In another implementation, a stored audio file can be transmitted when the UAV is flying in the proximity of an area of interest, or an Al image recognition system can trigger said audio transmission after identifying a related image signature in the UAV’s transmitted video. The invention is not limited to audio data, as other media data can be generated or provided based on the location of the UAV.

[0045] Figure 1 depicts an example system for an audio and video narrated virtual tour of a remote destination navigated by an end user-controlled drone. 100 is an unmanned aerial vehicle (UAV) equipped with a video camera, preferably high-definition, and a means to transmit captured video data over a radio network.

[0046] 101 is a remote controller device that communicates with the UAV (100) and acts as a network bridge between the UAVs (100) and a wide-area-network (102). 102 is a wide- area-network that is usually referred to as the Internet. Management systems 103 is an in- network component that manages, provisions, and stores essential data and credentials used by the operations of this system. The management system 103 may include or may exchange data with an Al system 106, such as described below, for identifying digital acoustic and image signatures in the data transmitted by the unmanned aerial vehicle.

[0047] 104 is an end user computing device, such as a PC or mobile device, connected to the wide area network 102 which executes a software application that transmits user-control information to the UAV 100. The end user computing device 104 has access to an audio communications channel and receives video streaming from the UAV 100, all through the proxy provided by the remote controller device 101. There may be multiple end user computing devices.

[0048] 105 is an audio guide computing device with a network connection to the wide area network 102. This computing device executes a software application that transmits supplemental audio data to the end user computing device 104, the management systems 103, and the remote controller device 101. The audio guide computing device also receives data over the WAN 102 from the management systems 103. In some embodiments, a multiplicity of interconnected audio guide computing devices 105 can be used.

[0049] One class of unmanned aerial vehicles (UAV) are quadcopter drones. Drones have become commonplace for use by both businesses and consumers. Most drones are equipped with digital imaging devices that produce high-definition digital pictures and video. These digital images can be stored on the drone’s on-board storage units and, in most implementations, can be transmitted over a radio network link back to a remote controller device. Commercial entities can use these drones and their imaging capabilities in dangerous or physically challenging environments, such as investigating an advancing forest fire, checking for leaks in a gas pipeline, or active surveillance and pursuit of wildlife poachers.

[0050] In some implementations, the remote controller device acts as a network bridge between the drone’s attached radio network and a wide-area-network (WAN). In this case, the drone’s transmitted real-time image data is proxy by the remote controller over the WAN to a remote end user device. The end user can view this real-time video data from the remote device and transmit control information, such as changes to the flight path or camera direction and focus, back to the drone bridged by the remote controller.

[0051] Embodiments with the audio guide extend the drone transmitted video experience with the addition of an audio communications channel over a WAN that is accessible by the remote controller device, the end user device, and one or more other computing devices used primarily for broadcasting supplemental audio data that is synchronized and multiplexed with the drone’s transmitted video streams.

[0052] This configuration enables a virtual tourism scenario for the end user in the case that the computing devices are used by audio guides who can assist and educate the end user on the various points of high value interest consistent with the drone’s flight path. In this configuration, the end user can also communicate with a person operating the remotecontrol unit who can 1) participate as an audio guide for the flight, 2) serve as a remote pilot who can resume control of the drone in case of emergencies, and 3) act as a visual observer communicating local information such a proximity to other aircraft, a flock of birds encroaching into the drone’s airspace, or a weather front moving into the region.

[0053] In this same configuration, in some implementations, the devices used by the audio guides are provided priority access to the drone controls. This allows the audio guides to remotely assume control of the drone’s movement and its imaging devices, such as camera direction and the lens's magnification power. A use case for this configuration is an audio guide which has spotted a quickly moving animal or located a highly camouflaged animal in the dense flora and wants to guide the video experience to these areas of high value interest.

[0054] In some implementations, the audio guides communications are replaced by stored, contextually relevant, pre-recorded audio files on the audio guide devices. Each of the plurality of pre-recorded audio files individually corresponds to information pertinent to a specific point of interest within the geofenced boundary of the drone’s flight path. When the drone’s flight path is in the geographical proximity of a point of interest, the audio guide device will stream the geographically appropriate pre-recorded audio over the WAN to the end user device.

[0055] Additionally, these points of interest can be determined at the time of flight by an Al-based image processing system. This Al-based image processing system uses a pretrained object detection model with specified object classes, such as various animals. The system processes each captured video frame to map image signatures to object classes with confidence factors. If the confidence factor exceeds a predefined threshold, the system generates a notification that an image signature is consistent with a specific object class. In the case that image signature is consistent with multiple object classes, the system will select the object class with highest confidence level for identifying the object. An example of this algorithm written in Python using OpenCV, an open-source computer vision and machine learning software library, and the YOLO libraries for object detection is the following: import cv2

# Load a pre-trained YOLO model for animal detection including YOLO weights and cfg fde. yolo net = cv2.dnn.readNet("yolov3.weights", "yolov3.cfg")

# Define a list of animal classes that the model can detect. animal classes = ["rhino", "leopard", "giraffe", "elephant", "lion"] # Add more as needed.

# Open a connection to the live video stream (you may replace 'O' with your video source). cap = cv2. VideoCapture(0) while cap.isOpened() : ret, frame = cap.read() if not ret: break

# Perform object detection on the frame. blob = cv2.dnn.blobFromImage(frame, 0.00392, (416, 416), (0, 0, 0), True, crop=False) yolo net. setlnput(blob) outs = yolo net.forward(yolo net.getUnconnectedOutLayersNames(j)

# Process the detected objects. for out in outs: for detection in out: scores = detection[5:j class id = detection[ 1 ] confidence = scores [class id] if confidence > 0.5: # Adjust the confidence threshold as needed. class name = animal classes [int(class id)] print(f 'Detected {class name] with confidence: {confidence}") ifcv2.waitKey(l) & OxFF == ord('q)' : break cap.release() cv2. destroy A UWindows()

[0056] After the Al image processing system identifies an image signature in the drone’s streaming video transmission with an object class, the Al system may 1) navigate the drone to the identified area of interest, 2) assume control of the drone’s camera to focus on the identified area of interest, 3) imbedded a graphic overlay in the video transmission to aid in identifying said area of interest, and 4) trigger the transmission of a contextually relevant pre-recorded audio stream to the end user

[0057] For example, a pride of sleeping lions camouflaged by dense underbrush can be identified by said Al system in the drone’s video transmission. The Al system navigates the drone to hover over the sleeping lions, focus the video camera on the pride, highlight the pride with a graphic in the video transmission to aid the end user identification, and transmit the relevant pre-recorded audio stream describing lion sleep cycles. [0058] In some configurations, the human audio guide narration provided by the audio guide computing device is replaced by a large language model (“LLM”) combined with a text-to-speech engine. In some implementations, the LLM based narration is triggered by the aforementioned Al system used to identify an image signature in the drone streaming video transmission. The LLM narration and text to speech generation can be implemented on a computing device supporting the Al system 106, a computing device supporting the audio guide computing device 105, or other computing device.

[0059] To enable this function for providing a large language model (LLM) narration based on an Al-based object classification model, the below python code illustrates an implementation that uses the PYTTSx3 text-to-speech library, OpenAI’s LLM, OpenCV machine learning library, and the YOLO libraries for object detection: import cv2 import pyttsx3 # Text-to-speech library for basic narration import openai # GPT-3 API for advanced narration

# Initialize the video capture interface and YOLO model cap = cv2. VideoCapture(0) # Replace 'O' with the appropriate video source if needed. yolo net = cv2.dnn.readNet("yolov3.weights", "yolov3.cfg") # Adjust the YOLO model paths.

# Initialize the Text-to-Speech engine for basic narration. narration engine = pyttsx3.init()

# Initialize the OpenAI GPT-3 API (you need to have an API key and set up the OpenAI environment). openai. api key = 'YOUR API KEY' # Added API key configuration.

# Initialize an empty list to store detected animals. detected animals = [] # Added the detected animals list. while cap.isOpened() : ret, frame = cap.read() if not ret: break

# Pass the received frame through the YOLO model for object detection. blob = cv2.dnn.blobFromImage(frame, 0.00392, (416, 416), (0, 0, 0), True, crop=False) yolo net. setlnput(blob) outs = yolo net.forward(yolo net.getUnconnectedOutLayersNames(j) detected animals = [] # A list to store detected animals.

# Process the detected objects. for out in outs: for detection in out: scores = detection[5:j class id = detection[ 1 ] confidence = scores [class id] if confidence > 0.5: # Adjust the confidence threshold as needed. class name = animal classes [int(class id)] detected animals.append(class name) # Added storing detected animals without bounding boxes. if detected animals:

# Generate a full narrative about the detected animals using GPT-3. full narrative = "In this video, we found the following animals:\n" for class name in detected animals: prompt = f 'Describe the {class name] in the video frame and its relevance in the ecosystem. " description = openai.Completion.create( engine = "text-davinci-002 ", prompt=prompt, max tokens=200 # Adjust as needed for detailed descriptions.

) full narrative += description.choices[0].text.strip() + "\n"

# Use the Text-to-Speech engine to play the generated audio narration. narration engine, say (full narrative )

# Clear the list of detected animals. detected animals = []

# Display the video frame (without bounding boxes). cv2.imshow ("Video", frame) ifcv2.waitKey(l) & OxFF == ord('q)' : break cap.release() cv2. destroy A UWindows()

[0060] In all configurations mentioned above, the end user receives an audio-narrated video of a virtual tour experience.

[0061] Given that the audio and video streams may be created by a multiplicity of sources, any one of the above-mentioned devices (remote control unit, Al system, audio guide computing devices, LLM-generated narration and text-to-speech system, and end user device), or a dedicated device in a public or private cloud infrastructure component connected to the WAN, such as central system management server described in Figure 1 (103), may be used to multiplex the various streams into a single file for local storage. The archived file preserves, in digital form, the end user audio and video experience which can be downloaded over the WAN to the end user’s device for off-line playback and/or downloaded (shared) to other devices approved by the end user.

[0062] A drone service provider offers drones for remote piloting at scenic destinations, wildlife conservancies, historical attractions, and the like. To provide the supplemental audio accompaniment synchronized with a video transmission from a UAV uses the following components:

[0063] 1. A drone equipped with imaging sensors, e.g., high-definition video cameras, and a radio for transmitting the drone’ s captured imaging data and for receiving control data for directing the imaging sensors and drone navigation information.

[0064] 2. Remote Controller device: For Remote Controller devices that communicate to the drones using a radio operating in the unlicensed frequency bands (ISM), such as the 2.4GHz and 5.8GHz frequencies, most countries have instituted regulations that limit the effective isotropic radiated power (EIRP) for the transmitting radio to minimize interference with other radios operating in same ISM band. The EIRP limitation means that the effective spatial range for radio transmission and reception to the drone from the Remote Controller is limited, i.e., the noise to signal saturates the communications link making communication and control ineffective. To extend the spatial range for communications, the Remote Controller can be connected to a wide area network (WAN) and can support a network bridging functionally between the radio network with the WAN. If the drone radio uses another frequency band with less stringent EIRP restrictions, the drone may be connected directly to a WAN, for example, through a low orbit satellite or a remote cellular tower.

[0065] 3. End user computing device: The end user may use a computing device such as a PC or mobile device, connected to the WAN, to execute a software application that transmits control information to the drone, has access to an audio communications channel, and receives video streaming data from the drone, with all network data traffic bridged by the remote controller device described above.

[0066] 4. Remote Audio Guide Computing device (s): one or a plurality of computing devices that is connected to the WAN, that can, through an application, access the drone’s video transmission, remotely control the drone movement and camera, and access a communications channel connected to the End user computing device and the remote controller.

[0067] Geofencing for an UAV will now be addressed.

[0068] The allowed drone airspace, or interior geometry, can be defined by a three- dimensional space, circumscribed by a virtual two-dimensional boundary referred to as a virtual geofence. In embodiments involving 3-dimensional geometries consistent with commonly defined controlled airspace geometries, and for other simply connected, convex boundaries, the interior geometry can be approximately constructed by the union of a multiplicity of polygonal prisms. A specific implementation of these polygon prisms is rectangular prisms, commonly referred to as voxels. The voxel’s simple geometry is useful when determining which polygon prism contains the drone. A drone is deemed within the allowable airspace if its location is interior to one of the polygonal prisms; and similarly, the drone is deemed in the restricted, exterior, region if not located in one of the polygonal prisms.

[0069] In embodiments incorporating geofencing, a system and method 1) generates a virtualized geofence airspace defined as the boundary of the union of polygonal prisms, 2) assumes the UAV flight controls to assure containment within the interior geometry, i.e., the allowable airspace, and 3) and a graphical notification triggered by the UAV’s proximity to the external geometry, i.e., the restricted airspace.

[0070] With the increasing UAV airspace congestion, national aviation authorities are imposing drone airspace restrictions for 1) mitigating collisions with other aerial vehicles and damage to people, wildlife, and property; 2) reducing noise disturbance in sensitive regions such near a nursing eagle’s nest; and 3) averting flights on or around areas of high national interest such as military bases.

[0071] Other reasons for imposing drone airspace restrictions include collision avoidance with physical obstructions such as trees, mountains, and towers; reducing privacy incursions caused by the UAVs image capturing; and assisting UAV remote pilots with navigation around areas of interest, such as an iconic waterfall or a wildlife watering hole. [0072] Figure 3 illustrates an example system for an audio and video narrated virtual tour of a remote destination navigated by an end user-controlled drone, 300 is an unmanned aerial vehicle equipped with a video camera, preferably high-definition, and a means to transmit captured video data over a radio network.

[0073] 301 is a remote controller device that communicates with the UAV (300) and acts as a network bridge between the UAVs (300) and a wide-area-network (302). 302 is a wide- area-network that is usually referred to as the Internet. Management systems 303 is an in- network component that manages, provisions, and stores essential data and credentials used by the operations of this system. This management system may include or may exchange data with a geofencing system 306, such as described below. The geofencing system is used to 1) store a representation of the exterior, interior, and boundary regions for the UAV (300) navigation, 2) track the spatial coordinates of the unmanned aerial vehicle, 3) evoke a notification if the UAV’s location is either in the exterior space or in the proximity of the geofence, 4) generate a graphical warning superimposed on the UAV’s transmitted video, and 5) send control information to the UAV to avoid navigating through the geofence.

[0074] 304 is an end user computing device, such as a PC or mobile device, connected to the wide area network 302 which executes a software application that transmits user-control information to the UAV 300. The end user computing device has access to an audio communications channel and receives video streaming from the UAV 300, all through the remote controller device 301 acting as a proxy. There may be multiple end user computing devices.

[0075] 305 is an audio guide computing device with a network connection to the wide area network 302. This computing device executes a software application that transmits supplemental audio data to the end user computing device 304, the management systems 303, and the remote controller device 301. The audio guide computing device also receives data over the WAN 302 from the management systems 303. In some embodiments a multiplicity of interconnected audio guide computing device 305 can be used.

[0076] Figure 4 illustrates the geometric interpretation of a voxel, or rectangular prism used for creating the interior region. 400 represents a voxel with a top voxel base (401) and a bottom voxel base (402). 402 is a point represented by 2-tuple, i.e., latitude and longitude coordinates, which is a projection of the voxel vertices on to a 2-surface.

[0077] Figure 5 represents geometry of multiple voxels used to define an example of allowable airspace for UAV navigation above a terrain. 500 is a boundary voxel side, a side in which there is no other adjacent voxel and represents a region of the geofence.

[0078] Figure 6 illustrates an example graphic (600) for an overlay on a UAV (300) when it approaches a boundary voxel side.

[0079] A method to construct a set of voxels for defining an interior geometry is to 1) generate a course-sampling map that projects the three-dimensional convex interior geometry onto the Earth’s latitude and longitude. This projection map generates a set of 2- tuples, i.e. {(latl , longl), (lat2, long2), etc.}, where each 2-tuple can be associated with the virtual geofence’s minimum and maximum altitude above ground level, (AGL). A voxel is created by associating a 2-tuple with its three nearest-neighbors as a voxel base with a minimum AGL defined as the average of the selected point and its three nearest neighbors minimum AGLs. Similarly, a voxel top base is created in a similar manner except calculated by averaging the 4 points’ maximum AGL. In a spherical coordinate system, the voxel vertices are defined as follows: Bottom Voxel Base= {(latl , longl, ave_min_agl), (lat2, long2, ave_min_agl), (lat3, long3, ave_min_agl), (lat4, long4, ave_min_alg)}

Top Voxel Base = {(latl, longl, ave_max_agl), (lat2, long2, ave_max_agl), (lat3, long3, ave_max_agl), (lat4, long4, ave_max_agl)}

[0080] To determine whether a drone is located within the allowed airspace, a system checks whether the drone’s spatial coordinates are contained within one of the defined voxels. If the drone coordinates are not located within any of the voxels, the drone is deemed flying outside of the allowed airspace. For defining the virtual geofence, introduced is the concept of a boundary voxel side. A boundary voxel side is a side that is NOT adjacent to another voxel. For example, in the current construction, every voxel top and bottom base is a boundary voxel side.

[0081 ] Using the drone’ s current speed and direction, a method can be used to predict when a drone will travel into the exterior geometry, i.e., the restricted airspace, meaning the drone is projected to cross a boundary voxel side. If the drone, located at A = (LAT_a, LONG_a, ALT_a) with velocity vector V(A) approaches a boundary voxel side defined by four vertices:

P₄ = (Lat_lt Long_lt Alt^; P₂ = (Lat₂, Long₂, Alt^ P₃ = Lat₄, Long ₄, Alt ₂) and P₄ = (Lat₂, Long₂, Alt₂)

[0082] A method is used to calculate the distance and the predicted elapsed time for the drone to reach the geofence. The first step is an elementary exercise to calculate the distance of the drone, point A, to the boundary voxel side as defined by P₄ , P₂ , P₃, and P₄ above:

1. Two vectors can be formed from these four coplanar points; for example, P₂ ~ P3 and P₄ - P₃

2. The cross-product of these two vectors generates a normal vector, N, i.e., a vector that is perpendicular to the plane formed by the 3 points.

3. Define a new vector W, which is a vector from the drone’s location to one of the points defining the boundary, i.e., A — /^The distance between the drone location and the given plane is the length of the projection of vector W onto the unit normal vector 7V7|N| calculated by the vector dot product. [0083] Dividing the normal component of the velocity vector (that component of the velocity vector that is perpendicular to the boundary voxel side) by the calculated normal distance, a predicted time can be calculated for reaching the virtual geofence. Because a drone may have a positive velocity component that can potentially intercept up to three additional boundary voxel sides, the predicted time calculation needs to be performed for those boundary voxel sides in which the drone has a positive velocity component in the direction of the boundary’s normal vector. The predicted time for reaching the virtual geofence is the minimum of the predicted times associated with reaching any of the individual boundary voxel sides.

[0084] A boundary threshold time can be defined and used as a benchmark to trigger an alert notification. For example, the alert can be generated if the predicted time to reach the virtual geofence is less than the defined threshold time. In another implementation, a boundary threshold time can be dynamically calculated based on the drone’s velocity normal to the boundary voxel side, the drone’s maximal deceleration rate, and a pilot response time.

[0085] To avoid breaching the geofence, the drone pilot must alter the drone’s velocity normal to the boundary voxel side to zero (no further advance toward the boundary) or negative (changing direction away from the boundary). For example, if the drone’s normal distance to the boundary voxel side is Ddrone, its speed normal to the boundary voxel side is Vdrone, and the maximal drone deceleration is Deaccdrone, then the notification threshold time, T threshold, and the time to reach the virtual geofence, T_geofence, are:

^threshold ^drone I ' cicer ne + Pilot reaction time

Tgeofence = ^dr one /D done?

If T threshold ~ T_geofence, a system notification is issued to alert the remote pilot of a pending breach of the geofence unless immediate action is taken.

[0086] In some embodiments, the notification approach is based on threshold distance, defined by converting the threshold time, Threshold, into a distance using the below formula:

If Dthreshold ~ Ddrone, then a system notification is generated, alerting of a pending breach of airspace unless corrective action is immediately taken. [0087] If a notification is to be issued for a pending breach of the virtual geofence, an alert can take form of a loud warning signal transmitted to the pilot’s remote controller device and/or a graphic image of a virtual mesh, or ‘cage’ graphic, superimposed on the video displayed on the pilot’s remote controller device.

[0088] To eliminate the variability of the pilot reaction time to cure a pending incursion into restricted airspace, a system may assume control over the drone’s navigation to either redirect its direction away from the geofence or decelerate the drone speed so that it will gracefully stop at the geofence and await control by the remote drone pilot to safely navigate away from the geofence.

[0089] The upper altitude limit for drones is often regulated by government aviation authorities as a fixed maximal height above the ground level (AGL). Hence the top voxel base AGL, in practice, will usually be set to have this maximum altitude. The lower voxel base AGL is determined by viewshed analysis between the drone and the remote controller unit. To maintain continuous communications, the radio transmission link between drone and remote controller unit must be free of obstructions, such as physical occlusions due to changes in the terrain or destructive signal interference due to reflections off the terrain and based on a calculation of the Fresnel zone.

[0090] The lower voxel base AGL is determined by a viewshed analysis based on 1) the location of the remote controller unit (xO, yO, h) and 2) a terrain mapping of the entire geofence region that provides an elevation map H(x,y). The output of this analysis is a minimum AGL defined for every point within the interior of the geofence.

[0091] In the code below, frm = (x0,y0), the location of the remote controller unit: to=(x,y) is the boundary of the geofence in a particular angular direction; dem is an array that contains the terrain elevation H(x,y) between frm and to CLEARANCE is a minimum offset above the ground to avoid obstacles such as trees, buildings, etc.; and RADIO ELEV is the h, the elevation of the remote control unit. def addViewshedPath(frm, to, dem, output array, CLEARANCE, RADIO ELEV, returnPlot=False, includeApproxFresnelZone=TRUE):

#print(" starting addViewshedPath")

#count points along the line elevs = dem.read(l) frmxy = dem.index(frm[0],frm[l]) toxy = dem.index(to[0],to[l]) xy distance = math.dist(frmxy,toxy) distance = haversine. haversine(swap(frm),swap(to), unit=haversine.Unit.METERS) pixelDi stance = di stance/xy di stance pathPoints = np.linspace(frmxy,toxy, int(xy di stance))

#distance between each point distance = haversine. haversine(swap(frm),swap(to), unit=haversine.Unit.METERS) pixelDi stance = di stance/xy di stance if returnPlot: toplot = np.zeros([pathPoints.shape[0],3])

# first point x,y=roundXY(pathPoints[0])

TAKEOFF ALT = elevs[x,y] last_point = output_array[:,x,y] = [0,0,math.radians(-89.999), 0, CLEARANCE] if returnPlot: toplot[0] = [0,0, CLEARANCE]

#print("looping") for idx,p in enumerate(pathPoints[l:]):

#index of point i = idx+1 x,y=roundXY(p) if output_array[l,x,y] > 0: continue agl = elevs[x,y] atl = agl - TAKEOFF ALT di st = i* pixelDi stance 1 astangl e=l ast point [IDX_AN G] viewshed min = dist*math.tan(lastangle)+RADIO_ELEV fresnel add = fresnelR(di st, di stance) if includeApproxFresnelZone else 0 if (viewshed_min < (atl+fresnel_add)): last_point = output_array[:,x,y] = [ atl, di st, math. atan2(atl+fresnel_add-RADIO_ELE V, di st), atl+fresnel_add, atl+fresnel_add+CLEARANCE

] else: last_point = output_array[:,x,y] = [ atl, di st, lastangle, viewshed_min, max(viewshed_min, atl+CLEARANCE+fresnel_add)

] if returnPlot: last_point = output_array[:,x,y] toplotfi] = [dist, atl, last_point[4]] if returnPlot: return toplot

[0092] The output array is a sequence of points in the radial direction between the remote controller and the sampled geofence boundary. As an example, in Figure 7, the end points of the red line 702 imposed on the terrain map 700 correspond to the remote controller location and the edge of the geofence boundary. Figure 8 is a graph 800 of the output results of the addViewshedPath function where the orange line 802 is the minimum AGL generated for this radial direction. To create a complete lower AGL map for the interior of the geofence, the addViewshedPath is called for additional samples of the geofence boundary.

[0093] Example Management System

[0094] The various components of Figures 1 and 3, i.e., management systems 103, remote controller device 101, UAV 100, Al system 106, audio guide computing device, end user computing device 104, management systems 303, remote controller device 301, UAV 300, geofencing system 306, audio guide computing device 305, end user computing device 304, may be embodied by one or more computing devices, processing systems, or servers, shown as apparatus 200 in FIG. 2.

[0095] As illustrated in FIG. 2, the apparatus 200 may include processing circuitry 202, memory 204, communications hardware 206, each of which will be described in greater detail below. While the various components are only illustrated in FIG. 2 as being connected with processing circuitry 202, it will be understood that the apparatus 200 may further comprise a bus (not expressly shown in FIG. 2) for passing information amongst any combination of the various components of the apparatus 200. The apparatus 200 may be configured to execute various operations described above in connection with FIGS. 1 and 3-6 and in connection with the above using computer program instructions.

[0096] The processing circuitry 202 (and/or co-processor or any other processor assisting or otherwise associated with the processor) may be in communication with the memory 204 via a bus for passing information amongst components of the apparatus. The processing circuitry 202 may be embodied in several different ways and may, for example, include one or more processing devices configured to perform independently. Furthermore, the processor may include one or more processors configured in tandem via a bus to enable independent execution of software instructions, pipelining, and/or multithreading. The use of the term “processor” may be understood to include a single core processor, a multi-core processor, multiple processors of the apparatus 200, remote or “cloud” processors, or any combination thereof.

[0097] The processing circuitry 202 may be configured to execute software instructions stored in the memory 204 or otherwise accessible to the processor (e.g., software instructions stored on a separate storage device (not shown)). In some cases, the processor may be configured to execute hard-coded functionality. As such, whether configured by hardware or software methods, or by a combination of hardware with software, the processing circuitry 202 represents an entity (e.g., physically embodied in circuitry) capable of performing operations described herein while configured accordingly. Alternatively, as another example, when the processing circuitry 202 is embodied as an executor of software instructions, the software instructions may specifically configure the processing circuitry 202 to perform the algorithms and/or operations described herein when the software instructions are executed.

[0098] The memory 204 is non-transitory and may include, for example, one or more volatile and/or non-volatile memories. In other words, for example, the memory 204 may be an electronic storage device (e.g., a computer readable storage medium). The memory 204 may be configured to store information, data, content, applications, software instructions, or the like, for enabling the apparatus to carry out various functions in accordance with example embodiments contemplated herein.

[0099] The communications hardware 206 may be any means such as a device or circuitry embodied in either hardware or a combination of hardware and software that is configured to receive and/or transmit data from/to a network and/or any other device, circuitry, or module in communication with the apparatus 200. In this regard, the communications hardware 206 may include, for example, a network interface for enabling communications with a wired or wireless communication network. For example, the communications hardware 206 may include one or more network interface cards, antennas, buses, switches, routers, modems, and supporting hardware and/or software, or any other device suitable for enabling communications via a network. Furthermore, the communications hardware 206 may include the processing circuitry for causing transmission of such signals to a network or for handling receipt of signals received from a network.

[0100] The communications hardware 206 may be configured to provide output to a user and, in some embodiments, to receive an indication of user input. The communications hardware 206 may comprise a user interface, such as a display, and may further comprise the components that govern use of the user interface, such as a web browser, mobile application, dedicated user device, or the like. In some embodiments, the communications hardware 206 may include a keyboard, a mouse, a touch screen, touch areas, soft keys, a microphone, a display, a speaker, or other input devices, output devices, or presentation devices. The communications hardware 206 may utilize the processing circuitry 202 to control one or more functions of one or more of these user interface elements through software instructions (e.g., application software and/or system software, such as firmware) stored on a memory (e.g., memory 204) accessible to the processing circuitry 202.

[0101] Conclusion [0102] Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0103] What is claimed is:

Claims

1. A system comprising: an unmanned aerial vehicle comprising an imaging sensor and a radio, wherein the radio is connected to transmit real time imaging data from the imaging sensors over a radio network, and wherein the radio is connected to receive control data from the radio network; a remote controller device comprising a radio tuned to receive the real-time imaging data from the unmanned aerial vehicle over the radio network; wherein the remote controller device is further configured to transmit the control data to the unmanned aerial vehicle over the radio network, and to transmit a video transmission based on the received real-time image data over the wide-area network; an end user computing device connected to the wide-area network, and including one or more presentation devices, wherein the end user computing device is configured to transmit instructions to the remote controller device to generate the control data for transmission to the unmanned aerial vehicle; and an audio guide computing device connected to the wide-area network and configured to transmit supplemental audio data, wherein the audio guide computing device is configured to provide the supplemental audio data; and wherein the end user computing device is configured to receive the video transmission and the supplemental audio data synchronized and multiplexed over a communications channel of the wide-area network, and to present the received video transmission and supplemental audio data through the one or more presentation devices.

2. The system of claim 1, wherein the unmanned aerial vehicle is a quadcopter drone.

3. The system of claim 1, further comprising one or more additional audio guide computing devices connected to the wide-area network, wherein each audio guide computing device provides respective supplemental audio data to be synchronized and multiplexed with the video transmission from the unmanned aerial vehicle.

4. The system of claim 3, wherein the respective supplemental audio data transmitted by the one or more additional audio guide computing devices comprises data from a file local on the additional audio guide computing device, and wherein transmission of the supplemental audio data is triggered by the geographical location of the unmanned aerial vehicle.

5. The system of claim 4, where the remote controller device incorporates the audio guide computing device.

6. The system of claim 1, wherein the supplemental audio data is based on data related to a geographic location of the unmanned aerial vehicle.

7. The system of claim 1, further comprising a low orbit satellite providing a direct radio link from the remote controller device to the unmanned aerial vehicle.

8. The system of claim 1, further comprising a cellular tower providing a direct radio link between the remote controller device and the unmanned aerial vehicle.

9. The system of claim 1, wherein the remote controller device is configured to function as a network bridge between the radio network and a wide-area network;

10. The system of claim 1, further comprising an artificial intelligence system configured to: identify an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data; and assume control of the unmanned aerial vehicle and the image sensor to focus on the object having the identified image signature.

11. The system of claim 1, further comprising an artificial intelligence system configured to: identify an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data; and insert a graphic overlay on the video transmission based on the object having the identified image signature.

12. The system of claim 1, further comprising an artificial intelligence system configured to: identify an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data; and trigger transmission of audio information associated with the object having the identified image signature.

13. The system of claim 10, wherein the artificial intelligence system is further configured to insert a graphic overlay on the video transmission based on the object having the identified image signature.

14. The system of claim 13, wherein the artificial intelligence system is further configured to trigger transmission of audio information associated with the object having the identified image signature.

15. The system of claim 10, wherein the artificial intelligence system is further configured to trigger transmission of audio information associated with the object having the identified image signature.

16. A method comprising: transmitting over a radio network, by an unmanned aerial vehicle, real time imaging data from imaging sensors of the unmanned aerial vehicle; receiving over the radio network, by the unmanned aerial vehicle, control data from the radio network; receiving from the radio network, by a remote controller device, the real time imaging data; transmitting over the radio network, by the remote controller device, the control data; transmitting over a wide-area network, by an end user computing device, instructions to the remote controller device to generate the control data for the unmanned aerial vehicle; providing, by an audio guide computing device, supplemental audio data; synchronizing and multiplexing on a communications channel of the wide-area network a video transmission based on the real-time imaging data and the supplemental audio data for transmission over the communications channel to the end user computing device; and presenting the video transmission and the supplemental audio data through one or more presentation devices of the end user computing device.

17. The method of claim 16, wherein the unmanned aerial vehicle is a quadcopter drone.

18. The method of claim 1, further comprising: providing, by one or more additional audio guide computing devices, a plurality of streams of supplemental audio data to be synchronized and multiplexed with the video transmission on the communications channel.

19. The method of claim 16, further comprising triggering transmission of the supplemental audio data by the geographical location of the unmanned aerial vehicle.

20. The method of claim 16, wherein the remote controller device incorporates the audio guide computing device.

21. The method of claim 16, wherein the supplemental audio data is based on data related to a geographic location of the unmanned aerial vehicle.

22. The method of claim 16, wherein the radio network comprises a low orbit satellite providing a direct radio link from the remote controller device to the unmanned aerial vehicle.

23. The method of claim 16, wherein the radio network comprises a cellular tower providing a direct radio link between the remote controller device and the unmanned aerial vehicle.

24. The method of claim 16, wherein the remote controller device is configured to function as a network bridge between the radio network and a wide-area network;

25. The method of claim 16, further comprising, identifying an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data; and assuming control of the unmanned aerial vehicle and the image sensor to focus on the object having the identified image signature.

26. The method of claim 16, further comprising: identifying an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data; and inserting a graphic overlay on the video transmission based on the object having the identified image signature.

27. The method of claim 16, further comprising: identifying an image signature of an object in the location of the unmanned aerial vehicle using the real-time imaging data; and triggering transmission of audio information associated with the object having the identified image signature.

28. The method of claim 25, further comprising: inserting a graphic overlay on the video transmission based on the object having the identified image signature.

29. The method of claim 29, further comprising: triggering transmission of audio information associated with the object having the identified image signature.

30. The method of claim 25, further comprising: triggering transmission of audio information associated with the object having the identified image signature.

31. The system of claim 1, further comprising: a large language model configured to generate text describing content of the video transmission; and a text-to-speech engine configured to generate audio corresponding to the generated text, wherein the supplemental audio data includes the generated audio.

32. The method of claim 16, further comprising: using a large language model, generating text describing content of the video transmission; and using a text-to-speech engine, generating audio corresponding to the generated text, wherein the supplemental audio data includes the generated audio.

33. An audio guide computing device for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network, the audio guide computing device comprising a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the audio guide computing device to provide supplemental audio data; and synchronize and multiplex the supplemental audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.

34. A method for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits a video transmission based on the realtime imaging data over the wide-area network, the method comprising: providing supplemental audio data; and synchronizing and multiplexing the supplemental audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.

35. A computing device for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network, the computing device comprising a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the computing device to: track spatial coordinates for a geographic location of the unmanned aerial vehicle; and generate information based on the geographic location of the unmanned aerial vehicle location to be transmitted to the end user computing device.

36. A method for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits a video transmission based on the realtime imaging data over the wide-area network, the method comprising: tracking spatial coordinates for a geographic location of the unmanned aerial vehicle; and generating information based on the geographic location of the unmanned aerial vehicle location to be transmitted to the end user computing device.

37. A system comprising: an unmanned aerial vehicle comprising an imaging sensor and a radio, wherein the radio is connected to transmit real time imaging data from the imaging sensors over a radio network, and wherein the radio is connected to receive control data from the radio network; a remote controller device comprising a radio tuned to receive the real-time imaging data from the unmanned aerial vehicle over the radio network; wherein the remote controller device is further configured to transmit the control data to the unmanned aerial vehicle over the radio network, and to transmit a video transmission based on the received real-time imaging data over the wide-area network; an end user computing device connected to the wide-area network, and including one or more presentation devices, wherein the end user computing device is configured to transmit instructions to the remote controller device to generate the control data for transmission to the unmanned aerial vehicle; and a geofencing computing device connected to the wide-area network and configured to: create a representation of a 3 -dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence; track spatial coordinates for a geographic location of the unmanned aerial vehicle; and determine a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

38. The system of claim 37, wherein the unmanned aerial vehicle is a quadcopter drone.

39. The system of claim 37, wherein the geofencing computing device is further configured to generate a notification to the end user computing device based on the determined spatial relationship between the geographic location of the unmanned aerial vehicle and the geofence.

40. The system of claim 39, wherein to determine the spatial relationship includes determining whether the geographical location of the unmanned aerial vehicle is either in the exterior space or in the proximity of the geofence.

41. The system of claim 39, wherein to generate the notification, the geofencing computing device is configured to: store a minimum threshold time and a minimum threshold distance; evoke the notification based on either of:

(i) a distance of the unmanned aerial vehicle to the geofence is less than or equal to the minimum threshold distance; or

(ii) a predicted time of the unmanned aerial vehicle to reach the geofence is less than or equal to the minimum threshold time.

42. The system of claim 41, wherein the minimum threshold time and minimum threshold distance for evoking a notification are dynamically computed values based on kinematics of the unmanned aerial vehicle.

43. The system of claim 41, wherein the minimum threshold distance is based on a maximum deceleration of the unmanned aerial vehicle in a direction normal to the geofence.

44. The system of claim 41, wherein the minimum threshold time is based on a maximum deceleration of the unmanned aerial vehicle in a direction normal to the geofence.

45. The system of claim 39, wherein the notification comprises a graphic representation overlaid on the video transmission sent from the remote controller device to the end user computing device.

46. The system of claim 39, wherein the geofencing computing device, after evoking a notification, causes the remote controller device to send control data to the unmanned aerial vehicle to decelerate in a direction away from the geofence to avoid navigating into the exterior space.

47. The system of claim 37, wherein the geofencing computing device causes the remote controller device to send control data to the unmanned aerial vehicle to decelerate in a direction away from the geofence to avoid navigating into the exterior space.

48. A method, comprising: transmitting over a radio network, by an unmanned aerial vehicle, real time imaging data from imaging sensors of the unmanned aerial vehicle; receiving over the radio network, by the unmanned aerial vehicle, control data from the radio network; receiving from the radio network, by a remote controller device, the real time imaging data; transmitting over the radio network, by the remote controller device, the control data; transmitting over a wide-area network, by an end user computing device, instructions to the remote controller device to generate the control data for the unmanned aerial vehicle; creating a representation of a 3 -dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence; tracking spatial coordinates for a geographic location of the unmanned aerial vehicle; and determining a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

49. The method of claim 48, wherein the unmanned aerial vehicle is a quadcopter drone.

50. The method of claim 48, further comprising: generating a notification to the end user computing device based on the determined spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

51. The method of claim 50, wherein determining the spatial relationship includes determining whether the geographical location of the unmanned aerial vehicle is either in the exterior space or in the proximity of the geofence.

52. The method of claim 50, wherein generating the notification comprises: storing a minimum threshold time and a minimum threshold distance; evoking the notification based on either of: (i) a distance of the unmanned aerial vehicle to the geofence is less than or equal to the minimum threshold distance; or

53. The method of claim 52, wherein the minimum threshold time and minimum threshold distance for evoking a notification are dynamically computed values based on kinematics of the unmanned aerial vehicle.

54. The method of claim 52, wherein the minimum threshold distance is based on a maximum deceleration of the unmanned aerial vehicle in a direction normal to the geofence.

55. The method of claim 52, wherein the minimum threshold time is based on a maximum deceleration of the unmanned aerial vehicle in a direction normal to the geofence.

56. The method of claim 50, wherein the notification comprises a graphic representation overlaid on the video transmission sent from the remote controller device to the end user computing device.

57. The method of claim 50, further comprising, after evoking a notification, causing the remote controller device to send control data to the unmanned aerial vehicle to decelerate in a direction away from the geofence to avoid navigating into the exterior space.

58. The system of claim 48, further comprising causing the remote controller device to send control data to the unmanned aerial vehicle to decelerate in a direction away from the geofence to avoid navigating into the exterior space.

59. A geofencing computing device for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits real-time imaging data as a video transmission over the wide-area network, the geofencing computing device comprising a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the geofencing computing device to: create a representation of a 3 -dimensional spatial region based on a union of polygon prisms to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence; and track spatial coordinates for a geographic location of the unmanned aerial vehicle; and determine a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

60. A method for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits real-time imaging data as a video transmission over the wide-area network, the method comprising: creating a representation of a 3 -dimensional spatial region to define an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence; and tracking spatial coordinates for a geographic location of the unmanned aerial vehicle; and determining a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

61. A geofencing computing device for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits real-time imaging data as a video transmission over the wide-area network, the geofencing computing device comprising a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the geofencing computing device to: create a representation of a 3 -dimensional spatial region defining an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence, wherein a minimum above the ground level for each point within the interior of the geofence is determined by a viewshed analysis based on three dimensional coordinates for the geographic location of the remote controller device and an elevation map of the geographic terrain of the geofence region; and track three-dimensional spatial coordinates for a geographic location of the unmanned aerial vehicle; and determine a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

62. A method for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits real-time imaging data as a video transmission over the wide-area network, the method comprising: creating a representation of a 3 -dimensional spatial region defining an interior space, an exterior space, and a boundary between the interior and exterior space as a geofence, wherein a minimum above the ground level for each point within the interior of the geofence is determined by a viewshed analysis based on 1) three dimensional coordinates for the geographic location of the remote controller device and 2) an elevation map of the geographic terrain of the geofence region; and tracking three-dimensional spatial coordinates for a geographic location of the unmanned aerial vehicle; and determining a spatial relationship between the geographic location of the unmanned aerial vehicle location and the geofence.

63. A computing device for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits a video transmission based on the real-time imaging data over the wide-area network, the computing device comprising a processing system including processing circuitry and a memory storing computer program instructions, the computer program instructions configuring the computing device to: generate text describing content of the video transmission based on a large language model; generate audio data corresponding to the generated text using a text-to-speech engine; and synchronize and multiplex the supplemental audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.

64. A method for use in a system with a remote controlled unmanned aerial vehicle, wherein the unmanned aerial vehicle includes imaging sensors and a radio that transmits real time imaging data from the imaging sensors over a radio network and that receives control data from the radio network from a remote controller device, wherein the remote controller device is connected to a wide-area network and receives instructions from an end user computing device to generate the control data for the unmanned aerial vehicle, wherein the remote controller device transmits a video transmission based on the realtime imaging data over the wide-area network, the method comprising: using a large language model, generating text describing content of the video transmission; using a text-to-speech engine, generating audio data corresponding to the generated text; and synchronizing and multiplexing the audio data with the video transmission over a communications channel of the wide-area network for transmission to the end user computing device.