WO2018089716A1

WO2018089716A1 - Remotely-controlled device control system, device and method

Info

Publication number: WO2018089716A1
Application number: PCT/US2017/060971
Authority: WO
Inventors: Ronald Wilson PULLING; Yang Hu
Original assignee: Aatonomy, Inc.
Priority date: 2016-11-09
Filing date: 2017-11-10
Publication date: 2018-05-17

Abstract

A system, device, and method for controlling a remotely-controlled device are disclosed. The remotely-controlled device control system could include one or more remotely-controlled devices; a controller comprised of a data collection module, at least one active module, at least one module description file (MDF), a robot description file (RDF) for each remotely-controlled device, and a processing unit (PU); at least one data source, and a marketplace. The PU may be configured to receive at least one active module; receive, for each received active module, input data of its associated MDF provided by the data collection module; generate, for each received active module, output data as a function of to the input data received by the processing unit, and provide the output data to a destination system.

Description

REMOTELY-CONTROLLED DEVICE CONTROL SYSTEM, DEVICE AND METHOD

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. provisional applications No.

62/419,818 entitled "DEVICES, SYSTEMS, AND METHODS FOR A CINEMATIC FOLLOWING USING VISUAL TRACKING SYSTEMS" filed November 9, 2016 and No. 62/419,838 entitled "MODULAR INTELLIGENCE PLATFORM FOR AUTONOMOUS REMOTELY-CONTROLLED APPLICATIONS" filed November 9, 2016, which are herein incorporated by reference in their entirety.

BACKGROUND

[0002] Remotely-controlled devices have become increasingly popular.

For example, a wealth of drones (i.e., remotely-controlled devices or autonomous vehicles) have appeared in the market for hobbyists to enjoy; however, many drones, robots, and other types of remotely-controlled devices each rely on their own proprietary control methods. For example, Drone Manufacturer A may design a drone which programmatically turns left or right in a completely different way from Drone

Manufacturer B. Because each manufacturer employs its own proprietary functionality or software, the functionality designed for use with Drone A is unlikely to be compatible with Drone B, and vice versa. As a result, it has heretofore not been practical to offer general software solutions to a broad customer base within this industry through a centralized, self-service marketplace. Not Furnished Upon Filing

SUMMARY

[0003] A modular intelligence platform for autonomous remotely-controlled applications is disclosed. The platform may include a general purpose computing device (or controller), a set of standards for the transfer of sensor and other data, and a mechanism for the aggregation of data, code modules ("Modules") which can be easily turned on, turned off, or exchanged, and make decisions on behalf of a connected robot, a set of standards for the communication of instructions to a robot, and a mechanism for aggregating and disseminating such instructions, and a marketplace from which such modules may be acquired

[0004] In one aspect, embodiments of the inventive concepts disclosed herein are directed to a system for controlling remotely-controlled devices. The system could include one or more remotely-controlled devices; a controller comprised of a data collection module, at least one active module, at least one module description file (MDF), a robot description file (RDF) for each remotely-controlled device, and a processing unit (PU); at least one data source, and a marketplace.

[0005] In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a device for controlling remotely-controlled devices. The device could include the PU configured to perform the method in the paragraph that follows.

[0006] In a further aspect, embodiments of the inventive concepts disclosed herein are directed to a method for controlling remotely-controlled devices. When properly configured, the PU may receive at least one active module; receive, for each received active module, input data of its associated MDF provided by the data collection module; generate, for each received active module, output data as a function of to the input data received by the processing unit, and provide the output data to a destination system.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a fuller understanding of the inventive embodiments, reference is made to the following description taken in connection with the accompanying drawings in which:

[0008] FIG. 1 depicts a functional block diagram of the remotely-controlled device controlling system, in accordance with various embodiments;

[0009] FIGS. 2A-2B depict an exemplary configuration of the remotely- controlled device controlling system, in accordance with various embodiments; and

[00010] FIG. 3 depicts an exemplary embodiment of a flowchart disclosing a method for controlling remotely-controlled devices, in accordance with various embodiments.

DETAILED DESCRIPTION

[00011] In the following description, several specific details are presented to provide a thorough understanding of embodiments of the inventive concepts disclosed herein. One skilled in the relevant art will recognize, however, that embodiments of the inventive concepts disclosed herein can be practiced without one or more of the specific details, or in combination with other components, etc. In other instances, well-known implementations or operations are not shown or described in detail to avoid obscuring aspects of various embodiments of the inventive concepts disclosed herein.

[00012] Referring now to FIG. 1 , an embodiment of a control system of a remotely-controlled device suitable for implementation of the inventive concepts described herein includes at least one remotely-controlled device 1 10, a controller 120, at least one data source 140, and a marketplace 150.

[00013] Remotely-controlled devices 1 10 could include any remotely- controlled device(s), vehicle(s), and/or swarms of vehicles which is capable of flying through the air, atmosphere, and/or space including, but not limited to, lighter than air vehicles and heavier than air vehicles, wherein the latter may include remotely- controlled quadcopters, fixed-wing, and rotary-wing vehicles. Additionally, remotely- controlled devices 1 10 could include any remotely-controlled system including legged robots and/or vehicles capable of traversing the surface of the Earth, and/or any watercraft capable of unmanned/autonomous operation on or beneath water.

Additionally, remotely-controlled devices 1 10 could include any in-place mechanical or electrical system capable of autonomous operation. [00014] Controller 120 could include a data collection module 122, at least one active module 124, at least one module description file (MDF) 126, a remotely- controlled device description file (RDF) 128 for each remotely-controlled device, a processing unit (PU) 130, and an optional output aggregator 132. Controller 120 may include a software architecture which facilitates the execution of exchangeable active module(s) 124 that are capable of making decisions on behalf of remotely-controlled devices 1 10. Controller 120 may also include a means of controlling those remotely- controlled devices 1 10 in such a way as to achieve a desired outcome, independent of the type of remotely-controlled device 1 10 connected to controller 120. Controller 120 may be communicatively connected to remotely-controlled device(s) 1 10.

[00015] Data collection module 122 may be configured to receive input data from at least one data source 140. In some embodiments, data collection module 122 may receive data from remotely-controlled device 1 10 and/or other remotely-controlled devices working in conjunction with the remotely-controlled device 1 10 in a variety of ways. These could include continual publish / subscribe paradigms and TCP-style request and respond paradigms.

[00016] Active modules 124 may enable a wide range of capabilities. Each active module 124 may include executable code of a functional algorithm which, when employed, could enable a specific operation of the remotely-controlled device 1 10. If purchased, subscribed, and/or licensed from marketplace 150, active modules 124 may automatically be made unavailable once a time period has expired or a usage limit has been reached. In some embodiments, multiple active modules 124 may be used simultaneously. [00017] The following is a small, non-exhaustive list of example active modules 124 provided for the purpose of illustration and not of limitation: cinematic following; pedestrian detection and avoidance for autonomous vehicles (e.g., self- driving cars); obstacle avoidance; gesture recognition; signal processing; path planning in complex, obstructed environments; SLAM (Simultaneous Localization And Mapping); object recognition; object tracking; voice commands / natural language processing; emotion detection (e.g., the person I'm interacting with is sad / happy / angry);

manipulator path planning (for remotely-controlled arms); full-coverage path planning (e.g. make sure a cleaning robot covers the entire floor, or a farming robot covers an entire field); returning to a charging station (e.g., when batteries are low, or when a task is completed); searching for objects (e.g., in an unconstrained way or within an established search radius/area); following a line; and enabling swarm behavior (e.g., groups of robots maneuvering together, sometimes to accomplish some task).

[00018] Active modules 124 could also include foundational capabilities that can be used as inputs into more complex and higher-level decision-making processes, or that can be used directly in the execution of basic tasks. For example, an optical flow module (e.g., measuring how far things move between frames in a video) could be employed in quadcopter hovering stabilization. In another example, plug-and-play computer vision modules such as edge detection could be used as a pre-processing step in transforming image data into the format required by a higher-level semantic segmentation module, the output of which could then in turn be used as an input into an object recognition module. [00019] Active modules 124 could receive input data from data collection module 122. Active modules 124 may also receive information found in RDF 126 directly to retrieve relevant static variable values (e.g. the dimensions of the robot). Each active module 124 may require a specific type(s) of data as inputs. Information regarding data required by each active module 124 may be provided in MDF 126.

[00020] MDF 126 may be associated with one active module 122. In some embodiments, MDF 126 may identify specific categories of remotely-controlled devices 1 10 which may employ output data generated by the active module 124 associated with the MDF 126. In some embodiments, MDF 126 may include information including, but not limited to, requirements for data types, data formats, data update frequency, and sensor placement on the robot (e.g. "in the center, at the front, within 2 inches of the ground"). In some embodiments, MDF 126 could include subscription/purchase/license status or expiration date information, which may be capable of turning the active module 124 associated with the MDF 126 off/on, and/or may be replaced. In some

embodiments, MDF 126 may describe categories or specific instances of other available modules 152 with which the active module 124 associated with the MDF 126 is or is not interoperable with.

[00021] RDF 128 could be associated with one remotely-controlled device

1 10 and include physical descriptions and dimensions of the remotely-controlled device 1 10. In some embodiments, RDF 128 could provide input to at least one active module 124. In some embodiments, RDF 128 could be used to automatically determine the compatibility of the remotely-controlled device 1 10 with an available module 152 in the marketplace 150, and to automatically prevent that available module 152 from being acquired as an active module 124 for use with the remotely-controlled device 1 10 described by RDF 128.

[00022] PU 130 could include any electronic data processing unit which executes software or computer instruction code that could be stored, permanently or temporarily, in a digital memory storage device or a non-transitory computer-readable media including, but not limited to, random access memory (RAM), read-only memory (ROM), compact disc (CD), hard disk drive, diskette, solid-state memory, Personal Computer Memory Card International Association card (PCMCIA card), secure digital cards, and compact flash cards. PU 130 may be driven by the execution of software or computer instruction code containing algorithms developed for the specif ic functions embodied herein. PU 130 may be an application-specific integrated circuit (ASIC) customized for the embodiments disclosed herein. Common examples of electronic data processing units are microprocessors, Digital Signal Processors (DSPs),

Programmable Logic Devices (PLDs), Programmable Gate Arrays (PGAs), and signal generators; however, for the embodiments herein, the term "processor" is not limited to such processing units and its meaning is not intended to be construed narrowly. For instance, PU 130 could also include more than one electronic data processing unit. In some embodiments, PU 130 could be a processor(s) used by or in conjunction with any other system employed herein. In some embodiments, PU 130 could be embedded within the robotic device 1 10. In some embodiments, PU 130 could be embedded within a manual input device. In some embodiments, PU 130 could be located remotely and connected to the controller 120 via wired or wireless data transmission linkages. [00023] In some embodiments, the terms "programmed" and "configured" are synonymous. PU 130 may be electronically coupled to systems and/or sources to facilitate the receipt of input data. In some embodiments, operatively coupled may be considered as interchangeable with electronically coupled. It is not necessary that a direct connection be made; instead, such receipt of input data and the providing of output data could be provided through a bus, through a wireless network, or as a signal received and/or transmitted by PU 130 via a physical or a virtual computer port. PU 130 may be programmed or configured to execute the method discussed in detail below.

[00024] Output aggregator 132 may be configured to receive output data generated by one or more active modules 124. Active modules 124 may or may not be required to format output data in standardized forms (e.g. "a single number with two decimal places"). An output standardizer may be used to transform raw outputs into specific formats. Outputs from each Module are collected by an Output Aggregator (e.g., Output Aggregator 132), combined, and sent to the robot using a data

transmission protocol. Results may be also be transmitted to a mobile device (e.g. a smartphone or dedicated receiver) for visualization and real-time feedback, or used as input data provided to data collection module 122 for subsequent use by an active module 124.

[00025] Data source 140 could include any device(s) configured to provide as input data the specific data required by the active module(s) 124 that are specified in the MDF 126 associated with the active module 124. Data source 140 could include an embedded sensor, a directly connected sensor, a remotely-controlled device-attached sensor, and non-sensor data such as, but not limited to, data input manually to a device (e.g. a smartphone) data pulled from an online database, data accessed through a third- party API, or data provided as an output of a different active module 124. Sensors may be embedded in controller 120, directly connected (e.g., "plugged in") to controller 120, and/or attached to the remotely-controlled device 1 10. Data collection module 122 may be configured to read values directly from the sensors, or it may receive such data by way of intermediate processes (for example, such an intermediate process may normalize or otherwise transform the data before sending the data to data collection module 122).

[00026] Marketplace 150 may be any source from which a plurality of available modules 152 and a plurality of available MDFs 154 associated with the plurality of available modules 152 may be obtained; once obtained and provided to controller 120, available modules 152 and available MDFs 154 may become active modules 124 and active MDFs 126, respectively. Available modules 152 may be applied to specific remotely-controlled devices and/or fleets of remotely-controlled devices. In some embodiments, only available modules 152 compatible with a specific remotely-controlled device 1 10 will be allowed to be purchased. Marketplace 150 may include descriptions of the different available modules 152, a ranking or rating system, comments from other users, lists of similar available modules 152, lists of available modules 152 that can work in concert with other available modules 152, price

information, and promotions. In some embodiments, a trial period may be available so that available modules 152 may be employed temporarily without payment.

[00027] Marketplace 150 may include a dashboard for controlling which available modules 152 are now active modules 124 for a given fleet of remotely- controlled devices 1 10. This Dashboard may be used to turn active modules 124 on/off and/or re-order a priority given to various active modules 124. A user may also configure its active modules 124 or remotely-controlled devices 1 10, for example, by entering specific required or optional details (e.g. the location of a sensor on the robot, or the type of object to be tracked). Changes made in the Dashboard may be reflected in associated active MDF(s) 126 and RDF(s) 128.

[00028] Referring now to FIGS. 2A and 2B, an exemplary embodiment of one possible configuration of the system 100 is illustrated in accordance with various embodiments. For the purpose of discussion, it will be assumed that the remotely- controlled device 1 10 is an autonomous vehicle (AV) which the operator would like to employ to capture cinema-quality videos of a person (object of the video) strolling along a beach, and that the AV is equipped with a stationary camera. Instead of manually operating the AV while watching the video to keep the person centered in the image, the operator would like the AV to follow the person autonomously.

[00029] As shown in FIGS. 2A and 2B, system 100 includes active module

124 obtained from marketplace 150 that is configured with a visual tracking algorithm, i.e., a visual tracking module. Data inputs required for the visual tracking module are identified in the associated MDF 126 as image data (representative of an image located outside the AV), a sequence of following modes, and a selected object (within the image). As shown, these required data inputs are provided to data collection module 122 by MDF 126. The image data input originates from the fixed camera present on the AV, is transmitted to the Display Unit for the user to view, and is passed on from the Display Unit to the data collection module 122. The sequence of following modes and the selected object to be tracked are set by the user via the Main Input Device (e.g. via a touch interface on a smartphone), then passed to the data collection module 122. The data collection module 122 then passes all required inputs to the visual tracking module 124 which, in turn, generates output data identified as course correction data employed by the AV control system that will be provided to the AV control system (i.e., the destination system).

[00030] As shown in RDF 128, the AV is configured with a GPS system as well as the visual tracking system and the AV control system. If the operator wished to track the strolling person via a series of waypoints, system 100 could be configured with an active module 124 obtained from marketplace 150 that is configured with a GPS system algorithm, i.e., a GPS system module. Data inputs required for the GPS system module are identified in the associated MDF 126 as GPS location and waypoints (which define the course of flight of the AV). GPS location is periodically provided by a GPS sensor embedded in the AV, and is transmitted by way of the Main Input Device to the Data Collection Module 122. GPS waypoints are periodically provided by a GPS sensor in the Display Unit, which the strolling person carries with them, and are passed to the Data Collection Module 122. The output data generated by GPS system module is representative of course correction data employed by the AV control system that will be provided to the AV control system (i.e., the destination system).

[00031] FIG. 3 depicts flowchart 200 disclosing an example of a method for controlling a remotely-controlled device, where PU 130 may be programmed or configured with instructions corresponding to the modules embodied in flowchart 200. In some embodiments, PU 130 may be a processor or a combination of processors found in controller 120, or any other system suitable for performing the task. Also, PU 130 may be a processor of a module such as, but not limited to, a printed circuit card having one or more input interfaces to facilitate the two-way data communications of PU 130, i.e., the receiving and sending of data, the latter being synonymous with a providing and/or transmitting of data. As necessary for the accomplishment of the following modules embodied in flowchart 200, the acquiring of data is synonymous and/or interchangeable with the receiving and/or retrieving of data, and the providing of data is synonymous and/or interchangeable with the making available or supplying of data.

[00032] The method of flowchart 200 begins with module 202 with PU 130 receiving one or more active modules 124 that have been obtained from marketplace 150 as one or more available modules 152. In some embodiments, marketplace 150 may be available to one or more manufacturers of the remotely-controlled device(s). In some embodiments, marketplace 150 may be available to one or more owners and/or operators of the remotely-controlled device(s).

[00033] The method of flowchart 200 continues with module 204 with PU

130 receiving input data required by each active module 124 from data collection module 122. Data collection module 122 could have received an active MDF 126 for each active module 124, where each active MDF 126 may include identification data. The identification data may be representative of the specific input(s) required by its associated active module 124. The identification data may be representative of the name and/or address of data source 140 for each specific input. In some embodiments, a plurality of specific inputs may be provide by one data source 140. Once the identification of the input data source(s) is known, data collection module 122 could receive input data from each and provide it to the PU 130.

[00034] The method of flowchart 200 continues with module 206 with PU

130 generating, for each active module of the plurality of active modules, output data by applying a functional algorithm which, when employed, could enable a specific operation of a remotely-controlled device 1 10.

[00035] The method of flowchart 200 continues with module 208 with PU

130 providing the output data to a destination system. In some embodiments, the destination system could be a system found on the remotely-controlled device. In some embodiments, the destination system could be an output aggregator configured to combine data and send the combined data to a system found on the remotely-controlled device. Then, the method of flowchart 200 ends.

[00036] It should be noted that the steps of the method described above may be embodied in computer-readable media stored in a non-transitory computer- readable medium as computer instruction code. The method may include one or more of the steps described herein, which one or more steps may be carried out in any desired order including being carried out simultaneously with one another. For example, two or more of the steps disclosed herein may be combined in a single step and/or one or more of the steps may be carried out as two or more sub-steps. Further, steps not expressly disclosed or inherently present herein may be interspersed with or added to the steps described herein, or may be substituted for one or more of the steps described herein as will be appreciated by a person of ordinary skill in the art having the benefit of the instant disclosure.

[00037] As used herein, the term "embodiment" means an embodiment that serves to illustrate by way of example but not limitation.

[00038] It will be appreciated to those skilled in the art that the preceding examples and embodiments are exemplary and not limiting to the broad scope of the inventive concepts disclosed herein. It is intended that all modifications, permutations, enhancements, equivalents, and improvements thereto that are apparent to those skilled in the art upon a reading of the specification and a study of the drawings are included within the broad scope of the inventive concepts disclosed herein. It is therefore intended that the following appended claims include all such modifications, permutations, enhancements, equivalents, and improvements falling within the broad scope of the inventive concepts disclosed herein.

Claims

CLAIMS What is claimed is:

1 . A system for controlling a remotely-controlled device, comprising: a marketplace having at least one available module and at least one available module description file (MDF); a plurality of input data sources; at least one remotely-controlled device; a description file for each remotely-controlled device or fleet of remotely-controlled devices; at least one active module received from the marketplace as at least one available module; at least one active MDF received from the marketplace as at least one available MDF, where each active MDF is associated with one active module and includes identification data representative of: input data required by its associated active module; a source identification of at least one input data source of the plurality of input data sources for the input data required by its associated active module, and data types and formats for the output data that will be created by its associated active module; a data collection module configured to: receive, from each MDF, the identification of the data sources required by its associated active module; , receive, for each MDF, input data from each identified data source, and provide, for each MDF, the input data to a processing unit; and the processing unit comprised of at least one processor coupled to a non-transitory processor-readable medium storing processor-executable code corresponding to at least one active module and configured to: receive at least one active module, receive, for each received active module, the input data of its associated MDF provided by the data collection module, generate, for each received active module, output data as a function of the input data received by the processing unit, and provide the output data to a destination system.

2. A device for controlling a remotely-controlled device, comprising: a processing unit comprised of at least one processor coupled to a non-transitory processor-readable medium storing processor-executable code corresponding to at least one active modules and configured to: receive at least one active module, receive, for each one active module, input data of active module description file (MDF) associated with the active module that is provided by a data collection module, generate, for each one active module, output data as a function of the input data received by the processing unit, and provide the output data to a destination system.

3. A method for controlling a remotely-controlled device, comprising: receiving, by a processing unit comprised of at least one processor coupled to a non- transitory processor-readable medium storing processor-executable code

corresponding to at least one active modules, at least one active module, receiving, for each one active module, input data of active module description file (MDF) associated with the active module that is provided by a data collection module, generating, for each one active module, output data as a function of the input data received by the processing unit, and providing the output data to a destination system.