JP2022133258A - Autonomous maneuver generation to mate connectors - Google Patents

Abstract

To provide a method of autonomous maneuver generation to mate connectors.SOLUTION: A method includes providing an image of feature data on the basis of a feature extraction model of connector mating. The image depicts a first portion of a first device and a second portion of a second device. The feature data includes coordinates representing key points of each of the first and second devices depicted in the image. The method also includes obtaining position data indicating a position in 3D space of a connector of the first device. The method further includes providing the feature data and the position data to a trained autonomous agent to generate a proposed maneuver to mate the connector with a connector of the second device.SELECTED DRAWING: Figure 1

Description

The present disclosure relates generally to generating autonomous procedures for connecting connectors.

Highly skilled operating personnel are typically used to guide complex high speed docking operations such as airborne refueling and spacecraft docking operations. Therefore, their actions rely heavily on human judgment, which is often supplemented by computer vision techniques. To illustrate, complex stereo vision systems can be used to assist operating personnel in connecting connectors (eg, receivers and refueling boom or docking connectors).

In one particular aspect, a method includes providing a first image as input to a feature extraction model to generate feature data. The first image depicts a first portion of the first device and a second portion of the second device. The feature data includes first coordinates representing keypoints of a first device depicted within the first image and keypoints of a second device depicted within the first image. Contains the second coordinate representing. The method also includes obtaining position data indicative of a position in three-dimensional space of the first connector of the first device from one or more sensors onboard the first device. A first connector is disposed on the first portion of the first device. The method includes providing feature data and location data as inputs to a trained autonomous agent to generate a suggested procedure for connecting the first connector to the second connector of the second device. Including further.

In another particular aspect, a device includes a moveable coupling system configured to move a first connector relative to a second connector of a second device. The device also includes one or more sensors configured to generate position data indicative of the position of the first connector in three-dimensional space. The device further includes a camera configured to generate image data depicting a portion of the movable coupling system and at least a portion of the second device. The device also includes a feature extraction model configured to receive a first image based on the image data and a feature extraction model configured to generate feature data based on the first image. The feature data includes first coordinates representing keypoints of the moveable coupling system depicted within the first image, and keypoints of the second device depicted within the first image. including a second coordinate representing . The device further comprises a trained autonomous agent configured to generate suggested procedures for connecting the first connector to the second connector of the second device based on the feature data and the location data. include.

In another particular aspect, a tanker aircraft includes a steerable boom including a first connector configured to couple with a second connector of a second aircraft during refueling of the second aircraft. including. The tanker aircraft also includes one or more sensors coupled to the steerable boom and configured to generate position data indicative of the position of the steerable boom in three-dimensional space. The tanker aircraft further includes a camera configured to generate image data depicting a portion of the steerable boom and at least a portion of the second aircraft. The tanker aircraft also has a feature extraction model configured to receive a first image based on the image data, the feature extraction model configured to generate feature data based on the first image. include. The feature data includes first coordinates representing keypoints of the steerable boom depicted within the first image and representing keypoints of the second aircraft depicted within the first image. contains a second coordinate that is The tanker aircraft further includes a trained autonomous agent configured to generate a suggested procedure for connecting the first connector to the second connector based on the feature data and the location data.

In another particular aspect, a spacecraft includes a steerable docking mechanism including a first connector configured to mate with a second connector of a second spacecraft during a docking operation. The spacecraft also includes one or more sensors coupled to the steerable docking mechanism and configured to generate position data indicative of the position of the steerable docking mechanism in three-dimensional space. . The spacecraft further includes a camera configured to generate image data depicting a portion of the steerable docking mechanism and at least a portion of the second spacecraft. The spacecraft also includes a feature extraction model configured to receive a first image based on the image data, the feature extraction model configured to generate feature data based on the first image. . The feature data includes first coordinates representing keypoints of the steerable docking mechanism depicted within the first image and keypoints of the second spacecraft depicted within the first image. including a second coordinate representing . The spacecraft also includes a trained autonomous agent configured to generate suggested procedures for connecting the first connector to the second connector based on the feature data and the position data.

The features, functions, and advantages described herein may be implemented independently in various implementations or may be combined in still other implementations. These additional details can be understood with reference to the following specification description and drawings.

FIG. 10 illustrates a system configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data; FIG. 10 illustrates a system configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data; Figure 3 shows an example of a display generated by the system of Figure 2; FIG. 3 is a diagram of various aspects of image processing and feature detection by the system of FIG. 2; FIG. 3 is a diagram of various aspects of image processing and feature detection by the system of FIG. 2; FIG. 3 is a diagram of various aspects of image processing and feature detection by the system of FIG. 2; FIG. 3 is a diagram of various aspects of image processing and feature detection by the system of FIG. 2; FIG. 2 illustrates a system configured to generate a suggested procedure for connecting a first connector to a second connector based on image data and position data; Figure 6 shows an example of a display generated by the system of Figure 5; FIG. 6 is a diagram of various aspects of image processing and feature detection by the system of FIG. 5; FIG. 6 is a diagram of various aspects of image processing and feature detection by the system of FIG. 5; FIG. 6 is a diagram of various aspects of image processing and feature detection by the system of FIG. 5; FIG. 6 is a diagram of various aspects of image processing and feature detection by the system of FIG. 5; FIG. 5 is a flowchart of one embodiment of a method for generating suggested procedures for connecting a first connector to a second connector based on image data and position data; FIG. FIG. 5 is a flowchart of another embodiment of a method for generating suggested procedures for connecting a first connector to a second connector based on image data and position data; FIG. Figure 10 is a flowchart of one embodiment of a method for training an autonomous agent to generate suggested procedures for connecting a first connector to a second connector based on image data and location data; 4 is a flow chart of one embodiment of an aircraft life cycle configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data; 1 is a block diagram of an embodiment of an aircraft configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data; FIG. 1 is a block diagram of a computing environment including a computing device configured to support aspects of computer-implementable methods and computer-executable program instructions (or code) according to the present disclosure; FIG.

Aspects disclosed herein present systems and methods that use trained autonomous agents to generate suggested procedures for connecting connectors of two different devices. For example, the autonomous agent may be one or more neural networks or others trained to generate procedural recommendations or instructions for guiding a refueling connector to an aircraft refueling port during airborne refueling. may include a machine learning model of As another example, autonomous agents may be trained to generate procedural recommendations or instructions to guide the docking of one spacecraft to another. In some implementations, trained autonomous agents can be used to assist workers in increasing reliability and standardizing operations during maneuvers to connect connectors. In other embodiments, trained autonomous agents are used in place of operating personnel to reduce costs, such as costs associated with training operating personnel, costs associated with operations to connect connectors, and the like. can be done.

In some contexts, the two devices that perform docking include a primary device and a secondary device. In some contexts (such as when two peer devices are docked), the terms may be arbitrarily assigned, but in general, a principal device may refer to a device serving a slave device. Alternatively, the primary device refers to the device on which the trained autonomous agent is installed. To illustrate, in the context of airborne refueling, the primary device is the tanker aircraft. Similarly, a slave device refers to the other device of a pair of devices. To illustrate, in the context of airborne refueling, the slave device is the receiving aircraft (eg, the aircraft being refueled). Further, the term device may be used to refer to any body, system, or component that is operated as a unit (e.g., in the case of a secondary device) or cooperates to accomplish a task (e.g., in the case of a primary device). Widely used to contain assemblies.

In one particular aspect, the system employs a camera (eg, a single camera) to capture monocular video of at least a portion of each of the devices undergoing a docking operation. For example, the camera may capture images of a portion of the refueling boom and a portion of the receiving aircraft. As another example, a camera may capture images of portions of two spacecraft docking ports. An autonomous agent is trained to output a real-time directional indicator of the procedure for positioning the coupler of the device undergoing docking. This directional indicator can be output in a manner that can be interpreted and executed by either operating personnel or an automated system. In some implementations, autonomous agents are also, or alternatively, trained to limit the steps that can be taken to reduce the likelihood of unintended contact between devices.

To illustrate, one form of airborne refueling employs complex targeting operations combined with controlling the docking of a boom from a tanker aircraft into a receiving aircraft receptacle. During this operation, the operator interprets the images from the cameras and guides the boom to dock with the receiving aircraft receptacle while both aircraft are in motion. Operators control the relative angles of the booms as well as the extended length of the booms. Its operation can be complicated by relative aircraft movements, poor visibility, poor lighting conditions, and the like. Further complications can arise when operators control the boom in three-dimensional (3D) space based on two-dimensional (2D) information from one or more cameras. For example, by interpreting 2D information from images, and for vibrations of the boom due to turbulence, the depth perception of the operator, the ability of the operator to predict the point of contact with the boom receiver, and the operator's ability to evaluation becomes complicated. Incorrect interpretation of 2D information can lead to unsuccessful refueling. This is because the receiving aircraft or boom can be damaged if the boom contacts a portion of the receiving aircraft other than the socket. The same or similar difficulties may exist when two spacecraft are docked.

In one particular aspect disclosed herein, machine learning is used to train autonomous agents to replace or assist workers. For example, based on 2D image data and sensor data (e.g., position encoder data), the autonomous agent processes the available information in a manner that enables precise localization of two devices in 3D space. good. As another example, an autonomous agent may determine (under certain circumstances) the optimal control scheme for connecting connectors of two devices.

In one particular implementation, a U-Net Convolutional Neural Network (CNN) is trained as a feature extractor to map a target image to a binary mask (or multiple masks). In this embodiment, the reliability of feature detection is improved by using images with different levels of detail to train the feature extractor. For example, an original image or a source image can be downsampled one or more times to generate one or more downsampled images, and the feature extractor uses the source image and the one or more downsampled images to be trained. By training the feature extractor using multiple images with varying levels of detail, the chances of the feature detector detecting specific details of features rather than general attributes are reduced. . In turn it improves the reliability of the feature detector. To illustrate, the feature extractor can detect general attributes even when specific and detailed features are ambiguous.

By downsampling the image, higher frequency content that represents feature detail is removed or reduced. The feature extractor is thereby encouraged to learn more general features such as general shape and geometry. The U-Net CNN architecture allows generating feature data at various levels of downsampling and combining the feature data to generate the feature data output. By generating combined feature data in this manner, reliability is improved (by using lower resolution images) while accuracy is preserved (by using higher resolution images). A feature extractor trained in this manner can have an error rate that is orders of magnitude lower than a feature extractor using a single image resolution.

The feature data generated by the feature extractor identifies the coordinates of keypoints of the device in the image. In one particular aspect, an imputation network is trained to recognize and repair outliers in the feature data based on the known geometries of the device. For example, the imputation network may compare known geometries of a refueling boom and a particular receiving aircraft to keypoints and may remove or relocate erroneous keypoints. . In some aspects, a Kalman filter is used to temporally filter the image and/or feature data to further improve accuracy. In some implementations, in addition to image data, the feature extractor uses supplemental sensor data, such as data from one or more of lidar systems, radar systems, etc., to generate feature data. you can

In one particular aspect, feature data is provided as input to a trained autonomous agent. In some aspects, the trained autonomous agent also receives as input position data indicating the position in 3D space of the primary device's first connector (e.g., boom refueling coupler in air refueling). ). A trained autonomous agent is configured to generate recommended procedures based on feature data and location data. The recommended procedure shows the movement of the master device's connector to connect to the slave device's connector.

In one particular implementation, autonomous agents are trained using reinforcement learning techniques. For example, in reinforcement learning, rewards are determined based on how well an autonomous agent performs a desired action. Additionally or alternatively, a penalty is determined based on how badly the autonomous agent performs the desired action. In one particular aspect, reinforcement learning is used to train an autonomous agent to determine the optimal procedure, such as the shortest or least expensive procedure for connecting connectors. In another particular aspect, reinforcement learning is used to train autonomous agents to imitate one or more highly skilled workers. If the autonomous agent successfully connects the connectors of the two devices without the master device's connector contacting any surface of the slave device except the slave device's connector, a reward can be given. Additionally or alternatively, penalties may be provided if an autonomous agent causes any unintended contact between a portion of the principal device and a portion of the receiving device.

One advantage of the disclosed systems and methods is that the machine learning-based docking process can be parallelized for execution on one or more graphical processing units (GPUs) rather than computer vision techniques that rely on pattern matching. It is also possible to operate quickly. For example, pattern matching techniques generally operate at about one frame every few seconds, while the techniques of this disclosure can operate at over 20 frames per second. In addition, the autonomous agent can mimic a worker by learning patterns used by highly skilled workers to connect connectors on devices, and improve these patterns to achieve sub-optimal behavior. can be eliminated or combined with the best practices of several different skilled workers.

The drawings and the following description set forth specific illustrative aspects. Those skilled in the art will recognize various configurations that embody the principles described herein and fall within the contemplated scope of the claims that follow this description, even though not explicitly described or illustrated herein. It will be appreciated that both can be devised. Furthermore, any examples given herein are intended to aid in understanding the principles of the disclosure and are not to be considered limiting. As a result, the disclosure is not limited to the specific embodiments or examples described below, but by the claims and their equivalents.

Particular implementations are described herein with reference to the drawings. In the present description, common features are indicated by common reference numerals throughout the drawings. In some drawings, multiple examples of a particular type of feature are used. Although these features are physically and/or logically different, the same reference number is used for each, and different instances are distinguished by adding a letter to the reference number. When features are referred to herein as groups or classes (eg, when no specific one of the features is referenced), reference numbers are used without distinguishing letters. However, when referring herein to one particular feature of a plurality of features of the same type, reference numerals are used with distinguishing letters. For example, referring to FIG. 1, it shows an embodiment encompassing any primary device (e.g., first device 102) and any secondary device (e.g., second device 112), while: 2, 3, 5, and 6 illustrate particular embodiments of a primary device (e.g., tanker vehicle 102A or first spacecraft 102B) and particular embodiments of a secondary device (e.g., receiving side). aircraft 112A or a second spacecraft 112B). Referring to a particular one of the specific examples, such as the one shown in FIG. 2, the primary device is referred to as tanker aircraft 102A. However, referring to any random or generic example of FIG. 1, the first device 102 is used without the distinguishing letter "A". Unless otherwise specified in a particular context, each generic description (eg, description of first device 102) is also a description of each specific example (eg, aircraft 102A).

Various terms, as may be used herein, are used for the purpose of describing particular embodiments only and are not intended to be limiting. For example, the singular forms "a, an" and "the" are intended to include the plural unless the context clearly indicates otherwise. Moreover, some features described in this specification may be present in the singular in some implementations and in the plural in other implementations. To illustrate, FIG. 1 depicts a system 100 including one or more processors (“(one or more) processor(s) 120” in FIG. 1). It illustrates that in some implementations system 100 includes a single processor 120 and in other implementations system 100 includes multiple processors 120 . As readily referred to herein, such features are generally introduced as "one or more" features and henceforth in the singular, unless aspects relating to a plurality of such features are described. point to the characteristics.

The terms "comprise, comprises, comprising" are used interchangeably with "include, includes, including." Further, the term "wherein" is used interchangeably with the term "wherein". As used herein, "exemplary" refers to an example, an implementation, and/or an aspect and is not limiting. It should not be construed as exclusive or as indicating one preferred or preferred embodiment. As used herein, sequential terms (e.g., "first," "second )", "third", etc.) does not in itself indicate a priority or order of that element over another, but rather (except for the use of ordering terms) rather It simply distinguishes the element from another one with the same name. As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to a plurality of elements.

As used herein, "generating," "calculating," "using," "selecting," "accessing," and "determining." "determining" are interchangeable unless the context indicates otherwise. For example, "generating," "calculating," or "determining" a parameter (or signal) refers to actively generating, calculating, or determining the parameter (or signal) or, for example, can refer to using, selecting, or accessing parameters (or signals) that have already been produced by a component or device of the As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) Any combination can be included. Two devices (or components) may be directly or indirectly coupled (communicatively coupled, electrically coupled, or physically coupled). Two devices (or components) that are electrically coupled can be included in the same device or different devices, by way of illustrative non-limiting examples, electronics, one or more connectors, or inductive coupling. can be combined with In some embodiments, two devices (or components) that are communicatively connected, such as by telecommunications, transmit electrical signals (digital or analog signals) can be sent and received. As used herein, "directly coupled" means coupled without intervening components (e.g., communicatively connected, electrically coupled, or physically coupled) Used to describe two devices.

1 to generate a suggested procedure for connecting the first connector 106 of the first device 102 to the second connector 116 of the second device 112 based on image data and location data. A system 100 is shown that includes several devices, including a first device 102 that has been configured. In the example shown in FIG. 1, the first device 102 comprises or corresponds to a primary device, as described above, and the second device 112 comprises a secondary device, as described above. Contain or correspond to. For example, first device 102 may include trained autonomous agent 132 and be configured to serve or support second device 112 . A second device 112 is a device or system configured to couple to the first device 102 and potentially a device or system configured to be serviced or supported by the first device 102 . include.

First device 102 includes a moveable coupling system 104 configured to move first connector 106 relative to second connector 116 of second device 112 . For example, movable coupling system 104 may include a steerable boom of a refueling system, as described in greater detail in FIGS. As another example, movable coupling system 104 may include a steerable docking arm of a docking system, as described in greater detail in FIGS. The examples referenced above are merely illustrative and non-limiting.

First device 102 also includes camera 122 , one or more sensors 124 , one or more processors 120 , and memory 140 . In the example shown in FIG. 1, first device 102 also includes one or more image processors 114 . In some implementations, image processor(s) 114 and processor(s) 120 are combined. Illustratively, one or more GPUs, one or more central processing units (CPUs), or one or more other multi-core or multi-threaded processing units are integrated with image processor(s) 114 and processor(s) 120. can work as both.

In FIG. 1, sensor(s) 124 include position encoder 126 and one or more complementary sensors 128 . Position encoder 126 is coupled to moveable coupling system 104 and configured to generate position data 178 indicative of the position of first connector 106 within 3D space 108 . For example, position encoder 126 may generate data indicative of angles of one or more joints of moveable coupling system 104, deployed length of moveable coupling system 104, other 3D position data, or a combination thereof. good. In this illustrative example, position data 178 indicates a position within a frame of reference associated with first device 102 and independent of second device 112 . To illustrate, the angles of the joints of the moveable coupling system 104 may be indicated relative to the joint's frame of reference or the first device's 102 frame of reference. The angle by itself does not indicate the position of the first connector 106 or joint relative to the second device 112 . This is because the second device 112 is movable with respect to the first device 102 .

Supplemental sensor(s) 128, if present, provide supplemental sensor data (e.g., additional position data). For example, the supplemental sensor(s) 128 may include a rangefinder (eg, a laser rangefinder), and the supplemental sensor data may transmit range data (eg, a rangefinder to the second device). 112). Additionally or alternatively, the supplemental sensor(s) 128 may include a radar system, and the supplemental sensor data may include radar data (eg, distance to the second device 112, distance to the second device 112 or both). Additionally or alternatively, the acquisition sensor(s) 128 may include a lidar system, and the supplemental sensor data may include lidar data (e.g., distance to second device 112, distance to second device 112, 112 or both). Additionally or alternatively, the capture sensor(s) 128 may include a sonar system, and the supplemental sensor data may include sonar data (e.g., distance to second device 112, distance to second device 112, or both). Additionally or alternatively, the capture sensor(s) 128 may include one or more additional cameras (eg, in addition to camera 122), and the supplemental sensor data includes stereoscopic image data. OK.

Camera 122 of first device 102 is configured to generate image data (eg, image(s) 170) depicting at least a portion of moveable coupling system 104 and at least a portion of second device 112. It is In some implementations, the image(s) 170 represents the relative position of the moveable coupling system 104 and the second device 112 in real-time (e.g., video front-end processing delays and buffering). contains a stream of video frames, which are few).

In one embodiment of FIG. 1, image(s) 170 are processed by image processor(s) 114 to generate processed image data. For example, image processor(s) 114 of FIG. 1 includes downsampler 134 configured to generate one or more downsampled images 172 based on image(s) 170 . In some implementations, downsampler 134 may include multiple downsampling stages that generate various levels of downsampled image(s) 172 . To illustrate, a source image from camera 122 may be downsampled for the first time to produce a first downsampled image, which is downsampled one or more additional times. A second downsampled image may be generated. Downsampled image(s) 172 may include a first downsampled image, a second downsampled image, one or more further images, or a combination thereof. Accordingly, in some implementations, multiple downsampled images 172 may be generated for each source image 170 from camera 122 .

In one embodiment of FIG. 1, image processor(s) 114 also generates one or more segmentation maps 174 (“(one or more) segmentation maps” in FIG. 1) based on image(s) 170 . It also includes a partition model 136 configured to generate The segmentation map(s) 174 associated with the first of the image(s) 170 represent distinguishable regions of the first image. For example, the partition map 174 may include boundaries between various portions of the second device 112, boundaries between the portions of the second device 112 and the first device 102 represented in the first image, A boundary between the second device 112, the first device 102, or both, and the background region, or a combination thereof, may be characterized.

Image(s) 170 , downsampled image(s) 172 , segmentation map(s) 174 , or combinations thereof are used as inputs to feature extraction model 130 to generate feature data 176 . provided to In one particular aspect, feature data 176 includes first coordinates representing keypoints of movable coupling system 104 and includes second coordinates representing keypoints of second device 112 . For example, an exemplary display 152 showing image 154 of image(s) 170 having at least a portion of feature data 176 is shown in FIG. In this illustrative example, image 154 depicts first portion 156 of first device 102 (such as a portion of moveable coupling system 104 ) and second portion 162 of second device 112 . A first portion 156 includes a keypoint 158 that represents or is represented by a first coordinate 160 and a second portion 162 represents or is represented by a second coordinate 166 . contains key points 164 represented by . Keypoints 158 , 164 and coordinates 160 , 166 are determined by feature extraction model 130 and represented within feature data 176 .

In some implementations, feature extraction model 130 includes or corresponds to a machine learning model. To illustrate, the feature extraction model 130 may include a neural network trained to detect keypoints 158, 164 within the image 154 and determine the coordinate location within the image 154 associated with each keypoint. or correspond to it. In some implementations, feature extraction model 130 includes or corresponds to one or more convolutional neural networks, such as U-CNN. In such implementations, the downsampler 134 and the feature extraction model 130 may be combined within the U-CNN architecture. Thereby, a particular image (one or more) of images 170 is evaluated by the CNN at multiple different resolutions to generate feature data 176 . Additionally or alternatively, the downsampler 134 and the partitioning model 136 are combined within the U-CNN architecture. This allows a segmentation map to be generated based on multiple resolutions of image 170 .

In one particular aspect, feature extraction model 130 is configured to generate feature data 176 based, at least in part, on known geometry 142 of second device 112 . For example, memory 140 of FIG. 1 stores data representing known geometric dimensions 142 of second device 112 and feature extraction model 130 detected in image(s) 170. Keypoints are compared to known geometry 142 to detect and correct misplaced keypoints. As one particular example, the feature extraction model 130 is an imputation network that compares known geometric dimensions 142 to keypoints and provides an imputation to remove or relocate erroneous keypoints. may include a computational network. In some aspects, feature extraction model 130 also includes a Kalman filter to temporally filter image(s) 170 and/or feature data 176 to improve accuracy of feature data 176 .

Feature data 176 and location data 178 are provided as inputs to trained autonomous agent 132 . In one particular aspect, trained autonomous agent 132 generates suggested procedures 180 for connecting first connector 106 to second connector 116 based on feature data 176 and location data 178. Configured. In one particular implementation, trained autonomous agent 132 includes or corresponds to a neural network. As one example, the neural network of trained autonomous agents 132 is trained using one or more reinforcement learning techniques. To illustrate, during the training phase, the reinforcement learning technique applies the proposed procedure 180 output by the neural network in a particular situation (e.g., for a particular set of input feature data 176 and a particular set of position data 178). ) may train the neural network based in part on a reward determined by comparison to an optimal or target procedure. In this context, the optimal or target procedure is, for example, the shortest or least expensive procedure for connecting the connectors 106, 116, mimicking the procedure performed by one or more skilled personnel under similar circumstances. a set of safety conditions that do not result in any undesired contact between parts of the device 102, 112, a procedure corresponding to the handling characteristics specified during or prior to training, or a combination thereof. may contain

Suggested procedures 180 generated by trained autonomous agent 132 may be output to navigation system 144, to graphical user interface (GUI) engine 150, or both. Steering system 144 is configured to generate and/or execute instructions to reposition first device 102, moveable coupling system 104, or both. To illustrate, steering system 144 may include or correspond to a control system configured to command repositioning of moveable coupling system 104 based on proposed procedures 180 . As a particular example, when the first device 102 includes a tanker aircraft, the maneuvering system 144 includes a tanker aircraft flight control system, a refueling boom boom control system, or both. .

In some embodiments, steering system 144 includes or is coupled to one or more instruments 146, procedure limiters 148, or both. The instrument(s) 146 include or are coupled to control and/or safety sensors associated with the first device 102 . In one particular implementation, data from device(s) 146 is provided to trained autonomous agent 132 to generate suggested procedure 180 . For example, when the first device 102 corresponds to a tanker aircraft, the equipment(s) 146 may correspond to the tanker aircraft's aircraft equipment (e.g., altimeter, angle of attack indicator, heading indicator, airspeed indicator). etc.). In this example, flight data generated by device(s) 146 (along with feature data 176 and position data 178) may be provided to trained autonomous agent 132 to generate suggested procedure 180. .

A procedure limiter 148 performs checks associated with the steering system 144 to determine if procedure limit criteria are met. If procedure limiter 148 determines that the procedure limit criteria are met, procedure limiter 148 causes steering system 144 to limit the maneuver operations that may be performed. As one example, procedure limiter 148 may provide data to trained autonomous agent 132 to limit the set of procedures that may be selected as suggested procedures 180 . Illustratively, the procedure limit criteria may indicate a minimum altitude for airborne refueling, and the procedure limiter 148 indicates that the altimeter (e.g., one of the instruments 146) When the first device 102) indicates that it is below the minimum altitude, the trained autonomous agent 132 moves the boom refueling connector (eg, first connector 106) to the fuel receptacle (eg, second connector). 116) may be prevented from proposing a procedure for connecting to . Additionally or alternatively, procedure limiter 148 may limit steering actions that may be performed by first device 102 or moveable coupling system 104 based on suggested procedure 180 .

GUI engine 150 is configured to generate display 152 and to provide display 152 to a display device that may or may not be mounted on first device 102 . Display 152 includes image 154 depicting first portion 156 of first device 102, second portion 162 of second device 112, or both. In some implementations, keypoints 158 , 164 , coordinates 160 , 166 , or both are depicted within display 152 . In other embodiments, keypoints 158 , 164 and coordinates 160 , 166 are determined as part of feature data 176 but not drawn within display 152 . In some implementations, display 152 includes one or more graphical elements 168 that overlap at least a portion of image 154 and indicate suggested procedures 180 . For example, graphical element(s) 168 may indicate a direction to steer moveable coupling system 104, an amount (e.g., angular displacement or distance) to steer moveable coupling system 104, or both. .

The trained autonomous agent 132, in conjunction with other features of the first device 102, increases the efficiency of operations (e.g., by reducing training costs) for connecting the first connector 106 and the second connector 116. ), improving reliability and reproducibility. For example, trained autonomous agents 132 can mimic procedures performed by highly skilled workers without the time and expense required to train the workers. For example, trained autonomous agent 132 can improve procedures performed by skilled workers by determining procedures that are more optimal than those performed by skilled workers.

Although FIG. 1 depicts first device 102 including supplemental sensor(s) 128 , in some implementations supplemental sensors 128 are omitted or trained autonomous agent 132 . not used to generate input to For example, position data 178 may be determined from the output of position encoder 126 alone.

Although FIG. 1 depicts first device 102 including image processor(s) 114, in other implementations, image(s) 170 are provided to feature extraction model 130 as inputs. are not preprocessed by the image processor(s) 114 before being processed.

Although FIG. 1 depicts navigation system 144 , GUI engine 150 , and display 152 not onboard first device 102 , in other implementations, none of navigation system 144 , GUI engine 150 , or display 152 may be included. are mounted on the first device 102 .

Although segmentation model 136, downsampler 134, and feature extraction model 130 are depicted as separate components in FIG. 1, in other embodiments, segmentation model 136, downsampler 134, and feature extraction model 130 are Two or more of the described functions of may be performed by a single component. In some implementations, one or more of segmentation model 136, downsampler 134, and feature extraction model 130 are implemented in hardware, such as via an application specific integrated circuit (ASIC) or field programmable gate array (FPGA). can be expressed as In some implementations, the operations described with reference to one or more of segmentation model 136, downsampler 134, and feature extraction model 130 are performed by a single processor (eg, processor(s) 120 or by the image processor(s) 114), or by multiple processors using serial operation, parallel operation, or a combination thereof.

FIG. 2 illustrates a system 200 configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data. System 200 is one specific, non-limiting example of system 100 . In one embodiment of FIG. 2, first device 102 corresponds to tanker aircraft 102A and movable coupling system 104 corresponds to refueling boom 104A. In this example, first connector 106 corresponds to refueling connector 106A of refueling boom 104A. Further, the second device 112 corresponds to the receiving aircraft 112A and the second connector 116 corresponds to the receiving receptacle 116A of the receiving aircraft 112A.

In FIG. 2 , tanker aircraft 102 A includes sensor(s) 124 and camera 122 . Each of them operates as described with reference to FIG. To illustrate, the sensor(s) 124 may include a position encoder 126 that produces position data 178 indicative of the position of the refueling boom 104A or of the refueling connector 106A within the frame of reference of the tanker aircraft 102A. .

Additionally, tanker aircraft 102A includes fuel tanks 202 for supplying fuel to receiving aircraft 112A via refueling boom 104A. Tanker aircraft 102 A also includes a flight control system 204 coupled to one or more aircraft 208 . Flight control system 204 is configured to control or facilitate control of the flight operations of tanker aircraft 102A. Tanker aircraft 102A also includes boom controller 206 for controlling or facilitating control of refueling boom 104A. Airplane instrument(s) 208, flight control system 204, or a combination thereof are coupled to aircraft sensors 210 for receiving flight data. Flight data may be used by trained autonomous agent 132 to generate suggested procedures 180 or may be used by procedure limiter 148 to limit suggested procedures 180 . In some implementations, flight control system 204, boom controller 206, or both include, correspond to, or are included in steering system 144 of FIG.

Camera 122 of tanker aircraft 102A is positioned to capture image(s) 170 depicting at least a portion of refueling boom 104A and receiving aircraft 112A during an airborne refueling operation. . FIG. 3 is a diagram showing one particular embodiment of display 152A generated based on images from camera 122 of FIG. In FIG. 3, display 152A is a specific example of display 152 in FIG. For example, display 152A depicts a portion of receiving aircraft 112A and a portion of refueling boom 104A. In this particular example, display 152A also depicts refueling connector 106A and refueling refueling receptacle 116A of receiving aircraft 112A. In one embodiment, shown in FIG. 3, display 152A also includes graphical element 302 that shows suggested procedure 180. As shown in FIG. In one example shown in FIG. 3, the proposed procedure 180 suggests moving the refueling connector 106A to the left (within the frame of reference shown).

4A, 4B, 4C, and 4D are diagrams illustrating various aspects of image processing and feature detection by system 200 of FIG. 2, according to some implementations. In particular, FIG. 4A shows an example of a source image 400 depicting receiving aircraft 112A of FIG. FIG. 4B depicts an example 410 of keypoints, such as exemplary keypoints 412 and 414 detected by the feature extraction model 130 of FIG. In one example 410 of FIG. 4B, a keypoint 414 is an example of a wrong position keypoint to be placed at a position 416 based on the known geometry 142 of the receiving aircraft 112A. is. FIG. 4C depicts an example 420 in which the keypoints 414 of FIG. 4B have been relocated based on the known geometry 142 of the receiving aircraft 112A. FIG. 4D shows an example 430 in which the source image 400 was processed by the segmentation model 136 to generate the segmentation map 174 . For example, in FIG. 4D, various segments, such as exemplary segment 432 of source image 400, are distinguished by different fill patterns.

FIG. 5 illustrates a system 500 configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data. System 500 is one specific, non-limiting example of system 100 . In one example of FIG. 5, first device 102 corresponds to first spacecraft 102B and movable coupling system 104 corresponds to docking arm 104B. In this example, first connector 106 corresponds to docking connector 106B of docking arm 104B. Additionally, a second device 112 corresponds to a second spacecraft 112B and a second connector 116 corresponds to a docking connector 116B of the second spacecraft 112B.

In FIG. 5, first spacecraft 102B includes sensor(s) 124 and camera 122, each of which operates as described with reference to FIG. To illustrate, the sensor(s) 124 may include a position encoder 126 that produces position data 178 indicative of the position of the docking arm 104B or docking connector 106B relative to the frame of reference of the first spacecraft 102B.

First spacecraft 102B includes control system 504 coupled to one or more instruments 506 . Control system 504 is configured to control or facilitate control of the operation of first spacecraft 102B. First spacecraft 102B also includes docking controller 508 for controlling or facilitating control of docking arm 104B. Control system 504, docking arm 104B, or both are coupled to sensor 510 to receive data. The data may be used by trained autonomous agent 132 to generate suggested procedures 180 or may be used by procedure limiter 148 to limit suggested procedures 180 . In some implementations, control system 504, docking controller 508, or both include, correspond to, or are included in steering system 144 of FIG.

Camera 122 of first spacecraft 102B is positioned to capture image(s) 170 depicting at least a portion of docking arm 104B and second spacecraft 112B during a docking operation. FIG. 6 is a diagram showing one particular example of a display 152B generated based on images from camera 122 of FIG. In FIG. 6, display 152B is a specific example of display 152 in FIG. For example, display 152B depicts a portion of the second spacecraft and a portion of docking arm 104B. In this particular example, display 152B also depicts docking connector 106B and docking connector 116B of second spacecraft 112B. In one embodiment, shown in FIG. 6, display 152B also includes graphical element 602 that shows suggested procedure 180. FIG. In one example shown in FIG. 6, suggested procedure 180 suggests moving docking connector 106B to the left (within the frame of reference shown).

7A, 7B, 7C, and 7D are diagrams illustrating various aspects of image processing and feature detection by system 500 of FIG. 5, according to some implementations. In particular, FIG. 7A shows an example of a source image 700 depicting second spacecraft 112B of FIG. FIG. 7B depicts an example of keypoints 710 including exemplary keypoints 712 and 714 detected by the feature extraction model 130 of FIG. In one embodiment 710 of FIG. 7B, a keypoint 714 is an implementation of a wrong position keypoint to be placed at a position 716 based on the known geometry 142 of the second spacecraft 112B. For example. FIG. 7C depicts an example 720 in which the keypoints 714 of FIG. 7B have been relocated based on the known geometry 142 of the second spacecraft 112B. FIG. 7D shows an example 730 in which the source image 700 was processed by the segmentation model 136 to generate the segmentation map 174 . For example, in FIG. 7D, various segments of source image 700, such as exemplary segment 732, are distinguished by different fill patterns.

FIG. 8 is a flowchart of one embodiment of a method 800 for generating suggested procedures for connecting a first connector to a second connector based on image data and position data. Method 800 may be performed by first device 102 or one or more components thereof, including image processor(s) 114, processor(s) 120, downsampler 134, segmentation model 136, feature extraction model. 130, a trained autonomous agent 132, or a combination thereof.

The method 800 includes, at 802, providing the first image as input to a feature extraction model to generate feature data. The first image depicts a first portion of the first device and a second portion of the second device. For example, one or more of the images 170 (one or more), one or more of the downsampled images 172 (one or more), or both may form the first portion 156 and the second portion 162. may be drawn and provided as input to the feature extraction model 130 of FIG. The feature data generated by the feature extraction model includes first coordinates representing keypoints of a first device depicted within the first image, and a first coordinate depicted within the first image. 2 includes second coordinates representing keypoints of the device. For example, feature data 176 may indicate a first coordinate 160 representing keypoint 158 depicted within image 154 and a second coordinate 160 representing keypoint 164 depicted within image 154 . Coordinates 166 may be indicated.

The method 800 also includes, at 804, obtaining position data indicative of a position in 3D space of the first connector of the first device from one or more sensors onboard the first device. . In that case, the first connector is arranged on the first portion of the first device. For example, sensor(s) 124 generate position data 178 . Position data 178 indicates the position of first connector 106 within the frame of reference of first device 102 .

At 806, the method 800 provides feature data and location data as inputs to a trained autonomous agent to generate a suggested procedure for connecting the first connector to the second connector of the second device. further comprising: For example, processor(s) 120 may be trained with feature data 176 and location data 178 as input to generate suggested procedures 180 for connecting first connector 106 and second connector 116 . provided to the autonomous agent 132 .

FIG. 9 is a flowchart of another embodiment of a method 900 for generating suggested procedures for connecting a first connector to a second connector based on image data and position data. Method 900 may be performed by first device 102 or one or more components thereof, including image processor(s) 114, processor(s) 120, downsampler 134, segmentation model 136, feature extraction model. 130, a trained autonomous agent 132, or a combination thereof.

Method 900 includes, at 902, receiving a source image. For example, the source image may include or correspond to one of the image(s) 170 captured by camera 122 .

At 904, method 900 includes downsampling the source image one or more times to generate a first image. For example, downsampler 134 downsamples one of image(s) 170 to produce one of downsampled image(s) 172 as the first image. good. In some embodiments, downsampler 134 downsamples the source image more than one time (eg, two or more times) to generate the first image. To illustrate, downsampler 134 may downsample the source image one or more times to generate a first image, and one or more further samples of the first image to generate a second image. You can downsample as many times as you like.

At 906, the method 900 combines at least one of a plurality of images (eg, a source image, a first image, a second image, or a combination thereof) as input to generate feature data. Including providing to the extraction model. In some examples, the method 900 includes generating a segmentation map based on at least one of the plurality of images and providing the segmentation map as input to the feature extraction model as well. For example, one or more of the image(s) 170, one or more of the downsampled image(s) 172, the segmentation map 174, or a combination thereof may be input to the feature extraction model of FIG. 130. In this example, the feature data generated by feature extraction model 130 includes first coordinates 160 representing keypoints 158 depicted within image 154 and keypoints depicted within image 154 . A second coordinate 166 representing 164 may be shown.

The method 900 includes, at 908, obtaining position data indicative of a position in 3D space of the first connector of the first device. For example, sensor(s) 124 may generate position data 178 . Position data 178 indicates the position of first connector 106 within the frame of reference of first device 102 .

The method 900 includes, at 910, performing a comparison of the coordinates representing the keypoints of the second device to the known geometry of the second device. For example, the second coordinates 166 of the keypoints 164 are compared to the known geometry 142 of the second device 112 to determine if any keypoints are misplaced. good.

At 912, method 900 includes modifying the feature data based on the comparison. For example, in response to determining that one or more of the keypoints 164 are misplaced based on a comparison of the known geometry 142 and the second coordinates 166, the feature data 176 may be modified. In this example, misplaced keypoints 164 may be relocated within feature data 176 or omitted from feature data 176 .

At 914, the method 900 provides feature data and location data as inputs to a trained autonomous agent to generate a suggested procedure for connecting the first connector to the second connector of the second device. including doing For example, processor(s) 120 may be trained with feature data 176 and location data 178 as input to generate suggested procedures 180 for connecting first connector 106 and second connector 116 . provided to the autonomous agent 132 . In some examples, position data 178 includes supplemental sensor data from supplemental sensors 128 . That data is also provided to the trained autonomous agent 132 as input.

The method 900 includes, at 916, generating a procedure-limited output if the procedure-limited criteria are met. For example, the procedure limiter 148 of FIG. 1 may determine whether the procedure limit criteria are met and may generate a procedure limit output in response to the procedure limit criteria being met. In some embodiments, a procedure limit output is also provided as an input to the trained autonomous agent to limit the procedures that can be selected as suggested procedures.

At 918, the method 900 includes generating a GUI having graphical elements that indicate the proposed procedure, instruct relocation of the first device based on the proposed procedure, or both. include. For example, suggested procedure 180 of FIG. 1 may be provided to GUI engine 150 to generate display 152 . In this example, display 152 includes graphical element(s) representing proposed procedure 180 . As another example, suggested procedure 180 may be provided to steering system 144 that may cause first device 102 or portions thereof (such as movable coupling system 104 ) to perform suggested procedure 180 .

FIG. 10 is a flowchart of one embodiment of a method 1000 for training an autonomous agent to generate suggested procedures for connecting a first connector to a second connector based on image data and location data. be. The method 1000 may be performed by one or more processors on or off the first device 102 of FIG. For example, method 1000 may be performed by image processor(s) 114, processor(s) 120, or processor(s) 1320 of FIG.

At 1002, method 1000 includes providing a first image as input to a feature extraction model to generate feature data. The first image depicts a first portion of the first device and a second portion of the second device. For example, one or more of the images 170 (one or more), one or more of the downsampled images 172 (one or more), or both may form the first portion 156 and the second portion 162. may be drawn and provided as input to the feature extraction model 130 of FIG. The feature data generated by the feature extraction model includes first coordinates representing keypoints of a first device depicted within the first image, and a first coordinate depicted within the first image. 2 includes second coordinates representing keypoints of the device. For example, feature data 176 may indicate a first coordinate 160 representing keypoint 158 depicted within image 154 and a second coordinate 160 representing keypoint 164 depicted within image 154 . Coordinates 166 may be indicated.

The method 1000 also includes, at 1004, obtaining position data indicative of a position in 3D space of the first connector of the first device from one or more sensors onboard the first device. . In that case, the first connector is arranged on the first portion of the first device. For example, sensor(s) 124 generate position data 178 . Position data 178 indicates the position of first connector 106 within the frame of reference of first device 102 .

At 1006, the method 1000 provides feature data and location data as inputs to a machine learning model to generate suggested procedures for connecting the first connector to the second connector of the second device. further includes For example, feature data 176 and location data 178 may be provided as inputs to a machine learning model to generate suggested procedures 180 for connecting first connector 106 and second connector 116 .

Method 1000 further includes, at 1008, generating an estimated reward based on the proposed procedure. For example, the reward value may be determined using the estimation function of reinforcement learning techniques.

At 1010, method 1000 further includes modifying a decision scheme of the machine learning model based on the reward value to generate a trained autonomous agent. For example, the decision scheme that generates the suggested procedures may be updated.

In some implementations, image data, location data, or other data used to train a machine learning model to generate a trained autonomous agent is simulated or pre-recorded. For example, the simulator of system 100 can be used to generate image data, position data, or both.

FIG. 11 is a flowchart of one embodiment of an aircraft lifecycle 1100 configured to generate suggested procedures for connecting a first connector to a second connector based on image data and position data. be. In the pre-manufacturing stage, the exemplary lifecycle 1100 includes specifications and designs of an aircraft, such as the tanker aircraft 102A described at 1102 with reference to FIG. During aircraft specification and design, lifecycle 1100 may include specification and design of feature extraction model 130, trained autonomous agent 132, or both. At 1104, lifecycle 1100 includes procurement of materials. It may involve procuring material for the feature extraction model 130, the trained autonomous agent 132, or both.

During manufacturing, the lifecycle 1100 includes component and subassembly manufacturing at 1106 and aircraft systems integration at 1108 . For example, the lifecycle 1100 may include the manufacture of components and subassemblies of the feature extraction model 130, the trained autonomous agent 132, or both, and the production of the system of the feature extraction model 130, the trained autonomous agent 132, or both. May include integration. Lifecycle 1100 includes certification and delivery of the aircraft at 1110 and placing the aircraft in service at 1112 . Authorization and delivery may include authorization of the feature extraction model 130, the trained autonomous agent 132, or both, thereby operating the feature extraction model 130, the trained autonomous agent 132, or both. . While in service with the customer, the aircraft may be scheduled for periodic maintenance and maintenance (which may also include modifications, reconfigurations, refurbishments, etc.). At 1114, lifecycle 1100 includes performing maintenance and maintenance on the aircraft. It may include performing maintenance and maintenance of feature extraction models 130, trained autonomous agents 132, or both.

Each of the lifecycle 1100 processes may be performed or performed by system integrators, third parties, and/or operating personnel (eg, customers). For purposes of this specification, system integrators may include, without limitation, any number of aircraft manufacturers and major system subcontractors, and third parties may include, without limitation, any number of vendors, subcontractors, Vendors and suppliers may be included, and workers may be airlines, leasing companies, military organizations, service organizations, and the like.

Aspects of the present disclosure may be described in the context of one embodiment of a vehicle such as a spacecraft or aircraft. One particular example of a vehicle is aircraft 1200 shown in FIG.

In the example of FIG. 12, aircraft 1200 includes fuselage 1218 with multiple systems 1220 and interior 1222 . Examples of systems 1220 include one or more of propulsion system 1224 , electrical system 1226 , environmental system 1228 , and hydraulic system 1230 . Any number of other systems may also be included. System 1220 of FIG. 12 also includes processor(s) 120 , feature extraction model 130 , and trained autonomous agent 132 .

FIG. 13 is a block diagram of a computing environment 1300 including a computing device 1310 configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. be. For example, computing device 1310, or portions thereof, may be configured to execute instructions to initiate, perform, or control one or more of the operations described with reference to FIGS. 1-10.

Computing device 1310 includes one or more processors 1320 . Processor 1320 is configured to communicate with system memory 1330, one or more storage devices 1340, one or more input/output interfaces 1350, one or more communication interfaces 1360, or any combination thereof. In some implementations, processor(s) 1320 may correspond to or be included in image processor(s) 114 or processor(s) 120 of FIG.

System memory 1330 can include volatile memory devices (eg, random access memory (RAM) devices), non-volatile memory devices (eg, read-only memory (ROM) devices), programmable read-only memory devices (eg, read-only memory (ROM) devices), dedicated memory, and flash memory), or both. System memory 1330 stores an operating system 1332, which is the basic input/output system for powering computing device 1310 and for enabling computing device 1310 to operate by users, other programs, or other devices. and a complete operating system that allows it to work with System memory 1330 stores system (program) data 1336, such as data representing known geometry 142 of FIG.

System memory 1330 includes one or more applications 1334 (eg, sets of instructions) executable by processor 1320 . As one example, one or more applications 1334 may include a program executable by processor(s) 1320 to initiate, control, or perform one or more operations described with reference to FIGS. 1-10. Includes instructions. Illustratively, one or more applications 1334 may include (one or more ) includes instructions executable by processor 1320 .

In one particular implementation, system memory 1330 includes a non-transitory computer-readable medium that stores instructions. The instructions, when executed by the processor(s) 1320, cause the processor(s) 1320 to initiate, perform, and perform operations to generate suggested procedures based on image data and location data. Or let it do what it controls. For example, the operations may include providing a first image as input to a feature extraction model to generate feature data, position data indicating the position in three-dimensional space of a first connector of a first device and providing feature data and location data as inputs to a trained autonomous agent to generate a suggested procedure for connecting the first connector to the second connector.

The one or more storage devices 1340 include non-volatile storage devices such as magnetic disks, optical disks, or flash memory devices. In one particular implementation, storage device 1340 includes both removable and non-removable memory devices. Storage device 1340 is configured to store an operating system, images of the operating system, applications (eg, one or more of applications 1334), and program data (eg, program data 1336). In one particular aspect, system memory 1330, storage device 1340, or both include tangible computer-readable media. In one particular aspect, one or more of storage devices 1340 reside external to computing device 1310 .

One or more input/output interfaces 1350 enable computing device 1310 to communicate with one or more input/output devices 1370 to facilitate interaction with a user. For example, one or more input/output interfaces 1350 may include GUI engine 150 of FIG. 1, a display interface, an input interface, or both. For example, input/output interface 1350 is adapted to receive input from a user, receive input from other computing devices, or a combination thereof. In some implementations, the input/output interface 1350 is a serial interface (e.g., universal serial bus (USB) interface or (IEEE (Institute of Electrical and Electronics Engineers) interface standard), parallel interface, display adapter, audio adapter). or conforms to one or more standard interface protocols, including custom interfaces ("IEEE" is a registered trademark of the Institute of Electrical and Electronics Engineers, Inc. of Piscataway, NJ). In embodiments, input/output devices 1370 include one or more user interface devices and displays, including any combination of buttons, keyboards, pointing devices, displays, speakers, microphones, touch screens, and other devices.

Processor(s) 1320 are configured to communicate with devices or controllers 1380 via one or more communication interfaces 1360 . For example, one or more communication interfaces 1360 may include network interfaces. Device or controller 1380 may include, for example, steering system 144 of FIG. 1, one or more other devices, or any combination thereof.

In conjunction with the systems and methods described, the apparatus includes means for providing the first image as input to a feature extraction model to generate feature data. In some implementations, the means for providing the first image as input to the feature extraction model include first device 102, camera 122, image processor(s) 114, downsampler 134, (one or more ) processor 120, one or more other circuits or devices configured to provide an image as input to the feature extraction model, or a combination thereof.

The apparatus also includes means for obtaining position data indicative of a position in three-dimensional space of the first connector of the first device. For example, means for obtaining position data may include first device 102, sensor(s) 124, position encoder 126, processor(s) 120, one or more other devices, or combinations thereof.

The apparatus also provides feature data and location data as input to a trained autonomous agent to generate a suggested procedure for connecting the first connector to the second connector of the second device. including the means of For example, the means for providing feature data and location data as inputs to a trained autonomous agent include the first device 102, the feature extraction model 130, the processor(s) 120, the feature data and location data as inputs. It may correspond to one or more other devices configured to serve autonomous agents, or a combination thereof.

In some implementations, a non-transitory computer-readable medium stores instructions. The instructions, when executed by one or more processors, cause the one or more processors to initiate, perform, or control operations to perform some or all of the functions described above. For example, the instructions may be executable to perform one or more of the acts or methods of FIGS. 1-10. In some implementations, part or all of one or more of the acts or methods of FIGS. It may be implemented by a central processing unit (CPU), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), or any combination thereof.

The example illustrations described herein are intended to provide a general understanding of the structure of the various implementations. These figures are not intended to be an exhaustive description of all the elements and features of the devices and systems utilizing the structures or methods described herein. Many other implementations may become apparent to those of skill in the art upon review of this disclosure. Other implementations may be utilized and other implementations may be derived from the present disclosure such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. For example, method operations may be performed in a different order than shown in the figures, or one or more method operations may be omitted. Accordingly, the present disclosure and figures are to be regarded as illustrative rather than restrictive.

Aspects of the present disclosure are further described with reference to the following set of interrelated clauses.
Clause 1.
providing a first image as input to a feature extraction model to generate feature data, the first image being a first portion of a first device and a second portion of a second device; and wherein the feature data includes first coordinates representing keypoints of the first device depicted in the first image, and the second device depicted in the first image. providing a first image as input to a feature extraction model, including second coordinates representing keypoints; obtaining position data indicating a position in three-dimensional space of a connector of the first connector, wherein the first connector is positioned on the first portion of the first device; and to a trained autonomous agent with the feature data and the location data as inputs to generate a suggested procedure for connecting the first connector to the second connector of the second device. including providing, including methods;
Clause 2.
said first device comprising a tanker aircraft, said first portion of said first device comprising a refueling boom, said first connector comprising a refueling connector of said refueling boom; The method of clause 1, wherein the second device includes an aircraft and the second connector includes a refueling refueling receptacle of the aircraft.
Article 3.
The first device comprises a first spacecraft, the first connector comprises a first docking connector, the second device comprises a second spacecraft, the second connector comprises a second docking connector. including the method of Clause 1, including the docking connector of the
Article 4.
4. The method of any one of clauses 1-3, wherein the position data indicates a position within a frame of reference associated with the first device and independent of the second device.
Article 5.
5. The method of any one of clauses 1-4, wherein the position data is generated based on outputs of one or more position encoders associated with the first portion.
Clause 6.
Further comprising receiving a source image and downsampling the source image one or more times to generate the first image prior to providing the first image as an input to the feature extraction model. , including the method of any one of clauses 1 to 5.
Article 7.
receiving a source image before providing the first image as an input to the feature extraction model; downsampling the source image to generate a second image; generating the first image; and providing the second image as an input to the feature extraction model to generate the feature data. or the method of claim 1.
Article 8.
8. The method of any one of clauses 1-7, further comprising generating a graphical user interface including graphical elements indicative of the suggested procedure.
Article 9.
9. The method of clause 8, wherein the graphical user interface further comprises the first image and the graphical element overlapping at least a portion of the first image within the graphical user interface.
Clause 10.
further comprising generating a procedure-limited output if a procedure-limited criterion is met, the procedure-limited output being a maneuvering action that may be performed by the first device, the first portion of the first device; or a set of procedures that may be selected as said suggested procedures, resulting in a restriction being applied to at least one of the maneuvering actions that may be performed by including the method of
Clause 11.
said trained autonomous agent comprises or corresponds to a neural network trained via a reinforcement learning process to determine a maneuvering action for connecting said first connector and said second connector; Including a method according to any one of clauses 1-10.
Clause 12.
Acquiring supplemental sensor data indicating the relative positions of the first device and the second device in three-dimensional space from the one or more sensors mounted on the first device. and wherein the supplemental sensor data comprises one or more of range data, radar data, lidar data, sonar data, or stereo image data. include.
Article 13.
The first device comprises an aircraft, and the method further obtains flight data from one or more of a flight control system or flight instruments of the aircraft; as an input to the trained autonomous agent.
Article 14.
The first device comprises an aircraft, the method further comprises obtaining flight data from one or more of a flight control system or flight instruments of the aircraft, and the proposed procedure comprises: including the method of any one of clauses 1-13, generated based on the determination that the screening criteria are met.
Article 15.
providing the first image to a segmentation model to generate a segmentation map representing distinguishable regions of the first image; and using the segmentation map as input to generate the feature data. 15. The method of any one of clauses 1-14, further comprising providing said feature extraction model.
Article 16.
performing a comparison of said second coordinates representing said keypoints of said second device with known geometrical dimensions of said second device; and said feature data based on said comparison. 16. The method of any one of clauses 1-15, further comprising modifying the
Article 17.
17. The method of any one of clauses 1-16, further comprising generating a reward value based on the proposed procedure and modifying link weights of the trained autonomous agent based on the reward value. including the method of
Article 18.
18. The method of Clause 17, wherein the reward value is based, at least in part, on how closely the proposed procedure matches a procedure performed by operating personnel.
Article 19.
19, comprising the method of clause 17 or 18, wherein the reward value is based, at least in part, on the time required to connect the first connector and the second connector.
Clause 20.
the reward value is based, at least in part, on whether the suggested procedure results in the first connector contacting any portion of the second device other than the second connector; Including the method of any one of clauses 17-19.
Article 21.
21. The method of any one of clauses 1-20, further comprising instructing relocation of the first portion of the first device based on the suggested procedure.
Article 22.
a movable coupling system configured to move a first connector relative to a second connector of a second device, to generate position data indicative of a position in three-dimensional space of the first connector; a camera configured to generate image data depicting a portion of the moveable coupling system and at least a portion of the second device; a first sensor based on the image data; A feature extraction model configured to receive an image and to generate feature data based on said first image, said feature data being of said movable coupling system depicted in said first image. A feature extraction model comprising first coordinates representing keypoints and comprising second coordinates representing keypoints of said second device depicted in said first image, and said feature data and said location data. a device comprising a trained autonomous agent configured to generate a suggested procedure for connecting said first connector to a second connector of said second device based on a.
Article 23.
said movable coupling system comprising a refueling boom of a tanker aircraft, said first connector comprising a refueling connector of said refueling boom, said second device comprising an aircraft, said A second connector includes the device of Clause 22, including a refueling refueling receptacle of said aircraft.
Article 24.
wherein the movable coupling system includes a first docking connector of a first spacecraft, the second device includes a second spacecraft, and the second connector includes a second docking connector; Including the device according to Clause 22.
Article 25.
25. The position data comprises a device according to any one of clauses 22-24, indicating a position within a frame of reference associated with the device and independent of the second device.
Article 26.
26. The device of any one of clauses 22-25, wherein the position data is generated based on the output of one or more position encoders associated with the moveable coupling system.
Article 27.
27. The device of any one of clauses 22-26, further comprising an image processor configured to receive a source image from the camera and downsample the source image to generate the first image. include.
Article 28.
The image processor is further configured to downsample the first image to generate a second image, wherein the feature extraction model is based on the first image and the second image. 28. A device according to any one of clauses 22 to 27 for generating said characteristic data.
Article 29.
29. Any one of clauses 22-28, further comprising a graphical user interface engine coupled to said trained autonomous agent and configured to output a graphical user interface comprising graphical elements indicative of said proposed procedure. devices.
Clause 30.
30. The device of clause 29, wherein the graphical user interface further comprises the first image and the graphical element overlapping at least a portion of the first image within the graphical user interface.
Article 31.
of a maneuvering action that may be performed by the device, a maneuvering action that may be performed by the steerable boom, or a set of procedures that may be selected as the suggested procedure based on a procedure restriction criterion being met; 31. The device of any one of clauses 22-30, further comprising, at least one, a procedure limiter configured to cause the restriction to be applied.
Article 32.
said trained autonomous agent comprises or corresponds to a neural network trained via a reinforcement learning process to determine a maneuvering action for connecting said first connector and said second connector; A device according to any one of clauses 22-31.
Article 33.
The one or more sensors are further configured to generate supplemental sensor data indicative of relative positions of the device and the second device in three-dimensional space, the supplemental sensor data comprising: 33. A device according to any one of clauses 22-32, comprising one or more of range data, radar data, lidar data, sonar data or stereo image data.
Article 34.
further at least one of a flight control system or a flight instrument configured to generate flight data and provide said flight data as input to said trained autonomous agent to generate said proposed procedure; A device according to any one of clauses 22 to 33, including.
Article 35.
The proposed procedure includes the device of Clause 34, generated based on determining that the flight data satisfies screening criteria.
Article 36.
A segmentation model configured to generate a segmentation map representing distinguishable regions of the first image, wherein the feature extraction model generates the feature data based, at least in part, on the segmentation map. comprising a device according to any one of clauses 22 to 35, adapted to
Article 37.
Further comprising a memory storing data representing known geometric dimensions of said second device, said feature extraction model being at least partially based on said known geometric dimensions of said second device. 37. The device of any one of clauses 22-36, wherein the device is configured to generate the characteristic data based on physical dimensions.
Article 38.
38. The device of any one of clauses 22-37, further comprising a control system configured to command repositioning of the movable coupling system based on the suggested procedure.
Article 39.
a steerable refueling boom including a refueling connector configured to be coupled to a refueling well of said second aircraft during refueling of said second aircraft, to said steerable refueling boom one or more sensors coupled and configured to generate position data indicative of the position of the steerable refueling boom in three-dimensional space; a portion of the steerable refueling boom and the second aircraft; a camera configured to generate image data depicting at least a portion thereof; a feature extractor configured to receive a first image based on said image data and generate feature data based on said first image; a model, wherein the feature data includes first coordinates representing key points of the steerable refueling boom depicted in the first image; and a proposal for connecting the refueling connector to the refueling refueling receptacle based on the feature data and the location data, the feature extraction model comprising second coordinates representing key points of the aircraft of Including a tanker aircraft including a trained autonomous agent configured to generate procedures.
Clause 40.
40. The position data includes the tanker aircraft of clause 39, indicating a position within a frame of reference associated with the tanker aircraft and independent of the second aircraft.
Article 41.
Clause 39 or 40, wherein the one or more sensors include one or more position encoders associated with the steerable refueling boom, and wherein the position data is generated based on outputs of the one or more position encoders including tanker aircraft described in .
Article 42.
42. Air refueling according to any one of clauses 39 to 41, further comprising an image processor configured to receive a source image from the camera and downsample the source image to generate the first image. Including machine.
Article 43.
The image processor is further configured to downsample the first image to generate a second image, wherein the feature extraction model is based on the first image and the second image. 43, including a tanker aircraft according to Clause 42 for generating said characteristic data.
Article 44.
44. A tanker aircraft according to any one of clauses 39 to 43, further comprising a graphical user interface engine configured to output a graphical user interface including graphical elements indicative of said proposed procedure.
Article 45.
45. The tanker aircraft of clause 44, wherein the graphical user interface further comprises the first image and the graphical element overlapping at least a portion of the first image within the graphical user interface.
Article 46.
a set of procedures that may be selected as a maneuvering action that may be performed by the tanker aircraft, a maneuvering action that may be performed by the steerable refueling boom, or the suggested procedure based on a procedure restriction criterion being met; 46, further comprising a procedure limiter configured to cause the restriction to apply to at least one of the tanker aircraft of any one of clauses 39-45.
Article 47.
wherein said trained autonomous agent comprises or corresponds to a neural network trained via a reinforcement learning process to determine a maneuvering action for connecting said refueling connector to said refueling receptacle; A tanker aircraft according to any one of claims 39 to 46.
Article 48.
The one or more sensors are further configured to generate supplemental sensor data indicative of relative positions of the tanker aircraft and the second aircraft in three-dimensional space, wherein the supplemental sensor data contains one or more of range data, radar data, lidar data, sonar data or stereoscopic image data.
Article 49.
further at least one of a flight control system or a flight instrument configured to generate flight data and provide said flight data as input to said trained autonomous agent to generate said proposed procedure; A tanker aircraft according to any one of Clauses 39 to 48, including.
Clause 50.
A segmentation model configured to generate a segmentation map representing distinguishable regions of the first image, wherein the feature extraction model generates the feature data based, at least in part, on the segmentation map. including a tanker aircraft according to any one of Clauses 39 to 49, adapted to
Article 51.
Further comprising a memory storing data representing known geometric dimensions of said second device, said feature extraction model being at least partially based on said known geometric dimensions of said second device. 51. A tanker aircraft according to any one of Clauses 39 to 50, wherein the tanker aircraft is configured to generate the characteristic data based on the physical dimensions.
Article 52.
52. The tanker aircraft of any one of Clauses 39-51, further comprising a control system configured to command repositioning of the steerable refueling boom based on the suggested procedure.
Article 53.
a steerable docking mechanism including a first docking connector configured to be coupled to a second docking connector of a second spacecraft during a docking operation; one or more sensors configured to generate position data indicative of the position of the steerable docking mechanism; and to generate image data depicting a portion of the steerable docking mechanism and at least a portion of the second spacecraft. a feature extraction model configured to receive a first image based on said image data and to generate feature data based on said first image, said feature data comprising said comprising a first coordinate representing a keypoint of said steerable docking mechanism depicted in a first image, and a second coordinate representing a keypoint of said second spacecraft depicted within said first image; a feature extraction model, and training configured to generate a suggested procedure for connecting the first docking connector to the second docking connector based on the feature data and the location data. Includes spacecraft, including autonomous agents.
Article 54.
54. The spacecraft of clause 53, wherein the position data indicates a position within a frame of reference associated with the spacecraft and independent of the second spacecraft.
Article 55.
Clause 53 or 54, wherein the one or more sensors include one or more position encoders associated with the steerable docking mechanism, and wherein the position data is generated based on outputs of the one or more position encoders. including spacecraft.
Article 56.
56. The spacecraft of any one of clauses 53-55, further comprising an image processor configured to receive source images from the camera and downsample the source images to generate the first image. including.
Article 57.
The image processor is further configured to downsample the first image to generate a second image, wherein the feature extraction model is based on the first image and the second image. 57, including the spacecraft of Clause 56 for generating said characterization data.
Article 58.
58. The spacecraft of any one of clauses 53-57, further comprising a graphical user interface engine configured to output a graphical user interface including graphical elements indicative of said proposed procedure.
Article 59.
59. The spacecraft of clause 58, wherein the graphical user interface further comprises the first image and the graphical element overlapping at least a portion of the first image within the graphical user interface.
Article 60.
a set of maneuvers that may be selected as the proposed maneuver, based on which a maneuver limit criterion has been met; 59. The spacecraft of any one of Clauses 53-59, further comprising a procedure limiter configured to cause at least one of the limits to be applied.
Article 61.
said trained autonomous agent comprises or corresponds to a neural network trained via a reinforcement learning process to determine a maneuvering action for connecting said first connector and said second connector; A spacecraft according to any one of Clauses 53 to 60.
Article 62.
The one or more sensors are further configured to generate supplemental sensor data indicative of relative positions of the spacecraft and the second spacecraft in three-dimensional space, wherein the supplementary sensor data includes a spacecraft according to any one of Clauses 53 to 61, including one or more of range data, radar data, lidar data, sonar data, or stereoscopic image data.
Article 63.
A segmentation model configured to generate a segmentation map representing distinguishable regions of the first image, wherein the feature extraction model generates the feature data based, at least in part, on the segmentation map. 63. A spacecraft according to any one of clauses 53 to 62, adapted to
Article 64.
Further comprising a memory storing data representing known geometrical dimensions of the second spacecraft, the feature extraction model being at least in part the known geometry of the second spacecraft. 64. A spacecraft according to any one of clauses 53-63, wherein the spacecraft is arranged to generate said characterization data based on geometrical dimensions.
Article 65.
65. The spacecraft of any one of Clauses 53-64, further comprising a control system configured to command repositioning of the steerable docking mechanism based on the suggested procedure.

Moreover, although specific examples have been illustrated and described herein, any subsequent configuration designed to achieve the same or similar results may be substituted for the specific implementations shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon inspection of the specification.

The "Abstract" of this disclosure is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the above Detailed Description, various features may be grouped together or described within a single implementation for the purpose of conciseness of the disclosure. It should also be understood that the above examples are intended to illustrate rather than limit the present disclosure, and that many modifications and variations may be made in accordance with the principles of the disclosure. As the following claims reflect, claimed subject matter may not cover all features of any of the disclosed examples. The scope of the disclosure is defined by the following claims and their equivalents.

Claims (12)

A method (800, 900) comprising:
providing (802, 906) a first image (170) as an input to a feature extraction model (130) to generate feature data (176), said first image being a first device ( 102) and a second portion (162) of a second device (112), wherein the characteristic data is of the first device depicted in the first image; comprising a first coordinate (160) representing a keypoint (158) and comprising a second coordinate (166) representing a keypoint (164) of said second device depicted in said first image; providing an image of 1 as an input to a feature extraction model;
position data (178) indicative of a position in three-dimensional space (108) of a first connector (106) of said first device from one or more sensors (124) on said first device; obtaining (804, 908), wherein the first connector is located on the first portion of the first device, obtaining location data; providing said feature data and said location data as input to a trained autonomous agent (132) to generate a suggested procedure (180) for connecting to a second connector of said second device; 806, 914). Said first device comprises a tanker aircraft (102A), said first portion of said first device comprises a refueling boom (104A), said first connector comprises fuel for said refueling boom. The method of claim 1, comprising a refueling connector (106A), said second device comprising an aircraft (112A), said second connector comprising a refueling refueling well (116A) of said aircraft. The first device includes a first spacecraft (102B), the first connector includes a first docking connector (106B), and the second device includes a second spacecraft (112B). 3. The method of claim 1 or 2, wherein said second connector comprises a second docking connector (116B). 4. A method according to any one of claims 1 to 3, wherein said position data indicates a position within a frame of reference associated with said first device and independent of said second device. Receiving a source image and down-sampling the source image one or more times to generate the first image before providing the first image as input to the feature extraction model (904). 5. The method of any one of claims 1-4, further comprising: providing said first image to a segmentation model (136) to generate a segmentation map (174) representing distinguishable regions (432, 732) of said first image; and generating said feature data. 6. The method of any one of claims 1 to 5, further comprising providing (906) the segmentation map as an input to the feature extraction model to do so. performing (910) a comparison between said second coordinates representing said keypoints of said second device and a known geometry (142) of said second device; 7. The method of any one of claims 1 to 6, further comprising modifying (912) the feature data based on. 8. The method of any one of claims 1 to 7, further comprising instructing (918) relocation of the first portion of the first device based on the suggested procedure. A device (102),
a moveable coupling system (104) configured to move the first connector (106) relative to the second connector (116) of the second device (112);
one or more sensors (124) configured to generate position data (178) indicative of the position of the first connector in three-dimensional space (108);
a camera (122) configured to generate image data (170) depicting a portion (156) of said moveable coupling system and at least a portion (162) of said second device;
A feature extraction model (130) configured to receive a first image based on said image data and to generate feature data (176) based on said first image, said feature data comprising said comprising a first coordinate (160) representing a keypoint (158) of said moveable coupling system depicted in said first image; a keypoint of said second device depicted within said first image ( 164) for connecting the first connector to a second connector of the second device based on the feature extraction model and the feature data and the location data; a device comprising a trained autonomous agent (132) configured to generate a suggested procedure (180) of The movable coupling system includes a refueling boom (104A) of a tanker aircraft (102A), the first connector includes a refueling boom refueling connector (106A), and the first connector includes a refueling boom (106A). 10. The device of claim 9, wherein two devices comprise an aircraft (112A) and said second connector comprises a refueling refueling receptacle (116A) of said aircraft. The moveable coupling system includes a first docking connector (106B) of a first spacecraft (102B), the second device includes a second spacecraft (112B), the second connector 11. The device according to claim 9 or 10, wherein the includes a second docking connector (116B). The position data indicates a position within a frame of reference associated with the device and independent of the second device, the position data being one or more position encoders (126) associated with the moveable coupling system. 12. A device according to any one of claims 9 to 11, wherein the device is generated based on the output of

Images