WO2023072389A1

WO2023072389A1 - Controlling execution of a perception algorithm

Info

Publication number: WO2023072389A1
Application number: PCT/EP2021/079881
Authority: WO
Inventors: José ARAÚJO; Paula CUBERO; Sandra Hernandez
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2021-10-27
Filing date: 2021-10-27
Publication date: 2023-05-04

Abstract

An electronic device includes at least one processor which operates to obtain performance requirements (110) for a perception algorithm (104) executed by a mobile device (100) to compute at least one of: poses of features of a real-world environment, a pose of the mobile device relative to features of the real-world environment, and/or recognition of objects based on the features of the real-world environment, using data from at least one sensor. The operations obtain runtime performance metrics (120) relating to the perception algorithm. The operations adapt parameters (134, 136) controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm.

Description

CONTROLLING EXECUTION OF A PERCEPTION ALGORITHM

TECHNICAL FIELD

[001] The present disclosure relates to an electronic device for controlling execution of a perception algorithm by a mobile device, a method by an electronic device for controlling execution of a perception algorithm by a mobile device, and a corresponding computer program product.

BACKGROUND

[002] Mobile devices such as augmented reality (AR), virtual reality (VR), and extended reality (XR) glasses, smartphones and robots are becoming much more aware of their real- world environment. Perception algorithms are used to compute pose of mobile device using Simultaneous Localization and Mapping (SLAM), compute a map of the environment using 3D reconstruction, and compute a semantic understanding of the environment properties using scene/object detection, classification, and segmentation. These mobile devices may operate to localize themselves in the real-world environment, algorithmically understand and characterize the real-world environment and may aid users with decision making associated with the real-world environment (e.g. XR) or autonomously execute tasks associated with the real-world environment (e.g. robotic navigation).

[003] Designing mobile devices to run perception algorithms involves complex trade-offs between the performance of sensor-based perception pipelines (e.g., localization error, feature detection error, 3D reconstruction error, classification precision, latency of the algorithms, etc.) and the associated power consumption in resource-constrained mobile devices. Some assumptions made during the design process may be violated during real-world operation when excessive power is consumed, when insufficient computing resources of a mobile device are available to execute a perception algorithm, and/or when more performance is required from the perception algorithm than what is being provided.

[004] Hence, there is a need for more adaptable control of the configuration of perception algorithms and the configuration of computing resources used for executing perception algorithms on mobile devices. SUMMARY

[005] Some embodiments disclosed herein are directed to an electronic device comprising at least one processor which operates to obtain performance requirements for a perception algorithm executed by a mobile device to compute at least one of poses of features of a real- world environment, a pose of the mobile device relative to features of the real-world environment, and/or recognition of objects based on features of the real -world environment, using data from at least one sensor. The operations obtain runtime performance metrics relating to the perception algorithm. The operations adapt parameters controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm.

[006] Some other corresponding embodiments disclosed herein are directed to a method by an electronic device. The method includes obtaining performance requirements for a perception algorithm executed by a mobile device to compute at least one of poses of features of a real-world environment, a pose of the mobile device relative to features of the real -world environment, and/or recognition of objects based on features of the real -world environment, using data from at least one sensor. The method further includes obtaining runtime performance metrics relating to the perception algorithm. The method further includes adapting parameters controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm.

[007] Some other corresponding embodiments disclosed herein are directed to a computer program product comprising a non-transitory computer readable medium storing program instructions executable by at least one processor of an electronic device. The program instructions obtain performance requirements for a perception algorithm executed by a mobile device to compute at least one of poses of features of a real -world environment, a pose of the mobile device relative to features of the real-world environment, and/or recognition of objects based on features of the real -world environment, using data from at least one sensor. The program instructions obtain runtime performance metrics relating to the perception algorithm. The program instructions adapt parameters controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm.

[008] Potential advantages that may be provided by one or more of these embodiments include that the parameters which are used to control computation by the perception algorithm can be dynamically or iteratively adapted to reduce differences between the performance requirements and the runtime performance metrics. Alternatively or additionally, the amount of computing resources and/or runtime properties of the mobile device allocated to execute the perception algorithm can be dynamically or incrementally controlled to reduce differences between the performance requirements and the runtime performance metrics. Runtime performance of the perception algorithm and/or the mobile device can thereby be more optimally tuned so the runtime performance metrics can more optimally satisfy the performance requirements despite a myriad of possible real-world situations, such as when several perception algorithms and/or applications being executed on the mobile device compete for computing resources, when the real-world environment is difficult to perceive using the at least one sensor, and/or when the environment in which the mobile device is located or moving rapidly changes.

[009] Other electronic devices, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such electronic devices, methods, and computer program products be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and/or combination.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:

[0011] Figure 1 illustrates a mobile device that runs a perception algorithm which can be controlled by a performance optimization algorithm based on performance requirements and runtime performance metrics feedback, in accordance with some embodiments of the present disclosure;

[0012] Figure 2 illustrates components of a mobile device that runs a perception algorithm which can be controlled by illustrated components of a server running a performance optimization algorithm based on performance requirements and runtime performance metrics feedback, in accordance with some embodiments of the present disclosure;

[0013] Figure 3 illustrates a flowchart of operations performed by the performance optimization algorithm of Figures 1 and 2 in accordance with some embodiments of the present disclosure;

[0014] Figure 4 illustrates a graph showing a Pareto curve extending through experimental measurements of pose error from a SLAM algorithm as a function of power consumed to run the SLAM algorithm on a mobile device, in accordance with some embodiments of the present disclosure; and

[0015] Figure 5 illustrates a zoomed-in close-up view of a portion of the graph of Figure 4 and shows a performance requirement point on the Pareto curve which is offset from a measured runtime performance metric of the SLAM algorithm, and which offset is used to adapt parameters controlling computation by the SLAM algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the SLAM, in accordance with some embodiments of the present disclosure.

DETAILED DESCRIPTION

[0016] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present/used in another embodiment.

[0017] Some embodiments of the present disclosure have arisen from ongoing experimentation in cross-layer modelling and optimization of perception pipelines. The complexity and non-optimal results of attempting to perform trade-offs between the accuracy of vision-based or structure-based perception pipelines (e.g., localization error, object recognition error, and object classification precision) and the power consumption when perception algorithms are executed in resource-constrained mobile devices. Object classification may include recognizing an object as a table, chair, door, hallway, room, building, street sign, building sign, etc. As used herein, recognition of an object may include recognition of a scene, such as a hallway, room, etc. [0018] Some of this experimentation has been performed on a newly developed computational testbed which instrumentalizes and executes various state-of-the-art localization and mapping algorithms as maplab from ETH Zurich, various object detection and recognition (classifier) algorithms from MaskRCNN, using various computing platforms based on the Nvidia Jetson NX and Nvidia Jetson AGX. 3D reconstruction may be performed using densification algorithms. Example algorithms are described in the following publications: T. Schneider et al., “Maplab: An Open Framework for Research in Visual- Inertial Mapping and Localization”, IEEE Robotics and Automation Letters, vol. 3, pages 1418-1425, 2018; K. He et al., “Mask R-CNN”, arXiv: 1703.06870v3, 2018;

H. Hirschmuller, "Stereo Processing by Semiglobal Matching and Mutual Information", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, pages 328-341, 2007; and H. Oleynikova et al., “Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planning”, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), DOI: 10.1109/IROS.2017.8202315, 2017.

[0019] Performing cross-layer modelling and designing a cross-layer optimizer/resource manager for perception pipelines has been determined to be particularly challenging because of, for example, the output performance (e.g. localization error and power consumption) being highly dependent upon controllable inputs such as the parameters controlling computation by various perception algorithms (e.g., how often to run subprocesses of algorithms, algorithm mathematical coefficients and/or default values, algorithm threshold values, number of algorithm iterations to perform, number of features to be detected, etc.) and/or controlling an amount of computing resources of the mobile device 200 allocated to execute of the various perception algorithms (e.g., number of features to detect per image, number of central processing unit (CPU) cores, power mode of various computing resources, etc.) and, moreover, highly dependent on many non-controllable inputs such computational characteristics of the various computing platforms, (e.g., CPU and/or graphical processing unit (GPU) and effects of utilization of processing resources and memory resources by other concurrently executing application(s)), the environmental conditions (e.g. brightness of the real -world environment perceived by one of more sensors, number of features (e.g., objects) detectable in the scene), the environment knowledge (e.g. existing map availability and its quality), and mobile device operation (e.g. speed of the mobile device). It has been determined that there are large correlations between a large number of these inputs and other characteristics. If the resources of a computing platform are managed in a too aggressive manner (e.g., run in power save mode with limited resources when the mobile device is moving fast), the perception pipeline may deliver unacceptable runtime performance (e.g., localization error may exceed a distance threshold) or even fail to provide an output within a time deadline requirement.

[0020] Hence, when adapting parameters that control computation by a perception algorithm and/or control an amount of computing resources of a mobile device allocated to execute the perception algorithm, the adaptation may be improved by being performed based on a complete system model which is based on the model uncertainty and perform a more optimized design of the adaptation policies in order to better satisfy performance requirements while providing system robustness.

[0021] Through experimentation, these parameters have been adapted to obtain pareto configurations relating runtime performance metrics, e.g., error, and performance requirements, e.g., power consumed by a SLAM algorithm running on a mobile device. Pareto efficiency or Pareto optimality refers to a situation where no individual performance metric can be better off without making at least one individual performance metric criterion worse off or without any loss thereof.

[0022] Figure 4 illustrates a graph showing a Pareto curve extending through experimental measurements of pose error (e.g., room-mean-square error, RMSE) of a SLAM algorithm as a function of power consumed to run the SLAM algorithm on a mobile device, in accordance with some embodiments of the present disclosure. Although various embodiments are described in the context of power requirements and runtime metrics, these and other embodiments may be used with energy requirements and runtime metrics.

[0023] Referring to Figure 4, the pareto curve extends through Pareto configurations 400 of power versus pose error (RMSE) with a few runtime performance samples illustrated from the computational testbed running on an Nvidia Jetson AGX for 10000 different SLAM algorithm and platform configurations. The Pareto curve may alternatively be obtained using multi -objective optimization software tools such as Hypermapper. Figure 4 also illustrates a default configuration 402 and runtime performance samples 404. Various of the SLAM algorithm and platform configurations were set by adapting parameters controlling computation by the SLAM algorithm and/or controlling an amount of computing resources of the computational testbed allocated to execute the SLAM algorithm, to reduce differences between performance requirements and runtime performance metrics during further computation of the SLAM algorithm.

[0024] It has been determined that the Pareto configurations can guide the design and runtime adaptation of parameters controlling performance of SLAM algorithms based on the best expected runtime performance, such as in terms of error and power, for a particular SLAM algorithm. The Pareto region can be used to guide how to adapt a configuration and/or adaptation policy during runtime to tune the SLAM algorithm and computational testbed parameters while staying in, or close to, Pareto configuration.

[0025] The embodiments disclosed herein are provided as non-liming examples for ease of explanation. Although various embodiments of the present disclosure are explained in the context of SLAM pose error and computing power consumption as a type of runtime performance metrics and Pareto analysis, these and other embodiments are not limited thereto. Through experimentation it has been determined that corresponding behavior occurs for other perception metrics characterizing the performance of perception algorithms, such as: feature (or object) detection accuracy vs power consumption, feature (or object) detection latency vs power consumption, feature (or object) detection error vs feature (or object) detection latency, 3D reconstruction error vs power consumption and/or latency, etc.

[0026] Moreover, although various embodiments of the present invention are explained in the context of SLAM algorithms, they are not limited thereto and instead can be used for any type of perception algorithm. A perception algorithm can be any algorithm executable by a mobile device to compute at least one of: poses of features of a real -world environment, a pose of the mobile device relative to features of the real-world environment, and/or recognition of objects based on features of the real -world environment, using data from at least one sensor. Accordingly, a perception algorithm may operate to understand the real- world environment. A perception algorithm may operate to create a computer model of the structure of the real-world environment, which is also referred to as "3D reconstruction, using the computed poses of features of the real-world environment and/or recognized objects based on features of the real -world environment."

[0027] Example embodiments are now described in which a perception algorithm is executed on a mobile device. The perception algorithm and computing resources of the mobile device can have many parameters which can be adapted. In some embodiments, these parameters are adapted using a model-based optimizer to determine a more optimal configuration of the perception algorithm and computing resources which are indicated to satisfy certain defined performance requirements. In some embodiment, operations can: 1) adapt parameters controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm; and 2) control when adaptation of the parameters and/or a parameter optimization algorithm (or model) is to be performed. These operations can adapt the parameters to reduce differences between the performance requirements and the runtime performance metrics relating to the perception algorithm during further computation of the perception algorithm.

[0028] Potential advantages that may be provided various embodiments disclosed herein include that the parameters which are used to control computation by the perception algorithm can be dynamically or incrementally adapted to reduce differences between the performance requirements and the runtime performance metrics. Alternatively or additionally, the amount of computing resources of the mobile device allocated to execute the perception algorithm can be dynamically or incrementally controlled to reduce differences between the performance requirements and the runtime performance metrics. Runtime performance of the perception algorithm and/or the mobile device can thereby be more optimally tuned so the runtime performance metrics can more optimally satisfy the performance requirements despite a myriad of possible real-world situations, such as when various mobile device applications or several perception algorithms compete for computing resources, when the real-world environment is difficult to perceive using the at least one sensor, and/or when the environment rapidly changes.

[0029] Figure 1 illustrates a mobile device 200 that runs a perception algorithm 104 that can be controlled by a performance optimization algorithm 130 based on performance requirements and runtime performance metrics feedback, in accordance with some embodiments of the present disclosure. The mobile device 200 may include, without limitation, a smartphone, tablet computer, AR, VR, or XR glasses. Figure 3 illustrates a flowchart of operations performed by the performance optimization algorithm 130 of Figures 1 and 2 in accordance with some embodiments of the present disclosure.

[0030] The performance optimization algorithm 130 may be executed by the mobile device 200 of Figures 1 or 2 and/or may be a separate networked device, such as the server 230 of Figure 2 which communicates with the mobile device 200. The performance optimization algorithm 130 is therefore described in the context of being executed by an electronic device which may correspond to the mobile device 200 of Figures 1 and 2 and/or may correspond to the server 230 of Figure 2.

[0031] Referring initially to Figures 1 and 2, mobile device 200 executes a perception algorithm 104 (e.g., SLAM algorithm) using data from at least one sensor (e.g., 210 in Figure 2) to compute at least one of: pose of features of a real-world environment, a pose of the mobile device 200 relative to features of the real-world environment, and/or recognition of objects based on features of the real -world environment. The mobile device 200 may include a resource manager that controls an amount of the computing resources of the mobile device 200 which are allocated to execute the perception algorithm 104. The computing resources of the mobile device which can be used for executing the perception algorithm 104 can include, without limitation, central processing unit(s), graphics processing unit(s), volatile memory(ies), non-volatile memory(ies), application specific integrated circuits, wireless/wired interface to the sensor(s), etc.

[0032] As used herein, the term "pose" refers to the position and/or the orientation of the mobile device 200 relative to a defined coordinate system for the real-world environment. A pose may therefore be defined based on only the multidimensional position of the mobile device 200 relative to the defined coordinate system, based only on the multidimensional orientation of the mobile device 200 relative to the defined coordinate system, or based on a combination of the multidimensional position and the multidimensional orientation of the mobile device 200 relative to the defined coordinate system. The term "pose" therefore is used to refer to position, orientation, or combination thereof.

[0033] A performance optimization algorithm 130 obtains 300 performance requirements 110 for the perception algorithm 104 executed by the mobile device 200 to compute at least one of: pose of features of a real -world environment, a pose of the mobile device 200 relative to features of the real-world environment, and/or recognition of objects based on features of the real -world environment, using data from at least one sensor 210 (Figure 2). The performance optimization algorithm 130 obtains 302 runtime performance metrics 120 relating to the perception algorithm 104. The performance optimization algorithm 130 adapts 304 parameters controlling 134 computation by the perception algorithm 104 and/or controlling 136 an amount of computing resources of the mobile device 200 allocated by the resource manager 102 to execute the perception algorithm 104, to reduce differences between the performance requirements 110 and the runtime performance metrics 120 during further computation of the perception algorithm 104. The performance optimization algorithm 130 may be a multi -objective performance optimization algorithm. [0034] Operations for obtaining 302 runtime performance metrics relating to pose of the mobile device 200 may according to some embodiments include computing pose using two primary operational steps, a localization step and an odometry step. The first “localization step” may include matching the current features of the real-world environment captured by the at least one sensor 210 with features which are part of a map (e.g., image based map), to determine the pose of the mobile device 200 relative to a coordinate system of the map. The second “odometry step” may include computing translational movement and rotational movement (change in pose) of the mobile device 200 based on a relation between the current features of the environment captured by the at least one sensor 210 and the past features observed in the last N inputs (e.g., frames) from the at least one sensor 210. The outputs from the first “localization step” and the second “odometry step” can be fused together by transforming the odometry pose into the localization coordinate system (e.g., coordinate system of the map). In a region where the map is not available, only the odometry step is performed.

[0035] The term "features" used herein can refer to properties of the real-world environment captured by the at least one sensor 210. Examples of features include, without limitation, a 2D image from an image based sensor (e.g., camera), 3D point cloud from a structure based sensor (e.g., Lidar), part of a 2D image and/or 3D point cloud (e.g., set of pixels, region, object), etc. Features may be described by a “descriptor” which enables features to be efficiently compressed and saved, e.g., as part of a map.

[0036] As will be explained in further detail below, the performance optimization algorithm 130 may include a machine learning model 132 of the perception algorithm 104, which is used to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104, in order to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm 104.

[0037] As was explained above, the performance optimization algorithm 130 may be executed by the mobile device 200 and/or by another networked device, such as by a server. Figure 2 illustrates components of the mobile device 200 that runs the perception algorithm 104 which can be controlled by illustrated components of a server 230 running the performance optimization algorithm 130 based on performance requirements and runtime performance metrics feedback, in accordance with some embodiments of the present disclosure.

[0038] Referring to Figure 2, the mobile device 200 includes at least one sensor 210, which may be depth sensor(s) 212, camera(s) 214, inertial measurement unit(s) (IMU) 216, etc. The depth sensor(s) 212 actively senses distance to a real-world feature, such as by bouncing a laser (e.g., Lidar), RF signal (e.g., radar), sound (e.g., ultrasonic sensor), etc. on the real- world feature. The depth sensor(s) 212 generates depth data which includes a set of data points indicating locations in 3D space corresponding to features in the real-world environment which are sensed by the depth sensor(s) 212. The camera(s) 214 may be any type of image-based sensor, including without limitation a monocular camera, stereo cameras, etc. The IMU(s) 216 or other sensor (e.g., wheel odometer) may be configured to generate data indicating movement of the mobile device 200.

[0039] The mobile device 200 includes at least one processor 202 ("processor"), at least one memory 204 ("memory") storing program instructions executable by the processor 202, and at least one wireless transceiver 206 ("wireless transceiver") to communicate with a server 230, e.g., via a radio access network 220 and networks 222 (e.g., private networks and/or public networks such as the Internet). The illustrated memory 204 stores executable program instructions for the resource manager 102 in the perception algorithm 104.

[0040] The server 230 may be an edge server and, accordingly, may be positioned between the radio access network 220 and the networks 222, or may be positioned within the networks 222 proximately located to the radio access networks 220.

[0041] The server 230 includes at least one processor 240 ("processor"), at least one memory 250 ("memory") storing program instructions executable by the processor 240, and at least network interface 260 ("network interface") to communicate with the mobile device 200, e.g., via the radio access network 220 and networks 222. The illustrated memory 250 stores executable program instructions for the reception optimization algorithm 130 and may further store a structure-based map 252 and/or an image-based map 256. The optional structure-based map 252 may be generated using an depth-based localization algorithm which processes depth data from a structure-based sensor, such as the depth sensor 212 (e.g., Lidar, radar, etc.). The optional image-based map 256 may be generated using an image-based localization algorithm which processes image data from an image sensor, such as the camera(s) 214. The memory 250 may store program instructions for the perception algorithm 104 which is executed by the processor 240 to provide output therefrom to the mobile device 200 for display to a user (e.g., via XR glasses), providing guidance to the user and/or a guidance program controlling speed, heading, and/or orientation of the mobile device 200, etc.

[0042] The server 230 is operative to receive the performance requirements 110 and the runtime performance metrics 120 from the mobile device 200, and the at least one processor 240 is operative to communicate a message through the network 222 connection to the mobile device 200 indicating the adapted parameters to be used to control computations of the perception algorithm 104 by the mobile device 200 and/or to be used to control an amount of computing resource of the mobile device 200 allocated by the resource manager 102 to execute the perception algorithm 104. When indicating the adapted parameters, the server 230 may indicate the actual values of the parameters that are to be used or may indicate an amount of change that should be made to the actual values of the parameters that are to be used.

[0043] In contrast, when the mobile device 200 executes the performance optimization algorithm 130, e.g., processor 202 executing instructions from memory 204, the mobile device 200 is operative to control computations of the perception algorithm 104 using the adapted parameters and/or to control an amount of computing resource of the mobile device 200 allocated by the resource manager 102 to execute the perception algorithm 104 based on the adapted parameters.

[0044] The rate or time instance of adaptation of the parameters may be controlled based on various defined sensed characteristics of the environment. For example, in some embodiments the processor 202 or 240 is further operative to repeat the adaptation of the parameters, using updated runtime performance metrics that are obtained, at a time interval or time instance that is defined based on at least one of the following:

1. amount of change in ambient brightness of the real-world environment detectable by the at least one sensor between instances of the time interval or time instance (e.g., measured brightness relative to one or more thresholds or previous measurements);

2. amount of change in features of the real-world environment which are detectable by the sensor 210 between instances of the time interval or time instance;

3. amount of change in computing resources of the mobile device 200 available to be allocated to execute the perception algorithm 104 between instances of the time interval or time instance; and

4. amount of change in pose, e.g., linear speed and/or angular speed, of the mobile device 200 between instances of the time interval or time instance.

[0045] The parameters may be adapted using the performance optimization algorithm 130 shown in Figures 1 and 2. In some embodiments, the operation to adapt parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104, includes processing the performance requirements and the runtime performance metrics through the performance optimization algorithm 130 to generate the adapted parameters which reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm 104. The performance optimization algorithm 130 is configured to approximate relationships between: 1) using particular values for the parameters to control computation by the perception algorithm 104 and/or control the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104; and 2) the runtime performance metrics which result from computation of the perception algorithm 104 when using the particular values for the parameters.

[0046] The relationships approximated by the performance optimization algorithm 130 may be initially defined through experiments before deployment to mobile devices, and then adapted after deployment based on runtime performance metrics generated during operation of the perception algorithm in various real-world environments.

[0047] In one embodiment, the performance optimization algorithm 130 includes a Pareto optimization algorithm which contains Pareto curves approximating the relationships, and the parameters are adapted based on selecting values among the Pareto curves using the performance requirements as value selection pointers. The Pareto curves may be defined during a setup process and subsequently updated based on feedback from runtime performance metrics to better approximate Pareto configuration under various runtime conditions.

[0048] As explained above and further below, the performance optimization algorithm 130 may include a machine learning model 132 that relates inputs to the perception algorithm 104 to outputs of the perception algorithm 104. The perception algorithm 104 is used to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated by the resource manager 102 to execute the perception algorithm 104, in order to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm 104.

[0049] In some embodiments, the machine learning model 132 is adapted based on feedback of new runtime performance metrics, such as by adapting the machine learning model 132 to reduce differences between the performance requirements and runtime performance metrics relating to the at least one of:

1. the poses of features of the real-world environment;

2. the pose of the mobile device (200) relative to features of the real-world environment; and/or

3. the recognition of objects based on features of the real -world environmentcomputed by the perception algorithm (104) using earlier adapted parameters, to control computation by the perception algorithm 104 and/or control the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104.

[0050] Some further related embodiments are directed to controlling what triggers adaptation of the machine learning model 132. Operations to adapt the machine learning model 132 may be triggered by at least one of the following:

1. determining that one of the runtime performance metrics does not satisfy one of the performance requirements;

2. determining that a plurality of the runtime performance metrics do not satisfy a plurality of prioritized ones of the performance requirements;

3. determining that a maximum rate of adaptation of the machine learning algorithm will not be exceeded;

4. determining that a minimum rate of adaptation of the machine learning algorithm will be exceeded if adaptation of the machine learning algorithm is not presently triggered; and

5. determining that at least a threshold change occurs between at least one of the runtime performance metrics that are presently obtained and at least one of the runtime performance metrics that were most recently previously obtained.

[0051] Some further related embodiments are directed to various types of controlling actions, performance requirements 110, runtime performance metrics 120, and the parameters.

[0052] The resource manager 102 can be configured to respond to use the parameters to control at least one of the following:

1. a maximum number of features of the real-world environmentidentified in images from a camera of the at least one sensor 210 during further execution of the perception algorithm 104;

2. a priority of execution provided to the perception algorithm 104 by an operating system managing multi-tasking access to computational resources of the mobile device 200;

3. a number of central processing unit (CPU) cores in the mobile device 200 that are allocated to execute the perception algorithm 104 during further computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 using the perception algorithm 104;

4. a number of graphics processing unit (GPU) cores in the mobile device 200 that are allocated to execute the perception algorithm 104 during further computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 using the perception algorithm 104;

5. a clock rate of a CPU core in the mobile device 200 when processing the perception algorithm 104 during further computation of the poses of features of the real-world environmentand/or the mobile device 200 using the perception algorithm 104;

6. a clock rate of a GPU core in the mobile device 200 when processing the perception algorithm 104 during further computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 using the perception algorithm 104;

7. a power mode of computational resources of the mobile device 200 during further computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 using the perception algorithm 104; and

8. a speed of the mobile device 200 moving through the real-world environment during further computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 using the perception algorithm 104.

[0053] The performance requirements 110 may characterize at least one of the following:

1. a maximum, minimum, and/or acceptable range of localization error when computing localization of the mobile device 200 relative to features of the real-world environmentand/or relative to objects recognized based on features of the real -world environment by the perception algorithm 104 using the data from the at least one sensor 210,

2. a maximum, minimum, and/or acceptable range of feature detection error when computing the poses of features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210,

3. a maximum, minimum, and/or acceptable range of 3D reconstruction error when computing a 3D representation of features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210;

4. a maximum, minimum, and/or acceptable range of latency when computing localization of the mobile device 200 relative to features of the real-world environmentand/or relative to objects recognized based on features of the real -world environment by the perception algorithm 104 using the data from the at least one sensor 210, 5. a maximum, minimum, and/or acceptable range of latency when computing the poses of features of the real-world environmentand/or pose of the mobile device 200 by the perception algorithm 104 using the data from the at least one sensor 210,

6. a maximum, minimum, and/or acceptable range of latency when computing the 3D representation of features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210,

7. a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device 200 which can be used for localization of the mobile device 200 relative to features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210,

8. a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device 200 which can be used for computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 by the perception algorithm 104 using the data from the at least one sensor 210, and

9. a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device 200 which can be used for computing the 3D representation of features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210.

[0054] The runtime performance metrics 120 may be generated (e.g., by the perception algorithm 104, the performance optimization algorithm 130, and/or another component) to characterize at least one of the following:

1. a localization error in computed localization of the mobile device 200 relative to features of the real -world environmentby the perception algorithm 104 using the data from the at least one sensor 210,

2. a feature detection error in the pose of features by the perception algorithm 104 computed by the perception algorithm 104 using the data from the at least one sensor 210,

3. a 3D reconstruction error in a 3D representation of features of the real-world environmentcomputed by the perception algorithm 104 using the data from the at least one sensor 210,

4. an object recognition error, 5. a latency in computing localization of the mobile device 200 relative to features of the real -world environmentand/or relative to objects recognized based on features of the real -world environment by the perception algorithm 104 using the data from the at least one sensor 210,

6. a latency in computing the poses of features of the real-world environmentand/or the mobile device 200 by the perception algorithm 104 using the data from the at least one sensor 210,

7. a latency in computing the 3D representation of features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210,

8. a power consumption of computational resources of the mobile device 200 used for localization of the mobile device 200 relative to features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210,

9. a power consumption of computational resources of the mobile device 200 used for computation of the poses of features of the real-world environmentand/or pose of the mobile device 200 by the perception algorithm 104 using the data from the at least one sensor 210,

10. a power power consumption of computational resources of the mobile device 200 used for recognizing objects based on features of the real -world environmentusing the data from the at least one sensor 210,

11. a power consumption of computational resources of the mobile device 200 used for computing the 3D representation of features of the real-world environmentby the perception algorithm 104 using the data from the at least one sensor 210.

[0055] Examples of possible non-controllable inputs (e.g., runtime performance metrics 120) include utilization of the computing resources such as CPU loading and/or GPU loading, and include measured environmental properties such a how dynamic or non-static objects are sensed in the environment, e.g., object speed and/or level of erratic movement. Examples of possible controllable inputs (e.g., runtime performance metrics 120) include application-level parameters such as the localization cycle frequency or the number of features of the real- world environmentto be identified in a frame, platform level parameters such as the CPU and GPU frequency, the number of processor cores to be used, etc. Another example of a controllable inputs (e.g., runtime performance metrics 120) includes multi-tasking priority provided by an operating system to various applications one of which is the perception algorithm 104. In case of a robot or other motorized device, speed can be a controllable metric.

[0056] The performance optimization algorithm 130 and, more particularly, the machine learning (ML) model 132 can be adapted to relate various non-controllable and controllable runtime performance metrics to the output performance of the perception algorithm 104. For example, the ML model 132 may describe the relation between the output performance and the non-controllable and controllable inputs, such as a function f where output of the perception algorithm 104 equals f(inputs), which in some embodiments is a set [RMSE, Power] = f (application parameters, platform parameters, speed, CPU load, GPU load, brightness), where "RMSE" may be absolute pose error. The function f can also be split between f_power, f_APE, f mAp (mean average precision of object detector algorithm) which would encode the relationship for each output performance separately, and the function f may be an example of the relation between inputs to the performance metrics of, e.g., localization, object detection, and power.

[0057] The ML model 132 may be trained using experiment traces with input and output data which can be obtained from the same mobile device or another mobile device (e.g., collaborative learning, federated learning, etc.) with the same or similar operational properties. For example, the ML model 132 may be trained with different controllable inputs (application and platform parameters which are adaptable) and different non-controllable inputs (mobile device speed, feature brightness sensed by sensor(s), computing resource(s) loading, etc.), for which the output performance of the perception algorithm 104 is measured (e.g., localization error, power usage, etc.).

[0058] In some embodiments, the performance optimization algorithm 130 computes the desired controllable inputs (to adapt the parameters) in order to, for example, reduce or minimize runtime localization error while satisfying a power consumption requirement, or to reduce or minimize runtime power consumption while satisfying a localization error requirement, based on measurements of non-controllable inputs, the ML model 132, and the level of uncertainty in the ML model 132. As described below, the runtime error may be a measured true error or a quantity that approximates the error, e.g., proxy for error.

[0059] As explained above, the ML model 132 or more generally the performance optimization algorithm 130, may include a Pareto optimization algorithm which contains Pareto curves approximating the relationships between: 1) using particular values for the parameters to control computation by the perception algorithm 104 and/or control the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104; and 2) the runtime performance metrics which result from computation of the perception algorithm 104 when using the particular values for the parameters. The Pareto curve can be defined to provide an optimal trade-off between two metrics, where in order to improve one metric, the other metric will have to be negatively impacted, such illustrated by the Pareto confirmations along the curve shown in Figure 4.

[0060] Figure 5 illustrates a zoomed-in close-up view of a portion of the graph of Figure 4 and shows a performance requirement point 500 on the Pareto curve 400 which is offset from a measured runtime performance metric 506 of a SLAM algorithm, and which offset is used to adapt parameters controlling computation by the SLAM algorithm and/or controlling an amount of computing resources of the mobile device 200 allocated to execute the SLAM, in accordance with some embodiments of the present disclosure.

[0061] Referring to Figure 5, the offset is illustrated by a Delta E (error difference) and Delta P (power difference) between the performance requirement point 500 and the measured runtime performance metric 506.

[0062] In accordance with the operations of Figure 3, the performance requirements are obtained for the perception algorithm 104, such max error Emax and/or max power Pmax. The performance requirements can be used to identify a Pareto configuration curve, such the curve 400 in Figure 5. During runtime operation of the perception algorithm 104, the operations obtain runtime performance metrics relating to the perception algorithm 104, such as measured error and measured power, and the differences between the required and measured metrics are determined (e.g., Delta E and Delta P). The differences Delta E and Delta P are used to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling an amount of computing resources of the mobile device 200 allocated by the resource manager 102 to execute the perception algorithm 104, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm 104.

[0063] In the example scenario of Figure 5, the Pareto curve for pose error E and power P defines a relationship between the runtime measurements of minimum error E or power P that can be achieved for a required Pmax or Emax. As an example, the Pareto curve point 500 corresponds to when the perception algorithm 104 is allowed a particular maximum power consumption Pmax of =, which then sets a corresponding lowest error that is achievable as Emin according to the Pareto curve 400.

[0064] From an implementation point of view in a device-edge-cloud system, the Pareto curve can be computed by an edge-cloud platform (i.e., server 230) from runtime performance metrics collected from several mobile devices with the same or similar characteristics (in a collaborative manner), which the resulting adapted parameters being transmitted to the mobile device 200. Alternatively, as explained above, the mobile device can be operationally configured to compute the pareto curve based on its own runtime performance metrics.

[0065] The parameters are adapted to provide a desired runtime performance. For example, when the performance requirement is E < Emax, which is the maximum allowed error, then the parameters are adapted to achieve Emax with the lowest power P (e.g., the point 500 in Figure 5) since these configured parameters are expected to achieve the desired requirement of consuming the lowest power.

[0066] In contrast, when the performance requirement is P < Pmax, which is the maximum allowed power, then the parameters are adapted to achieve Pmax with the lowest error E (e.g., the point 500 in Figure 5) since these configured parameters are expected to achieve the desired requirement of the lowest error possible.

[0067] During runtime operation of the perception algorithm 104, the pose error and power are determined as runtime performance metrics. The error can be measured directly if a ground truth signal exists (e.g. the pose at a certain location was known by relying on a specific known position anchor by using visual markers at known locations) while a proxy of the error can also be estimated by the estimated pose itself which is referred in the art here as the “pose distance” where the error is taken as the difference in poses between two time steps). Other ways of estimating or computing proxies may be used, such as based on a combination of pose, speed, change in pixels between consecutive images, etc. The power can be measured directly by the mobile device. If the perception algorithm 104 runs on a dedicated processing unit or a dedicated core, the power consumption can be more easily measured or determined, e.g., based on manufacturer specification.

[0068] The error and power values can be relative, for example, to the average from the current time t and the last T seconds as [t - T, t], where T is a design parameter. Advantageously, when the pose error can be estimated with higher accuracy over a time period [t_i, t_f], the measurement of the error and power may then performed over such time period. This may be typical in a SLAM algorithm, where the estimated pose drifts over time and whenever a loop closure occurs the current and past poses (trajectory) are re-optimized and so a more accurate estimation of the pose is available. It would therefore be advantageous to analyze the pose error over intervals where the pose estimation quality is the best (e.g. when loop closures occurred within t_i and t_f). Moreover, it may also be that a ground truth pose estimate becomes available for at least a region which was covered by the device during the interval [t_i, t_f] as for example another mobile device with a positioning system which is no worse than the one used by the current mobile device reports its pose and the pose error E can be computed against such signal.

[0069] The time T would define the time period in which measurements should be used to compute the perception metric and power consumption to then take decisions on reconfiguration (adapting the parameters) and re-modelling (adapting the performance optimization algorithm, such as the ML model 132). This time should not be too short, otherwise it will capture only transient behavior, but it can also not be too large otherwise it will not capture any transient behavior, so it acts like a filter in the system.

[0070] The time T may be set by heuristics, depending on how often it is reasonable to run the reconfiguration. For example, one could set T to be a fixed time of 1 minute or 10 minutes.

[0071] The time T may be defined by occurrence of various defined events which relate to the operation of the mobile device. For example, time T may be related to a specific distance travelled by the mobile device. For example, the runtime performance metrics may be evaluated when the mobile device has travelled 10 meters, so that the time T would actually depend on the speed of the mobile device.

[0072] After the value of the error and power runtime performance metrics are measured, the difference between the current values and the Pareto values are calculated to obtain Delta E and Delta P as illustrated in the Figure 5.

[0073] Depending on the values of Delta E and Delta P and depending on the performance requirements, different actions are taken regarding the adaption of parameters and/or remodelling (adapting the ML model 132 used by the performance optimization algorithm). The values of Delta E and Delta P indicate accuracy of the model of E and P respectively with respect to the performance requirements, which will have an impact on the way the reconfiguration should be done in order to achieve the performance requirements. Example actions are now described for several different scenarios.

[0074] In a first scenario, the computed Delta E > X and Delta P > Y. This may be a critical situation where both the differences are larger than the performance requirements and which therefore neither is satisfied. This situation requires aggressive action to reduce the delta differences for both error and power and have the system perform closer to the expected Pareto configuring. Since both deltas exceed the performance requirements, this means that the performance optimization algorithm 130 describing the relation between a configuration and E and P is inaccurate. Consequently, the relationship of the performance optimization algorithm 130 should not be relied upon to adapt the parameters. The responsive action is therefore to take the most conservative action possible which would have the expectation to achieve the performance requirements without considering the other runtime performance metrics since the relationship shouldn't be relied upon.

[0075] When the performance requirement is E < Emax while without trying to reduce or minimize P, the operations taking the most conservative action are performed by computing the controllable input values (ci) and non-controllable input values (nci) which minimize the error E, i.e. argmin E (ci, net) without trying to minimize P. In this way, the parameters are ci adapted to allow the perception algorithm 104 to use as much power as needed to have the error satisfy the performance requirement.

[0076] In contrast, when the performance requirement is P < Pmax while trying to minimize E, the same solution as in the previous case can be applied, where the operations taking the most conservative action are performed by computing ci which minimizes the power P, i.e., argmin P(ci, net) without trying to minimize E. ci

[0077] Corresponding operations for this first scenario performed in the context of Figure 3 are now explained. One of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error. Another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device 200 used for execution of the perception algorithm 104. The at least one processor 202 or 240 is further operative to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104, by operations as follows. Responsive thereto, the operations adapt the parameters to minimize differences of one of: 1) the maximum error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error without seeking to reduce the runtime power consumption, during further computation of the perception algorithm 104; and 2) the runtime power consumption and the maximum power consumption without seeking to reduce the runtime localization error, during further computation of the perception algorithm 104.

[0078] In a second scenario, the computed Delta E > X and Delta P is within a power requirement range [-Z, Y], This may be a critical situation only for E, while P is within the power requirement range. This situation indicates that the performance optimization algorithm 130 describing the relation is accurate for P and inaccurate for E, the relation can be trusted for the P model to control adapting of the parameters while the E model should not be used.

[0079] When the performance requirement is E < Emax while trying to reduce or minimize P, because there is acceptable accuracy in the model for P, the parameter configuration to be selected (or adapted to conform to) is the one that decreases the error with a low required increase in power, which means computing the controllable input values (ci) which minimizes the error E i.e. argmin E(ci, net) such that Delta P < Ymax, where Ymax ci would be a value of power increase (penalty) which would satisfy the performance requirements. The operations may select configurations which would achieve the lowest error possible (which would be lower than Emax), but only with an acceptable power consumption penalty of maximum Ymax Watts. This can be done since the model is sufficiently accurate relating power consumption to the chosen configuration ci.

[0080] In contrast, when the performance requirement is P < Pmax while trying to reduce or minimize E, the operations select the parameter configuration which is expected to decrease the error E while keeping Delta P < Y, i.e. argmin E(ci, net) such that Delta P < Y, so the ci operations only use any available margin (Y-Delta_P) to try to reduce Delta E. This is done since the P model is sufficiently accurate.

[0081] Corresponding operations for this first scenario performed in the context of Figure 3 are now explained. One of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error. Another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device 200 used for execution of the perception algorithm 104. The at least one processor 202 or 240 is further operative to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104, by operations as follows: Responsive thereto, the operations adapt the parameters to minimize differences between the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error while allowing increase of the runtime power consumption up to a maximum of the acceptable range of power consumption, during further computation of the perception algorithm 104. [0082] In a third scenario, the computed Delta E > X and Delta P < -Z. This may be a critical situation only for E, and there is margin in terms of power. This implies that both models (E and P) are inaccurate, but the P model is not critical since the mobile device 200 is using less power than expected. Because the mobile device 200 is using less power than expected, when adapting the parameters to achieve the performance requirements in terms of power the parameters can be adapted to use the power margin to improve the error.

[0083] When the performance requirement is E < Emax while trying to minimize P, the same operation actions may be performed as in the first scenario, in view of the the inaccuracy of the model for P so the action to adapt the parameters to reduce the error E would be the most conservative as in the first scenario where Delta E > X and Delta P > Y.

[0084] In contrast, when the performance requirement is P < Pmax while trying to minimize E, the parameters can be adapted to to achieve a lower Delta E by using the power margin that exists with Delta P < -Z. Because the model of P is inaccurate, an option is to iteratively adapt the parameters in order to satisfy the performance requirement. For example, a new configuration can be obtained by solving argmin E ci, nci) such that P < Pbar, where ci

Pbar > Pmax, the parameters are reconfigured and then a new observation of the new Delta P and Delta E values can be obtained. These operations can be performed iteratively until it is observed that P is closer to the performance requirement Pmax or exceeds it by no more than Y.

[0085] Corresponding operations for this third scenario performed in the context of Figure 3 are now explained. One of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error. Another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device 200 used for execution of the perception algorithm 104. The at least one processor 202 or 240 is further operative to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104, by operations as follows. Responsive thereto, the operations adapt the parameters to minimize differences between the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error while allowing increase of the runtime power consumption up to the maximum power consumption, during further computation of the perception algorithm 104. [0086] In a fourth scenario, the computed Delta E < -W or Delta E in [-W, X], and correspondingly for Delta P in [-Z, Y] or Delta P < -Z. This means that there is margin in both deltas and it is a non-critical situation. This implies that both E and P models are incorrect, but the E and P models are not critical because there is acceptable error and acceptable power consumption satisfying the performance requirements. Because running the perception algorithm 104 uses less power than expected by the model, when trying to adapt the parameters to satisfy the performance requirement for power, the operations can advantageously use the power margin to improve the error. Similarly, since there is a smaller error than expected, when trying to adapt the parameters to satisfy the performance requirement for error, the operations can advantageously use the the error margin to improve the power.

[0087] When the performance requirement is E < Emax while trying to minimize P, the operations apply a parameter configuration which is expected to increase the error up to E < Emax while keeping P to the minimum. As described in the previous scenario, this can be performed iteratively by solving argmin P(ci, net) such that E < Ebar, where Ebar > Emax, ci the parameters are reconfigured and a new observation of the new Delta E and Delta P values is obtained. These operations can be performed iteratively until it is observed that E is closer to the performance requirement for Emax or exceeds it by no more than X.

[0088] In contrast, when the performance requirement is P < Pmax while trying to minimize E, the operations apply a parameter configuration which is expected to increase the power up to P < Pmax while keeping E to the minimum. As described in the previous scenario, this can be performed iteratively by solving argmin E(ci, nci) such that P < Pbar, ci where Pbar > Pmax, the system is reconfigured and a new observation of the new Delta E and Delta P values is obtained. These operations can be performed iteratively until it is observed that P is closer to the desired Pmax or exceeds it by no more than Y.

[0089] Corresponding operations for this fourth scenario performed in the context of Figure 3 are now explained. One of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error. Another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device 200 used for execution of the perception algorithm 104. The at least one processor 202 or 240 is further operative to adapt the parameters controlling computation by the perception algorithm 104 and/or controlling the amount of computing resources of the mobile device 200 allocated to execute the perception algorithm 104, by by operations as follows. Responsive thereto, the operations adapt the parameters to minimize differences of one of: 1) the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error without the runtime power consumption exceeding the maximum power consumption, during further computation of the perception algorithm 104; and 2) the runtime power consumption and the maximum power consumption without the runtime localization error exceeding the maximum localization error, during further computation of the perception algorithm 104.

[0090] Similar operations can be used for inverse scenarios, i.e., switching E for P and vice- versa, for the problems where Delta E is in a range [-W, X] and Delta E is < -W when Delta P > X.

[0091] The thresholds X, Y, Z and W are adaptable parameter variables which can be selected experimentally. For example, the values thereof can be defined as a percentage of the performance requirements maximum error Emax or power Pmax, for example X = 1.1 *Emax which means that the error constraint can be violated by up to 10%. The performance requirements may be time varying, where when few data is available to train the system model and the Pareto curve, the performance requirements are larger, where when the system runs for a longer time and more data is available, the performance requirements become smaller. The reason is that the better the model, the more aggressive with the requirement to be closer to the performance requirements the operations can more robustly achieve. Besides time, another heuristic to set the performance requirements is to compute the 2 or 3-sigma bounds of the values of Delta E and Delta P over an interval, and set the value of the threshold to a value as (or close to the) “2 or 3-sigma bounds” since the 2 or 3-sigma bound would mean that with high probability the values of Delta E and Delta P would be most likely under this bound at all times if the models are performing as expected.

[0092] In another embodiment, the thresholds for Delta E and Delta P are defined to a higher or lower value depending if the metric E or P is the variable that is defined as a performance requirement. The objective is that the user and/or performance optimization algorithm 130 impose a higher performance requirement if a metric is set as a constraint rather than just a minimization metric. As an example, if it is required that E < Emax, then performance requirement X for Delta E < X can be set to a value XI, while if it is required that P < Pmax, than performance requirement X for Delta E < X can be set to a value X2, where XI < X2. [0093] In some further embodiments, model updates are triggered depending on the delta values computed based on differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm 104.

[0094] In one embodiment, model updates are performed when there is a violation of a performance requirement, e.g., Delta E > X, >Delta_P > Y, or a violation of a performance requirement, e.g., Delta P < -Z, Delta E < -W.

[0095] In another embodiment, the model updates are performed with different priorities, where a positive violation (Delta E > X, Delta P > Y) has higher priority as it is more critical than a negative violation (Delta P < -Z, Delta E < -W). The reason is that a negative violation means that there is a margin with respect to power or error in the runtime performance, so the perception algorithm 104 is performing better than expected, and so it is less critical to perform a model update in this case.

[0096] If a certain configuration triggers model updates more often than a desired limit N, this means that this configuration is not robust and more data may be needed before this configuration can be used when the application has desired performance requirements. In this case a solution is for the operations to use such configuration only when the perception algorithm 104 has no strict performance requirements in order to gather more data to train the system model, while the Delta E and Delta P conditions could still be evaluated in order to understand if the configuration triggers model updates less often than the desired limit N and can then be used when the perception algorithm 104 satisfies performance requirements. [0097] In another embodiment, the data used for model updates is only the data captured during the interval [t-T,t] where it is detected that a performance requirement violation in terms of Delta P or Delta E occurred in order to reduce the amount of data to be used for model update. This means that if there is no violation triggering a model update at time t, the data captured between time [t-T,t] will not be used to retrain the model since the model was able to predict the performance of the perception algorithm 104 correctly during that interval and so it is unnecessary to use this data when performing a model update.

[0098] As explained above, the model update can performed at the server 230. The mobile device 200 may transmit to the server 230 all the input and output data that should be used for model training. The server 230 re-trains the model and transmits back the model to the mobile device 200.

[0099] Further definitions and embodiments are now explained below.

[00100] In the above description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein. [00101] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and/or clarity. The term "and/or" includes any and all combinations of one or more of the associated listed items.

[00102] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements/operations, these elements/operations should not be limited by these terms. These terms are only used to distinguish one element/operation from another element/operation. Thus, a first element/operation in some embodiments could be termed a second element/operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[00103] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id Est," may be used to specify a particular item from a more general recitation.

[00104] Example embodiments are described herein with reference to block diagrams and/or flowchart illustrations of computer-implemented methods, apparatus (systems and/or devices) and/or computer program products. It is understood that a block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and/or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions/acts specified in the block diagrams and/or flowchart block or blocks, and thereby create means (functionality) and/or structure for implementing the functions/acts specified in the block diagrams and/or flowchart block(s).

[00105] These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions/acts specified in the block diagrams and/or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and/or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[00106] It should also be noted that in some alternate implementations, the functions/acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Moreover, the functionality of a given block of the flowcharts and/or block diagrams may be separated into multiple blocks and/or the functionality of two or more blocks of the flowcharts and/or block diagrams may be at least partially integrated. Finally, other blocks may be added/inserted between the blocks that are illustrated, and/or blocks/operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[00107] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. An electronic device (200, 230) comprising at least one processor (202, 240) operative to: obtain performance requirements for a perception algorithm (104) executed by a mobile device (200) to compute at least one of: poses of features of a real -world environment, a pose of the mobile device (200) relative to features of the real- world environment, and/or recognition of objects based on features of the real- world environment, using data from at least one sensor (210); obtain runtime performance metrics relating to the perception algorithm (104); and adapt parameters controlling computation by the perception algorithm (104) and/or controlling an amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm (104).

2. The electronic device (200, 230) of Claim 1, wherein: the electronic device (200, 230) comprises a server (230) operative to receive the performance requirements and the runtime performance metrics from the mobile device (200) through a network (220, 222) connection; and the at least one processor (202, 240) is operative to communicate a message through the network (220, 222) connection to the mobile device (200) indicating the adapted parameters to be used to control computations of the perception algorithm (104) by the mobile device (200) and/or to be used to control an amount of computing resource of the mobile device (200) allocated to execute the perception algorithm (104).

3. The electronic device (200, 230) of Claim 1, wherein: the electronic device (200, 230) comprises the mobile device (200); and the at least one processor (202, 240) is operative to control computations of the perception algorithm (104) using the adapted parameters and/or to control an amount of computing resource of the mobile device (200) allocated to execute the perception algorithm (104) based on the adapted parameters.

4. The electronic device (200, 230) of any of Claims 1 to 3, where the at least one processor (202, 240) is further operative to repeat the adaptation of the parameters, using updated runtime performance metrics that are obtained, at a time interval or time instance that is defined based on at least one of the following: amount of change in ambient brightness of the real-world environment detectable by the at least one sensor between instances of the time interval or time instance; amount of change in features of the real-world environment which are detectable by the at least one sensor (210) between instances of the time interval or time instance; amount of change in computing resources of the mobile device (200) available to be allocated to execute the perception algorithm (104) between instances of the time interval or time instance; and amount of change in speed of the mobile device (200) between instances of the time interval or time instance.

5. The electronic device (200, 230) of any of Claims 1 to 4, wherein the operation to adapt parameters controlling computation by the perception algorithm (104) and/or controlling the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm (104), comprises: processing the performance requirements and the runtime performance metrics through a performance optimization algorithm (130) to generate the adapted parameters which reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm (104), wherein the performance optimization algorithm (130) approximates relationships between: 1) using particular values for the parameters to control computation by the perception algorithm (104) and/or control the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104); and 2) the runtime performance metrics which result from computation of the perception algorithm (104) when using the particular values for the parameters.

6. The electronic device (200, 230) of Claim 5, wherein the performance optimization algorithm (130) comprises a Pareto optimization algorithm which contains Pareto curves approximating the relationships, and the parameters are adapted based on selecting values among the Pareto curves using the performance requirements as value selection pointers.

7. The electronic device (200, 230) of any of Claims 5 to 6, wherein the performance optimization algorithm (130) comprises a machine learning model (132) that is adapted based on differences between the performance requirements and runtime performance metrics relating to the at least one of: the poses of features of the real-world environment, the pose of the mobile device (200) relative to features of the real-world environment, and/or the recognition of objects based on features of the real -world environment computed by the perception algorithm (104) using earlier adapted parameters to control computation by the perception algorithm (104) and/or control the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104).

8. The electronic device (200, 230) of Claim 7, wherein the adaptation of the machine learning model (132) is triggered by at least one of the following: determining that one of the runtime performance metrics does not satisfy one of the performance requirements; determining that a plurality of the runtime performance metrics do not satisfy a plurality of prioritized ones of the performance requirements; determining that a maximum rate of adaptation of the machine learning algorithm will not be exceeded; determining that a minimum rate of adaptation of the machine learning algorithm will be exceeded if adaptation of the machine learning algorithm is not presently triggered; and determining that at least a threshold change occurs between at least one of the runtime performance metrics that are presently obtained and at least one of the runtime performance metrics that were most recently previously obtained.

9. The electronic device (200, 230) of any of Claims 1 to 8, wherein: the parameters controlling computation by the perception algorithm (104) and/or controlling an amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), control at least one of the following: a maximum number of features of the real-world environment identified in images from a camera of the at least one sensor (210) during further execution of the perception algorithm (104); a priority of execution provided to the perception algorithm (104) by an operating system managing multi-tasking access to computational resources of the mobile device (200); a number of central processing unit, CPU, cores in the mobile device (200) that are allocated to execute the perception algorithm (104) during further computation of the poses of features of the real-world environment and/or pose of the mobile device (200) using the perception algorithm (104); a number of graphics processing unit, GPU, cores in the mobile device (200) that are allocated to execute the perception algorithm (104) during further computation of the poses of features of the real-world environment and/or pose of the mobile device (200) using the perception algorithm (104); a clock rate of a CPU core in the mobile device (200) when processing the perception algorithm (104) during further computation of the poses of features of the real- world environment and/or pose of the mobile device (200) using the perception algorithm (104); a clock rate of a GPU core in the mobile device (200) when processing the perception algorithm (104) during further computation of the poses of features of the real- world environment and/or pose of the mobile device (200) using the perception algorithm (104); a power mode of computational resources of the mobile device (200) during further computation of the poses of features of the real-world environment and/or pose of the mobile device (200) using the perception algorithm (104); and a speed of the mobile device (200) moving through the real-world environment during further computation of the poses of features of the real-world environment and/or pose of the mobile device (200) using the perception algorithm (104).

10. The electronic device (200, 230) of any of Claims 1 to 9, wherein: the performance requirements characterize at least one of the following: a maximum, minimum, and/or acceptable range of localization error when computing localization of the mobile device (200) relative to features of the real -world environmentand/or relative to objects recognized based on features of the real -world environment by the perception algorithm (104) using the data from the at least one sensor (210), a maximum, minimum, and/or acceptable range of feature detection error when computing the poses of features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210), a maximum, minimum, and/or acceptable range of 3D reconstruction error when computing a 3D representation of features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210); a maximum, minimum, and/or acceptable range of latency when computing localization of the mobile device (200) relative to features of the real-world environmentand/or relative to objects recognized based on features of the real- world environment by the perception algorithm (104) using the data from the at least one sensor (210), a maximum, minimum, and/or acceptable range of latency when computing the poses of features of the real-world environment and/or pose of the mobile device (200) by the perception algorithm (104) using the data from the at least one sensor (210), a maximum, minimum, and/or acceptable range of latency when computing the 3D representation of features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210), a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device (200) which can be used for localization of the mobile device (200) relative to features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210), a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device (200) which can be used for computation of the poses of features of the real-world environment and/or pose of the mobile device (200) by the perception algorithm (104) using the data from the at least one sensor (210), and a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device (200) which can be used for computing the 3D representation of features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210).

11. The electronic device (200, 230) of any of Claims 1 to 10, wherein: the runtime performance metrics characterize at least one of the following: a localization error in computed localization of the mobile device (200) relative to features of the real -world environment by the perception algorithm (104) using the data from the at least one sensor (210), a feature detection error in the pose of features by the perception algorithm (104) computed by the perception algorithm (104) using the data from the at least one sensor (210), a 3D reconstruction error in a 3D representation of features of the real-world environment computed by the perception algorithm (104) using the data from the at least one sensor (210), an object recognition error, a latency in computing localization of the mobile device (200) relative to features of the real-world environmentand/or relative to objects recognized based on features of the real -world environment by the perception algorithm (104) using the data from the at least one sensor (210), a latency in computing the poses of features of the real-world environment and/or pose of the mobile device (200) by the perception algorithm (104) using the data from the at least one sensor (210), a latency in computing the 3D representation of features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210), a power consumption of computational resources of the mobile device (200) used for localization of the mobile device (200) relative to features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210), a power consumption of computational resources of the mobile device (200) used for computation of the poses of features of the real-world environment and/or the mobile mobile device (200) by the perception algorithm (104) using the data from the at least one sensor (210), a power power consumption of computational resources of the mobile device (200) used for recognizing objects based on features of the real -world environment using the data from the at least one sensor (210), a power consumption of computational resources of the mobile device (200) used for computing the 3D representation of features of the real-world environment by the perception algorithm (104) using the data from the at least one sensor (210). The electronic device (200, 230) of any of Claims 1 to 11, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device (200) used for execution of the perception algorithm (104); and the at least one processor (202, 240) is further operative to adapt the parameters controlling computation by the perception algorithm (104) and/or controlling the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, the object recognition error, and the 3D reconstruction error exceeding a maximum error characterized by one of the performance requirements and the runtime power consumption exceeding a maximum power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences of one of: 1) the maximum error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error without seeking to reduce the runtime power consumption, during further computation of the perception algorithm (104); and 2) the runtime power consumption and the maximum power consumption without seeking to reduce the runtime localization error, during further computation of the perception algorithm (104). The electronic device (200, 230) of any of Claims 1 to 12, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device (200) used for execution of the perception algorithm (104); and the at least one processor (202, 240) is further operative to adapt the parameters controlling computation by the perception algorithm (104) and/or controlling the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error exceeding a maximum error characterized by one of the performance requirements and the runtime power consumption being within an acceptable range of power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences between the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error while allowing increase of the runtime power consumption up to a maximum of the acceptable range of power consumption, during further computation of the perception algorithm (104). The electronic device (200, 230) of any of Claims 1 to 13, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device (200) used for execution of the perception algorithm (104); and the at least one processor (202, 240) is further operative to adapt the parameters controlling computation by the perception algorithm (104) and/or controlling the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error exceeding a maximum localization error characterized by one of the performance requirements and the runtime power consumption being less than a maximum power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences between the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error while allowing increase of the runtime power consumption up to the maximum power consumption, during further computation of the perception algorithm (104). The electronic device (200, 230) of any of Claims 1 to 14, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device (200) used for execution of the perception algorithm (104); and the at least one processor (202, 240) is further operative to adapt the parameters controlling computation by the perception algorithm (104) and/or controlling the amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error being less than a maximum localization error characterized by one of the performance requirements and the runtime power consumption being less than a maximum power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences of one of: 1) the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error without the runtime power consumption exceeding the maximum power consumption, during further computation of the perception algorithm (104); and 2) the runtime power consumption and the maximum power consumption without the runtime localization error exceeding the maximum localization error, during further computation of the perception algorithm (104).

16. A method by an electronic device (200, 230) comprising: obtaining (300) performance requirements for a perception algorithm executed by a mobile device to compute at least one of: poses of features of a real -world environment, a pose of the mobile device relative to features of the real-world environment, and/or recognition of objects based on features of the real -world environment, using data from at least one sensor; obtaining (302) runtime performance metrics relating to the perception algorithm; and adapting (304) parameters controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm.

17. The method of Claim 16, wherein the electronic device (200, 230) comprises a server (230), and further comprising: receiving the performance requirements and the runtime performance metrics from the mobile device through a network connection; and communicating a message through the network connection to the mobile device indicating the adapted parameters to be used to control computations of the perception algorithm by the mobile device and/or to be used to control an amount of computing resource of the mobile device allocated to execute the perception algorithm.

18. The method of Claim 16, wherein the electronic device (200, 230) comprises the mobile device (200), and further comprising controlling computations of the perception algorithm using the adapted parameters and/or to controlling an amount of computing resource of the mobile device allocated to execute the perception algorithm based on the adapted parameters.

19. The method of any of Claims 16 to 18, further comprising repeating the adaptation of the parameters, using updated runtime performance metrics that are obtained, at a time interval or time instance that is defined based on at least one of the following: amount of change in ambient brightness of the real-world environment detectable by the at least one sensor between instances of the time interval or time instance; amount of change in features of the real-world environment which are detectable by the at least one sensor (210) between instances of the time interval or time instance; amount of change in computing resources of the mobile device (200) available to be allocated to execute the perception algorithm (104) between instances of the time interval or time instance; and amount of change in speed of the mobile device (200) between instances of the time interval or time instance.

21. The method of any of Claims 16 to 20, wherein the adapting (304) parameters controlling computation by the perception algorithm (104) and/or controlling the amount of computing resources of the mobile device allocated to execute the perception algorithm, to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm, comprises: processing the performance requirements and the runtime performance metrics through a performance optimization algorithm to generate the adapted parameters which reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm, wherein the performance optimization algorithm approximates relationships between:

1) using particular values for the parameters to control computation by the perception algorithm and/or control the amount of computing resources of the mobile device allocated to execute the perception algorithm; and 2) the runtime performance metrics which result from computation of the perception algorithm when using the particular values for the parameters.

22. The method of Claim 21, wherein the performance optimization algorithm comprises a Pareto optimization algorithm which contains Pareto curves approximating the relationships, and the parameters are adapted based on selecting values among the Pareto curves using the performance requirements as value selection pointers.

23. The method of any of Claims 21 to 22, wherein the performance optimization algorithm comprises a machine learning model that is adapted based on differences between the performance requirements and runtime performance metrics relating to the at least one of: the poses of features of the real-world environment, the pose of the mobile device relative to features of the real -world environment, and/or the recognition of objects based on features of the real-world environment computed by the perception algorithm using earlier adapted parameters to control computation by the perception algorithm and/or control the amount of computing resources of the mobile device allocated to execute the perception algorithm.

24. The method of Claim 23, wherein the adaptation of the machine learning model is triggered by at least one of the following: determining that one of the runtime performance metrics does not satisfy one of the performance requirements; determining that a plurality of the runtime performance metrics do not satisfy a plurality of prioritized ones of the performance requirements; determining that a maximum rate of adaptation of the machine learning algorithm will not be exceeded; determining that a minimum rate of adaptation of the machine learning algorithm will be exceeded if adaptation of the machine learning algorithm is not presently triggered; and determining that at least a threshold change occurs between at least one of the runtime performance metrics that are presently obtained and at least one of the runtime performance metrics that were most recently previously obtained.

25. The method of any of Claims 16 to 24, wherein: the parameters controlling computation by the perception algorithm and/or controlling an amount of computing resources of the mobile device allocated to execute the perception algorithm, control at least one of the following: a maximum number of features of the real-world environment identified in images from a camera of the at least one sensor during further execution of the perception algorithm; a priority of execution provided to the perception algorithm by an operating system managing multi-tasking access to computational resources of the mobile device; a number of central processing unit, CPU, cores in the mobile device that are allocated to execute the perception algorithm during further computation of the poses of features of the real-world environment and/or pose of the mobile device using the perception algorithm; a number of graphics processing unit, GPU, cores in the mobile device that are allocated to execute the perception algorithm during further computation of the poses of features of the real-world environment and/or pose of the mobile device using the perception algorithm; a clock rate of a CPU core in the mobile device when processing the perception algorithm during further computation of the poses of features of the real-world environment and/or pose of the mobile device using the perception algorithm; a clock rate of a GPU core in the mobile device when processing the perception algorithm during further computation of the poses of features of the real-world environment and/or pose of the mobile device using the perception algorithm; a power mode of computational resources of the mobile device during further computation of the poses of features of the real-world environment and/or pose of the mobile device using the perception algorithm; and a speed of the mobile device moving through the real-world environment during further computation of the poses of features of the real-world environment and/or pose of the mobile device using the perception algorithm.

26. The method of any of Claims 16 to 25, wherein: the performance requirements characterize at least one of the following: a maximum, minimum, and/or acceptable range of localization error when computing localization of the mobile device relative to features of the real- world environmentand/or relative to objects recognized based on features of the real-world environment by the perception algorithm using the data from the at least one sensor, a maximum, minimum, and/or acceptable range of feature detection error when computing the poses of features of the real-world environment by the perception algorithm using the data from the at least one sensor, a maximum, minimum, and/or acceptable range of 3D reconstruction error when computing a 3D representation of features of the real-world environment by the perception algorithm using the data from the at least one sensor; a maximum, minimum, and/or acceptable range of latency when computing localization of the mobile device relative to features of the real-world environment and/or relative to objects recognized based on features of the real-world environment by the perception algorithm using the data from the at least one sensor, a maximum, minimum, and/or acceptable range of latency when computing the poses of features of the real-world environment and/or pose of the mobile device by the perception algorithm using the data from the at least one sensor, a maximum, minimum, and/or acceptable range of latency when computing the 3D representation of features of the real-world environment by the perception algorithm using the data from the at least one sensor, a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device which can be used for localization of the mobile device relative to features of the real-world environment by the perception algorithm using the data from the at least one sensor, a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device which can be used for computation of the poses of features of the real-world environment and/or pose of the mobile device by the perception algorithm using the data from the at least one sensor, and a maximum, minimum, and/or acceptable range of power consumption of computational resources of the mobile device which can be used for computing the 3D representation of features of the real-world environment by the perception algorithm using the data from the at least one sensor.

27. The method of any of Claims 16 to 26, wherein: the runtime performance metrics characterize at least one of the following: a localization error in computed localization of the mobile device relative to features of the real-world environment by the perception algorithm using the data from the at least one sensor, a feature detection error in the poses of features of the real-world environment by the perception algorithm computed by the perception algorithm using the data from the at least one sensor, a 3D reconstruction error in a 3D representation of features of the real-world environment computed by the perception algorithm using the data from the at least one sensor, an object recognition error, a latency in computing localization of the mobile device relative to features of the real -world environment and/or relative to objects recognized based on features of the real-world environment by the perception algorithm using the data from the at least one sensor, a latency in computing the poses of features of the real-world environment and/or pose of the mobile device by the perception algorithm using the data from the at least one sensor, a latency in computing the 3D representation of features of the real-world environment by the perception algorithm using the data from the at least one sensor, a power consumption of computational resources of the mobile device used for localization of the mobile device relative to features of the real-world environment by the perception algorithm using the data from the at least one sensor, a power consumption of computational resources of the mobile device used for computation of the poses of features of the real-world environment and/or pose of the mobile device by the perception algorithm using the data from the at least one sensor, a power power consumption of computational resources of the mobile device used for recognizing objects based on features of the real -world environment using the data from the at least one sensor, a power consumption of computational resources of the mobile device used for computing the 3D representation of features of the real-world environment by the perception algorithm using the data from the at least one sensor. The method of any of Claims 16 to 27, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device used for execution of the perception algorithm; and further comprising adapting the parameters controlling computation by the perception algorithm and/or controlling the amount of computing resources of the mobile device allocated to execute the perception algorithm, by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, the object recognition error, and the 3D reconstruction error exceeding a maximum error characterized by one of the performance requirements and the runtime power consumption exceeding a maximum power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences of one of: 1) the maximum error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error without seeking to reduce the runtime power consumption, during further computation of the perception algorithm; and 2) the runtime power consumption and the maximum power consumption without seeking to reduce the runtime localization error, during further computation of the perception algorithm. The method of any of Claims 16 to 28, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device used for execution of the perception algorithm; and further comprising adapting the parameters controlling computation by the perception algorithm and/or controlling the amount of computing resources of the mobile device allocated to execute the perception algorithm, by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error exceeding a maximum error characterized by one of the performance requirements and the runtime power consumption being within an acceptable range of power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences between the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error while allowing increase of the runtime power consumption up to a maximum of the acceptable range of power consumption, during further computation of the perception algorithm.

The method of any of Claims 16 to 29, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device used for execution of the perception algorithm; and further comprising adapting the parameters controlling computation by the perception algorithm and/or controlling the amount of computing resources of the mobile device allocated to execute the perception algorithm, by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error exceeding a maximum localization error characterized by one of the performance requirements and the runtime power consumption being less than a maximum power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences between the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error while allowing increase of the runtime power consumption up to the maximum power consumption, during further computation of the perception algorithm. The method of any of Claims 16 to 30, wherein: one of the runtime performance metrics characterizes at least one of runtime localization error, feature detection error, object recognition error, and 3D reconstruction error; another one of the runtime performance metrics characterizes runtime power consumption of computational resources of the mobile device used for execution of the perception algorithm; and further comprising adapting the parameters controlling computation by the perception algorithm and/or controlling the amount of computing resources of the mobile device allocated to execute the perception algorithm, by operations comprising: responsive to the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error being less than a maximum localization error characterized by one of the performance requirements and the runtime power consumption being less than a maximum power consumption characterized by another one of the performance requirements, adapt the parameters to minimize differences of one of: 1) the maximum localization error and the at least one of the runtime localization error, the feature detection error, the object recognition error, and the 3D reconstruction error without the runtime power consumption exceeding the maximum power consumption, during further computation of the perception algorithm; and 2) the runtime power consumption and the maximum power consumption without the runtime localization error exceeding the maximum localization error, during further computation of the perception algorithm.

32. A computer program product comprising a non-transitory computer readable medium storing program instructions executable by at least one processor (202, 240) of an electronic device (200, 230) to: obtain performance requirements for a perception algorithm (104) executed by a mobile device (200) to compute at least one of: poses of features of a real-world environment, a pose of the mobile device (200) relative to features of the real- world environment, and/or recognition of objects based on features of the real- world environment, using data from at least one sensor (210); obtain runtime performance metrics relating to the perception algorithm (104); and adapt parameters controlling computation by the perception algorithm (104) and/or controlling an amount of computing resources of the mobile device (200) allocated to execute the perception algorithm (104), to reduce differences between the performance requirements and the runtime performance metrics during further computation of the perception algorithm (104).