WO2023025791A1

WO2023025791A1 - Object tracking for lower latency and less bandwidth

Info

Publication number: WO2023025791A1
Application number: PCT/EP2022/073468
Authority: WO
Inventors: Gabor Sandor Enyedi; Dávid JOCHA; Bence FORMANEK; Balázs Peter GERÖ; András Kern
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2021-08-27
Filing date: 2022-08-23
Publication date: 2023-03-02

Abstract

A UE obtains (601) a data stream having RGB and/or RGBD image frames. The stream is split (603) into first and second data streams at a first and second frame rates respectively where the first frame rate is higher than the second frame rate. The second stream is transmitted (605) to at least one remote processing component for object detection, location and pose detection, and associated keypoint selection. A keypoint refresh is received (607) including a 2D position of one or more keypoints in the RGB / RGBD image frames in the second data stream and a region of interest (Rol). The keypoints are tracked (609) in the first stream based on the Rol. Changes in object position and pose of objects are determined (611) based on associated key point positions tracked for each of the frames in the first data stream. The changes in position and pose of objects are transmitted towards an application.

Description

OBJECT TRACKING FOR LOWER LATENCY AND LESS BANDWIDTH

TECHNICAL FIELD

The present disclosure relates generally to communications, and more particularly to communication methods and related devices and nodes supporting object detection.

BACKGROUND

Object detection in three dimensions (3D), based on camera input, is one of the most important tasks of computer vision, being needed in both robotics and in augmented reality (AR) use-cases. When an object has already been found in a 3D space, the next step is determining its position and orientation. It is also important also to know the position and orientation of those parts of the object which are not seen. For example, if, in an AR scenario, there is a table with a leg not seen in any of the input data, then a virtual ball is still expected to bounce off the hidden leg.

There currently exist certain challenge(s). State-of-art object and pose detection algorithms are using complex solutions, typically involving neural networks. If such algorithms are executing on a mobile/wearable device with limited processing power and battery, this may be challenging, even with proper hardware accelerators.

SUMMARY

It is a straightforward idea to move the processing to where more resource is available, for example to an edge node of a cloud processing arrangement. Unfortunately, previous solutions simply forward a video stream to the cloud, which means that those solutions are especially dependent on perfect network conditions, for example appropriate levels of network throughput and latency, which may not always be possible to achieve in real mobile networks.

Various embodiments described herein use two different methods for object location and pose tracking. One is computationally challenging, but robust, that runs in the (edge) cloud, and the second is a keypoint tracking method which is regularly reinitialized.

According to some embodiments, a method performed by a user equipment, UE, includes obtaining a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames. The method further includes splitting the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate. The method further includes transmitting the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream. The method further includes receiving a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more key points. The method further includes determining and tracking the 3D position of the one or more keypoints of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest. The method further includes determining a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame. The method further includes transmitting the changes in object position and pose towards an application.

Analogous UEs and computer programs are provided.

Certain embodiments may provide one or more of the following technical advantage(s). The various embodiments use less networking load and computation resources than when pose and position is computed for each frame, since only a fraction of the frames is processed by the computationally expensive robust algorithm. Moreover, since keypoint tracking can be performed on the UE with high speed, the latency with respect to the real world becomes very low. Finally, mobile networks may have varying transport speed where with the various embodiments, it is possible to wait a bit for better radio environment when needed. Thus, the various embodiments may provide one or more of lower average throughput and computational complexity, relaxed latency and throughput requirements, and have a lower delay to the real world.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of inventive concepts. In the drawings:

Figure 1 is an illustration of a system for detecting and tracking the position and pose of one or multiple objects according to some embodiments of inventive concepts;

Figure 2 is a flowchart illustrating operations of the system of Figure 1 according to some embodiments of inventive concepts;

Figure 3 is a block diagram illustrating a wireless device UE according to some embodiments of inventive concepts;

Figure 4 is a block diagram illustrating a radio access network RAN node (e.g., a base station eNB/gNB) according to some embodiments of inventive concepts;

Figure 5 is a block diagram illustrating a core network CN node (e.g., an AMF node, an SMF node, etc.) according to some embodiments of inventive concepts;

Figures 6-9 are flow charts illustrating operations of a UE according to some embodiments of inventive concepts;

Figure 10 is a block diagram of a communication system in accordance with some embodiments;

Figure 11 is a block diagram of a user equipment in accordance with some embodiments

Figure 12 is a block diagram of a network node in accordance with some embodiments;

Figure 13 is a block diagram of a host computer communicating with a user equipment in accordance with some embodiments;

Figure 14 is a block diagram of a virtualization environment in accordance with some embodiments; and

Figure 15 is a block diagram of a host computer communicating via a base station with a user equipment over a partially wireless connection in accordance with some embodiments in accordance with some embodiments.

DETAILED DESCRIPTION

Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings in which examples of embodiments of inventive concepts are shown. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present/used in another embodiment. Currently, with the rapid development of neural networks, most of the object position and pose detection algorithms use artificial intelligence, Al, techniques. However, there are some, much simpler methods already known, such as those described in the document titled "Comparison of tracking algorithms implemented in OpenCV," to P. Janku, K. Koplik, T. Dulik, and I. Szabo, MATEC Web of Conferences, vol. 76, no. 04031, 2016, which describes optical keypoint tracking.

Other methods are more complex, such as the complex keypoint-based methodology described in "6DoF Object Tracking based on 3D Scans for Augmented Reality Remote Live Support" to J. Rambach, A. Pagani, M. Schneider, O. Artemenko and D. Stricker, which aims to find much more “stable” keypoints, and streams all the frames to the cloud where the processing is done.

According to various embodiments of inventive concepts, a combination of two methods is used for object pose detection and tracking, the two methods being operated in parallel. The first method is a robust, complex, and relatively slow method implemented in the cloud. The second method is a fast but less robust method implemented on user equipment, UE. The various embodiments use the slower method when possible (e.g. the radio condition is good enough), periodically and/or when unavoidable, to reinitialize the fast method, and use the fast method to decrease computational load, network load and latency.

Various embodiments disclosed herein have the advantage of using much less networking load and computation resource than known techniques in by which pose and position are computed for each frame. As will be shown below, only a fraction of the frames is communicated to and processed by the computationally expensive robust algorithm. Moreover, since tracking of objects may be performed to some level of accuracy on the UE with high speed, the latency with respect to the real world becomes very low. Finally, mobile networks may have varying transport speed; with the present solution, it is possible to wait a period of time for a better radio environment, when needed to communicate new frames to the cloud-hosted components.

A system for detecting and tracking the position and pose of one or multiple objects detectable in received image frames is illustrated in Figure 1. When embodiments described herein are applied to the problem of tracking rigid bodies, the pose of the body typically means “rotation” or “orientation” in the following description, and there are six degrees of freedom (DoF), i.e. 3 dimensions for position and 3 angles for orientation. Note that the various embodiments described herein are not limited to rigid bodies. Deformations (e.g. people with joints) can be handled in similar ways, except that the number of degrees of freedom, n, is greater than 6.

In overview, the system illustrated in Figure 1 implements functionality to determine the position and orientation (pose) of objects detectable in a received stream of images. The images are preferably RGBD images output (Create RGBD Stream component 100), for example, by one or more image sensors of a UE or other portable/mobile device, the RGBD images comprising Red, Green and Blue color component values and a Depth component value for pixels in each image.

In one example implementation of the system in Figure 1, functionality may be implemented in the UE to receive the stream of images and to “split” (Split Stream component 102) the stream into a low frame rate stream and a higher frame rate stream. The low frame rate stream is intended for communication to and use by cloud-hosted processing for robust object detection and tracking. The high frame rate stream is intended for consumption by UE-implemented functionality providing a simpler object tracking technique that requires a lower level of processing. However, the simpler UE-implemented tracking functionality requires regular reinitialization using data output by the cloud-hosted processing, as network conditions allow.

The low frame rate stream is communicated to cloud-hosted components for robust object detection and determination of object 3D position and pose (Robust nDoF component 104) and finding and selecting keypoints associated with the detected objects (Keypoint Finding component 106) and their 2D position within each frame.

The high frame rate stream is passed (102) to less robust and less processing-intensive components, implemented in the UE, for tracking the 3D position and pose of objects detected in the low frame rate frames by the cloud-hosted components. In particular, the UE- hosted components determine, in each high frame rate frame, the 3D position of keypoints found in the low frame rate frames by the cloud-hosted component (106) and uses those keypoint positions to track (108), in each high frame rate frame, changes in the 3D position and pose of the associated objects based upon received, robustly determined 3D position and pose of those objects by the cloud-hosted component (104) in the low frame rate frames.

The UE is therefore able to track the position and pose of objects in the high frame rate stream of frames, with low latency, and to supply the object position and pose data for each high frame rate frame to one or more applications (112) executing on the UE or elsewhere. These applications include, for example, applications providing augmented (AR), mixed (XR) or virtual reality (VR) display experiences to a user of the UE.

The components and functionality of the system shown in Figure 1 will now be described in more detail.

Create RGBD Stream 100

In the system shown in Figure 1, a Create RGBD Stream component 100 supplies a high frame rate image stream as an input to object tracking functionality of the system. In the example shown in Figure 1, the component 100 generates a stream of RGBD images, i.e. traditional color (RGB) images with depth information for each pixel if available. The component 100 may comprise one or more UE-hosted cameras able to determine the depth information (in the best case, there is depth information for all the pixels). This means that with proper camera calibration, it is possible to determine the precise 3D position of each pixel with respect to the camera. The depth information can be determined in multiple ways, usually by stereo camera or LIDAR. The cost of equipment producing RGBD images is currently reducing. For example, high-end cell phones are already RGBD capable. Thus, RGBD sensors are expected to be common in future UEs.

Note that it is not essential that the component 100 provides RGBD images. The RGB part of the “RGBD” may comprise a monochromatic image, or, after some preprocessing, a vector graphic image. For the depth data, in theory it is possible to restore the same information later in the process by tracking more points, but this is very undesirable as shall be explained below in a "Keypoint tracking" component. In the description below, the particular example of an RGBD frame input or equivalent shall be used in the description of various embodiments of inventive concepts.

Stream Split component 102

For tracking, multiple RGBD images of the same object are expected, i.e. a video stream. A stream split component 102 receives a stream of RGBD images from the Create RGBD Stream component 100, e.g. a camera system, and comprises functionality for generating two versions of the received stream: a high frame rate RGBD stream which may, for example, comprise the received stream without alteration; and a low frame rate RGBD stream. The component 102 may generate the low frame rate stream by selection of images from the RGBD stream received from the Create RGBD Stream component 100. Alternatively the component 102 may generate the low frame rate stream using one of various known techniques to perform transcoding or other re-coding methods to generate a stream of RGBD images at a lower frame rate than that received. The stream split component 102 directs the lower frame rate stream to functional components configured to perform the above-mentioned robust determination of position and pose of one or more objects detectable in the frames of the low frame rate stream, implemented for example in a higher capacity processing environment separate from the UE. The stream split component 102 directs the high frame rate stream to functional components configured to perform the above-mentioned less robust and less processing-intensive keypoint tracking and, from that, changes in the 3D position and pose of objects in frames of the high frame rate stream, which may for example be implemented in the UE.

Note that the RGBD camera(s) (100) and the stream splitting functionality (102) is typically implemented at the UE, but this is not the case for the robust determination functionality. The "Stream Split" component 102 is therefore responsible for sending the lower frame rate data to an appropriate endpoint through the network where the robust determination functionality is accessible or implemented.

The Stream Split component 102 may also be configured to add a timestamp and/or a sequence number to each one of the RGB image frames and/or the RGBD image frames that it receives from the create RGBD stream component 100 to aid in subsequent matching of image frames from high and low frame rate streams that it outputs. For example, the Keypoint Tracking component 108 and the Position+Pose Computing Component 110 may use the added timestamp/sequence number to match frames received in the high frame rate stream to the low frame rate stream frames for which object and keypoint data are received from the cloud-hosted components 106, 108.

Robust nDoF component 104

According to various embodiments of inventive concepts, the slow but Robust nDoF component 104 implements a known method for determining the position in three dimensions (3D position) and pose of one or more objects detectable in a single received RGBD image. The Robust nDoF component 104 may be implemented within a cloud-hosted processing arrangement or other high capacity processing arrangement that is remote from the UE that is configured to receive the low frame rate RGBD frames. The Robust nDoF component 104 may comprise a neural network or may implement another known method for detecting one or more objects in the received frames and for determining the 3D position and pose of one or more detected objects. This component (104) is also configured to create and to output a 2D mask for each detectable object, selecting the pixels of the object in the RGB image (this is called “image segmentation”, which is typically a necessary preprocessing part of many 6D positioning algorithms).

Running a neural network implementation, in particular, of this robust position and pose-determining component (104) is typically computationally intensive. Such a neural network may typically have hundreds of layers and tens or hundreds of millions of variables. Even if it is possible to utilize high level data parallelism inside a single layer, the layers themselves should be processed sequentially (albeit with a few exceptions), which takes time even with state-of-art hardware. Special purpose circuits optimized for neural networks can help to speed up this process, but due to the computational intensity required, the various embodiments of inventive concepts put this function into a cloud-hosted processing arrangement in order to speed up computation, increase precision and/or save UE battery life for other things.

However, relying on remote cloud-hosted processing means having to forward RGBD images (in a video stream) from the UE to the cloud-hosted components in a real-time manner. This requires the network to maintain relatively low latency and high throughput, which may not always be possible in mobile networks, where the link condition varies. Moreover, having many users in the same network cell may be challenging for both the network and a respective processing node in an edge cloud. Finally, sending the information through the network may improve the relatively high computational latency of the complex location/pose estimation algorithm, which means that the tracking information received from the cloud may be already slightly outdated when received.

Thus, according to inventive concepts disclosed herein, the particular implementation and distribution of object tracking functionality enables a tradeoff to be made between remote processing of some or all of the frames and latency, real-time throughput, etc. To account for this, in the various embodiments of inventive concepts, the Stream Split component 102 is configured to adjust the frame rate of the low frame rate output that it sends to the cloud- hosted Robust nDoF component 104, for example to reduce the frame rate to at most a few times per second (less than 1 frame per second (fps) is possible in many instances). Thus, the calculation of a 3D position and the pose of an object, and the creation of a 2D mask of an object may be carried out, at most, a few times per second.

Key point Finding component 106

The Key point Finding component 106 implements a known key point detection and selection algorithm, configured to find a required number of keypoints for each of one or more objects detected by the Robust nDoF component 104. The Keypoint Finding component 106 receives from the component 104 a 2D mask defining the pixels of each detected object in an RGBD image and the 3D position and pose of each object. The Keypoint Finding component 106 may find and select keypoints anywhere on the object, for example at an edge of the object or away from an edge of the object, according to the level of texturing and detail that may be detected on the object. For each keypoint that is found, the Keypoint Finding component 106 outputs a 2D position of the key point in the image frame and an indication of a “region of interest” around the keypoint. The output 2D keypoint positions and associated region of interest indications are sent to the UE for use by a Keypoint Tracking component 108, implemented on the UE.

Note that when there is a need to select a keypoint for tracking, a single point alone cannot be tracked. Thus, when a keypoint is to be tracked, a tracking algorithm (108) is looking for an area around it, not for just a pixel. In extreme cases, this area can be quite huge, i.e. when used for keypoint tracking, some convolutional neural networks can consider even the whole image. Thus, various embodiments use optical keypoint tracking, so a small area around the keypoint (called a Rol - region of interest) is tracked.

Note that these keypoints are not exactly similar to those used for long term tracking by known Simultaneous Localization and Mapping (SLAM) algorithms. A reason for this is that since tracking is needed only for a short amount of time, the key points do not need to be selected so carefully, and in many cases they can be almost any point belonging to the object in the 2D image. However, if the object is very poorly textured, some care in selecting the keypoint and the Rol is needed by the Keypoint Finding component 106. In the case of poor textures, the key points can be selected e.g. at the edge of the object.

The number of keypoints is important in some embodiments. If one considers rigid objects, RGBD images and calibrated cameras (i.e. the 3D position of the pixel with respect to the camera can be calculated) 3 points are often sufficient. If more precision is required, in the case of missing depth data or when the degree of freedom is higher than 6, more keypoints are needed. Note that this is a tradeoff as each keypoint needs some computation, thus the tracking complexity increases and it is beneficial to keep the number of keypoints as low as possible.

Besides sending the 2D position of selected keypoints and associated indications of their Rol to the Keypoint Tracking component 108, which is configured to track these keypoints, the original 3D positions of the objects and hence of the selected keypoints is needed by the UE for computing their displacement; this and the original 3D position of the object is therefore sent from the cloud-hosted components 104, 106 to a Position+pose Computing component 110 implemented on the UE.

The Keypoint Finding component 106 may be configured to reinitialize the Keypoint Tracking component 108 with updated details of the tracked keypoints according to one or more different criteria. These updates may be regular, may be done when some keypoint is lost, when sufficient radio resources are available, or any combination thereof. Moreover, some objects may move faster or are more important than others, in which case the reinitialization may be more frequent.

Keypoint Tracking component 108

The Keypoint Tracking component 108 is configured to receive the RGBD frames output at a high frame rate from the Stream Split component 102 and to receive, over the network, the 2D positions and indicated regions of interest for keypoints in those frames, found and selected by the cloud-hosted Key point Finding component 106. The 2D positions and indicated regions of interest for keypoints received from the Key point Finding component 106 are used to initialize and periodically to reinitialize the Keypoint Tracking component 108. The interval between successive receipts of keypoint data from the component 106 may be pre-configured in the component 106, or it may be adjusted in response to particular conditions arising in the finding of keypoints in received RGBD frames, as determined by the component 106.

The Key point Tracking component 108 may implement one of numerous known optical object tracking algorithms for tracking selected points in a 2D video and will not be described in detail here. These optical object tracking algorithms are configured to receive the 2D position of a selected point on an object in a 2D image, an indication of an area around the point, and to track the selected point in a video stream.

One problem that can occur with some of these optical tracking algorithms is that they are prone to lose the tracked point if the optical environment changes, for example, in the presence of occlusion, or if the associated object turns too much. In many of these algorithms, if a selected point is lost, it will never be found again, even if the image is very similar to a previous one where the point was easily tracked (e.g. the occlusion is removed). However, the impact of such a situation is reduced in the various embodiments described herein because tracking without reinitialization is needed for only a short amount of time. The optical environment typically does not change significantly between successive reinitializations of key point data from the Keypoint Finding component 106. Moreover, advantageously, the algorithms selected for implementation by the Keypoint Tracking component 108 are typically much simpler and computationally less expensive than known neural network-based keypoint tracking techniques.

An output of the Key point Tracking component 108 is the 2D position of each keypoint in each frame received from the Stream Split component 102. The 2D position is exactly what keypoint tracking algorithms provide. However, the Keypoint Tracking component 108 is also configured to determine and output a 3D position for each keypoint in each received frame. This is a simple calculation if the received frames are RGBD frames, using the determined 2D position for each keypoint combined with the received depth information for pixels in the frame. If the received frames are RGB images, the determination of 3D position is more difficult, but still possible, if the number of key points is sufficiently high. However, the additional processing required to estimate depth information for keypoints of an object, when not available in received RGBD data for the corresponding pixels, makes this an undesirable option for UE-implemented functionality.

The Keypoint Tracking component 108 therefore outputs, for each received frame, the 3D position of each of the tracked keypoints and forwards these data to the Position+pose Computing component 110.

Note that the processing in the “Robust nDoF” component 104 may take some time. Therefore, when the positions of objects and associated key points are received from the cloud-hosted components 104, 106, that information may hold some lag to reality. Fortunately, this is not an issue for most keypoint tracking algorithms (108): the keypoints found and selected an image processed by the Robust nDoF component 104 may be found in newer frames, as long as the difference is not “too much”, i.e. if only a few frames pass without a further keypoint update being received from the component 106, that is OK. Position+pose computing component 110

The Position+pose computing component 110 receives 3D positions for each keypoint in a frame of the high frame rate stream from the Keypoint Tracking component 108. The Position+pose computing component 110 compares those 3D positions to the last position of the keypoints sent by the Keypoint Finding component 108. If at least 3 keypoints on the object can be found, they describe a triangle, which moves and turns with the object, thus based on their displacement, the new position and pose of the original object can be calculated. If there are more key points, this computation can be more precise, or, in the case of well-placed keypoints, non-rigid objects (e.g. moving joints) can be tracked. The result is the n coordinates describing the position and the pose of the object. Application 112

An application 112 is an end user of the object tracking data. An application may be executing on the UE. Alternatively, the application 112 or be executing in the cloud-hosted processing arrangement and may be receiving and merging data from multiple UEs.

Figure 3 is a block diagram illustrating elements of a UE 300 (also referred to as a mobile terminal, a mobile communication terminal, a wireless device, a wireless communication device, a wireless terminal, mobile device, a wireless communication terminal, user equipment, UE, a user equipment node/terminal/device, etc.) configured to provide wireless communication according to embodiments of inventive concepts. (UE 300 may be provided, for example, as discussed below with respect to wireless devices UE 1012A, UE 1012B, and wired or wireless devices UE 1012C, UE 1012D of Figure 10, UE 1100 of Figure 11, virtualization hardware 1404 and virtual machines 1408A, 1408B of Figure 14, and UE 1506 ofFigure 15, all of which should be considered interchangeable in the examples and embodiments described herein and be within the intended scope of this disclosure, unless otherwise noted.) As shown, UE may include an antenna 307 (e.g., corresponding to antenna 1122 of Figure 11), and transceiver circuitry 301 (also referred to as a transceiver, e.g., corresponding to interface 1112 of Figure 11 having transmitter 1118 and receiver 1120) including a transmitter and a receiver configured to provide uplink and downlink radio communications with a base station(s) (e.g., corresponding to network node 1010A, 1010B ofFigure 10, network node 1200 ofFigure 12, and network node 1504 of Figure 15 also referred to as a RAN node) of a radio access network. UE may also include processing circuitry 303 (also referred to as a processor, e.g., corresponding to processing circuitry 1102 ofFigure 11, and control system 1412 of Figure 14) coupled to the transceiver circuitry, and memory circuitry 305 (also referred to as memory, e.g., corresponding to memory 1110 of Figure 10) coupled to the processing circuitry. The memory circuitry 305 may include computer readable program code that when executed by the processing circuitry 303 causes the processing circuitry to perform operations according to embodiments disclosed herein. According to other embodiments, processing circuitry 303 may be defined to include memory so that separate memory circuitry is not required. UE may also include an interface (such as a user interface) coupled with processing circuitry 303, and/or UE may be incorporated in a vehicle.

As discussed herein, operations of UE may be performed by processing circuitry 303 and/or transceiver circuitry 301. For example, processing circuitry 303 may control transceiver circuitry 301 to transmit communications through transceiver circuitry 301 over a radio interface to a radio access network node (also referred to as a base station) and/or to receive communications through transceiver circuitry 301 from a RAN node over a radio interface. Moreover, modules may be stored in memory circuitry 305, and these modules may provide instructions so that when instructions of a module are executed by processing circuitry 303, processing circuitry 303 performs respective operations (e.g., operations discussed below with respect to Example Embodiments relating to wireless communication devices). According to some embodiments, a UE 300 and/or an element(s)/function(s) thereof may be embodied as a virtual node/nodes and/or a virtual machine/machines.

Figure 4 is a block diagram illustrating elements of a radio access network RAN node 400 (also referred to as a network node, base station, eNodeB/eNB, gNodeB/gNB, etc.) of a Radio Access Network (RAN) configured to provide cellular communication according to embodiments of inventive concepts. (RAN node 400 may be provided, for example, as discussed below with respect to network node 1010A, 1010B of Figure 10, network node 1200 of Figure 3, hardware 1404 or virtual machine 1408A, 1408B of Figure 14, and/or base station 1504 of Figure 15, all of which should be considered interchangeable in the examples and embodiments described herein and be within the intended scope of this disclosure, unless otherwise noted.) As shown, the RAN node 400 may include transceiver circuitry 401 (also referred to as a transceiver, e.g., corresponding to portions of RF transceiver circuitry 1212 and radio front end circuitry 1218 of Figure 12) including a transmitter and a receiver configured to provide uplink and downlink radio communications with mobile terminals. The RAN node 400 may include network interface circuitry 507 (also referred to as a network interface, e.g., corresponding to portions of communication interface 1206 of Figure 12) configured to provide communications with other nodes (e.g., with other base stations) of the RAN and/or core network CN. The network node may also include processing circuitry 403 (also referred to as a processor, e.g., corresponding to processing circuitry 1202 of Figure 12) coupled to the transceiver circuitry, and memory circuitry 405 (also referred to as memory, e.g., corresponding to memory 1204 of Figure 12) coupled to the processing circuitry. The memory circuitry 405 may include computer readable program code that when executed by the processing circuitry 403 causes the processing circuitry to perform operations according to embodiments disclosed herein. According to other embodiments, processing circuitry 403 may be defined to include memory so that a separate memory circuitry is not required.

As discussed herein, operations of the RAN node may be performed by processing circuitry 403, network interface 407, and/or transceiver 401. For example, processing circuitry 403 may control transceiver 401 to transmit downlink communications through transceiver 401 over a radio interface to one or more mobile terminals UEs and/or to receive uplink communications through transceiver 401 from one or more mobile terminals UEs over a radio interface. Similarly, processing circuitry 403 may control network interface 407 to transmit communications through network interface 407 to one or more other network nodes and/or to receive communications through network interface from one or more other network nodes. Moreover, modules may be stored in memory 405, and these modules may provide instructions so that when instructions of a module are executed by processing circuitry 403, processing circuitry 403 performs respective operations (e.g., operations discussed below with respect to Example Embodiments relating to RAN nodes). According to some embodiments, RAN node 400 and/or an element(s)/function(s) thereof may be embodied as a virtual node/nodes and/or a virtual machine/machines.

According to some other embodiments, a network node may be implemented as a core network CN node without a transceiver. In such embodiments, transmission to a wireless UE may be initiated by the network node so that transmission to the wireless UE is provided through a network node including a transceiver (e.g., through a base station or RAN node). According to embodiments where the network node is a RAN node including a transceiver, initiating transmission may include transmitting through the transceiver.

Figure 5 is a block diagram illustrating elements of a core network (CN) node (e.g., an SMF (session management function) node, an AMF (access and mobility management function) node, etc.) of a communication network configured to provide cellular communication according to embodiments of inventive concepts. (CN node 500 may be provided, for example, as discussed below with respect to core network node 1008 of Figure 10, hardware 1404 or virtual machine 1408A, 1408B of Figure 14, all of which should be considered interchangeable in the examples and embodiments described herein and be within the intended scope of this disclosure, unless otherwise noted) As shown, the CN node may include network interface circuitry 507 configured to provide communications with other nodes of the core network and/or the radio access network RAN. The CN node may also include a processing circuitry 503 (also referred to as a processor,) coupled to the network interface circuitry, and memory circuitry 505 (also referred to as memory) coupled to the processing circuitry. The memory circuitry 505 may include computer readable program code that when executed by the processing circuitry 503 causes the processing circuitry to perform operations according to embodiments disclosed herein. According to other embodiments, processing circuitry 503 may be defined to include memory so that a separate memory circuitry is not required.

As discussed herein, operations of the CN node may be performed by processing circuitry 503 and/or network interface circuitry 507. For example, processing circuitry 503 may control network interface circuitry 507 to transmit communications through network interface circuitry 507 to one or more other network nodes and/or to receive communications through network interface circuitry from one or more other network nodes. Moreover, modules may be stored in memory 505, and these modules may provide instructions so that when instructions of a module are executed by processing circuitry 503, processing circuitry 503 performs respective operations (e.g., operations discussed below with respect to Example Embodiments relating to core network nodes). According to some embodiments, CN node 500 and/or an element(s)/function(s) thereof may be embodied as a virtual node/nodes and/or a virtual machine/machines.

In the description that follows, while the UE may be any of the UE 300, wireless device 1012A, 1012B, wired or wireless devices UE 1012C, UE 1012D, UE 1100, virtualization hardware 1404, virtual machines 1408A, 1408B, or UE 1506, the UE 300 shall be used to describe the functionality of the operations of the UE. Operations of the UE 300 (implemented using the structure of the block diagram of Figure e) will now be discussed with reference to the flow chart of Figure 6 according to some embodiments of inventive concepts. For example, modules may be stored in memory 305 of Figure 3, and these modules may provide instructions so that when the instructions of a module are executed by respective communication device processing circuitry 303, processing circuitry 303 performs respective operations of the flow chart.

Figure 6 illustrates operations a UE 300 performs in determining an object position and pose according to some embodiments of inventive concepts.

Turning to Figure 6, in component 601, the processing circuitry 303 obtains a data stream having one of RGB, red-green-blue, image frames and/or RGB depth, RGBD image frames. The data stream can be obtained from a camera, from a local or remote storage device, from a network device, etc.

In component 603, the processing circuitry 303 splits the data stream into a first data stream at a first rate and a second data stream at a second rate where the first rate is higher than the second rate. As indicated above, one reason for this is due to the computation time for the Robust nDoF component 104 to determine position, pose, etc.

In component 605, the processing circuitry 303, responsive to a keypoint refresh being triggered, transmits the second data stream to at least one remote module for object location and pose detection and for keypoint selection. In some embodiments a timestamp and/or a sequence number is added to each one of the RGB image frames and/or the RGBD image frames so that the processing circuitry 303 can match image frames in the first data stream to image frames in the second data stream.

The remote module may reside on a RAN node 400, a core network node 800, etc. Thus, in some embodiments, the processing circuitry 303 transmits the second data stream to at least one remote module for object location and pose detection and for keypoint selection by transmitting the second data stream to at least one base station of a mobile communications network that implements the at least one remote module for object location and pose detection and for keypoint selection.

Figure 7 illustrates various ways that the keypoint refresh is triggered. The processing circuitry 303 determines if the keypoint refresh condition has been triggered in component 701. In component 703, the processing circuitry 303 determines if the keypoint refresh condition has been triggered by determining if a timer has expired. Thus, the keypoint refresh condition has been triggered if the timer has expired. The timer can be adjusted. For example, the processing circuitry 303 decreases a time length of the timer responsive to there being one of an object moving faster than a normal speed and an object of importance.

In component 705, the processing circuitry 303 determines if the keypoint refresh condition has been triggered by determining if one or more keypoints have been lost. Thus, the keypoint refresh condition has been triggered if one or more keypoints have been lost.

In component 707, the processing circuitry 303 determines if the key point refresh condition has been triggered by determining if radio resources are available. Thus, if radio resources are available, the keypoint refresh condition has been triggered.

Returning to Figure 6, in component 607, the processing circuitry 303 receives a keypoint refresh from the at least one remote module, the keypoint refresh comprising a 2D position of one or more keypoints in the one of the RGB image frames and/or the RGBD image frames in the second data stream and a region of interest of the one or more key points.

In the embodiments where the processing circuitry 303 transmits the second data stream to at least one base station, the processing circuitry 303 receives the keypoint refresh from the at least one base station.

In component 609, the processing circuitry 303 tracks the one or more keypoints in the one of the RGB image frames and/or the RGBD image frames in the first data stream based on the region of interest.

In component 611, the processing circuitry 303 determines an object position and pose based on positions of the one or more key points tracked for each of the one of the RGB image frames and/or the RGBD image frames in the first data stream.

Figure 8 illustrates an embodiment of determining the object position and pose. Turning to Figure 8, in component 801, the processing circuitry 303 determines 3D coordinates of the one or more keypoints. In component 803, the processing circuitry 303 determines the object position and pose by determining the object position and pose based on 3D coordinates of the one or more keypoints.

Figure 9 illustrates another embodiment of determining the object position and pose based on the one or more keypoints. Turning to Figure 9, in component 901, the processing circuitry 303 compares positions of the one or more key points to last positions of the one or more keypoints. In component 903, the processing circuitry 303, responsive to at least 3 keypoints on the object being found, calculates a new 3D position of the object based on a displacement of the at least 3 keypoints from last positions of the at least 3 keypoints.

Returning to Figure 6, in component 613, the processing circuitry 303 transmits the object position and pose towards an application.

Figure 10 shows an example of a communication system 1000 in accordance with some embodiments.

In the example, the communication system 1000 includes a telecommunication network 1002 that includes an access network 1004, such as a radio access network (RAN), and a core network 1006, which includes one or more core network nodes 1008. The access network 1004 includes one or more access network nodes, such as network nodes 1010a and 1010b (one or more of which may be generally referred to as network nodes 1010), or any other similar 3^rd Generation Partnership Project (3GPP) access node or non-3GPP access point. The network nodes 1010 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 1012a, 1012b, 1012c, and 1012d (one or more of which may be generally referred to as UEs 1012) to the core network 1006 over one or more wireless connections. Example wireless communications over a wireless connection include transmitting and/or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and/or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 1000 may include any number of wired or wireless networks, network nodes, UEs, and/or any other components or systems that may facilitate or participate in the communication of data and/or signals whether via wired or wireless connections. The communication system 1000 may include and/or interface with any type of communication, telecommunication, data, cellular, radio network, and/or other similar type of system.

The UEs 1012 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and/or operable to communicate wirelessly with the network nodes 1010 and other communication devices. Similarly, the network nodes 1010 are arranged, capable, configured, and/or operable to communicate directly or indirectly with the UEs 1012 and/or with other network nodes or equipment in the telecommunication network 1002 to enable and/or provide network access, such as wireless network access, and/or to perform other functions, such as administration in the telecommunication network 1002.

In the depicted example, the core network 1006 connects the network nodes 1010 to one or more hosts, such as host 1016. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 1006 includes one more core network nodes (e.g., core network node 1008) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and/or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 1008. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and/or a User Plane Function (UPF).

The host 1016 may be under the ownership or control of a service provider other than an operator or provider of the access network 1004 and/or the telecommunication network 1002, and may be operated by the service provider or on behalf of the service provider. The host 1016 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio/video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

As a whole, the communication system 1000 of Figure 10 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and/or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and/or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and/or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

In some examples, the telecommunication network 1002 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 1002 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 1002. For example, the telecommunications network 1002 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and/or Massive Machine Type Communication (mMTC)ZMassive loT services to yet further UEs.

In some examples, the UEs 1012 are configured to transmit and/or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 1004 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 1004. Additionally, a UE may be configured for operating in single- or multi-RAT or multistandard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi -radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

In the example, the hub 1014 communicates with the access network 1004 to facilitate indirect communication between one or more UEs (e.g., UE 1012c and/or 1012d) and network nodes (e.g., network node 1010b). In some examples, the hub 1014 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 1014 may be a broadband router enabling access to the core network 1006 for the UEs. As another example, the hub 1014 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 1010, or by executable code, script, process, or other instructions in the hub 1014. As another example, the hub 1014 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 1014 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 1014 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 1014 then provides to the UE either directly, after performing local processing, and/or after adding additional local content. In still another example, the hub 1014 acts as a proxy server or orchestrator for the UEs, in particular in if one or more of the UEs are low energy loT devices.

The hub 1014 may have a constant/persistent or intermittent connection to the network node 1010b. The hub 1014 may also allow for a different communication scheme and/or schedule between the hub 1014 and UEs (e.g., UE 1012c and/or 1012d), and between the hub 1014 and the core network 1006. In other examples, the hub 1014 is connected to the core network 1006 and/or one or more UEs via a wired connection. Moreover, the hub 1014 may be configured to connect to an M2M service provider over the access network 1004 and/or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 1010 while still connected via the hub 1014 via a wired or wireless connection. In some embodiments, the hub 1014 may be a dedicated hub - that is, a hub whose primary function is to route communications to/from the UEs from/to the network node 1010b. In other embodiments, the hub 1014 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 1010b, but which is additionally capable of operating as a communication start and/or end point for certain data channels. Figure 11 shows a UE 1100 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and/or operable to communicate wirelessly with network nodes and/or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded/integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and/or an enhanced MTC (eMTC) UE.

A UE may support device-to-device (D2D) communication, for example by implementing a 3 GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and/or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

The UE 1100 includes processing circuitry 1102 that is operatively coupled via a bus 1104 to an input/output interface 1106, a power source 1108, a memory 1110, a communication interface 1112, and/or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 11. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

The processing circuitry 1102 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1110. The processing circuitry 1102 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 1102 may include multiple central processing units (CPUs).

In the example, the input/ output interface 1106 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and/or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 1100. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

In some embodiments, the power source 1108 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 1108 may further include power circuitry for delivering power from the power source 1108 itself, and/or an external power source, to the various parts of the UE 1100 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 1108. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 1108 to make the power suitable for the respective components of the UE 1100 to which power is supplied.

The memory 1110 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1110 includes one or more application programs 1114, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 1116. The memory 1110 may store, for use by the UE 1100, any of a variety of various operating systems or combinations of operating systems.

The memory 1110 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and/or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1110 may allow the UE 1100 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1110, which may be or comprise a device-readable storage medium.

The processing circuitry 1102 may be configured to communicate with an access network or other network using the communication interface 1112. The communication interface 1112 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 1122. The communication interface 1112 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 1118 and/or a receiver 1120 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 1118 and receiver 1120 may be coupled to one or more antennas (e.g., antenna 1122) and may share circuit components, software or firmware, or alternatively be implemented separately.

In the illustrated embodiment, communication functions of the communication interface 1112 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and/or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol/intemet protocol (TCP/IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 1112, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door/window sensor, a flood/moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and/or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 1100 shown in Figure 11.

As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and/or measurements, and transmits the results of such monitoring and/or measurements to another UE and/or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and/or reporting on its operational status or other functions associated with its operation.

In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and/or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

Figure 12 shows a network node 1200 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and/or operable to communicate directly or indirectly with a UE and/or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)).

Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units and/or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell/multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and/or Minimization of Drive Tests (MDTs).

The network node 1200 includes a processing circuitry 1202, a memory 1204, a communication interface 1206, and a power source 1208. The network node 1200 may be composed of multiple physically separate components (e.g., aNodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 1200 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 1200 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 1204 for different RATs) and some components may be reused (e.g., a same antenna 1210 may be shared by different RATs). The network node 1200 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 1200, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 1200.

The processing circuitry 1202 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and/or encoded logic operable to provide, either alone or in conjunction with other network node 1200 components, such as the memory 1204, to provide network node 1200 functionality.

In some embodiments, the processing circuitry 1202 includes a system on a chip (SOC). In some embodiments, the processing circuitry 1202 includes one or more of radio frequency (RF) transceiver circuitry 1212 and baseband processing circuitry 1214. In some embodiments, the radio frequency (RF) transceiver circuitry 1212 and the baseband processing circuitry 1214 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1212 and baseband processing circuitry 1214 may be on the same chip or set of chips, boards, or units.

The memory 1204 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and/or any other volatile or non-volatile, non-transitory device-readable and/or computerexecutable memory devices that store information, data, and/or instructions that may be used by the processing circuitry 1202. The memory 1204 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and/or other instructions capable of being executed by the processing circuitry 1202 and utilized by the network node 1200. The memory 1204 may be used to store any calculations made by the processing circuitry 1202 and/or any data received via the communication interface 1206. In some embodiments, the processing circuitry 1202 and memory 1204 is integrated.

The communication interface 1206 is used in wired or wireless communication of signaling and/or data between a network node, access network, and/or UE. As illustrated, the communication interface 1206 comprises port(s)/terminal(s) 1216 to send and receive data, for example to and from a network over a wired connection. The communication interface 1206 also includes radio front-end circuitry 1218 that may be coupled to, or in certain embodiments a part of, the antenna 1210. Radio front-end circuitry 1218 comprises filters 1220 and amplifiers 1222. The radio front-end circuitry 1218 may be connected to an antenna 1210 and processing circuitry 1202. The radio front-end circuitry may be configured to condition signals communicated between antenna 1210 and processing circuitry 1202. The radio front-end circuitry 1218 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 1218 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 1220 and/or amplifiers 1222. The radio signal may then be transmitted via the antenna 1210. Similarly, when receiving data, the antenna 1210 may collect radio signals which are then converted into digital data by the radio front-end circuitry 1218. The digital data may be passed to the processing circuitry 1202. In other embodiments, the communication interface may comprise different components and/or different combinations of components.

In certain alternative embodiments, the network node 1200 does not include separate radio front-end circuitry 1218, instead, the processing circuitry 1202 includes radio front-end circuitry and is connected to the antenna 1210. Similarly, in some embodiments, all or some of the RF transceiver circuitry 1212 is part of the communication interface 1206. In still other embodiments, the communication interface 1206 includes one or more ports or terminals 1216, the radio front-end circuitry 1218, and the RF transceiver circuitry 1212, as part of a radio unit (not shown), and the communication interface 1206 communicates with the baseband processing circuitry 1214, which is part of a digital unit (not shown).

The antenna 1210 may include one or more antennas, or antenna arrays, configured to send and/or receive wireless signals. The antenna 1210 may be coupled to the radio front-end circuitry 1218 and may be any type of antenna capable of transmitting and receiving data and/or signals wirelessly. In certain embodiments, the antenna 1210 is separate from the network node 1200 and connectable to the network node 1200 through an interface or port.

The antenna 1210, communication interface 1206, and/or the processing circuitry 1202 may be configured to perform any receiving operations and/or certain obtaining operations described herein as being performed by the network node. Any information, data and/or signals may be received from a UE, another network node and/or any other network equipment. Similarly, the antenna 1210, the communication interface 1206, and/or the processing circuitry 1202 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and/or signals may be transmitted to a UE, another network node and/or any other network equipment. The power source 1208 provides power to the various components of network node 1200 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 1208 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 1200 with power for performing the functionality described herein. For example, the network node 1200 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 1208. As a further example, the power source 1208 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

Embodiments of the network node 1200 may include additional components beyond those shown in Figure 12 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and/or any functionality necessary to support the subject matter described herein. For example, the network node 1200 may include user interface equipment to allow input of information into the network node 1200 and to allow output of information from the network node 1200. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 1200.

Figure 13 is a component diagram of a host 1300, which may be an embodiment of the host 1016 of Figure 10, in accordance with various aspects described herein. As used herein, the host 1300 may be or comprise various combinations hardware and/or software, including a standalone server, a blade server, a cloud-implemented server, a distributed server, a virtual machine, container, or processing resources in a server farm. The host 1300 may provide one or more services to one or more UEs.

The host 1300 includes processing circuitry 1302 that is operatively coupled via a bus 1304 to an input/output interface 1306, a network interface 1308, a power source 1310, and a memory 1312. Other components may be included in other embodiments. Features of these components may be substantially similar to those described with respect to the devices of previous figures, such as Figures 11 and 12, such that the descriptions thereof are generally applicable to the corresponding components of host 1300.

The memory 1312 may include one or more computer programs including one or more host application programs 1314 and data 1316, which may include user data, e.g., data generated by a UE for the host 1300 or data generated by the host 1300 for a UE. Embodiments of the host 1300 may utilize only a subset or all of the components shown. The host application programs 1314 may be implemented in a container-based architecture and may provide support for video codecs (e.g., Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG, VP9) and audio codecs (e.g., FLAC, Advanced Audio Coding (AAC), MPEG, G.711), including transcoding for multiple different classes, types, or implementations of UEs (e.g., handsets, desktop computers, wearable display systems, heads-up display systems). The host application programs 1314 may also provide for user authentication and licensing checks and may periodically report health, routes, and content availability to a central node, such as a device in or on the edge of a core network. Accordingly, the host 1300 may select and/or indicate a different host for over-the-top services for a UE. The host application programs 1314 may support various protocols, such as the HTTP Live Streaming (HLS) protocol, Real-Time Messaging Protocol (RTMP), Real-Time Streaming Protocol (RTSP), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc.

Figure 14 is a block diagram illustrating a virtualization environment 1400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 1400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized.

Applications 1402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and/or benefits of some of the embodiments disclosed herein.

Hardware 1404 includes processing circuitry, memory that stores software and/or instructions executable by hardware processing circuitry, and/or other hardware devices as described herein, such as a network interface, input/output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 1406 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 1408a and 1408b (one or more of which may be generally referred to as VMs 1408), and/or perform any of the functions, features and/or benefits described in relation with some embodiments described herein. The virtualization layer 1406 may present a virtual operating platform that appears like networking hardware to the VMs 1408.

The VMs 1408 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 1406. Different embodiments of the instance of a virtual appliance 1402 may be implemented on one or more of VMs 1408, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

In the context of NFV, a VM 1408 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 1408, and that part of hardware 1404 that executes that VM, be it hardware dedicated to that VM and/or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 1408 on top of the hardware 1404 and corresponds to the application 1402.

Hardware 1404 may be implemented in a standalone network node with generic or specific components. Hardware 1404 may implement some functions via virtualization. Alternatively, hardware 1404 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 1410, which, among others, oversees lifecycle management of applications 1402. In some embodiments, hardware 1404 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 1412 which may alternatively be used for communication between hardware nodes and radio units.

Figure 15 shows a communication diagram of a host 1502 communicating via a network node 1504 with a UE 1506 over a partially wireless connection in accordance with some embodiments. Example implementations, in accordance with various embodiments, of the UE (such as a UE 1012a of Figure 10 and/or UE 1100 of Figure 11), network node (such as network node 1010a of Figure 10 and/or network node 1200 of Figure 12), and host (such as host 1016 of Figure 10 and/or host 1300 of Figure 13) discussed in the preceding paragraphs will now be described with reference to Figure 15.

Like host 1300, embodiments of host 1502 include hardware, such as a communication interface, processing circuitry, and memory. The host 1502 also includes software, which is stored in or accessible by the host 1502 and executable by the processing circuitry. The software includes a host application that may be operable to provide a service to a remote user, such as the UE 1506 connecting via an over-the-top (OTT) connection 1550 extending between the UE 1506 and host 1502. In providing the service to the remote user, a host application may provide user data which is transmitted using the OTT connection 1550.

The network node 1504 includes hardware enabling it to communicate with the host 1502 and UE 1506. The connection 1560 may be direct or pass through a core network (like core network 1006 of Figure 10) and/or one or more other intermediate networks, such as one or more public, private, or hosted networks. For example, an intermediate network may be a backbone network or the Internet.

The UE 1506 includes hardware and software, which is stored in or accessible by UE 1506 and executable by the UE’s processing circuitry. The software includes a client application, such as a web browser or operator-specific “app” that may be operable to provide a service to a human or non-human user via UE 1506 with the support of the host 1502. In the host 1502, an executing host application may communicate with the executing client application via the OTT connection 1550 terminating at the UE 1506 and host 1502. In providing the service to the user, the UE's client application may receive request data from the host's host application and provide user data in response to the request data. The OTT connection 1550 may transfer both the request data and the user data. The UE's client application may interact with the user to generate the user data that it provides to the host application through the OTT connection 1550. The OTT connection 1550 may extend via a connection 1560 between the host 1502 and the network node 1504 and via a wireless connection 1570 between the network node 1504 and the UE 1506 to provide the connection between the host 1502 and the UE 1506. The connection 1560 and wireless connection 1570, over which the OTT connection 1550 may be provided, have been drawn abstractly to illustrate the communication between the host 1502 and the UE 1506 via the network node 1504, without explicit reference to any intermediary devices and the precise routing of messages via these devices.

As an example of transmitting data via the OTT connection 1550, in step 1508, the host 1502 provides user data, which may be performed by executing a host application. In some embodiments, the user data is associated with a particular human user interacting with the UE 1506. In other embodiments, the user data is associated with a UE 1506 that shares data with the host 1502 without explicit human interaction. In step 1510, the host 1502 initiates a transmission carrying the user data towards the UE 1506. The host 1502 may initiate the transmission responsive to a request transmitted by the UE 1506. The request may be caused by human interaction with the UE 1506 or by operation of the client application executing on the UE 1506. The transmission may pass via the network node 1504, in accordance with the teachings of the embodiments described throughout this disclosure. Accordingly, in step 1512, the network node 1504 transmits to the UE 1506 the user data that was carried in the transmission that the host 1502 initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In step 1514, the UE 1506 receives the user data carried in the transmission, which may be performed by a client application executed on the UE 1506 associated with the host application executed by the host 1502.

In some examples, the UE 1506 executes a client application which provides user data to the host 1502. The user data may be provided in reaction or response to the data received from the host 1502. Accordingly, in step 1516, the UE 1506 may provide user data, which may be performed by executing the client application. In providing the user data, the client application may further consider user input received from the user via an input/output interface of the UE 1506. Regardless of the specific manner in which the user data was provided, the UE 1506 initiates, in step 1518, transmission of the user data towards the host 1502 via the network node 1504. In step 1520, in accordance with the teachings of the embodiments described throughout this disclosure, the network node 1504 receives user data from the UE 1506 and initiates transmission of the received user data towards the host 1502. In step 1522, the host 1502 receives the user data carried in the transmission initiated by the UE 1506.

One or more of the various embodiments improve the performance of OTT services provided to the UE 1506 using the OTT connection 1550, in which the wireless connection 1570 forms the last segment. More precisely, the teachings of these embodiments may improve the

In an example scenario, factory status information may be collected and analyzed by the host 1502. As another example, the host 1502 may process audio and video data which may have been retrieved from a UE for use in creating maps. As another example, the host 1502 may collect and analyze real-time data to assist in controlling vehicle congestion (e.g., controlling traffic lights). As another example, the host 1502 may store surveillance video uploaded by a UE. As another example, the host 1502 may store or control access to media content such as video, audio, VR or AR which it can broadcast, multicast or unicast to UEs. As other examples, the host 1502 may be used for energy pricing, remote control of non-time critical electrical load to balance power generation needs, location services, presentation services (such as compiling diagrams etc. from data collected from remote devices), or any other function of collecting, retrieving, storing, analyzing and/or transmitting data.

In some examples, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve. There may further be an optional network functionality for reconfiguring the OTT connection 1550 between the host 1502 and UE 1506, in response to variations in the measurement results. The measurement procedure and/or the network functionality for reconfiguring the OTT connection may be implemented in software and hardware of the host 1502 and/or UE 1506. In some embodiments, sensors (not shown) may be deployed in or in association with other devices through which the OTT connection 1550 passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software may compute or estimate the monitored quantities. The reconfiguring of the OTT connection 1550 may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not directly alter the operation of the network node 1504. Such procedures and functionalities may be known and practiced in the art. In certain embodiments, measurements may involve proprietary UE signaling that facilitates measurements of throughput, propagation times, latency and the like, by the host 1502. The measurements may be implemented in that software causes messages to be transmitted, in particular empty or ‘dummy’ messages, using the OTT connection 1550 while monitoring propagation times, errors, etc.

Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and/or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and/or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and/or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and/or by end users and a wireless network generally. Further definitions and embodiments are discussed below.

In the above-description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and/or clarity. The term "and/or" (abbreviated “/”) includes any and all combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements/operations, these elements/operations should not be limited by these terms. These terms are only used to distinguish one element/operation from another element/operation. Thus a first element/operation in some embodiments could be termed a second element/operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id est," may be used to specify a particular item from a more general recitation.

Example embodiments are described herein with reference to block diagrams and/or flowchart illustrations of computer-implemented methods, apparatus (systems and/or devices) and/or computer program products. It is understood that a component of the block diagrams and/or flowchart illustrations, and combinations of components in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and/or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions/acts specified in the block diagrams and/or flowchart block or blocks, and thereby create means (functionality) and/or structure for implementing the functions/acts specified in the block diagrams and/or flowchart block(s).

These computer program instructions may also be stored in a tangible computer- readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions/acts specified in the block diagrams and/or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and/or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

It should also be noted that in some alternate implementations, the functions/acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the components may sometimes be executed in the reverse order, depending upon the functi onality/acts involved. Moreover, the functionality of a given component of the flowcharts and/or block diagrams may be separated into multiple components and/or the functionality of two or more blocks of the flowcharts and/or block diagrams may be at least partially integrated. Finally, other components may be added/inserted between the components that are illustrated, and/or components/operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

EMBODIMENTS

Group A Embodiments

1. A method performed by a user equipment, UE, (300, 1012A, 102 IB, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) comprising: obtaining (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; splitting (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmitting (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receiving (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determining and tracking (609) the 3D position of the one or more keypoints of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; determining (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmitting (613) the changes in object position and pose towards an application.

2. The method of Embodiment 1, wherein the obtained data stream comprises RGBD image frames and wherein determining the 3D position of the one or more key points in an RGBD image frame of the first data stream comprises combining the received 2D position of the one or more keypoints with depth information for respective pixels in the RGBD image frame of the first data stream.

3. The method of Embodiment 1 or Embodiment 2, further comprising: receiving from the at least one remote processing component the 3D coordinates of the one or more key points of the key point refresh in an image frame of the second data stream; and determining the 3D position and pose, in an image frame of the first data stream, of an object associated with the keypoints of the keypoint refresh, based upon the received 3D coordinates of the key points of the keypoint refresh and any said determined change in position or pose of the object.

4. The method of any of Embodiments 1-3, wherein determining the object position and pose based on the one or more key points comprises: comparing (901) positions of the one or more keypoints to previous positions of the one or more keypoints; and responsive to at least 3 keypoints on the object being found, calculating (903) a new 3D position of the object based on a displacement of the at least 3 key points from the previous positions of the at least 3 key points.

5. The method of any of Embodiments 1-4, further comprising: transmitting the second data stream to the at least one remote processing component responsive to determining (701) that a key point refresh condition has been triggered.

6. The method of Embodiment 5, wherein determining that the keypoint refresh condition has been triggered comprises determining (703) that a timer has expired.

7. The method of Embodiment 6, further comprising decreasing a time length of the timer responsive to determining that a change in position or pose of an object has been faster than a normal speed, or that an object has been designated as an object of importance.

8. The method of any of Embodiments 5-7, wherein determining that the keypoint refresh condition has been triggered comprises determining (705) that one or more keypoints have been lost.

9. The method of any of Embodiments 5-8, wherein determining that the keypoint refresh condition has been triggered comprises determining (707) that radio resources are available.

10. The method of any of Embodiments 1-9, further comprising adding a timestamp and/or a sequence number to each one of the RGB image frames and/or the RGBD image frames of the obtained data stream.

11. The method of any of Embodiments 1-10, wherein splitting the data stream comprises transcoding or re-encoding the obtained data stream to generate the second data stream at the second frame rate.

12. The method of any of Embodiments 1-11, further comprising determining the second frame rate according to determined network conditions between the UE and the at least one remote processing component. 13. The method of any of Embodiments 1-12, wherein transmitting (605) the second data stream to at least one remote processing component comprises transmitting the second data stream to at least one base station of a mobile communications network that implements the at least one remote processing component.

Group B Embodiments

14. A user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) configured to: obtain (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; split (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmit (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receive (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determine and tracking (609) the 3D position of the one or more key points of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; determine (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmit (613) the changes in object position and pose towards an application.

15. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Embodiment 14, wherein the obtained data stream comprises RGBD image frames and wherein the the UE (300) is configured to determine the 3D position of the one or more keypoints in an RGBD image frame of the first data stream by combining the received 2D position of the one or more keypoints with depth information for respective pixels in the RGBD image frame of the first data stream. 16. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Embodiment 14 or Embodiment 15, wherein the UE is further configured to: receive from the at least one remote processing component the 3D coordinates of the one or more keypoints of the keypoint refresh in an image frame of the second data stream; and determine the 3D position and pose, in an image frame of the first data stream, of an object associated with the keypoints of the keypoint refresh, based upon the received 3D coordinates of the key points of the keypoint refresh and any said determined change in position or pose of the object.

17. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 15-16, wherein the UE (300) is configured to determine the object position and pose based on the one or more keypoints by: comparing (901) positions of the one or more keypoints to previous positions of the one or more keypoints; and responsive to at least 3 keypoints on the object being found, calculating (903) a new 3D position of the object based on a displacement of the at least 3 key points from the previous positions of the at least 3 key points.

18. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 14-17, wherein the UE is further configured to: transmit the second data stream to the at least one remote processing component responsive to determining (701) that a key point refresh condition has been triggered.

19. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Embodiment 18, wherein the UE (300) is configured to determine that the keypoint refresh condition has been triggered by determining (703) if a timer has expired.

20. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Embodiment 19, wherein the UE is further configured to decrease a time length of the timer responsive to determining that a change in position or pose of an object has been faster than a normal speed, or that an object has been designated as an object of importance.

21. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 18-20, wherein the UE is configured to determine that the keypoint refresh condition has been triggered by determining (705) that one or more keypoints have been lost.

22. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 18-21, wherein the UE is configured to determine if the keypoint refresh condition has been triggered by determining (707) that radio resources are available.

23. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 14-22, wherein the UE is further configured to add a timestamp and/or a sequence number to each one of the image frames of the obtained data stream.

24. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 14-23, wherein the UE is further configured to transmit (605) the second data stream to at least one remote processing component by transmitting the second data stream to at least one base station of a mobile communications network that implements the at least one remote processing component.

25. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Embodiments 14-24 wherein the UE comprises one of a smart phone, a mobile phone, a cell phone, a voice over IP (VoIP) phone, a wireless local loop phone, a desktop computer, a personal digital assistant (PDA), a camera, a gaming console or device, a playback appliance, a wearable terminal device, a wireless endpoint, a mobile station, a tablet, a laptop, a laptop-embedded equipment (LEE), a laptop-mounted equipment (LME), a smart device, a wireless customer-premise equipment (CPE), a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, or an enhanced MTC (eMTC) UE.

27. A user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), comprising: processing circuitry configured to perform any of the steps of any of the Group A embodiments; and power supply circuitry configured to supply power to the processing circuitry.

28. A user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) comprising: an antenna configured to send and receive wireless signals; radio front-end circuitry connected to the antenna and to processing circuitry, and configured to condition signals communicated between the antenna and the processing circuitry; the processing circuitry being configured to perform any of the steps of any of the Group A embodiments; an input interface connected to the processing circuitry and configured to allow input of information into the UE to be processed by the processing circuitry; an output interface connected to the processing circuitry and configured to output information from the UE that has been processed by the processing circuitry; and a battery connected to the processing circuitry and configured to supply power to the UE.

29. A user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) comprising: processing circuitry (303, 1102, 1412); memory (305, 1110) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the UE to perform operations comprising: obtaining (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; splitting (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmitting (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receiving (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determining and tracking (609) the 3D position of the one or more keypoints of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; determining (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmitting (613) the changes in object position and pose towards an application.

30. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Embodiment 29, wherein the memory includes further instructions that when executed by the processing circuitry causes the UE to perform in accordance with any of Embodiments 2-13.

31. A host configured to operate in a communication system to provide an over-the-top (OTT) service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a cellular network for transmission to a user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), wherein the UE comprises a communication interface and processing circuitry, the communication interface and processing circuitry of the UE being configured to perform any of the steps of any of the Group A embodiments to receive the user data from the host.

32. The host of the previous embodiment, wherein the cellular network further includes a network node configured to communicate with the UE to transmit the user data to the UE from the host.

33. The host of the previous 2 embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application.

34. A method implemented by a host operating in a communication system that further includes a network node and a user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), the method comprising: providing user data for the UE; and initiating a transmission carrying the user data to the UE via a cellular network comprising the network node, wherein the UE performs any of the operations of any of the Group A embodiments to receive the user data from the host.

35. The method of the previous embodiment, further comprising: at the host, executing a host application associated with a client application executing on the UE to receive the user data from the UE.

36. The method of the previous embodiment, further comprising: at the host, transmitting input data to the client application executing on the UE, the input data being provided by executing the host application, wherein the user data is provided by the client application in response to the input data from the host application.

37. A host configured to operate in a communication system to provide an over-the-top (OTT) service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a cellular network for transmission to a user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) wherein the UE comprises a communication interface and processing circuitry, the communication interface and processing circuitry of the UE being configured to perform any of the steps of any of the Group A embodiments to transmit the user data to the host.

38. The host of the previous embodiment, wherein the cellular network further includes a network node configured to communicate with the UE to transmit the user data from the UE to the host.

39. The host of the previous 2 embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application.

40. A method implemented by a host configured to operate in a communication system that further includes a network node and a user equipment, UE, (300, 1012A, 102 IB, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), the method comprising: at the host, receiving user data transmitted to the host via the network node by the UE, wherein the UE performs any of the steps of any of the Group A embodiments to transmit the user data to the host.

41. The method of the previous embodiment, further comprising: at the host, executing a host application associated with a client application executing on the UE to receive the user data from the UE.

42. The method of the previous embodiment, further comprising: at the host, transmitting input data to the client application executing on the UE, the input data being provided by executing the host application, wherein the user data is provided by the client application in response to the input data from the host application.

43. A computer program comprising program code to be executed by processing circuitry (303, 1102, 1412) of a communication device (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), whereby execution of the program code causes the communication device (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) to perform operations according to any of embodiments 1-13.

44. A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry (303, 1102, 1412) of a communication device (300, QQ112A, QQ112B, QQ112C, QQ112D, QQ200, QQ504, QQ508A, QQ508B, QQ606), whereby execution of the program code causes the communication device (300, QQ112A, QQ112B, QQ112C, QQ112D, QQ200, QQ504, QQ508A, QQ508B, QQ606) to perform operations according to any of embodiments 1-13.

Claims

48 CLAIMS

2. The method of Claim 1, wherein the obtained data stream comprises RGBD image frames and wherein determining the 3D position of the one or more keypoints in an RGBD image frame of the first data stream comprises combining the received 2D position of the one or more keypoints with depth information for respective pixels in the RGBD image frame of the first data stream.

3. The method of Claim 1 or Claim 2, further comprising: receiving from the at least one remote processing component the 3D coordinates of the one or more key points of the key point refresh in an image frame of the second data 49 stream; and determining the 3D position and pose, in an image frame of the first data stream, of an object associated with the keypoints of the keypoint refresh, based upon the received 3D coordinates of the key points of the keypoint refresh and any said determined change in position or pose of the object.

4. The method of any of Claims 1-3, wherein determining the object position and pose based on the one or more keypoints comprises: comparing (901) positions of the one or more keypoints to previous positions of the one or more keypoints; and responsive to at least 3 keypoints on the object being found, calculating (903) a new 3D position of the object based on a displacement of the at least 3 key points from the previous positions of the at least 3 keypoints.

5. The method of any of Claims 1-4, further comprising: transmitting the second data stream to the at least one remote processing component responsive to determining (701) that a key point refresh condition has been triggered.

6. The method of Claim 5, wherein determining that the key point refresh condition has been triggered comprises determining (703) that a timer has expired.

7. The method of Claim 6, further comprising decreasing a time length of the timer responsive to determining that a change in position or pose of an object has been faster than a normal speed, or that an object has been designated as an object of importance.

8. The method of any of Claims 5-7, wherein determining that the keypoint refresh condition has been triggered comprises determining (705) that one or more keypoints have been lost.

9. The method of any of Claims 5-8, wherein determining that the keypoint refresh condition has been triggered comprises determining (707) that radio resources are available.

10. The method of any of Claims 1-9, further comprising adding a timestamp and/or a 50 sequence number to each one of the RGB image frames and/or the RGBD image frames of the obtained data stream.

11. The method of any of Claims 1-10, wherein splitting the data stream comprises transcoding or re-encoding the obtained data stream to generate the second data stream at the second frame rate.

12. The method of any of Claims 1-11, further comprising determining the second frame rate according to determined network conditions between the UE and the at least one remote processing component.

13. The method of any of Claims 1-12, wherein transmitting (605) the second data stream to at least one remote processing component comprises transmitting the second data stream to at least one base station of a mobile communications network that implements the at least one remote processing component.

14. A user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) configured to: obtain (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; split (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmit (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receive (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determine and tracking (609) the 3D position of the one or more key points of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; 51 determine (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmit (613) the changes in object position and pose towards an application.

15. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Claim 14, wherein the obtained data stream comprises RGBD image frames and wherein the the UE (300) is configured to determine the 3D position of the one or more keypoints in an RGBD image frame of the first data stream by combining the received 2D position of the one or more keypoints with depth information for respective pixels in the RGBD image frame of the first data stream.

16. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Claim 14 or Claim 15, wherein the UE is further configured to: receive from the at least one remote processing component the 3D coordinates of the one or more keypoints of the keypoint refresh in an image frame of the second data stream; and determine the 3D position and pose, in an image frame of the first data stream, of an object associated with the keypoints of the keypoint refresh, based upon the received 3D coordinates of the key points of the keypoint refresh and any said determined change in position or pose of the object.

17. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 15-16, wherein the UE (300) is configured to determine the object position and pose based on the one or more keypoints by: comparing (901) positions of the one or more keypoints to previous positions of the one or more keypoints; and responsive to at least 3 keypoints on the object being found, calculating (903) a new 3D position of the object based on a displacement of the at least 3 key points from the previous positions of the at least 3 key points.

18. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 14-17, wherein the UE is further configured to: transmit the second data stream to the at least one remote processing component responsive to determining (701) that a key point refresh condition has been triggered.

19. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Claim 18, wherein the UE (300) is configured to determine that the keypoint refresh condition has been triggered by determining (703) if a timer has expired.

20. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of Claim 19, wherein the UE is further configured to decrease a time length of the timer responsive to determining that a change in position or pose of an object has been faster than a normal speed, or that an object has been designated as an object of importance.

21. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 18-20, wherein the UE is configured to determine that the keypoint refresh condition has been triggered by determining (705) that one or more keypoints have been lost.

22. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 18-21, wherein the UE is configured to determine if the key point refresh condition has been triggered by determining (707) that radio resources are available.

23. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 14-22, wherein the UE is further configured to add a timestamp and/or a sequence number to each one of the image frames of the obtained data stream.

24. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 14-23, wherein the UE is further configured to transmit (605) the second data stream to at least one remote processing component by transmitting the second data stream to at least one base station of a mobile communications network that implements the at least one remote processing component.

25. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of any of Claims 14-24 wherein the UE comprises one of a smart phone, a mobile phone, a cell phone, a voice over IP (VoIP) phone, a wireless local loop phone, a desktop computer, a personal digital assistant (PDA), a camera, a gaming console or device, a playback appliance, a wearable terminal device, a wireless endpoint, a mobile station, a tablet, a laptop, a laptop- embedded equipment (LEE), a laptop-mounted equipment (LME), a smart device, a wireless customer-premise equipment (CPE), a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, or an enhanced MTC (eMTC) UE.

26. A user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) comprising: processing circuitry (303, 1102, 1412); memory (305, 1110) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the UE to perform operations comprising: obtaining (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; splitting (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmitting (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receiving (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determining and tracking (609) the 3D position of the one or more keypoints of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; determining (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmitting (613) the changes in object position and pose towards an application.

27. The UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) of 54

Claim 26, wherein the memory includes further instructions that when executed by the processing circuitry causes the UE to perform in accordance with any of Claims 2-13.

28. A computer program comprising program code to be executed by processing circuitry (303, 1102, 1412) of a user equipment, UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), whereby execution of the program code causes the UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) to: obtain (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; split (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmit (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receive (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determine and tracking (609) the 3D position of the one or more key points of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; determine (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmit (613) the changes in object position and pose towards an application.

29. The computer program of Claim 28 comprising further program code to be executed by the processing circuitry (303, 1102, 1412), whereby execution of the further program code causes the UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) to perform in accordance with any of Claims 2-13.

30. A computer program product comprising a non-transitory storage medium including 55 program code to be executed by processing circuitry (303, 1102, 1412) of a user equipment, UE, (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506), whereby execution of the program code causes the UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) to: obtain (601) a data stream having one of red-green-blue, RGB, image frames and/or RGB depth, RGBD image frames; split (603) the data stream into a first data stream at a first frame rate and a second data stream at a second frame rate where the first frame rate is higher than the second frame rate; transmit (605) the second data stream to at least one remote processing component configured for object detection, determination of object location and pose, and for selection of keypoints associated with the object detected in an image frame of the second data stream; receive (607) a keypoint refresh from the at least one remote processing component, the keypoint refresh comprising a 2D position of one or more keypoints associated with an object detected in one of the image frames in the second data stream and an indication of a respective region of interest of the one or more keypoints; determine and tracking (609) the 3D position of the one or more key points of the keypoint refresh in the image frames in the first data stream based on the respective indicated region of interest; determine (611) a change in object position and pose in each image frame in the first data stream based on the determined and tracked 3D position of the one or more associated keypoints in the respective image frame; and transmit (613) the changes in object position and pose towards an application.

31. The computer program product of Claim 30 wherein the non-transitory storage medium includes further program code to be executed by the processing circuitry (303, 1102, 1412), whereby execution of the program code causes the UE (300, 1012A, 1021B, 1012C, 1012D, 1100, 1404, 1408A, 1408B, 1506) to perform operations according to any of Claims 2-13.