EP4659198A1

EP4659198A1 - System and apparatus suitable for facilitating object reconstruction of at least one object associated with an image, and a processing method in association thereto

Info

Publication number: EP4659198A1
Application number: EP24700967.3A
Authority: EP
Inventors: Mo SHAN
Original assignee: Aumovio Germany GmbH
Current assignee: Aumovio Germany GmbH
Priority date: 2023-02-03
Filing date: 2024-01-16
Publication date: 2025-12-10
Also published as: GB2626778A; GB2626778A8; WO2024160527A1

Abstract

There is provided a processing method suitable for facilitating object reconstruction of at least one object associated with an image. The processing method can include an initializing step, an inference step and an optimizing step. The initializing step includes an encoding step wherein a shared latent code can be generated and a decoding step wherein the shared latent code can be decoded to Signed Distance Function (SDF) values and 3 Dimensional (3D) semantic keypoints. The inference step includes regressing the 3D semantic keypoints based on obtained point cloud data, and deriving initial pose associated with the object based on a reference pose, the regressed 3D semantic keypoints and obtained 2D (2 dimensional) semantic keypoints. In regard to the optimizing step, the initial pose can be optimized based on an error parameter derivable in association with the SDF values and another error parameter derivable in association with the 3D semantic keypoints.

Description

SYSTEM AND APPARATUS SUITABLE FOR FACILITATING OBJECT RECONSTRUCTION OF AT LEAST ONE OBJECT ASSOCIATED WITH AN IMAGE, AND A PROCESSING METHOD IN ASSOCIATION THERETO

Field Of Invention

The present disclosure generally relates to one or both of a system and an apparatus suitable for facilitating object reconstruction of at least one object associated with an image. The present disclosure further relates a processing method which can be associated with the system and/or the apparatus.

Background

Signed Distance Function (SDF) can generally be used in connection with objectcentric reconstruction as SDF can enable high quality shape representation from partial point cloud observations.

SDF related techniques can possibly be improved by manner of Neural Networks. For example, neural networks can possibly be utilized in connection with SDF (e.g., regress SDF value(s)).

The present disclosure contemplates that there is still need for improvement of SDF- related technique(s).

Summary of the Invention

In accordance with an aspect of the disclosure, there is provided an apparatus which can, for example, include an embedding part configurable to encode a shared latent code, in accordance with an embodiment of the disclosure.

The apparatus can, for example, further include a keypoint decoder which can be configured to decode the shared latent code to 3 Dimensional (3D) semantic keypoints, in accordance with an embodiment of the disclosure. The apparatus can, for example, yet further include a signed distance function (SDF) decoder which can be configured to decode the shared latent code to SDF values, in accordance with an embodiment of the disclosure.

In one embodiment, the apparatus can, for example, be configured to perform one or more processing tasks in association with one or both of inference-based processing (e.g., in association with deriving an initial pose) and optimization-based processing (i.e. , in association with optimizing the derived initial pose).

In accordance with an embodiment of the disclosure, it is contemplated that the embedding part can, for example, be configured to encode the shared latent code in association with an encoding step of a processing method, as will be discussed later in further detail in accordance with an aspect of the disclosure. Furthermore, in one embodiment, the keypoint decoder can, for example, be configured to decode the shared latent code to 3 Dimensional (3D) semantic keypoints in association a decoding step (302b) of a processing method, as will be discussed later in further detail in accordance with an aspect of the disclosure. Moreover, in one embodiment, the signed distance function (SDF) decoder can, for example, be configured to decode the shared latent code to SDF values in association with a decoding step of a processing method, as will be discussed later in further detail in accordance with an aspect of the disclosure.

It is contemplated that in the above manner in the above manner, it is contemplated that at least a robust way for initialization can be provided/facilitated, in accordance with an embodiment of the disclosure. Moreover, optimization in association with, for example, derived initial pose can possibly be facilitated, in accordance with an embodiment of the disclosure.

The above-described advantageous aspect(s) of the apparatus of the present disclosure can also apply analogously (all) the aspect(s) of a below described processing method of the present disclosure. Likewise, all below described advantageous aspect(s) of the processing method of the disclosure can also apply analogously (all) the aspect(s) of above described apparatus of the disclosure. In accordance with an aspect of the disclosure, there is provided a processing method which can, for example, be suitable for facilitating object reconstruction of at least one object associated with an image, in accordance with an embodiment of the disclosure.

The processing method can, for example, include an initializing step, an inference step and an optimizing step, in accordance with an embodiment of the disclosure.

The initializing step can, for example, include an encoding step and a decoding step. In regard to the encoding step, a shared latent code can, for example, be generated. In regard to the decoding step, the shared latent code can, for example, be capable of being decoded to Signed Distance Function (SDF) values and 3 Dimensional (3D) semantic keypoints.

The inference step can, for example, include regressing the 3D semantic keypoints based on obtained point cloud data, and deriving initial pose associated with the object based on a reference pose, the regressed 3D semantic keypoints and obtained 2D (2 dimensional) semantic keypoints.

In regard to the optimizing step, the initial pose can, for example, be capable of being optimized based on an error parameter derivable in association with the SDF values and another error parameter derivable in association with the 3D semantic keypoints.

It is contemplated that in the above manner, it is contemplated that at least a robust way for initialization can be provided/facilitated, in accordance with an embodiment of the disclosure. Moreover, optimization in association with, for example, derived initial pose can possibly be facilitated, in accordance with an embodiment of the disclosure.

The present disclosure further contemplates a computer program which can include instructions which, when the program is executed by a computer, cause the computer to carry out the initializing step, the inference step and/or the optimization step as discussed with reference to the processing method.

The present disclosure yet further contemplates a computer readable storage medium (not shown) having data stored therein representing software executable by a computer, the software including instructions, when executed by the computer, to carry out the initializing step, the inference step and/or the optimization step as discussed with reference to the processing method.

Brief Description of the Drawings

Embodiments of the disclosure are described hereinafter with reference to the following drawings, in which:

Fig. 1 shows a system which can include at least one apparatus, according to an embodiment of the disclosure;

Fig, 2 shows the apparatus of Fig, 1 in further detail, according to an embodiment of the disclosure; and

Fig. 3 shows a processing method in association with the system of Fig. 1 , according to an embodiment of the disclosure.

Detailed Description

The present disclosure contemplates that aside shape, there could possibly be need for optimization of pose to reconstruct one or more objects in a scene. Conventional Signed Distance Function (SDF) techniques would usually initialize and then optimize the pose based on multi-view measurements. The present disclosure contemplates that conventional SDF possibly lack a robust way for initialization as due to reliance on heuristics.

The present disclosure contemplates that for object pose initialization, one or more objects can be represented by manner of a dual latent space that can possibly encode the SDF and/or 3D (three-dimensional) semantic keypoints. Groundtruth point clouds and/or 3D keypoints can be provided for training. For example, groundtruth point clouds can be provided for training in association with SDF loss and 3D keypoints can be provided for training in association with keypoint loss. During inference, multi-view RGB-D (Red Green Blue-Depth) data/measurements can be taken/used as inputs, and the optimized pose and shape of the object(s) in the scene can be output.

In one example, the present disclosure contemplates the use of one or more latent embedding spaces and two decoders. Shared latent code can be decoded to both 3D semantic keypoints and SDF values. The 3D keypoints can be used to provide an initial estimate of the pose. It is contemplated that corresponding 2D keypoints from images can be obtained and PnP (Perspective-n-Point) based solution(s)/techniques(s) (e.g., PnP camera pose estimation) can be used to determine the pose.

In the above manner, it is contemplated that at least a robust way for initialization can be provided/facilitated.

The foregoing will be discussed in further detail with reference to Fig. 1 to Fig. 3 hereinafter.

Referring to Fig. 1 , a system 100 is shown, according to an embodiment of the disclosure. The system 100 can, for example, be suitable for facilitating data-driven optimization. Optimization can, for example, be in association with directly regressing & optimizing shape and pose of one or more objects. For example, the system 100 can be suitable for facilitating object reconstruction of at least one object associated with an image, in accordance with an embodiment of the disclosure.

Moreover, the system 100 can be associated with the utilization of key point(s) (e.g., 3D keypoints) and SDF. The system 100 can, for example, be facilitate for use in association with technology areas in relation to, for example, autonomous driving, robotics and/or augmented reality, in accordance with an embodiment of the disclosure. As shown, the system 100 can include one or more apparatuses 102, at least one device 104 and, optionally, a communication network 106, in accordance with an embodiment of the disclosure.

The apparatus(es) 102 can be coupled to the device(s) 104. Specifically, the apparatus(es) 102 can, for example, be coupled to the device(s) 104 via the communication network 106c, in accordance with an embodiment of the disclosure.

In one embodiment, the apparatus(es) 102 can be coupled to the communication network 106 and the device(s) 104 can be coupled to the communication network 106. Coupling can be by manner of one or both of wired coupling and wireless coupling. The apparatus(es) 102 can, in general, be configured to communicate with the device(s) 104 via the communication network 106, according to an embodiment of the disclosure.

Generally, in accordance with an embodiment of the disclosure, the apparatus(es) 102 can be configured to receive one or more input signals and process the input signal(s) to generate/derive one or more output signals. Moreover, in accordance with an embodiment of the disclosure, the device(s) 104 can, for example, be configured to one or both of generate the input signal(s) and communicate the input signal(s) to the apparatus(es) 102.

The apparatus(es) 102 can, for example, be configured to process the input signal(s) by manner of any one of optimization-based processing, inference-based processing, encoding-based processing and decoding-based processing, or any combination thereof (i.e., optimization-based processing, inference-based processing, encodingbased processing and/or decoding-based processing) to generate/derive the output signal(s), in accordance with an embodiment of the disclosure. The apparatus(es) 102 will be discussed in further detail with reference to Fig. 2, in accordance with an embodiment of the disclosure The device(s) 104 can, for example, be configured to generate the input signal(s) and/or communicate the input signal(s), in accordance with an embodiment of the disclosure. For example, the input signal(s) can be communicated from the device(s) 104 to the apparatus(es) 102. In one example, a device 104 can be associated with/correspond to/include one or more image capturing devices such as RGB-D sensors (e.g., a RGB-D camera) and the input signal(s) can be associated with/correspond to/include one or both of at least one image and RGB-D data/measurement(s), in accordance with an embodiment of the disclosure. In another example, a device 104 can be associated with/correspond to/include one or more databases/networks (e.g., KeypointNet, OcclusionNet) which can be capable of carrying datasets and/or capable of prediction/learning (e.g., a framework to predict 2D and 3D locations of occluded keypoints for objects). In yet another example, the device(s) 104 can be associated with/correspond to/include a combination of image capturing device(s) and database(s)/network(s). In this regard, the input signal(s) can, for example, include/be associated with/correspond to any one of captured image(s) (e.g., which can include RGB-D data/measurement(s), dataset(s) and predicted data, or any combination of (i.e., RGB-D data/measurement(s), dataset(s) and/or predicted data), in accordance with an embodiment of the disclosure.

The communication network 106 can, for example, correspond to an Internet communication network, a wired-based communication network, a wireless-based communication network, or any combination thereof. Communication (i.e., between the apparatus(es) 102 and the device(s) 104) via the communication network 106 can be by manner of one or both of wired communication and wireless communication.

The aforementioned apparatus(es) 102 will be discussed in further detail with reference to Fig. 2 hereinafter.

Referring to Fig. 2, an apparatus 102 is shown in further detail in the context of an example implementation 200, according to an embodiment of the disclosure. In the example implementation 200, the apparatus 102 can correspond to an electronic module 200a which can, for example, be capable of performing one or more processing tasks in association with optimization-based processing, inferencebased processing, encoding-based processing and/or decoding-based processing, in accordance with an embodiment of the disclosure.

The electronic module 200a can, for example, include a casing 200b. Moreover, the electronic module 200a can, for example, carry any one of a first module 202, a second module 204, a third module 206, or any combination thereof.

In one embodiment, the electronic module 200a can carry a first module 202, a second module 204 and/or a third module 206. In a specific example, the electronic module 200a can carry a first module 202, a second module 204 and a third module 206, in accordance with an embodiment of the disclosure.

In this regard, it is appreciable that, in one embodiment, the casing 200b can be shaped and dimensioned to carry any one of the first module 202, the second module 204 and the third module 206, or any combination thereof.

The first module 202 can be coupled to one or both of the second module 204 and the third module 206. The second module 204 can be coupled to one or both of the first module 202 and the third module 206. The third module 206 can be coupled to one or both of the first module 202 and the second module 204. In one example, the first module 202 can be coupled to the second module 204 and the second module 204 can be coupled to the third module 206, in accordance with an embodiment of the disclosure. Coupling between the first module 202, the second module 204 and/or the third module 206 can, for example, be by manner of one or both of wired coupling and wireless coupling. Each of the first module 202, the second module 204 and the third module 206 can correspond to one or both of a hardware-based module and a software-based module, according to an embodiment of the disclosure.

In one example, the first module 202 can correspond to a hardware-based receiver which can be configured to receive one or more input signals. The second module 204 can, for example, correspond to a network-based/software- based and/or hardware-based processing module which can be configured to perform one or more processing tasks in association with any one of, or any combination of, the following:

• optimization-based processing

• inference-based processing

• encoding-based processing

• decoding-based processing

Specifically, the second module 204 can, for example, be configured to process the received input signal(s) by manner of optimization-based processing, inferencebased processing, encoding-based processing and/or decoding-based processing in a manner so as to generate/derive one or more output signal(s), in accordance with an embodiment of the disclosure.

The third module 206 can, in one example, correspond to a hardware-based transmitter which can be configured to communicate the output signal(s) from the electronic module 200a, in accordance with an embodiment of the disclosure. The output signal(s) can, for example, be communicated from the electronic module 200a to one or more devices 104 and/or one or more other apparatuses 102, in accordance with an embodiment of the disclosure.

The present disclosure contemplates the possibility that the first and second modules 202/204 can be an integrated software-hardware based module (e.g., an electronic part which can carry a software program/algorithm in association with receiving and processing functions/an electronic module programmed to perform the functions of receiving and processing). The present disclosure further contemplates the possibility that the first and third modules 202/206 can be an integrated softwarehardware based module (e.g., an electronic part which can carry a software program/algorithm in association with receiving and transmitting functions/an electronic module programmed to perform the functions of receiving and transmitting). The present disclosure yet further contemplates the possibility that the first and third modules 202/206 can be an integrated hardware module (e.g., a hardware-based transceiver) capable of performing the functions of receiving and transmitting.

The above example implementation 200 will now be discussed in further detail with reference to an example scenario, in accordance with an embodiment of the disclosure hereinafter.

In the example scenario, the second module 204 can, for example, include any one of an embedding part (e.g., an embedding module carrying/associable with a dual latent embedding space) 204a, one decoder (e.g., a keypoint decoder) 204b and another decoder (e.g., a Signed Distance Function (SDF) decoder) 204c, or any combination thereof. For example, the second module 204 can include an embedding part 204a, a keypoint decoder 204b and a SDF decoder 204c. One or both of the keypoint decoder 204b and the SDF decoder 204c can, for example, be coupled to the embedding part 204a, in accordance with an embodiment of the disclosure. For example, each of the keypoint decoder 204b and the SDF decoder 204c can be coupled to the embedding part 204a, in accordance with an embodiment of the disclosure.

Moreover, the keypoint decoder 204b can, for example, correspond to/be representative of/be indicative of a keypoint network (fe) and the SDF decoder 204c can, for example, correspond to/be representative of/be indicative of a SDF network (ge), in accordance with an embodiment of the disclosure. The decoders (i.e., the keypoint decoder 204b and the SDF decoder 204c) can, for example, share a joint embedding space (e.g., also referable to as “a dual latent embedding space”), which can be represented by Zj e R64, for each object category (j), in accordance with an embodiment of the disclosure. Generally, in one embodiment, given Zj and input point cloud (P_c), fe can, for example, decode to Nj semantic keypoints in 3D canonical space, where the number Nj can be category dependent. Further generally, in one embodiment, given Zj and P_c, g<t> can, for example, decode to corresponding SDF values for P_c. Specifically, in the example scenario, a dataset with the object meshes and 3D keypoint annotations can be provided for reconstructing pose and shape of one or more objects from one or more multi-view RGB-D observations (e.g., RGB-D data can be indicative of/associable with/correspond to/include the multi-view RGB-D observation(s)). It is contemplated that KeypointNet (e.g., Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13647-13656, 2020) can, for example, provide such a dataset using, for example, object models in ShapeNet (e.g., Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012, 2015), in accordance with an embodiment of the disclosure.

Moreover, in the example scenario, object pose can be initialized by manner of a dual latent embedding space Zj (i.e., an embedding part 204a) and two decoders (i.e., the keypoint decoder 204b and the SDF decoder 204c). The dual latent embedding space Zj can encode (by manner of, for example, encoding-based processing) both SDF and 3D semantic keypoints simultaneously. In regard to the two decoders, the keypoint network fe and the SDF network g can decode (by manner of, for example, decoding-based processing) the shared latent code to both 3D semantic keypoints and SDF values, as follows:

• for the given Zj and input point cloud P_c, fe can decode to Nj semantic keypoints in 3D canonical space, where the number Nj is category dependent; and

• for the given Zj and P_c, go can decode to the corresponding SDF values for P_c.

Furthermore, in the example scenario, at least one processing task associated with optimized-based processing can be performed by manner of training using groundtruth point clouds for SDF loss and training using 3D keypoints for keypoint loss to (jointly) optimize the keypoint network fe and the SDF network go..

Additionally, in the example scenario, at least one processing task associated with inference-based processing can be performed for inferring multi-view RGB-D data/measurement(s) as inputs, and at least one optimized initial pose (e.g., which can include pose and shape) associated with at least one object in a scene (e.g., based on a captured image) can be output. In this regard, the output signal(s) can, for example, correspond to.be/be associated with/include optimized initial pose, in accordance with an embodiment of the disclosure. Moreover, 3D keypoints can be regressed using 3D measurements (feeding P_o,) and a latent space can be used to represent both 3D keypoints and SDF. 2D semantic keypoints K_pcan be detected in association with the images. Given Ks, K_p and the object pose wT₀, initial pose of an object can be recovered using EPnP.

In this regard, it is appreciable that processing task(s) associated with inferencebased processing can be associated with/include any one of, or any combination of: a) Obtaining object pose (coT₀) denoting the transformation in Lie group from the object frame to the world frame b) Obtaining point cloud observation (P_o) from multi-view RGB-D dataZmeasurement(s) and instance segmentation c) Detecting 2D (two-dimensional) semantic keypoints (K_p) in association with the image(s) (captured by an image capturing device such as a camera) d) Regressing the 3D semantic keypoints e) Recovering/deriving initial pose of an object using EPnP (Efficient Perspective-n-Point) based on regressed 3D semantic keypoints (Ks), 2D semantic keypoints (K_p) and a reference pose (e.g., a known camera pose, coTc, which can be the known camera pose of an image capturing device).

After obtaining the initial pose, the initial pose (e.g., shape and pose) can be (further) optimized (i.e., in association with optimized-based processing) based on error parameters such as the SDF residual (E_s) and the reprojection residual (E_r). Specifically, SDF residual (E_s) and the reprojection residual (E_r) can be estimated and combined to derive a final residual (E) parameter. The shape and pose can be optimized through stochastic gradient descent by manner of calculating the Jacobians of E with respect to Zj and coT₀. As mentioned earlier, the output signal(s) can, for example, correspond to.be/be associated with/include optimized initial pose, in accordance with an embodiment of the disclosure. In one specific example, in association with inference, object pose coT₀ e SE(3) can be obtained and transformation in Lie Group from the object frame to the world frame can be denoted. Point cloud observation P_o from multi-view RGB-D data/measurement(s) and instance segmentation can be obtained, and P_o can be fed to fe to regress the 3D semantic keypoints Ks e R3xNj. 2D semantic keypoints Kp e R2xNj can be detected on the images, for example, using OcclusionNet (e.g., Occlusion-net: 2d/3d occluded keypoint localization using graph networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7326-7335, 2019.) for cars. With Ks, Kp and the known object pose wT₀, initial pose of the car can be recovered using EPnP (e.g., Epnp: An accurate o (n) solution to the pnp problem. International journal of computer vision, 81 (2): 155-166, 2009). After obtaining the initial pose, shape and pose can be further optimized based on the one or more error parameters (e.g., a first error parameter and a second error parameter).

The first error parameter can, for example, be a SDF residual (E_s) which can be defined by: where X e R⁴ can be a point in P_o with homogeneous coordinates. coT₀’¹X can be the transformed point in object frame with inhomogeneous coordinates. Since X lies on the object surface, its real SDF value can be zero, while the estimated SDF value can be g<t>(Zj , coT₀’¹X).

The second error parameter can, for example, be a reprojection residual (E_r) which can be defined by: where fe(Zj) can be the regressed semantic keypoint in an object frame, and the keypoint can be transformed to a camera frame via cTcocoT₀ ^_1fe(Zj), and projected on an image plane using, for example, camera intrinsics via K(cTco x coT₀’¹fe(Zj)). The product “x” can refer to matrix multiplication, p can be the detected semantic keypoint on an image.

The error parameters (i.e., the first and second error parameters) can be combined to derive a final residual (E) parameter as follows:

E = (JUSES + COrEr where E_s can be a measurement/indication of consistency between the output and observed SDF values, whereas Er can be a measurement/indication of discrepancy between predicted object pose and 2D observations. Jacobians of E can be calculated/derived with respect to Zj and the object pose coT₀ to optimize the shape and pose through stochastic gradient descent. co_s and co_r can be weights to balance the residual(s).

In view of the foregoing, it is appreciable that the present disclosure generally contemplates an apparatus 102 (e.g., corresponding to an electronic module 200a) which can, for example, include an embedding part 204a configurable to encode a shared latent code, in accordance with an embodiment of the disclosure. The apparatus 102 can, for example, further include a keypoint decoder 204b which can be configured to decode the shared latent code to 3 Dimensional (3D) semantic keypoints, in accordance with an embodiment of the disclosure. The apparatus 102 can, for example, yet further include a signed distance function (SDF) decoder 204c which can be configured to decode the shared latent code to SDF values, in accordance with an embodiment of the disclosure.

In one embodiment, the apparatus 102 can, for example, be configured to perform one or more processing tasks in association with one or both of inference-based processing (e.g., in association with deriving an initial pose) and optimization-based processing (e.g., in association with optimizing the derived initial pose). In the above manner, it is contemplated that at least a robust way for initialization can be provided/facilitated, in accordance with an embodiment of the disclosure. Moreover, optimization in association with, for example, derived initial pose can possibly be facilitated, in accordance with an embodiment of the disclosure.

The above-described advantageous aspect(s) of the apparatus 102 of the present disclosure can also apply analogously (all) the aspect(s) of a below described processing method of the present disclosure. Likewise, all below described advantageous aspect(s) of the processing method of the disclosure can also apply analogously (all) the aspect(s) of above described apparatus 102 of the disclosure. It is to be appreciated that these remarks apply analogously to the earlier discussed system 100 of the present disclosure.

Referring to Fig. 3, a processing method in association with the system 100 is shown, according to an embodiment of the disclosure. The processing method 300 can, for example, be suitable for facilitating object reconstruction of at least one object associated with an image, in accordance with an embodiment of the disclosure.

The processing method 300 can, for example, include any one of an initializing step 302, an inference step 304 and an optimizing step 306, or any combination thereof, in accordance with an embodiment of the disclosure.

In one embodiment, the processing method 300 can include an initializing step 302, an inference step 304 and an optimizing step 306. In another embodiment, the processing method 300 can include an initializing step 302 and an inference step 304. In yet another embodiment, the processing method 300 can include an inference step 304 and an optimizing step 306. In yet a further embodiment, the processing method 300 can include one of an initializing step 302, an inference step 304 and an optimizing step 306 (i.e. , an initializing step 302, an inference step 304 or an optimizing step 306). With regard to the initializing step 302, one or more processing tasks in association with one or both of encoding-based processing and decoding-based processing can be performed, in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to perform the processing task(s) in association with the encoding-based processing and/or decoding-based processing, in accordance with an embodiment of the disclosure. In one specific example, the initializing step 302 can include one or both of an encoding step 302a (e.g., in association with encoding-based processing) and a decoding step 302b (e.g., in association with decoding-based processing), in accordance with an embodiment of the disclosure.

With regard to the inference step 304, one or more processing tasks in association with inference-based processing can be performed, in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to perform the processing task(s) in association with the inference-based processing, in accordance with an embodiment of the disclosure.

With regard to the optimization step 306, one or more processing tasks in association with optimization-based processing can be performed, in accordance with an embodiment of the disclosure. For example, the apparatus(es) 102 can be configured to perform the processing task(s) in association with the optimizationbased processing, in accordance with an embodiment of the disclosure.

The present disclosure further contemplates a computer program (not shown) which can include instructions which, when the program is executed by a computer (not shown), cause the computer to carry out the initializing step 302, the inference step 304 and/or the optimization step 306 as discussed with reference to the processing method 300.

The present disclosure yet further contemplates a computer readable storage medium (not shown) having data stored therein representing software executable by a computer (not shown), the software including instructions, when executed by the computer, to carry out the initializing step 302, the inference step 304 and/or the optimization step 306 as discussed with reference to the processing method 300.

In view of the foregoing, it is appreciable that the present disclosure generally contemplates a processing method 300 which can, for example, be suitable for facilitating object reconstruction of at least one object associated with an image, in accordance with an embodiment of the disclosure.

The processing method 300 can, for example, include an initializing step 302, an inference step 304 and an optimizing step 306, in accordance with an embodiment of the disclosure.

The initializing step 302 can, for example, include an encoding step 302a and a decoding step 302b. In regard to the encoding step 302a, a shared latent code can, for example, be generated. In regard to the decoding step 302, the shared latent code can, for example, be capable of being decoded to Signed Distance Function (SDF) values and 3 Dimensional (3D) semantic keypoints. The encoding step 302a can, for example, be associated with/based on the earlier mentioned encodingbased processing, in accordance with an embodiment of the disclosure. The decoding step 302b can, for example, be associated with/based on the earlier mentioned decoding-based processing, in accordance with an embodiment of the disclosure.

The inference step 304 can, for example, include regressing the 3D semantic keypoints based on obtained point cloud data, and deriving initial pose associated with the object based on a reference pose, the regressed 3D semantic keypoints and obtained 2D (2 dimensional) semantic keypoints. The inference step 304 can, for example, be associated with/based on the earlier mentioned inference-based processing, in accordance with an embodiment of the disclosure.

In regard to the optimizing step 306, the initial pose can, for example, be capable of being optimized based on an error parameter derivable in association with the SDF values and another error parameter derivable in association with the 3D semantic keypoints. The optimizing step 306 can, for example, be associated with/based on the aforementioned optimization-based processing, in accordance with an embodiment of the disclosure.

In one embodiment, the initial pose can be associated with shape and pose of the object. The optimizing step 306 can, for example, include optimizing shape and pose of the object.

In one embodiment, the shared latent code can correspond to a joint embedding space for each object category.

In one embodiment, the shared latent code can be capable of being decoded to SDF values based on the obtained point cloud data.

In one embodiment, the shared latent code can be capable of being decoded to the 3D semantic keypoints based on the obtained point cloud data.

In one embodiment, the SDF values can be capable of being optimized by manner of training using groundtruth point cloud data in association with SDF loss.

In one embodiment, the 3D semantic keypoints can be capable of being optimized by manner of training using 3D keypoints for keypoint loss.

In one embodiment, point cloud data can be capable of being obtained from multiview RGB-D (Red Green Blue - Depth) data communicable from an image capturing device such as a camera.

In one embodiment, point cloud data can be based on measurements and instance segmentation in association with the multi-view RGB-D data.

In one embodiment, the 2D semantic keypoints can be capable of being obtained by manner of detection of 2D semantic keypoints on images captured by the image capturing device via a framework such as OcclusionNet. It is contemplated that OcclusionNet can, for example, be capable of predicting 2D and 3D locations of occluded keypoints for objects.

In one embodiment, the error parameter derivable in association with the SDF values corresponds to a first error parameter and the another error parameter derivable in association with the 3D semantic keypoints corresponds to a second error parameter. Moreover, in one embodiment, the first and second error parameters can be capable of being combined to derive a combined error parameter. For example, the first error parameter can correspond to SDF residual, the second error parameter can correspond to reprojection residual, and the combined error parameter can correspond to final residual. In a specific example, the final residual can be based on a combination of the SDF residual and the reprojection residual.

In the above manner, it is contemplated that at least a robust way for initialization can be provided/facilitated, in accordance with an embodiment of the disclosure. Moreover, optimization in association with, for example, derived initial pose can possibly be facilitated, in accordance with an embodiment of the disclosure.

It should be appreciated that the embodiments described above can be combined in any manner as appropriate (e.g., one or more embodiments as discussed in the “Detailed Description” section can be combined with one or more embodiments as described in the “Summary of the Invention” section).

It should be further appreciated by the person skilled in the art that variations and combinations of embodiments described above, not being alternatives or substitutes, may be combined to form yet further embodiments.

In one example, the communication network 106 can be omitted. Communication (i.e., between the apparatus(es) 102 and the device(s) 104) can be by manner of direct coupling. Such direct coupling can be by manner of one or both of wired coupling and wireless coupling. In another example, the system 100/one or more portions of the system 100 can be integrated into existing perception system software stack of autonomous driving and/or robotics and/or augmented reality to reconstruct shape and pose of one or more objects in a scene.

In the foregoing manner, various embodiments of the disclosure are described for addressing at least one of the foregoing disadvantages. Such embodiments are intended to be encompassed by the following claims, and are not to be limited to specific forms or arrangements of parts so described and it will be apparent to one skilled in the art in view of this disclosure that numerous changes and/or modification can be made, which are also intended to be encompassed by the following claims.

Claims

Claim(s)

1 . A processing method (300) suitable for facilitating object reconstruction of at least one object associated with an image, the processing method (300) comprising: an initializing step (302) including: an encoding step (302a) to generate a shared latent code; a decoding step (302b) wherein the shared latent code is capable of being decoded to Signed Distance Function (SDF) values and 3 Dimensional (3D) semantic keypoints; an inference step (304) including: regressing the 3D semantic keypoints based on obtained point cloud data, and deriving initial pose associated with the object based on a reference pose, the regressed 3D semantic keypoints and obtained 2D (2 dimensional) semantic keypoints; and an optimizing step (306) wherein the initial pose is capable of being optimized based on an error parameter derivable in association with the SDF values and another error parameter derivable in association with the 3D semantic keypoints.

2. The processing method (300) of claim 1 , wherein the initial pose is associable with shape and pose of the object, and wherein the optimizing step (306) includes optimizing shape and pose of the object.

3. The processing method (300) of any of the preceding claims, wherein the shared latent code corresponds to a joint embedding space for each object category.

4. The processing method (300) of any of the preceding claims, wherein the shared latent code is capable of being decoded to SDF values based on the obtained point cloud data.

5. The processing method (300) of any of the preceding claims, wherein the shared latent code is capable of being decoded to the 3D semantic keypoints based on the obtained point cloud data.

6. The processing method (300) of any of the preceding claims, the SDF values capable of being optimized by manner of training using groundtruth point cloud data in association with SDF loss.

7. The processing method (300) of any of the preceding claims, the 3D semantic keypoints capable of being optimized by manner of training using 3D keypoints for keypoint loss.

8. The processing method (300) of any of the preceding claims, point cloud data capable of being obtained from multi-view RGB-D (Red Green Blue - Depth) data communicable from an image capturing device such as a camera.

9. The processing method (300) of any of the preceding claims, point cloud data being based on measurements and instance segmentation in association with the multi-view RGB-D data.

10. The processing method (300) of any of the preceding claims, the 2D semantic keypoints capable of being obtained by manner of detection of 2D semantic keypoints on images captured by the image capturing device via a framework, such as OcclusionNet, capable of predicting 2D and 3D locations of occluded keypoints for objects.

11 . The processing method (300) of any of the preceding claims, wherein the error parameter derivable in association with the SDF values corresponds to a first error parameter and the another error parameter derivable in association with the 3D semantic keypoints corresponds to a second error parameter, and wherein the first and second error parameters are capable of being combined to derive a combined error parameter.

12. The processing method (300) of any of the preceding claims, wherein the first error parameter corresponds to SDF residual, the second error parameter corresponds to reprojection residual, and the combined error parameter corresponds to final residual and wherein the final residual is based on a combination of the SDF residual and the reprojection residual.

13. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out at least one of the initializing step (302), the inference step (304) and the optimizing step (306) according to the processing method (300) of any of the preceding claims.

14. A computer readable storage medium having data stored therein representing software executable by a computer, the software including instructions, when executed by the computer, to carry out at least one of the initializing step (302), the inference step (304) and the optimizing step (306) according to the processing method (300) of any of the preceding claims.

15. An apparatus (102) comprising: an embedding part (204a) configurable to encode a shared latent code in association with the encoding step (302a) of the processing method (300) of any of the preceding claims; a keypoint decoder (204b) configurable to decode the shared latent code to 3 Dimensional (3D) semantic keypoints in association with the decoding step (302b) of the processing method (300) of any of the preceding claims; and a signed distance function (SDF) decoder (204c) configurable to decode the shared latent code to SDF values in association with the decoding step (302b) of the processing method (300) any of the preceding claims.