WO2023091129A1 - Localisation de caméra sur la base d'un plan - Google Patents

Localisation de caméra sur la base d'un plan Download PDF

Info

Publication number
WO2023091129A1
WO2023091129A1 PCT/US2021/059638 US2021059638W WO2023091129A1 WO 2023091129 A1 WO2023091129 A1 WO 2023091129A1 US 2021059638 W US2021059638 W US 2021059638W WO 2023091129 A1 WO2023091129 A1 WO 2023091129A1
Authority
WO
WIPO (PCT)
Prior art keywords
plane
query
candidate
image
training
Prior art date
Application number
PCT/US2021/059638
Other languages
English (en)
Inventor
Pan JI
Jiachen Liu
Yi Xu
Original Assignee
Innopeak Technology, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Innopeak Technology, Inc. filed Critical Innopeak Technology, Inc.
Priority to PCT/US2021/059638 priority Critical patent/WO2023091129A1/fr
Publication of WO2023091129A1 publication Critical patent/WO2023091129A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements

Definitions

  • This application relates generally to data processing technology including, but not limited to, methods, systems, and non-transitory computer-readable media for localizing a camera in an environment of an extended reality application that is executed by an electronic system.
  • SLAM Simultaneous localization and mapping
  • VR virtual reality
  • AR augmented reality
  • autonomous driving and navigation.
  • high frequency pose estimation is enabled by sensor fusion.
  • Asynchronous time warping is often applied with SLAM in an AR system to warp a query image before it is sent to a display to correct for head movement that occurs after the query image is rendered.
  • relevant image data and inertial sensor data are synchronized and used for localizing a camera in a scene, e.g., estimating and predicting camera poses.
  • the same query image is optionally used as background images to render virtual objects according to the camera poses.
  • the camera pose corresponding to the query image is determined from one or more previously-recorded images of the scene which share one or more sparse features with the query image.
  • the camera poses are not consistently accurate across the scene when camera localization depends on the sparse features. It would be beneficial to have a more accurate and efficient camera localization mechanism than the current practice.
  • Various embodiments of this application are directed to a camera localization process that utilizes planar information and semantic information of a query image to retrieve one or more candidate images that has similar camera poses with the query image from a database.
  • Image features of the query image and the candidate images in the database are augmented with semantic and planar features to form global descriptors, which are applied for distance metric learning and retrieval of the one or more candidate images.
  • the one or more candidate images is selected as being substantially similar to the query image based on semantic and structural features.
  • the selected candidate image(s) are used to enable place recognition and camera localization, and a query camera pose of the query image is determined based on a camera pose of the selected candidate image(s).
  • a camera localization method is implemented at an electronic system.
  • the method includes obtaining a query image, a candidate image, and information of a candidate camera pose at which the candidate image is captured in a scene.
  • the method further includes generating a plurality of query plane embeddings corresponding to a plurality of query planes in the query image, obtaining a plurality of candidate plane embeddings corresponding to a plurality of candidate planes of the candidate image, and comparing the plurality of query plane embeddings and the plurality of candidate plane embeddings to identify a first number of matching plane pairs.
  • Each matching plane pair includes a respective query plane of the query image and a respective candidate plane of the candidate image that substantially matches the respective query plane.
  • the method further includes based on the first number of matching plane pairs, iteratively determining that the query image is captured at a query camera pose related to the candidate camera pose of the candidate image.
  • a method is implemented at an electronic system to identify a candidate image.
  • the method includes obtaining a query image in a scene and generating a query feature map, a query plane instance feature, and a query plane semantic feature from the query image.
  • the query plane instance feature identifies one or more bounding boxes defining one or more planes in the query image, and the query plane semantic feature includes semantic information of the one or more planes.
  • the method further includes aggregating the query feature map, query plane instance feature, and query plane semantic feature to a plane-assisted global descriptor.
  • the method further includes for each of a plurality of images in a database, obtaining a respective global image descriptor, and determining a respective distance of the plane-assisted global descriptor and the respective global image descriptor.
  • the method further includes selecting a candidate image from the plurality of images based on the respective distance corresponding to each of the plurality of images.
  • some implementations include an electronic system that includes one or more processors and memory having instructions stored thereon, which when executed by the one or more processors cause the processors to perform any of the above methods.
  • some implementations include a non-transitory computer-readable medium, having instructions stored thereon, which when executed by one or more processors cause the processors to perform any of the above methods.
  • Figure 1 is an example data processing environment having one or more servers communicatively coupled to one or more client devices, in accordance with some embodiments.
  • FIG. 2 is a block diagram illustrating an electronic system configured to process content data (e.g., image data), in accordance with some embodiments.
  • content data e.g., image data
  • Figure 3 is an example data processing environment for training and applying a neural network-based data processing model for processing visual and/or audio data, in accordance with some embodiments.
  • Figure 4A is an example neural network applied to process content data in an NN-based data processing model, in accordance with some embodiments
  • Figure 4B is an example node in the neural network, in accordance with some embodiments.
  • Figure 5A is a flowchart of a process for processing inertial sensor data and image data of an electronic system using a SLAM module, in accordance with some embodiments.
  • Figure 5B is a flow diagram of a camera localization process that determines a query camera pose corresponding to a query image, in accordance with some embodiments.
  • Figure 6 is a flow diagram of an image retrieval process that retrieves one or more candidate images from a database for a query image, in accordance with some embodiments.
  • Figure 7 is a flow diagram of a training process that trains an image retrieval model applied in an image retrieval process, in accordance with some embodiments.
  • Figure 8A is a flow diagram of a camera localization process that determines a query camera pose associated with a query image based on a candidate camera pose associated with a candidate image, in accordance with some embodiments.
  • Figure 8B is a flow diagram of an iterative camera pose process that determines a query camera pose based on matching plane pairs of a query image and candidate image, in accordance with some embodiments.
  • Figure 8C is a flow diagram of another iterative camera pose process that determines a query camera pose based on matching plane pairs of a query image and candidate image, in accordance with some embodiments.
  • Figure 9 is a flow diagram of a training process that trains a camera localization model applied in a camera localization process, in accordance with some embodiments.
  • Figure 10A is a flow diagram of an image retrieval method, in accordance with some embodiments
  • Figure 10B is a flow diagram of a camera localization method, in accordance with some embodiments.
  • the camera localization pipeline includes three main functional blocks: (i) image retrieval, (ii) plane retrieval, and (iii) plane Perspective-n-Points (PnP).
  • image retrieval techniques are applied to find one or more candidate images from a database based on a query image.
  • An example image retrieval method augments image features with semantic and planar features to form global image descriptors.
  • the augmented global descriptors lead to a discriminative capability for distance metric learning and improve image retrieval performance.
  • a plane retrieval network is trained to get information of matching planes in the query and candidate images.
  • a plane PnP method is applied to final a camera pose of the query image based on the information of matching planes in the query and candidate images.
  • Augmented reality enhances images that are viewed on a screen or other display and overlays computer-generated images, sounds, or other data on a real world environment.
  • Mixed reality overlays and anchors virtual objects to real world and allows a user to interact with the virtual or real objects in a field of view.
  • Semantic segmentation is applied to assign a semantic label (e.g., floor, wall, table) to each pixel of an image.
  • Plane detection generates plane masks on a two-dimensional (2D) image.
  • a plane mask is an image of the same size as an input image, and in an example, each pixel in the plane mask is assigned with a plane identifier (ID) if the respective pixel belongs to a corresponding plane region.
  • ID plane identifier
  • a query image corresponds to a portion of a scene that has been scanned and mapped to a three-dimensional (3D) virtual space.
  • a 6-Degree-of-Freedom (6-DoF) camera pose is estimated for the query image with respect to the scene, and the 6-DoF camera pose optionally includes a position and an orientation of a camera in the scene when the query image is captured.
  • Distance metric learning also called metric learning
  • FIG. 1 is an example data processing environment 100 having one or more servers 102 communicatively coupled to one or more client devices 104, in accordance with some embodiments.
  • the one or more client devices 104 may be, for example, desktop computers 104A, tablet computers 104B, mobile phones 104C, head-mounted display (HMD) (also called augmented reality (AR) glasses) 104D, or intelligent, multi-sensing, network- connected home devices (e.g., a surveillance camera 104E, a smart television device, a drone).
  • HMD head-mounted display
  • AR augmented reality
  • Each client device 104 can collect data or user inputs, executes user applications, and present outputs on its user interface.
  • the collected data or user inputs can be processed locally at the client device 104 and/or remotely by the server(s) 102.
  • the one or more servers 102 provide system data (e.g., boot files, operating system images, and user applications) to the client devices 104, and in some embodiments, processes the data and user inputs received from the client device(s) 104 when the user applications are executed on the client devices 104.
  • the data processing environment 100 further includes a storage 106 for storing data related to the servers 102, client devices 104, and applications executed on the client devices 104.
  • the one or more servers 102 are configured to enable real-time data communication with the client devices 104 that are remote from each other or from the one or more servers 102. Further, in some embodiments, the one or more servers 102 are configured to implement data processing tasks that cannot be or are preferably not completed locally by the client devices 104.
  • the client devices 104 include a game console (e.g., the HMD 104D) that executes an interactive online gaming application.
  • the game console receives a user instruction and sends it to a game server 102 with user data.
  • the game server 102 generates a stream of video data based on the user instruction and user data and providing the stream of video data for display on the game console and other client devices that are engaged in the same game session with the game console.
  • the client devices 104 include a networked surveillance camera 104E and a mobile phone 104C.
  • the networked surveillance camera 104E collects video data and streams the video data to a surveillance camera server 102 in real time. While the video data is optionally pre-processed on the surveillance camera 104E, the surveillance camera server 102 processes the video data to identify motion or audio events in the video data and share information of these events with the mobile phone 104C, thereby allowing a user of the mobile phone 104 to monitor the events occurring near the networked surveillance camera 104E in the real time and remotely.
  • the one or more servers 102, one or more client devices 104, and storage 106 are communicatively coupled to each other via one or more communication networks 108, which are the medium used to provide communications links between these devices and computers connected together within the data processing environment 100.
  • the one or more communication networks 108 may include connections, such as wire, wireless communication links, or fiber optic cables. Examples of the one or more communication networks 108 include local area networks (LAN), wide area networks (WAN) such as the Internet, or a combination thereof.
  • the one or more communication networks 108 are, optionally, implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol.
  • a connection to the one or more communication networks 108 may be established either directly (e.g., using 3G/4G connectivity to a wireless carrier), or through a network interface 110 (e.g., a router, switch, gateway, hub, or an intelligent, dedicated whole-home control node), or through any combination thereof.
  • a network interface 110 e.g., a router, switch, gateway, hub, or an intelligent, dedicated whole-home control node
  • the one or more communication networks 108 can represent the Internet of a worldwide collection of networks and gateways that use the Transmission Control Protocol/Intemet Protocol (TCP/IP) suite of protocols to communicate with one another.
  • TCP/IP Transmission Control Protocol/Intemet Protocol
  • At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, consisting of thousands of commercial, governmental, educational and other computer systems that route data and messages.
  • deep learning techniques are applied in the data processing environment 100 to process content data (e.g., video data, visual data, audio data) obtained by an application executed at a client device 104 to identify information contained in the content data, match the content data with other data, categorize the content data, or synthesize related content data.
  • the content data may broadly include inertial sensor data captured by inertial sensor(s) of a client device 104.
  • data processing models are created based on one or more neural networks to process the content data. These data processing models are trained with training data before they are applied to process the content data. Subsequently to model training, the mobile phone 104C or HMD 104D obtains the content data (e.g., captures video data via an internal camera) and processes the content data using the data processing models locally.
  • both model training and data processing are implemented locally at each individual client device 104 (e.g., the mobile phone 104C and HMD 104D).
  • the client device 104 obtains the training data from the one or more servers 102 or storage 106 and applies the training data to train the data processing models.
  • both model training and data processing are implemented remotely at a server 102 (e.g., the server 102A) associated with a client device 104 (e.g. the client device 104A and HMD 104D).
  • the server 102A obtains the training data from itself, another server 102 or the storage 106 applies the training data to train the data processing models.
  • the client device 104 obtains the content data, sends the content data to the server 102 A (e.g., in an application) for data processing using the trained data processing models, receives data processing results (e.g., recognized hand gestures) from the server 102A, presents the results on a user interface (e.g., associated with the application), renders virtual objects in a field of view based on the poses, or implements some other functions based on the results.
  • the client device 104 itself implements no or little data processing on the content data prior to sending them to the server 102 A.
  • data processing is implemented locally at a client device 104 (e.g., the client device 104B and HMD 104D), while model training is implemented remotely at a server 102 (e.g., the server 102B) associated with the client device 104.
  • the server 102B obtains the training data from itself, another server 102 or the storage 106 and applies the training data to train the data processing models.
  • the trained data processing models are optionally stored in the server 102B or storage 106.
  • the client device 104 imports the trained data processing models from the server 102B or storage 106, processes the content data using the data processing models, and generates data processing results to be presented on a user interface or used to initiate some functions (e.g., rendering virtual objects based on device poses) locally.
  • a pair of AR glasses 104D are communicatively coupled in the data processing environment 100.
  • the AR glasses 104D includes a camera, a microphone, a speaker, one or more inertial sensors (e.g., gyroscope, accelerometer), and a display.
  • the camera and microphone are configured to capture video and audio data from a scene of the AR glasses 104D, while the one or more inertial sensors are configured to capture inertial sensor data.
  • the camera captures hand gestures of a user wearing the AR glasses 104D, and recognizes the hand gestures locally and in real time using a two-stage hand gesture recognition model.
  • the microphone records ambient sound, including user’s voice commands.
  • both video or static visual data captured by the camera and the inertial sensor data measured by the one or more inertial sensors are applied to determine and predict device poses.
  • the video, static image, audio, or inertial sensor data captured by the AR glasses 104D is processed by the AR glasses 104D, server(s) 102, or both to recognize the device poses.
  • deep learning techniques are applied by the server(s) 102 and AR glasses 104D jointly to recognize and predict the device poses.
  • the device poses are used to control the AR glasses 104D itself or interact with an application (e.g., a gaming application) executed by the AR glasses 104D.
  • the display of the AR glasses 104D displays a user interface, and the recognized or predicted device poses are used to render or interact with user selectable display items (e.g., an avatar) on the user interface.
  • deep learning techniques are applied in the data processing environment 100 to process video data, static image data, or inertial sensor data captured by the AR glasses 104D.
  • 2D or 3D device poses are recognized and predicted based on such video, static image, and/or inertial sensor data using a first data processing model.
  • Visual content is optionally generated using a second data processing model.
  • Training of the first and second data processing models is optionally implemented by the server 102 or AR glasses 104D.
  • Inference of the device poses and visual content is implemented by each of the server 102 and AR glasses 104D independently or by both of the server 102 and AR glasses 104D jointly.
  • FIG 2 is a block diagram illustrating an electronic system 200 configured to process content data (e.g., image data), in accordance with some embodiments.
  • the electronic system 200 includes a server 102, a client device 104 (e.g., AR glasses 104D in Figure 1), a storage 106, or a combination thereof.
  • the electronic system 200 typically, includes one or more processing units (CPUs) 202, one or more network interfaces 204, memory 206, and one or more communication buses 208 for interconnecting these components (sometimes called a chipset).
  • the electronic system 200 includes one or more input devices 210 that facilitate user input, such as a keyboard, a mouse, a voice-command input unit or microphone, a touch screen display, a touch-sensitive input pad, a gesture capturing camera, or other input buttons or controls.
  • the client device 104 of the electronic system 200 uses a microphone for voice recognition or a camera 260 for gesture recognition to supplement or replace the keyboard.
  • the client device 104 includes one or more optical cameras 260 (e.g., an RGB camera), scanners, or photo sensor units for capturing images, for example, of graphic serial codes printed on the electronic devices.
  • the electronic system 200 also includes one or more output devices 212 that enable presentation of user interfaces and display content, including one or more speakers and/or one or more visual displays.
  • the client device 104 includes a location detection device, such as a GPS (global positioning system) or other geolocation receiver, for determining the location of the client device 104.
  • GPS global positioning system
  • the client device 104 includes an inertial measurement unit (IMU) 280 integrating sensor data captured by multi-axes inertial sensors to provide estimation of a location and an orientation of the client device 104 in space.
  • IMU inertial measurement unit
  • the one or more inertial sensors of the IMU 280 include, but are not limited to, a gyroscope, an accelerometer, a magnetometer, and an inclinometer.
  • Memory 206 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. Memory 206, optionally, includes one or more storage devices remotely located from one or more processing units 202. Memory 206, or alternatively the non-volatile memory within memory 206, includes a non-transitory computer readable storage medium. In some embodiments, memory 206, or the non- transitory computer readable storage medium of memory 206, stores the following programs, modules, and data structures, or a subset or superset thereof:
  • Operating system 214 including procedures for handling various basic system services and for performing hardware dependent tasks
  • Network communication module 216 for connecting each server 102 or client device 104 to other devices (e.g., server 102, client device 104, or storage 106) via one or more network interfaces 204 (wired or wireless) and one or more communication networks 108, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on;
  • User interface module 218 for enabling presentation of information (e.g., a graphical user interface for application(s) 224, widgets, websites and web pages thereof, and/or games, audio and/or video content, text, etc.) at each client device 104 via one or more output devices 212 (e.g., displays, speakers, etc.);
  • information e.g., a graphical user interface for application(s) 224, widgets, websites and web pages thereof, and/or games, audio and/or video content, text, etc.
  • output devices 212 e.g., displays, speakers, etc.
  • Input processing module 220 for detecting one or more user inputs or interactions from one of the one or more input devices 210 and interpreting the detected input or interaction;
  • Web browser module 222 for navigating, requesting (e.g., via HTTP), and displaying websites and web pages thereof, including a web interface for logging into a user account associated with a client device 104 or another electronic device, controlling the client or electronic device if associated with the user account, and editing and reviewing settings and data that are associated with the user account;
  • One or more user applications 224 for execution by the electronic system 200 e.g., games, social network applications, smart home applications, and/or other web or non-web based applications for controlling another electronic device and reviewing data captured by such devices;
  • Model training module 226 for receiving training data and establishing a data processing model for processing content data (e.g., video, image, audio, or textual data) to be collected or obtained by a client device 104;
  • content data e.g., video, image, audio, or textual data
  • Data processing module 228 for processing content data using data processing models 250, thereby identifying information contained in the content data, matching the content data with other data, categorizing the content data, or synthesizing related content data, where in some embodiments, the data processing module 228 is associated with one of the user applications 224 to process the content data in response to a user instruction received from the user application 224, and in an example, the data processing module 228 is applied to implement an image retrieval process 600 in Figure 6 or a camera localization process 800 in Figure 8A;
  • Pose determination and prediction module 230 for determining and predicting a pose of the client device 104 (e.g., AR glasses 104D), where in some embodiments, the pose is determined and predicted jointly by the pose determination and prediction module 230 and data processing module 228, and the module 230 further includes an SLAM module 232 for mapping a scene where a client device 104 is located and identifying a pose of the client device 104 within the scene using image and IMU sensor data, a camera localization module 234 for determining a query camera pose associated with a query image based on one or more candidate images stored in a database, and an image retrieval module 236 for extracting one or more candidate images from a database to match a query image for camera localization;
  • SLAM module 232 for mapping a scene where a client device 104 is located and identifying a pose of the client device 104 within the scene using image and IMU sensor data
  • a camera localization module 234 for determining a query camera pose associated with a query image based on one or
  • Pose-based rendering module 238 for rendering virtual objects on top of a field of view of the camera 260 of the client device 104 or creating mixed reality content using images captured by the camera 260;
  • the one or more databases 240 are stored in one of the server 102, client device 104, and storage 106 of the electronic system 200 .
  • the one or more databases 240 are distributed in more than one of the server 102, client device 104, and storage 106 of the electronic system 200 .
  • more than one copy of the above data is stored at distinct devices, e.g., two copies of the data processing models 250 are stored at the server 102 and storage 106, respectively.
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 206 optionally, stores a subset of the modules and data structures identified above.
  • memory 206 optionally, stores additional modules and data structures not described above.
  • FIG. 3 is an example data processing system 300 for training and applying a neural network based (NN-based) data processing model 250 for processing content data (e.g., video, image, audio, or textual data), in accordance with some embodiments.
  • the data processing system 300 includes a model training module 226 for establishing the data processing model 250 and a data processing module 228 for processing the content data using the data processing model 250.
  • both of the model training module 226 and the data processing module 228 are located on a client device 104 of the data processing system 300, while a training data source 304 distinct form the client device 104 provides training data 306 to the client device 104.
  • the training data source 304 is optionally a server 102 or storage 106.
  • both of the model training module 226 and the data processing module 228 are located on a server 102 of the data processing system 300.
  • the training data source 304 providing the training data 306 is optionally the server 102 itself, another server 102, or the storage 106.
  • the model training module 226 and the data processing module 228 are separately located on a server 102 and client device 104, and the server 102 provides the trained data processing model 250 to the client device 104.
  • the model training module 226 includes one or more data pre-processing modules 308, a model training engine 310, and a loss control module 312.
  • the data processing model 250 is trained according to a type of the content data to be processed.
  • the training data 306 is consistent with the type of the content data, so is a data pre-processing module 308 applied to process the training data 306 consistent with the type of the content data.
  • an image pre-processing module 308A is configured to process image training data 306 to a predefined image format, e.g., extract a region of interest (ROI) in each training image, and crop each training image to a predefined image size.
  • ROI region of interest
  • an audio pre-processing module 308B is configured to process audio training data 306 to a predefined audio format, e.g., converting each training sequence to a frequency domain using a Fourier transform.
  • the model training engine 310 receives pre-processed training data provided by the data pre-processing modules 308, further processes the pre-processed training data using an existing data processing model 250, and generates an output from each training data item.
  • the loss control module 312 can monitor a loss function comparing the output associated with the respective training data item and a ground truth of the respective training data item.
  • the model training engine 310 modifies the data processing model 250 to reduce the loss function, until the loss function satisfies a loss criteria (e.g., a comparison result of the loss function is minimized or reduced below a loss threshold).
  • the modified data processing model 250 is provided to the data processing module 228 to process the content data.
  • the model training module 226 offers supervised learning in which the training data is entirely labelled and includes a desired output for each training data item (also called the ground truth in some situations). Conversely, in some embodiments, the model training module 226 offers unsupervised learning in which the training data are not labelled. The model training module 226 is configured to identify previously undetected patterns in the training data without pre-existing labels and with no or little human supervision. Additionally, in some embodiments, the model training module 226 offers partially supervised learning in which the training data are partially labelled.
  • the data processing module 228 includes a data pre-processing modules 314, a model -based processing module 316, and a data post-processing module 318.
  • the data preprocessing modules 314 pre-processes the content data based on the type of the content data. Functions of the data pre-processing modules 314 are consistent with those of the preprocessing modules 308 and covert the content data to a predefined content format that is acceptable by inputs of the model -based processing module 316. Examples of the content data include one or more of: video, image, audio, textual, and other types of data.
  • each image is pre-processed to extract an ROI or cropped to a predefined image size
  • an audio clip is pre-processed to convert to a frequency domain using a Fourier transform.
  • the content data includes two or more types, e.g., video data and textual data.
  • the model -based processing module 316 applies the trained data processing model 250 provided by the model training module 226 to process the pre-processed content data.
  • the model -based processing module 316 can also monitor an error indicator to determine whether the content data has been properly processed in the data processing model 250.
  • the processed content data is further processed by the data postprocessing module 318 to present the processed content data in a preferred format or to provide other related information that can be derived from the processed content data.
  • Figure 4A is an example neural network (NN) 400 applied to process content data in an NN-based data processing model 250, in accordance with some embodiments
  • Figure 4B is an example node 420 in the neural network (NN) 400, in accordance with some embodiments.
  • the data processing model 250 is established based on the neural network 400.
  • a corresponding model-based processing module 316 applies the data processing model 250 including the neural network 400 to process content data that has been converted to a predefined content format.
  • the neural network 400 includes a collection of nodes 420 that are connected by links 412. Each node 420 receives one or more node inputs and applies a propagation function to generate a node output from the one or more node inputs.
  • the node output is provided via one or more links 412 to one or more other nodes 420
  • a weight w associated with each link 412 is applied to the node output.
  • the one or more node inputs are combined based on corresponding weights wi, W2, W3, and W4 according to the propagation function.
  • the propagation function is a product of a non-linear activation function and a linear weighted combination of the one or more node inputs.
  • the collection of nodes 420 is organized into one or more layers in the neural network 400.
  • the one or more layers includes a single layer acting as both an input layer and an output layer.
  • the one or more layers includes an input layer 402 for receiving inputs, an output layer 406 for providing outputs, and zero or more hidden layers 404 (e.g., 404A and 404B) between the input and output layers 402 and 406.
  • a deep neural network has more than one hidden layers 404 between the input and output layers 402 and 406. In the neural network 400, each layer is only connected with its immediately preceding and/or immediately following layer.
  • a layer 402 or 404B is a fully connected layer because each node 420 in the layer 402 or 404B is connected to every node 420 in its immediately following layer.
  • one of the one or more hidden layers 404 includes two or more nodes that are connected to the same node in its immediately following layer for down sampling or pooling the nodes 420 between these two layers.
  • max pooling uses a maximum value of the two or more nodes in the layer 404B for generating the node of the immediately following layer 406 connected to the two or more nodes.
  • a convolutional neural network is applied in a data processing model 250 to process content data (particularly, video and image data).
  • the CNN employs convolution operations and belongs to a class of deep neural networks 400, i.e., a feedforward neural network that only moves data forward from the input layer 402 through the hidden layers to the output layer 406.
  • the one or more hidden layers of the CNN are convolutional layers convolving with a multiplication or dot product.
  • Each node in a convolutional layer receives inputs from a receptive area associated with a previous layer (e.g., five nodes), and the receptive area is smaller than the entire previous layer and may vary based on a location of the convolution layer in the convolutional neural network.
  • Video or image data is pre-processed to a predefined video/image format corresponding to the inputs of the CNN.
  • the pre-processed video or image data is abstracted by each layer of the CNN to a respective feature map.
  • a recurrent neural network is applied in the data processing model 250 to process content data (particularly, textual and audio data). Nodes in successive layers of the RNN follow a temporal sequence, such that the RNN exhibits a temporal dynamic behavior.
  • each node 420 of the RNN has a time-varying real-valued activation.
  • the RNN examples include, but are not limited to, a long short-term memory (LSTM) network, a fully recurrent network, an Elman network, a Jordan network, a Hopfield network, a bidirectional associative memory (BAM network), an echo state network, an independently RNN (IndRNN), a recursive neural network, and a neural history compressor.
  • LSTM long short-term memory
  • BAM bidirectional associative memory
  • an echo state network an independently RNN (IndRNN)
  • a recursive neural network a recursive neural network
  • a neural history compressor examples include, but are not limited to, a long short-term memory (LSTM) network, a fully recurrent network, an Elman network, a Jordan network, a Hopfield network, a bidirectional associative memory (BAM network), an echo state network, an independently RNN (IndRNN), a recursive neural network, and a neural history compressor.
  • the RNN can be used for hand
  • the training process is a process for calibrating all of the weights w, for each layer of the learning model using a training data set which is provided in the input layer 402.
  • the training process typically includes two steps, forward propagation and backward propagation, which are repeated multiple times until a predefined convergence condition is satisfied.
  • forward propagation the set of weights for different layers are applied to the input data and intermediate results from the previous layers.
  • backward propagation a margin of error of the output (e.g., a loss function) is measured, and the weights are adjusted accordingly to decrease the error.
  • the activation function is optionally linear, rectified linear unit, sigmoid, hyperbolic tangent, or of other types.
  • a network bias term b is added to the sum of the weighted outputs from the previous layer before the activation function is applied.
  • the network bias b provides a perturbation that helps the NN 400 avoid over fitting the training data.
  • the result of the training includes the network bias parameter b for each layer.
  • FIG. 5A is a flowchart of a process 500 for processing inertial sensor data and image data of an electronic system (e.g., a server 102, a client device 104, or a combination of both) using a SLAM module 232, in accordance with some embodiments.
  • the process 500 includes measurement preprocessing 502, initialization 504, local visual- inertial odometry (VIO) with relocation 506, and global pose graph optimization 508.
  • VIO local visual- inertial odometry
  • an RGB camera 260 captures image data of a scene at an image frame rate (e.g., 30 FPS), and features are detected and tracked (510) from the image data.
  • An IMU 280 measures inertial sensor data at a sampling frequency (e.g., 1000 Hz) concurrently with the RGB camera 260 capturing the image data, and the inertial sensor data are pre-integrated (512) to provide data of a variation of device poses 540.
  • a sampling frequency e.g. 1000 Hz
  • the image data captured by the RGB camera 260 and the inertial sensor data measured by the IMU 280 are temporally aligned (514).
  • a vision-only structure from motion (SfM) techniques 514 are applied (516) to couple the image data and inertial sensor data, estimate three-dimensional structures, and map the scene of the RGB camera 260.
  • a sliding window 518 and associated states from a loop closure 520 are used to optimize (522) a VIO.
  • the VIO corresponds (524) to a keyframe of a smooth video transition and a corresponding loop is detected (526)
  • features are retrieved (528) and used to generate the associated states from the loop closure 520.
  • a multi-degree-of-freedom (multi- DOF) pose graph is optimized (530) based on the states from the loop closure 520, and a keyframe database 532 is updated with the keyframe associated with the VIO.
  • the features that are detected and tracked (510) are used to monitor (534) motion of an object in the image data and estimate image-based poses 536, e.g., according to the image frame rate.
  • the inertial sensor data that are pre-integrated (512) may be propagated (538) based on the motion of the object and used to estimate inertial-based poses 540, e.g., according to a sampling frequency of the IMU 280.
  • the image-based poses 536 and the inertial-based poses 540 are stored in the pose data database 252 and used by the module 230 to estimate and predict poses that are used by the pose-based rendering module 238.
  • the SLAM module 232 receives the inertial sensor data measured by the IMU 280 and obtains image-based poses 536 to estimate and predict more poses 540 that are further used by the pose-based rendering module 238.
  • SLAM high frequency pose estimation is enabled by sensor fusion, which relies on data synchronization between imaging sensors and the IMU 280.
  • the imaging sensors e.g., the RGB camera 260, a LiDAR scanner
  • the IMU 280 can measure inertial sensor data and operate at a very high frequency (e.g., 1000 samples per second) and with a negligible latency (e.g., ⁇ 0.1 millisecond).
  • Asynchronous time warping is often applied in an AR system to warp an image before it is sent to a display to correct for head movement and pose variation that occurs after the image is rendered.
  • ATW algorithms reduce a latency of the image, increase or maintain a frame rate, or reduce judders caused by missing image frames.
  • relevant image data and inertial sensor data are stored locally, such that they can be synchronized and used for pose estimation/predication.
  • the image and inertial sensor data are stored in one of multiple STL containers, e.g., std::vector, std::queue, std: :list, etc., or other selfdefined containers. These containers are generally convenient for use.
  • the image and inertial sensor data are stored in the STL containers with their timestamps, and the timestamps are used for data search, data insertion, and data organization.
  • FIG. 5B is a flow diagram of a camera localization process 550 that determines a query camera pose 552 corresponding to a query image 554, in accordance with some embodiments.
  • the camera localization process 550 includes an image retrieval stage 556, a plane retrieval stage 558, a plane perspective-n-point (PnP) stage 560.
  • an electronic system 200 retrieves one or more candidate images 562 from a database 564, and the one or more candidate images 562 substantially match the query image 554.
  • query planes and candidate planes are retrieved from the query image 554 and each candidate image 562, respectively.
  • the query planes of the query image 554 and the candidate planes of each candidate image 562 are matched to one another, if a first number of matching plane pairs are identified in the query image 554 and the respective candidate image 562.
  • an image in the database 564 is qualified and identified as a candidate image 562 in accordance with a determination that the first number is greater than a predefined matching pair number threshold (e.g., 3), i.e., in accordance with a determination that there are more than the predefined threshold of matching plane pairs in the query image 554 and each candidate image 562.
  • plane embeddings are determined for the query image 554 or candidate image 562 and applied to identify the corresponding matching plane pairs thereof.
  • the electronic system 200 iteratively determines that the query image 554 is captured at a query camera pose 552 related to a known candidate camera pose of the candidate image 562. For example, for each iteration (e.g., iteration 812 in Figure 8B), a second number of matching plane pairs (e.g., plane pairs 806’ in Figure 8B) are selected in the first number of matching plane pairs (e.g., plane pairs 806 in Figure 8A).
  • a second number of matching plane pairs e.g., plane pairs 806’ in Figure 8B
  • the query camera pose 552 is determined to be aligned with and equal to the candidate camera pose of the candidate image 562.
  • an iteration fitting error e.g., error 816 in Figure 8B is determined between the query planes and candidate planes in the second number of matching plane pairs.
  • a smallest fitting error (e.g., error 818 in Figure 8B) is identified among the iteration fitting error of each of the plurality of iterations, and used to determine a relative pose change (e.g., pose change 820-1 in Figure 8B) between the query camera pose and the candidate camera pose in the respective iteration associated with the smallest fitting error.
  • the query camera pose 552 is determined based on the candidate camera pose of the candidate image and the relative pose change.
  • an iteration fitting error (e.g., error 816 in Figure 8C) is determined between the query planes and candidate planes in the second number of matching plane pairs, and compared with a threshold error (e.g., threshold error 838 in Figure 8C).
  • a threshold error e.g., threshold error 838 in Figure 8C.
  • the plurality of iterations are terminated, and a relative pose change (e.g., pose change 820-2 in Figure 8C) is determined between the query camera pose and the candidate camera pose and used to determine the query camera pose. More details on the plane PnP stage 560 are explained below with respect to Figures 8A-8C.
  • FIG 6 is a flow diagram of an image retrieval process 556 that retrieves one or more candidate images 562 from a database 564 for a query image 554, in accordance with some embodiments.
  • the query image 554 is captured in a scene by a camera (e.g., a camera 260 in Figure 2).
  • a camera e.g., a camera 260 in Figure 2.
  • an electronic device e.g., a mobile phone 104C
  • an electronic device e.g., a mobile phone 104C
  • the electronic device does not include the camera, and is configured to receive the query image 554 via one or more communication networks 108 and retrieve the one or more candidate images 562 from the database 564 for the query image 554.
  • the electronic device generates a query feature map 602, a query plane instance feature 604, and a query plane semantic feature 606 from the query image 554.
  • the query plane instance feature 604 identifies one or more bounding boxes defining one or more query planes in the query image 554, and the query plane semantic feature 606 includes semantic information of the one or more query planes of the query image 554.
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are aggregated to a plane-assisted global descriptor 608.
  • the one or more bounding boxes associated with the query plane(s) of the query image 554 are applied to perform an operation of extracting the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 having a predefined spatial size, e.g., 14x 14.
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are concatenated to form a plane-assisted query feature map 610, which is further converted to the plane-assisted global descriptor 608.
  • a respective global image descriptor 612 is obtained.
  • each image is stored with the respective global image descriptor 612 in the database 564.
  • the electronic device processes the query image, it extracts an image from the database 564, determines a corresponding feature map, plan instance feature, and plane semantic feature of the extracted image, and aggregates the determined map and features to the respective global image descriptor 612.
  • the electronic device determines a respective distance 614 of these two descriptors 608 and 612.
  • a candidate image 562 is selected from the plurality of images based on the respective distance 614 corresponding to each of the plurality of images in the database 564.
  • the images of the database 564 are ranked based on the respective distance 614 of the global image descriptor 612 of each image from the plane- assisted global descriptor 608, and a predefined number of (Ac) candidate images 562 having the smallest respective distances 614 are identified.
  • the selected candidate image 562 is among the predefined number of (Ac) images.
  • the predefined number of (Ac) images have smaller respective distances 614 than remaining images of the plurality of images in the database 564.
  • the smaller respective distance 614 indicate that the query image 554 has a higher similarity level to the predefined number of (Ac) images than to the remaining images in the database 564.
  • the respective distance 614 is compared with a predefined distance threshold DTH. In accordance with a determination that the respective distance 614 is less than the predefined distance threshold DTH, the electronic device identifies the respective image corresponding to the respective distance 614 as a candidate image 562. Further, in some embodiments, the electronic device identifies the predefined number of (Ac) candidate images (e.g., 5 candidate images) from the database 564 and aborts processing more images in the database 564. Alternatively and additionally, in some embodiments, the electronic device is not limited by the predefined number (Ac) and identifies in the database 564 all images having the respective distances 614 that satisfy the predefined distance threshold DTH.
  • the image retrieval process 556 is enabled by deep learning techniques, e.g., by an image retrieval model, and the image retrieval model includes at least a feature backbone network 616, a plane processing network 618, and a vector aggregation network 620. Both of the feature backbone network 616 and plane processing network 618 are configured to receive and process the query image 554.
  • the plane processing network 618 is a single network.
  • the plane processing network 618 includes a plane detection network (e.g., PlaneRCNN) and a semantic segmentation network (e.g., DeepLab).
  • the plane detection network is configured to characterize structural features of the query image 554 with the query plane instance feature 604, and the semantic segmentation network is configured to characterize semantic features of the query image 554 with the query plane semantic feature 606.
  • the query image 554 is passed through three parallel branches, i.e., the feature backbone network 616, plane detection network, and semantic segmentation network.
  • the one or more bounding boxes are generated by the plane detection network of the plane processing network 618, and used to perform a region of interest align (ROIAlign) operation on each of the three parallel branches to extract a respective feature map for each region of interest (e.g., for each bounding box) in the query image 554.
  • ROIAlign region of interest align
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are aggregated to a plane-assisted query feature map 610 using a feature aggregation network 622.
  • the vector aggregation network 620 is configured to generate the plane-assisted global descriptor 608 associated with the query image 554.
  • an Net VLAD network is applied to discern a large amount of content clutter (e.g., people, cars), changes in viewpoint, and different illumination conditions (e.g., night and daytime illuminations).
  • the Net VLAD network encodes features of different planes defined by the bounding boxes into a high-dimensional vector (i.e., the plane-assisted global descriptor 608) using one or more clustering centers.
  • the vector aggregation network 620 has a single layer.
  • additional operations are performed to localize the camera that captures the query image 554, i.e., identify a query camera pose including a camera position and a camera orientation of the camera in association with the query image 554.
  • the plurality of images of the database 564 are captured in the same scene where the query image 554 is captured.
  • the electronic device extracts from the database 564 information of a candidate camera pose at which the candidate image 562 is captured in the scene, and determines that the query image is captured at the query camera pose related to the candidate camera pose of the candidate image 562.
  • the query camera pose associated with the query image 554 is identical to the candidate camera pose at which the candidate image 562 is captured.
  • the query camera pose associated with the query image 554 is moved by a relative pose change with reference to the candidate camera pose.
  • a plurality of query feature points are extracted from the query image 554, and a plurality of reference feature points are identified from the candidate image 562.
  • the plurality of query feature points and the plurality of reference feature points are compared to determine the query camera pose relative to the candidate camera pose.
  • the query camera pose associated with the query image 554 is identical to the candidate camera pose of the candidate image 562.
  • a plurality of query plane embeddings are generated for the plurality of query planes in the query image 554.
  • a plurality of candidate plane embeddings correspond to a plurality of candidate planes of the candidate image 562.
  • the plurality of query plane embeddings and the plurality of candidate plane embeddings are compared to identify a first number of matching plane pairs (e.g., plane pairs 806 in Figure 8A).
  • Each matching plane pair includes a respective query plane of the query image and a respective candidate plane of the candidate image that substantially matches the respective query plane, e.g., with a tolerance.
  • the first number of matching plane pairs are applied to determine the query camera pose 552 related to the candidate camera pose of the candidate image 562. Further, in some embodiments, the electronic device determines a cost matrix (e.g., cost matrix 808 in Figure 8A) including a plurality of matrix elements from the plurality of query plane embeddings and the plurality of candidate plane embeddings. Each matrix element indicates a distance between a respective query plane embedding and a respective candidate plane embedding.
  • a Hungarian method e.g., method 810 in Figure 8 A
  • a subset of matching plane pairs are selected.
  • a fitting error (e.g., fitting error 818 in Figure 8B or 8C) is determined between the query planes and candidate planes in the subset of matching plane pairs.
  • the electronic device determines whether the fitting error is below a threshold error (e.g., threshold error 838 in Figure 8C).
  • a threshold error e.g., threshold error 838 in Figure 8C.
  • the electronic device determines a relative pose change (e.g., pose changes 820 in Figures 8B and 8C) between the query camera pose 552 and the candidate camera pose, and determines the query camera pose 552 based on the candidate camera pose of the candidate image 562 and the relative pose change. More details on determining the query camera pose 552 associated with the query image 554 based on the candidate camera pose are explained below with reference to Figures 8A-8C.
  • the image retrieval process 556 leverages structural and semantic information of planes of the query image 554 and leads to accurate and robust image retrieval, particularly for query images 554 that are captured in indoor environments and contain multiple structured planes.
  • a scene is mapped to a 3D virtual space, and the camera is localized in the 3D virtual space.
  • the image retrieval process 556 enables a smooth AR experience. After a camera pose is determined in the 3D virtual space, a pre-built map of the 3D virtual space is loaded to determine the camera pose, and virtual objects are rendered in the 3D virtual space in a realistic manner.
  • FIG. 7 is a flow diagram of a training process 700 that trains an image retrieval model applied in an image retrieval process 600, in accordance with some embodiments.
  • the image retrieval model includes a feature backbone network 616, a plane processing network 618, and a vector aggregation network 620. Both of the feature backbone network 616 and plane processing network 618 are configured to receive and process the query image 554.
  • the plane processing network 618 is a single network.
  • the plane processing network 618 includes two separate and parallel networks, a plane detection network (e.g., PlaneRCNN) and a semantic segmentation network (e.g., DeepLab).
  • the plane detection network is configured to characterize structural features of the query image 554 with the query plane instance feature 604, and the semantic segmentation network is configured to characterize semantic features of the query image 554 with the query plane semantic feature 606.
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are aggregated to a plane-assisted query feature map 610 without using deep learning techniques.
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are aggregated to a plane-assisted query feature map 610 using a feature aggregation network 622.
  • the vector aggregation network 620 is configured to generate the plane-assisted global descriptor 608 associated with the query image 554.
  • each individual network of the image retrieval model is trained separately using a respective data set.
  • Each of the feature backbone network 616, plane detection network, semantic segmentation network, vector aggregation network 620, and feature aggregation network 622 is trained separately.
  • all of the feature backbone network 616, plane detection model, semantic segmentation model, feature aggregation network 622, and vector aggregation network 620 are trained end-to-end based on a triplet loss L using an anchor training image a a positive sample pi of the anchor training image, and a negative sample /z, of the anchor training image.
  • (1) where d(x £ )
  • and r is a preset margin.
  • the image retrieval model includes a first subset of networks and a second subset of networks.
  • Each of the first subset of networks is trained separately using a respective data set, and the second subset of networks are trained jointly in an end-to-end manner.
  • each of PlaneRCNN and DeepLab is pretrained individually and integrated in the image retrieval model.
  • the feature backbone network 616, vector aggregation network 620, and feature aggregation network 622 are trained jointly in an end-to-end manner, e.g., using the triplet loss L.
  • the training process 700 is implemented at a server 102 to train the image retrieval model, and the image retrieval model is provided to a client device 104 that implements the image retrieval process 600.
  • the training process 700 is implemented at a client device 104, which thereby both trains and applies the image retrieval model.
  • the training process 700 is implemented at a server 102 to train the image retrieval model.
  • a client device 104 sends the query image 554 to the server 102, and the image retrieval model is implemented by the server 102 to process the query image 554.
  • the candidate image 562 is returned by the server 102 to the client device 104.
  • FIG. 8A is a flow diagram of a camera localization process 800 that determines a query camera pose 552 associated with a query image 554 based on a candidate camera pose 852 associated with a candidate image 562, in accordance with some embodiments.
  • the camera localization process 800 is implemented at an electronic system (e.g., an electronic system 200 including a client device 104).
  • Information of the candidate camera pose 852 associated with the candidate image 562 is known, e.g., obtained with the candidate image 562.
  • the electronic system generates a plurality of query plane embeddings 802 corresponding to a plurality of query planes in the query image 554 and obtains a plurality of candidate plane embeddings 804 corresponding to a plurality of candidate planes of the candidate image 562.
  • the plurality of candidate plane embeddings 804 of the plurality of candidate planes of the candidate image 562 are optionally extracted from memory of the electronic system (e.g., memory 206 in Figure 2) or generated from the candidate image 562.
  • the plurality of query plane embeddings 802 and the plurality of candidate plane embeddings 804 are compared to identify a first number of matching plane pairs 806.
  • Each matching plane pair 806 includes a respective query plane of the query image 554 and a respective candidate plane of the candidate image 562 that matches the respective query plane. Based on the first number of matching plane pairs 806, the electronic system iteratively determines (850 or 880) that the query image 554 is captured at the query camera pose 552 related to the candidate camera pose 852 of the candidate image 562.
  • a cost matrix 808 is applied to compare the query and candidate plane embeddings 802 and 804.
  • the cost matrix 808 includes a plurality of matrix elements, and each matrix element indicates a distance (i.e., a similarity level) between a respective query plane embedding 802 and a respective candidate plane embedding 804.
  • each matrix element of the cost matrix 808 measures the distance using a cosine similarity of the respective plane embeddings 802 and 804.
  • a Hungarian method is further applied to the cost matrix 808 to identify the first number of matching plane pairs 806. Under some circumstances, the image retrieval process 500 identifies a predefined number of (e.g., 10) candidate images 562.
  • a first candidate image 562 has only one matching plane pair 806 with the query image 554 based on comparison of their query plane embeddings 802 and 804, and each of the other nine candidate images 562 has at least 5 matching plane pairs 806 with the query image 554 based on comparison of their query plane embeddings 802 and 804.
  • a predefined matching pair number threshold e.g. 3
  • the query camera pose 552 is determined from the matching plane pairs 806.
  • the corresponding first candidate image 562 is not used to determine the query camera pose 552 of the query image 554.
  • the respective candidate plane of the candidate image 562 matches the respective query plane of the query image 554, when the respective candidate plane is parallel with the respective query plane.
  • the respective candidate plane has a candidate plane normal direction that does not align with and deviates slightly from a query plane normal direction of the respective query plane.
  • the respective candidate plane of the candidate image 562 is regarded as matching the respective query plane of the query image 554, when a deviation of the candidate plane normal direction is less than a threshold angle (e.g., less than 5 degrees).
  • the plurality of query plane embeddings 802 and the plurality of candidate plane embeddings 804 are generated from the query image 554 and the candidate image 562 using a plane retrieval network 822, respectively.
  • the plane retrieval network 822 includes a feature backbone network 824, a plane detection model 826, and a region of interest (ROI) polling network 828.
  • a query feature map 830 is generated from the query image 554 using the feature backbone network 824.
  • a plurality of query bounding boxes 832 correspond to the plurality of query planes of the query image 554, and information of the query bounding boxes 832 is generated from the query image 554 using the plane detection model 826.
  • Information of the query bounding box 832 includes plane masks and local 3D plane parameters of the query planes of the query image 554.
  • the plurality of query plane embeddings 802 are generated from the information of the plurality of bounding boxes 832 and the query feature map 830 using the ROI polling network 828.
  • a candidate feature map 834 is generated from the candidate image 562 using the feature backbone network 824.
  • a plurality of candidate bounding boxes 836 correspond to the plurality of candidate planes of the candidate image 562, and information of the candidate bounding box 836 is generated from the candidate image 562 using the plane detection model 826.
  • Information of the candidate bounding box 836 includes plane masks and local 3D plane parameters of the candidate planes of the candidate image 562.
  • the plurality of candidate plane embeddings 804 are generated from the plurality of bounding boxes 836 and the candidate feature map 834 using the ROI polling network 828.
  • Figure 8B is a flow diagram of an iterative camera pose process 850 that determines a query camera pose 552 based on matching plane pairs 806 of a query image 554 and candidate image 562, in accordance with some embodiments.
  • the query camera pose 552 is determined during a plurality of iterations 812.
  • a second number of matching plane pairs 806’ are selected from the first number of matching plane pairs 806, and the second number is less than the first number.
  • each iteration 812 processes 3 matching plane pairs 806’ among total 7 matching plane pairs 806 of the query image 554 and candidate image 562.
  • the electronic system determines whether the respective query plane and the respective candidate plane are (814) substantially parallel (e.g., having an angle less than the threshold angle, such as less than 5 degrees). In accordance with a determination that in each of the selected second number of matching plane pairs 806’ of the respective query plane and the respective candidate plane are parallel, it is determined that the query camera pose 552 is aligned with and equal to the candidate camera pose 852 of the candidate image 562.
  • the electronic system determines an iteration fitting error 816 between the query planes and candidate planes in the second number of matching plane pairs 806’ during each iteration 812.
  • the electronic system identifies a smallest fitting error 818 among the iteration fitting error 816 of each of the plurality of iterations 812.
  • a relative pose change 820-1 between the query camera pose 552 and the candidate camera pose 852 is determined for the respective iteration 812 associated with the smallest fitting error 818.
  • the query camera pose 552 is determined based on the candidate camera pose 852 of the candidate image 562 and the relative pose change 820-1.
  • Figure 8C is a flow diagram of another iterative camera pose process 880 that determines a query camera pose 552 based on matching plane pairs 806 of a query image 554 and candidate image 562, in accordance with some embodiments.
  • the electronic system determines an iteration fitting error 816 between the query planes and candidate planes in the second number of matching plane pairs 806’ and whether the iteration fitting error 816 is below a threshold error 838 during each of a subset of iterations 812.
  • the electronic system terminates the plurality of iterations 812, determines a relative pose change 820-2 between the query camera pose 552 and the candidate camera pose 852.
  • the query camera pose 552 is determined based on the candidate camera pose 852 of the candidate image 562 and the relative pose change 820-2.
  • FIG 9 is a flow diagram of a training process 900 that trains a camera localization model applied in a camera localization process 800 as shown in Figure 8, in accordance with some embodiments.
  • a plurality of query plane embeddings 802 and a plurality of candidate plane embeddings 804 are generated from a query image 554 and a candidate image 562 using a plane retrieval network 822, respectively.
  • the plane retrieval network 822 includes a feature backbone network 824, a plane detection model 826, and an ROI polling network 828.
  • a query feature map 830 is generated from the query image 554 using the feature backbone network 824.
  • a plurality of query bounding boxes 832 correspond to the plurality of query planes of the query image 554, and information of the query bounding boxes 832 is generated from the query image 554 using the plane detection model 826.
  • Information of the query bounding box 832 includes plane masks and local 3D plane parameters of the query planes of the query image 554.
  • the plurality of query plane embeddings 804 are generated from the plurality of bounding boxes 832 and the query feature map 830 using the ROI polling network 828.
  • a candidate feature map 834 is generated from the candidate image 562 using the feature backbone network 824.
  • a plurality of candidate bounding boxes 836 correspond to the plurality of candidate planes of the candidate image 562, and information of the candidate bounding box 836 is generated from the candidate image 562 using the plane detection model 826.
  • Information of the candidate bounding box 836 includes plane masks and local 3D plane parameters of the candidate planes of the candidate image 562.
  • the plurality of candidate plane embeddings 804 are generated from the plurality of bounding boxes 836 and the candidate feature map 834 using the ROI polling network 828.
  • the feature backbone network 824 and ROI polling network 828 are trained jointly in a supervised manner, while the plane detection model 826 is predefined and configured to generate plane masks and local 3D plane parameters 832 and 836 for planes of the query image 554 and candidate image 562, respectively.
  • an electronic system e.g., a server 102, a client device 104 configured to implement the process 800
  • Each of the first and second ground truths 904 and 908 include respective ground truth plane labels.
  • the ground truth plane labels of paired images are leveraged to form positive and negative plane pairs.
  • a first training feature map 910 is generated from the first training image 902 using the feature backbone network 824, and a second training feature map 912 is generated from the second training image 906 using the feature backbone network 824.
  • a plurality of first training plane embeddings 914 are generated from the first ground truth 904 and the first training feature map 910 using the ROI polling network 828.
  • a plurality of second training plane embeddings 916 are generated from the second ground truth 908 and the second training feature map 912 using the ROI polling network 828.
  • a triplet loss LT EMBDI is determined between the first and second training plane embeddings 914 and 196.
  • the feature backbone network 824 and the ROI polling network 828 are trained based on the triplet loss LT EMBDI.
  • both the plane detection model 826 and the feature backbone network 824 are predefined.
  • the ROI polling network 828 is trained in a supervised manner. Specifically, the electronic system receives a first training image 902, a first ground truth 904 of the first training image 902, a second training image 906, a second ground truth 908 of the second training image 906. Each of the first and second ground truths 904 and 908 include respective ground truth plane labels.
  • the ground truth plane labels of paired images (the first and second training images 902 and 906) are leveraged to form positive and negative plane pairs.
  • a first training feature map 910 is generated from the first training image 902 using the feature backbone network 824
  • a second training feature map 912 is generated from the second training image 906 using the feature backbone network 824.
  • a plurality of first training plane embeddings 914 are generated from the first ground truth 904 and the first training plane embeddings 914 using the ROI polling network 828.
  • a plurality of second training plane embeddings 916 are generated from the second ground truth 908 and the second training plan embeddings 916 using the ROI polling network 828.
  • a triplet loss LT_EMBD2 is determined between the first and second training plane embeddings 914 and 916, and the ROI polling network 828 is trained based on the triplet loss LT_EMBD2.
  • the ground truth plane labels are used to pool the feature maps 910 and 912.
  • the triplet loss LT EMBDI or LT_EMBD2 can be built on those corresponding training plane embeddings 914 and 916.
  • the plane embeddings 914/916 of positive plane pairs become closer to each other.
  • distance between their plane embeddings 914/916 become larger.
  • ; and (1) d(a £ , n £ )
  • the pretrained plane detection model 826 is applied to generate plane masks and 3D plane parameters for the query image 554 and the candidate image 562 that is optionally retrieved in an image retrieval process 600.
  • An example of the pretrained plane detection model 826 is PlaneRCNN.
  • the plane embeddings 914 are pooled using the ROI polling network 828.
  • a cost matrix 808 is formed via a pairwise distance of plane feature embeddings 802 and 804. Plane matches are found by applying (808) a Hungarian method on the cost matrix.
  • a plane perspective-n-point (Plane- PNP) method is implemented based on random sample consensus (RANSAC).
  • the electronic system randomly sample three plane pairs from the first number of matching plane pairs 806 and solve a pose candidate in a closed form.
  • the electronic system randomly pick three matching plane pairs 806’ during each iteration 812.
  • a query camera pose 552 associated with a query image 554 is T q
  • a candidate camera pose 852 of a retrieved candidate image 562 is T r , which is known during a camera localization process 800.
  • relationships among the query camera pose 552, candidate camera pose 852, query planes, and candidate planes are described as follows: where n q and n r are a query plane of the query image 554 and a candidate plane of the retrieved candidate image 562, respectively; d q and d r are a query offset of the query plane and a candidate offset of the candidate plane, respectively; R qr is a rotation matrix describing rotational transformation of the query plane of the query image 554 and the candidate plane of the retrieved candidate image 562; and t qr is a translation vector describing the relative pose change 820 between the query plane and the candidate plane.
  • singular value decomposition is applied to solve the rotation matrix R qr (i.e., the relative pose change 820) from equation (3).
  • the translation vector t qr is directly solved in equation (4).
  • the relative pose change 820 (Z 9r ) is a combination of the rotation matrix R qr and translation vector t qr .
  • a fitting error Ze is defined as follows:
  • At least three matching plane pairs 806’ are applied to solve the query camera pose T q (552).
  • the query camera pose T q (552) is equal to the candidate camera pose 852.
  • the Plane-PNP method is implemented to solve the query camera pose T q (552) associated with the query image 554.
  • predictions of the query planes from the query image 554 are not sufficiently accurate, and the fitting error is required to be less than 0.5.
  • the fitting error is less than a threshold error 838 (e.g., 0.5)
  • the query camera pose T q (552) is used.
  • the candidate camera pose 852 is used.
  • planar objects with homogenous color are inserted into the scene. A camera localization system performs much better at localizing a camera after the insertion due to inclusion of planar geometry features.
  • the training process 900 is implemented at a server 102 to train the camera localization model, and the camera localization model is provided to a client device 104 to implement the camera localization process 800.
  • the training process 900 is implemented to train the camera localization model locally at a client device 104, which applies the camera localization model to implement the camera localization process 800.
  • the training process 900 is implemented at a server 102 to train the camera localization model.
  • a client device sends the query image 554 to the server 102, and the camera localization model is implemented by the server 102 to process the query image 554.
  • FIG 10A is a flow diagram of an image retrieval method 1000, in accordance with some embodiments.
  • the method 1000 is described as being implemented by an electronic system 200 (e.g., a client device 104, a server 102, or a combination thereof).
  • the client device 104 is a mobile phone 104C, AR glasses 104D, smart television device, or drone.
  • Method 1000 is, optionally, governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of the computer system.
  • Each of the operations shown in Figure 6 may correspond to instructions stored in a computer memory or non-transitory computer readable storage medium (e.g., memory 206 in Figure 2).
  • the computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices.
  • the instructions stored on the computer readable storage medium may include one or more of: source code, assembly language code, object code, or other instruction format that is interpreted by one or more processors. Some operations in method 1000 may be combined and/or the order of some operations may be changed.
  • the electronic system obtains (1002) a query image 554 in a scene and generates (1004) a query feature map 602, a query plane instance feature 604, and a query plane semantic feature 606 from the query image 554.
  • the query plane instance feature 604 identifies (1006) one or more bounding boxes defining one or more planes in the query image 554, and the query plane semantic feature includes semantic information of the one or more planes.
  • the electronic system aggregates (1008) the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 to a plane-assisted global descriptor 608.
  • the electronic system For each of a plurality of images in a database 564, the electronic system obtains (1010) a respective global image descriptor 612, and determines (1012) a respective distance 614 of the plane-assisted global descriptor 608 and the respective global image descriptor 612.
  • a candidate image 562 is selected (1014) from the plurality of images based on the respective distance 614 corresponding to each of the plurality of images of the database 564.
  • the candidate image 562 is (1016) among a predefined number of images having smaller respective distances than remaining image of the plurality of images in the database 564.
  • the query image 554 has a higher similarity level to the predefined number of images than to the remaining images in the database 564.
  • the one or more bounding boxes are applied to perform an operation of extracting the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 having a predefined spatial size.
  • each of the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 has a size of 14x 14.
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are concatenated form a plane-assisted query feature map 610.
  • the plane-assisted query feature map 610 is converted to the plane-assisted global descriptor 608.
  • the query feature map 602, query plane instance feature 604, and query plane semantic feature 606 are aggregated to a plane-assisted query feature map 610 using a feature aggregation network 622.
  • the plane-assisted query feature map 610 is converted to the plane-assisted global descriptor 608 using a vector aggregation network 620 (e.g., a NETVLAD layer).
  • the query feature map 602 is generated using a feature backbone network 616.
  • the query plane instance feature 604 and query plane semantic feature 606 are generated from a plane detection model and a semantic segmentation model, respectively.
  • a subset of the feature backbone network 610, plane detection model, semantic segmentation model, feature aggregation network, and vector aggregation network 620 is trained based on a triplet loss L using an anchor training image a a positive sample pi of the anchor training image, and a negative sample /z, of the anchor training image.
  • the respective global image descriptor 612 is obtained by generating a first feature map, a first plane instance feature, and a first plane semantic feature from the first image.
  • the first plane instance feature identifies one or more first bounding boxes defining one or more planes in the first image
  • the first plane semantic feature includes semantic information of the one or more planes in the first image.
  • the first feature map, first plane instance feature, and first plane semantic feature are aggregated to the respective global image descriptor of the first image.
  • the respective distance 614 is determined between the plane- assisted global descriptor of the query image 554 and the respective global image descriptor 612 of the first image.
  • the respective global image descriptor 612 is extracted from memory of the electronic system.
  • the plurality of images of the database 564 are captured in the scene where the query image 554 is captured.
  • the electronic system extracts, from the database 564, information of a candidate camera pose at which the candidate image 562 is captured in the scene.
  • the query image is captured at a query camera pose related to the candidate camera pose of the candidate image 562.
  • the query camera pose is determined based on the candidate camera pose.
  • the electronic system extracts a plurality of query feature points of the query image 554, identifies a plurality of reference feature points of the candidate image, and compares the plurality of query feature points and the plurality of reference feature points to determine the query camera pose relative to the candidate camera pose.
  • the electronic system generates a plurality of query plane embeddings 802 corresponding to a plurality of query planes in the query image 554.
  • a plurality of candidate plane embeddings 804 correspond to a plurality of candidate planes of the candidate image 562.
  • the plurality of query plane embeddings 802 and the plurality of candidate plane embeddings 804 are compared to identify a first number of matching plane pairs 806.
  • Each matching plane pair 806 includes a respective query plane of the query image 554 and a respective candidate plane of the candidate image 562 that substantially matches the respective query plane.
  • the first number of matching plane pairs 806 are applied to determine the query camera pose 552 related to the candidate camera pose 852 of the candidate image 562. Further, in some embodiments, the electronic system determines a cost matrix 808 including a plurality of matrix elements and applies a Hungarian method 810 to the cost matrix to identify the first number of matching plane pairs. Each matrix element indicates a distance between a respective query plane embedding and a respective candidate plane embedding. Additionally, in some embodiments, a subset of matching plane pairs 806’ is selected. A fitting error 816 is determined between the query planes and candidate planes in the subset of matching plane pairs and compared with a threshold error 838.
  • the electronic system determines a relative pose change 820-2 between the query camera pose 552 and the candidate camera pose 852 and determines the query camera pose 552 based on the candidate camera pose 852 of the candidate image 562 and the relative pose change 820-2.
  • FIG 10B is a flow diagram of a camera localization method 1050, in accordance with some embodiments.
  • the method 1050 is described as being implemented by an electronic system 200 (e.g., a client device 104, a server 102, or a combination thereof).
  • the client device 104 is a mobile phone 104C, AR glasses 104D, smart television device, or drone.
  • Method 1050 is, optionally, governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of the computer system.
  • Each of the operations shown in Figure 6 may correspond to instructions stored in a computer memory or non-transitory computer readable storage medium (e.g., memory 206 in Figure 2).
  • the computer readable storage medium may include a magnetic or optical disk storage device, solid state storage devices such as Flash memory, or other non-volatile memory device or devices.
  • the instructions stored on the computer readable storage medium may include one or more of: source code, assembly language code, object code, or other instruction format that is interpreted by one or more processors. Some operations in method 1050 may be combined and/or the order of some operations may be changed.
  • the electronic system obtains (1052) a query image 554, a candidate image 562, and information of a candidate camera pose 852 at which the candidate image 562 is captured in a scene, and generates (1054) a plurality of query plane embeddings 802 corresponding to a plurality of query planes in the query image 554.
  • the electronic system obtains (1056) a plurality of candidate plane embeddings 804 corresponding to a plurality of candidate planes of the candidate image 562.
  • the plurality of candidate plane embeddings 804 of the plurality of candidate planes of the candidate image 562 are extracted from memory of the electronic system.
  • the plurality of candidate plane embeddings 804 are generated from the candidate image 562.
  • the plurality of query plane embeddings 802 and the plurality of candidate plane embeddings 804 are compared (1058) to identify a first number of matching plane pairs 806.
  • Each matching plane pair 806 includes (1060) a respective query plane of the query image 554 and a respective candidate plane of the candidate image 562 that substantially matches the respective query plane.
  • the plurality of query plane embeddings 802 and the plurality of candidate plane embeddings 804 are compared by determining (1062) a cost matrix 808 including a plurality of matrix elements and applying (1064) a Hungarian method 810 to the cost matrix 808 to identify the first number of matching plane pairs 806.
  • Each matrix element indicates a distance between a respective query plane embedding 802 and a respective candidate plane embedding 804. Based on the first number of matching plane pairs 806, the electronic device iteratively determines (1066) that the query image 554 is captured at a query camera pose 552 related to the candidate camera pose 852 of the candidate image 562.
  • the electronic system determines that the first number is great than a predefined matching pair number threshold.
  • the query camera pose 552 is determined in accordance with a determination that the first number is greater than the predefined matching pair number threshold.
  • the predefined matching pair number threshold is equal to 3
  • the query camera pose 552 is determined in accordance with a determination that the query image 554 and candidate image 562 have at least 3 matching plane pairs.
  • the query camera pose 552 is determined during a plurality of iterations 812. For each iteration 812, the electronic system selects (1066) a second number of matching plane pairs 806’ in the first number of matching plane pairs 806. The second number is less than the first number. Further, in some embodiments, during each iteration 812, for each of the selected second number of matching plane pairs 806’, the electronic system determines (1070) whether the respective query plane and the respective candidate plane are parallel.
  • the electronic system determines (1072) that the query camera pose 552 is aligned with and equal to the candidate camera pose 852 of the candidate image 562.
  • the electronic system determines an iteration fitting error 816 between the query planes and candidate planes in the second number of matching plane pairs 806’.
  • a smallest fitting error is identified among the iteration fitting error 816 of each of the plurality of iterations 812.
  • a relative pose change 820 is determined between the query camera pose 552 and the candidate camera pose 852 in the respective iteration 812 associated with the smallest fitting error.
  • the query camera pose 552 is determined based on the candidate camera pose 852 of the candidate image and the relative pose change 820.
  • the electronic system determines an iteration fitting error 816 between the query planes and candidate planes in the second number of matching plane pairs 806’ and whether the iteration fitting error 816 is below a threshold error 838. For a last iteration, in accordance with a determination that the iteration fitting error 816 is below the threshold error 838, the electronic system terminates the plurality of iterations 812, determines a relative pose change 820 between the query camera pose 552 and the candidate camera pose 852, and determines the query camera pose 552 based on the candidate camera pose 852 of the candidate image 562 and the relative pose change 820.
  • the plurality of query plane embeddings 802 and the plurality of candidate plane embeddings 804 are generated from the query image 554 and the candidate image 562 using a plane retrieval network 822, respectively.
  • the plane retrieval network 822 includes a feature backbone network 824, a plane detection model 826, and an ROI polling network 828.
  • a query feature map 830 is generated from the query image 554 using the feature backbone network 824.
  • Information of a plurality of bounding boxes 832 corresponding to the plurality of query planes are generated using the plane detection model 826.
  • the plurality of query plane embeddings 802 are generated from the information of the plurality of bounding boxes 832 and the query feature map 830 using the ROI polling network 828.
  • the plane detection model 826 is predefined and configured to generate plane masks and local 3D plane parameters (i.e., the information of the bounding boxes 832) for planes of the query or candidate image 554 or 562.
  • the feature backbone network 824 and ROI polling network 828 are trained jointly in a supervised manner, e.g., based on a triplet loss LT EMBDI.
  • the plane detection model 826 and the feature backbone network 824 are predefined, and the ROI polling network 828 is trained in a supervised manner, e.g., based on a triplet loss LT_EMBD2. More details on training the ROI polling network 828 are explained above with reference to Figure 9.
  • the term “if’ is, optionally, construed to mean “when” or “upon” or “in response to determining” or “in response to detecting” or “in accordance with a determination that,” depending on the context.
  • the phrase “if it is determined” or “if [a stated condition or event] is detected” is, optionally, construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event]” or “in accordance with a determination that [a stated condition or event] is detected,” depending on the context.
  • stages that are not order dependent may be reordered and other stages may be combined or broken out. While some reordering or other groupings are specifically mentioned, others will be obvious to those of ordinary skill in the art, so the ordering and groupings presented herein are not an exhaustive list of alternatives. Moreover, it should be recognized that the stages can be implemented in hardware, firmware, software or any combination thereof.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Health & Medical Sciences (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)

Abstract

Une application vise à localiser une caméra dans une scène. Un système électronique obtient une image d'interrogation, une image candidate et des informations d'une pose de caméra candidate au niveau de laquelle l'image candidate est capturée dans une scène. Une pluralité d'incorporations de plans d'interrogation est générée pour une pluralité de plans d'interrogation dans l'image d'interrogation. Une pluralité d'incorporations de plans candidats est obtenue pour une pluralité de plans candidats de l'image candidate. Les incorporations de plans d'interrogation et les incorporations de plans candidats sont comparées pour identifier des paires de plans correspondants. Chaque paire de plans correspondants comprend un plan d'interrogation respectif de l'image d'interrogation et un plan candidat respectif de l'image candidate qui correspond sensiblement au plan d'interrogation respectif. Sur la base des paires de plans correspondants, le système électronique détermine de manière itérative que l'image d'interrogation est capturée au niveau d'une pose de caméra d'interrogation associée à la pose de caméra candidate de l'image candidate.
PCT/US2021/059638 2021-11-17 2021-11-17 Localisation de caméra sur la base d'un plan WO2023091129A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/US2021/059638 WO2023091129A1 (fr) 2021-11-17 2021-11-17 Localisation de caméra sur la base d'un plan

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2021/059638 WO2023091129A1 (fr) 2021-11-17 2021-11-17 Localisation de caméra sur la base d'un plan

Publications (1)

Publication Number Publication Date
WO2023091129A1 true WO2023091129A1 (fr) 2023-05-25

Family

ID=86397618

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2021/059638 WO2023091129A1 (fr) 2021-11-17 2021-11-17 Localisation de caméra sur la base d'un plan

Country Status (1)

Country Link
WO (1) WO2023091129A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150142716A1 (en) * 2013-11-15 2015-05-21 Disney Enterprises, Inc. Tracking player role using non-rigid formation priors
US20160180181A1 (en) * 2014-12-22 2016-06-23 Hyundai Mobis Co., Ltd. Obstacle detecting apparatus and obstacle detecting method
US20190221003A1 (en) * 2015-12-30 2019-07-18 Tsinghua University Method and device for interactive calibration based on 3d reconstruction in 3d surveillance system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150142716A1 (en) * 2013-11-15 2015-05-21 Disney Enterprises, Inc. Tracking player role using non-rigid formation priors
US20160180181A1 (en) * 2014-12-22 2016-06-23 Hyundai Mobis Co., Ltd. Obstacle detecting apparatus and obstacle detecting method
US20190221003A1 (en) * 2015-12-30 2019-07-18 Tsinghua University Method and device for interactive calibration based on 3d reconstruction in 3d surveillance system

Similar Documents

Publication Publication Date Title
US20220044352A1 (en) Cross-domain image translation
WO2021184026A1 (fr) Fusion audiovisuelle avec attention intermodale pour la reconnaissance d'actions vidéo
US20240037948A1 (en) Method for video moment retrieval, computer system, non-transitory computer-readable medium
WO2023101679A1 (fr) Récupération inter-modale d'image de texte sur la base d'une expansion de mots virtuels
CN112509047A (zh) 基于图像的位姿确定方法、装置、存储介质及电子设备
WO2021077140A2 (fr) Systèmes et procédés de transfert de connaissance préalable pour la retouche d'image
WO2021092600A2 (fr) Réseau pose-over-parts pour estimation de pose multi-personnes
WO2021092631A9 (fr) Récupération de moment vidéo à base de texte faiblement supervisé
CN114897039A (zh) 一种数据处理方法及相关设备
WO2022103877A1 (fr) Génération d'avatar 3d à commande audio réaliste
WO2023091131A1 (fr) Procédés et systèmes pour récupérer des images sur la base de caractéristiques de plan sémantique
WO2023086398A1 (fr) Réseaux de rendu 3d basés sur des champs de radiance neurale de réfraction
WO2023277877A1 (fr) Détection et reconstruction de plan sémantique 3d
WO2023091129A1 (fr) Localisation de caméra sur la base d'un plan
WO2023027712A1 (fr) Procédés et systèmes permettant de reconstruire simultanément une pose et des modèles humains 3d paramétriques dans des dispositifs mobiles
WO2023277888A1 (fr) Suivi de la main selon multiples perspectives
WO2023069086A1 (fr) Système et procédé de ré-éclairage de portrait dynamique
WO2023229600A1 (fr) Détection et mappage de pose en boucle fermée dans un mappage slam
US20240153184A1 (en) Real-time hand-held markerless human motion recording and avatar rendering in a mobile platform
WO2023063937A1 (fr) Procédés et systèmes de détection de régions planes à l'aide d'une profondeur prédite
WO2023063944A1 (fr) Reconnaissance de gestes de la main en deux étapes
WO2023023162A1 (fr) Détection et reconstruction de plan sémantique 3d à partir d'images stéréo multi-vues (mvs)
US20240087344A1 (en) Real-time scene text area detection
WO2022250689A1 (fr) Reconnaissance d'action vidéo progressive à l'aide d'attributs de scène
WO2023211435A1 (fr) Estimation de profondeur pour systèmes slam à l'aide de caméras monoculaires