TW202405757A - Computing apparatus and model generation method - Google Patents

Abstract

A computing apparatus and a model generation method are provided. In the method, sensing data is fused to determine depth information of multiple sensing points, moving trajectories of one or more pixels in the image data are tracked according to the image data and the inertial measurement data through the visual inertial odometry (VIO) algorithm, and those sensing points are mapped into a coordinate system according to the depth information and the moving trajectories through the simultaneous localization and mapping (SLAM) algorithm, to generate a three-dimensional environment model. An object is set on the three-dimensional environment model through a setting operation. The shopping information of the object is provided.

Description

Computing device and model generation method

The present invention relates to a spatial modeling technology, and in particular, to a computing device and a model generation method.

To simulate a real environment, the space of the real environment can be scanned to produce a simulated environment that looks like the real environment. The simulation environment can be implemented in applications such as games, home decoration, and robot movement. It is worth noting that the sensing data obtained by scanning the space may have errors, which may cause distortion of the simulated environment.

Embodiments of the present invention provide a computing device and a model generation method that can compensate for errors and thereby improve the fidelity of the simulation environment.

The model generation method in the embodiment of the present invention includes: fusing the sensing data to determine the depth information of multiple sensing points. These sensing data include image data and inertial measurement data. Based on the image data and inertial measurement data, the movement trajectory of one or more pixels in the image data is tracked through the Visual Inertial Odometry (VIO) algorithm. Based on the depth information and movement trajectories, those sensing points are mapped to the coordinate system through the Simultaneous Localization And Mapping (SLAM) algorithm to generate a three-dimensional environment model. Positions in the 3D environment model are defined by coordinate systems.

The computing device in the embodiment of the present invention includes a memory and a processor. Memory is used to store program code. The processor is coupled to the memory. The processor loads the program code to execute the computing device configured to fuse multiple sensing data to determine depth information of multiple sensing points, and track the depth information in the image data through a visual inertial odometry algorithm based on the image data and inertial measurement data. The movement trajectories of one or more pixels are mapped to the coordinate system through simultaneous positioning and mapping algorithms based on the depth information and movement trajectories to generate a three-dimensional environment model. Sensing data includes image data and inertial measurement data. Positions in the 3D environment model are defined by coordinate systems.

Based on the above, according to the computing device and model generation method of the present invention, VIO and SLAM algorithms are used to estimate the positions of sensing points in the environment, and thereby establish a three-dimensional environment model. This can improve the accuracy of position estimation and the fidelity of the three-dimensional model.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

Figure 1 is a schematic diagram of a model generation system 1 according to an embodiment of the present invention. Referring to FIG. 1 , the model generation system 1 includes (but is not limited to) a mobile device 10 and a computing device 30 .

The mobile device 10 may be a mobile phone, a tablet computer, a scanner, a robot, a wearable device, a self-propelled vehicle or a vehicle-mounted system. The mobile device 10 includes (but is not limited to) multiple sensors 11 .

The sensor 11 may be an image capture device, a LiDAR, a Time-of-Flight (ToF) detector, an Inertial Measurement Unit (IMU), an accelerometer, a gyroscope or an electronic sensor. compass. In one embodiment, the sensor 11 is used to obtain sensing data. Sensing data includes image data and inertial sensing data. The image data may be one or more images and the sensing intensity of their pixels. Inertial sensing data can be attitude, three-axis acceleration, angular velocity or displacement.

The computing device 30 may be a mobile phone, a tablet computer, a desktop computer, a notebook computer, a server or an intelligent assistant device. The computing device 30 is communicatively connected to the mobile device 10 . For example, through Wi-Fi, Bluetooth, infrared or other wireless transmission technologies, or through circuit internal wiring, Ethernet, optical fiber network, Universal Serial Bus (USB) or other wired transmission technologies or Receive data and may be implemented with additional communication transceivers (not shown). The computing device 30 includes (but is not limited to) a memory 31 and a processor 32 .

The memory 31 can be any type of fixed or removable random access memory (Radom Access Memory, RAM), read only memory (Read Only Memory, ROM), flash memory (flash memory), traditional hard disk (Hard Disk Drive, HDD), solid-state drive (Solid-State Drive, SSD) or similar components. In one embodiment, the memory 31 is used to store program codes, software modules, data (for example, sensing data, or three-dimensional models) or files, the details of which will be described in subsequent embodiments.

The processor 32 is coupled to the memory 31 . The processor 32 may be a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), programmable chemical controller, Application-Specific Integrated Circuit (ASIC) or other similar components or a combination of the above components. In one embodiment, the processor 32 is used to execute all or part of the operations of the computing device 30 and can load and execute program codes, software modules, files and/or data stored in the memory 31 . In one embodiment, the processor 32 performs all or part of the operations of embodiments of the present invention. In some embodiments, the software modules or program codes recorded in the memory 31 may also be implemented by physical circuits.

In some embodiments, the mobile device 10 and the computing device 30 may be integrated into independent devices.

In the following, the method described in the embodiment of the present invention will be explained in conjunction with various devices and components in the model generation system 1 . Each process of this method can be adjusted according to the implementation situation, and is not limited to this.

Figure 2 is a flow chart of a model generation method according to an embodiment of the present invention. Referring to FIG. 2 , the processor 32 of the computing device 30 fuses multiple sensing data to determine depth information of multiple sensing points (step S210 ). Specifically, multiple sensing points can be formed by scanning the environment through the sensor 11 . The depth information of the sensing point may be the distance between the sensor 11 and the sensing point. In one embodiment, the processor 32 may match images in the image data into multiple image blocks. For example, the processor 32 can identify objects in the image (for example, walls, ceilings, floors, or cabinets) through image feature comparison or deep learning models, and segment the objects into image blocks based on the outline of the area where the object is located. Then, the processor 32 can determine the depth information corresponding to those image blocks. For example, the processor 32 can extract features through a deep learning model and predict the depth information of the image block or the corresponding object based on the features. Deep learning models/algorithms can analyze training samples to obtain patterns from them, and then predict unknown data through patterns. Generally speaking, depth information is usually related to the size ratio and posture of objects in the scene. The deep learning model is a machine learning model constructed after learning, and inferences are made based on the evaluation data (for example, image area). For another example, the processor 32 can compare the image area with the characteristic information of objects located at different locations stored in the memory 31 . The processor 32 may determine the depth information based on the locations with a degree of similarity higher than a corresponding threshold.

In another embodiment, the sensor 11 is a depth sensor or a distance sensor. The processor 32 can determine the depth information of multiple sensing points in the environment based on the sensing data of the depth sensor or the distance sensor.

The processor 32 tracks the movement trajectory of one or more pixels in the image data through a visual inertial odometry (VIO) algorithm based on the image data and inertial measurement data (step S220). Specifically, VIO is a technology that uses one or more image capture devices and one or more IMUs to perform state measurements. The aforementioned state refers to the posture, speed or other physical quantity of the carrier of the sensor 11 (for example, the mobile device 10) under a specific degree of freedom. Since the image capture device can capture photons within a certain exposure time to obtain a two-dimensional (2D) image, when moving at low speed, the image data obtained by the image capture device records quite rich environmental information. However, at the same time, image data are easily affected by the environment and have dimensional ambiguities. In contrast, IMU is used to sense its own angular acceleration and acceleration. Although the inertial measurement data is relatively single and has a large cumulative error, it is not affected by the environment. In addition, inertial measurement data also have the characteristics of exact scale units, which just makes up for the shortage of image data. By integrating both image data and inertial measurement data, more accurate inertial navigation can be obtained.

Figure 3 is a schematic diagram of inertial navigation according to an embodiment of the present invention. Referring to FIG. 3 , the processor 32 can determine the position difference of the object in the image data between time point T1 and time point T2. Time point T1 is earlier than time point T2. The object occupies some of the pixels in the image. The processor 32 can identify the object, determine the image position of the object in the image, and define it as a landmark (landmark) L. The processor 32 can compare the position difference of the same object captured by the image capturing device 112 at two different time points T1 and T2.

Then, the processor 32 can determine the movement trajectory from time point T1 to time point T2 based on the initial position and position difference of time point T1. The initial position is determined based on the inertial measurement data (obtained through the IMU 111) at time point T1. For example, integrating the inertia of IMU 111 may yield the initial position. The processor 32 may further convert the position of the landmark L from the sensing coordinate system to the world coordinate system WC. VIO has many data fusion methods. For example, loosely coupled and tight coupled. The loose coupling algorithm performs pose estimation based on image data and inertial measurement data respectively, and then fuses the pose estimation results. The tightly coupled algorithm directly fuses image data and inertial measurement data, constructs motion and observation equations based on the fused data, and performs state estimation accordingly.

Referring to FIG. 2 , the processor 32 maps the sensing points to the coordinate system through a Simultaneous Localization And Mapping (SLAM) algorithm based on the depth information and movement trajectories to generate a three-dimensional (3D) environment model (step S230 ). Specifically, the SLAM algorithm can convert the depth information of sensing points in the environment at different locations at different times into the same coordinate system through coordinate conversion, thereby generating a complete three-dimensional environment model of the environment. The position in the three-dimensional environment model is defined by this coordinate system.

However, a bias/error-free and highly accurate 3D environment model requires unbiased movement trajectories and depth information. However, various sensors 11 usually have errors to varying degrees. In addition, noise usually exists in the real environment, so the SLAM algorithm must consider not only the unique mathematical solution, but also the interaction with the physical concepts related to the result. It is worth noting that in the next iterative step of building a three-dimensional model, the measured distance and direction/attitude have a predictable series of errors. These errors are usually caused by the limited accuracy of the sensor 11 and other noise from the environment, and are reflected in errors in points or features on the three-dimensional environment model. As time passes and movement changes, errors in positioning and map construction accumulate, affecting the accuracy of the map itself.

In one embodiment, the processor 32 may match the first correlation at the first point in time and the second correlation at the second point in time. The first time point is earlier than the second time point. The first correlation is the correlation between the sensing data at the first time point and the corresponding position in the three-dimensional environment model, and the second correlation is the correlation between the sensing data at the second time point and the three-dimensional environment model. The correlation between the corresponding positions. That is, the sensing data at a specific time point and the corresponding landmark. The SLAM algorithm uses an iterative mathematical problem to solve the deviations of various sensed data. Mathematical problems include, for example, forming equations of motion and equations of observation based on sensed data (as states).

The processor 32 may correct the positions of those sensing points on the coordinate system according to the matching results between the first correlation and the second correlation. To compensate for these errors, processor 32 may match the current three-dimensional environment model with previous three-dimensional environment models. For example, through the loop closure algorithm, it is known that repeated locations in the three-dimensional environment model have been traveled. Or, for algorithms related to SLAM probabilistics. For example, Kalman filtering, particle filtering (a certain Monte Carlo method), and scanning the data range for matching. Through these algorithms, the processor 32 can gradually optimize past and present trajectory positions and depth information by comparing current (eg, second time point) and past (eg, first time point) sensing data. . Through recursive optimization, accurate estimates of each point in the environment can be obtained. From the above description, it can be seen that the algorithm of the embodiment of the present invention can form a closed loop, and can also accumulate a complete and accurate three-dimensional environment model as the trajectory progresses. On the contrary, if a closed loop is not formed, the errors may continue to accumulate and amplify, eventually leading to incoherence in the previous and later data, thereby producing a useless three-dimensional environmental model.

In one embodiment, the processor 32 can minimize errors in the positions of the sensing points on the coordinate system through an optimization algorithm based on the first correlation and the second correlation, and through a filtering algorithm based on the second correlation. method to estimate the position of those sensing points on the coordinate system. The optimization algorithm converts the SLAM state estimate into an error term and minimizes the error term. For example, Newton's method, Gauss-Newton method or Levenberg-Marquardt method. Examples of filtering algorithms include Kalman filtering, extended Kalman filtering, and particle filtering. The optimization algorithm can refer to the sensing data at different time points, while the filtering algorithm introduces noise to the current sensing data.

The difference from the existing technology is that compared with the existing technology that only uses an optimization algorithm or a filtering algorithm, the embodiment of the present invention combines the two algorithms. The proportion of the optimization algorithm and the filtering algorithm is related to the software and hardware resources of the computing device 30 and the accuracy of the predicted position. For example, if the software and hardware resources or accuracy requirements are low, the filtering algorithm has a higher weight than the optimization algorithm. And if the software and hardware resources or accuracy requirements are high, the proportion of the optimization algorithm will be higher than that of the filtering algorithm.

In one embodiment, processor 32 may receive a set operation. Setting operations can be obtained through input devices such as touch panels, mice, keyboards, or other input devices. For example, a swipe, press, or click. The processor 32 can set objects in the three-dimensional environment model according to this setting operation. Depending on the application scenario, the objects may be furniture, picture frames or home appliances. The processor 32 can move the object according to the setting operation and place the object at a designated position in the three-dimensional environment model. Then, the processor 32 can provide the shopping information of the object through the display (not shown). For example, item name, amount, shipping method, payment options and other options. The processor 32 can also connect to the store server through a communication transceiver (not shown) and complete the shopping process accordingly.

In an application scenario, the mobile device 10 can quickly scan the space and sense all dimensional information in the space, so that the user no longer needs any manual measurements and can directly and easily arrange furniture in the three-dimensional environment model. Embodiments of the present invention can also provide a Software as a Service (SaaS) system, allowing users to refer to the actual space for presentation or adjustment of placement, and the shopping program loaded on the computing device 30 can add products to the shopping cart. to shop directly. In addition, the cloud connection method allows users to cooperate with each other to coordinate space remotely, thus becoming the largest online home community. However, it is not limited to furniture arrangement, and the rapid modeling characteristics of embodiments of the present invention can also be introduced into other applications.

To sum up, in the computing device and model generation method of the present invention, data from sensors such as LiDAR, cameras, and IMUs of mobile phones or other portable mobile devices are fused to obtain in-depth information, and then through VIO Algorithm is used to track the movement trajectories of different pixels on the camera, and the depth information and movement trajectories are used together with the SLAM algorithm framework for optimization to obtain accurate estimates of each sensing point in the environment.

Although the present invention has been disclosed above through embodiments, they are not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some modifications and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the appended patent application scope.

1: Model generation system 10:Mobile device 11: Sensor 30:Computing device 31:Memory 32: Processor S210~S230: steps T1, T2: time points 111:IMU 112:Image capture device L:Landmark WC: world coordinate system

FIG. 1 is a schematic diagram of a model generation system according to an embodiment of the present invention. Figure 2 is a flow chart of a model generation method according to an embodiment of the present invention. Figure 3 is a schematic diagram of inertial navigation according to an embodiment of the present invention.

S210~S230: steps

Claims (10)

A model generation method including: Fusion of multiple sensing data to determine depth information of multiple sensing points, where the sensing data includes image data and inertial measurement data; Track a movement trajectory of at least one pixel in the image data through a visual inertial odometry (VIO) algorithm based on the image data and the inertial measurement data; and Based on the depth information and the movement trajectory, the sensing points are mapped to a coordinate system through a Simultaneous Localization and Mapping (SLAM) algorithm to generate a three-dimensional environment model, in which The position is defined by this coordinate system. The model generation method as described in claim 1, wherein the step of mapping the sensing points to the coordinate system includes: Matching a first correlation at a first time point and a second correlation at a second time point, wherein the first time point is earlier than the second time point, and the first correlation is the first time point The correlation between the sensing data and the corresponding position in the three-dimensional environment model, and the second correlation is between the sensing data at the second time point and the corresponding position in the three-dimensional environment model relevance; and The positions of the sensing points on the coordinate system are modified according to the matching result between the first correlation and the second correlation. The model generation method of claim 2, wherein the step of modifying the positions of the sensing points on the coordinate system based on the matching result between the first correlation and the second correlation includes: Minimizing the position error of the sensing points on the coordinate system through an optimization algorithm based on the first correlation and the second correlation; and Estimating the positions of the sensing points on the coordinate system through a filtering algorithm based on the second correlation, wherein the proportion of the optimization algorithm and the filtering algorithm is related to the resources and predicted position of the computing device accuracy. As for the model generation method described in claim 1, the step of fusing the sensing data includes: Split the image data into multiple image blocks; The steps of determining the depth information corresponding to the image blocks and tracking the movement trajectory of the at least one pixel in the image data include: Determining a position difference between an object in the image data between a third time point and a fourth time point, wherein the third time point is earlier than the fourth time point; and The movement trajectory from the third time point to the fourth time point is determined based on an initial position at the third time point and the position difference, wherein the initial position is determined based on the inertial measurement data at the third time point. The model generation method as described in request item 1 further includes: receive a setting operation; Set an object in the three-dimensional environment model according to the setting operation; and Provide shopping information for this item. A computing device including: a memory to store a program code; and a processor, coupled to the memory, configured to load to execute: Fusion of multiple sensing data to determine depth information of multiple sensing points, where the sensing data includes image data and inertial measurement data; Track a movement trajectory of at least one pixel in the image data through a visual inertial odometry algorithm based on the image data and the inertial measurement data; and Based on the depth information and the movement trajectory, the sensing points are mapped to a coordinate system through simultaneous positioning and mapping to generate a three-dimensional environment model, where the positions in the three-dimensional environment model are defined by the coordinate system. The computing device of claim 6, wherein the processor is further configured to execute: Matching a first correlation at a first time point and a second correlation at a second time point, wherein the first time point is earlier than the second time point, and the first correlation is the first time point The correlation between the sensing data and the corresponding depth information and the corresponding movement trajectory, and the second correlation is the sensing data at the second time point, the corresponding depth information and the corresponding movement trajectory the correlation between; and The positions of the sensing points on the coordinate system are modified according to the matching result between the first correlation and the second correlation. The computing device of claim 7, wherein the processor is further configured to execute: Minimizing the position error of the sensing points on the coordinate system through an optimization algorithm based on the first correlation and the second correlation; and Estimating the positions of the sensing points on the coordinate system through a filtering algorithm based on the second correlation, wherein the proportion of the optimization algorithm and the filtering algorithm is related to the resources and predicted positions of the computing device accuracy. The computing device of claim 6, wherein the processor is further configured to execute: Split the image data into multiple image blocks; Determine the depth information corresponding to these image blocks; Determining a position difference between an object in the image data between a third time point and a fourth time point, wherein the third time point is earlier than the fourth time point; and The movement trajectory from the third time point to the fourth time point is determined based on an initial position at the third time point and the position difference, wherein the initial position is determined based on the inertial measurement data at the third time point. The computing device of claim 6, wherein the processor is further configured to execute: receive a setting operation; Set an object in the three-dimensional environment model according to the setting operation; and Provide shopping information for this item.

Images