TWI822423B - Computing apparatus and model generation method - Google Patents

Abstract

A computing apparatus and a model generation method are provided. In the method, sensing data is fused to determine depth information of multiple sensing points, moving trajectories of one or more pixels in the image data are tracked according to the image data and the inertial measurement data through the visual inertial odometry (VIO) algorithm, and those sensing points are mapped into a coordinate system according to the depth information and the moving trajectories through the simultaneous localization and mapping (SLAM) algorithm, to generate a three-dimensional environment model. An object is set on the three-dimensional environment model through a setting operation. The shopping information of the object is provided.

Description

Computing device and model generation method

The present invention relates to a spatial modeling technology, and in particular, to a computing device and a model generation method.

To simulate a real environment, the space of the real environment can be scanned to produce a simulated environment that looks like the real environment. The simulation environment can be implemented in applications such as games, home decoration, and robot movement. It is worth noting that the sensing data obtained by scanning the space may have errors, which may cause distortion of the simulated environment.

Embodiments of the present invention provide a computing device and a model generation method that can compensate for errors and thereby improve the fidelity of the simulation environment.

The model generation method in the embodiment of the present invention includes: fusing the sensing data to determine the depth information of multiple sensing points. These sensing data include image data and inertial measurement data. Based on the image data and inertial measurement data, the movement trajectory of one or more pixels in the image data is tracked through the Visual Inertial Odometry (VIO) algorithm. Based on the depth information and movement trajectories, those sensing points are mapped to the coordinate system through the Simultaneous Localization And Mapping (SLAM) algorithm to generate a three-dimensional environment model. Positions in the 3D environment model are defined by coordinate systems.

The computing device in the embodiment of the present invention includes a memory and a processor. Memory is used to store program code. The processor is coupled to the memory. The processor loads the program code to execute the computing device configured to fuse multiple sensing data to determine depth information of multiple sensing points, and track the depth information in the image data through a visual inertial odometry algorithm based on the image data and inertial measurement data. The movement trajectories of one or more pixels are mapped to the coordinate system through simultaneous positioning and mapping algorithms based on the depth information and movement trajectories to generate a three-dimensional environment model. Sensing data includes image data and inertial measurement data. Positions in the 3D environment model are defined by coordinate systems.

Based on the above, according to the computing device and model generation method of the present invention, VIO and SLAM algorithms are used to estimate the positions of sensing points in the environment, and thereby establish a three-dimensional environment model. This can improve the accuracy of position estimation and the fidelity of the three-dimensional model.

In order to make the above-mentioned features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

Figure 1 is a schematic diagram of a model generation system 1 according to an embodiment of the present invention. Referring to FIG. 1 , the model generation system 1 includes (but is not limited to) a mobile device 10 and a computing device 30 .

The mobile device 10 may be a mobile phone, a tablet computer, a scanner, a robot, a wearable device, a self-propelled vehicle or a vehicle-mounted system. The mobile device 10 includes (but is not limited to) multiple sensors 11 .

The sensor 11 may be an image capture device, a LiDAR, a Time-of-Flight (ToF) detector, an Inertial Measurement Unit (IMU), an accelerometer, a gyroscope or an electronic sensor. compass. In one embodiment, the sensor 11 is used to obtain sensing data. Sensing data includes image data and inertial sensing data. The image data may be one or more images and the sensing intensity of their pixels. Inertial sensing data can be attitude, three-axis acceleration, angular velocity or displacement.

The computing device 30 may be a mobile phone, a tablet computer, a desktop computer, a notebook computer, a server or an intelligent assistant device. The computing device 30 is communicatively connected to the mobile device 10 . For example, through Wi-Fi, Bluetooth, infrared or other wireless transmission technologies, or through circuit internal wiring, Ethernet, optical fiber network, Universal Serial Bus (USB) or other wired transmission technologies or Receive data and may be implemented with additional communication transceivers (not shown). The computing device 30 includes (but is not limited to) a memory 31 and a processor 32 .

The memory 31 can be any type of fixed or removable random access memory (Radom Access Memory, RAM), read only memory (Read Only Memory, ROM), flash memory (flash memory), traditional hard disk (Hard Disk Drive, HDD), solid-state drive (Solid-State Drive, SSD) or similar components. In one embodiment, the memory 31 is used to store program codes, software modules, data (for example, sensing data, or three-dimensional models) or files, the details of which will be described in subsequent embodiments.

The processor 32 is coupled to the memory 31 . The processor 32 may be a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor (Microprocessor), digital signal processor (Digital Signal Processor, DSP), programmable chemical controller, Application-Specific Integrated Circuit (ASIC) or other similar components or a combination of the above components. In one embodiment, the processor 32 is used to execute all or part of the operations of the computing device 30 and can load and execute program codes, software modules, files and/or data stored in the memory 31 . In one embodiment, the processor 32 performs all or part of the operations of embodiments of the present invention. In some embodiments, the software modules or program codes recorded in the memory 31 may also be implemented by physical circuits.

In some embodiments, the mobile device 10 and the computing device 30 may be integrated into independent devices.

In the following, the method described in the embodiment of the present invention will be explained in conjunction with various devices and components in the model generation system 1 . Each process of this method can be adjusted according to the implementation situation, and is not limited to this.

Figure 2 is a flow chart of a model generation method according to an embodiment of the present invention. Referring to FIG. 2 , the processor 32 of the computing device 30 fuses multiple sensing data to determine depth information of multiple sensing points (step S210 ). Specifically, multiple sensing points can be formed by scanning the environment through the sensor 11 . The depth information of the sensing point may be the distance between the sensor 11 and the sensing point. In one embodiment, the processor 32 may match images in the image data into multiple image blocks. For example, the processor 32 can identify objects in the image (for example, walls, ceilings, floors, or cabinets) through image feature comparison or deep learning models, and segment the objects into image blocks based on the outline of the area where the object is located. Then, the processor 32 can determine the depth information corresponding to those image blocks. For example, the processor 32 can extract features through a deep learning model and predict the depth information of the image block or the corresponding object based on the features. Deep learning models/algorithms can analyze training samples to obtain patterns from them, and then predict unknown data through patterns. Generally speaking, depth information is usually related to the size ratio and posture of objects in the scene. The deep learning model is a machine learning model constructed after learning, and inferences are made based on the evaluation data (for example, image area). For another example, the processor 32 can compare the image area with the characteristic information of objects located at different locations stored in the memory 31 . The processor 32 may determine the depth information based on the locations with a degree of similarity higher than a corresponding threshold.

In another embodiment, the sensor 11 is a depth sensor or a distance sensor. The processor 32 can determine the depth information of multiple sensing points in the environment based on the sensing data of the depth sensor or the distance sensor.

The processor 32 tracks the movement trajectory of one or more pixels in the image data through a visual inertial odometry (VIO) algorithm based on the image data and inertial measurement data (step S220). Specifically, VIO is a technology that uses one or more image capture devices and one or more IMUs to perform state measurements. The aforementioned state refers to the posture, speed or other physical quantity of the carrier of the sensor 11 (for example, the mobile device 10) under a specific degree of freedom. Since the image capture device can capture photons within a certain exposure time to obtain a two-dimensional (2D) image, when moving at low speed, the image data obtained by the image capture device records quite rich environmental information. However, at the same time, image data are easily affected by the environment and have dimensional ambiguities. In contrast, IMU is used to sense its own angular acceleration and acceleration. Although the inertial measurement data is relatively single and has a large cumulative error, it is not affected by the environment. In addition, inertial measurement data also have the characteristics of exact scale units, which just makes up for the shortage of image data. By integrating both image data and inertial measurement data, more accurate inertial navigation can be obtained.

Figure 3 is a schematic diagram of inertial navigation according to an embodiment of the present invention. Referring to FIG. 3 , the processor 32 can determine the position difference of the object in the image data between time point T1 and time point T2. Time point T1 is earlier than time point T2. The object occupies some of the pixels in the image. The processor 32 can identify the object, determine the image position of the object in the image, and define it as a landmark (landmark) L. The processor 32 can compare the position difference of the same object captured by the image capturing device 112 at two different time points T1 and T2.

Then, the processor 32 can determine the movement trajectory from time point T1 to time point T2 based on the initial position and position difference of time point T1. The initial position is determined based on the inertial measurement data (obtained through the IMU 111) at time point T1. For example, integrating the inertia of IMU 111 may yield the initial position. The processor 32 may further convert the position of the landmark L from the sensing coordinate system to the world coordinate system WC. VIO has many data fusion methods. For example, loosely coupled and tight coupled. The loose coupling algorithm performs pose estimation based on image data and inertial measurement data respectively, and then fuses the pose estimation results. The tightly coupled algorithm directly fuses image data and inertial measurement data, constructs motion and observation equations based on the fused data, and performs state estimation accordingly.

Referring to FIG. 2 , the processor 32 maps the sensing points to the coordinate system through a Simultaneous Localization And Mapping (SLAM) algorithm based on the depth information and movement trajectories to generate a three-dimensional (3D) environment model (step S230 ). Specifically, the SLAM algorithm can convert the depth information of sensing points in the environment at different locations at different times into the same coordinate system through coordinate conversion, thereby generating a complete three-dimensional environment model of the environment. The position in the three-dimensional environment model is defined by this coordinate system.

However, a bias/error-free and highly accurate 3D environment model requires unbiased movement trajectories and depth information. However, various sensors 11 usually have errors to varying degrees. In addition, noise usually exists in the real environment, so the SLAM algorithm must consider not only the unique mathematical solution, but also the interaction with the physical concepts related to the result. It is worth noting that in the next iterative step of building a three-dimensional model, the measured distance and direction/attitude have a predictable series of errors. These errors are usually caused by the limited accuracy of the sensor 11 and other noise from the environment, and are reflected in errors in points or features on the three-dimensional environment model. As time passes and movement changes, errors in positioning and map construction accumulate, affecting the accuracy of the map itself.

In one embodiment, the processor 32 may match the first correlation at the first point in time and the second correlation at the second point in time. The first time point is earlier than the second time point. The first correlation is the correlation between the sensing data at the first time point and the corresponding position in the three-dimensional environment model, and the second correlation is the correlation between the sensing data at the second time point and the three-dimensional environment model. The correlation between the corresponding positions. That is, the sensing data at a specific time point and the corresponding landmark. The SLAM algorithm uses an iterative mathematical problem to solve the deviations of various sensed data. Mathematical problems include, for example, forming equations of motion and equations of observation based on sensed data (as states).

The processor 32 may correct the positions of those sensing points on the coordinate system according to the matching results between the first correlation and the second correlation. To compensate for these errors, processor 32 may match the current three-dimensional environment model with previous three-dimensional environment models. For example, through the loop closure algorithm, it is known that repeated locations in the three-dimensional environment model have been traveled. Or, for algorithms related to SLAM probabilistics. For example, Kalman filtering, particle filtering (a certain Monte Carlo method), and scanning the data range for matching. Through these algorithms, the processor 32 can gradually optimize past and present trajectory positions and depth information by comparing current (eg, second time point) and past (eg, first time point) sensing data. . Through recursive optimization, accurate estimates of each point in the environment can be obtained. From the above description, it can be seen that the algorithm of the embodiment of the present invention can form a closed loop, and can also accumulate a complete and accurate three-dimensional environment model as the trajectory progresses. On the contrary, if a closed loop is not formed, the errors may continue to accumulate and amplify, eventually leading to incoherence in the previous and later data, thereby producing a useless three-dimensional environmental model.

In one embodiment, the processor 32 can minimize errors in the positions of the sensing points on the coordinate system through an optimization algorithm based on the first correlation and the second correlation, and through a filtering algorithm based on the second correlation. method to estimate the position of those sensing points on the coordinate system. The optimization algorithm converts the SLAM state estimate into an error term and minimizes the error term. For example, Newton's method, Gauss-Newton method or Levenberg-Marquardt method. Examples of filtering algorithms include Kalman filtering, extended Kalman filtering, and particle filtering. The optimization algorithm can refer to the sensing data at different time points, while the filtering algorithm introduces noise to the current sensing data.

The difference from the existing technology is that compared with the existing technology that only uses an optimization algorithm or a filtering algorithm, the embodiment of the present invention combines the two algorithms. The proportion of the optimization algorithm and the filtering algorithm is related to the software and hardware resources of the computing device 30 and the accuracy of the predicted position. For example, if the software and hardware resources or accuracy requirements are low, the filtering algorithm has a higher weight than the optimization algorithm. And if the software and hardware resources or accuracy requirements are high, the proportion of the optimization algorithm will be higher than that of the filtering algorithm.

In one embodiment, processor 32 may receive a set operation. Setting operations can be obtained through input devices such as touch panels, mice, keyboards, or other input devices. For example, a swipe, press, or click. The processor 32 can set objects in the three-dimensional environment model according to this setting operation. Depending on the application scenario, the objects may be furniture, picture frames or home appliances. The processor 32 can move the object according to the setting operation and place the object at a designated position in the three-dimensional environment model. Then, the processor 32 can provide the shopping information of the object through the display (not shown). For example, item name, amount, shipping method, payment options and other options. The processor 32 can also connect to the store server through a communication transceiver (not shown) and complete the shopping process accordingly.

In an application scenario, the mobile device 10 can quickly scan the space and sense all dimensional information in the space, so that the user no longer needs any manual measurements and can directly and easily arrange furniture in the three-dimensional environment model. Embodiments of the present invention can also provide a Software as a Service (SaaS) system, allowing users to refer to the actual space for presentation or adjustment of placement, and the shopping program loaded on the computing device 30 can add products to the shopping cart. to shop directly. In addition, the cloud connection method allows users to cooperate with each other to coordinate space remotely, thus becoming the largest online home community. However, it is not limited to furniture arrangement, and the rapid modeling characteristics of embodiments of the present invention can also be introduced into other applications.

To sum up, in the computing device and model generation method of the present invention, data from sensors such as LiDAR, cameras, and IMUs of mobile phones or other portable mobile devices are fused to obtain in-depth information, and then through VIO Algorithm is used to track the movement trajectories of different pixels on the camera, and the depth information and movement trajectories are used together with the SLAM algorithm framework for optimization to obtain accurate estimates of each sensing point in the environment.

Although the present invention has been disclosed above through embodiments, they are not intended to limit the present invention. Anyone with ordinary knowledge in the technical field may make some modifications and modifications without departing from the spirit and scope of the present invention. Therefore, The protection scope of the present invention shall be determined by the appended patent application scope.

1: Model generation system 10:Mobile device 11: Sensor 30:Computing device 31:Memory 32: Processor S210~S230: steps T1, T2: time point 111:IMU 112:Image capture device L:Landmark WC: world coordinate system

FIG. 1 is a schematic diagram of a model generation system according to an embodiment of the present invention. Figure 2 is a flow chart of a model generation method according to an embodiment of the present invention. Figure 3 is a schematic diagram of inertial navigation according to an embodiment of the present invention.

S210~S230: steps

Claims (6)

A model generation method is suitable for a computing device, wherein the computing device includes a processor. The method includes: fusing multiple sensing data through the processor to determine depth information of multiple sensing points, wherein The sensing data includes an image data and an inertial measurement data. The step of determining the depth information of the sensing points includes: using the processor to identify multiple objects matching to multiple objects based on the image data. object image blocks; through the processor, input the object image blocks to the deep learning model to output the characteristics and depth information of the objects, and then determine the depth of the sensing points corresponding to the objects Information; through the processor, track a movement trajectory of at least one pixel in the image data through a visual inertial odometry (VIO) algorithm based on the image data and the inertial measurement data; and through the processor , based on the depth information and the movement trajectory, the sensing points are mapped to a coordinate system through a Simultaneous Localization and Mapping (SLAM) algorithm to generate a three-dimensional environment model, wherein the three-dimensional environment model The position of is defined by the coordinate system, wherein the step of mapping the sensing points to the coordinate system includes: via the processor, matching a first correlation of a first time point and a second time point of A second correlation, wherein the first time point is earlier than the second time point, and the first correlation is between the sensing data at the first time point and the The correlation between the corresponding positions in the three-dimensional environment model, and the second correlation is the correlation between the sensing data at the second time point and the corresponding positions in the three-dimensional environment model; and through the processing The device corrects the position of the sensing points on the coordinate system according to the matching result between the first correlation and the second correlation, wherein according to the matching between the first correlation and the second correlation As a result, the step of modifying the positions of the sensing points on the coordinate system includes: via the processor, minimizing the position of the sensing points based on the first correlation and the second correlation through an optimization algorithm. the error of the position on the coordinate system; and through the processor, estimating the position of the sensing points on the coordinate system through a filtering algorithm based on the second correlation, wherein the optimization algorithm and the The proportion of the filtering algorithm is related to the resources of the computing device and the accuracy of the predicted position. If the resource or the accuracy requirement is low, the proportion of the filtering algorithm is higher than the optimization algorithm. If the The requirements for resources or accuracy are higher, and the optimization algorithm has a higher proportion than the filtering algorithm. The model generation method as described in claim 1, wherein the step of fusing the sensing data includes: dividing the image data into a plurality of image blocks through the processor; determining the image blocks through the processor Corresponding depth information, and the step of tracking the movement trajectory of the at least one pixel in the image data includes: Determine, through the processor, a position difference between an object in the image data between a third time point and a fourth time point, wherein the third time point is earlier than the fourth time point; and through the processing The device determines the movement trajectory from the third time point to the fourth time point based on an initial position at the third time point and the position difference, wherein the initial position is determined based on the inertial measurement data at the third time point. . The model generation method as described in claim 1 further includes: receiving a setting operation through the processor; setting an object in the three-dimensional environment model according to the setting operation through the processor; and providing, through the processor, Shopping information for this item. A computing device includes: a memory for storing a program code; and a processor coupled to the memory and configured to load and execute: fuse multiple sensing data to determine multiple sensing A depth information of points, wherein the sensing data includes an image data and an inertial measurement data, wherein the step of determining the depth information of the sensing points includes: identifying a plurality of objects matched to a plurality of objects according to the image data. Object image blocks; input the object image blocks to the deep learning model to output the characteristics and depth information of the objects, and then determine the depth information corresponding to the sensing points of the objects; Tracking a movement trajectory of at least one pixel in the image data based on the image data and the inertial measurement data through a visual inertial odometry algorithm; and based on the depth information and the movement trajectory through a synchronized positioning and mapping of these senses The measuring points are mapped to a coordinate system to generate a three-dimensional environment model, where the positions in the three-dimensional environment model are defined by the coordinate system. The step of mapping the sensing points to the coordinate system includes: matching a first A first correlation at a time point and a second correlation at a second time point, wherein the first time point is earlier than the second time point, and the first correlation is the feelings at the first time point. The correlation between the sensing data and the corresponding position in the three-dimensional environment model, and the second correlation is the correlation between the sensing data at the second time point and the corresponding position in the three-dimensional environment model; and modifying the positions of the sensing points on the coordinate system based on the matching results between the first correlation and the second correlation, wherein based on the matching results between the first correlation and the second correlation The step of modifying the positions of the sensing points on the coordinate system includes: minimizing the positions of the sensing points on the coordinate system through an optimization algorithm based on the first correlation and the second correlation. error; and estimating the positions of the sensing points on the coordinate system through a filtering algorithm based on the second correlation, wherein the proportion of the optimization algorithm and the filtering algorithm is related to the computing device. The accuracy of the resource and predicted location, where if the resource or the accuracy If the accuracy requirement is low, the filtering algorithm has a higher weight than the optimization algorithm. If the resource or the accuracy requirement is high, the optimization algorithm has a higher weight than the filtering algorithm. The computing device as described in claim 6, wherein the processor is further used to: divide the image data into a plurality of image blocks; determine the depth information corresponding to the image blocks; determine an object in the image data a position difference between a third time point and a fourth time point, wherein the third time point is earlier than the fourth time point; and the determination is based on an initial position of the third time point and the position difference. The movement trajectory from the third time point to the fourth time point, wherein the initial position is determined based on the inertial measurement data at the third time point. The computing device of claim 6, wherein the processor is further configured to: receive a setting operation; set an object in the three-dimensional environment model according to the setting operation; and provide shopping information for the object.

Images