RU2812950C1 - System for constructing three-dimensional space model - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates namely to the modelling of three-dimensional space. A system for constructing a model of three-dimensional space, containing an image capturing device that moves in three-dimensional space, an interface for obtaining images of three-dimensional space and a system for modelling three-dimensional space, wherein the image capturing device is made in the form of an IOS-based phone with LIDAR, and before constructing the three-dimensional space, calibration of the optical part, the camera of the image capture device for each frame of the video image simultaneously captures RGB colour data, depth data, IMU data and odometry data, and the three-dimensional space modelling system translates the RGB colour data and depth data into a pixel matrix, calculates the exact positions for each frame received by a phone moving in space for subsequent reconstruction of three-dimensional space.

EFFECT: increasing the accuracy of constructing three-dimensional space models.

2 cl, 12 dwg

Description

The invention relates to modeling three-dimensional space.

The main goal of the proposed solution is to create a system for communication and collaboration of a new generation. The idea is based on the fact that many work tasks cannot be solved remotely using video alone (solutions like Skype, Zoom), since it is not informative and convenient enough compared to personal presence on site.

To solve this problem, it is necessary to create an end-to-end solution based on the user's smartphone with LIDAR for streaming video with depth data (RGB-D) and subsequent 3D reconstruction from this data by a remote device in real time.

The closest solution is RF patent No. 2713611, dated 02/05/2020, which discloses a device and method for generating a three-dimensional space model. This device for generating a three-dimensional space model includes: an image acquisition interface configured to receive image data generated by an image capture device, this image data represents an observation result, so that there is a relative movement between the three-dimensional space and the image capture device in time; and a modeling engine configured to process the image data received by the imaging interface and to compute a three-dimensional model of the three-dimensional space, where the modeling engine comprises: a model segmenter configured to segment the three-dimensional model and partition it into at least active and inactive parts based on at least one property of the model, wherein the modeling engine is configured to use active parts of the three-dimensional model to update the model over time; and the alignment mechanism is configured to align active portions of the 3D model with inactive portions of the 3D model over time.

The disadvantages of this device are the lack of integrity of the resulting three-dimensional model, as well as its quality, due to the presence of a large amount of noise.

The proposed system allows for three-dimensional reconstruction in real time and does this much more accurately than built-in methods or other analogues implemented on a mobile device.

The technical result is increasing the accuracy of the three-dimensional space model.

The technical result is achieved by the fact that

The proposed system is illustrated in Figs. 1-5.

Figure 1 shows a diagram of the operation of the proposed solution for data translation and processing,

Figure 2 shows a diagram of the steps and stages of the proposed system for creating a three-dimensional reconstruction,

Fig. 3 shows a pattern for camera calibration,

Fig. 4 shows an example of determining key points in a frame and comparing them with an adjacent frame to determine the trajectory,

Fig. 5 shows an example of a reconstructed movement map,

Figure 6 shows an example of reconstruction before the Loop Closure event fires. The area of trajectory divergence is marked in red,

Fig. 7 shows the reconstruction after the Loop Closure algorithm is triggered. Red lines highlight the trajectories of correction of points from the current frame,

Fig.8 is an example of a dataset element, frame No. 10,

Fig. 9 shows the structure of an octree for storing points,

Fig. 10 shows real-time rendering at intervals of 500 frames,

Fig. 11 shows the result without excluding neighboring points, the result with filtering,

Fig. 12 shows the results of the measurement comparison between the real world and the reconstruction.

To implement the proposed system, Iphone 13 pro Max was chosen as an image capture device, which allows you to obtain a depth map with a resolution of 192 by 256 with a frequency of 60 frames per second. Combining this with camera agility, we can easily and quickly obtain a dense and accurate map of points.

In this case, data transfer from phone 13 pro Max to the three-dimensional space modeling system was carried out using the WebRTC interface via the TCP protocol, see Fig. 1.

From an architectural point of view, a software package of three components was implemented

1) Smartphone application "ARstream"

2) Server application for signaling "Signal"

3) Client application for receiving and processing data "Viewer"

The main part of the tasks is implemented directly on the sending device in the ARstream application.

On it, using ARkit 6 and its built-in functions, the ability to obtain data from LIDAR was realized with the assignment of depth parameters and a degree of confidence for each pixel.

LIDAR devices iPhone 13 pro Max allows you to obtain a depth map with a resolution of 192 by 256 with a frequency of 60 frames per second. Combining this with camera mobility, we can easily and quickly obtain a dense and accurate map of points. However, careful filtering around the edges of physical objects is required, since at this depth map resolution, these areas contain potential sources of reconstruction errors. To do this, in addition to the depth of each pixel, we also transmit the degree of confidence in the range from 0 to 1, which is calculated by methods built into ARkit 6.

The ARstream application receives a full set of data from the device’s sensors, marks them with a serial number and then sends video and audio to be sent using WebRTC methods to ensure fast and high-quality transmission, and the remaining data using the packet transmission method.

This technique, in combination with a signaling server, will potentially allow secure transmission and encryption of data, since we can use any encryption methods for transmission between client devices, and the signaling server will provide all user authentication requirements.

The implementation of three-dimensional reconstruction using LIDAR (RGB-D data) is generally feasible when solving the following problems (see Fig. 2):

1) Receiving data from device sensors and associating them. At the same time, various approaches make it possible to implement this either adjusted for data from the device’s IMU sensors or without them

2) Calculating the camera's trajectory

3) Construction of a point cloud from frame to frame

4) Cleaning the resulting data from noise

5) Trajectory correction mechanisms to eliminate accumulated errors (Loop Closure)

6) Construction of the final mesh

General methodology of work using our approach and results as an example.

Example input data:

1) RGB. Available in standard 16 or 8 bit digital encoding

2) Depth. They are PNGs that store data only in the black-white range, where pure white (0) indicates the minimum distance, and black (1) the maximum distance captured by the depth sensor.

3) IMU data. Stored in text format like

All types of incoming data can be processed both in real time and in the form of a pre-prepared group of data (dataset).

The data transfer mechanism sends the following types of information

1) Color

2) Depth

3) Confidence in depth

4) Camera matrix

5) IMU data

6) Odometry data

A key step in high-quality 3D reconstruction is the mechanism for accurately defining the camera in 3D space, as well as calibrating the camera's optical part and lens to correctly construct the point cloud.

Standard algorithms were used for calibration. In general terms, it consists of a series of photographs of a chessboard pattern (see Fig. 3).

The task of calibration is to calculate the parameters of the matrix and lens to determine the distortions of the incoming light, which are typical for any photographic equipment, and take these distortions into account during reconstruction.

The main difficulty in our task is directly determining the trajectory and this is due to the features of the camera's rolling shutter.

This feature means that in the frames that are captured during turns there are distortions that are invisible to the eye, but very significant when calculating. They occur because the top of the frame is captured 15 milliseconds earlier than the bottom. In this way, a shift is formed, which, when calculated by the method of visual odometry (comparing a sequence of frames), will produce a certain error, which sooner or later will lead to significant distortion or breakdown of the reconstruction.

To compensate for this problem, we take into account additional data from IMU trackers and the capabilities built into the AR Kit to calculate the trajectory of the phone’s movements.

A brief description of the position determination technique is as follows:

1) For a picture with a resolution of 720 × 480, 1000 key points are determined in the frame. The key point is the area of high contrast between adjacent pixels. If possible, points are determined evenly across the frame, which is divided into a grid consisting of 60 squares, in each square the algorithm tries to detect at least 5 points.

2) If the search for points was successful for the previous frame, a constant velocity measurement model is used, which is used to calculate and predict the camera movement. If tracking of the previous frame was not successful, then a broader location determination method is used.

3) Absolute position calculation.

If frame-by-frame tracking is lost, the frame is converted into a so-called bag of words and queued for searching for matches. Thus, the current frame is checked against previous frames to find the one with the most matches. If found, it is checked against a number of neighboring ones to confirm compliance. After this, tracking is optimized and position calculation continues.

4) Map construction.

After calculating the map position and identifying the key points and their matches, we can project the map onto the frame and find a match with the overall reconstruction map. This local map contains a set of key frames K1, which stores it together with the current frame and a set K2, which represents neighboring frames to K1 in the visibility graph (the points are in the field of view of all frames). The map also has a reference key frame Kref ∈ K1, which stores all matching key points with the current frame. Now every map point visible in K1 and K2 is searched in the current frame as follows:

- The projection of the key point x in the current frame is calculated.

Cancel if it goes beyond the image boundaries.

- Calculate the angle between the current line of sight v and the key point. Cancel if v ⋅ n < cos(60°).

- Calculate the distance d from the map point to the camera center. Cancel if it goes beyond the map point scale invariance region d /∈ [dmin, dmax].

- Calculate the scale in the frame using the d/dmin ratio.

5) Associate key points with the best match from the current map and thus calculate the camera position f. New key points are added to the general map (see Fig. 4).

The resulting algorithm allows you to build a map of space based on abstract key points and an approximate geometry of the space with a fairly high level of accuracy. However, this method is of interest to us primarily for reconstructing the trajectory of movement, see Fig. 5; further reconstruction occurs using data from the depth camera.

In our case, due to the problems outlined above, correction is also required for data from other sensors of the device - the accelerometer and gyroscope. Thanks to them, the trajectory delta between frames is calculated in parallel, and this information is used to limit the search area for key frames.

Also, in any reconstruction methods existing today, there is an accumulation of errors, see Fig. 6. Using different methods and types of cameras, it can be greatly reduced, but not completely eliminated. To solve this problem, the Loop Closure algorithm is used. For it to work, the shooter must periodically return to areas already captured by the camera. When the algorithm operates, when a situation arises when returning to an already visited area, the process of checking the key points of the current frame with those already stored on the map occurs. When a sufficient number of matches are found, the algorithm adjusts the entire trajectory between the current frame and the last frame when this area was captured by the camera.

There is a direct correction method using the interpolation method. The vector deviation between the current frame position and the expected one is calculated, and the entire previous sequence of frames is similarly adjusted. The closer the corrected frame is to the past event of entering the frame, the less adjustment occurs (see Fig. 7).

The Loop Closure mechanism is an extremely important element when building a reconstruction. In the case of a constantly moving camera, the error occurs regularly and tends to accumulate. Without the work of such an algorithm, the final reconstruction very quickly begins to break down and loses its appearance with which it can actually be worked and, in general, adequately perceived.

The essence of this algorithm is a certain assumption and approximation of the previous data series. Thus, objective restrictions are formed on the degree of reconstruction accuracy regarding the size and accuracy of angles. At the current stage, we do not see any possible ways to ensure extremely high accuracy, comparable, for example, to manually collecting size data with a laser tape measure.

This is an important feature that must be kept in mind when choosing areas of application for such a development. At the current stage, these methods should be used where an error of 5-10 millimeters is acceptable.

An example of constructing a three-dimensional space is the following data:

Sequence of color static frames, 19961 pieces

Sequence of black and white images with a color depth of 16 bits, 19961 pieces

A sequence of device movements containing a timestamp, frame number, position in a three-dimensional coordinate system, quaternion rotation (hereinafter referred to as pose)

About your device's camera settings

An example of a dataset element, frame No. 10 (see Fig. 8):

Also, in addition to the above, the dataset contains:

Depth Format (Z16)

Width and height of the depth map (192×256)

Depth map scale value (1000)

Matrix parameters:

Focal length (475.8907)

Coordinates of the main point of the image (0,0)

Affinity and non-orthogonality coefficients (0, 475.8907)

Radial distortion coefficients (0, 322.10426, 240.97993, 1)

The number of frames per second at which the images were taken

Using this data set (hereinafter referred to as the dataset), we collected an initial test version of the system.

To transfer data from the real world to the virtual scene, we need two tools - conversion of coordinate systems, and compliance with units of measurement.

Based on the fact that the coordinates and rotation angles were recorded in the coordinate system of the Unity scene, we assumed that it was enough for us to install a virtual camera on the virtual scene in the pose recorded from a real device, neglecting differences in coordinate systems. This satisfies the first need, then we proceeded to compare units of measurement.

We learned that the LIDAR depth map encodes depth information in a gradient, where black represents the furthest distance from the camera and white represents the closest proximity to the camera. According to technical data, the maximum range of LIDAR readings on the iPhone is 14-5 meters. Knowing this information, we decoded the pixel values into the distance of the points from the virtual camera. The decoding algorithm looks like this:

Get a set of data corresponding to one frame,

Place the virtual camera in a pose recorded from the device,

Take a frame from a color camera and a depth camera,

For each pixel of the image, create a point,

Based on the pixel color of the depth map, set the Z-axis distance for the point,

Based on the pixel position inside the images, set the X and Y position as follows:

We create an abstract instance of a point and place it in temporary storage

Since the image points are stored in a one-dimensional array, we create two-dimensional coordinates using the following formulas:

Where xPos, yPos are the desired two-dimensional coordinates, - pixel index inside a one-dimensional image array, HV - image height,

We convert the depth value into a numerical representation using the current processed pixel of the depth map

We check for the presence of noise in values neighboring the current pixel; if neighboring pixels do not have a value other than zero, then we ignore this pixel

We check the pixel value on the depth map to see if the value is zero, and whether the value fits into the filtering threshold parameter, and if true, we create a three-dimensional representation of the point using the formulas:

Where - coordinate of a point in the space of the virtual scene,

- depth value,

- the main point of the image,

- focal length,

- three-dimensional coordinate of a point on the scene,

- three-dimensional coordinate of the camera on the scene,

- relative vector directed upward,

- relative vector directed to the right,

- relative vector directed forward.

Based on the pixel color from the color image, set the color of the point,

Send the resulting point to the visualizer, repeat for all pixels,

Thus, we have assembled the first version of the depth determination mechanism. Using SLAM technology on the test device allowed us to ignore differences in the coordinate system and synchronize the position of the device in the real world and inside the virtual scene. Because upon initialization, ARK.it localizes the device's position relative to the ground, and sets the initial position. In the coordinate system, the position recorded by ARKit has a Y value (the height of the device above the ground), and a negligible value in X and Z (close to zero). Therefore, the position in the real world receives a zero coordinate - the ARKit initialization location, which can also be a zero coordinate in the virtual scene.

Development of a mechanism for identifying and filtering noise, as well as three-dimensional reconstruction based on a server part equipped with a video accelerator

Having tested the prototype of the depth determination mechanism, we discovered that the data obtained from the depth map contained a large number of useless points. This is due to the LIDAR operating mechanism. Since LIDAR is an optical system, surfaces located outside the radiator's coverage area are not read. The mechanism of operation of LIDAR is determined by the reflection of radiation in a refracted medium, and the receipt of radiation back to the receiver. Since surfaces located further than the radiator coverage area do not have the ability to reflect light, these surfaces are not registered and are marked in black on the depth map. These unregistered surfaces can be considered noise.

We supplemented the algorithm with a filter that eliminates points that are completely black in order to initially get rid of noise. Within the processing loop, such points are skipped until the loop reaches points that are shaded white.

The pre-processed data set, as well as a test mechanism for localizing points inside the virtual scene, were ready, so we set about visualization.

First of all, we have a test dataset of 39,922 images. Each image has a resolution of 640x480, therefore, each image contains 307,200 pixels, and two images need to be processed, therefore, 614,400 pixels need to be processed in 1 frame. The video stream must send frames at 60 frames per second, which means we need to process 36,864,000 pixels per second. In total, we get very serious requirements for system performance and processing speed.

Add a table in which how much data appears over a period of time, 4 lines - how many pixels, points, how many images were used

Having these requirements, we chose a particle system as a visualization tool. Traditionally, for interactive systems, a particle system is used to render a large number of points, since particle systems were designed to create and manipulate large numbers of graphical elements. First of all, we tried out the standard particle system included in the Unity kit. First of all, we were interested in the maximum available number of points that we can draw using a particle system, because visualizing point clouds, to ensure density and display quality, requires on average at least 1 million points. Experiments showed that the standard particle system cannot cope with such a number of elements, and also does not provide the ability to control individual points, after which it was decided to abandon its use.

We tested a visualization engine using Unity VFX Graph. This component was created to implement complex special effects within Unity projects. Being a particle system at its core, VFX Graph supports working with a large number of points without over-taxing the device's resources. In addition, VFX Graph was originally created to work with complex calculations, so it has functionality for accessing the graphics buffer.

The decoding algorithm sent data to a test version of VFX Graph, creating points, giving them a position and color. This option performed best, maintaining a high FPS (-400) and a large number of points (307,200 points).

Thus, we chose VFX Graph as the visualization subsystem.

Initially, we sent data directly to VFX Graph, but this created a limitation on the processing speed of the data. To avoid this problem, we created a compute shader. We moved the decoding algorithm inside this shader. This scheme made it possible to free the main computing thread of the application from heavy calculations and operate at the graphics core level. The decoding algorithm described in Chapter 1 has been transferred entirely, without changes, adjusted to the syntax of the HLSL language.

Next, the question arose of how to avoid repeatedly placing points on the same coordinate. Since we are working with a video stream, each subsequent frame will most likely differ little from the previous one. Because of this, in the process of decoding and placing points, there is a disadvantage - for one point in space there may be other duplicate points. These duplicates are noise and need to be filtered out, since these points repeat useful information and are not useful, loading computing resources. Our system requires all the maximum available computing resources, and by cleaning these duplicates, we free up more resources for the payload.

Since we are limited by performance and amount of data, we cannot use classical methods (singular decomposition, nearest iterative point method). Additionally, we need to be able to interact with points after the fact without having to perform complex calculations.

However, in the process of familiarizing ourselves with academic materials, we discovered material devoted to voxel hashing using an octree. This technique dealt with both factors, and we decided to use octrees. The first reason to use octrees is the ability to store information in a structured way that allows you to quickly access a specific point. Based on the first reason, we can interact with the processed data, which will be required for further creation of tools. Also, the octree structure requires less memory overhead than other available storage methods. Taken together, the use of octrees allows us to work within constraints.

Work has begun on implementing the octree mechanism into our solution.

First of all, a division of space into blocks was created, to which a coordinate was assigned in integer form. This allows us to obtain, from these coordinates, an area in memory that is responsible for a strictly defined area containing pointers to descending levels of the octree. Next, since each block stores a pointer to a strictly defined octree, we take the point obtained as a result of decoding and round its coordinate to an integer. Coordinate rounding allows us to localize the block most suitable for a point in space in the most efficient way, since the coordinate of the point and the indexing of the blocks will coincide.

Initially, we divide the space into volumes (chunks) of arbitrary size, thereby we obtain a region of space that includes the coordinate of the point. The next step, since we have the most suitable block, is to place the point in the octree belonging to the suitable block. This happens thanks to a recursive function that divides the space in half and checks which half the point ends up in. By dividing the space, we obtain the levels of the octree, recursively dividing until the space most closely matches the position of the point. Having reached the lowest level, we check whether there is allocated memory in this region, and if not, then we mark up the memory region and place a point there.

Thus, an algorithm for placing points was implemented, and one of the noise reduction methods - when filling a block, we do not fill an already filled point, and thus perform secondary filtering from noise.

During testing, the results of which are described in the last chapter, another serious problem emerged, the emergence of which we did not take into account before starting work - the appearance of noise and distortion of depth data when scaling the depth map, associated with the use of various image scaling methods.

Since these problems are quite serious, we began developing an algorithm to increase the resolution of the depth map. Initially, the resolution of the LIDAR depth map is 256x192 pixels, and in order to be able to combine the color image and the depth map, we upscale the depth map using the neighbor pixel method to a resolution of 640x480. This resolution is optimal because it allows you to maintain a balance between the quality of the image and the amount of data contained in it. Obviously, the neighbor pixel method is not able to provide a depth map of sufficient quality, so we began research to improve the situation.

We carried out an analysis of existing solutions for image enlargement. Previously, we processed images manually using the Converseen program. The process looked like this:

Loading depth maps into Converseen

Specify the parameters - format and final resolution of the images

Let's start the conversion

Despite the simplicity of the process, Converseen did not provide the required flexibility in setting up image enlargement methods and required manual processing, which ultimately led to unsatisfactory results in testing and slowed down the preparation process. Also, Converseen interpolated the edges of contrasting areas in the image, which ultimately produced a rounded geometry in the scan that did not match the real object.

Therefore, the next step was research on options for mechanisms built into the solution that allow you to enlarge the image.

First, we considered OpenCV as a provider of image enlargement methods. However, since using OpenCV required a wrapper that did not include the methods we needed, this option was rejected.

Next, we looked at the image enlargement tools built into .NET. These facilities were provided within the System.Drawing namespace. This method did not suit us, because in practice, this option turned out to be very slow and provided a limited set of tools.

Finally, we looked at the Magicklmage library, connected using a wrapper for the C# language, namely the Magicklmage.Scale() method, which averages the average pixel value taking into account neighboring ones, thus forming the final interpolated image.

The latter method showed the greatest efficiency, since it made it possible to control the degree of interpolation and perform processing in real time, in parallel with the work of the main mechanics. Also, the method showed high performance, which is one of the most important conditions for the operation of our solution.

Real-time 3D model rendering engine

We designed and created important parts of the system - data decoding and visualization, and then began real-time visualization.

Since at this stage we were working with a dataset, and not with a live stream, a system was assembled that emulated sending a data stream from a portable device. The stream emulator contained a dataset, mechanisms for starting and stopping playback, as well as virtual camera control. Initially, we set the camera position corresponding to one frame, but now having a stream emulator, we changed the position of the virtual camera according to the pose of the device. To track the path traveled, a graphical representation of the trajectory in the form of a line was added.

By running the real-time stream emulator with point decoding disabled, we obtained a trajectory that coincided with the actual movement of the device while recording the dataset. This confirmed our hypothesis about position synchronization by SLAM. The measurements of the segments of the resulting trajectory on the stage corresponded to the actual dimensions of the object in meters.

Having received confirmation of the functionality of the stream emulator and its mechanics for emulating the movement of the device’s camera, we began processing data in real time. Initially, we rendered the VFX Graph every 500 frames, resulting in the result shown in Fig. 10.

At the same time, we added noise reduction based on the exclusion of neighboring points located at a sharp distance from each other, see Fig. 11.

Satisfied with these results, we reduced the frame interval until, as a result of algorithm optimizations, we began to process every frame.

Development of a prototype for visualizing user movement in physical space

After implementing the mechanism for rendering a three-dimensional model in real time, it was decided to add visualization of the user’s movements in the real world, on a virtual stage.

The rendering mechanism involves moving the virtual camera according to the position of the scanning device, therefore, it would be logical to rely on the virtual camera.

First of all, a mechanism was added to the virtual camera that draws a linear trajectory of movement. First of all, the Line Renderer component was tested.

Line Renderer is part of the standard components of the Unity engine, and is designed for drawing lines. This component contains:

Setting the position and thickness of intermediate points

Setting the line material

Adjusting the position and thickness of intermediate points allows you to create complex shapes by creating a record of an intermediate point, indicating the coordinate position of this point. The principle of operation is similar to creating Bezier curves, however, unlike a Bezier curve, Line Renderer does not smooth the trajectory; each intermediate point is connected in a straight line.

The material setting allows you to select the color and characteristics of the line. For rendering, the component uses flat sprite images rotated perpendicular to the camera angle to create a simulated volume. This approach allows you to save graphic resources by not drawing 3D line models, since to create intermediate points, a 3D model would require adding polygons and deforming them. The amount of information contained in a 3D model of a curved line exceeds the amount of information required to construct a sprite curved line.

Line Renderer was chosen as a visualization tool because it met the following criteria:

- low resource requirements,

- possibility of flexible configuration,

- the ability to create intermediate points when changing the trajectory.

Next, to work with the Line Renderer, a script called CameraTrajectoryRenderer was created, which controls the creation of intermediate line values. The algorithm for this script is as follows:

When initialized, the script subscribes the line update procedure to the line update event.

The line update method checks that the value of the current frame is a multiple of 50, and then inserts a new intermediate point into the Line Renderer component, under the index number, and with position coordinates obtained from the current value of the dataset sample. Then, the index number is incremented, after which the process is repeated when the event is triggered.

As a result, we received a drawn trajectory of the user's movement.

Next, to obtain visual information about the current angle of rotation of the camera, it was decided to attach a three-dimensional model, schematically depicting a person, to the camera object. This model has the average height of an adult (1 m 80 cm), and the position of the eyes is schematically indicated. The position of the model is tied to the position of the camera, where the center of the model is the schematic “eyes”.

Thus, we were able to obtain information about which direction the virtual camera is looking in the context of the scene.

Development of functionality for client-server interaction between the visualized space and the real world

Having ready tools for calculating and visualizing the model inside the virtual scene, the next step was to add client-server interaction, as part of the task of creating a technological solution for next-generation communication and collaboration.

Since such a step requires interaction tools, and not just visualization, we added the following functionality:

Mechanisms for saving a rendered model

One of the important criteria for the usefulness of our technological complex is the ability to create, save, download and process information. For this, we have created mechanisms for saving the point cloud obtained as a result of scanning.

First, a mechanism was created to save the resulting cloud in the PLY format. This format is a standard for storing point clouds and platform-independent 3D models, and also does not require proprietary modules for operation.

A script was created that had the following structure:

- link to the current reconstruction management script,

- a variable responsible for the name of the saved file,

- a variable responsible for the system path of the saved file,

- variable responsible for the scaling factor of the target point cloud,

- a method responsible for saving the point cloud in text format,

- a method responsible for saving a point cloud in a binary format.

Initially, a scheme was implemented to save the file in text format, in ASCII 1.0 encoding. This was necessary because the binary format stores data in a form that is not human-readable, and we needed to track anomalies and errors that occur when saving a point cloud. The structure of the PLY file was studied, and it was found that the PLY file consists of a series of header data, followed by an array describing the structure of the 3D model.

The following headers have been added to the method:

The string "ply", which means that this file is in PLY format

The string "format ascii 1.0" indicating the encoding of the target file

The string “element vertex” with the addition of the number of points already calculated by our reconstruction system. This line allows third-party applications to expect a certain number of 3D mesh structural elements

Next comes a series of properties that describe the structure of the array. These properties are X, Y, Z coordinates, as well as color, in the format R, G, B.

The last header is the line "end_header", indicating the transition to the structure array.

Next, follows the structure of the 3D mesh in the format:

X coordinate, Y coordinate, Z coordinate, R value, G value, B value

The data is filled in a cycle, the size of the number of calculated points. The coordinates are filled in according to the formula:

Where - coordinate value along one of the axes,

shift coefficient along one of the axes, D - scaling factor coefficient

After filling in the coordinate information, the color parameters of the point are filled in. To comply with PLY format color standards, the 32-bit color value obtained from the color frame is converted to a single-byte, per-channel value, and then written. For the convenience of the user, the time spent on saving is counted.

The algorithm for saving the model through a text format showed its efficiency; a point cloud consisting of 11,744,136 vertices was saved in 33.4 seconds, occupying 285 megabytes on disk.

However, due to the fact that information in text form takes up more space, it was decided to implement a mechanism for saving the file in a binary format. The algorithm of this mechanism is identical to the mechanism for saving in text form, with the difference that for binary saving, the values were first converted into binary form. When testing the finished mechanism, a similar amount of data was stored in a similar time, but it took up approximately 140 megabytes on disk, which is half as much as in the text format.

Next, a mechanism was implemented to save the scanned point cloud for subsequent visualization within our solution.

Given an array of octree indices, we write it to a binary file in a format readable by our solution. This saving method allows us to continue working with previously scanned space, for example, to continue reconstruction scanning. Also, since this method works directly with the data format used within the application engines, we eliminate the possibility of introducing interpretation errors. The disadvantage of this method is the inability to work with saved data in third-party applications.

Next, a mechanism for measuring distances on a scanned point cloud was implemented. This mechanism was implemented in order to be able to verify the value of size readings relative to real objects. Considering that our application is intended for use in the construction industry, the mechanics of measuring distances is integral to this area.

The measurement mechanism was implemented by using an octree structure to store the coordinates of points, it works as follows:

An iterative reprojection of the octree is created and populated by the Box Collider component.

Using the Raycast mechanism, we find the smallest octree level containing points.

An iterative reprojection is created for each point inside the resulting chunk, filled with the Box Collider component.

Using the functionality of the Raycast mechanism, we obtain the coordinate of the collision of the ray with the desired point.

This mechanism works twice, first the user places the first point of the measurement segment, then places the second point. Having two coordinates in space, the distance between them is calculated using the Vector3.Distance function, the result is graphically displayed on the drawn segment.

So, having a set of tools to interact with the application, we began to implement client-server communication. We chose TCP as the protocol for transmitting events and coordinates. The decisive factor for choosing TCP was the need to use different platforms and different implementation languages. The TCP protocol is a multifunctional solution for networking a wide variety of platforms.

We chose the Agora solution as the video stream transmission protocol. First of all, TCP socket implementations were written for the server part and applications for the phone. The TCP socket, inside our solution, is responsible for transmitting events and data. Since we want to build in the ability to create tags in the server application that will then be seen by the user on the portable device, we need to have a system of events that would be sent to the remote device, interpreted by the portable solution as “create a tag object” or “delete a tag object”. Together with the creation/deletion event, we need to pass the current coordinate of the label on which to place the label. Thus, the TCP stack on the server sends a data packet of the following type in real time:

(label name), (create\delete\update), (coordinates), (rotation angle)

This packet is received and interpreted by the algorithm inside the portable application. As described earlier, the coordinates inside the real world and the coordinates in the virtual scene are the same, so we send coordinates according to the user's location of the object in the virtual scene. The label name is necessary so that the algorithm has a link to a specific label (if it already exists), or to create a new label with the ability to link to it. The event defines the operation to be performed on the label - in case of creation we place the label, in case of deletion we delete the label, in case of update we update the coordinates of the label. Next are the coordinates and rotation angles that are applied to the label object on the remote application side.

Agora is used to transmit a stream of video frames from a color camera and a depth camera. These frames enter the reconstruction system and are used according to the algorithms described above. Via a TCP socket, the device sends camera parameters (position, rotation, matrix parameters) to the server, and this data is subsequently used as described above.

When activating the Mark tool on the server, the user is required to click on the reconstructed model, placing the mark where necessary. Next, the mechanism sends a string via the TCP socket containing the event of creating a new label under a specific name, and with coordinates corresponding to the position of the click. This string is received by the device and processed to create a tag with a specific name.

The user can also edit the position of an already placed mark. To do this, the first click selects the mark, and the second click determines its new position. After the second click, the TCP socket sends a label update event under the name obtained after selecting the label. This event is received by the device and interpreted as a change in the position of the selected mark, giving it new coordinates.

In addition to creating and updating, the user can delete a label. To do this, you need to select an already placed label, and then click on the delete button. The TCP socket sends a delete event and the name of the selected label. The device receives this event and deletes this tag.

System prototype testing

Having a ready version, we started testing.

Reconciliation of measurements between the real world and reconstruction

We chose a testing ground in the form of a staircase landing. Also, in the real world, a ruler 30 centimeters long was placed. As a result of the reconstruction, we measured the length of the ruler with a virtual instrument, and the result was 30 centimeters with errors of less than one thousandth of a meter, see Fig. 12.

Based on the results of the work done, we were able to confirm all the starting hypotheses. We have proven that it is possible to implement an accurate 3D reconstruction system in real time using low-resolution LIDAR, which is located in the iPhone family of devices.

At the same time, a significant problem was solved - the device dynamically moves in space and each frame requires calculating its exact position for reconstruction. Solving this problem allowed us to create a tool that allows us to obtain a three-dimensional copy of any object or room without expensive equipment or specific skills in photogrammetry or the use of industrial scanners. Moreover, in the process of implementing the solution, we discovered new approaches to constructing point clouds and working with this data, which allowed us to create the ability to regulate the resolution of the final scan. We can now also tailor accuracy to the different needs and computing capabilities of end devices.

The created solution has no analogues in terms of accuracy and ability to work in real time. We surpass them both in terms of speed and accuracy.

The prospects and new tasks are already clear - perhaps, based on the existing solution, both building an accurate mesh (3D model), and using AI, turning the created point clouds into 2D or 3D plans for architectural programs.

Claims (12)

1. A system for constructing a three-dimensional space model, containing: - an image capture device that moves in three-dimensional space to obtain an image of three-dimensional space in time, - a three-dimensional image acquisition interface receiving image data from the image capture device, and - a three-dimensional space modeling system that processes three-dimensional space image data obtained by an image capture device in time, characterized in that The image capture device is made in the form of an IOS phone with LIDAR, before constructing a three-dimensional space, the optical part is calibrated cameras and lenses of the image capture device, The camera of the image capture device simultaneously captures RGB color data, depth data, IMU data and odometry data for each frame of the video image, and the three-dimensional space modeling system translates color data in RGB format and depth data into a pixel matrix, each element of which is assigned a position value based on the depth values, and an index of the element within the matrix, after which the resulting coordinates of the points are distributed throughout the virtual scene, based on the processed data element, as well as IMU and odometry data, during the distribution of points on the scene, an octree structure is created that serves to store points in space, with the ability to quickly access specific points, then, based on the filled octree, visualization elements are created that display color and position of the points, and according to the data received: calculates the exact position for each frame received by a phone moving in space for subsequent reconstruction of three-dimensional space. 2. The system according to claim 1 of the formula, characterized in that when calibrating the optical part of the camera and the lens of the image capture device, the parameters of the matrix and lens are calculated to determine the distortions of the incident light.