RU2736628C1 - Method and system for rendering 3d models in a browser using distributed resources - Google Patents

Abstract

FIELD: image processing means.

SUBSTANCE: invention relates to means of performing 3D rendering (visualization) on user configurations in conditions of limited computational resources. At least one three-dimensional model is obtained on the server. Each object in the three-dimensional model is broken down into an array of triangles and placed into the objects storage by its unique identifier. A spatial index is formed in the database of all objects from the obtained three-dimensional model. A visibility pyramid is formed, which consists of a predetermined number of cubes and a request is sent to the server to obtain a list of objects included in the given cubes. A list of objects for visualization is formed by primary rejection of three-dimensional objects according to the method of clipping by entering into the requested cubes at the server side based on the spatial index of each object. Objects are sorted from the found list from the larger to the lower based on a predetermined criterion. An array of unique identifiers of the sorted objects is sent to the user's browser from the server. Objects are rendered based on unique identifiers obtained at the previous step.

EFFECT: high speed of processing and visualizing 3D objects.

13 cl, 8 dwg

Description

FIELD OF TECHNOLOGY

[001] This technical solution generally relates to the computing field, and in particular to methods and systems for performing 3D rendering (rendering) on custom configurations in conditions of limited computing resources.

TECHNOLOGY LEVEL

[002] Currently, modern solutions in computer-aided design (CAD) systems, architectural modeling, computer games development, VR and AR technologies require interactive visualization of dynamic data of very large volumes, orders of magnitude exceeding the available computing power.

[003] An Autodesk online viewer (https // viewer.autoriesk.com /) is known in the art, which allows you to view 2D and 3D CAD files. This solution works very well with 3D in a browser. However, the optimization vector is based on compression of the original model, and since the compression techniques of models have a finite limit, this approach does not work on large models (the number of objects in the model (s)> 1 million).

[004] Also known from the prior art patent No. US 10339696 B2 "On demand geometry and acceleration structure creation with discrete production scheduling" (patent holder: Imagination Technologies Ltd, publication date: 02.07.2019). This technical solution discloses geometry processing systems and methods for rasterization and ray tracing processes that involve preprocessing the source geometry, for example, by ray tracing or other procedural modification of the source geometry, to create the final geometry on which the rendering will be based. The acceleration structure (or part of it) for use during ray tracing is determined based on the final geometry. Only the coarse-grained elements of the accelerating structure can be rendered or stored, and the fine-grained structure in a particular coarse element can be obtained in response to a set of beams ready to pass through the coarse element. The target geometry can be recreated in response to requests from the rasterization engine and from ray intersection units that require such geometry to test intersection with primitives.

[005] Also known as a source of information US 8593459 B2 "Tree insertion depth adjustment based on view frustum and distance culling" (patent holder: International Business Machines Corp, publication date: 11/26/2013). This solution discloses a method of using a hierarchy of bounding volumes for ray tracing, in which the distance between the bounding volume of a plurality of bounding volumes and a reference is determined on a computing device, in which the plurality of bounding volumes contains divided portions of image data, and wherein the data structure includes a plurality of nodes corresponding to a set of bounding volumes; setting, in accordance with the distance, a threshold value of the end node for the node corresponding to the limiting volume, the threshold value of the end node being the number of primitives to be stored in the node before splitting; and resizing the bounding volume based at least in part on the distance and based at least in part on the determination that the number of primitives to be placed in a node corresponding to the bounding volume exceeds the target node's threshold.

[006] However, prior art solutions are slow and computationally intensive when running in a web browser on a client.

SUMMARY OF THE INVENTION

[001] This technical solution is aimed at eliminating the disadvantages inherent in solutions known from the prior art.

[002] A technical problem or a technical problem solved in this technical solution is the creation of a technical solution that allows visualizing and rebuilding 3D models, to carry out user interaction with 3D models in real time. This solution should provide the ability for the user to work with models that cannot be fully loaded onto the user's equipment (for example, a personal computer, or a tablet, or a mobile phone).

[003] The technical result achieved by solving the above technical problem is to increase the speed of processing and visualization of 3D objects.

[004] An additional technical result is the provision of working with 3D models on one scene, consisting of a large number of original CAD models (files) and a large number of 3D objects (more than 1 million).

[005] Also, when implementing this technical solution, server capacities are effectively used, including due to horizontal scaling. Thus, a user working on a desktop / tablet / phone uses one or more servers to calculate the rendering of a 3D scene. All high-load operations are transferred and processed from the client to the server side.

[006] The specified technical result is achieved by implementing a method for rendering 3D models in a browser using distributed resources, performed by at least one computing device, in which at least one three-dimensional model is obtained on the server; break each object in the 3D model into an array of triangles and place it in the object storage by its unique identifier; form a spatial index in the database of all objects from the obtained three-dimensional model; form a pyramid of visibility, consisting of a predetermined number of cubes and send a request to the server to obtain a list of objects included in these cubes; generating a list of objects for rendering by first culling three-dimensional objects using the clipping method by entering the requested cubes on the server side based on the spatial index of each object; sorting objects from the found list based on a predetermined criterion; sending an array of unique identifiers of sorted objects to the user's browser from the server; render objects based on the unique identifiers obtained in the previous step.

[007] In some embodiments of the technical solution, at least one three-dimensional model is obtained on the server, downloaded by the user through a web browser.

[008] In some embodiments of the technical solution, at least one three-dimensional model is obtained on the server, loaded by the user from a CAD file.

[009] In some embodiments of the technical solution, at least one 3D model is obtained on the server in RVM / ATT or IFC format.

[0010] In some embodiments of the technical solution of the obtained three-dimensional model, objects are represented in the form of a tree.

[0011] In some embodiments of the technical solution, each object in the resulting three-dimensional model contains a unique object identifier and / or a reference to the parent object, and / or a spatial transformation matrix relative to the parent object, and / or materials of the object, and / or 2D / 3D geometry object, and / or properties of each object (attributes).

[0012] In some embodiments of the technical solution, as a spatial index, a partition of the entire area of space with a fixed step for each dimension is used, or octal trees, or kD trees, or BSP trees, or BVH (Bounding Volume Hierarchy), or Interval Tree, or R-trees.

[0013] In some embodiments of the technical solution, the spatial index is generated as follows:

a. for each object of the three-dimensional model, find its transformation matrix relative to the origin, taking into account the tree structure of the three-dimensional model;

b. for each geometry of the object, a transformation operation is applied, obtaining the geometry of all objects relative to one origin in world coordinates

c. the resulting transformed geometry is placed in the database by its unique identifier;

d. for each object with the transformed geometry, a spatial index is calculated and placed in the database.

[0014] In some embodiments of the technical solution, a request for display is sent to the server, which is a JSON array of coordinates.

[0015] In some embodiments of the technical solution, when an array of unique identifiers of the found sorted objects is received from the server, this list is compared with the available cache of objects, and

• all UID objects missing from the cache are marked as new;

• objects that are present in the cache of objects, but are not in the truncated pyramid of visibility, are marked as obsolete and removed from the cache;

• for new objects, background loading of materials and geometries into the corresponding caches is performed.

[0016] In some implementations of the technical solution, only those objects that are present in the cache are rendered based on unique identifiers.

[0017] In some embodiments of the technical solution, when sorting objects, a predetermined criterion is the angular size of each object and / or the distance from the camera and / or the occupied volume and / or area of the object.

[0018] Also, the specified technical result is achieved by implementing a system for rendering 3D models in a browser using distributed resources, which contains at least one server configured to receive at least one three-dimensional model from the client; splitting each object in a three-dimensional model into an array of triangles and placing it in the object storage by its unique identifier; forming a spatial index in the database of all objects from the obtained three-dimensional model; generating a list of objects for visualization by first culling three-dimensional objects using the clipping method by entering the requested cubes based on the spatial index of each object; sorting objects from the found list from largest to smallest based on a predetermined criterion for sending an array of unique identifiers of sorted objects to the client; at least one client configured to send at least one three-dimensional model to the server; forming a visibility pyramid consisting of a predetermined number of cubes and sending a request to the server to obtain a list of objects included in these cubes; rendering objects based on unique identifiers obtained earlier.

BRIEF DESCRIPTION OF DRAWINGS

[0019] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0020] FIG. 1 shows the general scheme of the method of rendering 3D models in a browser using distributed resources.

[0021] FIG. 2 shows an example of the implementation of loading a 3D model to the server from the client.

[0022] FIG. 3 shows an example of the implementation of triangulation of objects of a three-dimensional model and assignment of a unique identifier to each object.

[0023] FIG. 4 shows an embodiment of a system for rendering 3D models in a browser using distributed resources in the form of a block diagram.

[0024] FIG. 5 shows an example of the implementation of the selection from a three-dimensional model of a set of various information and storage in various data stores.

[0025] FIG. 6 shows an example of the implementation of the formation of a list of objects for visualization by means of primary culling of three-dimensional objects using the clipping method along the visibility pyramid on the server side.

[0026] FIG. 7 shows an embodiment of dividing the truncated visibility pyramid into n cubes.

[0027] FIG. 8 shows a block diagram of rendering (rendering) of a three-dimensional scene and working with the cache on the client.

DETAILED DESCRIPTION OF THE TECHNICAL SOLUTION

[0028] This technical solution can be implemented on a computer, in the form of a system or computer-readable medium containing instructions for performing the above method.

[0029] The technical solution can be implemented as a distributed computer system.

[0030] In this invention, a system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, well-defined sequence of operations (actions, instructions).

[0031] By a command processor is meant an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

[0032] An instruction processor reads and executes machine instructions (programs) from one or more storage devices. Data storage devices can be, but are not limited to, hard disks (HDD), flash memory, ROM (read only memory), solid-state drives (SSD), optical drives.

[0033] The following will describe the concepts and definitions necessary for a detailed disclosure of the implemented technical solution.

[0034] Model (3D model) - an initial three-dimensional model (consisting of 3D objects), represented in one or more of the existing 3D formats (Autodesk FBX, etc.), in relation to which the task of interactive 3D visualization is being solved.

[0035] Mesh (Geometry of an object) - a three-dimensional mesh, a collection of points and faces which describe the geometry of the corresponding 3D-object.

[0036] Layout of the model (Layout) - a set of data describing the list and hierarchical nesting (parent-child) of 3D-model objects, links to geometry (in the case of batches - to the geometry of the reference object) in the form of UUIDs or other identifiers, and their parameters spatial transformation relative to parent objects (transforms from local to parent coordinate system).

[0037] Spatial transformation matrices (Transforms) are 4 × 4 numerical matrices that describe transformations over the spatial position of an object: shift, rotation, scale.

[0038] Client part (client, client renderer, client viewer) is a client part of a software solution executed in a web browser (JS), WASM, etc., but not limited to, and is responsible for interactive 3D rendering of the original model.

[0039] Registry of objects to be drawn - a list of objects containing only objects visible in the current frame that are to be drawn on the screen. It is the result of performing OS calculations.

[0040] A communication channel (web-socket, http, https, grpc, etc.) is a distributed part of a software solution (used on both the client and server side of the solution) used to exchange information about the state of 3D objects and the scene in general.

[0041] State information - a set of data describing the current position of the camera in the frame, screen resolution, window size (in which rendering is performed), parameters for clipping and culling objects, as well as spatial displacements of objects from their previous position in the form of transforms (if objects moved).

[0042] Read Buffer - A buffer in GPU memory used to draw the current frame. Each drawing (graphics pipeline) is performed based on the information stored in this buffer (continuous loop).

[0043] Write buffer is a buffer in GPU memory, which is used to prepare a new read buffer in case the scene in the "current" frame has changed (objects moved, the camera turned, etc.). After the write buffer is fully prepared, rendering switches to it (it becomes a read buffer), and the previous read buffer is used later as a write buffer (swapped).

[0044] A batch of objects (Batch, instances, instances) is a grouping of objects of identical geometry (ie, which differ only in different shifts / rotations / scales) into one batch, for which a description of the geometry of the reference object is set, and the rest are set in the form transforms from an object (i.e. a reference object) to world space.

[0045] Object-Oriented Bounding Box (OBB, OOBB, Object-Oriented Bounding Box) is one of the variations of the bounding volume technology used to simplify auxiliary computations in 3D visualizations by performing tests (calculations) on a simplified representation of objects (used in cases when calculations on the exact geometry of objects take an unreasonable amount of time / resources).

[0046] 3D-rendering (rendering) - the creation of a digital raster image on the developed three-dimensional scene using special computer programs. The term is used in artistic and technical computer graphics.

[0047] Key-value storage - a database based on key-value pairs is a type of non-relational database that uses a simple key-value method to store data. A key-value-pair database stores data as a collection of key-value pairs in which the key serves as a unique identifier. Both keys and values can be anything from simple to complex composite objects. Databases using key / value pairs are highly separable and provide unprecedented scale-out not possible with other types of databases.

[0048] Step 110: obtain at least one 3D model on the server.

[0049] Initially, the user of the technical solution initiates the download of a 3D model 230 (sometimes a 3D model) as shown in FIG. 2, in the web browser of the client 210 to the server 240 or via the user's mobile communication device, for example, via a mobile application via the data network 220.

[0050] The 3D model 230 can be loaded in RVM / ATT, IFC, etc. data formats, but not limited to. This technical solution supports many different file formats (.dwg, .dgn, .ipt, .rvt, .rvm, etc.), which allows you to assemble a consolidated 3D model using models downloaded from different CAD systems (CAD or CADD , computer-aided design system) in their own formats.

[0051] In some implementations, 3D model 230 may be partially displayed as content is loaded. If there are textures, the original model can be displayed with and without them, with the model structure or not - this is all depending on the view template.

[0052] In some implementations, WebGL technology may be used to download and view 3D model 230 on client 210.

[0053] Server 240, in one embodiment, may include separate servers or servers running on a data center virtual machine where multiple servers may be virtualized to provide the required amount of computation. Client 210 is located at the user's location to receive and process input and transmit it to servers 240, as well as receive data for rendering, which will be described below. Client 210 and servers 240 communicate over a communications network 220, such as, for example, the Internet. In other embodiments, client 210 may be any device, portable or non-portable, wireless or non-wireless, such that client 210 can interact with network 220 and provide display access to display 3D model 230 and download, and allow a user to input for possibilities of interactive control. In one embodiment, the client 210 is a thin client. However, in other embodiments, the client 210 may be a general-purpose computer, a special-purpose computer, a game console, a personal computer, a laptop computer, a tablet computer, a mobile computing device, a portable gaming device, a mobile phone, a set-top box, an interface / a streaming media device, high definition television or network display, or any other computing device that can be configured to provide the desired functionality of client 210 as defined herein. In one embodiment, the server 240 is configured to detect the type of client device being used by the user and provide a sense of the 3D model 230 corresponding to the user's client device. For example, picture settings, audio settings, and other types of settings on the user's client device can be optimized.

[0054] In various embodiments, the degree of processing performed by client 210 may vary with respect to input and output processing. However, in simple terms, the state of the 3D model is mainly stored and executed on the server 240, while the client 210 mainly functions to receive and transmit data entered by the user, as well as for final rendering at the final stage of the technical solution. The client 210 can be a stand alone device that is connected to the display and provides the ability to download and display on the display. In other embodiments, the client 210 may be embedded in a display. In one embodiment, the display is a network display, providing an operating system environment for programs or "applications" that use the display's network connectivity. In such an embodiment, the client 210 may be defined by an application running in an environment provided by the display operating system.

[0055] Step 120: Each object in the 3D model is split into an array of triangles and placed in the object store by its unique identifier.

[0056] Then, each object is split into an array of triangles and placed in the 3D object store by its unique identifier, as shown in FIG. 3. In other implementations, there may be four-point objects, etc.

[0057] Accordingly, a 3D object store is just one example of a suitable information store, and any suitable type of data store can be used. For example, a 3D object store may include a relational database, a hierarchical database, a networked database, an in-memory database, an object-oriented database, a data store, any other suitable store or database to maintain information, or a combination of these. ... Indeed, in some embodiments, the 3D object store may contain a single physical database, while in other embodiments, the 3D object store may be distributed across multiple different physical locations.

[0058] The division of objects into triangles is due to the following reasons:

• a triangle is the simplest polygon, the vertices of which uniquely define a face;

• any area can be guaranteed to be divided into triangles;

• the computational complexity of the algorithms for partitioning into triangles is significantly less than when using other polygons;

• implementation of rendering procedures is the most simple for the area bounded by a triangle;

• for a triangle, it is easy to determine its three nearest neighbors that have common faces with it.

[0059] The process of dividing a polygonal region with a complex configuration into a set of triangles is called triangulation. When analyzing or synthesizing complex surfaces, they are approximated by a grid of triangles, and subsequently operated with the simplest polygonal areas, i.e. with each of the triangles.

[0060] In some implementations, any surface of an object can be approximated with the required accuracy by a grid of triangles. The accuracy of the approximation is determined by the number of triangles and the way they are selected. For high-quality visualization of an object located near the observation point, it is required to take into account many times more triangles than in a situation when the same object is located at a distance. Even a coarse mesh is useful in practice, as the anti-aliasing techniques applied during the rendering can greatly improve the representation of the curvature of the surface. In some implementations, this technical solution may use Delaunay triangulation.

[0061] In general, the advantage of this technical solution is that you do not have to simplify the original 3D objects, instead, the data for the spatial index is calculated once. And for the calculation of Frustrum View it does not matter how detailed the 3D objects were, thus the problem of "high-quality visualization of the object" is not solved.

[0062] The three-dimensional model can be composed of a set of objects (eg, represented as a tree). Each object of the model can be represented in general terms:

a. The unique identifier of the object (usually a UUID), which is usually a random value;

b. A reference to the parent object, if one exists, i.e. a link means that an object stores information on its parent object (its unique identifier is UUID);

c. Spatial transformation matrix relative to the parent object (if a parent is specified, then all transformation operations occur relative to it). If the parent is absent, it is assumed to be a spatial transformation matrix relative to the origin;

d. Object material. A material can describe color, transparency, intensity, reflectivity, texture, etc .;

e. 2D / 3D geometry of an object in the form of a set of coordinates of the vertices of triangles;

f. Properties of each object (attributes). This can be the service data of CAD systems (date of object editing, author of the change, etc.), auxiliary attributes (type of construction, class of object, reference data, etc.).

[0063] For example, in FIG. 3 shows how the roof of a building as a 3D scene object has a unique object identifier UUID = 301, a wall has a unique identifier 302, etc. In a specific, rather than schematic, implementation, the identifier may take the value UUID = 00112233-4455-6677-8899-aabbccddeeff.

[0064] In the next step, a set of different information is extracted from the model and stored in different data stores, as shown in FIG. 5. For each format there is a parser that reads the CAD file information described above.

[0065] Step 130: generate a spatial index in the database of all objects from the obtained 3D model.

[0066] First, a spatial index is generated in the database of all 3D objects from the obtained 3D model. The spatial index is a standard technique known from the prior art, which allows to significantly reduce the complexity of queries about the relative position of objects in space, there are various types of so-called spatial indexes. A spatial index is some kind of data structure (most often hierarchical), usually built at the stage of stage preparation. As a spatial index, a partition of the entire area of space can be used (usually a bounding rectangular parallelepiped is described around this area, which is broken) with a fixed step for each dimension, octal trees, kD trees, BSP trees, BVH (Bounding Volume Hierarchy), Interval Tree, R-trees, etc., not limited to.

[0067] In a particular implementation, R-trees are used as the spatial index.

[0068] The spatial index of objects is generated as follows:

a. for each object of the three-dimensional model, its transformation matrix is found relative to the origin (world space), taking into account the structure of the tree of the three-dimensional model (i.e., taking into account all parents), which allows, instead of using the model tree, go to work with a flat list of objects where you need to take into account the hierarchy of the tree structure of the 3D model.

b. then, for each geometry of the object (each 3D object has a mesh, a description of its geometry, in a specific implementation it is a set of "triangles"), a transformation operation is applied, for example, shift, rotation or scale, which allows you to get the geometry of all objects relative to one origin coordinates (in world coordinates), for example, (0; 0,0). The resulting transformed geometry is placed in the database by its unique identifier;

c. for each object with the transformed geometry (in world coordinates), a spatial index is calculated and placed in the database.

[0069] Next, material data of each 3D object is placed in the material store 520 by its unique identifier, as shown in FIG. 5 through the operation of the parser 510.

[0070] Information is read by a parser 510 (sometimes a parser) from a CAD file on any 3D object and stored in an attribute store 530, for example, in the following form:

a. Material = steel;

b. Designer = Vasin A.A .;

c. Installation date = 19.12.2019.

[0071] For example, a 3D building model consists of a set of properties (attributes) of an object, it can be a wall, a column or a door, etc. An object property (or attribute) contains the necessary information about the object, which complements the geometric characteristics, and the parser 510 similarly analyzes the CAD file and stores all geometric information in the storage of 3D geometry 540. Consider, for example, the "Door" object - it has there is the height and width of the opening, the number of canvases (single-floor, one-and-a-half or double-floor), filling parameters (blind or glazed) - all these characteristics are determined by the geometry of the element. But this is not enough for a full description of the project. Doors, for example, can be fireproof and, in this case, it is necessary to indicate the fire resistance limit. This is the informational component, which is the metainformation of objects.

[0072] Step 140: form a pyramid of visibility, consisting of a predetermined number of cubes and send a request to the server to obtain a list of objects included in these cubes.

[0073] Step 150: a list of objects to be rendered is generated by first culling three-dimensional objects using a clipping method by entering the requested cubes on the server side based on the spatial index of each object.

[0074] Next, a list of objects to be rendered is generated by first culling three-dimensional objects using the server-side pyramid clipping method using the spatial index of all objects as shown in FIG. 6, which implies a priori exclusion of objects outside the camera's field of view.

[0075] In the prior art, it is known that for all vertices in the space of a three-dimensional scene, the bounding volume is a View Frustum (Frustum), and in the camera space the truncated pyramid is transformed into a unit cube.

[0076] Thus, there is no need to draw objects that are behind or to the side of the camera and do not fall into the field of view. Only those objects are drawn that are in the camera's field of view, or in the camera's field of view. This volume is a truncated pyramid, everything that is outside the pyramid is also outside the screen.

[0077] Based on the above, to determine the visibility of an object, it is necessary to check if the object being processed is inside the truncated pyramid, or multiply the coordinates of the object by the projection matrix and check if the object is inside the unit cube. In some embodiments, only simple geometrical figures consisting of several vertices are checked for visibility, and if the entire figure is out of sight, then the part of the scene enclosed in it can be discarded. The role of such figures (bounding volumes) can be figures known and used from the prior art, for example, a sphere, parallelepipeds (OOBB, AABB), etc. Thus, it is not difficult to organize cutting off of compact models (including dynamic ones).

[0078] Next, the pyramid is split into n cubes (for example, by 100) and a request is sent to the server to display (representing a JSON array of coordinates) only the n cubes data, as shown in FIG. 7. The larger n, the higher the accuracy of the approximate calculation of the truncated visibility pyramid.

[0079] The server has a service that finds the intersections of n cubes with 3D model objects. A spatial index is used to optimize this operation. The spatial index significantly speeds up the operation "does an object fall into one of n cubes". As a result, only those objects are obtained that have intersections with cubes, i.e. those objects that fall into the truncated pyramid of visibility. Computing the intersection by cubes is much more efficient than by the pyramid, i.e. we roughly calculate what falls into the truncated pyramid of visibility, lose in accuracy, but gain in speed.

[0080] In a particular implementation, for all of these resulting 3D objects from the intersection found from the spatial index, the server calculates an angular size, which is the size of the object that is observed on the retina or camera. The angular size (sometimes also the angle of view) is the angle between straight lines connecting the diametrically opposite extreme points of the measured (observed) object and the observer's eye, which takes values from 0 to 360 degrees. Angular dimension can also mean not the plane angle at which the object is visible, but the solid angle.

[0081] Step 160: sorting the objects from the found list from highest to lowest based on a predetermined criterion.

[0082] Next, objects from the found list are sorted from largest to smallest in angular size, and then the first ones are taken to objects (from 1000 to 50,000). This implementation of sorting is not limiting, which is obvious to the expert in the prior art. In some implementations, the predetermined criterion for sorting may be the distance to the camera and / or the occupied volume and / or area of the object, and the like. The value of k depends on the resources of the computing device and the complexity of the rendering algorithms. In a specific example of implementation on the target minimum hardware (4cpu, 4gb ram), at k => 50,000, the technical solution starts to work significantly slowly. The angular size is measured in degrees, respectively, a comparison of degrees is made, from larger to smaller.

[0083] Step 170: sending an array of unique identifiers of sorted objects to the user's browser from the server.

[0084] Then, the server 810 (backend), as shown in FIG. 8, sends to the client 820 (frontend) an array of UUIDs of the found sorted objects.

[0085] Client 820 (frontend) receives from server 810 (backend) an array of UUID objects. Because the object UUID itself does not contain geometry, the client 820 starts loading the corresponding geometry for each object (described above) and its material from the corresponding storages (from the server 810). The loaded geometries and materials on the client are added to local caches. If subsequent requests return a new array of UUID objects, information on new objects (missing in the cache) is loaded into the local caches, and information on obsolete objects (present in the cache, but not in the array of UUID objects) is removed from the cache of geometry and materials.

[0086] Step 180: objects are rendered based on the unique identifiers obtained in the previous step.

[0087] The rendering (rendering) of at least one object and work with the cache on the client 820 will be described below. On the client 820 (for example, in a web browser) there are several caches:

a. object cache 830, which is a key-value store. The key in this storage is the unique identifier of the object (UUID), value stores the following information:

i. unique material identifier (UUID);

ii. the unique identifier of the geometry (UUID);

iii. transformation matrix.

b. material cache 840, which is a key-value store, where the key is a unique material identifier (UUID), and value stores information about color, transparency, intensity, etc.

c. geometry cache 850, which is a key-value storage, where the key is a unique geometry identifier (UUID), and value stores the coordinates of the vertices of the triangles from which the 2D / 3D object is built.

[0088] As an exemplary storage implementation on the server 810, Redis (an in-memory database with additional fault tolerance), Riak (a distributed, replication storage), Memcached / MemcacheDB (a distributed in-memory database) and etc. As an example implementation, key-value stores can be used without being limited to hash maps.

[0089] The size of each cache is limited by the number of k, which in a particular implementation takes values from 1,000 to 50,000.

[0090] Each time the client 820 receives the results of the frustum fetch as an array of object UUIDs from the server 810, the client 820 compares this list with the available object cache, as shown in FIG. 8. All UUIDs missing in the object cache are marked as new, and objects that are present in the object cache, but not in the truncated visibility pyramid, are marked as obsolete and removed from the cache (including from the material cache and geometry cache). For new objects, materials and geometries are loaded in the background into the corresponding caches (background - it means that it does not block the user's work). All new objects are divided into batches. They send a request for data on the first packet, how the response came, send a request on the second packet, etc.

[0091] One of the features is that the client 820 may not wait for the complete download of data in all caches, and render those 3D objects, the data for which is already present in the cache (s).

[0092] Referring to FIG. 4, this technical solution can be implemented as a computing system 400 for rendering 3D models in a browser using distributed resources, which contains one or more of the following components:

• a processing component 401 comprising at least one processor 402,

• memory 403,

• component 405 multimedia,

• component 406 audio,

• interface 407 input / output (I / O),

• sensor component 408,

• component 409 data transmission.

[0093] The processing component 401 generally controls all of the operations of the system 400, such as processing data about the user or his ticket search request, and also controls the display, phone call, data communication, camera operation, and recording operation of the mobile communication device. The processing component 401 may include one or more processors 402 that implement instructions to complete all or part of the steps from the above methods. In addition, the processing component 401 may include one or more modules for convenient communication between other processing modules 401 and other modules. For example, the processing component 401 may include a media module for convenient, lightweight interaction between the media component 405 and the processing component 401.

[0094] Memory 403 is configured to store various types of data to support the operation of system 400, such as a user profile database. Examples of such data include instructions from any application or method, contact information, address book data, messages, images, videos, etc., all of which operate on the system 400. The memory 403 can be implemented as any type of volatile storage device, nonvolatile memory, or a combination thereof, such as static random access memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (EPROM), Read Only Memory device (ROM), magnetic memory, flash memory, magnetic disk or optical disk and others, but not limited to.

[0095] The multimedia component 405 includes a screen that provides an output interface between a system 400 that may be installed on a user's mobile communications device and a user. In some implementations, the screen may be a liquid crystal display (LCD) or a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen for receiving input from a user. The touchpad includes one or more touch sensors in terms of gestures, touching, and sliding across the touchpad. The touch sensor can not only sense the subject's touch boundary or swipe gesture, but also detect the length of time and pressure associated with touch and slide behavior. In some embodiments, the multimedia component 405 includes one front camera and / or one rear camera. When the system 400 is in an operating mode, such as shooting mode or video mode, the front camera and / or the rear camera can receive media data from the outside. Each front camera and rear camera can be one fixed lens optical system, or can have focal length or optical zoom.

[0096] The audio component 406 is configured to output and / or input an audio signal. For example, the audio component 406 includes a single microphone (MIC) that is configured to receive an external audio signal when the system 400 is in a mode of operation such as a call mode, a recording mode, and a speech recognition mode. The resulting audio signal can be further stored in memory 403 or routed to data transfer component 409. In some embodiments, the audio component 406 also includes one speaker configured to output an audio signal.

[0097] An input / output (I / O) interface 407 provides an interface between the processing component 401 and any peripheral interface module. The above peripheral interface module can be keyboard, steering wheel, button, etc. These buttons may include, but are not limited to, a start button, a volume button, a start button, and a lock button.

[0098] The sensor component 408 comprises one or more sensors and is configured to provide various aspects of assessing the state of the system 400. For example, the sensor component 408 can detect on / off states of the system 400, the relative position of components, such as a display and a keypad, of a single component. system 400, presence or absence of contact between a subject and system 400, and orientation or acceleration / deceleration and temperature change of system 400. Sensor component 408 includes a proximity sensor configured to detect the presence of an object in the vicinity when there is no physical contact. Touch component 408 includes an optical sensor (eg, CMOS or CCD image sensor) configured for use in rendering an application. In some embodiments, sensor component 408 comprises an acceleration sensor, gyroscope sensor, magnetic sensor, pressure sensor, or temperature sensor.

[0099] The communication component 409 is configured to facilitate wired or wireless communication between the system 400 and other devices. System 400 can access a wireless network based on a communication standard such as WiFi, 2G, 3G, 5G, or a combination thereof. In one exemplary embodiment, the communication component 409 receives a broadcast signal or broadcast associated information from an external broadcast control system via a broadcast channel. In one embodiment, communication component 409 comprises a Near Field Communication (NFC) module to facilitate near field communication. For example, the NFC module may be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[00100] In an exemplary embodiment, system 400 may be implemented by one or more Application-Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Units (DSPs), Programmable Logic Devices (PLCs), a logic chip programmed into operating conditions (FPGA), controller, microcontroller, microprocessor, or other electronic components, and can be configured to implement a method 100 for rendering 3D models in a browser using distributed resources.

[00101] In an exemplary embodiment, a nonvolatile computer readable medium comprises memory 403 that includes instructions where instructions are executed by processor 401 of system 400 to implement the above-described methods for rendering 3D models in a browser using allocated resources. For example, a nonvolatile computer-readable medium can be ROM, random access memory (RAM), compact disc, magnetic tape, floppy disks, optical storage devices, and the like.

[00102] Computing system 400 may include a display interface that transmits graphics, text, and other data from a communications infrastructure (or from a framebuffer, not shown) for display on a multimedia component 405. Computing system 400 further includes input devices or peripherals. Peripheral devices may include one or more devices for interacting with a user's mobile communications device, such as a keyboard, microphone, wearable device, camera, one or more audio speakers, and other sensors. Peripheral devices can be external or internal to the user's mobile communications device. A touchscreen can display, typically graphics and text, and also provides a user interface (such as, but not limited to, a graphical user interface (GUI)) through which a subject can interact with a user's mobile communication device, such as access and interact with applications running on the device.

[00103] The elements of the proposed technical solution are in a functional relationship, and their joint use leads to the creation of a new and unique technical solution. Thus, all blocks are functionally linked.

[00104] All blocks used in the system can be implemented with electronic components used to create digital integrated circuits, which is obvious to a person skilled in the art. Not limited to, microcircuits can be used, the logic of which is determined during manufacture, or programmable logic integrated circuits (FPGA), the logic of which is set through programming. For programming, programmers and debugging environments are used that allow you to set the desired structure of a digital device in the form of a circuit diagram or a program in special hardware description languages: Verilog, VHDL, AHDL, etc. An alternative to FPGAs can be programmable logic controllers (PLCs), basic matrix crystals ( BMK) requiring a factory production process for programming; ASICs are specialized custom large integrated circuits (LSIs), which are significantly more expensive for small-scale and single-piece production.

[00105] Typically, the FPGA itself consists of the following components:

• configurable logic blocks that implement the required logic function;

• programmable electronic links between configurable logic blocks;

• programmable input / output blocks providing connection of the external output of the microcircuit with the internal logic.

[00106] Blocks can also be implemented with read-only memory devices.

[00107] Thus, the implementation of all the blocks used is achieved by standard means based on the classical principles of implementing the foundations of computing.

[00108] As will be understood by one skilled in the art, aspects of the present technical solution may be implemented as a system, method, or computer program product. Accordingly, various aspects of the present technical solution may be implemented solely as hardware, as software (including application software and so on), or as an embodiment combining software and hardware aspects, which may generally be referred to as a "module" , "System" or "architecture". In addition, aspects of the present technical solution may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code that is implemented thereon.

[00109] Any combination of one or more computer readable media can also be used. The computer-readable storage medium can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination thereof. More specifically, examples (non-exhaustive list) of a computer-readable storage medium include: electrical connection using one or more wires, a portable computer diskette; hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), fiber optic connection, compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any combination of the above. As used herein, a computer-readable storage medium can be any flexible storage medium that can contain or store a program for use by the system itself, device, apparatus, or in connection therewith.

[00110] Program code embedded in a computer-readable medium can be transmitted using any medium, including, without limitation, wireless, wired, fiber optic, infrared, and any other suitable network or combination of the above.

[00111] Computer program code for performing operations for the steps of the present technical solution may be written in any programming language or combinations of programming languages, including an object-oriented programming language such as Python, R, Java, Smalltalk, C ++, and so on, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed on the user's computer in whole, in part, or as a separate software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN), a wide area network (WAN), or a connection to an external computer (for example, via the Internet using Internet providers).

[00112] Aspects of the present technical solution have been described in detail with reference to block diagrams, schematic diagrams, and / or diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the present technical solution. It will be appreciated that each block from the block diagram and / or diagrams, as well as combinations of blocks from the block diagram and / or diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other data processing device to create a procedure, such that instructions executed by a computer processor or other programmable data processing device create means to implement the functions / actions specified in block or blocks of flowchart and / or diagram.

[00113] These computer program instructions may also be stored on a computer-readable medium that can control a computer other than a programmable data processing device or other devices that function in a particular way, such that the instructions stored on the computer-readable medium create a device including instructions that perform the functions / actions specified in the block diagram and / or diagram.

Claims (39)

1. A method for rendering 3D models in a browser using distributed resources, performed by at least one computing device, and includes the following steps: • get at least one 3D model on the server; • split each object in the 3D model into an array of triangles and place it in the object storage by its unique identifier; • form a spatial index in the database of all objects from the obtained three-dimensional model; • form a visibility pyramid consisting of a predetermined number of cubes and send a request to the server to obtain a list of objects included in these cubes; • form a list of objects for visualization by primary culling of three-dimensional objects using the clipping method by entering the requested cubes on the server side based on the spatial index of each object; • carry out sorting of objects from the found list based on a predetermined criterion; • send an array of unique identifiers of sorted objects to the user's browser from the server; • render objects based on the unique identifiers obtained in the previous step. 2. The method according to claim 1, characterized in that at least one three-dimensional model is obtained on the server, downloaded by the user via a web browser. 3. The method according to claim 1, characterized in that at least one three-dimensional model is obtained on the server, downloaded by the user from a CAD file. 4. The method according to claim 1, characterized in that at least one three-dimensional model is obtained on the server in RVM / ATT or IFC format. 5. The method according to claim 1, characterized in that the resulting three-dimensional model of the objects are represented in the form of a tree. 6. The method according to claim 1, characterized in that each object in the obtained three-dimensional model contains a unique object identifier and / or a reference to the parent object, and / or a spatial transformation matrix relative to the parent object, and / or materials of the object, and / or 2D / 3D geometry of the object, and / or properties of each object (attributes). 7. A method according to claim 1, characterized in that as a spatial index, a partition of the entire area of space with a fixed step along each dimension is used, either octal trees, or kD trees, or BSP trees, or BVH (Bounding Volume Hierarchy), or Interval Tree, or R-trees. 8. The method according to claim 1, characterized in that the spatial index is formed as follows: a) for each object of the three-dimensional model, find its transformation matrix relative to the origin, taking into account the tree structure of the three-dimensional model; b) for each object geometry, a transform operation is applied, obtaining the geometry of all objects relative to one origin in world coordinates c) the resulting transformed geometry is placed into the database by its unique identifier; d) for each object with the transformed geometry, the spatial index is calculated and placed in the database. 9. The method according to claim 1, characterized in that a request is sent to the server for display, which is a JSON array of coordinates. 10. The method according to claim 1, characterized in that when an array of unique identifiers of the found sorted objects is received from the server, this list is compared with the existing cache of objects, and • all UID objects missing from the cache are marked as new; • objects that are present in the cache of objects, but are not in the truncated pyramid of visibility, are marked as obsolete and removed from the cache; • for new objects, background loading of materials and geometries into the corresponding caches is performed. 11. The method according to claim 1, characterized in that only those objects that are present in the cache are rendered based on unique identifiers. 12. The method according to claim 1, characterized in that when sorting objects, the predetermined criterion is the angular size of each object and / or the distance to the camera and / or the occupied volume and / or area of the object. 13. A system for rendering 3D models in a browser using distributed resources, containing: • at least one server capable of a) receiving at least one 3D model from the client; b) splitting each object in the 3D model into an array of triangles and placing it in the object store by its unique identifier; c) forming a spatial index in the database of all objects from the obtained three-dimensional model; d) generating a list of objects for rendering by first culling three-dimensional objects using a clipping method by entering the requested cubes based on the spatial index of each object; e) sorting objects from the found list based on a predetermined criterion; f) sending to the client an array of unique identifiers of sorted objects; • at least one client capable of a) sending at least one 3D model to the server; b) forming a pyramid of visibility, consisting of a predetermined number of cubes and sending a request to the server to obtain a list of objects included in these cubes; c) rendering objects based on the unique identifiers obtained earlier.

Images