RU2783218C1 - Method and system for controlling display of virtual tours in multi-user mode - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to the field of computer technology, in particular to methods for managing user interaction in virtual tours. The method for navigating through the generated scenes of a virtual tour in a multi-user mode includes the steps at which the surrounding space is surveyed using a 3D camera containing a lidar, while during the survey a group of images is obtained, based on which HDR images are formed, and point clouds of the surrounding space, transmit the data obtained during shooting with a 3D camera to the server, which generates a three-dimensional model of a virtual tour, which is a 3D model textured with 360-degree photo panoramas, while the virtual tour is displayed in the user device interface, and the 3D model contains predefined viewpoints (POV), and each point contains 3D coordinates, a command is received on the server from the device of the first user to generate a virtual tour in multi-user mode, then the first user is appointed as the leader and the virtual tour is displayed in the interface of the user device.

EFFECT: technical result consists in increasing the accuracy of processing the surrounding space to obtain a virtual scene.

15 cl, 12 dwg

Description

FIELD OF TECHNOLOGY

[0001] The present solution relates to the field of computer technology, in particular to methods for managing user interaction in virtual tours.

BACKGROUND OF THE INVENTION

[0002] A virtual tour is a method of realistically displaying a three-dimensional multi-element space on a user's tool, which can be a computer device (computer, tablet or smartphone) or a virtual reality tool. The elements of a virtual tour, as a rule, are spherical panoramas, interconnected by interactive transition links (also transition points or hotspots). Virtual tours may also include cylindrical panoramas, virtual 3D objects, ordinary photos, video, sound, etc. At the same time, the panoramas in the virtual tour are linked to the map according to the coordinates of the shooting location and are oriented to the cardinal points.

[0003] In other words, a virtual tour is a general term for a three-dimensional environment model formed using several spherical panoramas, as well as point clouds obtained using Lidar - Light Detection and Ranging laser shooting, connected into a single object for display using hotspots that you can virtually “navigate” while browsing. Other interactive elements may also be included in virtual tours: pop-up information windows, explanatory labels, graphical control keys, etc.

[0004] Existing solutions for displaying and allowing users to view and interact with a virtual tour are typically limited to a player that renders the environment of the virtual tour and allows multiple users to collaboratively view the environment model within the tour. An example of such a technology can be the technology disclosed in the application WO 2021074924 A1 (Mital, 04/22/2021), which describes a method for simultaneous viewing of a virtual tour by several participants, with the possibility of live communication and tracking the movement of the cursor of the leading participant.

[0005] The disadvantage of this kind of solutions is the principle of displaying the interface when connecting new participants, which provides the formation of a tour model display either at the original start point of the tour, or at some random point, which leads to inefficient navigation within the tour of connected users.

SUMMARY OF THE INVENTION

[0006] The claimed solution solves the technical problem of effectively managing the display of a virtual tour in a multi-user mode.

[0007] The technical result consists in increasing the efficiency of virtual tour users' navigation by linking the display point of the tour model of the participants' connected devices to the current display point of the host device.

[0008] In a preferred embodiment of the claimed solution, a method is claimed for controlling the display of virtual tours in a multi-user mode, performed using a computing device containing at least one processor and a memory storing machine-readable instructions that, when executed by at least one processor, provide:

receiving a command from the device of the first user to generate a virtual tour in a multi-user mode;

generating a virtual tour, which is a 3D model textured with 360-degree photo panoramas, while the virtual tour is displayed in the user device interface, and the 3D model contains predetermined viewpoints (POV), each point containing three-dimensional coordinates;

displaying the virtual tour for the first user, with the first user designated as the leader;

receiving a command from user devices to connect to the said virtual tour of new users, while

receive data POV, which is the first user;

forming a virtual tour display for each new user in the POV in which the first user is located, and the virtual tour display for each new user after connection duplicates the display of the first user;

display a virtual tour for users with the ability to view and navigate through the specified POV.

[0009] In one of the private embodiments, the connection of users to the virtual tour is carried out using the formation of a hyperlink to connections by the first user.

[0010] In another particular embodiment, a virtual avatar is formed for each participant in the virtual tour.

[0011] In another particular embodiment, when users connect to a virtual tour, a text and/or voice chat session is generated.

[0012] In another particular embodiment, the first user may transfer the host rights to another user.

[0013] In another particular embodiment, when transferring rights, the display of the virtual tour is set in accordance with the POV of the user designated as host.

[0014] In another particular embodiment, once new users connect to the POV of the first user, the function of manually moving the view camera in the POV or the free movement function of the virtual tour POV is activated on the devices of the new connected users.

[0015] In another particular embodiment, the host user controls the microphone mute/unmute of other users in the chat session.

[0016] In another particular embodiment, the virtual tour display for each new connected user begins by repeating the first user's POV path.

[0017] In another particular embodiment, the host avatar is configured to generate a virtual pointer.

[0018] In another particular embodiment, the virtual tour 3D model contains 3D objects and is a room model capable of changing at least the texture of the 3D model and/or 3D objects.

[0019] In another particular embodiment, 3D objects contain metadata that makes it possible to obtain additional information about the object, including at least one of: text information, video, a link to an external resource.

[0020] In another particular embodiment, the information is displayed in an additional window located on top of the virtual tour interface.

[0021] In another particular embodiment, the virtual tour interface contains a function that provides a measurement of the size of objects.

[0022] The claimed solution is also implemented using a virtual tour display control system in a multi-user mode, which contains at least one processor operatively associated with at least one memory storing machine-readable instructions that, when executed by the processor, implement the above method.

BRIEF DESCRIPTION OF THE DRAWINGS

[0023] FIG. 1 illustrates the general scheme for connecting users to a virtual tour.

[0024] FIG. 2 illustrates an example of a shooting process using a 3D camera.

[0025] FIG. 3 illustrates the raw data obtained during the survey.

[0026] FIG. 4 illustrates an example of an HDR image.

[0027] FIG. 5 illustrates an example of a sweep floor map.

[0028] FIG. 6 illustrates a floor map of a floor.

[0029] FIG. 7 illustrates the scheme of device interaction.

[0030] FIG. 8 illustrates an example of a virtual tour display.

[0031] FIG. 9 illustrates a virtual tour minimap.

[0032] FIG. 10 illustrates the virtual tour object modification window.

[0033] FIG. 11 illustrates an example of moving through survey points within a tour.

[0034] FIG. 12 illustrates a general view of the computing device.

IMPLEMENTATION OF THE INVENTION

[0035] In FIG. 1 shows the general scheme of user interaction with joint participation and viewing of virtual tours. The virtual tour (200) is generated on the server (120), to which users (110) connect using various types of computing devices (111-113). Devices can be, for example, a smartphone, computer, laptop, tablet, smart TV, virtual reality tool and other devices that can functionally reproduce and form an interface for user interaction with a virtual tour (200), regardless of the principle of its display (display devices, virtual reality) and interactions with objects within the tour (using inputs or controllers designed for virtual reality).

[0036] The server (120) is a standard computing computing device and can be either a server cluster or a cloud server. The main task of the server (120) is to process data to provide virtual tours (200), providing subsequent data transfer to user devices (110). The server (120) can also be used to directly generate virtual tours (200) by processing data (point clouds, photo panoramas, images) coming from a specialized 3D camera (140).

[0037] The exchange of information between users (110) and the server (120) is carried out via a data link (130), which is, as a rule, the global network "Internet". The data link implementation protocol (130) may be selected from any known type of protocol such as: LAN, WAN, PAN, WLAN, IEEE 802.11, GSM, and the like. The appropriate protocol and type of connection of a specific device (111 - 113) to the server (120) is determined based on the technical implementation of this type of device and the corresponding hardware for communicating with the Internet.

[0038] As a 3D camera (140) for capturing 3D virtual tours (200), for example, a FilinCam V6/V7/X1 3D camera and others can be used. The 3D camera (140) is designed to capture indoors and outdoors for further creation of their photorealistic digital models and virtual tours. The survey process includes the acquisition and pre-processing of raw data from the 3D camera (140), which also contains at least: a computing module (processor or microcontroller), memory, lidar, an IMU accelerometer sensor and an encoder (rotation angle sensor), which are necessary for formation of a data set for the subsequent formation of virtual tours (200). The camera (140) is controlled via a software application installed on the operator's mobile device, such as a tablet, smartphone or laptop. Upon completion of the survey, the data obtained are uploaded from the 3D camera (140) to the post-processing server, where they are stored, processed, and 3D models of the scanned space and virtual tours (200) are created based on them. Ready-made virtual tours (200) are transferred from the post-processing server to the server (120) for subsequent access to them by users (110).

[0039] In FIG. 2 shows an example of the process of capturing an area using a 3D camera for subsequent construction of a virtual tour (200) based on it. The shooting process begins with the installation of the camera (140), mounted on a tripod, at the starting point of the shooting (points 1 - 6 in Fig. 2). The camera (140) makes a full rotation around the axis, scanning the surrounding area with lidar and stops several times to take a series of photographs. As a result, the so-called "sweep" (from the English. Sweep) is formed, i.e. a dataset consisting of several HDR images, a lidar point cloud, and metadata.

[0040] An example of data stored in a sweep.

- three series of 6 or 7 (depending on the selected mode) camera shots (140), which are used to create HDR images;

- A set of lidar slices to create a point cloud;

- Quaternions and angles of camera deviation (140) from the horizon at the start and end of the sweep survey, calculated on the basis of IMU accelerometer and gyroscope data;

- Encoder data (time stamps, camera angles).

- Metadata (for example, sweep shooting mode: shooting resolution, time, etc.).

[0041] Upon acquisition of data from the first sweep at point #1, the camera (140) moves to a new survey point (eg, point #2) and the next sweep is taken. A set of captured sweeps, reduced to a single coordinate system, is called a "scene".

[0042] As shown in FIG. 2, sweeps in a scene are always organized into a tree-like acyclic graph with a single starting node. For each sweep in the scene, except for the first one, a so-called "parent" sweep is defined. By default, the last sweep taken is considered the parent, but if necessary, any of the captured sweeps can be assigned as the parent sweep. For example, in FIG. 2, the parent of sweep #4 is not sweep #3, as it would be by default, but sweep #2.

[0043] Each sweep has its own coordinate system, which coincides with the coordinate system of the 3D camera (140) at the start of the survey of this sweep. The coordinate system of the first sweep also serves as the scene coordinate system. Thus, in order to build a scene from sweeps, it is necessary to solve the problem of transferring the point cloud of each sweep from its own coordinate system to the scene coordinate system.

[0044] For this, a binding algorithm is used that analyzes point clouds and photo panoramas of two neighboring (i.e., connected by one graph edge) sweeps in the tree, searches for matching sections, based on them determines the relative position and orientation of these sweeps in the scene and calculates the matrix transition between them.

[0045] The anchoring algorithm calculates transition matrices T _ij for each pair of neighboring sweeps S _i and S _j . To transfer the point cloud P _i of the sweep S _i to the coordinate system of the sweep S _j , it is sufficient to multiply it by T _ij .

Pi ^j =P _i *T _ij

Here and below, the following notation is used:

• S _i - sweep number i. For example, S ₃ is sweep #3.

• P _i - sweep point cloud S _i in own coordinate system. For example, P ₃ is a point cloud of sweep No. 3.

• P _i ^j - point cloud of the sweep S _i , converted to the coordinate system of the sweep S _j . For example, P ₃ ⁵ is the point cloud of sweep #3 in the coordinate system of sweep #5.

• T _ij - transition matrix used to transform the sweep point cloud S _i into the sweep coordinate system S _j . For example, T ₃₅ is a transition matrix that converts a cloud of points P ₃ to P ₃ ⁵ .

[0046] To translate the point cloud P _j of the sweep S _j into the coordinate system of the sweep S _i , it is sufficient to multiply this point cloud by the matrix T _ji inverse to T _ij .

T _ji =(T _ij ) ^-1

P _ji =P _j *(T _ij ) ^-1

[0047] If the sweeps are not adjacent in the tree, the task is divided into several consecutive steps. For example, so that in FIG. 1 to convert the point cloud of sweep S ₆ to the coordinate system of sweep S ₁ , it should first be translated to the coordinate system of sweep S ₅ , then S ₂ , and finally S ₁ . To do this, it is necessary to successively multiply the cloud of points of the sweep S ₆ by the transition matrices T ₆₅ , T ₅₂ and T ₂₁ :

R ₆ ⁵ \u003d R ₆ * T ₆₅

R ₆ ² \u003d R ₆ ⁵ * T ₅₂

R ₆ ¹ \u003d R ₆ ² * T ₂₁

Or, more succinctly:

R ₆ ¹ \u003d R ₆ * T ₆₅ * T ₅₂ * T ₂₁

[0048] It should be noted that the tree structure of the graph guarantees the uniqueness of the path between any two sweeps in the scene. Thus, in order to bring all sweeps to a single coordinate system, it is necessary to determine the only possible path on the graph from each sweep to the initial sweep No. 1 and successively apply multiplication by the corresponding transition matrices. As a result of this procedure, a scene is formed that contains all the captured panoramas and point clouds in a single coordinate system.

[0049] The survey of the sweep consists of two main stages: the survey itself and pre-processing of the captured raw data. When shooting, the 3D camera (140) makes a complete rotation around the axis, scanning the surrounding area with a 40 Hz lidar, making several stops (for example, three stops) during the shooting to take a series of photographs with different exposures. As a result of this process, the memory of the 3D camera (140) stores several series of images from the camera, a set of lidar slices, as well as auxiliary metadata, including timestamps and rotation angles of the 3D camera (140) at the time of each image and each lidar slice.

[0050] The processing of the received raw data includes the following operations:

- From each series of pictures shown in FIG. 3, the HDR image shown in FIG. four;

- A single cloud of points is formed from the lidar slices. To do this, each slice is multiplied by the rotation matrix around the vertical Y axis by the angle specified by the 3D camera encoder readings (140) at the time the slice was taken, and by the lidar calibration matrix.

The resulting HDR images and point clouds are then used as input to calculate the pairwise sweep tie.

[0051] In most digital cameras, the dynamic range of the sensor (the range of brightness reproduced without distortion) is significantly inferior to human vision. A narrow dynamic range means that the sensor is not able to simultaneously capture very light and very dark areas of the image without loss of information. As a result, either the light area is displayed as a white spot, or the dark area is displayed as black.

[0052] To solve this problem, HDR imaging technology is used. An HDR image is formed from a series of normal shots taken at different exposures. Short-exposure shots show brighter areas well, while long-exposure shots show darker areas. The resulting HDR image contains all the details from all the original shots, both in the darkest and brightest areas. Such an image is as close as possible to the perception of the surrounding world by the human eye.

[0053] Next, consider the principle of binding adjacent sweeps using the example of sweeps S_i and S_j in the process of generating the virtual tour scene (200) with the corresponding viewpoints (shooting points). First, on the HDR images of sweeps S_i and S_j the search for key points and descriptors (vectors describing the neighborhood of the found key points) is performed. Key points are considered to be points with characteristic geometry, such as corners, or intersections of planes. The search for key points and the calculation of their descriptors is performed using algorithms from the OpenCV open library. The descriptors found on the HDR images of both sweeps are matched in pairs using algorithms from the FLANN open library. The coincidence of two descriptors is called a "match" (from the English "to match" - to match). The most accurate matches found are projected from HDR images onto point clouds and stored in the camera's memory (140) for further use.

[0054] In parallel, key points and descriptors are searched on point clouds of sweeps S _i and S _j using algorithms from the open library PCL (Point cloud library). The descriptors found on the point clouds of both sweeps are compared in pairs using algorithms from the FLANN open library. The most accurate matches found are stored in the camera's memory (140) for further use.

[0055] Thus, a combined set of best matches from two sources is formed: 2D (HDR images) and 3D (point clouds). However, situations are also possible when one of the sources is missing. For example, if at least one of the matched sweeps was shot in the "total darkness" mode, the search for key points and descriptors in its HDR images is not performed, and no matches are received from the 2D source. Similarly, situations are possible when the geometry of the scanned space does not allow finding any key points in the sweep point cloud (for example, this is possible on a large flat football field). In these cases, you should limit yourself to a set of matches from only one 2D or 3D source, which slightly reduces the final binding accuracy.

[0056] Then, based on a set of best matches from available sources, a search is made for possible transition matrices T _IJ using the RANSAC algorithm. The algorithm takes several randomly selected matches as a basis (for example, 4-5 matches) and calculates the transition matrix for them, after which it checks how many matches correspond to the found matrix. The process is repeated until a transition matrix is found that corresponds to a sufficiently large number of matches, or the maximum number of search iterations is reached.

[0057] There may be cases where the RANSAC algorithm finds more than one transition matrix. Typically, this occurs in situations of uniformly interleaved environments (for example, a long corridor with identical windows at an equal distance). In this case, the found matrices are checked for compliance with the indicators of additional sensors of the 3D camera (140), such as IMU, magnetometer, etc. The matrix that most closely matches these criteria is selected as the final transition matrix T _IJ .

[0058] Depending on the geometry of the scanned object, the scene may contain one or more floors. Each floor contains a group of sweeps linked to each other by matching. Each sweep always belongs to a certain floor, because a sweep cannot be located outside a floor or on several floors at the same time.

[0059] For each floor, a so-called "floor map" is generated, schematically displaying the point clouds of the sweeps of that floor in a top view. The floor map of a floor consists of the floor maps of all sweeps related to the given floor.

[0060] The sweep floor map is generated as follows: a horizontal plane is drawn through the origin of the sweep, cutting off all points above it. Cloud points lying below the clipping plane are projected onto the sweep HDR panoramas in accordance with a hard-coded "meters/pixels" ratio. Then, for each point, a display color is determined that corresponds to the color of the HDR panorama pixel onto which the point was projected. Finally, all points are displayed in the "top view" horizontal projection using the colors defined above.

[0061] An example of a sweep floor map is shown in FIG. 5. Floor map of a floor consists of floor maps of all sweeps of a given floor, taking into account their relative position. Information about the relative position of the sweeps is obtained at the matching stage, which makes it possible to generate the floor map shown in FIG. 6.

[0062] The floor map is not static and is updated as sweeps are taken or removed. As soon as the camera (140) captures a sweep and matches it with the parent sweep, the sweep is immediately displayed on the floor map of the corresponding floor of the scene. When one or more sweeps are deleted, the floor map is also automatically redrawn.

[0063] In FIG. 7 shows the general scheme of operation of the claimed solution in a multi-user mode. As shown in the example, the device (111) or the host device activates the creation of a multi-user room for the virtual tour (210). To do this, using the device (111), the user activates on the server (120) the choice to generate the corresponding virtual true (210) and then the activation of the multi-user mode is carried out using the corresponding function of the graphical user interface (GUI). An invitation to participate in a single virtual tour (210) of several users (for example, users 112, 113) is carried out by generating a connection link (211), which is created by the server (120) based on a request from the master device (111). The link (211) is a connection hyperlink that is transmitted from the device (111) to the connected devices (112, 113) using any suitable type of data exchange (CMC, messenger, e-mail, etc.). The link can also be encoded into a QR code, which is also transmitted from the host device (111) to the connecting device.

[0064] In FIG. 8 shows an example of displaying a virtual tour (300) on a user's device. The virtual tour (300) contains a number of elements of the surrounding space (311), for example, furniture, interior items, etc. Each element is a virtual space object that the user can interact with. When moving inside the tour model (300), the interface also displays a minimap (310) showing a schematic floor plan and the point where the user is currently located. The minimap (310) can display POV points for moving to them both in the current room and on the general tour model. When activated on the minimap (310), such a point, using the graphical interface, moves the user to this point.

[0065] As shown in FIG. 9, on the minimap (310), in addition to the location of the user (for example, the user 111), the direction of his gaze (1111) is also displayed, showing the point of view of the tour (300) from the position of his camera.

[0066] In FIG. 10 shows an example of interface display when interacting with objects (311) of virtual tour (300). When interacting with the object (311) using an input means, for example, clicking with the mouse or pressing the touch screen of a smartphone or tablet, an additional window (312) opens, displaying additional information about the object. Such information can be: the name of the object (311), its cost, the possibility of ordering and the possibility of changing the texture (313) of the object (311). 3D tour objects (311) contain metadata that provide the possibility of obtaining additional information about the object, such as: text information with a description of the object, video, a link to an external resource.

[0067] Objects (311) are actual photocopies of furniture, interiors, and the like. The tour interface (300) allows you to change the texture of the object (311) by changing it through an additional interface (313). This allows you to change the color scheme of the object (311) within the current tour (300). At the same time, each of the users participating in the tour (300) can independently change the texture of objects (311), which provides a display of changes in objects (311) inside the tour (300) for each user separately. Also, the software logic implemented by the server (120) within the framework of generating a virtual tour display (300) allows not only changing the color/texture of objects (311), but their geometric dimensions, thereby allowing a unique tour display to be generated for each user, which can be applied when selecting the design of premises or real estate as part of the use of virtual tours. The virtual tour model (300) provides the possibility of decorating it (changing the color of the walls, adding objects, etc.), which can be carried out both by one user for display to all other participants, and individually by each participant separately.

[0068] FIG. 11 illustrates a schematic display of survey points P1-P7 (POV) that are generated based on sweeps. At the time of connection of new participants (112, 113) of the tour (300), when following the link sent from the device (111), the server (120) receives information about the current location within the tour (300) of the user (111), who is currently leading moment. As shown in FIG. 11 at the time of connection, the user (111) is in P6. When users click on the link (112, 113), they will generate a virtual tour display from the user's camera view area (111). By default, when new users first connect to the host (111) device, the display of the view area (1111) of the user (111) is duplicated, in which the camera quaternion of the participant’s view area (112, 113) matches the camera quaternion of the view area of the leader (111).

[0069] After connecting, users (112, 113) duplicate the view of the user's camera (111), but can also select one of the modes of further participation in the virtual tour (300) using the tour interface. There are several such modes: 1) Full copying of all movements, turns and current increase (zoom) of the leader's camera (111); 2) Copying all movements of the presenter (111), but the ability to control the rotation and zoom of the camera manually; 3) Full freedom mode for independent movement within the tour (300).

[0070] Each of the connected participants (112, 113) can generate a command for other participants to follow, providing a repetition of moving around the tour and duplicating the camera view of such a user. This kind of command may be transmitted by interacting with the virtual tour display interface (300). At the same time, the participant to whom this notification is sent can accept or reject it.

[0071] The leader's device (111) can also activate a command to gather tour participants at a single point where the leader (111) is currently located. This is implemented by moving the participants in the POV of finding the leader (111). Additionally, a command to change the display area of the camera that duplicates the host's camera (111) can be forcibly generated.

[0072] When connecting to the tour of several users (111-113) in the multi-user mode of participation for each of the users, a unique avatar can be formed, displayed, for example, on the minimap (310) or in the general interface when reviewing the tour (300). The tour interface (300) also allows users (111-113) to communicate with each other via text and/or voice chat. When connected to the host device (111), it is the main device that manages and moderates the chat and allows you to turn on / off the microphones of other participants, as well as block text messages from them if necessary. All users participating in a single view of the virtual tour (300) have unique identifiers and indicate the login under which they join via a link (211). The chat function may further comprise a built-in translation module that provides translation of text or audio from one language to another. Each of the participants of the tour can indicate the preferred language of communication. In this case, a marker of the language of his communication is set on the server (120) for each user identifier, which allows using the translation module to automatically translate the generated text or audio messages. In this case, a separate chat translation will be generated for each user, depending on the language chosen by him.

[0073] The rights of the host can be transferred to another participant, this is done when the session of the virtual tour of the host device (111) exits, or when the host itself (111) directly transfers to another participant in this role. The transfer of the role of the presenter means that another user receives similar functionality that was originally available to the presenter (111). When a new leader is appointed, other participants are automatically assigned to the assigned leader's field of view in the corresponding POV in which he is located. Participants can change the view and navigation of the tour (300) at any time after the appointment of a new leader.

[0074] When new users (112, 113) are connected, the virtual tour interface (300) can implement the function of repeating the route of movement along the POV of the leader (111) with displaying frames of the display area of his camera. This function allows you to repeat the path of the leader (111) along the POV points where the leader (111) was before.

[0075] Additionally, the virtual tour interface (300) may include a virtual pointer function that provides a function for pointing to objects within the virtual tour (300). The pointer may be a virtual beam, or a geometric primitive (for example, a circle), displayed in the interface on users' devices and allowing you to point to an object (311) or an element of the virtual tour. When pointing at an object (311), it may be highlighted or otherwise highlighted within the tour display (300). The host device (111) can also activate the "Object Pointer Broadcast" function, in which highlighting objects (311) using the host pointer (111) is displayed not only in the application window of the presenter, but also in the application windows of all participants in the room. If the pointer object (311) has a metadata tag, then when highlighted, this tag is displayed in the panorama for all participants who see the object highlighted. The metadata tag may contain textual, visual, audio, and video information about the item, as well as hyperlinks and controls (such as "Buy" or "Add to Favorites" buttons) as described earlier in this application materials.

[0076] In addition to the function of pointing to objects within the tour (300), the interface provides a function for measuring the geometric dimensions between points in the tour (the so-called "roulette" function), which measures the distance between two or more points, because The 3D model of the virtual tour is built on the actual dimensions of the premises during their shooting. Dimensional measurement using the presenter's GUI (111) can be displayed not only in its application window, but also for all other users.

[0077] The tour display interface on the host device (111) has the function of drawing lines and shapes "on top" of the virtual tour panorama (300). All shapes drawn using this feature are displayed in a panorama in the application window for each participant.

[0078] For the host device (111), a "Video Playback" tool is available, with which the host (111) can broadcast selected video materials to the participant interfaces (112, 113). Also, the tool allows you to play the selected video material automatically, if predetermined conditions are met, for example, advertising video playback can start as soon as the presenter (111) enters a certain part of the tour (300) or points to a certain object (311) with a pointer.

[0079] For all participants (112, 113) of the tour, except for the leader (111), a picture-in-picture mode is available. When this mode is active, the user display interface (112, 113) displays not only the panorama of the virtual tour from the camera display area of this participant, but also the presenter's camera display area (111), for example, in a separate window on top of the main window. The location and size of the picture-in-picture window are configurable.

[0080] Participants of the tour (300) in addition to the chat session can communicate via videoconference. When this mode is activated, the user window displays not only a panorama of the virtual chat, but also a video stream from the webcam of the participant who is currently speaking (in picture-in-picture mode). The videoconference mode can also be activated via avatars. When this mode is active, the participant's avatar, like a video screen, displays the video stream from the participant's webcam. This feature is only available for geometric shape avatars.

[0081] The implementation of the above functions and capabilities within the framework of generating and displaying virtual tours for several participants allows not only to improve the efficiency of user navigation by binding within their connection to the display point of the main user (master), but also to expand the functionality, providing not only the formation of a display of virtual tours for viewing them from different viewpoints, but also additional opportunities in terms of virtual interior decor and the ability to select decor elements displayed on a 3D tour model.

[0082] Virtual tours can be widely used in real estate sales, tours, education, and the like.

[0083] In FIG. 12 shows an example of a general view of a computing system (400) or a computing device that provides the implementation of the claimed method or is part of a computer system, for example, a server, a personal computer, a part of a computing cluster that processes the necessary data to implement the claimed technical solution. Based on the system (400), user devices (110), a server (120), and a 3D camera (140) can be manufactured.

[0084] In general, the system is a computer system (400) and contains one or more processors (401), memory facilities such as RAM (402) and ROM (403), input / output interfaces (404) connected by a common information exchange bus ), input/output devices (405), and a device for networking (406).

[0085] The processor (401) (or multiple processors, multi-core processor, etc.) can be selected from a range of devices currently widely used, for example, manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos ™, MediaTEK™, Qualcomm Snapdragon™, etc. Under the processor or one of the processors used in the system (400), it is also necessary to take into account the graphics processor, for example, NVIDIA GPU or Graphcore, the type of which is also suitable for full or partial execution of the method, and can also be used to train and apply machine learning models in various information systems.

[0086] RAM (402) is a random access memory and is designed to store machine-readable instructions executable by the processor (401) to perform the necessary data logical processing operations. The RAM (402) typically contains the executable instructions of the operating system and associated software components (applications, program modules, etc.). In this case, the RAM (402) may be the available memory of the graphics card or graphics processor.

[0087] A ROM (403) is one or more persistent storage devices such as a hard disk drive (HDD), a solid state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media ( CD-R/RW, DVD-R/RW, BlueRay Disc, MD), etc.

[0088] Various types of I/O interfaces (404) are used to organize the operation of system components (400) and organize the operation of external connected devices. The choice of appropriate interfaces depends on the particular design of the computing device, which can be, but not limited to: PCI, AGP, PS/2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS/Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0089] To ensure user interaction with the computing system (400), various means (405) of I/O information are used, for example, a keyboard, a display (monitor), a touch screen, a touchpad, a joystick, a mouse, a light pen, a stylus, touch panel, trackball, speakers, microphone, augmented reality, optical sensors, tablet, indicator lights, projector, camera, biometric identification tools (retinal scanner, fingerprint scanner, voice recognition module), etc.

[0090] The network communication means (406) provides data transmission via an internal or external computer network, for example, an Intranet, the Internet, a LAN, and the like. As one or more means (406) can be used, but not limited to: Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module and others

[0091] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be construed as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims (29)

1. A method for navigating through the generated scenes of a virtual tour in a multi-user mode, containing the steps in which: performing a survey of the surrounding space using a 3D camera containing a lidar, while shooting receive a group of images, on the basis of which HDR images are formed, and point clouds of the surrounding space; transmitting the data obtained during shooting with a 3D camera to the server, which generates a three-dimensional model of a virtual tour, which is a 3D model textured with 360-degree photo panoramas, while the virtual tour is displayed in the user device interface, and the 3D model contains specified points view (POV), with each point containing three-dimensional coordinates; receive on the server a command from the device of the first user to generate a virtual tour in multi-user mode, while using the server in response to the received command: designating the first user as a leader and displaying the virtual tour on the user device interface; receive commands from user devices to connect new users to the mentioned virtual tour, while receive data POV, which is the first user; forming a virtual tour display for each new user in the POV in which the first user is located, and the virtual tour display for each new user after connection duplicates the display of the first user; display a virtual tour for users with the ability to view and navigate through the specified POV. 2. The method according to claim 1, characterized in that the connection of users to the virtual tour is carried out by forming a hyperlink to connections by the first user. 3. The method according to claim 1, characterized in that a virtual avatar is formed for each participant of the virtual tour. 4. The method according to claim 1, characterized in that when users connect to the virtual tour, a text and/or voice chat session is generated. 5. The method according to claim 1, characterized in that the first user can transfer the rights of the presenter to another user. 6. The method according to claim 5, characterized in that when transferring the rights, the display of the virtual tour is set in accordance with the POV of the user designated as the host. 7. The method according to claim 1, characterized in that after the connection of new users to the POV of the first user, on the devices of new connected users, the function of manually moving the view camera in the POV or the function of free movement of the virtual tour POV is activated. 8. The method according to claim 4, characterized in that the host user controls the mute/unmute of the microphone of other users in the chat session. 9. The method according to claim 1, characterized in that the display of the virtual tour for each new connected user begins with repeating the route of movement along the POV of the first user. 10. The method according to claim 1, characterized in that the presenter's avatar is configured to form a virtual pointer. 11. The method according to claim 1, characterized in that the 3D virtual tour model contains 3D objects and is a room model configured to change at least the texture of the 3D model and/or 3D objects. 12. The method according to claim 11, characterized in that the 3D objects contain metadata that makes it possible to obtain additional information about the object, including at least one of: text information, video, a link to an external resource. 13. The method according to claim 12, characterized in that the information is displayed in an additional window located on top of the virtual tour interface. 14. The method according to claim 1, characterized in that the virtual tour interface contains a function that provides a measurement of the size of objects. 15. Navigation system for generated virtual tour scenes in multi-user mode, containing a 3D camera, a server and user devices, in which performing a survey of the surrounding space using a 3D camera containing a lidar, while shooting receive a group of images, on the basis of which HDR images are formed, and point clouds of the surrounding space; transmitting the received data during shooting with a 3D camera to a server, which generates a three-dimensional model of a virtual tour, which is a 3D model textured with 360-degree photo panoramas, while the virtual tour is displayed in the interface of the user device; receive commands from user devices to connect new users to the mentioned virtual tour, while receive data POV, which is the first user; forming a virtual tour display for each new user in the POV in which the first user is located, and the virtual tour display for each new user after connection duplicates the display of the first user; display a virtual tour for users with the ability to view and navigate through the specified POV.

Images