CN114900743A - Scene rendering transition method and system based on video plug flow - Google Patents
Scene rendering transition method and system based on video plug flow Download PDFInfo
- Publication number
- CN114900743A CN114900743A CN202210460639.2A CN202210460639A CN114900743A CN 114900743 A CN114900743 A CN 114900743A CN 202210460639 A CN202210460639 A CN 202210460639A CN 114900743 A CN114900743 A CN 114900743A
- Authority
- CN
- China
- Prior art keywords
- virtual point
- point location
- dimensional model
- scene
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000007704 transition Effects 0.000 title claims abstract description 51
- 238000009877 rendering Methods 0.000 title claims abstract description 39
- 238000004590 computer program Methods 0.000 claims description 22
- 230000004044 response Effects 0.000 claims description 11
- 238000009499 grossing Methods 0.000 claims description 10
- 238000007499 fusion processing Methods 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 6
- 238000012545 processing Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 4
- 230000004927 fusion Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
- H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The application is applicable to the technical field of image processing, and provides a scene rendering transition method, a device, a server, terminal equipment and a computer readable storage medium based on video plug flow, wherein the method comprises the following steps: acquiring a three-dimensional model; determining a browsing mode of a user viewing the three-dimensional model as a roaming mode; responding to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, and acquiring target video information between the current virtual point location and the next adjacent virtual point location; and playing the target video information between the current virtual point location and the next adjacent virtual point location to the user. Therefore, compared with the prior art that the scene where one virtual point position in the three-dimensional model is directly transited to the scene where the next virtual point position is located, the method and the device for processing the video plug flow slowly transit from the scene where one virtual point position in the three-dimensional model is located to the scene where the next virtual point position is located, and the effect that the scene transition is more real and smooth is achieved.
Description
Technical Field
The application belongs to the technical field of image processing, and particularly relates to a scene rendering transition method and device based on video plug flow, a server, terminal equipment and a computer-readable storage medium.
Background
In the prior art, three-dimensional scene rendering transition generally refers to directly transitioning from a scene where one virtual point location in a three-dimensional model is located to a scene where the next virtual point location is located, and the problem that the transition effect is not real and smooth enough still exists.
Disclosure of Invention
The embodiment of the application provides a scene rendering transition method, a scene rendering transition device, a scene rendering transition system, a scene rendering transition server, a terminal device and a computer-readable storage medium based on video plug flow.
In a first aspect, an embodiment of the present application provides a scene rendering transition method based on video plug flow, where the method is applied to a terminal device, and the method includes:
acquiring a three-dimensional model; the three-dimensional model comprises at least two virtual point positions;
determining that a browsing mode of a user watching the three-dimensional model is a roaming mode;
responding to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, and acquiring target video information between the current virtual point location and the next adjacent virtual point location;
and playing the target video information between the current virtual point location and the next adjacent virtual point location to the user.
In an optional implementation manner of the first aspect, determining that a browsing mode of the user viewing the three-dimensional model is a roaming mode includes:
calling a first projection camera to display the outline scene of the three-dimensional model to a user;
and responding to a roaming request of the user, replacing the first projection camera with a second projection camera, and displaying the internal scene of the three-dimensional model to the user.
In an optional implementation manner of the first aspect, acquiring, in response to a scene transition request from a user to transition from a current virtual point location to a next adjacent virtual point location, target video information between the current virtual point location and the next adjacent virtual point location includes:
determining that the virtual point position where the second projection camera is located is a current virtual point position, detecting the clicking operation of the user, and taking the virtual point position corresponding to the clicking operation of the user as a next adjacent virtual point position;
generating a feature code according to the current virtual point location and the next adjacent virtual point location, sending the feature code to a server to instruct the server to inquire target video information according to the feature code, rendering the target video information, and returning the target video information to the terminal equipment.
In a second aspect, an embodiment of the present application provides a scene rendering transition method based on video plug flow, where the method is applied to a server, and the method includes:
acquiring depth information and image information corresponding to each real point location;
acquiring video information between adjacent real point locations and storing the video information;
generating a three-dimensional model according to the depth information and the image information corresponding to each real point location;
determining a virtual point position corresponding to each real point position in the three-dimensional model;
establishing an association relation between adjacent virtual point locations and the video information according to the corresponding relation between the real point locations and the virtual point locations;
and sending the three-dimensional model to a terminal device.
In an optional implementation manner of the second aspect, generating a three-dimensional model according to the depth information and the image information corresponding to each real point location includes:
generating an initial point cloud according to the depth information and the image information;
performing fusion processing on the initial point cloud to obtain a fused point cloud;
smoothing the fused point cloud to obtain a target point cloud;
and recovering the geometric structure of the target point cloud to obtain a three-dimensional model.
In an optional implementation manner of the second aspect, after sending the three-dimensional model to the terminal device, the method further includes:
acquiring a feature code sent by terminal equipment;
inquiring target video information according to the feature codes;
rendering the target video;
and sending the target video to the terminal equipment.
In a third aspect, an embodiment of the present application provides a scene rendering transition system based on video plug flow, where the system includes:
the acquisition equipment is used for acquiring the depth information and the image information corresponding to each real point location and sending the depth information and the image information corresponding to each real point location to the server; collecting video information between adjacent real point locations and sending the video information to a server;
the server is connected with the acquisition equipment and is used for acquiring depth information and image information corresponding to each real point location; acquiring video information between adjacent real point locations and storing the video information; generating a three-dimensional model according to the depth information and the image information corresponding to each real point location; determining a virtual point position corresponding to each real point position in the three-dimensional model; establishing an association relation between adjacent virtual point locations and the video information according to the corresponding relation between the real point locations and the virtual point locations; sending the three-dimensional model to a terminal device;
the terminal equipment is connected with the server and used for acquiring the three-dimensional model; the three-dimensional model comprises at least two virtual point positions; determining that a browsing mode of a user watching the three-dimensional model is a roaming mode; responding to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, and acquiring target video information between the current virtual point location and the next adjacent virtual point location; and playing the target video information between the current virtual point location and the next adjacent virtual point location to the user.
In a fourth aspect, an embodiment of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor, when executing the computer program, implements the method according to the first aspect.
In a fifth aspect, embodiments of the present application provide a server, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor, when executing the computer program, implements the method according to the second aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, and the computer program realizes the method according to the first aspect when executed by a processor.
Compared with the prior art, the embodiment of the application has the advantages that:
in the embodiment of the application, three-dimensional reconstruction is carried out through a server according to depth information and image information corresponding to the real point location sent by the acquisition equipment to obtain a three-dimensional model; establishing an incidence relation between adjacent virtual point locations and the video information in the three-dimensional model according to the acquired video information between the adjacent real point locations; and after the feature code sent by the terminal equipment is obtained, inquiring target video information according to the feature code, and pushing the rendered target video to the terminal equipment. Therefore, compared with the prior art that the scene where one virtual point position in the three-dimensional model is directly transited to the scene where the next virtual point position is, the embodiment of the application uses a video plug flow mode to slowly transit from the scene where one virtual point position in the three-dimensional model is located to the scene where the next virtual point position is located, and achieves the effect that the scene transition is more real and smooth.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic structural diagram of a scene rendering transition system based on video plug flow according to an embodiment of the present application;
fig. 2 is a flowchart illustrating a scene rendering transition method based on video plug flow according to an embodiment of the present application;
fig. 3 is another flowchart illustrating a scene rendering transition method based on video plug flow according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a scene rendering transition device based on video plug flow according to an embodiment of the present application;
fig. 5 illustrates another scene rendering transition device based on video plug-streaming according to an embodiment of the present application;
FIG. 6 is a schematic structural diagram of a server provided in an embodiment of the present application;
fig. 7 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;
fig. 8 is a schematic plan view of a three-dimensional model provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather mean "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
The technical solutions provided in the embodiments of the present application will be described below with specific embodiments.
Referring to fig. 1, a schematic structural diagram of a scene rendering transition system based on video plug flow provided in an embodiment of the present application includes a capture device 10, a server 20, and a terminal device 30, where the capture device is connected to the server, and the server is connected to the terminal device.
The acquisition equipment is used for acquiring the depth information and the image information corresponding to each real point location and sending the depth information and the image information corresponding to each real point location to the server; and collecting video information between adjacent real point positions and sending the video information to a server.
The acquisition device may be a 3D camera, the 3D camera includes a laser camera and a dome camera, a relative position between the laser camera and the dome camera is fixed, the laser camera includes a first laser camera (i.e., an UpLidar laser camera), a second laser camera (i.e., a MidLidar laser camera) and a third laser camera (i.e., a DownLidar laser camera), which are fixed relative positions, and the exemplary UpLidar laser camera, MidLidar laser camera and DownLidar laser camera are arranged at intervals in the order of upper, middle and lower on the same vertical line. In addition, the 3D camera rotates for 6 times at the same point according to the rotation angle of 60 degrees to finally obtain a 360-degree panoramic image.
The server is used for acquiring depth information and image information corresponding to each real point location; acquiring video information between adjacent real point locations and storing the video information; generating a three-dimensional model according to the depth information and the image information corresponding to each real point location; determining a virtual point position corresponding to each real point position in the three-dimensional model; establishing an association relation between adjacent virtual point locations and the video information according to the corresponding relation between the real point locations and the virtual point locations; and sending the three-dimensional model to a terminal device.
The terminal equipment is used for acquiring the three-dimensional model; the three-dimensional model comprises at least two virtual point positions; determining a browsing mode of a user watching the three-dimensional model as a roaming mode; responding to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, and acquiring target video information between the current virtual point location and the next adjacent virtual point location; and playing the target video information between the current virtual point location and the next adjacent virtual point location to a user.
In the embodiment of the application, three-dimensional reconstruction is carried out through a server according to depth information and image information corresponding to the real point location sent by the acquisition equipment to obtain a three-dimensional model; establishing an incidence relation between adjacent virtual point locations and the video information in the three-dimensional model according to the acquired video information between the adjacent real point locations; and then, acquiring the three-dimensional model through the terminal equipment, determining that a browsing mode of a user watching the three-dimensional model is a roaming mode, responding to a scene transition request of the user from the current virtual point location to the next adjacent virtual point location, acquiring target video information between the current virtual point location and the next adjacent virtual point location, and playing the target video information between the current virtual point location and the next adjacent virtual point location to the user. Therefore, scene transition between the point locations is achieved through the video plug flow mode, so that transition effects between the point locations are more realistic and smoother.
Referring to fig. 2, a flowchart of a scene rendering transition method based on video plug-streaming provided in an embodiment of the present application is schematically illustrated, by way of example and not limitation, the method may be applied to a server, and the method may include the following steps:
step S201, obtaining depth information and image information corresponding to each real point location.
In specific application, the depth information and the image information are collected at different real point positions in a real scene through a 3D camera. For example, the operation flow of the user in the real scene may be: installing and fixing the 3D camera, starting the 3D camera, opening a WiFi connection function of the mobile phone or the ipad, starting a small program of the mobile phone or the ipad, planning a shooting route by a user, shooting by the 3D camera of the user, storing space data by the 3D camera, and automatically uploading the space data to the server by the 3D camera.
And S202, acquiring video information between adjacent real point positions and storing the video information.
In specific application, a section of video (for example, 60 frames) is recorded between adjacent real point positions through a 3D camera, then the video is encoded and packaged and then uploaded to a server for storage, and each section of video is marked to correspond to one feature code.
Step S203, generating a three-dimensional model according to the depth information and the image information corresponding to each real point location.
In specific application, the three-dimensional model is generated according to the depth information and the image information corresponding to each real point position, and the method comprises the following steps:
and step S203-1, generating an initial point cloud according to the depth information and the image information.
Specifically, generating an initial point cloud from depth information and image information includes:
and S203-1-1, registering the depth information and the image information to obtain panoramic image information.
Illustratively, the target depth image and the color image may be registered according to an Iterative Closest Point algorithm ICP) such that a coordinate system is unified between the target depth image and the color image.
And S203-1-2, obtaining the three-dimensional coordinates of the initial point cloud according to the panoramic image information.
Illustratively, the three-dimensional coordinates of the initial point cloud are obtained according to the following formula:
wherein, (u, v) is the pixel coordinate of each target feature point in the panoramic image, d is the depth value of each pixel point in the panoramic image, K is the internal reference of the laser camera, and (X, Y, Z) is the three-dimensional coordinate of the point cloud. The internal reference of the laser camera can be calculated by adopting a Zhangzhen calibration method.
And S203-2, performing fusion processing on the initial point cloud to obtain a fused point cloud.
The target depth image and the color image are obtained by shooting through a laser camera, the laser camera comprises a laser and a camera, the laser is used for collecting the depth image and sending the depth image to the server, and the camera is used for collecting the color image and sending the color image to the server.
In specific application, the initial point cloud is subjected to fusion processing to obtain a fused point cloud, and the method comprises the following steps:
and S203-2-1, acquiring pose information of the laser camera.
Illustratively, the pose-state information of the laser camera is obtained by using an SFM (structure-from-motion) algorithm, which is an off-line algorithm for performing three-dimensional reconstruction based on various collected disordered pictures. The name implies recovering the three-dimensional structure of an object from motion (a collection of pictures taken at different times), which requires the reconstruction of a sparse point cloud, together with camera parameters, of the picture.
And S203-2-2, carrying out reprojection probability fusion on the initial point cloud according to the pose information to obtain a fused point cloud.
It can be understood that after the pose of the laser camera is obtained by the SFM algorithm, the multiple initial point clouds are subjected to reprojection probability fusion, so that the point clouds observed by the laser camera at different positions are not layered and conflicted.
And S203-3, smoothing the fused point cloud to obtain a target point cloud.
Exemplarily, smoothing the fused point cloud to obtain a target point cloud, including:
and S203-3-1, extracting the feature vector of the fused point cloud by using a nonparametric regression algorithm.
The feature vector comprises a normal vector and a scaling degree.
It is understood that the non-parametric method is assumed to be of a good form, and the non-parametric method is equivalent to a smoothing technique, the form is not known in advance, but is generated by using samples, and the common methods include a nearest neighbor method, a kernel function method and the like.
And S203-3-2, smoothing the feature vector, and taking the smoothed fusion point cloud as a target point cloud.
Exemplary ways of smoothing include a Katz smoothing method and the like.
And S203-4, recovering the geometric structure of the target point cloud to obtain a three-dimensional model.
In specific application, a surface triangle of a target point cloud is extracted according to a preset reconstruction algorithm, and a target three-dimensional model is generated.
The preset reconstruction algorithm may be a dense point cloud reconstruction MVS algorithm.
And step S204, determining a virtual point position corresponding to each real point position in the three-dimensional model.
It can be understood that a three-dimensional model is generated by the depth information and the image information obtained by shooting a real scene through the 3D camera, and a virtual-real relationship between the real scene and the three-dimensional model is established, so that a virtual point location corresponding to each real point location in the three-dimensional model can be obtained according to the coordinates of the real point location in the real scene.
Step S205, according to the corresponding relation between the real point location and the virtual point location, the incidence relation between the adjacent virtual point location and the video information is established.
It can be understood that the video information is uploaded to a server for storage after the 3D camera shoots between the adjacent real point locations, and based on the corresponding relationship between each real point location and the virtual point location, the association relationship between the adjacent virtual point location and the video information can be established.
And step S206, sending the three-dimensional model to the terminal equipment.
In an optional implementation manner, after sending the three-dimensional model to the terminal device, the method further includes:
and step S206-1, acquiring the feature code sent by the terminal equipment.
The feature code is generated by the terminal device according to target video information involved in transition from a scene where the current virtual point location is located to a scene where the next adjacent virtual point location is located, and the generation mode can be a one-shot coding mode.
And S206-2, inquiring the target video information according to the feature code.
And step S206-3, rendering the target video.
Illustratively, the target video is rendered by way of UE4 rendering.
And step S206-4, sending the target video to the terminal equipment.
Specifically, a WebRTC protocol plug-flow mode is adopted to send the target video to the terminal device.
In the embodiment of the application, three-dimensional reconstruction is carried out through a server according to depth information and image information corresponding to the real point location sent by the acquisition equipment to obtain a three-dimensional model; establishing an incidence relation between adjacent virtual point locations and the video information in the three-dimensional model according to the acquired video information between the adjacent real point locations; and after the feature code sent by the terminal equipment is obtained, inquiring target video information according to the feature code, and pushing the rendered target video to the terminal equipment. Therefore, compared with the prior art in which the scene where one virtual point in the three-dimensional model is located is directly transited to the scene where the next virtual point is located, the embodiment of the present application uses a video plug flow mode to slowly transit from the scene where one virtual point in the three-dimensional model is located to the next virtual point.
Referring to fig. 3, another flow diagram of a scene rendering transition method based on video plug flow provided in an embodiment of the present application is shown, by way of example and not limitation, the method may be applied to a terminal device, and the method may include the following steps:
step S301, a three-dimensional model is obtained.
The three-dimensional model comprises at least two virtual point positions.
In specific application, the server pushes and sends the three-dimensional model rendered by the UE4 to the terminal device through the WebRTC protocol.
Step S302, determining that the browsing mode of the user watching the three-dimensional model is a roaming mode.
The browsing mode comprises a plane mode, a 3D mode and a roaming mode, wherein the plane mode is used for displaying a plane of the three-dimensional model, the 3D mode is used for displaying a profile scene outside the three-dimensional model, and the roaming mode is used for displaying a scene inside the three-dimensional model.
In specific application, determining that a browsing mode of a user watching the three-dimensional model is a roaming mode comprises the following steps:
and step S302-1, calling a first projection camera to display the outline scene of the three-dimensional model to a user.
Wherein, the first projection camera is an orthogonal projection camera. The first projection camera is a virtual camera that is simulated outside the three-dimensional scene.
It is understood that, initially, the browsing mode in which the user views the three-dimensional model is the 3D mode or the planar mode, the terminal device calls the first projection camera to display the outline scene of the three-dimensional model to the user.
And step S302-2, responding to the roaming request of the user, replacing the first projection camera with a second projection camera, and displaying the internal scene of the three-dimensional model to the user.
Wherein, the second projection camera is a perspective projection camera. It should be noted that the second projection camera refers to a virtual camera simulated in the three-dimensional scene.
It can be understood that the browsing mode of the three-dimensional model viewed by the user is switched from the 3D mode or the planar mode to the roaming mode, and then the terminal device replaces the first projection camera with the second projection camera to display the internal scene of the three-dimensional model to the user.
It should be noted that the viewing angle from which the user views the three-dimensional model is referred to as the third person viewing angle.
Step S303, in response to a scene transition request from the current virtual point location to the next adjacent virtual point location, obtaining target video information between the current virtual point location and the next adjacent virtual point location.
In a specific application, in response to a scene transition request from a current virtual point location to a next adjacent virtual point location from a user, obtaining target video information between the current virtual point location and the next adjacent virtual point location, including:
and step S303-1, determining the virtual point position where the second projection camera is located as the current virtual point position, detecting the clicking operation of the user, and taking the virtual point position corresponding to the clicking operation of the user as the next adjacent virtual point position.
Step S303-2, generating a feature code according to the current virtual point location and the next adjacent virtual point location, sending the feature code to a server to instruct the server to inquire the target video information according to the feature code, rendering the target video information, and returning the target video information to the terminal equipment.
It can be understood that the next adjacent virtual point location refers to an adjacent point location of the current virtual point location, and the server queries target video information corresponding to the current virtual point location and the next adjacent virtual point location according to the feature code.
Step S304, the target video information between the current virtual point location and the next adjacent virtual point location is played to the user.
Exemplarily, referring to fig. 5, a schematic plan view of a three-dimensional model provided in an embodiment of the present application is shown. The virtual point location A and the virtual point location B correspond to a material C on the same three-dimensional model, the current virtual point location of a user is the virtual point location A, the next adjacent virtual point location of the user is the virtual point location B, then, the terminal equipment obtains video information between the rendered virtual point location A and the rendered virtual point location B from the server, the video information between the virtual point location A and the virtual point location B is played to the user, and scene transition from the virtual point location A to the virtual point location B is achieved.
In the embodiment of the application, the three-dimensional model is obtained through the terminal device, the browsing mode of the user watching the three-dimensional model is determined to be a roaming mode, the scene transition request of the user from the current virtual point location to the next adjacent virtual point location is responded, the target video information between the current virtual point location and the next adjacent virtual point location is obtained, and the target video information between the current virtual point location and the next adjacent virtual point location is played to the user. Therefore, compared with the prior art that the scene where one virtual point position in the three-dimensional model is directly transited to the scene where the next virtual point position is, the embodiment of the application uses a video plug flow mode to slowly transit from the scene where one virtual point position in the three-dimensional model is located to the scene where the next virtual point position is located, and achieves the effect that the scene transition is more real and smooth.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Corresponding to the method described in the foregoing embodiment, fig. 4 shows a schematic structural diagram of a scene rendering transition device based on video plug flow provided in an embodiment of the present application, where the device is applied to a terminal device.
Referring to fig. 4, the apparatus includes:
an obtaining module 41, configured to obtain a three-dimensional model; the three-dimensional model comprises at least two virtual point positions;
a determining module 42, configured to determine that a browsing mode of the user viewing the three-dimensional model is a roaming mode;
a response module 43, configured to respond to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, to obtain target video information between the current virtual point location and the next adjacent virtual point location;
and the playing module 44 is configured to play the target video information between the current virtual point location and the next adjacent virtual point location to the user.
In one possible implementation manner, the determining module includes:
the calling unit is used for calling the first projection camera to display the outline scene of the three-dimensional model to a user;
and the response unit is used for responding to a roaming request of a user, replacing the first projection camera with a second projection camera, and displaying the internal scene of the three-dimensional model to the user.
In one possible implementation, the response module includes:
the determining unit is used for determining that the virtual point position where the second projection camera is located is a current virtual point position, detecting the clicking operation of the user, and taking the virtual point position corresponding to the clicking operation of the user as a next adjacent virtual point position;
and the generating unit is used for generating a feature code according to the current virtual point location and the next adjacent virtual point location, sending the feature code to a server so as to instruct the server to inquire target video information according to the feature code, rendering the target video information, and returning the target video information to the terminal equipment.
Corresponding to the method described in the foregoing embodiment, fig. 5 illustrates another scene rendering transition device based on video plug flow, which is applied to a server, according to an embodiment of the present application.
Referring to fig. 5, the apparatus includes:
a first obtaining module 51, configured to obtain depth information and image information corresponding to each real point location;
the second obtaining module 52 is configured to obtain video information between adjacent real point locations, and store the video information;
a generating module 53, configured to generate a three-dimensional model according to the depth information and the image information corresponding to each real point location;
a determining module 54, configured to determine a virtual point location corresponding to each real point location in the three-dimensional model;
the association module 55 is configured to establish an association relationship between the adjacent virtual point locations and the video information according to a corresponding relationship between the real point locations and the virtual point locations;
and a sending module 56, configured to send the three-dimensional model to a terminal device.
In one possible implementation manner, the generating module includes:
a generating unit for generating an initial point cloud according to the depth information and the image information;
the fusion unit is used for carrying out fusion processing on the initial point cloud to obtain a fused point cloud;
the smoothing unit is used for smoothing the fused point cloud to obtain a target point cloud;
and the recovery unit is used for recovering the geometric structure of the target point cloud to obtain a three-dimensional model.
In one possible implementation manner, the apparatus further includes:
the third acquisition module is used for acquiring the feature code sent by the terminal equipment;
the query module is used for querying the target video information according to the feature codes;
a rendering module for rendering the target video;
and the sending module is used for sending the target video to the terminal equipment.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
Fig. 6 is a schematic structural diagram of a server according to an embodiment of the present application. As shown in fig. 6, the terminal device 6 of this embodiment includes: at least one processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, the processor 60 implementing the steps in any of the various method embodiments described above when executing the computer program 62.
The terminal device 6 may be a computing device such as a cloud server. The server may include, but is not limited to, a processor 60, a memory 61. Those skilled in the art will appreciate that fig. 6 is only an example of the terminal device 6, and does not constitute a limitation to the terminal device 6, and may include more or less components than those shown, or combine some components, or different components, such as an input/output device, a network access device, and the like.
Fig. 7 is a schematic structural diagram of a terminal device according to an embodiment of the present application. As shown in fig. 7, the terminal device 7 of this embodiment includes: at least one processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, the processor 70 implementing the steps in any of the various method embodiments described above when executing the computer program 72.
The terminal device 7 may be a computing device such as a desktop computer. The terminal device may include, but is not limited to, a processor 70, a memory 71. Those skilled in the art will appreciate that fig. 7 is only an example of the terminal device 7, and does not constitute a limitation to the terminal device 7, and may include more or less components than those shown, or combine some components, or different components, for example, and may further include input/output devices, network access devices, and the like.
The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps in the above-mentioned method embodiments.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/terminal apparatus, a recording medium, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A scene rendering transition method based on video plug flow is applied to terminal equipment and is characterized by comprising the following steps:
acquiring a three-dimensional model; the three-dimensional model comprises at least two virtual point positions;
determining that a browsing mode of a user watching the three-dimensional model is a roaming mode;
responding to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, and acquiring target video information between the current virtual point location and the next adjacent virtual point location;
and playing the target video information between the current virtual point location and the next adjacent virtual point location to the user.
2. The method for transitioning from scene rendering based on video plug-streaming according to claim 1, wherein determining that the browsing mode of the user viewing the three-dimensional model is the roaming mode comprises:
calling a first projection camera to display the outline scene of the three-dimensional model to a user;
and responding to a roaming request of the user, replacing the first projection camera with a second projection camera, and displaying the internal scene of the three-dimensional model to the user.
3. The video-push-stream-based scene rendering transition method according to claim 2, wherein in response to a scene transition request from a user to transition from a current virtual point location to a next adjacent virtual point location, acquiring target video information between the current virtual point location and the next adjacent virtual point location, comprises:
determining that the virtual point position where the second projection camera is located is a current virtual point position, detecting the clicking operation of the user, and taking the virtual point position corresponding to the clicking operation of the user as a next adjacent virtual point position;
generating a feature code according to the current virtual point location and the next adjacent virtual point location, sending the feature code to a server to instruct the server to inquire target video information according to the feature code, rendering the target video information, and returning the target video information to the terminal equipment.
4. A scene rendering transition method based on video plug flow is applied to a server and is characterized by comprising the following steps:
acquiring depth information and image information corresponding to each real point location;
acquiring video information between adjacent real point positions and storing the video information;
generating a three-dimensional model according to the depth information and the image information corresponding to each real point location;
determining a virtual point position corresponding to each real point position in the three-dimensional model;
establishing an association relation between adjacent virtual point locations and the video information according to the corresponding relation between the real point locations and the virtual point locations;
and sending the three-dimensional model to a terminal device.
5. The video-plug-flow-based scene rendering transition method according to claim 4, wherein generating a three-dimensional model according to the depth information and the image information corresponding to each real point location comprises:
generating an initial point cloud according to the depth information and the image information;
performing fusion processing on the initial point cloud to obtain a fused point cloud;
smoothing the fused point cloud to obtain a target point cloud;
and recovering the geometric structure of the target point cloud to obtain a three-dimensional model.
6. The video-push-stream-based scene rendering transition method according to claim 4, further comprising, after sending the three-dimensional model to a terminal device:
acquiring a feature code sent by terminal equipment;
inquiring target video information according to the feature codes;
rendering the target video;
and sending the target video to the terminal equipment.
7. A video-plug-flow-based scene rendering transition system, comprising:
the acquisition equipment is used for acquiring the depth information and the image information corresponding to each real point location and sending the depth information and the image information corresponding to each real point location to the server; collecting video information between adjacent real point locations and sending the video information to a server;
the server is connected with the acquisition equipment and is used for acquiring depth information and image information corresponding to each real point location; acquiring video information between adjacent real point locations and storing the video information; generating a three-dimensional model according to the depth information and the image information corresponding to each real point location; determining a virtual point position corresponding to each real point position in the three-dimensional model; establishing an association relation between adjacent virtual point locations and the video information according to the corresponding relation between the real point locations and the virtual point locations; sending the three-dimensional model to a terminal device;
the terminal equipment is connected with the server and used for acquiring the three-dimensional model; the three-dimensional model comprises at least two virtual point positions; determining that a browsing mode of a user watching the three-dimensional model is a roaming mode; responding to a scene transition request from a current virtual point location to a next adjacent virtual point location by a user, and acquiring target video information between the current virtual point location and the next adjacent virtual point location; and playing the target video information between the current virtual point location and the next adjacent virtual point location to a user.
8. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 3 when executing the computer program.
9. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 4 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 3 or the method according to any one of claims 4 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460639.2A CN114900743A (en) | 2022-04-28 | 2022-04-28 | Scene rendering transition method and system based on video plug flow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210460639.2A CN114900743A (en) | 2022-04-28 | 2022-04-28 | Scene rendering transition method and system based on video plug flow |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114900743A true CN114900743A (en) | 2022-08-12 |
Family
ID=82719477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210460639.2A Pending CN114900743A (en) | 2022-04-28 | 2022-04-28 | Scene rendering transition method and system based on video plug flow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114900743A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116934984A (en) * | 2023-09-19 | 2023-10-24 | 成都中轨轨道设备有限公司 | Intelligent terminal and method for constructing virtual panoramic scene space |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111080799A (en) * | 2019-12-04 | 2020-04-28 | 广东康云科技有限公司 | Scene roaming method, system, device and storage medium based on three-dimensional modeling |
| CN112752155A (en) * | 2020-01-13 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Media data display method and related equipment |
| WO2021109376A1 (en) * | 2019-12-03 | 2021-06-10 | 深圳市商汤科技有限公司 | Method and device for producing multiple camera-angle effect, and related product |
| CN113824629A (en) * | 2021-09-30 | 2021-12-21 | 北京有竹居网络技术有限公司 | House display method, device, equipment and medium |
| CN114387400A (en) * | 2022-01-18 | 2022-04-22 | 北京有竹居网络技术有限公司 | Three-dimensional scene display method, display device, electronic equipment and server |
-
2022
- 2022-04-28 CN CN202210460639.2A patent/CN114900743A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021109376A1 (en) * | 2019-12-03 | 2021-06-10 | 深圳市商汤科技有限公司 | Method and device for producing multiple camera-angle effect, and related product |
| CN111080799A (en) * | 2019-12-04 | 2020-04-28 | 广东康云科技有限公司 | Scene roaming method, system, device and storage medium based on three-dimensional modeling |
| CN112752155A (en) * | 2020-01-13 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Media data display method and related equipment |
| CN113824629A (en) * | 2021-09-30 | 2021-12-21 | 北京有竹居网络技术有限公司 | House display method, device, equipment and medium |
| CN114387400A (en) * | 2022-01-18 | 2022-04-22 | 北京有竹居网络技术有限公司 | Three-dimensional scene display method, display device, electronic equipment and server |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116934984A (en) * | 2023-09-19 | 2023-10-24 | 成都中轨轨道设备有限公司 | Intelligent terminal and method for constructing virtual panoramic scene space |
| CN116934984B (en) * | 2023-09-19 | 2023-12-08 | 成都中轨轨道设备有限公司 | Intelligent terminal and method for constructing virtual panoramic scene space |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11270460B2 (en) | Method and apparatus for determining pose of image capturing device, and storage medium | |
| CN114119839B (en) | Three-dimensional model reconstruction and image generation method, equipment and storage medium | |
| US10388025B2 (en) | Interactive image based 3D panogragh | |
| CN112927362B (en) | Map reconstruction method and device, computer readable medium and electronic equipment | |
| US11044398B2 (en) | Panoramic light field capture, processing, and display | |
| CN112039937B (en) | Display method, position determination method and device | |
| CN112270736B (en) | Augmented reality processing method and device, storage medium and electronic equipment | |
| CN108668108B (en) | Video monitoring method and device and electronic equipment | |
| CN113223130B (en) | Path roaming method, terminal equipment and computer storage medium | |
| CN109688343A (en) | The implementation method and device of augmented reality studio | |
| CN110060201B (en) | Hot spot interaction method for panoramic video | |
| CN116057577A (en) | Map for augmented reality | |
| CN111669564A (en) | Image reconstruction method, system, device and computer readable storage medium | |
| CN114549289A (en) | Image processing method, image processing device, electronic equipment and computer storage medium | |
| CN113313832A (en) | Semantic generation method and device of three-dimensional model, storage medium and electronic equipment | |
| CN114900743A (en) | Scene rendering transition method and system based on video plug flow | |
| CN114900742A (en) | Scene rotation transition method and system based on video plug flow | |
| CN113838116A (en) | Method and device for determining target view, electronic equipment and storage medium | |
| CN110047035B (en) | Panoramic video hot spot interaction system and interaction equipment | |
| CN114089836B (en) | Labeling method, terminal, server and storage medium | |
| CN113810734B (en) | Video fusion method, device, equipment, system and computer readable storage medium | |
| CN108920598A (en) | Panorama sketch browsing method, device, terminal device, server and storage medium | |
| CN114596407A (en) | Resource object three-dimensional model generation interaction method and device, and display method and device | |
| Gilbert et al. | Inpainting of wide-baseline multiple viewpoint video | |
| CN117014698A (en) | Point location planning method, device and system based on video push stream |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220812 |
|
| RJ01 | Rejection of invention patent application after publication |