CN117676241A - Digital factory large scene display method and system based on server three-dimensional rendering - Google Patents

Digital factory large scene display method and system based on server three-dimensional rendering Download PDF

Info

Publication number
CN117676241A
CN117676241A CN202311595238.9A CN202311595238A CN117676241A CN 117676241 A CN117676241 A CN 117676241A CN 202311595238 A CN202311595238 A CN 202311595238A CN 117676241 A CN117676241 A CN 117676241A
Authority
CN
China
Prior art keywords
rendering
scene
dimensional
service
client
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311595238.9A
Other languages
Chinese (zh)
Inventor
李建勋
周云鹏
燕祥庆
张素梅
梅敬成
杨洪鹏
席瀚博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong Hoteam Software Co ltd
Original Assignee
Shandong Hoteam Software Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong Hoteam Software Co ltd filed Critical Shandong Hoteam Software Co ltd
Priority to CN202311595238.9A priority Critical patent/CN117676241A/en
Publication of CN117676241A publication Critical patent/CN117676241A/en
Pending legal-status Critical Current

Links

Abstract

The invention provides a method and a system for displaying a large scene of a digital factory based on three-dimensional rendering of a server, which relate to the technical field of digital twinning and specifically comprise the following steps: the client requests a background rendering service of the appointed scene from the cloud platform; the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode; the client performs image decoding and playing on the received video stream, and displays a large digital factory scene; according to the invention, complex software is not required to be installed and deployed, and detailed equipment states in a digital factory scene can be smoothly browsed at any terminal and any time and place.

Description

Digital factory large scene display method and system based on server three-dimensional rendering
Technical Field
The invention belongs to the technical field of digital twinning, and particularly relates to a method and a system for displaying a large scene of a digital factory based on three-dimensional rendering of a server.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
With the rise of the application of the digital twin and simulation fields in the industrial fields of petrochemical industry, nuclear power, electric power and the like, the three-dimensional visual browsing technology has a great demand on the high-performance browsing of large-scale digital factory construction and large-scale assembly.
In the digital industrial field, digital terminals are various in variety, terminal performance is uneven, especially factory with wider factory scale, digital twin models are complex, and due to hardware configuration, terminal diversity browsing and the like, the problem of insufficient performance exists in large-scene and complex-scene rendering model file browsing, and visual experience is affected.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides the method and the system for displaying the large digital plant scene based on the three-dimensional rendering of the server, which can smoothly browse the digital plant scene and the detailed equipment state in the scene at any terminal and any time and place without installing and deploying complex software.
To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:
the invention provides a digital factory large scene display method based on three-dimensional rendering of a server.
A digital factory large scene display method based on server three-dimensional rendering comprises the following steps:
the client requests a background rendering service of the appointed scene from the cloud platform;
the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode;
and the client performs image decoding and playing on the received video stream, and displays the large digital factory scene.
Further, the client sends a rendering request to the cloud platform through a portal of the cloud platform;
the portal of the cloud platform comprises rendering scene management, user session management, video streaming service, control streaming service, background rendering service agent and management end API interface gateway module.
Further, after the cloud platform receives the request, a background rendering service aiming at the three-dimensional scene of the client is started, a scene rendering model file is loaded in the cloud platform by the background rendering service, after the scene rendering model file is loaded, a background service GPU resource rendering model is utilized, and the background rendering service captures a rendered three-dimensional image in real time.
Further, the method of video streaming transmits the three-dimensional image captured in real time to the client in real time, specifically:
capturing a rendered three-dimensional image;
converting the three-dimensional image into video through video frame buffering;
encoding the video;
and transmitting the encoded video to the client.
Furthermore, the encoding adopts an H264 format, and the encoded video is pushed to the client in real time through a WebRTC or RTSP network protocol.
Further, the method also comprises the step that the client sends the control command to the background rendering service, and the background rendering service controls the background rendering scene according to the command.
Further, the control command is sent to the background rendering service by using the WebSocket technology.
A second aspect of the present invention provides a digital factory large scene display system based on server three-dimensional rendering.
The digital factory large scene display system based on the three-dimensional rendering of the server comprises a rendering request module, a scene rendering module and a scene display module:
a rendering request module configured to: the client requests a background rendering service of the appointed scene from the cloud platform;
a scene rendering module configured to: the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode;
a scene display module configured to: and the client performs image decoding and playing on the received video stream, and displays the large digital factory scene.
A third aspect of the present invention provides a computer readable storage medium having stored thereon a program which when executed by a processor performs the steps in a method for displaying a large digital plant scene based on server three-dimensional rendering according to the first aspect of the present invention.
A fourth aspect of the present invention provides an electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, the processor implementing the steps in a method for displaying a digitized factory large scene based on server three-dimensional rendering according to the first aspect of the invention when the program is executed.
The one or more of the above technical solutions have the following beneficial effects:
according to the invention, three-dimensional rendering calculation is completed on a cloud platform with higher calculation power, and pictures are transmitted to a client in real time in a video stream mode, so that the capability of browsing large scenes and complex digital and analog of a factory on a terminal with general performance and even an industrial personal computer is realized, and detailed equipment states in the scenes of a digital factory can be smoothly browsed at any terminal and any time and place without installing and deploying complex software.
Additional aspects of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.
Fig. 1 is a schematic diagram of a rendering principle.
Fig. 2 is a flow chart of a method of the first embodiment.
Fig. 3 is an overall flowchart of the first embodiment.
Fig. 4 is a service map of the first embodiment.
Fig. 5 is a communication relationship diagram of the first embodiment.
Detailed Description
It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
As described in the background art, rendering on a common terminal or a mobile terminal in the digital industrial field cannot meet the browsing performance, therefore, the invention provides a server three-dimensional rendering for displaying large scenes in a digital factory, the server three-dimensional rendering technology is based on cloud computing, and all models are rendered in three dimensions on a cloud server; the three-dimensional scene picture generated at the cloud is transmitted to the user terminal in real time for display by means of a 5G high-speed network in a video form, and the user terminal performs various operations on the twin scene and the model, so that the three-dimensional scene picture has almost no difference with the local rendering operation experience; the rendering principle is as shown in fig. 1:
image data stream: the background rendering service captures images output to a screen by a three-dimensional engine (capturing 30-60 times per second), sequentially compresses and codes the images into an H264 format video stream, and transmits the video stream to a client through RTSP or WebRTC protocol; and the client operates reversely, decodes the video stream and finally displays the decoded video stream on a screen.
Control flow: the operations of a mouse, a keyboard, commands and the like of the client are transmitted to the rendering service through the WebSocket, and the rendering service of the background controls the rendering engine to respond to the rendering response.
The user can smoothly browse details of the digital twin factories and each device in the factories in the Web browser without installing other software by utilizing the WebRTC technology, and the security of digital and analog is ensured by utilizing the video streaming technology.
Example 1
In one embodiment of the present disclosure, there is provided a method for displaying a large digital factory scene based on server three-dimensional rendering, as shown in fig. 2, including the steps of:
step S1: the client requests a background rendering service of the appointed scene from the cloud platform;
step S2: the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode;
step S3: and the client performs image decoding and playing on the received video stream, and displays the large digital factory scene.
Further, the client sends a rendering request to the cloud platform through a portal of the cloud platform;
the portal of the cloud platform comprises rendering scene management, user session management, video streaming service, control streaming service, background rendering service agent and management end API interface gateway module.
Further, after the cloud platform receives the request, a background rendering service aiming at the three-dimensional scene of the client is started, a scene rendering model file is loaded in the cloud platform by the background rendering service, after the scene rendering model file is loaded, a background service GPU resource rendering model is utilized, and the background rendering service captures a rendered three-dimensional image in real time.
Further, the method of video streaming transmits the three-dimensional image captured in real time to the client in real time, specifically:
capturing a rendered three-dimensional image;
converting the three-dimensional image into video through video frame buffering;
encoding the video;
and transmitting the encoded video to the client.
Furthermore, the encoding adopts an H264 format, and the encoded video is pushed to the client in real time through a WebRTC or RTSP network protocol.
Further, the method also comprises the step that the client sends the control command to the background rendering service, and the background rendering service controls the background rendering scene according to the command.
Further, the control command is sent to the background rendering service by using the WebSocket technology.
The implementation process of the digital factory large scene display method based on the server three-dimensional rendering in the embodiment is described in detail below.
The background service is deployed on the cloud platform, provides background three-dimensional scene rendering capability for the cloud platform, and the key problems for realizing large scene display of the digital factory include: the overall flow is shown in figure 3, and a user (i.e. a client) enters a server rendering management page through a cloud platform portal to request a background rendering service of a specified scene; after receiving the request, the platform starts a three-dimensional scene background rendering service instance aiming at the user; and the background rendering service loads a scene rendering model file in the cloud platform. After the scene rendering model file is loaded, a background service GPU resource rendering model is utilized, wherein the model is a digital three-dimensional model and is used for three-dimensional rendering, the model file comprises the layout (assembly relation) of a digital factory, triangle information of a rendering scene and the like, a background rendering service captures a rendered image in real time, the image is encoded into an H264 format, continuous image encoding (30-60 images per second) forms a video stream, and the image is pushed to a client in real time through webRTC or RTSP network protocol; the client sends the control command to the background rendering service by utilizing the WebSocket technology, and the background rendering service makes rendering scene transformation according to the command.
As a specific implementation manner, the background rendering service of the embodiment is mainly implemented by using c++ language, three-dimensional scene rendering is implemented by using an M3D three-dimensional rendering engine, a rendered image is captured by using a OpenGL readpixels interface, an H264 video coding library is used for compressing the image or NVENC hardware of an inflight video card is used for compressing the image (the encoding of the image by using an H264 algorithm is a video compression mode, and H265, VP8 and the like are also used in the mainstream), a video stream is pushed by using RTSP or WebRTC network protocol, the front-end client terminal decodes and plays the image based on WebRTC of a browser, and the control of the client terminal to the background rendering service is implemented by using the front-end and back-end communication protocol customized by using the embodiment through WebSocket technology.
The server rendering cloud platform, the background rendering service and the front-end and back-end communication protocol are described in detail below.
1. Server rendering cloud platform
Server rendering relies on a cloud platform. The user initiates a background rendering instance through the management page and browses the three-dimensional digitized factory scene presented in a video stream through the browsing page with interaction. The cloud platform comprises modules such as rendering instance management, user session management, video streaming service, control streaming service, background rendering service proxy, management end API interface gateway and the like, wherein the rendering instance management is used for managing background rendering service processes. When a user requests a rendering service, the server rendering platform initiates a background rendering instance of an independent process. The number and the state of the background rendering instances currently running can be queried through the rendering instance management module. User session management is responsible for managing users currently browsing three-dimensional scenes. The method can inquire the number of the users browsing the current scene and control the operation authority of the user to the scene. The video stream service pushes the three-dimensional scene presented by the video stream to the client of each user through a unified interface. And distributing the scene control information of each user to the appointed background rendering service through a unified interface of the control flow service. The background rendering service agent is used for identifying and mapping background rendering instances. One agent and one background render instance 1:1 mapping. And the rendering cloud platform API gateway provides the user with function call through a Restfu style network interface. Comprising an administrator interface: and starting, closing and inquiring the background rendering service. Common user interface: inquiring, setting rendering field functions, such as inquiring assembly structures, selecting and highlighting scene specified objects.
When the cloud platform main program is started, two main steps are executed:
(1) Starting an API gateway Web service, registering a rendering service management interface of a Restful style to the API gateway, wherein the registering interfaces are as follows:
the command structure body is contained in a communication protocol, the communication protocol defines communication data structures such as background command call, keyboard, mouse and the like, the command structure body is customized by adopting Protobuf, serialization and storage of a front JS grammar and a background C++ grammar are supported by adopting intermediate language definition, and the command structure is defined as follows:
(2) Video proxy (push, pull), user control flow listening service is started.
When a user wants to create a background rendering service through an API, a background service agent (AgentRS) is created firstly, and is responsible for mapping a user video session, a control flow session and the background rendering service so as to uniformly realize transparent control of the user using a portal to the background rendering service of each session, wherein the service mapping relationship is shown in fig. 4:
the user requests to initiate a background rendering instance through the API. Because each background rendering instance is an independent process, the platform manages the background rendering instance process through the background rendering instance agent, and the background rendering instance agent and the background rendering instance process are 1:1 relationship.
A background rendering instance agent comprises 1 background rendering session and N user sessions. The background rendering session is responsible for communicating with the background rendering instance; one background rendering scene may allow multiple people to browse at the same time, so the background rendering instance agent has a session with the user of 1: n mapping relation.
2. Background rendering services
Each rendering service corresponds to one rendering scene, SVLX three-dimensional scene data is loaded on a cloud platform, and an M3D engine is used for rendering three-dimensional graphics; each rendering scene allows multi-user viewing, only one of which is allowed to become a master at the same time, each rendering service is mapped one-to-one to a rendering service proxy (AgentRS) of the rendering portal, through which the proxy communicates with the rendering portal service, and the background rendering service pushes the video stream to a Media Server (Media Server), which redistributes the video stream to each accessed client. The Control flow uses an autonomous Control protocol network packet to carry out WebSocket communication through a Control service (Control Server), so that a user can Control a background rendering scene; the background rendering service, the server rendering portal, and the inter-user communication are as shown in fig. 5:
the upper diagram is a schematic diagram of the working principle of a certain background rendering instance in the platform and communicated with the platform. The video source processing thread captures and writes successive images rendered by the M3D three-dimensional rendering engine into the video source buffer. The video coding compression thread reads images from the video source buffer area and sends video frames coded and compressed by the H264 algorithm to the video propeller. The video streaming device pushes the video stream to the platform streaming media server through the RTSP protocol, converts the video stream into the video stream of the WebRTC protocol to be pushed to the Web client or the original client of the RTSP protocol through the unified interface of the platform streaming media service.
The platform provides a unified scene control service interface. The user sends the scene control information to the platform, and the platform distributes the control information to the appointed background scene rendering service.
3. Front-end and back-end communication protocols
In order to realize efficient and stable cross-language cross-platform front-end and back-end communication, the embodiment uses Protobuf to define a set of complete front-end and back-end communication protocols, uses intermediate linguistic definition, can compile a data structure compatible with JS/C++/C# development language, compresses communication data into binary system for transmission, greatly improves transmission efficiency, reduces delay and improves interaction experience; the protocol data structure is defined as follows:
the protocol is a communication interaction foundation of the whole system, is used for receiving user control information through control flow service of the platform, and the platform distributes the control information to a designated scene, so that efficient, stable and low-delay operation of the system is ensured.
Example two
In one embodiment of the present disclosure, a digital factory large scene display system based on server three-dimensional rendering is provided, which includes a rendering request module, a scene rendering module, and a scene display module:
a rendering request module configured to: the client requests a background rendering service of the appointed scene from the cloud platform;
a scene rendering module configured to: the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode;
a scene display module configured to: and the client performs image decoding and playing on the received video stream, and displays the large digital factory scene.
Example III
An object of the present embodiment is to provide a computer-readable storage medium.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps in a method for server three-dimensional rendering based digital factory large scene display according to an embodiment of the present disclosure.
Example IV
An object of the present embodiment is to provide an electronic apparatus.
An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor implements the steps in a method for displaying a large digital plant scene based on server three-dimensional rendering according to the first embodiment of the present disclosure when executing the program.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. The method for displaying the large digital factory scene based on the three-dimensional rendering of the server is characterized by comprising the following steps:
the client requests a background rendering service of the appointed scene from the cloud platform;
the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode;
and the client performs image decoding and playing on the received video stream, and displays the large digital factory scene.
2. The method for displaying the large digital factory scene based on the three-dimensional rendering of the server according to claim 1, wherein the client sends a rendering request to the cloud platform through a portal of the cloud platform;
the portal of the cloud platform comprises rendering scene management, user session management, video streaming service, control streaming service, background rendering service agent and management end API interface gateway module.
3. The method for displaying the large digital factory scene based on the three-dimensional rendering of the server according to claim 1, wherein after the cloud platform receives the request, a background rendering service for the three-dimensional scene of the client is started, the background rendering service loads a scene rendering model file in the cloud platform, and after the scene rendering model file is loaded, a background service GPU resource rendering model is utilized, and the background rendering service captures the rendered three-dimensional image in real time.
4. The method for displaying a large digital factory scene based on three-dimensional rendering of a server according to claim 1, wherein the method for transmitting the three-dimensional image captured in real time to a client in real time by means of a video stream specifically comprises the following steps:
capturing a rendered three-dimensional image;
converting the three-dimensional image into video through video frame buffering;
encoding the video;
and transmitting the encoded video to the client.
5. The method for displaying a large digital factory scene based on three-dimensional rendering of a server according to claim 4, wherein the encoding adopts an H264 format, and the encoded video is pushed to a client in real time through WebRTC or RTSP network protocol.
6. The method for displaying a large digital plant scene based on three-dimensional rendering of claim 1, further comprising the client sending a control command to a background rendering service, the background rendering service controlling the background rendering scene according to the command.
7. The method for displaying a digitized factory large scene based on three-dimensional rendering of claim 6, wherein the control command is sent to a background rendering service using WebSocket technology.
8. The digital factory large scene display system based on the three-dimensional rendering of the server is characterized by comprising a rendering request module, a scene rendering module and a scene display module:
a rendering request module configured to: the client requests a background rendering service of the appointed scene from the cloud platform;
a scene rendering module configured to: the background rendering service loads three-dimensional scene data of the appointed scene, renders a three-dimensional image by using a three-dimensional rendering engine, and transmits the three-dimensional image captured in real time to the client in real time in a video stream mode;
a scene display module configured to: and the client performs image decoding and playing on the received video stream, and displays the large digital factory scene.
9. An electronic device, comprising:
a memory for non-transitory storage of computer readable instructions; and
a processor for executing the computer-readable instructions,
wherein the computer readable instructions, when executed by the processor, perform the method of any of the preceding claims 1-7.
10. A storage medium, characterized by non-transitory storing computer-readable instructions, wherein the instructions of the method of any one of claims 1-7 are performed when the non-transitory computer-readable instructions are executed by a computer.
CN202311595238.9A 2023-11-27 2023-11-27 Digital factory large scene display method and system based on server three-dimensional rendering Pending CN117676241A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311595238.9A CN117676241A (en) 2023-11-27 2023-11-27 Digital factory large scene display method and system based on server three-dimensional rendering

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311595238.9A CN117676241A (en) 2023-11-27 2023-11-27 Digital factory large scene display method and system based on server three-dimensional rendering

Publications (1)

Publication Number Publication Date
CN117676241A true CN117676241A (en) 2024-03-08

Family

ID=90081834

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311595238.9A Pending CN117676241A (en) 2023-11-27 2023-11-27 Digital factory large scene display method and system based on server three-dimensional rendering

Country Status (1)

Country Link
CN (1) CN117676241A (en)

Similar Documents

Publication Publication Date Title
CN111681167B (en) Image quality adjusting method and device, storage medium and electronic equipment
WO2018121014A1 (en) Video play control method and apparatus and terminal device
CN103414751B (en) A kind of PC screen content sharing/interaction control method
CN101505365B (en) Real-time video monitoring system implementing method based on network television set-top box
CN1856819B (en) System and method for network transmission of graphical data through a distributed application
CN102203760B (en) Remote computing platforms providing high-fidelity display and interactivity for clients
CN105637886B (en) Server from graphic user interface to client and client for providing
US20230215076A1 (en) Image frame display method, apparatus, device, storage medium, and program product
CN112799891B (en) iOS device testing method, device, system, storage medium and computer device
CN103152573A (en) Method and system for transmitting image frame between mobile terminal and intelligent television set
WO2024060663A1 (en) Three-dimensional model scene interaction method, system and apparatus, and device, storage medium and computer program product
CN102664939A (en) Method and device for mobile terminal of screen mirror image
CN112601096A (en) Video decoding method, device, equipment and readable storage medium
CN109040786A (en) Transmission method, device, system and the storage medium of camera data
CN111464828A (en) Virtual special effect display method, device, terminal and storage medium
CN102770827A (en) Method for displaying multimedia content on a screen of a terminal
CN113973224A (en) Method for transmitting media information, computing device and storage medium
CN103581695A (en) System and method for achieving access of mobile terminal to global eye
CN117676241A (en) Digital factory large scene display method and system based on server three-dimensional rendering
CN114938408B (en) Data transmission method, system, equipment and medium of cloud mobile phone
CN105025359A (en) Remote video display method and system based on SPICE protocol
US10223997B2 (en) System and method of leveraging GPU resources to increase performance of an interact-able content browsing service
CN112565818B (en) Two-three-dimensional integrated operation station platform construction method and electronic equipment
CN110990109B (en) Spliced screen back display method, terminal, system and storage medium
CN111901628A (en) Cloud rendering method based on zSpace desktop VR all-in-one machine

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination