CN115225615B - Illusion engine pixel streaming method and device - Google Patents
Illusion engine pixel streaming method and device Download PDFInfo
- Publication number
- CN115225615B CN115225615B CN202210773055.0A CN202210773055A CN115225615B CN 115225615 B CN115225615 B CN 115225615B CN 202210773055 A CN202210773055 A CN 202210773055A CN 115225615 B CN115225615 B CN 115225615B
- Authority
- CN
- China
- Prior art keywords
- frame data
- rendering
- rendering target
- real
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000009877 rendering Methods 0.000 claims abstract description 231
- 238000004891 communication Methods 0.000 claims abstract description 97
- 238000012545 processing Methods 0.000 claims abstract description 39
- 238000006243 chemical reaction Methods 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 12
- 230000001965 increasing effect Effects 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 4
- 230000000694 effects Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/262—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists
- H04N21/26208—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists the scheduling operation being performed under constraints
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention provides a method and a device for streaming a pixel of a fantasy engine, wherein the method comprises the following steps: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data which can be processed by the central processing unit; and pushing the frame data of the first rendering target to the real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to the client to decode and display the frame data. The method and the device for the pixel streaming of the illusion engine realize the coding and the pixel streaming by using the real-time communication server, greatly reduce the delay of the pixel streaming and support a plurality of coding formats.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a method and a device for streaming virtual engine pixels.
Background
A illusion Engine (UE) Pixel Streaming allows illusion items to be played from any on-screen networking device (e.g., computer, cell phone, tablet, etc.). The advantage of using a pixel stream is that the output device does not need to handle all scenes, physical simulations and interactions, only one screen and internet connection is needed.
The existing pixel streaming plugin of the illusion engine is realized based on WebRTC (Web Real-Time Communication, web instant messaging) service, the illusion engine needs to send the illusion engine to WebRTC for pixel streaming after local encoding, the encoding format only supports the format of H264, and the encoding format has larger delay in video streaming with a client.
Disclosure of Invention
Aiming at the defects existing in the prior art, the embodiment of the invention provides a method and a device for streaming a pixel of a fantasy engine.
The embodiment of the invention provides a pixel streaming method of a fantasy engine, which comprises the following steps: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing; pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
According to the method for streaming the pixel of the illusion engine provided by the embodiment of the invention, the frame data after rendering is obtained in real time, and the frame data is stored to a first rendering target, and the method comprises the following steps: executing the real-time acquisition of frame data after rendering through a rendering thread, and storing the frame data to a first rendering target; the reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor, includes: executing the frame data of the first rendering target through a read-back and push thread, so as to convert the frame data into explicit data which can be processed by the central processing unit; the pushing the frame data of the first rendering target to a real-time communication server includes: and executing the frame data pushing the first rendering target to a real-time communication server through the readback and pushing threads.
According to the method for streaming the pixel of the illusion engine provided by the embodiment of the invention, the frame data after rendering is obtained in real time, and the frame data is stored to a first rendering target, and the method comprises the following steps: acquiring the frame data which is completely rendered from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data into a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server; copying the frame data in the second rendering target to the first rendering target located at the central processor.
According to an embodiment of the present invention, a method for streaming a pixel of a illusion engine, which converts a format of frame data and stores the frame data in a second rendering target in a corresponding format, includes: the frame data is subjected to format conversion by means of one-pass rendering and is stored to a second rendering target in a corresponding format.
According to the method for the pixel streaming of the illusion engine, which is provided by the embodiment of the invention, the method is realized on a server which is in the same intranet with the real-time communication server.
According to the embodiment of the invention, the method for streaming the illusion engine pixels further comprises the following steps: in response to detecting that a browser running on the client is stationary, a push frame rate at which the frame data of the first render target is pushed to the real-time communication server is reduced, and a number of render samples is increased.
The embodiment of the invention also provides a fantasy engine pixel streaming device, which comprises: an acquisition module for: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; a readback module for: reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing; streaming module for: pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
The embodiment of the invention also provides electronic equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the steps of the illusion engine pixel streaming method according to any one of the above when executing the program.
The embodiments of the present invention also provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the ghost engine pixel streaming method as described in any of the above.
Embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements the steps of the ghost engine pixel streaming method as described in any one of the above.
According to the pixel streaming method and device of the illusion engine, the frame data which are completely rendered are obtained in real time, the frame data are stored in the first rendering target, the frame data of the first rendering target are read back, so that the frame data are converted into explicit data which can be processed by the central processing unit, the frame data of the first rendering target are pushed to the real-time communication server, the real-time communication server encodes the frame data of the first rendering target to obtain the pixel stream in the target format, and the pixel stream is pushed to the client to decode and display the frame data, so that encoding and pixel streaming by the real-time communication server are realized, the delay of pixel streaming is greatly reduced, and multiple encoding formats are supported.
Drawings
In order to more clearly illustrate the technical solutions of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of a pixel streaming method of a illusion engine according to an embodiment of the present invention;
FIG. 2 is a second flow chart of a pixel streaming method of the illusion engine according to the embodiment of the present invention;
fig. 3 is a schematic structural diagram of a fantasy engine pixel streaming device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flow chart of a pixel streaming method of a fantasy engine according to an embodiment of the present invention. As shown in fig. 1, the method includes:
s1, acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is located in the central processing unit, and the first rendering target supports the graphic processor to write at a speed greater than a preset speed.
Frame data, such as image data, which is rendered by the illusion engine and can be displayed on a browser of a client, is acquired in real time. The rendered frame data is stored in the GPU, and the embodiment of the invention is encoded by using a Real-time communication (RTC, real-Time Communication) server, and communication is required to be established between the CPU and the Real-time communication server. Accordingly, the rendered frame data is stored to a first rendering target, which is located at the central processing unit CPU, so that the CPU can read back to the first rendering target to establish communication with the real-time communication server. And, the first render target supports graphics processor writing at greater than a preset speed. Namely, the storage area where the first rendering target is located supports CPU readable and GPU fast writing. Wherein the rendering target may be denoted as RT (Render Target), which is a buffer other than a back buffer and a screen frame buffer, for recording the result of the output after rendering.
The first rendering target which can be read back by the CPU and is written in by the GPU quickly is positioned in a special storage area of the central processing unit, and the storage area can be searched by a display card driver through the capability provided by the bottom graphic API. The preset speed is related to the writing speed supported by the hardware.
And step S2, reading back the frame data of the first rendering target, and converting the frame data into explicit data which can be processed by the central processing unit.
The first rendering target is located at the CPU, but the frame data of the first rendering target is implicit (non-explicit) data, and the frame data CPU in the first rendering target cannot be directly utilized. Therefore, in order for the CPU to be able to operate on the frame data of the first rendering target, it is necessary to read back the frame data of the first rendering target, convert the frame data into explicit data that the central processing unit can process, so that the CPU can process the read back frame data.
And S3, pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
If the data format of the read-back frame data is a format readable by the real-time communication server, the frame data of the first rendering target can be directly pushed to the real-time communication server; if the data format of the frame data read back is not the format readable by the real-time communication server, the frame data of the first rendering target needs to be pushed to the real-time communication server after the format of the frame data of the first rendering target is changed into the format readable by the real-time communication server.
The frame data of the first rendering target is pushed to the real-time communication server by using the SDK (Software Development Kit ) provided by the real-time communication server, the frame data encoded by the first rendering target is encoded into a pixel stream in a format required by the client by using the encoding capability (usually hard encoding) of the real-time communication server, and the pixel stream is pushed to the client for decoding, so that the rendered frame data is displayed on the client.
The real-time communication server can realize hard coding of various formats, and can code frame data of the first rendering target into a pixel stream of a target format according to specific requirements of the client.
According to the pixel streaming method of the illusion engine, the frame data after rendering is obtained in real time, the frame data is stored in the first rendering target, the frame data of the first rendering target is read back, so that the frame data are converted into explicit data which can be processed by the central processing unit, the frame data of the first rendering target are pushed to the real-time communication server, the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and the pixel stream is pushed to the client to decode and display the frame data, so that encoding and pixel streaming by the real-time communication server are realized, the delay of pixel streaming is greatly reduced, and multiple encoding formats are supported.
According to the method for streaming the pixel of the illusion engine provided by the embodiment of the invention, the frame data after rendering is obtained in real time, and the frame data is stored to a first rendering target, and the method comprises the following steps: executing the real-time acquisition of frame data after rendering through a rendering thread, and storing the frame data to a first rendering target; the reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor, includes: executing the frame data of the first rendering target through a read-back and push thread, so as to convert the frame data into explicit data which can be processed by the central processing unit; the pushing the frame data of the first rendering target to a real-time communication server includes: and executing the frame data pushing the first rendering target to a real-time communication server through the readback and pushing threads.
The rendering thread, and the readback and push threads may be executed asynchronously. Wherein the rendering thread is the rendering thread of the rendering engine. And acquiring frame data after rendering in real time through the rendering thread, and storing the frame data to a first rendering target. By setting the read-back and push threads to execute the frame data of the read-back first rendering target and pushing the frame data of the first rendering target to the real-time communication server, the read-back and push processes can not block the rendering, the frame rate of the rendering is ensured, and the processing speed is improved.
According to the pixel streaming method of the illusion engine, the frame data after rendering is acquired in real time by using the rendering thread, the frame data is stored in the first rendering target, and the frame data of the first rendering target is read back and pushed to the real-time communication server by the read back and pushing thread, so that the read back and pushing flow cannot block rendering, the frame rate of rendering is guaranteed, the processing speed is improved, and the delay of pixel streaming is further reduced.
According to the method for streaming the pixel of the illusion engine provided by the embodiment of the invention, the frame data after rendering is obtained in real time, and the frame data is stored to a first rendering target, and the method comprises the following steps: acquiring the frame data which is completely rendered from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data into a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server; copying the frame data in the second rendering target to the first rendering target located at the central processor.
To encode with the real-time communication server, it is necessary that the format of frame data of a rendering target is a format readable by the real-time communication server. The frame data rendered by the illusion engine does not support direct reading by a real-time communication server and needs format conversion. The format conversion can be performed before and after the readback theoretically, but it is verified through experiments that if the format conversion is performed after the readback, the format conversion needs to be performed by using a CPU, so that the speed is very slow, and the expected effect of reducing the pixel streaming delay cannot be achieved. Therefore, format conversion is advanced, and the image processor directly performs format conversion on the frame data to obtain the frame data in a format readable by the real-time communication server. Since frame data of a certain format needs to be stored to a rendering target of a corresponding format, the frame data is format-converted and stored to a second rendering target of the corresponding format. The frame data in the second render target is then copied to the CPU readable, GPU-written fast first render target of the central processing unit.
The rendered frame data can be obtained from a Back Buffer (Back Buffer) in real time through a callback function.
According to the pixel streaming method of the illusion engine, the frame data which is completely rendered is obtained from the rear buffer area in real time, the frame data is subjected to format conversion and stored in the second rendering target which is converted into the corresponding format, and the frame data in the second rendering target is copied to the first rendering target which is positioned in the central processing unit, so that the delay of pixel streaming is further reduced.
According to an embodiment of the present invention, a method for streaming a pixel of a illusion engine, which converts a format of frame data and stores the frame data in a second rendering target in a corresponding format, includes: the frame data is subjected to format conversion by means of one-pass rendering and is stored to a second rendering target in a corresponding format.
The conversion of the format of the frame data into the readable format of the real-time communication server can be realized in various ways, and through a great deal of experiments, the conversion of the frame data into the readable format of the real-time communication server by means of rendering one pass is faster.
According to the pixel streaming method of the illusion engine, provided by the embodiment of the invention, the frame data is subjected to format conversion in a one-pass rendering mode and is stored in the second rendering target in the corresponding format, so that the delay of pixel streaming is further reduced.
According to the method for the pixel streaming of the illusion engine, which is provided by the embodiment of the invention, the method is realized on a server which is in the same intranet with the real-time communication server.
The illusion engine pixel streaming method is implemented on a server which is in the same intranet as the real-time communication server, and is implemented on a cloud server which is in the same intranet as the real-time communication server. Thus, the frame data (such as original video data) of the first rendering target can be pushed to the real-time communication server at a high speed because of the intranet, the frame data of the first rendering target arrives at the real-time communication server and can be encoded into a pixel stream in a format required by the client in a hard-coding mode, and then the pixel stream is pushed to the client, so that the relatively low delay is ensured.
According to the pixel streaming method of the illusion engine, which is provided by the embodiment of the invention, the pixel streaming method of the illusion engine is realized on the server which is in the same intranet with the real-time communication server, and the pushing of the frame data of the first rendering target to the real-time communication server is realized through the intranet, so that the delay of the pixel streaming is further reduced.
According to the embodiment of the invention, the method for streaming the illusion engine pixels further comprises the following steps: in response to detecting that a browser running on the client is stationary, a push frame rate at which the frame data of the first render target is pushed to the real-time communication server is reduced, and a number of render samples is increased.
Optimization in terms of relevant traffic and rendering effects can be performed while the browser of the client is stationary. For example, when it is detected that the browser of the client is stationary (e.g., the screen does not change), the frame data of the first rendering target is pushed to the real-time communication server at a frequency of pushing only 1 frame per second, so that the pushed data amount is reduced, and the flow of acquiring data by the client is reduced. Because the frame rate requirement is low when the device is stationary, a large space is provided for enhancing the rendering effect, and at the moment, the image quality can be obviously improved by improving reflection, the sampling number of projection and the like, the noise of a picture is reduced, and the rendering effect is enhanced.
According to the pixel streaming method of the illusion engine, provided by the embodiment of the invention, the pushing frame rate of pushing the frame data of the first rendering target to the real-time communication server is reduced and the rendering sampling number is increased by responding to the detection that the browser running on the client is static, so that the rendering effect is enhanced.
Fig. 2 is a second flow chart of a pixel streaming method of the illusion engine according to the embodiment of the invention. As shown in fig. 2, in order to reduce the delay of video streaming as much as possible and support more coding formats to support more terminal views, the embodiment of the invention makes full use of the existing server communication cloud environment, applies the SDK provided by the communication Real-time audio and video service (Tencent Real-Time Communication, TRTC), proposes a phantom engine pixel streaming method, asynchronously reads each frame of video memory data (frame data) and submits the video memory data (frame data) to the TRTC during Real-time rendering, performs coding by using the powerful coding function of the TRTC server, then pushes the coded video memory data to a client for decoding, and optimizes the relevant flow and rendering effect when the browser is stationary. The illusion engine pixel streaming method provided by the embodiment of the invention is based on the illusion engine UE4, and each software functional module can be integrated into a TRTC-based UE4 pixel streaming plug-in. The pixel streaming method of the illusion engine provided by the embodiment of the invention specifically comprises the following steps:
in the UE4 rendering flow, reading frame data of a Back Buffer by a callback mode, rendering one time, changing the frame data format into an R8G8B8A8 format which can be read by TRTC, and storing the frame data in a second rendering target. Wherein the R8G8B8A8 format comprises 32-bit pixels, wherein 8 bits are given red, 8 bits are given green, 8 bits are given blue, and 8 bits are given to the ALPHA channel. After obtaining frame data in R8G8B8A8 format, copying each frame of RT data to a first rendering target which can be read back by a CPU and is written into by a GPU.
And reading back the frame data of the first rendering target to the CPU memory, so that the frame data of the first rendering target is converted into explicit data which can be processed by the CPU, and pushing the read back frame data to the TRTC server. The pixel streaming method operates in a communication cloud environment and is located in the same intranet with the TRTC server, so that the TRTC is used for greatly improving the streaming speed in the communication cloud environment.
If the browser of the client is detected to be static, pushing one frame of data to the TRTC server every second, increasing the rendering sampling number and enhancing the rendering effect.
The TRTC server encodes the frame data of the first rendering target into a pixel stream in a hard coding mode, and pushes the pixel stream to the client so that the client can decode and display the rendered frame data.
The pixel streaming plug-in can mutually communicate messages through WebSocket (full duplex communication protocol based on TCP) and the client, wherein the control server is an intermediate of message transmission.
The external interface of the plug-in is used for interfacing with the underlying logic code of the UE4, such as what kind of processing is performed by the server (the server operated by the pixel streaming method) after processing the client message, what kind of message is sent by the server to the client, etc.
The preferred embodiments of the present embodiment may be freely combined on the premise that the logic or structure does not conflict with each other, and the present invention is not limited to this.
The following describes the ghost engine pixel streaming device provided by the embodiment of the present invention, and the ghost engine pixel streaming device described below and the ghost engine pixel streaming method described above can be referred to correspondingly.
Fig. 3 is a schematic structural diagram of a fantasy engine pixel streaming device according to an embodiment of the present invention. As shown in fig. 3, the apparatus includes an acquisition module 10, a readback module 20, and a streaming module 30, where: the acquisition module 10 is configured to: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; the readback module 20 is for: reading back the frame data of the first rendering target, and converting the frame data into explicit data which can be processed by the central processing unit; the streaming module 30 is for: pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
According to the virtual engine pixel streaming device provided by the embodiment of the invention, the frame data after rendering is obtained in real time, the frame data is stored in the first rendering target and is read back to the frame data of the first rendering target, so that the frame data is converted into the explicit data which can be processed by the central processing unit, the frame data of the first rendering target is pushed to the real-time communication server, the real-time communication server encodes the frame data of the first rendering target to obtain the pixel stream in the target format, and the pixel stream is pushed to the client to decode and display the frame data, so that the encoding and pixel streaming by the real-time communication server are realized, the delay of pixel streaming is greatly reduced, and a plurality of encoding formats are supported.
According to the illusion engine pixel streaming device provided by the embodiment of the present invention, when the acquisition module 10 is configured to acquire frame data after rendering in real time, and store the frame data to a first rendering target, the acquisition module is specifically configured to: executing the real-time acquisition of frame data after rendering through a rendering thread, and storing the frame data to a first rendering target; the readback module 20 is specifically configured to, when configured to read back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processing unit: executing the frame data of the first rendering target through a read-back and push thread, so as to convert the frame data into explicit data which can be processed by the central processing unit; the streaming module 30, when configured to push the frame data of the first rendering target to a real-time communication server, is specifically configured to: and executing the frame data pushing the first rendering target to a real-time communication server through the readback and pushing threads.
According to the virtual engine pixel streaming device provided by the embodiment of the invention, the frame data after rendering is acquired in real time by using the rendering thread, the frame data is stored in the first rendering target, and the frame data of the first rendering target is read back and pushed to the real-time communication server by using the read back and pushing thread, so that the read back and pushing flow cannot block rendering, the frame rate of rendering is ensured, the processing speed is increased, and the delay of pixel streaming is further reduced.
According to the illusion engine pixel streaming device provided by the embodiment of the present invention, when the acquisition module 10 is configured to acquire frame data after rendering in real time, and store the frame data to a first rendering target, the acquisition module is specifically configured to: acquiring the frame data which is completely rendered from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data into a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server; copying the frame data in the second rendering target to the first rendering target located at the central processor.
According to the virtual engine pixel streaming device provided by the embodiment of the invention, the frame data after rendering is obtained from the rear buffer area in real time, the frame data is subjected to format conversion and stored in the second rendering target converted into the corresponding format, and the frame data in the second rendering target is copied to the first rendering target positioned in the central processing unit, so that the delay of pixel streaming is further reduced.
According to the illusion engine pixel streaming device provided by the embodiment of the present invention, when the acquisition module 10 is configured to perform format conversion on the frame data and store the frame data in the second rendering target in the corresponding format, the acquisition module is specifically configured to: the frame data is subjected to format conversion by means of one-pass rendering and is stored to a second rendering target in a corresponding format.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the frame data is subjected to format conversion in a one-pass rendering mode and is stored in the second rendering target in the corresponding format, so that the delay of pixel streaming is further reduced.
According to the embodiment of the invention, the illusion engine pixel streaming device is realized on a server which is in the same intranet with the real-time communication server.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the illusion engine pixel streaming device is realized on the server which is in the same intranet with the real-time communication server, and the frame data of the first rendering target is pushed to the real-time communication server through the intranet, so that the delay of pixel streaming is further reduced.
According to an embodiment of the present invention, there is provided a ghost engine pixel streaming apparatus, the apparatus further including a rendering enhancement module configured to: in response to detecting that a browser running on the client is stationary, a push frame rate at which the frame data of the first render target is pushed to the real-time communication server is reduced, and a number of render samples is increased.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the pushing frame rate of pushing the frame data of the first rendering target to the real-time communication server is reduced by responding to the detection that the browser running on the client is static, the rendering sampling number is increased, and the rendering effect is enhanced.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 4, the electronic device may include: processor 410, communication interface (Communications Interface) 420, memory 430 and communication bus 440, wherein processor 410, communication interface 420 and memory 430 communicate with each other via communication bus 440. The processor 410 may invoke logic instructions in the memory 430 to perform the illusion engine pixel streaming method, which includes: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing; pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
Further, the logic instructions in the memory 430 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, embodiments of the present invention further provide a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, is capable of executing the illusive engine pixel streaming method provided by the methods described above, the method comprising: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing; pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
In yet another aspect, embodiments of the present invention also provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the illusion engine pixel streaming method provided by the methods above, the method comprising: acquiring frame data after rendering in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write in at a speed greater than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing; pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (9)
1. A illusive engine pixel streaming method, comprising:
acquiring frame data after rendering from a graphic processor in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write at a speed greater than a preset speed, wherein the preset speed is related to a writing speed supported by hardware;
reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing;
pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
2. The illusion engine pixel streaming method of claim 1, wherein the acquiring the rendered frame data in real time, storing the frame data to a first rendering target, comprises: executing the real-time acquisition of frame data after rendering through a rendering thread, and storing the frame data to a first rendering target;
the reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor, includes: executing the frame data of the first rendering target through a read-back and push thread, so as to convert the frame data into explicit data which can be processed by the central processing unit;
the pushing the frame data of the first rendering target to a real-time communication server includes: and executing the frame data pushing the first rendering target to a real-time communication server through the readback and pushing threads.
3. The illusion engine pixel streaming method of claim 1, wherein the acquiring the rendered frame data in real time, storing the frame data to a first rendering target, comprises:
acquiring the frame data which is completely rendered from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data into a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server;
copying the frame data in the second rendering target to the first rendering target located at the central processor.
4. The illusion engine pixel streaming method of claim 3, wherein the converting the frame data format and storing to a second rendering target of a corresponding format comprises:
the frame data is subjected to format conversion by means of one-pass rendering and is stored to a second rendering target in a corresponding format.
5. The illusion engine pixel streaming method of claim 1, wherein the method is implemented on a server that is in the same intranet as the real-time communication server.
6. The illusion engine pixel streaming method of claim 1, wherein the method further comprises:
in response to detecting that a browser running on the client is stationary, a push frame rate at which the frame data of the first render target is pushed to the real-time communication server is reduced, and a number of render samples is increased.
7. A fantasy engine pixel streaming apparatus, comprising:
an acquisition module for: acquiring frame data after rendering from a graphic processor in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit, and the first rendering target supports the graphic processor to write at a speed greater than a preset speed, wherein the preset speed is related to a writing speed supported by hardware;
a readback module for: reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that the central processor is capable of processing;
streaming module for: pushing the frame data of the first rendering target to a real-time communication server, so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client to decode and display the frame data.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps of the illusion engine pixel streaming method according to any one of claims 1 to 6 when the program is executed.
9. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the illusion engine pixel streaming method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210773055.0A CN115225615B (en) | 2022-06-30 | 2022-06-30 | Illusion engine pixel streaming method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210773055.0A CN115225615B (en) | 2022-06-30 | 2022-06-30 | Illusion engine pixel streaming method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115225615A CN115225615A (en) | 2022-10-21 |
| CN115225615B true CN115225615B (en) | 2024-02-23 |
Family
ID=83610540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210773055.0A Active CN115225615B (en) | 2022-06-30 | 2022-06-30 | Illusion engine pixel streaming method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115225615B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116896684A (en) * | 2023-08-02 | 2023-10-17 | 广州颖上信息科技有限公司 | Virtual control system and method for stabilizer |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2011265373A1 (en) * | 2011-12-20 | 2013-07-04 | Canon Kabushiki Kaisha | Colour gradients from smooth-profiile and sharp-profile paths |
| CN109510990A (en) * | 2018-10-25 | 2019-03-22 | 深圳市腾讯信息技术有限公司 | Image processing method and device, computer readable storage medium, electronic equipment |
| CN112929682A (en) * | 2021-01-21 | 2021-06-08 | 广州虎牙科技有限公司 | Method, device and system for transparently processing image background and electronic equipment |
| CN113938705A (en) * | 2020-07-13 | 2022-01-14 | 阿里巴巴集团控股有限公司 | Method, device, server, terminal equipment and system for video coding and decoding |
| CN114006894A (en) * | 2020-12-30 | 2022-02-01 | 万翼科技有限公司 | Data processing system, method, electronic device, and computer storage medium |
| CN114040246A (en) * | 2021-11-08 | 2022-02-11 | 网易（杭州）网络有限公司 | Image format conversion method, device, equipment and storage medium of graphic processor |
| CN114079823A (en) * | 2021-11-17 | 2022-02-22 | 天翼数字生活科技有限公司 | Video rendering method, device, equipment and medium based on Flutter |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7586492B2 (en) * | 2004-12-20 | 2009-09-08 | Nvidia Corporation | Real-time display post-processing using programmable hardware |
-
2022
- 2022-06-30 CN CN202210773055.0A patent/CN115225615B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2011265373A1 (en) * | 2011-12-20 | 2013-07-04 | Canon Kabushiki Kaisha | Colour gradients from smooth-profiile and sharp-profile paths |
| CN109510990A (en) * | 2018-10-25 | 2019-03-22 | 深圳市腾讯信息技术有限公司 | Image processing method and device, computer readable storage medium, electronic equipment |
| CN113938705A (en) * | 2020-07-13 | 2022-01-14 | 阿里巴巴集团控股有限公司 | Method, device, server, terminal equipment and system for video coding and decoding |
| CN114006894A (en) * | 2020-12-30 | 2022-02-01 | 万翼科技有限公司 | Data processing system, method, electronic device, and computer storage medium |
| CN112929682A (en) * | 2021-01-21 | 2021-06-08 | 广州虎牙科技有限公司 | Method, device and system for transparently processing image background and electronic equipment |
| CN114040246A (en) * | 2021-11-08 | 2022-02-11 | 网易（杭州）网络有限公司 | Image format conversion method, device, equipment and storage medium of graphic processor |
| CN114079823A (en) * | 2021-11-17 | 2022-02-22 | 天翼数字生活科技有限公司 | Video rendering method, device, equipment and medium based on Flutter |
Non-Patent Citations (1)
| Title |
|---|
| 基于大数据技术的云端城市地质三维可视化框架;宋越;高振记;王鹏;;中国矿业(第06期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115225615A (en) | 2022-10-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11706484B2 (en) | Video processing method, electronic device and computer-readable medium | |
| US10594901B2 (en) | Game engine application direct to video encoder rendering | |
| US20210281718A1 (en) | Video Processing Method, Electronic Device and Storage Medium | |
| CN112843676B (en) | Data processing method, device, terminal, server and storage medium | |
| EP3410302B1 (en) | Graphic instruction data processing method, apparatus | |
| CN110290398B (en) | Video issuing method and device, storage medium and electronic equipment | |
| CN115225615B (en) | Illusion engine pixel streaming method and device | |
| CN111885346A (en) | Picture code stream synthesis method, terminal, electronic device and storage medium | |
| CN111343503A (en) | Video transcoding method and device, electronic equipment and storage medium | |
| CN111294600A (en) | Compression method, decompression method and device for video sequence frame | |
| US11562772B2 (en) | Video processing method, electronic device, and storage medium | |
| CN113973224A (en) | Method for transmitting media information, computing device and storage medium | |
| CN114302176A (en) | Video playing method and device | |
| US20240040147A1 (en) | Data processing method and apparatus, computer device, and storage medium | |
| CN115914745A (en) | Video decoding method and device, electronic equipment and computer readable medium | |
| CN116668741A (en) | Cloud desktop display method, image display method, device and storage medium | |
| CN114938408B (en) | Data transmission method, system, equipment and medium of cloud mobile phone | |
| CN114071190B (en) | Cloud application video stream processing method, related device and computer program product | |
| WO2021223577A1 (en) | Video processing method, related apparatus, storage medium, and program product | |
| CN116546262A (en) | Data processing method, device, equipment and storage medium | |
| US20190349414A1 (en) | Client side data stream processing | |
| CN115118922B (en) | Method and device for inserting motion picture in real-time video screen combination in cloud conference | |
| CN113052748B (en) | Graphics processor and video decoding display method | |
| CN111435995B (en) | Method, device and system for generating dynamic picture | |
| CN115426519A (en) | Method and system for playing H265 video in browser |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |