CN115225615A - Illusion engine pixel streaming method and device - Google Patents
Illusion engine pixel streaming method and device Download PDFInfo
- Publication number
- CN115225615A CN115225615A CN202210773055.0A CN202210773055A CN115225615A CN 115225615 A CN115225615 A CN 115225615A CN 202210773055 A CN202210773055 A CN 202210773055A CN 115225615 A CN115225615 A CN 115225615A
- Authority
- CN
- China
- Prior art keywords
- frame data
- real
- rendering
- rendering target
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000009877 rendering Methods 0.000 claims abstract description 207
- 238000004891 communication Methods 0.000 claims abstract description 93
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000006243 chemical reaction Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 19
- 230000003068 static effect Effects 0.000 claims description 9
- 230000001965 increasing effect Effects 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 4
- 230000000694 effects Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/262—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists
- H04N21/26208—Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists the scheduling operation being performed under constraints
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
- H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the invention provides a pixel streaming method and a pixel streaming device for a ghost engine, wherein the method comprises the following steps: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in the central processing unit and supports the graphic processor to write in at a speed higher than a preset speed; reading back frame data of the first rendering target, thereby converting the frame data into explicit data which can be processed by a central processor; and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream of a target format, and pushing the pixel stream to a client for decoding and displaying the frame data. The pixel streaming method and device of the illusion engine provided by the embodiment of the invention realize the coding and pixel streaming by using the real-time communication server, greatly reduce the delay of the pixel streaming and support various coding formats.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a pixel streaming method and device for an illusion engine.
Background
The illusion Engine (UE) Pixel Streaming (Pixel Streaming) allows the illusion items to be played from any networked device with a screen (e.g., computer, cell phone, tablet, etc.). The advantage of using a pixel stream is that the output device does not need to handle all scenes, physical simulations and interactions, only one screen and internet connection.
The existing illusion engine pixel streaming plug-in is realized based on WebRTC (Web Real-Time Communication, web page instant messaging) service, the illusion engine needs to be sent to the WebRTC for pixel streaming after local coding, the coding format only supports the format of H264, and a larger Time delay exists in the transmission of video stream with a client.
Disclosure of Invention
In view of the defects in the prior art, embodiments of the present invention provide a method and an apparatus for phantom engine pixel streaming.
The embodiment of the invention provides a pixel streaming method of an illusion engine, which comprises the following steps: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit and supports the graphic processor to write at a speed higher than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor; and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
According to an embodiment of the present invention, a method for pixel streaming of an illusion engine, where the method obtains rendered frame data in real time, and stores the frame data in a first rendering target, includes: executing the real-time acquired rendered frame data through a rendering thread, and storing the frame data to a first rendering target; the reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor, including: executing the frame data of the first rendering target through a read back and push thread, thereby converting the frame data into explicit data which can be processed by the central processor; the pushing the frame data of the first rendering target to a real-time communication server includes: and executing the pushing of the frame data of the first rendering target to a real-time communication server through the read back and push thread.
According to an embodiment of the present invention, a method for pixel streaming of an illusion engine, where the method obtains rendered frame data in real time, and stores the frame data in a first rendering target, includes: acquiring rendered frame data from a rear buffer area in real time, carrying out format conversion on the frame data and storing the frame data to a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server; copying the frame data in the second render target to the first render target at the central processor.
According to an embodiment of the present invention, a method for pixel streaming of a ghost engine, where the format conversion of the frame data and the storage of the frame data to a second rendering target in a corresponding format includes: and converting the format of the frame data in a rendering one-pass mode and storing the frame data to a second rendering target in a corresponding format.
According to the pixel streaming method of the illusion engine provided by the embodiment of the invention, the method is realized on a server which is positioned in the same intranet with the real-time communication server.
According to an embodiment of the present invention, a ghost engine pixel streaming method is provided, the method further includes: and in response to the detection that the browser running at the client is static, reducing a pushing frame rate for pushing the frame data of the first rendering target to the real-time communication server, and increasing a rendering sampling number.
The embodiment of the present invention further provides a ghost engine pixel streaming device, including: an acquisition module to: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit, and the first rendering target supports the graphic processor to write at a speed higher than a preset speed; a read back module to: reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor; a streaming module to: and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
An embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of any one of the above-mentioned ghost engine pixel streaming methods when executing the program.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the ghost engine pixel streaming method as described in any one of the above.
Embodiments of the present invention further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the ghost engine pixel streaming method according to any of the above embodiments are implemented.
The illusion engine pixel streaming method and the device provided by the embodiment of the invention have the advantages that the rendered frame data is obtained in real time, the frame data is stored in the first rendering target, the frame data of the first rendering target is read back, the frame data is converted into the explicit data which can be processed by the central processing unit, the frame data of the first rendering target is pushed to the real-time communication server, the real-time communication server encodes the frame data of the first rendering target to obtain the pixel stream of the target format, the pixel stream is pushed to the client to be decoded and the frame data is displayed, the encoding and pixel streaming by using the real-time communication server are realized, the delay of the pixel streaming is greatly reduced, and various encoding formats are supported.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a flow chart of a ghost engine pixel streaming method according to an embodiment of the present invention;
fig. 2 is a second flowchart of the ghost engine pixel streaming method according to the embodiment of the present invention;
fig. 3 is a schematic structural diagram of a ghost engine pixel streaming apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
Fig. 1 is a flowchart illustrating a ghost engine pixel streaming method according to an embodiment of the present invention. As shown in fig. 1, the method includes:
s1, acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is located in a central processing unit, and the first rendering target supports the graphic processor to write at a speed higher than a preset speed.
And acquiring frame data which is rendered by the illusion engine and can be displayed on a browser of the client, such as image data, in real time. The rendered frame data is stored in the GPU, and in the embodiment of the present invention, a Real-Time Communication (RTC) server is used for encoding, and Communication needs to be established with the Real-Time Communication server through the CPU. Accordingly, rendered frame data is stored to a first rendering target, which is located at the central processor CPU, so that the CPU can read back the first rendering target to establish communication with the real-time communication server. Also, the first render target supports the graphics processor to write at greater than a preset speed. Namely, the storage area where the first rendering target is located supports reading of a CPU and quick writing of a GPU. The Render Target may be represented as RT (Render Target), which is a back buffer and a buffer outside the screen frame buffer, and is used to record the output result after rendering.
The first rendering target of the CPU readable back and GPU writable fast is located in a special storage area of the CPU, and the storage area can be searched by the video card drive through the capacity provided by the bottom graphics API. The predetermined speed is related to the write speed supported by the hardware.
And S2, reading back the frame data of the first rendering target, and converting the frame data into explicit data which can be processed by the central processing unit.
Although the first rendering target is located in the CPU, the frame data of the first rendering target is implicit (non-explicit) data, and the frame data CPU in the first rendering target cannot be directly utilized. Therefore, in order to enable the CPU to operate the frame data of the first rendering target, it is necessary to read back the frame data of the first rendering target, convert the frame data into explicit data that the central processor can process, so that the CPU can process the read back frame data.
And S3, pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
If the data format of the read frame data is a format readable by the real-time communication server, the frame data of the first rendering target can be directly pushed to the real-time communication server; if the data format of the read frame data is not a format readable by the real-time communication server, the frame data of the first rendering target needs to be pushed to the real-time communication server after the format of the frame data of the first rendering target is changed into a format readable by the real-time communication server.
And pushing the read frame data of the first rendering target to a real-time communication server by using an SDK (Software Development Kit) provided by the real-time communication server, encoding the frame data encoded by the first rendering target into a pixel stream of a format required by the client by using the encoding capability (usually hard coding) of the real-time communication server, and pushing the pixel stream to the client for decoding, so as to display the rendered frame data at the client.
The real-time communication server can realize hard coding in various formats, and can code the frame data of the first rendering target into a pixel stream in a target format according to the specific requirements of the client.
The illusion engine pixel streaming method provided by the embodiment of the invention is characterized in that frame data after rendering is obtained in real time, the frame data is stored in a first rendering target, the frame data of the first rendering target is read back, the frame data is converted into explicit data which can be processed by a central processing unit, the frame data of the first rendering target is pushed to a real-time communication server, the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream of a target format, the pixel stream is pushed to a client to be decoded and the frame data is displayed, the encoding and pixel streaming by using the real-time communication server are realized, the delay of the pixel streaming is greatly reduced, and a plurality of encoding formats are supported.
According to an embodiment of the present invention, a method for pixel streaming of an illusion engine, where the method obtains rendered frame data in real time, and stores the frame data in a first rendering target, includes: executing the real-time acquired rendered frame data through a rendering thread, and storing the frame data to a first rendering target; the reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor, including: executing the frame data of the first rendering target through a read back and push thread, thereby converting the frame data into explicit data which can be processed by the central processor; the pushing the frame data of the first rendering target to a real-time communication server includes: and executing the pushing of the frame data of the first rendering target to a real-time communication server through the read back and push thread.
The rendering thread and the read back and push threads may be executed asynchronously. Wherein, the rendering thread is the rendering thread of the rendering engine. And executing real-time acquisition of rendered frame data through the rendering thread, and storing the frame data to a first rendering target. The read back and push thread is arranged to execute the read back of the frame data of the first rendering target and push the frame data of the first rendering target to the real-time communication server, so that the rendering cannot be blocked by the read back and push process, the rendering frame rate is guaranteed, and the processing speed is increased.
According to the pixel streaming method of the illusion engine, the rendering thread is used for executing the operation of acquiring the rendered frame data in real time, the frame data is stored in the first rendering target, and the reading back and pushing thread is used for executing the operation of reading back the frame data of the first rendering target and pushing the frame data of the first rendering target to the real-time communication server, so that the rendering cannot be blocked by the reading back and pushing process, the rendering frame rate is guaranteed, the processing speed is improved, and the delay of pixel streaming is further reduced.
According to an embodiment of the present invention, a method for pixel streaming of an illusion engine, where the method obtains rendered frame data in real time, and stores the frame data in a first rendering target, includes: acquiring rendered frame data from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data to a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server; copying the frame data in the second render target to the first render target at the central processor.
To perform encoding by the real-time communication server, it is necessary to make the format of the frame data of the rendering target a format readable by the real-time communication server. Frame data rendered by the illusion engine does not support direct reading of a real-time communication server, and format conversion is needed. The format conversion can be theoretically performed before and after the read back, but experiments prove that if the format conversion is performed after the read back, the format conversion needs to be performed by using a CPU, the speed is very slow, and the expected effect of reducing the pixel streaming delay cannot be achieved. Therefore, format conversion is carried out in advance, and the frame data is directly subjected to format conversion through the graphic processor to obtain the frame data in the format readable by the real-time communication server. Since frame data of a certain format needs to be stored to a rendering target of a corresponding format, the frame data is format-converted and stored to a second rendering target of the corresponding format. And then copying the frame data in the second rendering target to the first rendering target which can be read by a CPU and written by a GPU.
The rendered frame data can be obtained from a Back Buffer (Back Buffer) in real time through a callback function.
The pixel streaming method of the illusion engine provided by the embodiment of the invention obtains the rendered frame data from the back buffer area in real time, performs format conversion on the frame data, stores the frame data into the second rendering target in a corresponding format, copies the frame data in the second rendering target into the first rendering target in the central processing unit, and further reduces the delay of pixel streaming.
According to an embodiment of the present invention, a method for pixel streaming of a ghost engine, where the format conversion of the frame data and the storage of the frame data to a second rendering target in a corresponding format includes: and performing format conversion on the frame data in a rendering pass and storing the frame data to a second rendering target in a corresponding format.
The conversion of the format of the frame data into a format readable by the real-time communication server may be achieved in a variety of ways, and it has been found through a number of experiments that the conversion of the frame data into a format readable by the real-time communication server in one rendering pass is faster.
According to the pixel streaming method of the illusion engine provided by the embodiment of the invention, the frame data is subjected to format conversion in a rendering one-pass mode and is stored to the second rendering target of the corresponding format, so that the pixel streaming delay is further reduced.
According to the pixel streaming method of the illusion engine provided by the embodiment of the invention, the method is realized on a server which is positioned in the same intranet with the real-time communication server.
The illusion engine pixel streaming method is realized on a server in the same intranet with a real-time communication server, such as a cloud server in the same intranet with the real-time communication server. Thus, because of the intranet, the frame data (such as the original video data) of the first rendering target can be pushed to the real-time communication server at a fast speed, and the frame data of the first rendering target can be encoded into a pixel stream in a format required by the client in a hard-coding manner on the real-time communication server and then pushed to the client, so that the delay is ensured to be relatively low.
According to the pixel streaming method of the unreal engine provided by the embodiment of the invention, the pixel streaming method of the unreal engine is realized on the server which is positioned in the same intranet with the real-time communication server, and the frame data of the first rendering target is pushed to the real-time communication server through the intranet, so that the delay of pixel streaming is further reduced.
According to an embodiment of the present invention, a ghost engine pixel streaming method is provided, the method further includes: and in response to the detection that the browser running at the client is static, reducing a pushing frame rate for pushing the frame data of the first rendering target to the real-time communication server, and increasing a rendering sampling number.
The optimization of the related traffic and rendering effect can be performed when the browser of the client is static. For example, when it is detected that the browser of the client is still (e.g., the screen does not change), the frame data of the first rendering target is pushed to the real-time communication server at the frequency of only pushing 1 frame per second, which reduces the amount of pushed data, and thus reduces the flow of data acquired by the client. Because the frame rate requirement is low when the device is static, a large space is provided for enhancing the rendering effect, the image quality can be obviously improved by improving the reflection, the sampling number of projection and the like, the noise of the image is reduced, and the rendering effect is enhanced.
According to the pixel streaming method of the illusion engine, provided by the embodiment of the invention, the frame pushing rate of frame data of the first rendering target pushed to the real-time communication server is reduced by responding to the detection that the browser running at the client is static, the number of rendering samples is increased, and the rendering effect is enhanced.
Fig. 2 is a second flowchart of a ghost engine pixel streaming method according to an embodiment of the present invention. As shown in fig. 2, in order to reduce the transmission delay of a video stream as much as possible and support more encoding formats to support more terminal views, the embodiment of the present invention fully utilizes the existing server flight Communication cloud environment, applies an SDK provided by flight Real-Time audio and video service (TRTC), and provides a phantom engine pixel streaming method, which asynchronously reads each frame of display memory data (frame data) during Real-Time rendering and submits the frame data to the TRTC, encodes the frame data by using a powerful encoding function of a TRTC service end and then pushes the encoded frame data to a client for decoding, and optimizes related traffic and rendering effect when a browser is stationary. The ghost engine pixel streaming method provided by the embodiment of the invention is based on the ghost engine UE4, and each software functional module can be integrated into a TRTC-based UE4 pixel streaming plug-in. The illusion engine pixel streaming method provided by the embodiment of the invention specifically comprises the following steps:
in the rendering process of the UE4, frame data of the Back Buffer is read in a callback mode, rendering is performed for one time, the format of the frame data is changed into a format of R8G8B8A8 which can be read by TRTC, and the format of the frame data is stored in a second rendering target. Wherein the R8G8B8A8 format includes 32-bit pixels, where 8 bits are for red, 8 bits are for green, 8 bits are for blue, and 8 bits are for the ALPHA channel. After frame data in the R8G8B8A8 format is obtained, copying each frame of RT data to a first rendering target which can be read back by a CPU and written into a GPU quickly.
Reading the frame data of the first rendering target back to the CPU memory, converting the frame data of the first rendering target into explicit data which can be processed by the CPU, and pushing the read frame data to the TRTC server. The pixel streaming method runs in the Tengcong cloud environment and is located in the same intranet with the TRTC server, so that the TRTC is used for greatly improving the streaming speed in the Tengcong cloud environment.
And if the browser of the client is detected to be static, pushing one frame of data to the TRTC server every second, increasing the rendering sampling number and enhancing the rendering effect.
And the TRTC encodes the frame data of the first rendering target into a pixel stream in a hard coding mode, and pushes the pixel stream to the client so as to be decoded by the client and display the rendered frame data.
The pixel streaming plug-in can communicate messages with each other through WebSocket (full duplex communication protocol based on TCP) and the client, wherein the control server is a middleware for message transmission.
The plug-in external interface is used for processing the bottom logic code of the UE4, such as what kind of processing is done by a server (a server operated by the pixel streaming method) after processing a received client message, what message is sent by the server to the client, and the like.
It should be noted that, on the premise that logics or structures of a plurality of preferred embodiments provided in this embodiment do not conflict with each other, the preferred embodiments may be freely combined, and the present invention is not limited to this.
In the following, the ghost engine pixel streaming apparatus provided by the embodiment of the present invention is described, and the ghost engine pixel streaming apparatus described below and the ghost engine pixel streaming method described above can be referred to correspondingly.
Fig. 3 is a schematic structural diagram of a ghost engine pixel streaming apparatus according to an embodiment of the present invention. As shown in fig. 3, the apparatus includes an acquisition module 10, a readback module 20, and a streaming module 30, wherein: the acquisition module 10 is configured to: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit, and the first rendering target supports the graphic processor to write at a speed higher than a preset speed; the readback module 20 is for: reading back the frame data of the first rendering target, and converting the frame data into explicit data which can be processed by the central processor; streaming module 30 is used to: and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
The illusion engine pixel streaming device provided by the embodiment of the invention acquires the rendered frame data in real time, stores the frame data in the first rendering target, reads back the frame data of the first rendering target, converts the frame data into the explicit data which can be processed by the central processing unit, pushes the frame data of the first rendering target to the real-time communication server so that the real-time communication server encodes the frame data of the first rendering target to obtain the pixel stream of the target format, pushes the pixel stream to the client for decoding and displaying the frame data, realizes encoding and pixel streaming by using the real-time communication server, greatly reduces the delay of the pixel streaming, and supports various encoding formats.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the obtaining module 10 is specifically configured to, when obtaining frame data after rendering in real time and storing the frame data to a first rendering target: executing the real-time acquired rendered frame data through a rendering thread, and storing the frame data to a first rendering target; the read-back module 20, when configured to read back the frame data of the first rendering target, so as to convert the frame data into explicit data that can be processed by the central processing unit, is specifically configured to: executing the frame data of the first rendering target through a read back and push thread, thereby converting the frame data into explicit data which can be processed by the central processor; when the streaming module 30 is configured to push the frame data of the first rendering target to a real-time communication server, specifically configured to: and executing the pushing of the frame data of the first rendering target to a real-time communication server through the read back and push thread.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the rendering thread is utilized to execute the operation of acquiring the rendered frame data in real time, the frame data is stored in the first rendering target, and the reading and pushing thread is utilized to execute the operation of reading the frame data of the first rendering target and pushing the frame data of the first rendering target to the real-time communication server, so that the reading and pushing process does not block rendering, the rendering frame rate is ensured, the processing speed is increased, and the pixel streaming delay is further reduced.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the obtaining module 10 is specifically configured to, when obtaining frame data after rendering in real time and storing the frame data to a first rendering target: acquiring rendered frame data from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data to a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server; copying the frame data in the second render target to the first render target at the central processor.
The illusion engine pixel streaming device provided by the embodiment of the invention obtains the rendered frame data from the rear buffer area in real time, performs format conversion on the frame data, stores the frame data into the second rendering target in the corresponding format, copies the frame data in the second rendering target into the first rendering target in the central processing unit, and further reduces the delay of pixel streaming.
According to the illusion engine pixel streaming device provided in the embodiment of the present invention, when the obtaining module 10 is configured to perform format conversion on the frame data and store the frame data to the second rendering target in the corresponding format, the obtaining module is specifically configured to: and converting the format of the frame data in a rendering one-pass mode and storing the frame data to a second rendering target in a corresponding format.
The illusion engine pixel streaming device provided by the embodiment of the invention performs format conversion on the frame data in a rendering one-pass mode and stores the frame data to the second rendering target in a corresponding format, thereby further reducing the delay of pixel streaming.
According to the illusion engine pixel streaming device provided by the embodiment of the invention, the device is realized on a server which is in the same intranet with the real-time communication server.
The illusion engine pixel streaming device provided by the embodiment of the invention realizes that the illusion engine pixel streaming device is arranged on the server which is positioned in the same intranet with the real-time communication server, and the intranet pushes the frame data of the first rendering target to the real-time communication server, thereby further reducing the delay of pixel streaming.
According to an embodiment of the present invention, there is provided a ghost engine pixel streaming apparatus, further including a rendering enhancement module, configured to: and in response to the detection that the browser running at the client is static, reducing a pushing frame rate for pushing the frame data of the first rendering target to the real-time communication server, and increasing a rendering sampling number.
The illusion engine pixel streaming device provided by the embodiment of the invention reduces the frame pushing rate of pushing the frame data of the first rendering target to the real-time communication server by responding to the detection that the browser running at the client is static, improves the number of rendering samples and enhances the rendering effect.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 4, the electronic device may include: a processor (processor) 410, a communication Interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are in communication with each other via the communication bus 440. The processor 410 may invoke logic instructions in the memory 430 to perform a ghost engine pixel streaming method comprising: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit, and the first rendering target supports the graphic processor to write at a speed higher than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor; and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server encodes the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention or a part thereof which substantially contributes to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
In another aspect, an embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, a computer can execute the ghost engine pixel streaming method provided by the above methods, where the method includes: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit, and the first rendering target supports the graphic processor to write at a speed higher than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor; and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
In yet another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the ghost engine pixel streaming method provided by the above methods, the method including: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit and supports the graphic processor to write at a speed higher than a preset speed; reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor; and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A ghost engine pixel streaming method, comprising:
acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit, and the first rendering target supports the graphic processor to write at a speed higher than a preset speed;
reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor;
and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
2. The illusion-engine pixel streaming method of claim 1, wherein the obtaining rendered frame data in real-time, storing the frame data to a first rendering destination, comprises: executing the real-time acquired rendered frame data through a rendering thread, and storing the frame data to a first rendering target;
said reading back said frame data of said first rendering target, thereby converting said frame data into explicit data that can be processed by said central processor, comprising: executing the frame data of the first rendering target through a read back and push thread, thereby converting the frame data into explicit data which can be processed by the central processor;
the pushing the frame data of the first rendering target to a real-time communication server includes: and pushing the frame data of the first rendering target to a real-time communication server through the read back and push thread.
3. The illusion-engine pixel streaming method of claim 1, wherein the obtaining rendered frame data in real-time, storing the frame data to a first rendering destination, comprises:
acquiring rendered frame data from a rear buffer area in real time, performing format conversion on the frame data, and storing the frame data to a second rendering target in a corresponding format; wherein the corresponding format is an explicit format readable by the real-time communication server;
copying the frame data in the second render target to the first render target at the central processor.
4. The illusion-engine pixel streaming method of claim 3, wherein the format converting and storing the frame data to a second rendering destination in a corresponding format comprises:
and converting the format of the frame data in a rendering one-pass mode and storing the frame data to a second rendering target in a corresponding format.
5. The illusion-engine pixel streaming method of claim 1, wherein the method is implemented on a server on the same intranet as the real-time communication server.
6. The illusion-engine pixel streaming method of claim 1, further comprising:
and in response to the detection that the browser running on the client is static, reducing a pushing frame rate for pushing the frame data of the first rendering target to the real-time communication server, and increasing a rendering sampling number.
7. A ghost engine pixel streaming apparatus, comprising:
an acquisition module to: acquiring rendered frame data in real time, and storing the frame data to a first rendering target; the first rendering target is positioned in a central processing unit and supports the graphic processor to write at a speed higher than a preset speed;
a read back module to: reading back the frame data of the first rendering target, thereby converting the frame data into explicit data that can be processed by the central processor;
a streaming module to: and pushing the frame data of the first rendering target to a real-time communication server so that the real-time communication server can encode the frame data of the first rendering target to obtain a pixel stream in a target format, and pushing the pixel stream to a client for decoding and displaying the frame data.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the ghost engine pixel streaming method according to any of claims 1 to 6.
9. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the ghost engine pixel streaming method according to any of claims 1 to 6.
10. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, performs the steps of the ghost engine pixel streaming method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210773055.0A CN115225615B (en) | 2022-06-30 | 2022-06-30 | Illusion engine pixel streaming method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210773055.0A CN115225615B (en) | 2022-06-30 | 2022-06-30 | Illusion engine pixel streaming method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115225615A true CN115225615A (en) | 2022-10-21 |
| CN115225615B CN115225615B (en) | 2024-02-23 |
Family
ID=83610540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210773055.0A Active CN115225615B (en) | 2022-06-30 | 2022-06-30 | Illusion engine pixel streaming method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115225615B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116896684A (en) * | 2023-08-02 | 2023-10-17 | 广州颖上信息科技有限公司 | Virtual control system and method for stabilizer |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060132491A1 (en) * | 2004-12-20 | 2006-06-22 | Nvidia Corporation | Real-time display post-processing using programmable hardware |
| AU2011265373A1 (en) * | 2011-12-20 | 2013-07-04 | Canon Kabushiki Kaisha | Colour gradients from smooth-profiile and sharp-profile paths |
| CN109510990A (en) * | 2018-10-25 | 2019-03-22 | 深圳市腾讯信息技术有限公司 | Image processing method and device, computer readable storage medium, electronic equipment |
| CN112929682A (en) * | 2021-01-21 | 2021-06-08 | 广州虎牙科技有限公司 | Method, device and system for transparently processing image background and electronic equipment |
| CN113938705A (en) * | 2020-07-13 | 2022-01-14 | 阿里巴巴集团控股有限公司 | Method, device, server, terminal equipment and system for video coding and decoding |
| CN114006894A (en) * | 2020-12-30 | 2022-02-01 | 万翼科技有限公司 | Data processing system, method, electronic device, and computer storage medium |
| CN114040246A (en) * | 2021-11-08 | 2022-02-11 | 网易（杭州）网络有限公司 | Image format conversion method, device, equipment and storage medium of graphic processor |
| CN114079823A (en) * | 2021-11-17 | 2022-02-22 | 天翼数字生活科技有限公司 | Video rendering method, device, equipment and medium based on Flutter |
-
2022
- 2022-06-30 CN CN202210773055.0A patent/CN115225615B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060132491A1 (en) * | 2004-12-20 | 2006-06-22 | Nvidia Corporation | Real-time display post-processing using programmable hardware |
| AU2011265373A1 (en) * | 2011-12-20 | 2013-07-04 | Canon Kabushiki Kaisha | Colour gradients from smooth-profiile and sharp-profile paths |
| CN109510990A (en) * | 2018-10-25 | 2019-03-22 | 深圳市腾讯信息技术有限公司 | Image processing method and device, computer readable storage medium, electronic equipment |
| CN113938705A (en) * | 2020-07-13 | 2022-01-14 | 阿里巴巴集团控股有限公司 | Method, device, server, terminal equipment and system for video coding and decoding |
| CN114006894A (en) * | 2020-12-30 | 2022-02-01 | 万翼科技有限公司 | Data processing system, method, electronic device, and computer storage medium |
| CN112929682A (en) * | 2021-01-21 | 2021-06-08 | 广州虎牙科技有限公司 | Method, device and system for transparently processing image background and electronic equipment |
| CN114040246A (en) * | 2021-11-08 | 2022-02-11 | 网易（杭州）网络有限公司 | Image format conversion method, device, equipment and storage medium of graphic processor |
| CN114079823A (en) * | 2021-11-17 | 2022-02-22 | 天翼数字生活科技有限公司 | Video rendering method, device, equipment and medium based on Flutter |
Non-Patent Citations (1)
| Title |
|---|
| 宋越;高振记;王鹏;: "基于大数据技术的云端城市地质三维可视化框架", 中国矿业, no. 06 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116896684A (en) * | 2023-08-02 | 2023-10-17 | 广州颖上信息科技有限公司 | Virtual control system and method for stabilizer |
| CN116896684B (en) * | 2023-08-02 | 2024-05-17 | 广州颖上信息科技有限公司 | Virtual control system and method for stabilizer |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115225615B (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3886444A1 (en) | Video processing method and apparatus, and electronic device and computer-readable medium | |
| US20210281718A1 (en) | Video Processing Method, Electronic Device and Storage Medium | |
| SG173703A1 (en) | Method for generating gif, and system and media player thereof | |
| CN112843676B (en) | Data processing method, device, terminal, server and storage medium | |
| CN111294600A (en) | Compression method, decompression method and device for video sequence frame | |
| CN115225615B (en) | Illusion engine pixel streaming method and device | |
| CN113973224B (en) | Media information transmission method, computing device and storage medium | |
| CN110740352B (en) | SPICE protocol-based difference image display method in video card transparent transmission environment | |
| CN114071190B (en) | Cloud application video stream processing method, related device and computer program product | |
| EP3855754A1 (en) | Video processing method and apparatus, electronic device, and storage medium | |
| CN109587561B (en) | Video processing method and device, electronic equipment and storage medium | |
| CN111343503A (en) | Video transcoding method and device, electronic equipment and storage medium | |
| CN109587555B (en) | Video processing method and device, electronic equipment and storage medium | |
| CN113691531B (en) | WebSocket protocol-based data transmission method, system, equipment and storage medium | |
| CN114938408A (en) | Data transmission method, system, equipment and medium of cloud mobile phone | |
| CN115914745A (en) | Video decoding method and device, electronic equipment and computer readable medium | |
| CN111093091A (en) | Video processing method, server and system | |
| CN113052748B (en) | Graphics processor and video decoding display method | |
| WO2019217127A1 (en) | Client side data stream processing | |
| CN109803157A (en) | A kind of sequence frame picture transmission method, system and electronic equipment based on video | |
| KR20160082521A (en) | Chroma down-conversion and up-conversion processing | |
| CN114820660A (en) | Image processing method, image processing apparatus, electronic device, storage medium, and program product | |
| US10659826B2 (en) | Cloud streaming service system, image cloud streaming service method using application code, and device therefor | |
| CN113595885A (en) | Picture sending method and device, electronic equipment and storage medium | |
| CN111147928A (en) | Video processing method, server, terminal and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |