CN115761083A

CN115761083A - Rendering result processing method, device, equipment and medium

Info

Publication number: CN115761083A
Application number: CN202211350114.XA
Authority: CN
Inventors: 潘金龙; 熊小安; 袁振杰; 陈方正; 张德莹; 孙瑞鸿
Original assignee: Beijing Softong Intelligent Technology Co ltd
Current assignee: Beijing Softong Intelligent Technology Co ltd
Priority date: 2022-10-31
Filing date: 2022-10-31
Publication date: 2023-03-07

Abstract

The invention discloses a rendering result processing method, a rendering result processing device, rendering result processing equipment and a rendering result processing medium. The method is performed by a rendering service, the method comprising: obtaining a rendering result, and redirecting the rendering result to a rendering service cache region; performing image post-processing on the rendering result in a rendering service cache region, and encoding the rendering result output by the image post-processing to obtain a rendering video stream; and sending the rendered video stream to a user terminal. The technical scheme solves the problems of long transmission delay and poor display real-time performance of the rendering result, and can greatly reduce the delay while improving the transmission efficiency of the rendering result.

Description

Rendering result processing method, device, equipment and medium

Technical Field

The present invention relates to the field of cloud rendering technologies, and in particular, to a method, an apparatus, a device, and a medium for processing a rendering result.

Background

The cloud rendering is to place the 3D program in a remote server for rendering, the user terminal clicks a 'cloud rendering' button in the local 3D program through front-end software or directly and accesses to the access resource by means of high-speed internet, an instruction is sent out from the user terminal, the server executes a corresponding rendering task according to the instruction to obtain a rendering result, and the rendering result is obtained in a screen recording mode and is transmitted back to the user terminal for displaying.

However, in the prior art, a rendering result of real-time cloud rendering is generally obtained from a server in a screen recording manner and then transmitted to a user terminal. Therefore, the existing rendering result processing mode has extremely high requirements on network conditions, consumes a large amount of processing time of a processor, and is difficult to realize real-time presentation of rendering results on a user terminal.

Disclosure of Invention

The invention provides a rendering result processing method, a rendering result processing device, rendering result processing equipment and a rendering result processing medium, which are used for solving the problems of long transmission delay time and poor display real-time performance of rendering results, and can greatly reduce the delay time while improving the transmission efficiency of the rendering results.

According to an aspect of the present invention, there is provided a method of processing a rendering result, the method being performed by a rendering service, the method including:

obtaining a rendering result, and redirecting the rendering result to a rendering service cache region;

performing image post-processing on the rendering result in a rendering service cache region, and encoding the rendering result output by the image post-processing to obtain a rendering video stream;

and sending the rendered video stream to a user terminal.

According to another aspect of the present invention, there is provided an apparatus for processing a rendering result, the apparatus being configured to a rendering service, the apparatus comprising:

the redirection module is used for acquiring a rendering result and redirecting the rendering result to a rendering service cache region;

the video stream generation module is used for carrying out image post-processing on the rendering result in the rendering service cache region and coding the rendering result output by the image post-processing to obtain a rendering video stream;

and the video stream sending module is used for sending the rendered video stream to the user terminal.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method for processing the rendering result according to any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a rendering result processing method according to any one of the embodiments of the present invention when the computer instructions are executed.

According to the technical scheme of the embodiment of the invention, the rendering result is redirected to the rendering service cache region by acquiring the rendering result, and the rendering result is subjected to image post-processing in the rendering service cache region. And then coding a rendering result output by image post-processing to obtain a rendering video stream, and sending the rendering video stream to the user terminal. According to the scheme, the problems of long transmission delay time and poor display real-time performance of the rendering result are solved by redirecting the rendering result, and the time delay can be greatly reduced while the transmission efficiency of the rendering result is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a flowchart of a method for processing rendering results according to an embodiment of the present invention;

FIG. 2 is a flowchart of a method for processing rendering results according to a second embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a processing apparatus for rendering results according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device implementing a rendering result processing method according to an embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. According to the technical scheme, the data acquisition, storage, use, processing and the like meet the relevant regulations of national laws and regulations.

Example one

Fig. 1 is a flowchart of a method for processing a rendering result according to an embodiment of the present invention, where the method is applicable to real-time cloud rendering of a scene, and the method may be executed by a device for processing a rendering result, where the device may be implemented in a form of hardware and/or software, and the device may be configured in an electronic device. As shown in fig. 1, the method includes:

s110, obtaining a rendering result, and redirecting the rendering result to a rendering service cache region.

The method can be executed by a rendering service, and the rendering service can respond to a rendering request of the user terminal, schedule the rendering host to complete a rendering task and obtain a rendering result. Wherein the rendering result may be a three-dimensional rendering picture generated based on the three-dimensional scene data. In the prior art, after a rendering result is obtained, a rendering service generally needs to output a three-dimensional rendering picture to a screen for re-interception, and the three-dimensional rendering picture as image data needs to reenter a display memory for hardware coding, so that a large amount of consumed hardware resources are obtained. In the scheme, after the rendering result is obtained by the rendering service, the rendering result can be redirected to the rendering service cache region to directly perform subsequent processing, so that the cloud rendering efficiency is greatly improved. In a specific example, under the conditions of 1080p and 60Fps rendering, the cloud rendering delay of the redirection mode is less than one third of that of the screen recording mode in the state of good network transmission conditions.

And S120, carrying out image post-processing on the rendering result in the rendering service cache region, and encoding the rendering result output by the image post-processing to obtain a rendering video stream.

As can be easily understood, in the GPU phase of cloud rendering, the rendering service may perform image post-processing on the rendering result in the rendering service buffer to optimize the rendering result. Specifically, the image post-processing may include operations such as image scaling, image rotation, antialiasing, and color correction. After image post-processing, the rendering service may hardware encode the rendering results to output a rendered video stream. The hardware coding method may include coding methods such as H264, MPGE, and H265. The conventional encoding methods such as H264 and mpeg are advantageous for rendering video streams with a 720p and 1080p resolution of 30Fps, but for devices such as VR (Virtual Reality), AR (Augmented Reality), and large-screen, the conventional encoding methods are prone to have problems such as black edge, tearing, smearing, and color gamut narrowing. H265 encoding can achieve high resolution wide color gamut encoding under good network conditions, resulting in a high quality rendered video stream. The rendering service can detect network conditions, output devices and the like, and when the network meets preset conditions and the output devices are irregular devices such as VR, AR and large screens, H265 coding is adopted.

S130, sending the rendered video stream to a user terminal.

The rendering service may send the rendered video stream to the user terminal for application by the user terminal.

According to the technical scheme, the rendering result is redirected to the rendering service cache region through obtaining the rendering result, and the rendering result is subjected to image post-processing in the rendering service cache region. And then coding a rendering result output by image post-processing to obtain a rendering video stream, and sending the rendering video stream to the user terminal. According to the scheme, the problems of long transmission delay time and poor display real-time performance of the rendering result are solved by redirecting the rendering result, and the delay time can be greatly reduced while the transmission efficiency of the rendering result is improved.

Example two

Fig. 2 is a flowchart of a rendering result processing method according to a second embodiment of the present invention, which is detailed based on the second embodiment. As shown in fig. 2, the method includes:

s210, three-dimensional scene data are obtained, vertex coloring and viewport cutting operations are carried out on the three-dimensional scene data, and data to be mapped are obtained.

In this scheme, the three-dimensional scene data may include 3D programs such as UE \ Unity, CAD \3Dmax, and the like. The rendering service can perform operations such as vertex coloring and viewport clipping on the three-dimensional scene data after receiving a rendering instruction of the user terminal. Vertex shading may be the processing of vertex information in three-dimensional scene data, such as spatial transformations of vertices, vertex offsets, and the like. Viewport clipping may be to cull the parts outside the view frustum when the model intersects the view frustum boundary, generating new vertices and triangles. After vertex shading, viewport clipping, and the like are performed on the three-dimensional scene data, the rendering service may obtain data to be mapped.

S220, according to the virtual screen of the rendering service, performing screen mapping on the data to be mapped to obtain mapping data.

The rendering service may perform screen mapping on data to be mapped, and map vertices in Normalized Device Coordinates (NDC) to a screen to obtain mapping data. The rendering service may provide a virtual screen for the data to be mapped. The virtual screen provided by the rendering service can be adjusted at will in the geometric rendering stage to support display devices of different sizes.

And S230, fragment coloring is carried out on the mapping data to obtain a rendering result of the three-dimensional scene data.

The rendering service may map the data for fragment processing, such as map sampling, illumination calculation, and the like, to implement fragment coloring and output rendering results of the three-dimensional scene data.

S240, obtaining a rendering result, and redirecting the rendering result to a rendering service cache region.

And S250, carrying out image post-processing on the rendering result in the rendering service cache region, and encoding the rendering result output by the image post-processing to obtain a rendering video stream.

In this scheme, optionally, the encoding the rendering result output by the image post-processing to obtain a rendering video stream includes:

blocking each frame of image in a rendering result output by image post-processing to obtain block data;

determining an intra-frame prediction result and an inter-frame prediction result according to the block data;

determining information to be coded according to the intra-frame prediction result and the inter-frame prediction result; the information to be coded comprises transformation information, motion information and prediction information;

and entropy coding is carried out on the information to be coded to obtain a rendering video stream.

After the image post-processing, the rendering service may block each frame of image in the rendering result, for example, into 10 × 10 blocks of pixels. The size of the block determines the color reduction degree of the subsequent area to a certain extent, and a smaller block can obtain a better reduction degree and simultaneously reduce the compression rate. According to the block data, the rendering service may perform predictive coding to obtain an intra-frame prediction result and an inter-frame prediction result. The intra prediction result may include residual information, and the inter prediction result may include information such as motion estimation and motion compensation. The rendering service may transform the residual, quantize and scan the transform coefficients. The inter-frame prediction result and the intra-frame prediction result are quantized, and the rendering service can obtain transformation information, motion information and prediction information waiting coding information. Entropy coding is performed on information to be coded, and the rendering service may output a rendered video stream.

In a high-resolution large-screen application scene, the coding mode can enhance the strength of intra-frame prediction. Because the resolution is large at this time, the color flat area in the frame is relatively large, and the compression rate can be more effectively improved by intra-frame prediction. The rendering service can mark static areas in the three-dimensional scene data and provide the static areas to the encoder in a black-and-white image mode, so that the efficiency of interframe compression is greatly improved, and the interframe searching time is further reduced. In an environment with high requirements on color reduction degree, a smaller size can be selected for the rendering service during blocking, and a smaller step size is used in the encoding stage. For VR application scenarios with higher latency requirements, the rendering service may turn off inter-prediction to reduce latency.

And S260, sending the rendered video stream to a user terminal.

And S270, outputting the rendered video stream on a screen for the user terminal to preview.

The rendering service may output the rendered video stream on a screen and provide an equally scaled preview of the rendered picture. For example, a rendering service may be neatly arranged on a rendering host screen in a manner of 270p while running 5 1080p cloud rendering instances.

In one possible implementation, after sending the rendered video stream to a user terminal, the method further includes:

and if an operation instruction of the input equipment sent by the user terminal is received, driving scene change according to the rendered video stream and the operation instruction.

In a preferred scheme, the input device comprises a keyboard, a mouse, a touch screen, a gesture collecting device, a somatosensory collecting device, a virtual reality handle, a positioner and a gyroscope.

The user terminal can access one or more input devices and sends an operation instruction triggered by the input devices to the rendering service, and the rendering service can realize responses such as scene change, movement and the like based on the current rendering scene according to the operation instruction and the rendering video stream. The rendering service can formulate a set of standardized input interfaces to realize plug and play of various types of input devices, expand application scenes of the rendering service and bring good use experience for users.

In another possible approach, after obtaining the rendering result of the three-dimensional scene data, the method further includes:

and if output equipment access information sent by the user terminal is received, remapping the rendering result according to the output equipment access information so as to adapt to the display of the output equipment.

If the user terminal accesses output devices such as VR glasses, display screens, etc., output device access information may be sent to the rendering service. The output device access information may include information such as a type, a model, and a resolution of the device. If the attributes such as the resolution ratio of the output equipment are determined to be changed according to the access information of the output equipment, the rendering service can remap the rendering result so as to correct the rendering picture, realize self-adaptive display and provide good visual experience for users.

According to the technical scheme, the rendering result is obtained, the rendering result is redirected to the rendering service cache area, and the rendering result is subjected to image post-processing in the rendering service cache area. And then coding a rendering result output by image post-processing to obtain a rendering video stream, and sending the rendering video stream to the user terminal. According to the scheme, the problems of long transmission delay time and poor display real-time performance of the rendering result are solved by redirecting the rendering result, and the delay time can be greatly reduced while the transmission efficiency of the rendering result is improved.

EXAMPLE III

Fig. 3 is a schematic structural diagram of a rendering result processing apparatus according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes:

a redirection module 310, configured to obtain a rendering result, and redirect the rendering result to a rendering service cache region;

a video stream generating module 320, configured to perform image post-processing on the rendering result in the rendering service cache region, and encode the rendering result output by the image post-processing to obtain a rendering video stream;

a video stream sending module 330, configured to send the rendered video stream to a user terminal.

In this scheme, optionally, the apparatus further includes a rendering result output module, configured to:

acquiring three-dimensional scene data, and performing vertex coloring and viewport clipping operations on the three-dimensional scene data to obtain data to be mapped;

according to a virtual screen of rendering service, performing screen mapping on the data to be mapped to obtain mapping data;

and carrying out fragment coloring on the mapping data to obtain a rendering result of the three-dimensional scene data.

In one possible implementation, the apparatus further includes a screen output module configured to:

and outputting the rendered video stream on a screen for a user terminal to preview.

In a preferred embodiment, the video stream generating module 320 includes:

the blocking unit is used for blocking each frame of image in the rendering result output by the image post-processing to obtain blocking data;

a prediction unit for determining an intra prediction result and an inter prediction result according to the block data;

the transformation unit is used for determining information to be coded according to the intra-frame prediction result and the inter-frame prediction result; the information to be coded comprises transformation information, motion information and prediction information;

and the entropy coding unit is used for entropy coding the information to be coded to obtain a rendering video stream.

On the basis of the above scheme, the apparatus further includes a scene driving module, configured to:

Optionally, the rendering result output module is further configured to:

Preferably, the input device comprises a keyboard, a mouse, a touch screen, a gesture collecting device, a body sensing collecting device, a virtual reality handle, a positioner and a gyroscope.

The rendering result processing device provided by the embodiment of the invention can execute the rendering result processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

FIG. 4 shows a schematic block diagram of an electronic device 410 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 4, the electronic device 410 includes at least one processor 411, and a memory communicatively connected to the at least one processor 411, such as a Read Only Memory (ROM) 412, a Random Access Memory (RAM) 413, and the like, wherein the memory stores computer programs executable by the at least one processor, and the processor 411 may perform various appropriate actions and processes according to the computer programs stored in the Read Only Memory (ROM) 412 or the computer programs loaded from the storage unit 418 into the Random Access Memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other through a bus 414. An input/output (I/O) interface 415 is also connected to bus 414.

Various components in the electronic device 410 are connected to the I/O interface 415, including: an input unit 416 such as a keyboard, a mouse, or the like; an output unit 417 such as various types of displays, speakers, and the like; a storage unit 418, such as a magnetic disk, optical disk, or the like; and a communication unit 419 such as a network card, modem, wireless communication transceiver, or the like. The communication unit 419 allows the electronic device 410 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.

Processor 411 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 411 performs various methods and processes described above, such as a processing method of a rendering result.

In some embodiments, the processing of the rendering results may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of a computer program may be loaded onto and/or installed onto electronic device 410 via ROM 412 and/or communications unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the processing method of the rendering result described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured by any other suitable means (e.g., by means of firmware) to perform the processing method of the rendering result.

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above, reordering, adding or deleting steps, may be used. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired result of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of processing rendering results, the method being performed by a rendering service, the method comprising:

and sending the rendered video stream to a user terminal.

2. The method of claim 1, wherein prior to obtaining rendering results, the method further comprises:

3. The method of claim 1, wherein after obtaining the rendered video stream, the method further comprises:

4. The method according to claim 1, wherein said encoding a rendering result output by image post-processing to obtain a rendered video stream comprises:

5. The method of claim 1, wherein after sending the rendered video stream to a user terminal, the method further comprises:

6. The method of claim 2, wherein after obtaining the rendering results of the three-dimensional scene data, the method further comprises:

7. The method of claim 5, wherein the input device comprises a keyboard, a mouse, a touch screen, a gesture capture device, a somatosensory capture device, a virtual reality handle, a locator, and a gyroscope.

8. An apparatus for processing a rendering result, the apparatus being configured to a rendering service, the apparatus comprising:

9. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of processing rendering results of any of claims 1-7.

10. A computer-readable storage medium storing computer instructions for causing a processor to perform the method of processing the rendering result according to any one of claims 1 to 7 when the computer instructions are executed.