WO2023087827A1 - 渲染方法及装置 - Google Patents

渲染方法及装置 Download PDF

Info

Publication number
WO2023087827A1
WO2023087827A1 PCT/CN2022/114771 CN2022114771W WO2023087827A1 WO 2023087827 A1 WO2023087827 A1 WO 2023087827A1 CN 2022114771 W CN2022114771 W CN 2022114771W WO 2023087827 A1 WO2023087827 A1 WO 2023087827A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
processor
image
memory
image data
Prior art date
Application number
PCT/CN2022/114771
Other languages
English (en)
French (fr)
Other versions
WO2023087827A9 (zh
WO2023087827A8 (zh
Inventor
秦园
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Priority to EP22865904.1A priority Critical patent/EP4207047A4/en
Publication of WO2023087827A1 publication Critical patent/WO2023087827A1/zh
Publication of WO2023087827A9 publication Critical patent/WO2023087827A9/zh
Publication of WO2023087827A8 publication Critical patent/WO2023087827A8/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/445—Program loading or initiating
    • G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
    • G06F9/4451—User profiles; Roaming
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Definitions

  • the present application relates to the technical field of image processing, in particular to a rendering method and device.
  • the resolution of the image develops to a higher resolution, for example, the resolution of the image develops from 720P to 1080P, and then from 1080P to 2k, where P represents the total number of rows of pixels, such as 720P represents 720 rows of pixels ;k represents the total number of columns of pixels, such as 2k represents 2000 columns of pixels.
  • Electronic devices can take up too much computing power when rendering high-resolution images or ultra-high-resolution images, which will increase the power consumption of electronic devices, cause serious heat generation, and even cause problems with running freezes.
  • the rendering method and device provided in the present application solve the problems of increased power consumption, severe heating of the device, and stuck operation when electronic devices render images.
  • the present application provides a rendering method, the method is applied to an electronic device, the electronic device runs an application program, the electronic device includes a first processor and a second processor, and the method includes: the first processor receives The rendering command sent by the second processor is used to instruct the second processor to render the first image based on the first resolution; the first processor sends a rendering instruction to the second processor, and the rendering instruction is used to instruct the second processor to render the first image.
  • An image is rendered; the second processor generates image data of the first image at a second resolution based on the rendering instruction, and the second resolution is not greater than the first resolution; the second processor renders the first image at the second resolution
  • the image data under the rate is written into the first memory; the second processor reads the image data of the first image under the third resolution from the first memory, and the third resolution is greater than the second resolution; the second processor A first image is generated based on image data of the first image at a third resolution.
  • the second processor After the second processor writes the generated image data of the first image at the second resolution into the first memory, the second processor can read the image of the first image at the third resolution from the first memory Data, saving the second processor to generate the image data of the first image at the third resolution, saving the computing power of the second processor, shortening the time spent in the rendering process, thereby reducing power consumption and improving rendering fluency, Solve the problem of serious equipment heating and running freezes.
  • the third resolution is greater than the second resolution, indicating that the second processor can obtain image data with a relatively high resolution, and the second processor can draw the first image based on the image data with a relatively high resolution to improve the second image data. The image quality of an image.
  • the rendering instruction corresponds to the first frame buffer, and the number of drawing instructions executed by the first frame buffer is greater than the preset threshold; if the rendering method of the application is deferred rendering, the rendering instruction Corresponds to the framebuffers except the last framebuffer among all the framebuffers delivered by the application.
  • the rendering instructions sent by the first processor to the second processor target different frame buffers, which means that when the application programs adopt different rendering methods, the timing of the rendering instructions sent by the first processor is also different. , to realize the sending of rendering commands based on frame buffer control in different rendering modes.
  • the first frame buffer is the frame buffer that executes the largest number of drawing instructions among all frame buffers.
  • the method further includes: the first processor acquires the rendering mode of the application program from a configuration file of the application program.
  • the rendering instruction is used to instruct the second processor to render the first image based on the second resolution, where the second resolution is smaller than the first resolution.
  • the rendering instruction sent by the first processor to the second processor carries the second resolution of the first image, specifies the resolution of the image data generated by the second processor, and prevents the second processor from generating a resolution that is different from that required by the application Mismatched image data, improving the accuracy of the first image.
  • the second resolution is smaller than the first resolution, and the smaller the resolution, the smaller the amount of data, so by specifying the second resolution smaller than the first resolution, the amount of data processed by the second processor is reduced, and the power of the electronic device is reduced. consumption, which solves the serious problem of equipment heating.
  • the second resolution is smaller than the first resolution; the third resolution is the same as the first resolution, or the third resolution is greater than the first resolution.
  • the second resolution is smaller than the first resolution, and the third resolution is the same as the first resolution.
  • the third resolution is the same as the first resolution, which means that the second processor can read the image data corresponding to the first resolution, draw the first image with the first resolution, and ensure the accuracy of the drawn first image.
  • Image quality vs. the image quality quality required by the application. If the third resolution is greater than the first resolution, the second processor can read image data with a resolution greater than the first resolution, and the image quality of the drawn first image is better than the image quality required by the application program, improving the second processor.
  • the second resolution is equal to the first resolution.
  • the second resolution is equal to the first resolution, but the third resolution is greater than the second resolution, then the third resolution is also greater than the first resolution.
  • the second The processor may render the first image based on the third resolution, and the image quality of the drawn first image is better than the image quality required by the application program, thereby improving the image quality of the first image.
  • the electronic device further includes a third processor, and the image data of the first image at the third resolution is generated by the third processor.
  • the third processor After the second processor generates the image data of the first image at the second resolution, the third processor generates the image data of the first image at the third resolution, so as to actively invoke the computing power of the third processor, Meet the computing power requirements for high-resolution images and/or ultra-high-resolution images, reduce the power consumption of electronic equipment, and reduce the phenomenon of severe heating.
  • the third processor can share the calculation load of the second processor, shorten the time spent in the rendering process, thereby improving the smoothness of rendering and solving the problem of stuck operation.
  • the method further includes: the third processor reads the first image from the first internal memory and writes the image data in the second internal memory. image data under the resolution; the third processor generates image data of the first image under the third resolution based on the image data of the first image under the second resolution; the third processor generates the image data of the first image under the third resolution The image data at the resolution is written into the first memory.
  • the method further includes: the second processor writes the image data of the first image at the second resolution Write to the second memory, the second processor has the right to access the second memory and the first memory, the third processor has the right to access the first memory; the second processor sends a first notification to the first processor, the first The notification is used to indicate that the image data of the first image at the second resolution is successfully written into the second memory; in response to receiving the first notification, the first processor sends a second notification to the second processor, and the second notification is used to indicate The second processor writes the image data of the first image at the second resolution into the first memory, and the second notification carries the address pointer of the first memory; in response to receiving the second notification, the second processor from The image data of the first image under the second resolution is read from the second memory.
  • the first processor can monitor the image data reading and writing work of the second processor, so that after the second processor writes the image data of the first image at the second resolution into the second memory, the second processor is triggered in time to write The image data of the first image under the second resolution is written into the first memory, so as to improve efficiency.
  • the second notification sent by the first processor carries the address pointer of the first memory, and the second processor can write the image of the first image under the second resolution into the first memory based on the address pointer of the first memory. Image data, improve the accuracy of writing.
  • the third processor reads the image data of the first image at the second resolution from the first memory.
  • the method further includes: the first processor sends a third notification to the third processor, and the third notification is used to instruct the third processor to read the image of the first image at the second resolution from the first memory data.
  • the first processor can promptly notify the third processor to read it, and trigger the third processor to generate the first image at the second resolution. Image data at three resolutions to improve efficiency.
  • the second processor reads the image data of the first image at the third resolution from the first memory.
  • the method further includes: the first processor sends a fourth notification to the second processor, and the fourth notification is used to instruct the second processor to read the image of the first image at the third resolution from the first memory
  • the fourth notification carries the address pointer of the first memory.
  • the first processor can monitor the image data reading and writing work of the third processor, and after the third processor writes the image data of the first image at the third resolution into the first memory, it will trigger the second processor to read in time Image data of the first image at the third resolution, improving efficiency.
  • the fourth notification sent by the first processor carries the address pointer of the first memory, and the second processor can read the image of the first image at the third resolution from the correct address based on the address pointer of the first memory data to improve the accuracy of the read image data.
  • the method further includes: the second processor reads the image data of the first image at the third resolution To write into the second memory, the second processor has the authority to access the second memory and the first memory, and the third processor has the authority to access the first memory.
  • the electronic device further includes a third processor, and the image data of the first image at a third resolution is generated by the third processor; before the first processor sends a rendering instruction to the second processor, the method further includes : The first processor allocates the first memory from the third memory, and the first processor has the right to access the third memory; the first processor sends the pointer address of the first memory to the third processor, and the third processor based on The pointer address writes the image data of the first image at the third resolution into the first memory, and reads the image data of the first image at the second resolution from the first memory based on the pointer address.
  • the electronic device further includes a third processor, and the image data of the first image at a third resolution is generated by the third processor; before the first processor sends a rendering instruction to the second processor, the method further includes : the first processor allocates the first memory from the hardware buffer; the first processor sends the pointer address of the first memory to the third processor and the second processor, and the first processor, the second processor and the third The processor has the right to access the first memory, and the third processor and the second processor execute image data reading and writing to the first memory based on the pointer address.
  • the first processor may allocate the first memory from the hardware buffer, and the second processor does not need to do other processing before writing the image data of the first image at the second resolution into the first memory.
  • the image data of the first image at the second resolution is written in the memory, and the second processor and the third processor can share data based on the first memory, realizing the sharing of data between the second processor and the third processor Zero copy, improve processing efficiency.
  • the second processor reads the image data of the first image at the third resolution from the first memory.
  • the method also includes: the third processor reads the image data of the first image at the second resolution from the first memory; the third processor reads the image data of the first image at the second resolution based on the image data of the first image, generating image data of the first image at the third resolution; the third processor writes the image data of the first image at the third resolution into the first memory.
  • the third processor reads the image data of the first image at the second resolution from the first memory.
  • the method further includes: the first processor sends a fifth notification to the third processor, and the fifth notification is used to instruct the third processor to read an image of the first image at the second resolution from the first memory data.
  • the first processor can monitor the image data reading and writing work of the second processor, and after the second processor writes the image data of the first image at the second resolution into the first memory, it will trigger the third processor to read in time Image data of the first image at the second resolution, improving efficiency.
  • the second processor reads the image data of the first image at the third resolution from the first memory.
  • the method further includes: the first processor sends a sixth notification to the second processor, and the sixth notification is used to instruct the second processor to read the image of the first image at the third resolution from the first memory data.
  • the first processor can monitor the image data reading and writing work of the third processor, and after the third processor writes the image data of the first image at the third resolution into the first memory, it will trigger the second processor to read in time Image data of the first image at the third resolution, improving efficiency.
  • the electronic device also includes a third processor, and the image data of the first image at the third resolution is generated by the third processor; the third processor runs an artificial intelligence super-resolution model, and the third processor The artificial intelligence super-resolution model is used to perform super-resolution rendering on the image data of the first image at the second resolution to generate image data of the first image at the third resolution.
  • the method further includes: the first processor sends The first resolution and the second resolution; the third processor determines the super-resolution multiple of the artificial intelligence super-resolution model based on the first resolution and the second resolution, and the artificial intelligence resolution model is based on the super-resolution multiple for the first
  • the image data at the second resolution is super-resolution rendered.
  • the artificial intelligence super-resolution model can have at least one super-resolution multiple, and each super-resolution multiple can complete the conversion of different resolutions.
  • the first processor may send the first resolution and the second resolution to the third processor.
  • the third processor selects a super-resolution multiple that is not less than the multiple difference, so that the corresponding resolution of the image data output by the artificial intelligence super-resolution model is not less than first resolution.
  • the method also includes: the first processor performs artificial intelligence super-resolution
  • the model is initialized.
  • the initialization is used to determine the operation of the artificial intelligence super-resolution model and the normal operation of the artificial intelligence super-resolution model; the initialization includes runtime detection, model loading, model compilation, and memory configuration.
  • the runtime check is used to determine the operation manual Intelligent super-resolution model, model loading, model compilation, and memory configuration are used to determine the normal operation of the artificial intelligence super-resolution model.
  • the method further includes: the first processor reduces the resolution of the first image from the first resolution to the second resolution.
  • the electronic device further includes a third processor, and the image data of the first image at the third resolution is generated by the third processor; the third processor has a super-resolution factor, and the super-resolution factor is used to indicate the second The difference between the resolution and the third resolution; the third resolution is the same as the first resolution, the first processor reducing the resolution of the first image from the first resolution to the second resolution includes: the first processing The processor reduces the resolution of the first image from the first resolution to the second resolution based on the superresolution factor.
  • the artificial intelligence super-resolution model can have at least one super-resolution multiple, and each super-resolution multiple can complete the conversion of different resolutions.
  • different super-resolution multiples make the corresponding resolutions of the image data output by the artificial intelligence super-resolution model different, so the first processor can complete the first image based on the super-resolution multiples.
  • the reduction of the resolution makes the reduction of the resolution match a super-resolution multiple of the artificial intelligence super-resolution model, and the artificial intelligence super-resolution model can be used to complete the resolution improvement to ensure the accuracy.
  • the third processor is a neural network processor or a digital signal processor.
  • the present application provides a rendering method, the method is applied to the second processor of the electronic device, the electronic device runs an application program, the electronic device also includes a first processor, and the application program sends the rendering to the first processor command, the rendering command is used to instruct the second processor to render the first image based on the first resolution
  • the method includes: the second processor receives a rendering instruction sent by the first processor, and the rendering instruction is used to instruct the second processor to render the first image The first image is rendered; the second processor generates image data of the first image at a second resolution based on the rendering instruction, and the second resolution is not greater than the first resolution; the second processor renders the first image at the second The image data under the resolution is written into the first memory; the second processor reads the image data of the first image under the third resolution from the first memory, and the third resolution is greater than the second resolution; the second processing The device generates the first image based on the image data of the first image at the third resolution.
  • the second processor After the second processor writes the generated image data of the first image at the second resolution into the first memory, the second processor can read the image of the first image at the third resolution from the first memory Data, saving the second processor to generate the image data of the first image at the third resolution, saving the computing power of the second processor, shortening the time spent in the rendering process, thereby reducing power consumption and improving rendering fluency, Solve the problem of serious equipment heating and running freezes.
  • the third resolution is greater than the second resolution, indicating that the second processor can obtain image data with a relatively high resolution, and the second processor can draw the first image based on the image data with a relatively high resolution to improve the second image data. The image quality of an image.
  • the method further includes: the second processor writes the image data of the first image at the second resolution Write to the second memory, the second processor has the right to access the second memory and the first memory, the third processor has the right to access the first memory; the second processor sends a first notification to the first processor, the first The notification is used to indicate that the image data of the first image at the second resolution is successfully written into the second memory; in response to the second notification sent by the first processor, the second processor reads the first image from the second memory in For image data at the second resolution, the second notification is sent after the first processor receives the first notification, and the second notification is used to instruct the second processor to write the image data of the first image at the second resolution into To the first memory, the second notification carries the address pointer of the first memory.
  • the first processor can monitor the image data reading and writing work of the second processor, so that after the second processor writes the image data of the first image at the second resolution into the second memory, the second processor is triggered in time to write The image data of the first image under the second resolution is written into the first memory, so as to improve efficiency.
  • the second notification sent by the first processor carries the address pointer of the first memory, and the second processor can write the image of the first image under the second resolution into the first memory based on the address pointer of the first memory. Image data, improve the accuracy of writing.
  • the method further includes: the second processor reads the image data of the first image at the third resolution Write into the second memory, the second processor has access to the second memory and the first memory; the image data of the first image at the third resolution is generated by the third processor in the electronic device, and the third processor has Permission to access primary memory.
  • the method further includes: the second processor receives a notification sent by the first processor, and the notification is used to indicate The second processor reads the image data of the first image under the third resolution from the first memory, and the notification carries the address pointer of the first memory.
  • the second processor can read the image data of the first image at the third resolution from the correct address based on the address pointer of the first memory, so as to improve the accuracy of the read image data.
  • the present application provides an electronic device.
  • the electronic device includes: a first processor, a second processor, and a memory; wherein, the memory is used to store one or more computer program codes, and the computer program codes include computer instructions.
  • the first processor and the second processor execute the computer instructions
  • the first processor and the second processor execute the above rendering method.
  • the present application provides a system-on-a-chip, where the system-on-a-chip includes program codes that, when the program codes run on the electronic device, cause the first processor and the second processor in the electronic device to execute the rendering method described above.
  • the present application provides a processor, the processor is a second processor, and the second processor includes a processing unit and a memory; wherein, the memory is used to store one or more computer program codes, and the computer program codes include computer instructions , when the second processor executes the computer instructions, the second processor executes the above rendering method.
  • the present application provides a computer storage medium, the computer storage medium includes computer instructions, and when the computer instructions are run on the electronic device, the second processor in the electronic device executes the above rendering method.
  • the present application provides a rendering method, which is applied to rendering processing of the first image by an electronic device, where the electronic device runs an application program, and the electronic device includes a first processor, a second processor, and a third processor, the method Including: the first processor receives a rendering command issued by the application program, the rendering command is used to instruct rendering of the first image, and the first image has a second resolution; the third processor obtains the first image data, and the first image data is the image data of the first image with the first resolution; the third processor performs super-resolution rendering on the first image data to obtain the second image data, and the second image data is the image data of the first image with the second resolution image data, the first resolution is smaller than the second resolution; the second processor generates the first image with the second resolution based on the second image data.
  • the method further includes: in response to the rendering command, the first processor determines that the resolution of the first image is the first resolution, and instructs the third processor to obtain the first image data.
  • the method further includes: the first processor responds to the rendering command, determining that the resolution of the first image is the second resolution; the first processor reduces the resolution of the first image from the second resolution to the first resolution rate; the first processor instructs the third processor to obtain the first image data.
  • the method further includes: the first processor selects a zoom factor related to the superresolution factor from the zoom range according to the superresolution factor of the third processor, and the superresolution factor is used to indicate the first resolution and the second resolution
  • the first processor reducing the resolution of the first image from the second resolution to the first resolution includes: the first processor uses the scaling factor to reduce the resolution of the first image, so that The resolution of the first image is reduced from the second resolution to the first resolution, and the relationship between the zoom factor and the super-resolution factor is that the multiplication result of the zoom factor and the super-resolution factor is 1.
  • the AI super-resolution model is run in the third processor, and the third processor performs super-resolution rendering on the first image data, and obtaining the second image data includes: the third processor uses the AI super-resolution model to The first image data is subjected to super-resolution rendering.
  • the method further includes: the first processor initializes the AI super-resolution model, and the initialization is used to determine whether to run the AI super-resolution model and to determine that the AI super-resolution model can run normally; initialization includes runtime detection, model loading , model compilation and memory configuration, the runtime check is used to determine the running of the AI super-resolution model, and the model loading, model compilation and memory configuration are used to confirm that the AI super-resolution model can run normally.
  • the electronic device when the electronic device performs rendering processing on the first image, it invokes one or more frame buffers, and the method further includes: the first processor determines that the resolution of the first image is the second resolution, and the rendering method of the first image For delayed rendering and the currently processed frame buffer is not the last frame buffer, instruct the third processor to obtain the first image data, so as to perform rendering processing on the currently processed frame buffer.
  • the method further includes: when the first processor determines that the resolution of the first image is the second resolution, the rendering mode of the first image is delayed rendering, and the currently processed frame buffer is the last frame buffer, instructing the second The processor obtains fourth image data to perform rendering processing on the currently processed frame buffer, where the fourth image data is image data of the first image with the second resolution.
  • the electronic device when it performs rendering processing on the first image, it invokes one or more frame buffers, and the method further includes: the first processor determines the main frame buffer in the process of performing rendering processing on the first image; the main frame buffer is the electronic During the rendering process of the first image by the device, the frame buffer with the largest number of rendering operations is performed; the first processor determines that the resolution of the first image is the second resolution, the rendering method of the first image is forward rendering, and the current When the processed frame buffer is the main frame buffer, instruct the third processor to obtain the first image data, so as to perform rendering processing on the currently processed frame buffer.
  • the method further includes: when the first processor determines that the resolution of the first image is the second resolution, the rendering mode of the first image is forward rendering, and the currently processed frame buffer is not the main frame buffer, instruct the second The second processor obtains fourth image data to perform rendering processing on the currently processed frame buffer, where the fourth image data is image data of the first image with a second resolution.
  • the first processor determining the primary frame buffer during the rendering process of the first image includes: the first processor determines the primary frame buffer based on the frame buffer that performs the largest number of rendering operations during the rendering process of the second image. Framebuffer; rendering of the second image precedes rendering of the first image.
  • the method before determining the main frame buffer in the process of rendering the first image, the method further includes: in the process of rendering the second image, determining the number of drawing instructions executed on each frame buffer; The frame buffer that executes the largest number of drawing instructions is determined as the main frame buffer.
  • the second image is an image last frame of the first image.
  • the first processor is a central processing unit
  • the second processor is a graphics processor
  • the third processor is a neural network processor or a digital signal processor.
  • FIG. 1 is a schematic diagram of a rendered image provided by the present application
  • FIG. 2 is a schematic diagram of another rendered image provided by the present application.
  • FIG. 3 is a schematic interface diagram of a resolution enhancement setting provided by the present application.
  • FIG. 4 is a schematic diagram of another rendered image provided by the present application.
  • FIG. 5 is a schematic diagram of another rendered image provided by the present application.
  • FIG. 6 is a schematic diagram of determining the number of drawcalls on different frame buffers provided by the present application.
  • FIG. 7 is a schematic diagram for determining the number of drawcalls on another different frame buffer provided by the present application.
  • FIG. 8 is a schematic diagram of another rendered image provided by the present application.
  • FIG. 9 is a schematic diagram of a rendering method provided by the present application.
  • FIG. 10 is a flowchart of a rendering method provided by the present application.
  • FIG. 11 is a signaling diagram of the rendering method provided by the present application.
  • FIG. 12 is a schematic diagram of memory access provided by the present application.
  • FIG. 13 is another signaling diagram of the rendering method provided by the present application.
  • FIG. 14 is a schematic diagram of another memory access provided by the present application.
  • FIG. 15 is another signaling diagram of the rendering method provided by the present application.
  • words such as “exemplary” or “for example” are used as examples, illustrations or illustrations. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as “exemplary” or “such as” is intended to present related concepts in a concrete manner.
  • the electronic device may display frames of images to the user through a display screen.
  • a video stream may include multiple frames of images, and the electronic device may sequentially display each frame of images on the display screen to realize displaying the video stream on the display screen.
  • the image display can be triggered by an application program in the electronic device, and the application program can send a rendering command for different images to the electronic device, and the electronic device renders the image in response to the rendering command, and displays it based on the rendering result of the image.
  • each frame of image corresponds to multiple frame buffers (FrameBuffer, FB), and each FB is used to store the rendering results of some elements of the image, such as storing image data of some elements, and the image data is used to draw the corresponding elements .
  • the image includes characters, trees and other elements
  • the image data of characters and trees can be stored in the FB
  • the electronic device can draw based on the image data stored in the FB.
  • FIG. 1 an optional structure of an electronic device and a rendering process based on FB are shown in FIG. 1, and the electronic device may include a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), internal storage and display, internal storage can also be referred to as memory.
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • internal storage can also be referred to as memory.
  • the application program can display images through the display screen.
  • the application program sends rendering commands, and the CPU can intercept Rendering command issued by the application.
  • the CPU responds to the rendering command and creates corresponding FBs for the rendering process of the i-th frame image in the memory.
  • the CPU can create 3 FBs for the i-th frame image, which are respectively recorded as FB0, FB1 and FB2.
  • the CPU can issue a rendering instruction to the GPU according to the rendering command, and the GPU responds to the rendering instruction and executes the rendering corresponding to the rendering instruction.
  • the GPU performs rendering processing on FB1 and FB2 of the i-th frame image in response to the rendering instruction.
  • FB1 and FB2 respectively store the image data of the i-th frame image
  • FB1 can be used to store the image data of a part of the i-th frame image (called image data 1)
  • FB2 can be used to store the image data of the i-th frame image
  • image data 2 The image data of another part of the i-frame image.
  • the GPU fuses (or renders) image data 1 and image data 2 into FB0
  • FB0 stores the complete image data of the i-th frame image.
  • the GPU reads the image data of the i-th frame image from FB0, and displays the i-th frame image on the display screen according to the image data of the i-th frame image.
  • the screen refresh rate and image resolution of the display screen develop to a higher stage, such as the screen refresh rate of the display screen develops from 60Hz (Hertz, Hertz) to 90Hz, and then from 90Hz to 120Hz;
  • the resolution of the image has developed from 720P (1280 ⁇ 720) to 1080P (1920 ⁇ 1080), and from 1080P to 2k (2560 ⁇ 1440), which makes the rendering of electronic equipment run more smoothly, the scene is more complex and the picture quality is higher
  • the direction of development puts forward higher demands on the computing power of electronic equipment.
  • the present application provides a rendering method, in which processors other than the CPU and GPU are configured in the electronic device, and the CPU and GPU control the rendering process is adjusted to the CPU, GPU and other processors control the rendering process, for example
  • the computing power of the configured processor can be greater than that of the GPU, but the energy consumption can be less than that of the GPU, and the control ability can be weaker than that of the CPU and GPU.
  • other processors can generate high-resolution images and/or ultra-high-resolution images, and actively call the computing power of other processors to complete super-resolution rendering to meet high-resolution images.
  • the computing power requirements of ultra-high-resolution images reduce the power consumption of electronic equipment, and reduce the phenomenon of severe heat generation.
  • other processors can share the calculation load of the GPU, shorten the time spent in the rendering process, thereby improving the smoothness of rendering and solving the problem of running stuck.
  • super-resolution rendering is used to improve the resolution of image data.
  • the resolution improvement can be from low resolution to high resolution or super high resolution, or from high resolution to super high resolution. improvement.
  • a neural network processor (Neural-network Processing Unit, NPU) is configured in the electronic device, and the rendering process is controlled by the CPU, GPU, and NPU.
  • the NPU can perform super-resolution rendering to obtain high-resolution images with the assistance of the NPU. image data or image data for ultra-high resolution images.
  • a digital signal processor (Digital Signal Processor, DSP) is configured in the electronic device, and the rendering process is controlled by the CPU, GPU, and DSP, and the DSP can complete super-resolution rendering.
  • DSP Digital Signal Processor
  • NPU and DSP are configured in the electronic device, and the rendering process is controlled by the CPU, GPU, NPU and DSP.
  • the NPU and DSP can complete super-resolution rendering, wherein the super-resolution rendering of the NPU and DSP can correspond to different resolutions.
  • the NPU can correspond to the conversion from low resolution to high resolution
  • the DSP can correspond to the promotion from low resolution to ultra-high resolution
  • the NPU can correspond to the conversion from low resolution to ultra-high resolution.
  • Upgrading DSP can correspond to upgrading from low resolution to high resolution; in another example, NPU can correspond to upgrading from low resolution to high resolution, and DSP can correspond to upgrading from high resolution to ultra-high resolution.
  • the NPU and DSP can cooperate to complete super-resolution rendering.
  • the NPU and DSP can cooperate to cope with the improvement from low resolution to high resolution; Improvement, where cooperative response refers to the joint participation of NPU and DSP in the super-resolution rendering process of a frame of image.
  • Low resolution can be 540P (960 ⁇ 540) and 720P
  • high resolution can be 1080P
  • super high resolution can be 2k or even greater than 2k
  • GPU can generate image data of 540P images
  • NPU can complete the upgrade from 540P to 1080P, and can also complete the upgrade from 720P to 1080P, so as to use the NPU to generate image data of 1080P image, in addition, the NPU can also generate 2k or even resolution greater than 2k The image data for the image.
  • a usage scenario of the rendering method is: the quality of the picture output by the electronic device is higher than the picture quality required by the application program.
  • the method may be that the electronic device can increase the resolution of the image required by the application program.
  • the electronic device can increase its resolution to high resolution or super high resolution, automatically improving the picture quality of the application; or when the resolution of the image required by the application is At high resolutions, electronic devices upscaling to ultra-high resolutions can also automatically improve the picture quality of applications.
  • the resolution of the image required by the application is low resolution
  • the electronic device executes the rendering method to increase the low resolution of the image to a high resolution, thereby automatically changing the resolution of the image to improve the picture quality.
  • the processing process of the CPU, NPU and GPU includes: the CPU intercepts the rendering command sent by the application; the CPU responds to the rendering command and creates a corresponding FB for the image rendering process in the memory; the CPU obtains the resolution of the image based on the rendering command.
  • GPU After low resolution, send rendering instructions to GPU; GPU generates low-resolution image data based on rendering instructions, and sends low-resolution image data to NPU; NPU performs super-resolution rendering processing on low-resolution image data to obtain High-resolution image data, send the high-resolution image data to the GPU; after the GPU receives the high-resolution image data, perform user interface (UI) rendering and post-processing based on the high-resolution image data, To complete the drawing of a frame of image. For example, based on high-resolution image data, the GPU performs rendering processing on the FB pointed to by the rendering command to obtain a rendering result, which may be image data of some elements in the image. After obtaining the rendering results of all FBs, the GPU displays the drawn image on the display screen based on the rendering results.
  • UI user interface
  • the electronic device may provide a setting interface, in which there is a setting option of "resolution enhancement", and the setting option is used to provide resolution selection.
  • Figure 3 shows a schematic diagram of a setting interface.
  • the setting options provide a variety of resolution options, such as 1080P, 2k, and greater than 2k, and the user can choose one of these resolutions;
  • the electronic device executes the rendering method shown in FIG. 2 above.
  • FIG. 3 is just an example, and the setting options may also take other forms, as shown in the dashed box in FIG. 3 , and this embodiment does not limit the setting options.
  • the electronic device can set a white list, which records the applications that perform resolution enhancement.
  • the CPU intercepts the rendering command issued by the application, it determines whether the application is in the white list. If the application is in the white list , the CPU can trigger the NPU to increase the resolution through the GPU, the NPU sends the image data after the resolution increase to the GPU, and the GPU performs rendering processing based on the image data after the resolution increase; if the application is not in the whitelist, the CPU triggers the GPU Perform rendering processing, and the resolution of the image data at this time is the same as the resolution delivered by the application.
  • the image quality output by the electronic device is the same as the image quality required by the application program, but the electronic device can first reduce the resolution and then increase the resolution. As shown in Figure 4, the resolution of the image required by the application can be high resolution.
  • the processing process of the CPU, NPU and GPU includes: the CPU intercepts the rendering command issued by the application; Create the corresponding FB for rendering processing; after the CPU extracts the resolution of the image from the rendering command to a high resolution, the CPU reduces the resolution of the image, and then the CPU can send rendering instructions to the GPU; the GPU generates a low-resolution image based on the rendering instruction Image data, send low-resolution image data to NPU; NPU performs super-resolution rendering processing on low-resolution image data to obtain high-resolution image data, and sends high-resolution image data to GPU; GPU receives After receiving the high-resolution image data, UI rendering and post-processing are performed based on the high-resolution image data to complete the drawing of one frame of image.
  • the rendering result of the image is stored in the FB, and the GPU displays the drawn image on the display superior.
  • the NPU can restore the high-resolution image data through super-resolution rendering processing
  • the GPU can use the high-resolution image data to restore the high-resolution image, so that the image drawn by the GPU and the application
  • the picture quality of the image required by the program is the same, which meets the picture quality requirements of the application program and improves the rendering accuracy.
  • the amount of data input to the NPU is reduced, which can improve the rendering accuracy and speed up the rendering speed.
  • the CPU can reduce the resolution of the image, and then use the NPU to perform super-resolution rendering on the reduced-resolution image data, so that the rendering result after super-resolution rendering is the same as that before reduction. corresponding to the resolution. How much the resolution is reduced depends on the super-resolution capability of the NPU. For example, if the super-resolution capability is 2 times, the resolution of 1080P images is reduced by 0.5 times. It can be noted that 2 times and 0.5 times correspond to resolution The side lengths of the rate, not the pixels of the image.
  • the CPU can decide whether to reduce the resolution according to the resolution of the image required by the application program.
  • the resolution of the image required by the application program is 1080P or above, and the CPU reduces the Resolution, and then the NPU performs super-resolution rendering on the low-resolution image data; if the resolution of the image required by the application is less than 1080P, the CPU triggers the GPU to perform rendering processing while the image resolution remains unchanged.
  • Figure 4 shows the rendering method.
  • the NPU when the resolution of the image required by the application is high resolution, after the NPU receives the low-resolution image data sent by the GPU, the NPU performs super-resolution rendering processing on the low-resolution image data to obtain super-high resolution
  • the image data of the GPU is drawn based on the ultra-high-resolution image data, and the image quality of the image drawn by the GPU is higher than that of the image required by the application.
  • the resolution of the image required by the application is 1080P
  • the CPU reduces the resolution to 540P
  • the GPU generates 540P image data
  • the NPU feeds back 2K image data to the GPU
  • the GPU can draw an image with a resolution of 2K, and its picture quality is high
  • the picture quality of the application is improved.
  • Another usage scenario corresponding to the rendering method is: during the running of the application, the user can adjust the image quality of the application.
  • the user can adjust the image quality from low resolution to high resolution or super high resolution, or from high resolution to high resolution. high resolution or super high resolution to low resolution.
  • Figure 5 shows that the user adjusts the picture quality from low resolution to high resolution, and the corresponding processing process also changes after the resolution adjustment, and the processing process is as follows:
  • the CPU intercepts the rendering command issued by the application program; the CPU responds to the rendering command and creates a corresponding FB for image rendering processing in the memory.
  • the CPU After the CPU obtains a low-resolution image based on the rendering command, the CPU sends a rendering command to the GPU.
  • the GPU responds to the rendering command, performs rendering processing on the FB, and obtains a rendering result. After the GPU completes the rendering processing of all FBs, the GPU displays the image on the display screen based on the rendering result.
  • the CPU can obtain a high resolution image based on the rendering command, and the CPU reduces the image resolution to a low resolution, and then the CPU can send a rendering command to the GPU;
  • the GPU generates low-resolution image data based on rendering instructions, and sends the low-resolution image data to the NPU;
  • the NPU performs super-resolution rendering processing on the low-resolution image data to obtain high-resolution image data, and sends the high-resolution
  • the GPU receives the high-resolution image data, it performs UI rendering and post-processing based on the high-resolution image data to complete the drawing of a frame of image.
  • the rendering result of the image is stored in the FB, and the GPU Display the drawn image on the display.
  • the CPU can specify the FB for super-resolution rendering
  • the GPU can use the NPU and/or DSP to complete the super-resolution rendering when processing the specified FB, such as using the NPU and/or DSP to complete from low resolution to high resolution And/or ultra-high-resolution conversion, use GPU rendering for other FBs, GPU can no longer distinguish the resolution of the image.
  • the FB for performing super-resolution rendering can be different according to different rendering methods.
  • Rendering methods include deferred rendering and forward rendering.
  • color information color attachment
  • normal vector data color attachment
  • the normal vector data after upsampling is in There is a high probability of error when performing lighting calculations, and all FBs except FB0 are highly correlated (for example, the rendering result of one FB is bound to the next FB), if super-resolution rendering is performed on an FB , when other FBs use the rendering results of this FB, there is a high probability of error.
  • super-resolution rendering can be performed on all FBs corresponding to each frame image except FB0, that is, in deferred rendering, super-resolution rendering can be performed.
  • the FBs rendered by the resolution are the FBs except FB0 among all the FBs corresponding to one frame of image.
  • all FBs corresponding to a frame of image include: FB0, FB1, and FB2
  • FBs for super-resolution rendering include FB1 and FB2
  • FB0 is the last FB among all FBs delivered by the application.
  • the CPU can determine the FB for super-resolution rendering according to the number of rendering operations.
  • the CPU may determine the FB with the number of rendering operations greater than the preset number as the FB for super-resolution rendering; in another example, the CPU may determine the FB with the largest number of rendering operations as the FB for super-resolution rendering FB, the FB with the largest number of rendering operations can be the largest number of Drawcalls (drawing instructions related to drawing) executed by the GPU on the FB. This is because the more Drawcalls executed by the GPU on the FB, it indicates that the GPU executes on the FB.
  • the FB with the largest number of rendering operations can be determined as the super-resolution rate rendered FB.
  • the FB with the largest number of rendering operations is the FB that executes the largest number of Drawcalls.
  • the main FB For the convenience of description, the FB performing super-resolution rendering in the forward rendering is called the main FB.
  • the FB determining the largest number of Drawcalls will be described below with an example.
  • the CPU may be configured to receive a rendering command from an application program, and issue a corresponding rendering command to the GPU according to the rendering command, so that the GPU performs corresponding rendering according to the rendering command.
  • the rendering command may include glBindFrameBuffer() function, and one or more glDrawElement, glDrawArray, glDrawElementInstanced, glDrawArrayInstanced.
  • the rendering instruction may also include the glBindFrameBuffer() function, and one or more glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced.
  • the glBindFrameBuffer() function can be used to indicate the currently bound FB, and realize the binding between the corresponding rendering operation and the FB.
  • glBindFrameBuffer(1) can indicate that the currently bound frame buffer is FB1
  • the Drawcalls executed by the GPU on FB1 include: glDrawElement, glDrawArray, glDrawElementInstanced, and glDrawArrayInstanced.
  • the image of the Nth frame can use FB0, FB1 and FB2 as the frame buffer as an example.
  • the rendering command issued by the application may include the glBindFrameBuffer(1) function, thereby realizing the binding of the current rendering operation to FB1.
  • one glDrawElement command can correspond to one Drawcall.
  • the CPU can initialize the counter 1 when executing glBindFrameBuffer(1) according to the rendering command issued by the application program. For example, configure the corresponding counting frame bit for FB1 in the memory. By initializing the counter 1, the value of the frame bit can be initialized to 0. Each time glDrawElement is executed on FB1, the count of counter 1 is increased by 1, such as executing count1++. For example, after executing glDrawElement1-1, the CPU can execute count1++ on the counter 1, so that the value of the frame bit stored in the number of FB1Drawcalls changes from 0 to 1, and the number of Drawcalls executed on FB1 is 1 at this time. By analogy, the CPU can determine that the number of Drawcalls executed on FB1 is the current count of counter 1 (for example, the count can be A) during the rendering of the Nth frame of image.
  • the count can be A
  • the CPU can determine that the number of Drawcalls executed on FB2 is the current count of counter 2 (for example, the count can be B) during the rendering process of the image of the Nth frame.
  • the value of the frame bits storing the number of FB1Drawcalls in the memory can be A, and the number of frame bits storing the number of FB2Drawcalls can be B.
  • the CPU can select the FB corresponding to the larger count in A and B as the main FB. For example, when A is greater than B, the CPU can determine that the number of Drawcalls executed on FB1 is more, and determine that FB1 is the main FB. In contrast, when A is smaller than B, the CPU can determine that the number of Drawcalls executed on FB2 is more, and determine that FB2 is the main FB.
  • the CPU may determine the main FB whose number of Drawcalls is greater than the preset threshold, and the CPU may select the FB whose Drawcall number is greater than the preset threshold by means of counting, which will not be described in detail here.
  • the rendering process of the CPU, GPU, and NPU is shown in Figure 8.
  • the CPU triggers the rendering process according to the rendering command issued by the application program, and the CPU creates the FB.
  • the application program in Figure 8 indicates to render a high-resolution image.
  • the CPU receives the rendering command, it determines whether the rendering mode is forward rendering or delayed rendering. If the rendering method is forward rendering, the CPU determines whether the FB currently performing the rendering operation is FB1 (FB1 is the main FB).
  • the CPU reduces the resolution of the image and sends a rendering command to the GPU to
  • the GPU generates low-resolution image data
  • the GPU sends the low-resolution image data to the NPU, and the NPU completes super-resolution rendering;
  • the CPU determines that the rendering method is delayed rendering, the CPU determines whether the FB currently performing the rendering operation is FB0; if The FB currently performing the rendering operation is FB1 or FB2.
  • the CPU can also reduce the resolution of the image and instruct the GPU to generate low-resolution image data.
  • the GPU sends the low-resolution image data to the NPU, and the NPU completes super-resolution rendering.
  • the GPU After the GPU receives the high-resolution image data sent by the NPU, it performs UI rendering and post-processing based on the high-resolution image data to complete the drawing of a frame of image.
  • the rendering result of the image is stored in FB, and the GPU will render the drawn image The image is displayed on the monitor.
  • the two GPUs in FIG. 8 may be the same GPU.
  • the reason why the two GPUs are shown is to illustrate that different processing methods are adopted for different FBs in different rendering methods.
  • the CPU reduces the resolution of the image
  • the GPU can generate low-resolution image data corresponding to FB1, and can also generate low-resolution image data corresponding to FB2.
  • the low-resolution image data, and then the NPU performs super-resolution rendering on the low-resolution image data corresponding to the two FBs.
  • the last FB of FB0 corresponds to the image data of a frame of image
  • the GPU can generate low-resolution image data of the last FB of FB0
  • the NPU corresponds to the last FB of FB0 low-resolution image data for super-resolution rendering.
  • the GPU can generate low-resolution image data, and then the NPU performs super-resolution rendering on the low-resolution image; it can also be used when the FB is the previous one of FB0 During FB, the GPU generates low-resolution image data, and the NPU performs super-resolution rendering on the low-resolution image data.
  • the CPU can reduce the resolution of the image according to the super resolution of the NPU.
  • the super-resolution ratio is used to indicate the multiple of the resolution that the NPU can increase. For example, if the super-resolution ratio of the NPU is 2 times, the CPU can reduce the resolution of the image by 2 times; if the super-resolution ratio of the NPU is 4 times, the CPU can reduce the image resolution. The resolution is reduced by a factor of 4. For example, if the super resolution ratio of the NPU is 2 times, the CPU can reduce the resolution of the image from 1080P to 540P, and the NPU can convert 540P to 1080P to complete the super-resolution rendering from 540P to 1080P.
  • the CPU can reduce the resolution of the image from 2k to 540P, then the NPU can convert 540P to 2K to complete the super-resolution rendering from 540P to 2k; the CPU can also reduce the resolution of the image The resolution is reduced from 1080P to 270P, and accordingly the NPU can convert 207P to 1080P.
  • the schematic diagram corresponding to the rendering method is shown in 9.
  • the application program issues a rendering command for each frame of image, and the CPU intercepts the rendering command issued by the application program. Based on the rendering command, the CPU determines the value of the image corresponding to the current rendering command.
  • the resolution is high resolution, and the CPU can reduce the resolution of the image to a low resolution; the CPU sends rendering instructions to the GPU, and the GPU generates low-resolution image data based on the rendering instructions;
  • the resolution model completes the conversion from low-resolution to high-resolution to obtain high-resolution image data; GPU performs UI rendering and post-processing based on high-resolution image data, and displays the image on the display.
  • the flow chart of the rendering method shown in Figure 9 is shown in Figure 10.
  • the AI super-resolution model runs in the NPU, and the AI super-resolution model is used to complete the super-resolution rendering, which may include the following step:
  • the CPU determines that FB1 is the main FB during the rendering process of the N-1th frame image.
  • the main FB is the FB that executes the largest number of rendering operations during the rendering process.
  • the FB with the largest number of rendering operations may be the FB that executes the largest number of Drawcalls.
  • the CPU determines that FB1 is the FB that executes the largest number of Drawcalls.
  • One frame of image corresponds to multiple FBs, and the number of Drawcalls executed by each FB is different.
  • This embodiment can count all the FBs in a frame of image to obtain the number of Drawcalls executed by all FBs in the frame of image, and then in the frame of image
  • the FB that executes the largest number of Drawcalls is determined as the main FB, and the main FBs of other images after this frame of image can be the same as the main FB of this frame of image.
  • the CPU can determine the number of Drawcalls executed on different FBs of the N-1th frame image during the image rendering process of the previous frame (such as the N-1th frame image), and determine that FB1 is the main FB through the number of Drawcalls executed on different FBs , determine the main FB of the N-1th frame image as the main FB of the Nth frame image, the N+1th frame image, the N+2th frame image, etc.
  • the CPU intercepts the rendering command of the image of the Nth frame issued by the application program.
  • the CPU acquires that the initial resolution of the image of the Nth frame is high resolution and the currently processed FB.
  • the CPU can obtain the width and height of the image of the Nth frame based on the rendering command.
  • the width and height of the image at a resolution are fixed. For example, when the resolution of the image is 720P, the width of the image is 1280 and the height of the image is 720. ; When the resolution of the image is 1080P, the width of the image is 1920, and the height of the image is 1080, so the initial resolution of the image of the Nth frame can be determined by the width and height of the image of the Nth frame. Generally, images of 1920 ⁇ 1080 and above are considered to be high resolution. In this embodiment, when the width and height of the image meet 1920 ⁇ 1080 and above, the initial resolution of the image is determined to be high resolution. Based on the rendering command, the CPU can obtain the currently processed FB identifier, such as FB1. The initial resolution of the Nth frame image and the currently processed FB can be obtained from different rendering commands.
  • the CPU initializes the AI super-resolution model, determines whether to run the AI super-resolution model in the NPU through the initialization operation, and if running, ensures that the AI super-resolution model can run normally.
  • the initialization of the AI super-resolution model includes runtime inspection, model loading, model compilation, and memory configuration.
  • the runtime inspection is to determine whether to run the AI super-resolution model in the NPU.
  • Model loading, model compilation, and memory configuration are to ensure that AI The super-resolution model works fine.
  • the runtime check includes checking whether an NPU is configured in the electronic device and checking the resolution of the application; if an NPU is configured, running the AI super-resolution model in the NPU; if the resolution of the application is high resolution Or ultra-high resolution, run the AI super-resolution model; if the electronic device is not configured with NPU or the resolution of the application is low resolution, it is forbidden to run the AI super-resolution model, and the CPU sends rendering instructions to the GPU, and the GPU responds Render instructions to render.
  • Model loading is to convert the AI super-resolution model into a model file recognizable by the NPU.
  • the model file is loaded into the memory.
  • the model compilation is to verify that the model file can run successfully; the memory configuration is allocated for the AI super-resolution model.
  • Memory the allocated memory is used to store the input data and output data of the AI super-resolution model.
  • the memory allocated for the AI super-resolution model can be CPU memory, memory managed by Neural Processing API (such as SNPE ITensor buffer), or shared memory.
  • the CPU memory can be the memory allocated for the CPU.
  • the data used during the CPU operation is written into the CPU memory.
  • the AI super-resolution model occupies part of the CPU memory as the memory of the AI super-resolution model.
  • the link between the GPU and the NPU Data interaction can pass through the CPU memory, and when the GPU draws an image, the data in the CPU memory can be written into the GPU memory (the memory allocated for the GPU), and then the data can be read from the GPU memory for drawing.
  • the shared memory can be shared by the CPU, GPU, and NPU, and the GPU can directly read data from the shared memory for drawing.
  • the CPU determines that the initial resolution of the Nth frame image is high resolution, it triggers the initialization of the AI super-resolution model, which is a way of initializing the AI super-resolution model.
  • the AI super-resolution model may be initialized at other timings.
  • the CPU initializes the AI super-resolution model after detecting that the application program is started. Some applications have less image rendering times and low GPU usage.
  • the CPU can set a whitelist for the application. After the CPU detects that the application is started, it determines whether the application is in the whitelist. If the AI super-resolution is in the whitelist The rate model is initialized.
  • the CPU initializes the AI super-resolution model when the rendering method is forward rendering and the currently processed FB is FB1; when the rendering method is delayed rendering and the currently processed FB is not FB0, the AI super-resolution model is initialized.
  • the model is initialized.
  • the CPU acquires the rendering mode from the configuration file of the application program.
  • Rendering methods include deferred rendering and forward rendering.
  • the rendering methods are different, and the FBs that perform super-resolution rendering are also different.
  • deferred rendering super-resolution rendering can be performed on all FBs except FB0 corresponding to each frame image, that is, in deferred rendering, super-resolution can be performed on FBs except FB0 among all FBs corresponding to each frame image rate rendering.
  • the CPU After determining that the FB currently executing the rendering command is not FB0, the CPU performs super-resolution rendering on the FB currently executing the rendering command.
  • the computing power consumption of the GPU during the rendering process is concentrated in the main FB rendering.
  • the computing power of the GPU is concentrated in the calculation of the fragment shader (Fragment Shader, FS). From this, it can be inferred that the super-resolution rendering of the main FB in the forward rendering can be performed by the main FB. It is the FB that executes the largest number of Drawcalls. For example, in this embodiment, the CPU determines that FB1 is the main FB during the N-1 frame image rendering process, then the currently processed FB is FB1, and the CPU performs super-resolution rendering on FB1.
  • the CPU can send a rendering command to the GPU, and the GPU will render the current FB; if the rendering mode is delayed rendering and the currently processed FB is FB0, the CPU can send a rendering command to the GPU Send a rendering command, and the GPU will render the current FB.
  • the CPU reduces the initial resolution of the image to a low resolution based on the super-resolution factor; if the rendering method is delayed rendering and the FB is not FB0 (such as FB1 and FB2), The CPU reduces the initial resolution of the image to a lower resolution based on the overresolution factor.
  • the CPU can reduce the resolution of the image by reducing the width and height of the image.
  • the CPU can refer to the super-resolution multiple of the AI super-resolution model.
  • the AI super-resolution model can convert the image data from low resolution to high resolution, and the high resolution is greater than the low resolution.
  • the high resolution can be one of 2 times, 3 times and 4 times that of the low resolution, that is, the super resolution ratio of the AI super-resolution model is one of 2 times, 3 times and 4 times.
  • a 540P image is rendered with a 2x super-resolution to obtain a 1080P image, and a 4x super-resolution rendered to obtain a 2k image.
  • the CPU may reduce the resolution of the image according to the super-resolution factor of the AI super-resolution model.
  • the CPU can reduce the resolution of the image through a zoom factor. Reducing the resolution of the image can be achieved by reducing the width and height of the image.
  • the initial resolution of an image is 1080P, expressed as 1920 ⁇ 1080 in width and height
  • the super-resolution multiple of the AI super-resolution model is 2
  • the AI super-resolution model can complete the conversion from 540P to 1080P
  • the width and height of a 540P image The height is 960 ⁇ 540, indicating that the 1080P image is reduced by 2 times
  • the corresponding zoom factor is 0.5
  • the super-resolution factor of the AI super-resolution model is 4
  • the AI super-resolution model can complete the conversion from 270P to 1080P, and reduce the 1080P image by 4 times
  • the corresponding zoom factor is 0.25.
  • the scaling factor refers to the scaling of the side length of the image.
  • a scaling factor of 0.5 means that the width and height of the image are scaled by 0.5 at the same time. When the image and the width and height are scaled by 0.5 at the same time, the pixels of the image are scaled by 0.25.
  • the super-resolution multiple is used to indicate the super-resolution capability of the NPU, indicating the multiple of the NPU to increase the resolution.
  • the CPU can determine the super-resolution multiple during the initialization process of the AI super-resolution model; if the CPU does not determine the super-resolution multiple during the initialization process, the CPU can Send the high resolution and low resolution of the image to the NPU, that is, the resolution before reduction and the resolution after reduction, and the NPU selects the matching super-resolution multiple based on the resolution before reduction and the resolution after reduction.
  • the CPU notifies the GPU to generate low-resolution image data.
  • the superresolution ratio of the NPU can be a fixed multiple, and the resolution of the image required by each application program in the CPU can also be a fixed multiple, for example, the superresolution ratio of the NPU is 2 times, and the image resolution required by each application program If the resolution is 1080P, then the CPU can reduce the resolution to 540P when reducing the resolution. Because the superresolution and image resolution are fixed, the GPU knows that the resolution can be reduced to 540P, so when the CPU notifies the GPU to generate low-resolution image data, the notification does not need to carry the low-resolution.
  • the superresolution ratio of the NPU can be a fixed ratio, but the resolution of the image required by each application in the CPU is not fixed.
  • the resolution of the image required by some applications is 1080P, and the resolution of the image required by some applications is If the resolution of the CPU is 2k, the lower resolution of the CPU may be different, and the notification sent by the CPU to the GPU may carry the lower resolution.
  • the super-resolution ratio of the NPU may not be fixed, that is, there are multiple super-resolution ratios of the NPU, and the CPU may select one of the super-resolution ratios to reduce the resolution, and the corresponding notification sent by the CPU to the GPU may carry a low resolution.
  • the CPU can also send a notification to the NPU, and the notification can carry the resolution of the image of the Nth frame before and after reduction, such as step S107'.
  • Step S107' is an optional step, such as executing when the super-resolution ratio of the NPU is not fixed.
  • the CPU informs the NPU of the high resolution and low resolution of the Nth frame image, and the NPU can learn the resolution change of the Nth frame image through the high resolution and low resolution, and use the AI super-resolution model in the NPU
  • the AI super-resolution module can determine the super-resolution multiple to restore the image data that matches the high resolution of the Nth frame image.
  • the NPU can also restore high-resolution image data with a resolution greater than that of the Nth frame image.
  • the GPU generates low-resolution image data.
  • the GPU generates low-resolution RGB image data, and the RGB image data includes R channel data, G channel data, and B channel data.
  • the GPU sends the low-resolution image data to the NPU.
  • the NPU determines the super-resolution multiple of the AI super-resolution model.
  • the NPU uses the AI super-resolution model to perform super-resolution rendering on the low-resolution image data based on the super-resolution multiple to obtain high-resolution image data.
  • the super-resolution multiple of the AI super-resolution model can be used as the super-resolution multiple of the NPU.
  • the super-resolution multiple of the AI super-resolution model is a fixed multiple, and step S110' is omitted.
  • the super-resolution multiple of the AI super-resolution model may not be fixed, such as the super-resolution multiple of the AI super-resolution model includes 2 times and 4 times, and the NPU executes step S110', according to the Nth frame notified by step S107'
  • the high resolution and low resolution of the image determine the current super resolution ratio.
  • the AI super-resolution model can take low-resolution image data as input, high-resolution image data as output, and obtain through training on historical low-resolution image data, and during the training process
  • the high-resolution image rendered by the GPU is used as a reference to verify the AI super-resolution model.
  • Low resolution and high resolution are relative terms, and high resolution and low resolution can satisfy: high resolution is greater than low resolution, for example, low resolution is 540P, high resolution is 1080P; another example is low resolution It is 720P, and the high resolution is 2k.
  • the AI super-resolution model can be trained offline using training frameworks such as TensorFlow and PyTorch.
  • training frameworks such as TensorFlow and PyTorch.
  • some electronic devices do not support training frameworks such as TensorFlow and PyTorch.
  • the model format supported by mobile phones such as Qualcomm SNPE framework and NNAPI (Android Neural Networks API) is different from TensorFlow and PyTorch.
  • the model format supported by Qualcomm SNPE framework is DLC format;
  • the model format supported by NNAPI is TensorFlow Lite. If you use TensorFlow, PyTorch and other training frameworks to train the AI super-resolution model offline, you can convert the AI super-resolution model in the format of TensorFlow, PyTorch and other formats to DLC or TensorFlow Lite. Used in devices such as mobile phones.
  • the AI super-resolution model can be updated to adapt to image changes.
  • the electronic device can obtain the super-resolution rendering effect of the AI super-resolution model, and the AI super-resolution can be performed according to the super-resolution rendering effect.
  • the parameters of the rate model are adjusted, and the parameter adjustment can be offline adjustment or online adjustment.
  • the device that adjusts the parameters can be an electronic device that uses the AI super-resolution model for super-resolution rendering, or it can be other electronic devices.
  • Electronic equipment such as a computer, adjusts the parameters of the AI super-resolution model, and the computer sends the adjustment to the mobile phone for use.
  • the electronic device can collect data samples to obtain a data sample set, and use the data samples in the data sample set to adjust the AI super-resolution model.
  • the NPU sends high-resolution image data to the GPU.
  • the GPU performs rendering processing on the currently processed FB based on the high-resolution image data to obtain a rendering result, such as image data of elements in the Nth frame of image.
  • the GPU obtains the rendering results of all the FBs, based on the rendering results, the Nth frame of image is displayed on the display screen.
  • the above-mentioned rendering methods can be used in various scenes of GPU rendering images, such as game applications, home design applications, modeling applications, augmented reality applications, and virtual display applications, etc.
  • the super-resolution rendering of the AI super-resolution model can use the computing power of the NPU to share a part of the calculation of the GPU, so that the power consumption of electronic devices is lower and the rendering time is shorter.
  • the image data interaction between the GPU and the NPU can be completed by means of the GPU memory and the CPU memory.
  • the GPU memory can be the private memory of the GPU, and the CPU and the NPU are prohibited from accessing the GPU memory.
  • Image data is read from the GPU memory, and the image data read by the GPU can be written into the CPU memory or used when the GPU draws images, so the GPU memory can be used as the interactive memory and computing memory of the GPU.
  • the CPU can designate a storage space of the CPU memory (such as the first area) as the NPU input memory and the NPU output memory.
  • the NPU input memory is the input data used to write to the NPU for super-resolution rendering, and the NPU output memory is written to the NPU.
  • the output data when super-resolution rendering is completed, the NPU input memory and the NPU output memory can be the same area in the CPU memory. space taken up.
  • Either GPU or NPU, GPU and NPU can access CPU memory.
  • the corresponding memory mutual access process is shown in Figure 12, and may include the following steps:
  • the CPU designates the first area in the CPU memory as the NPU input memory and the NPU output memory, wherein the NPU input memory and the NPU output memory may be the first area, used as the NPU input memory when inputting data to the NPU, and used as the NPU output memory
  • the size of the first area can be determined according to the amount of data processed by the NPU, and will not be described here.
  • the CPU notifies the NPU of the pointer address of the first area.
  • the NPU can read image data from the NPU input memory based on the pointer address, and write image data into the NPU output memory based on the pointer address.
  • the GPU writes the low-resolution image data into the GPU memory.
  • the GPU notifies the CPU that the low-resolution image data has been written into the GPU memory.
  • the CPU notifies the GPU to write the low-resolution image data into the first area.
  • the notification sent by the CPU carries the pointer address of the first area.
  • the CPU may notify the GPU of the pointer address of the first area in step S201, that is, in step S201, the CPU notifies the NPU and the GPU of the pointer address of the first area.
  • the GPU reads low-resolution image data from the GPU memory.
  • the GPU writes the low-resolution image data into the first area based on the pointer address.
  • the CPU notifies the NPU to read low-resolution image data from the first area.
  • the NPU reads low-resolution image data from the first area based on the pointer address.
  • the NPU writes high-resolution image data into the first area based on the pointer address.
  • the NPU After the NPU reads the low-resolution image data, it inputs the low-resolution image data into the AI super-resolution model running on the NPU, and the AI super-resolution model performs super-resolution on the low-resolution image data Rendering, output high-resolution image data.
  • the NPU then writes high-resolution image data into the first area based on the pointer address.
  • the CPU notifies the GPU to read high-resolution image data from the first area.
  • the GPU reads high-resolution image data from the first area based on the pointer address.
  • the GPU writes the high-resolution image data into the GPU memory.
  • Figure 13 shows another sequence diagram of the rendering method, which may include the following steps:
  • the CPU determines that FB1 is the main FB during the rendering process of the N-1th frame image.
  • the CPU intercepts the rendering command of the image of the Nth frame issued by the application program.
  • the CPU acquires that the initial resolution of the image of the Nth frame is high resolution and the currently processed FB.
  • the CPU performs runtime check, model loading, and model compilation on the AI super-resolution model.
  • the CPU designates the first area in the CPU memory as the NPU input memory and the NPU output memory, completes the memory configuration for the AI super-resolution model, and completes the initialization of the AI super-resolution model through steps S304 and S305.
  • the CPU notifies the NPU of the pointer address of the first area.
  • the CPU obtains the rendering mode from the configuration file of the application program.
  • the CPU reduces the initial resolution of the image to a low resolution based on the super-resolution factor; if the rendering method is delayed rendering and the FB is not FB0 (such as FB1 and FB2), The CPU reduces the initial resolution of the image to a lower resolution based on the overresolution factor.
  • the CPU notifies the GPU to generate low-resolution image data.
  • the CPU notifies the NPU of the high resolution and low resolution of the Nth frame image.
  • the GPU generates low-resolution image data.
  • the GPU writes the low-resolution image data into the GPU memory.
  • the GPU notifies the CPU that the low-resolution image data has been written into the GPU memory.
  • the CPU notifies the GPU to write the low-resolution image data into the first area.
  • the notification sent by the CPU carries the pointer address of the first area.
  • the GPU reads low-resolution image data from the GPU memory.
  • the GPU writes the low-resolution image data into the first area based on the pointer address.
  • the CPU notifies the NPU to read low-resolution image data from the first area.
  • the NPU reads low-resolution image data from the first area based on the pointer address.
  • the NPU determines the super-resolution multiple of the AI super-resolution model.
  • the NPU performs super-resolution rendering on the low-resolution image data based on the super-resolution multiple to obtain high-resolution image data.
  • the NPU writes high-resolution image data into the first area based on the pointer address.
  • the CPU notifies the GPU to read high-resolution image data from the first area.
  • the GPU reads high-resolution image data from the first area.
  • the GPU writes the high-resolution image data into the GPU memory.
  • the GPU renders the currently processed FB based on the high-resolution image data, and obtains a rendering result, such as image data of elements in the Nth frame of image.
  • the GPU After obtaining the rendering results of all the FBs, the GPU displays the image of the Nth frame on the display screen based on the rendering results.
  • the CPU specifies an area from the CPU memory as the NPU input memory and another area as the NPU output memory
  • the CPU sends the pointer address of the NPU input memory and the pointer address of the NPU output memory to the NPU, and the NPU is based on the pointer of the NPU input memory
  • the address reads low-resolution image data, and reads high-resolution image data based on the pointer address of the NPU output memory.
  • the same CPU sends the pointer address of the NPU input memory and the pointer address of the NPU output memory to the GPU.
  • the GPU writes low-resolution image data based on the pointer address of the NPU input memory, and reads high-resolution image data based on the pointer address of the NPU output memory. image data.
  • GPU and NPU can interact with GPU memory and CPU memory. There is a problem that image data is copied between GPU memory and CPU memory, which prolongs the processing time of GPU and NPU. The length of time to fetch data, thus increasing rendering time.
  • GPU, CPU and NPU can carry out data interaction in the form of shared memory, wherein shared memory refers to the memory that GPU, CPU and NPU can visit, and shared memory can be used as the external cache memory of GPU, at least stores GPU The low-resolution image data input to the NPU, and the shared memory can be used as a processing cache of the NPU to at least store the high-resolution image data output by the NPU.
  • shared memory refers to the memory that GPU, CPU and NPU can visit
  • shared memory can be used as the external cache memory of GPU, at least stores GPU
  • the low-resolution image data input to the NPU, and the shared memory can be used as a processing cache of the NPU to at least store the high-resolution image data output by the NPU.
  • FIG. 14 The memory mutual access process of GPU and NPU using shared memory is shown in Figure 14, which may include the following steps:
  • the CPU applies for a shared memory from a hardware buffer (Hardware Buffer).
  • the CPU can apply for shared memory from the Hardware Buffer of Random Access Memory (RAM).
  • RAM Random Access Memory
  • One way to apply for shared memory is that the CPU applies for the storage space of the GPU SSBO (Shader Storage Buffer Object) from the Hardware Buffer.
  • Hardware Buffer is a kind of memory.
  • CPU, GPU and NPU access Hardware Buffer, there are certain requirements for the format of Hardware Buffer.
  • the format of Hardware Buffer For example, when applying for shared memory as a storage space accessible by Fragment Shader in GPU or Compute Shader in GPU, You can specify the format of the Hardware Buffer as AHARDWAREBUFFER_FORMAT_BLOB.
  • the CPU can apply for two pieces of shared memory from the Hardware Buffer, one piece of shared memory is used to store the low-resolution image data obtained by the GPU as the input data of the NPU; the other piece of shared memory is used to store The output data of the NPU, such as the high-resolution image data obtained by super-resolution rendering, is used as the input data of the GPU when rendering the UI.
  • the format of the two pieces of shared memory requested by the CPU is the same, for example, the format of the shared memory can be AHARDWAREBUFFER_FORMAT_BLOB.
  • the size of the shared memory requested can be determined according to the resolution of the image. For example, the size of the shared memory requested is width ⁇ height ⁇ byte size.
  • the type of image data is float type, and the float variable is 4 bytes.
  • the size is width ⁇ height ⁇ 3 ⁇ 4, and the width and height are determined by the resolution of the image. For example, for a 1080P image, the requested shared memory size is 1920 ⁇ 1080 ⁇ 3 ⁇ 4.
  • the CPU notifies the GPU of the pointer address of the shared memory, so as to use the shared memory as a storage memory for the low-resolution image data.
  • the CPU notifies the NPU of the pointer address of the shared memory, so as to use the shared memory as a storage memory for high-resolution image data.
  • the GPU writes the low-resolution image data into the shared memory based on the pointer address of the shared memory.
  • the CPU notifies the NPU to read the low-resolution image data from the shared memory.
  • the NPU reads the low-resolution image data from the shared memory based on the pointer address of the shared memory.
  • the NPU writes high-resolution image data into the shared memory based on the pointer address of the shared memory.
  • the CPU notifies the GPU to read high-resolution image data from the shared memory.
  • the GPU reads high-resolution image data from the shared memory based on the pointer address of the shared memory.
  • the CPU applies for two pieces of shared memory from the Hardware Buffer, one piece of shared memory is used to store the low-resolution image data obtained by the GPU as the input data of the NPU; the other piece of shared memory is used to store the output data of the NPU, such as super-resolution
  • the rendered high-resolution image data is used as the input data of the GPU when rendering the UI.
  • the CPU can send the pointer addresses of the two shared memories to the GPU and the NPU, so that the GPU and the NPU can complete the reading and writing of image data in the shared memory.
  • FIG 14 shows another sequence diagram of the rendering method, which may include the following steps:
  • the CPU determines that FB1 is the main FB during the rendering process of the N-1 frame image.
  • the CPU intercepts the rendering command of the image of the Nth frame issued by the application program.
  • the CPU acquires that the initial resolution of the image of the Nth frame is high resolution and the currently processed FB.
  • the CPU performs runtime check, model loading, and model compilation on the AI super-resolution model.
  • the CPU applies for a shared memory from the Hardware Buffer of the RAM.
  • the shared memory is used as a storage space for low-resolution image data and high-resolution image data.
  • Pass the steps S504 and step S505 complete the initialization of the AI super-resolution model.
  • the CPU notifies the GPU and the NPU of the pointer address of the shared memory.
  • the CPU obtains the rendering mode from the configuration file of the application program.
  • the CPU reduces the initial resolution of the image to a low resolution based on the super-resolution factor; if the rendering method is delayed rendering and the FB is not FB0 (such as FB1 and FB2), The CPU reduces the initial resolution of the image to a lower resolution based on the overresolution factor.
  • the CPU notifies the GPU to generate low-resolution image data.
  • the CPU notifies the NPU of the high resolution and low resolution of the Nth frame image.
  • the GPU generates low-resolution image data.
  • the GPU writes the low-resolution image data into the shared memory based on the pointer address of the shared memory.
  • the CPU notifies the NPU to read the low-resolution image data from the shared memory.
  • the NPU reads the low-resolution image data from the shared memory based on the pointer address of the shared memory.
  • the NPU determines the super-resolution multiple of the AI super-resolution model.
  • the NPU performs super-resolution rendering on the low-resolution image data based on the super-resolution multiple to obtain high-resolution image data.
  • the NPU writes the high-resolution image data into the shared memory based on the pointer address of the shared memory.
  • the CPU notifies the GPU to read high-resolution image data from the shared memory.
  • the GPU reads high-resolution image data from the shared memory based on the pointer address of the shared memory.
  • the GPU Based on the high-resolution image data, the GPU performs rendering processing on the currently processed FB to obtain a rendering result, such as image data of elements in the Nth frame of image.
  • the GPU After obtaining the rendering results of all the FBs, the GPU displays the image of the Nth frame on the display screen based on the rendering results.
  • Some embodiments of the present application provide an electronic device, and the electronic device includes: a first processor, a second processor, and a memory; wherein, the memory is used to store one or more computer program codes, and the computer program codes include computer instructions.
  • the first processor and the second processor execute the computer instructions
  • the first processor and the second processor execute the above rendering method.
  • Some embodiments of the present application provide a chip system, the chip system includes program code, and when the program code is run on the electronic device, the first processor and the second processor in the electronic device execute the above rendering method.
  • Some embodiments of the present application provide a processor, the processor is a second processor, and the second processor includes a processing unit and a memory; wherein, the memory is used to store one or more computer program codes, and the computer program codes include computer instructions , when the second processor executes the computer instructions, the second processor executes the above rendering method.
  • Some embodiments of the present application provide a computer storage medium, the computer storage medium includes computer instructions, and when the computer instructions are run on the electronic device, the second processor in the electronic device executes the above rendering method.
  • control device includes one or more processors, memory, the memory is used to store one or more computer program codes, the computer program codes include computer instructions, when the When the one or more processors execute the computer instructions, the control device performs the method described above.
  • the control device can be an integrated circuit IC, or a system-on-chip SOC.
  • the integrated circuit can be a general integrated circuit, a field programmable gate array FPGA, or an application specific integrated circuit ASIC.
  • the disclosed system, device and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of this embodiment may be integrated into one processing unit, or each unit may physically exist separately, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium
  • several instructions are included to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method described in each embodiment.
  • the aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk, and other various media capable of storing program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Processing (AREA)

Abstract

一种渲染方法及装置，其中第一处理器接收应用程序下发的渲染命令，渲染命令用于指示第二处理器基于第一分辨率对第一图像进行渲染；第一处理器向第二处理器发送渲染指令，渲染指令用于指示第二处理器对第一图像进行渲染；第二处理器基于渲染指令，生成第一图像在第二分辨率下的图像数据，第二分辨率不大于第一分辨率；第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存；第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据，第三分辨率大于第二分辨率；第二处理器基于第一图像在第三分辨率下的图像数据，生成第一图像。

Description

渲染方法及装置
本申请要求于2021年11月17日提交中国国家知识产权局、申请号为202111364418.7、发明名称为“渲染方法及装置”的中国专利申请的优先权，要求于2021年12月17日提交中国国家知识产权局、申请号为202111554611.7、发明名称为“渲染方法及装置”的中国专利申请的优先权，全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种渲染方法及装置。
背景技术
随着显示技术的发展，图像的分辨率向更高分辨率发展，如图像的分辨率从720P发展到1080P，又从1080P发展到2k，其中P表示像素总行数，如720P表示有720行像素；k表示像素总列数，如2k表示有2000列像素。电子设备在渲染高分辨率图像或超高分辨率图像时可以占用过多的算力开销，使电子设备的功耗提升以及出现发热严重，甚至出现运行卡顿的问题。
发明内容
本申请提供的渲染方法及装置，解决了电子设备渲染图像时功耗提升、设备发热严重以及运行卡顿的问题。
为达到上述目的，本申请采用如下技术方案：
第一方面，本申请提供一种渲染方法，方法应用于电子设备中，电子设备运行有应用程序，电子设备包括第一处理器和第二处理器，方法包括：第一处理器接收应用程序下发的渲染命令，渲染命令用于指示第二处理器基于第一分辨率对第一图像进行渲染；第一处理器向第二处理器发送渲染指令，渲染指令用于指示第二处理器对第一图像进行渲染；第二处理器基于渲染指令，生成第一图像在第二分辨率下的图像数据，第二分辨率不大于第一分辨率；第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存；第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据，第三分辨率大于第二分辨率；第二处理器基于第一图像在第三分辨率下的图像数据，生成第一图像。第二处理器将生成的第一图像在第二分辨率下的图像数据写入到第一内存后，第二处理器可以从第一内存中读取第一图像在第三分辨率下的图像数据，省去第二处理器生成第一图像在第三分辨率下的图像数据，节省了第二处理器的算力，缩短渲染过程所花费的时间，从而降低功耗和提高渲染流畅度，解决设备发热严重和运行卡顿的问题。并且第三分辨率大于第二分辨率，说明第二处理器可以得到一个分辨率相对较高的图像数据，第二处理器可基于分辨率相对较高的图像数据，绘制第一图像，提高第一图像的图像质量。
可选的，若应用程序的渲染方式为前向渲染，渲染指令对应于第一帧缓冲，第一帧缓冲执行绘制指令的数量大于预设阈值；若应用程序的渲染方式为延迟渲染，渲染指令对应于应用程序下发的所有帧缓冲中除最后一个帧缓冲之外的帧缓冲。在应用程序采用的渲染 方式不同时，第一处理器向第二处理器发送的渲染指令针对的帧缓冲也不同，说明应用程序采用不同渲染方式时，第一处理器发送渲染指令的时机也不同，实现在不同渲染方式下基于帧缓冲控制渲染指令的发送。
可选的，第一帧缓冲是所有帧缓冲中执行绘制指令的数量最多的帧缓冲。
可选的，在第一处理器向第二处理器发送渲染指令之前，方法还包括：第一处理器从应用程序的配置文件中获取应用程序的渲染方式。
可选的，渲染指令用于指示第二处理器基于第二分辨率对第一图像进行渲染，第二分辨率小于第一分辨率。第一处理器向第二处理器发送的渲染指令中携带第一图像的第二分辨率，指定第二处理器生成的图像数据的分辨率，防止第二处理器生成与应用程序要求的分辨率不匹配的图像数据，提高第一图像的准确度。并且第二分辨率小于第一分辨率，分辨率越小数据量越小，从而通过指定小于第一分辨率的第二分辨率，使第二处理器处理的数据量降低，降低电子设备的功耗，解决了设备发热严重的问题。
可选的，第二分辨率小于第一分辨率；第三分辨率与第一分辨率相同，或者，第三分辨率大于第一分辨率。第二分辨率小于第一分辨率，第三分辨率与第一分辨率相同。在第二处理器处理过程中，因为第二分辨率小于第一分辨率，所以第二处理器生成第一图像在第二分辨率下的图像数据时，第二处理器处理的数据量降低。但是第三分辨率与第一分辨率相同，说明第二处理器可以读取到对应第一分辨率的图像数据，绘制出具有第一分辨率的第一图像，保证绘制出的第一图像的图像质量与应用程序要求的图像质量质量。如果第三分辨率大于第一分辨率，第二处理器可以读取到分辨率大于第一分辨率的图像数据，绘制出的第一图像的图像质量优于应用程序要求的图像质量，提高第一图像的图像质量。
可选的，第二分辨率等于第一分辨率。第二分辨率等于第一分辨率，但第三分辨率大于第二分辨率，那么第三分辨率也大于第一分辨率，在应用程序要求基于第一分辨率渲染第一图像时，第二处理器可以基于第三分辨率渲染第一图像，绘制出的第一图像的图像质量优于应用程序要求的图像质量，提高第一图像的图像质量。
可选的，电子设备还包括第三处理器，第一图像在第三分辨率下的图像数据由第三处理器生成。在第二处理器生成第一图像在第二分辨率下的图像数据后，由第三处理器生成第一图像在第三分辨率下的图像数据，实现主动调用第三处理器的算力，满足满足高分辨率图像和/或超高分辨率图像的算力要求，降低电子设备的功耗，减少出现发热严重的现象。并且第三处理器可以分担第二处理器的计算量，缩短渲染过程所花费的时间，从而提高渲染流畅度，解决运行卡顿的问题。
可选的，第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存之后，方法还包括：第三处理器从第一内存中读取第一图像在第二分辨率下的图像数据；第三处理器基于第一图像在第二分辨率下的图像数据，生成第一图像在第三分辨率下的图像数据；第三处理器将第一图像在第三分辨率下的图像数据写入到第一内存。
可选的，第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存之前，方法还包括：第二处理器将第一图像在第二分辨率下的图像数据写入第二内存，第二处理器具有访问第二内存和第一内存的权限，第三处理器具有访问第一内存的权限；第二处理器向第一处理器发送第一通知，第一通知用于指示第一图像在第二分辨率下的图像数据成功 写入第二内存；响应于接收第一通知，第一处理器向第二处理器发送第二通知，第二通知用于指示第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存中，第二通知中携带有第一内存的地址指针；响应于接收第二通知，第二处理器从第二内存中读取第一图像在第二分辨率下的图像数据。第一处理器可监听第二处理器的图像数据读写工作，以在第二处理器将第一图像在第二分辨率下的图像数据写入第二内存后，及时触发第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存，提高效率。并且第一处理器发送的第二通知中携带有第一内存的地址指针，第二处理器可基于第一内存的地址指针，向第一内存中写入第一图像在第二分辨率下的图像数据，提高写入的准确度。
可选的，第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存之后，第三处理器从第一内存中读取第一图像在第二分辨率下的图像数据之前，方法还包括：第一处理器向第三处理器发送第三通知，第三通知用于指示第三处理器从第一内存中读取第一图像在第二分辨率下的图像数据。在第二处理器向第一内存写入第一图像在第二分辨率下的图像数据后，第一处理器可以及时通知第三处理器读取，触发第三处理器生成第一图像在第三分辨率下的图像数据，提高效率。
可选的，第三处理器将第一图像在第三分辨率下的图像数据写入到第一内存之后，第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据之前，方法还包括：第一处理器向第二处理器发送第四通知，第四通知用于指示第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据，第四通知中携带有第一内存的地址指针。第一处理器可监听第三处理器的图像数据读写工作，在第三处理器将第一图像在第三分辨率下的图像数据写入第一内存后，及时触发第二处理器读取第一图像在第三分辨率下的图像数据，提高效率。并且第一处理器发送的第四通知中携带有第一内存的地址指针，第二处理器可基于第一内存的地址指针，从正确地址中读取第一图像在第三分辨率下的图像数据，提高读取到的图像数据的准确度。
可选的，第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据之后，方法还包括：第二处理器将第一图像在第三分辨率下的图像数据写入第二内存，第二处理器具有访问第二内存和第一内存的权限，第三处理器具有访问第一内存的权限。
可选的，电子设备还包括第三处理器，第一图像在第三分辨率下的图像数据由第三处理器生成；在第一处理器向第二处理器发送渲染指令之前，方法还包括：第一处理器从第三内存中分配出第一内存，第一处理器具备访问第三内存的权限；第一处理器向第三处理器发送第一内存的指针地址，第三处理器基于指针地址向第一内存中写入第一图像在第三分辨率下的图像数据、以及基于指针地址从第一内存中读取第一图像在第二分辨率下的图像数据。
可选的，电子设备还包括第三处理器，第一图像在第三分辨率下的图像数据由第三处理器生成；在第一处理器向第二处理器发送渲染指令之前，方法还包括：第一处理器从硬件缓冲区中分配出第一内存；第一处理器向第三处理器和第二处理器发送第一内存的指针地址，第一处理器、第二处理器和第三处理器具有访问第一内存的权限，第三处理器和第二处理器基于指针地址，对第一内存执行图像数据的读写。在本实施例中，第一处理器可以从硬件缓冲区中分配出第一内存，第二处理器在将第一图像在第二分辨率下的图像数据 写入第一内存之前，无需在其他内存中写入第一图像在第二分辨率下的图像数据，第二处理器和第三处理器可以基于第一内存进行数据共享，实现第二处理器和第三处理器之间共享数据的零拷贝，提升处理效率。
可选的，第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存之后，第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据之前，方法还包括：第三处理器从第一内存中读取第一图像在第二分辨率下的图像数据；第三处理器基于第一图像在第二分辨率下的图像数据，生成第一图像在第三分辨率下的图像数据；第三处理器将第一图像在第三分辨率下的图像数据写入到第一内存。
可选的，第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存之后，第三处理器从第一内存中读取第一图像在第二分辨率下的图像数据之前，方法还包括：第一处理器向第三处理器发送第五通知，第五通知用于指示第三处理器从第一内存中读取第一图像在第二分辨率下的图像数据。第一处理器可监听第二处理器的图像数据读写工作，在第二处理器将第一图像在第二分辨率下的图像数据写入第一内存后，及时触发第三处理器读取第一图像在第二分辨率下的图像数据，提高效率。
可选的，第三处理器将第一图像在第三分辨率下的图像数据写入到第一内存之后，第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据之前，方法还包括：第一处理器向第二处理器发送第六通知，第六通知用于指示第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据。第一处理器可监听第三处理器的图像数据读写工作，在第三处理器将第一图像在第三分辨率下的图像数据写入第一内存后，及时触发第二处理器读取第一图像在第三分辨率下的图像数据，提高效率。
可选的，电子设备还包括第三处理器，第一图像在第三分辨率下的图像数据由第三处理器生成；第三处理器中运行有人工智能超分辨率模型，第三处理器利用人工智能超分辨率模型，对第一图像在第二分辨率下的图像数据进行超分辨率渲染，生成第一图像在第三分辨率下的图像数据。
可选的，第三处理器利用人工智能超分辨率模型，对第一图像在第二分辨率下的图像数据进行超分辨率渲染之前，方法还包括：第一处理器向第三处理器发送第一分辨率和第二分辨率；第三处理器基于第一分辨率和第二分辨率，确定人工智能超分辨率模型的超分倍数，人工智能分辨率模型基于超分倍数，对第一图像在第二分辨率下的图像数据进行超分辨率渲染。人工智能超分辨率模型可具有至少一个超分倍数，每个超分倍数可以完成不同分辨率的转换。例如以同一个分辨率下的图像数据为输入，不同超分倍数使得人工智能超分辨率模型输出的图像数据对应的分辨率不同。为了使得人工智能超分辨率模型输出的图像数据对应的分辨率不小于第一分辨率，第一处理器可以向第三处理器发送第一分辨率和第二分辨率。第三处理器根据第一分辨率和第二分辨率之间的倍数差异，选择一个不小于倍数差异的超分倍数，从而可以使工智能超分辨率模型输出的图像数据对应的分辨率不小于第一分辨率。
可选的，第三处理器利用人工智能超分辨率模型，对第一图像在第二分辨率下的图像数据进行超分辨率渲染之前，方法还包括：第一处理器对人工智能超分辨率模型进行初始化，初始化用于确定运行人工智能超分辨率模型以及确定人工智能超分辨率模型的正常运 行；初始化包括运行时检测、模型加载、模型编译以及内存配置，运行时检查用于确定运行人工智能超分辨率模型，模型加载、模型编译以及内存配置用于确定人工智能超分辨率模型的正常运行。
可选的，在第一处理器向第二处理器发送渲染指令之前，方法还包括：第一处理器将第一图像的分辨率从第一分辨率降低至第二分辨率。
可选的于，电子设备还包括第三处理器，第一图像在第三分辨率下的图像数据由第三处理器生成；第三处理器具有超分倍数，超分倍数用于指示第二分辨率和第三分辨率之间的差异；第三分辨率和第一分辨率相同，第一处理器将第一图像的分辨率从第一分辨率降低至第二分辨率包括：第一处理器基于超分倍数，将第一图像的分辨率从第一分辨率降低至第二分辨率。人工智能超分辨率模型可具有至少一个超分倍数，每个超分倍数可以完成不同分辨率的转换。例如以同一个分辨率下的图像数据为输入，不同超分倍数使得人工智能超分辨率模型输出的图像数据对应的分辨率不同，因此第一处理器可以基于超分倍数，对第一图像完成分辨率的降低，使分辨率的降低与人工智能超分辨率模型的一个超分倍数匹配，后续可以利用人工智能超分辨率模型完成分辨率的提升，保证准确度。
可选的，第三处理器是神经网络处理器或数字信号处理器。
第二方面，本申请提供一种渲染方法，方法应用于电子设备的第二处理器中，电子设备运行有应用程序，电子设备还包括第一处理器，应用程序向第一处理器下发渲染命令，渲染命令用于指示第二处理器基于第一分辨率对第一图像进行渲染，方法包括：第二处理器接收第一处理器发送的渲染指令，渲染指令用于指示第二处理器对第一图像进行渲染；第二处理器基于渲染指令，生成第一图像在第二分辨率下的图像数据，第二分辨率不大于第一分辨率；第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存；第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据，第三分辨率大于第二分辨率；第二处理器基于第一图像在第三分辨率下的图像数据，生成第一图像。第二处理器将生成的第一图像在第二分辨率下的图像数据写入到第一内存后，第二处理器可以从第一内存中读取第一图像在第三分辨率下的图像数据，省去第二处理器生成第一图像在第三分辨率下的图像数据，节省了第二处理器的算力，缩短渲染过程所花费的时间，从而降低功耗和提高渲染流畅度，解决设备发热严重和运行卡顿的问题。并且第三分辨率大于第二分辨率，说明第二处理器可以得到一个分辨率相对较高的图像数据，第二处理器可基于分辨率相对较高的图像数据，绘制第一图像，提高第一图像的图像质量。
可选的，第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存之前，方法还包括：第二处理器将第一图像在第二分辨率下的图像数据写入第二内存，第二处理器具有访问第二内存和第一内存的权限，第三处理器具有访问第一内存的权限；第二处理器向第一处理器发送第一通知，第一通知用于指示第一图像在第二分辨率下的图像数据成功写入第二内存；响应于第一处理器发送的第二通知，第二处理器从第二内存中读取第一图像在第二分辨率下的图像数据，第二通知在第一处理器接收到第一通知后发送，第二通知用于指示第二处理器将第一图像在第二分辨率下的图像数据写入到第一内存中，第二通知中携带有第一内存的地址指针。第一处理器可监听第二处理器的图像数据读写工作，以在第二处理器将第一图像在第二分辨率下的图像数据写入第二内存后，及时触发第二处理器 将第一图像在第二分辨率下的图像数据写入到第一内存，提高效率。并且第一处理器发送的第二通知中携带有第一内存的地址指针，第二处理器可基于第一内存的地址指针，向第一内存中写入第一图像在第二分辨率下的图像数据，提高写入的准确度。
可选的，第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据之后，方法还包括：第二处理器将第一图像在第三分辨率下的图像数据写入第二内存，第二处理器具有访问第二内存和第一内存的权限；第一图像在第三分辨率下的图像数据由电子设备中的第三处理器生成，第三处理器具有访问第一内存的权限。
可选的，第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据之前，方法还包括：第二处理器接收第一处理器发送的通知，通知用于指示第二处理器从第一内存中读取第一图像在第三分辨率下的图像数据，通知中携带有第一内存的地址指针。第二处理器可基于第一内存的地址指针，从正确地址中读取第一图像在第三分辨率下的图像数据，提高读取到的图像数据的准确度。
第三方面，本申请提供一种电子设备，电子设备包括：第一处理器、第二处理器和存储器；其中，存储器用于存储一个或多个计算机程序代码，计算机程序代码包括计算机指令，当第一处理器和第二处理器执行计算机指令时，第一处理器和第二处理器执行上述渲染方法。
第四方面，本申请提供一种芯片系统，芯片系统包括程序代码，当程序代码在电子设备上运行时，使得电子设备中的第一处理器和第二处理器执行上述渲染方法。
第五方面，本申请提供一种处理器，处理器为第二处理器，第二处理器包括处理单元和存储器；其中，存储器用于存储一个或多个计算机程序代码，计算机程序代码包括计算机指令，当第二处理器执行计算机指令时，第二处理器执行上述渲染方法。
第六方面，本申请提供一种计算机存储介质，计算机存储介质包括计算机指令，当计算机指令在电子设备上运行时，使得电子设备中的第二处理器执行上述渲染方法。
第七方面，本申请提供一种渲染方法，应用于电子设备对第一图像的渲染处理，电子设备运行有应用程序，电子设备包括第一处理器、第二处理器和第三处理器，方法包括：第一处理器接收应用程序下发的渲染命令，渲染命令用于指示对第一图像进行渲染，第一图像具有第二分辨率；第三处理器获得第一图像数据，第一图像数据是具有第一分辨率的第一图像的图像数据；第三处理器对第一图像数据进行超分辨率渲染，得到第二图像数据，第二图像数据是具有第二分辨率的第一图像的图像数据，第一分辨率小于第二分辨率；第二处理器基于第二图像数据生成具有第二分辨率的第一图像。
可选的，方法还包括：第一处理器响应渲染命令，确定第一图像的分辨率为第一分辨率，指示第三处理器获得第一图像数据。
可选的，方法还包括：第一处理器响应渲染命令，确定第一图像的分辨率为第二分辨率；第一处理器将第一图像的分辨率从第二分辨率降低至第一分辨率；第一处理器指示第三处理器获得第一图像数据。
可选的，方法还包括：第一处理器根据第三处理器的超分倍数，从缩放范围中选择与超分倍数相关的缩放倍数，超分倍数用于指示第一分辨率和第二分辨率之间的倍数关系；第一处理器将第一图像的分辨率从第二分辨率降低至第一分辨率包括：第一处理器利用缩 放倍数对第一图像的分辨率进行降低处理，使第一图像的分辨率从第二分辨率降低至第一分辨率，缩放倍数和超分倍数的关系是缩放倍数和超分倍数的相乘结果为1。
可选的，第三处理器中运行AI超分辨率模型，第三处理器对第一图像数据进行超分辨率渲染，得到第二图像数据包括：第三处理器利用AI超分辨率模型，对第一图像数据进行超分辨率渲染。
可选的，方法还包括：第一处理器对AI超分辨率模型进行初始化，初始化用于确定运行AI超分辨率模型以及确定AI超分辨率模型可正常运行；初始化包括运行时检测、模型加载、模型编译以及内存配置，运行时检查用于确定运行AI超分辨率模型，模型加载、模型编译以及内存配置用于确定AI超分辨率模型可正常运行。
可选的，电子设备对第一图像执行渲染处理时调用一个或多个帧缓冲，方法还包括：第一处理器在确定第一图像的分辨率为第二分辨率、第一图像的渲染方式为延迟渲染且当前处理的帧缓冲不是最后一个帧缓冲，指示第三处理器获得第一图像数据，以对当前处理的帧缓冲进行渲染处理。
可选的，方法还包括：第一处理器在确定第一图像的分辨率为第二分辨率、第一图像的渲染方式为延迟渲染且当前处理的帧缓冲是最后一个帧缓冲，指示第二处理器获得第四图像数据，以对当前处理的帧缓冲进行渲染处理，第四图像数据是具有第二分辨率的第一图像的图像数据。
可选的，电子设备对第一图像执行渲染处理时调用一个或多个帧缓冲，方法还包括：第一处理器确定对第一图像执行渲染处理过程中的主帧缓冲；主帧缓冲是电子设备对第一图像的渲染处理过程中，执行渲染操作数量最多的帧缓冲；第一处理器在确定第一图像的分辨率为第二分辨率、第一图像的渲染方式为前向渲染且当前处理的帧缓冲为主帧缓冲时，指示第三处理器获得第一图像数据，以对当前处理的帧缓冲进行渲染处理。
可选的，方法还包括：第一处理器在确定第一图像的分辨率为第二分辨率、第一图像的渲染方式为前向渲染且当前处理的帧缓冲不是主帧缓冲时，指示第二处理器获得第四图像数据，以对当前处理的帧缓冲进行渲染处理，第四图像数据是具有第二分辨率的第一图像的图像数据。
可选的，第一处理器确定对第一图像执行渲染处理过程中的主帧缓冲包括：第一处理器基于对第二图像的渲染处理过程中，执行渲染操作数量最多的帧缓冲，确定主帧缓冲；第二图像的渲染处理在第一图像的渲染处理之前。
可选的，在确定对第一图像执行渲染处理过程中的主帧缓冲之前，方法还包括：在对第二图像的渲染处理过程中，确定在每个帧缓冲上执行绘制指令的数量；将执行绘制指令的数量最多的帧缓冲确定为主帧缓冲。
可选的，第二图像是第一图像的上一帧图像。
可选的，第一处理器为中央处理器，第二处理器为图形处理器，第三处理器是神经网络处理器或数字信号处理器。
应当理解的是，本申请中对技术特征、技术方案、有益效果或类似语言的描述并不是暗示在任意的单个实施例中可以实现所有的特点和优点。相反，可以理解的是对于特征或有益效果的描述意味着在至少一个实施例中包括特定的技术特征、技术方案或有益效果。 因此，本说明书中对于技术特征、技术方案或有益效果的描述并不一定是指相同的实施例。进而，还可以任何适当的方式组合本实施例中所描述的技术特征、技术方案和有益效果。本领域技术人员将会理解，无需特定实施例的一个或多个特定的技术特征、技术方案或有益效果即可实现实施例。在其他实施例中，还可在没有体现所有实施例的特定实施例中识别出额外的技术特征和有益效果。
附图说明
图1为本申请提供的一种渲染图像的示意图；
图2为本申请提供的另一种渲染图像的示意图；
图3为本申请提供的一种分辨率提升设置的界面示意图；
图4为本申请提供的再一种渲染图像的示意图；
图5为本申请提供的再一种渲染图像的示意图；
图6为本申请提供的一种不同帧缓冲上drawcall数量的确定示意图；
图7为本申请提供的另一种不同帧缓冲上drawcall数量的确定示意图；
图8为本申请提供的再一种渲染图像的示意图；
图9为本申请提供的一种渲染方法的示意图；
图10为本申请提供的一种渲染方法的流程图；
图11为本申请提供的渲染方法的一种信令图；
图12为本申请提供的一种内存访问的示意图；
图13为本申请提供的渲染方法的另一种信令图；
图14为本申请提供的另一种内存访问的示意图；
图15为本申请提供的渲染方法的再一种信令图。
具体实施方式
本申请说明书和权利要求书及附图说明中的术语“第一”、“第二”和“第三”等是用于区别不同对象，而不是用于限定特定顺序。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
在用户使用电子设备的过程中，电子设备可以通过显示屏向用户显示一帧帧图像。以视频流为例，一个视频流可以包括多帧图像，电子设备可以在显示屏上依次显示每帧图像，实现在显示屏上显示视频流。其中图像显示可以由电子设备中的应用程序触发，应用程序可以向电子设备发出针对不同图像的渲染命令，电子设备响应渲染命令进行图像的渲染，并基于图像的渲染结果进行显示。
在一些实现中，每帧图像对应多个帧缓冲(FrameBuffer，FB)，每个FB用于存储图像的部分元素的渲染结果，如存储部分元素的图像数据，图像数据用于绘制出对应的元素。例如图像包括人物、树木等元素，FB中可以存储人物的图像数据和树木的图像数据，电子设备可以基于FB中存储的图像数据进行绘制。在本实施例中，电子设备的一种可选结构以 及基于FB进行渲染的过程如图1所示，电子设备可以包括中央处理器(Central Processing Unit，CPU)、图形处理器(Graphics Processing Unit，GPU)、内部存储器和显示屏，内部存储器也可以称为内存。
在图1中，电子设备中安装的应用程序(如游戏应用程序、视频应用程序)启动后，应用程序可以通过显示屏显示图像，在显示图像过程中，应用程序下发渲染命令，CPU可以拦截应用程序下发的渲染命令。CPU响应渲染命令，在内存中为第i帧图像的渲染处理创建对应的FB，例如在图1中CPU可以为第i帧图像创建3个FB，分别记为FB0、FB1和FB2。CPU可以根据渲染命令，向GPU下发渲染指令，GPU响应渲染指令，执行与渲染指令对应的渲染。如一种示例中，GPU响应渲染指令，对第i帧图像的FB1和FB2进行渲染处理。在完成渲染处理后，FB1和FB2中分别存储第i帧图像的图像数据，例如FB1可以用于存储第i帧图像的一部分元素的图像数据(称为图像数据1)，FB2可以用于存储第i帧图像的另一部分元素的图像数据(称为图像数据2)。在显示第i帧图像时，GPU将图像数据1以及图像数据2融合(或称为渲染)到FB0中，FB0中存储第i帧图像的完整的图像数据。GPU从FB0中读取第i帧图像的图像数据，根据第i帧图像的图像数据，在显示屏中显示出第i帧图像。
但是随着显示技术的发展，显示屏的屏幕刷新率和图像的分辨率向更高阶段发展，如显示屏的屏幕刷新率从60Hz(Hertz，赫兹)发展到90Hz，又从90Hz发展到120Hz；图像的分辨率从720P(1280×720)发展到1080P(1920×1080)，又从1080P发展到2k(2560×1440)，使得电子设备的渲染朝运行更流畅、场景更复杂和画质更高的方向发展，对电子设备的算力提出更高的需求。即便电子设备使用算力很强的芯片，电子设备以CPU和GPU的渲染方式进行渲染时仍可以占用过多的算力开销，且CPU和GPU的渲染方式难以满足快速增长的算力需求。在以CPU和GPU的渲染方式进行渲染时消耗过多的算力资源，使得电子设备的功耗提升以及出现发热严重，甚至出现运行卡顿的问题。
为了解决上述问题，本申请提供一种渲染方法，在电子设备中配置除CPU和GPU之外的其他处理器，将CPU和GPU控制渲染过程调整为CPU、GPU和其他处理器控制渲染过程，例如配置的处理器的算力可以大于GPU的算力但能耗可以小于GPU的能耗，控制能力可以弱于CPU和GPU。在CPU、GPU和其他处理器控制渲染过程时，其他处理器可以生成高分辨率图像和/或超高分辨率图像，主动调用其他处理器的算力完成超分辨率渲染，满足高分辨率图像和/或超高分辨率图像的算力要求，降低电子设备的功耗，减少出现发热严重的现象。并且其他处理器可以分担GPU的计算量，缩短渲染过程所花费的时间，从而提高渲染流畅度，解决运行卡顿的问题。其中超分辨率渲染用于对图像数据完成分辨率的提升，分辨率的提升可以是从低分辨率到高分辨率或超高分辨率的提升，也可以是从高分辨率到超高分辨率的提升。
在一些实现中，电子设备中配置神经网络处理器(Neural-network Processing Unit，NPU)，以CPU、GPU和NPU控制渲染过程，NPU可以完成超分辨率渲染，以利用NPU辅助得到高分辨率图像的图像数据或超高分辨率图像的图像数据。
在一些实现中，电子设备中配置数字信号处理器(Digital Signal Processor，DSP)，以CPU、GPU和DSP控制渲染过程，DSP可以完成超分辨率渲染。
在一些实现中，电子设备中配置NPU和DSP，以CPU、GPU、NPU和DSP控制渲染过程，NPU和DSP可以完成超分辨率渲染，其中NPU和DSP的超分辨率渲染可以对应不同分辨率。一种示例中，NPU可以对应低分辨率到高分辨率的转换，DSP可以对应低分辨率到超高分辨率的提升；另一种示例中，NPU可以对应低分辨率到超高分辨率的提升，DSP可以对应低分辨率到高分辨率的提升；再一种示例中，NPU可以对应低分辨率到高分辨率的提升，DSP可以对应高分辨率到超高分辨率的提升。
在一些实现中，NPU和DSP可以协同完成超分辨率渲染，例如NPU和DSP可以协同应对低分辨率到高分辨率的提升；又例如NPU和DSP可以协同应对低分辨率到超高分辨率的提升，其中协同应对是指一帧图像的超分辨率渲染过程中NPU和DSP共同参与。
低分辨率可以是540P(960×540)和720P，高分辨率可以是1080P，超高分辨率可以是2k甚至是大于2k，电子设备通过显示屏显示图像时，GPU可以生成540P图像的图像数据和720P图像的图像数据，NPU可以完成从540P到1080P的提升，也可以完成从720P到1080P的提升，以利用NPU生成1080P图像的图像数据，此外NPU还可以生成2k甚至是分辨率大于2k的图像的图像数据。
在本实施例中，渲染方法的一种使用场景是：电子设备输出的画面质量高于应用程序要求的画面质量，其方式可以是电子设备可以提升应用程序所需图像的分辨率，在应用程序所需图像的分辨率为低分辨率时，电子设备可以将其分辨率提升到高分辨率或超高分辨率，自动改善应用程序的画面质量；又或者在应用程序所需图像的分辨率为高分辨率时，电子设备提升至超高分辨率，也可以自动改善应用程序的画面质量。
如图2所示，应用程序所需图像的分辨率为低分辨率，电子设备执行渲染方法可以将图像的低分辨率提升到高分辨率，从而自动改变图像的分辨率，以此来提升画面质量。其中，CPU、NPU和GPU的处理过程包括：CPU截取应用程序下发的渲染命令；CPU响应渲染命令，在内存中为图像的渲染处理创建对应的FB；CPU基于渲染命令获得图像的分辨率为低分辨率后，向GPU发送渲染指令；GPU基于渲染指令生成低分辨率的图像数据，将低分辨率的图像数据发送给NPU；NPU对低分辨率的图像数据进行超分辨率渲染处理，得到高分辨率的图像数据，将高分辨率的图像数据发送给GPU；GPU接收到高分辨率的图像数据后，基于高分辨率的图像数据进行用户界面(User Interface，UI)渲染与后处理，以完成一帧图像的绘制。例如GPU基于高分辨率的图像数据，对渲染命令指向的FB进行渲染处理，得到渲染结果，渲染结果可以是图像中部分元素的图像数据。在得到所有FB的渲染结果后，GPU基于渲染结果将绘制后的图像显示在显示屏上。
在本实施例中，电子设备可提供一个设置界面，设置界面中具有“分辨率提升”的设置选项，设置选项用于提供分辨率选择。如图3示出了一种设置界面的示意图，在图3中设置选项提供了多种分辨率选择，如1080P、2k、大于2k，用户可以从这些分辨率中选择一种；在应用程序的分辨率小于用户选择的分辨率时，电子设备执行上述图2所示的渲染方法。图3只是一种示例，设置选项还可以采用其他形式，如图3中虚线框中所示，本实施例不对设置选项进行限定。
此外，电子设备可以设置一个白名单，白名单中记录了进行分辨率提升的应用程序，CPU拦截到应用程序下发的渲染命令后，确定应用程序是否在白名单，如果应用程序在白 名单中，CPU可以通过GPU触发NPU进行分辨率的提升，NPU将分辨率提升后的图像数据发送给GPU，GPU基于分辨率提升后的图像数据进行渲染处理；如果应用程序不在白名单中，CPU触发GPU进行渲染处理，此时图像数据的分辨率与应用程序下发的分辨率相同。
渲染方法对应的另一种使用场景是：电子设备输出的画面质量与应用程序要求的画面质量相同，但是电子设备可以先降低分辨率再提升分辨率。如图4所示，应用程序所需图像的分辨率可以为高分辨率，CPU、NPU和GPU的处理过程包括：CPU截取应用程序下发的渲染命令；CPU响应渲染命令，在内存中为图像的渲染处理创建对应的FB；CPU从渲染命令中提取出图像的分辨率为高分辨率后，CPU降低图像的分辨率，然后CPU可以向GPU发送渲染指令；GPU基于渲染指令生成低分辨率的图像数据，将低分辨率的图像数据发送给NPU；NPU对低分辨率的图像数据进行超分辨率渲染处理，得到高分辨率的图像数据，将高分辨率的图像数据发送给GPU；GPU接收到高分辨率的图像数据后，基于高分辨率的图像数据进行UI渲染与后处理，以完成一帧图像的绘制，图像的渲染结果存储在FB中，GPU将绘制后的图像显示在显示屏上。
CPU降低图像的分辨率后，NPU可以通过超分辨率渲染处理还原出高分辨率的图像数据，GPU可以利用高分辨率的图像数据还原出高分辨率的图像，使GPU绘制出的图像和应用程序所需图像的画面质量相同，满足应用程序对画面质量的要求，提高渲染准确度。并且CPU降低图像的分辨率后输入到NPU中的数据量降低，可以在提高渲染准确度，加快渲染速度。
例如图像的分辨率为1080P或者在1080P以上，CPU可以降低图像的分辨率，然后利用NPU对降低分辨率后的图像数据进行超分辨率渲染，使得超分辨率渲染后的渲染结果与降低前的分辨率相对应。对于分辨率降低多少可根据NPU的超分能力而定，如超分能力为2倍超分，则对1080P图像的分辨率降低0.5倍，其中可以注意一点的是，2倍和0.5倍对应分辨率的边长，而不是图像的像素。
对于图4所示渲染方法，CPU可根据应用程序所需图像的分辨率，决定是否降低分辨率，例如本实施例中应用程序所需图像的分辨率为1080P或者在1080P以上，CPU降低图像的分辨率，再由NPU对低分辨率的图像数据进行超分辨率渲染；如果应用程序所需图像的分辨率小于1080P，在图像的分辨率维持不变的情况下，CPU触发GPU进行渲染处理。例如在图4所示渲染方法中，用户点击桌面中的游戏应用程序(图4中的游戏1)，游戏应用程序所需图像的分辨率为高分辨率，在游戏应用程序启动后电子设备执行图4所示渲染方法。
此外，应用程序所需图像的分辨率为高分辨率时，NPU接收到GPU发送的低分辨率的图像数据后，NPU对低分辨率的图像数据进行超分辨率渲染处理，得到超高分辨率的图像数据，GPU基于超高分辨率的图像数据进行绘制，GPU绘制出的图像的画面质量高于应用程序所需图像的画面质量。例如应用程序所需图像的分辨率为1080P，CPU降低分辨率至540P，GPU生成540P的图像数据，NPU向GPU反馈2k的图像数据，GPU可以绘制出分辨率为2k的图像，其画面质量高于1080P的图像的画面质量，实现对应用程序的画面质量的提升。
渲染方法对应的再一种使用场景是：在应用程序运行过程中，用户可以调整应用程序的画面质量，用户可将画面质量从低分辨率调整至高分辨率或超高分辨率，也可以从高分辨率或超高分辨率调整至低分辨率。图5示出了用户将画面质量从低分辨率调整至高分辨 率，分辨率调整后对应的处理过程也发生变化，处理过程如下：
1)CPU截取应用程序下发的渲染命令；CPU响应渲染命令，在内存中为图像的渲染处理创建对应的FB。
2)CPU基于渲染命令获得图像的分辨率为低分辨率后，CPU向GPU发送渲染指令。GPU响应渲染指令，对FB进行渲染处理，得到渲染结果，GPU完成所有FB的渲染处理后，GPU基于渲染结果，将图像显示在显示屏上。
3)在用户调整应用程序的画面指令至高分辨率后，CPU可以基于渲染命令获得图像的分辨率为高分辨率，CPU降低图像的分辨率至低分辨率，然后CPU可以向GPU发送渲染指令；GPU基于渲染指令生成低分辨率的图像数据，将低分辨率的图像数据发送给NPU；NPU对低分辨率的图像数据进行超分辨率渲染处理，得到高分辨率的图像数据，将高分辨率的图像数据发送给GPU；GPU接收到高分辨率的图像数据后，基于高分辨率的图像数据进行UI渲染与后处理，以完成一帧图像的绘制，图像的渲染结果存储在FB中，GPU将绘制后的图像显示在显示屏上。
在本实施例中，CPU可以指定超分辨率渲染的FB，GPU处理指定的FB时利用NPU和/或DSP完成超分辨率渲染，如利用NPU和/或DSP完成从低分辨率到高分辨率和/或超高分辨率的转换，对其他FB利用GPU渲染，GPU可以不再区分图像的分辨率。
其中进行超分辨率渲染的FB可以根据渲染方式的不同而不同。渲染方式包括延迟渲染和前向渲染，在延迟渲染中，颜色信息(color attachment)不仅记录有颜色数据，还记录有每一个像素的法向量数据和深度数据，经过上采样后的法向量数据在进行光照计算时出错概率很大，且所有FB中除FB0之外的FB之间的关联性较大(如一个FB的渲染结果绑定在下一个FB中)，如果对一个FB进行超分辨率渲染，其他FB使用该FB的渲染结果时出错概率很大，因此在延迟渲染中，可以对每帧图像对应的所有FB中除FB0之外的FB进行超分辨率渲染，即延迟渲染中，进行超分辨率渲染的FB为一帧图像对应的所有FB中除FB0之外的FB。例如一帧图像对应的所有FB包括：FB0、FB1和FB2，进行超分辨率渲染的FB包括FB1和FB2，FB0是应用程序下发的所有FB中最后一个FB。
前向渲染中FB之间的关联性较小，可以至少对所有FB中的一个FB进行超分辨率渲染，在前向渲染下，CPU可以根据渲染操作数量，确定进行超分辨率渲染的FB。一种示例中，CPU可以将渲染操作数量大于预设数量的FB确定为进行超分辨率渲染的FB；另一种示例中，CPU可以将渲染操作数量最多的FB确定为进行超分辨率渲染的FB，其中渲染操作数量最多的FB可以是GPU对该FB执行的Drawcall(与绘制相关的绘制指令)数量最多，这是因为GPU对该FB执行的Drawcall数量越多，表明GPU在该FB上执行的渲染操作越多，在GPU渲染该FB时会消耗更多的算力资源，也会产生更高的功耗以及发热，由此本实施例可以将渲染操作数量最多的FB确定为进行超分辨率渲染的FB。渲染操作数量最多的FB是执行Drawcall数量最多的FB。为了便于说明，将前向渲染中进行超分辨率渲染的FB称为主FB，下面结合示例对确定Drawcall数量最多的FB进行说明。
在实施例中，CPU可以用于接收来自应用程序的渲染命令，并根据该渲染命令，向GPU下发对应的渲染指令，以便于GPU根据渲染指令执行对应的渲染。作为一种示例，渲染命令中可以包括glBindFrameBuffer()函数，以及一个或多个glDrawElement、glDrawArray、 glDrawElementInstanced、glDrawArrayInstanced。对应的，渲染指令中也可以包括glBindFrameBuffer()函数，以及一个或多个glDrawElement、glDrawArray、glDrawElementInstanced、glDrawArrayInstanced。其中，glBindFrameBuffer()函数可以用于指示当前绑定的FB，实现对应渲染操作与FB的绑定。如，glBindFrameBuffer(1)可以指示当前绑定的帧缓冲为FB1，GPU在FB1上执行的Drawcall包括：glDrawElement、glDrawArray、glDrawElementInstanced、glDrawArrayInstanced。
比如，第N帧图像可以使用FB0，FB1以及FB2作为帧缓冲为例。结合图6，为FB1对应的渲染命令的一种示例。应用程序下发的渲染命令中，可以包括glBindFrameBuffer(1)函数，由此实现当前渲染操作与FB1的绑定。在绑定FB1之后，可以通过glDrawElement指令，实现在该FB1上的渲染操作指示。在本示例中，1个glDrawElement指令可以对应1个Drawcall。在不同实现中，在FB1上执行的glDrawElement指令可以为多个，则对应的在FB1上执行的Drawcall就可以是多个。
CPU可以在根据应用程序下发的渲染命令，执行glBindFrameBuffer(1)时，初始化计数器1。比如在内存中为FB1配置对应的计数帧位，通过初始化该计数器1，可以将该帧位的值初始化为0。后续每在FB1上执行glDrawElement时，计数器1计数加1，如执行count1++。比如，在执行glDrawElement1-1后，CPU可以对计数器1执行count1++，从而使得存储在FB1Drawcall数量的帧位的数值由0变为1，此时在FB1上执行的Drawcall数量为1。以此类推，CPU就可以确定在进行第N帧图像的渲染过程中，FB1上执行Drawcall的数量为计数器1的当前计数(比如，该计数可以为A)。
类似的，对于FB2，GPU可以通过glBindFrameBuffer(2)函数绑定FB2。此后GPU通过glDrawElement、glDrawArray、glDrawElementInstanced、glDrawArrayInstanced等指令，实现在该FB2上的渲染操作指示。与FB1类似的，CPU还可以在根据应用程序下发的渲染命令，调用glBindFrameBuffer(2)时，初始化计数器2。如初始化count2＝0。后续每在FB2上执行glDrawElement时，计数器2计数加1，如执行count2++。在完成FB2上图像的渲染处理之后，CPU就可以确定在进行第N帧图像的渲染过程中，FB2上执行Drawcall的数量为计数器2的当前计数(比如，该计数可以为B)。如参考图7。在完成FB1和FB2的渲染处理之后，内存中存储FB1Drawcall数量的帧位的数值可以为A，存储FB2Drawcall数量的帧位可以为B。
在本示例中，CPU可以选取A和B中较大的计数对应的FB作为主FB。比如，在A大于B时，则CPU可以确定FB1上执行的Drawcall个数更多，确定FB1为主FB。相对的，在A小于B时，则CPU可以确定FB2上执行的Drawcall个数更多，确定FB2为主FB。
在其他示例中，CPU可以将执行Drawcall的数量大于预设阈值的确定为主FB，CPU可以通过计数方式选择执行Drawcall的数量大于预设阈值的FB，对此不再详述。
相对应的，CPU、GPU和NPU进行渲染的过程如图8所示，CPU根据应用程序下发的渲染命令，触发渲染流程，由CPU创建FB。其中图8中应用程序指示对高分辨率的图像进行渲染。CPU接收到渲染命令后，判断渲染方式是前向渲染还是延迟渲染。如果渲染方式是前向渲染，CPU确定当前执行渲染操作的FB是否是FB1(FB1为主FB)，如果当前执行渲染操作的FB是FB1，CPU降低图像的分辨率，向GPU发送渲染指令，指示GPU生成低分辨率 的图像数据，GPU将低分辨率的图像数据发送给NPU，NPU完成超分辨率渲染；如果CPU确定渲染方式是延迟渲染，CPU确定当前执行渲染操作的FB是否是FB0；如果当前执行渲染操作的FB是FB1或FB2，CPU也可以降低图像的分辨率，指示GPU生成低分辨率的图像数据，GPU将低分辨率的图像数据发送给NPU，NPU完成超分辨率渲染。GPU接收到NPU发送的高分辨率的图像数据后，基于高分辨率的图像数据进行UI渲染与后处理，以完成一帧图像的绘制，图像的渲染结果存储在FB中，GPU将绘制后的图像显示在显示屏上。
图8中的两个GPU可以是同一个GPU，之所以示出两个GPU是为了阐述在不同渲染方式下针对不同FB，采用不同的处理方式。
在一些实施例中，在延迟渲染下，CPU在当前执行渲染操作的FB是FB1、FB2时，CPU降低图像的分辨率，GPU可以生成FB1对应的低分辨率的图像数据，也可以生成FB2对应的低分辨率的图像数据，然后由NPU对两个FB对应的低分辨率的图像数据进行超分辨率渲染。在另一些实施例中，在延迟渲染下，FB0的上一个FB对应一帧图像的图像数据，GPU可以生成FB0的上一个FB的低分辨率的图像数据，由NPU对FB0的上一个FB对应的低分辨率的图像数据进行超分辨率渲染。即在延迟渲染下，在FB为除FB0之外的FB时，GPU可以生成低分辨率的图像数据，然后NPU对低分辨率的图像进行超分辨率渲染；也可以在FB为FB0的上一个FB时，GPU生成低分辨率的图像数据，NPU对低分辨率的图像数据进行超分辨率渲染。
CPU可根据NPU的超分倍率，降低图像的分辨率。超分倍数用于指示NPU可提升的分辨率的倍数，例如NPU的超分倍率是2倍，CPU可以将图像的分辨率降低2倍；如果NPU的超分倍数是4倍，CPU可以将图像的分辨率降低4倍。例如如果NPU的超分倍数是2倍，CPU可以将图像的分辨率从1080P降低至540P，NPU可以将540P转换成1080P，完成从540P到1080P的超分辨率渲染。如果NPU的超分辨率是4倍，CPU则可以将图像的分辨率从2k降低至540P，那么NPU可以将540P转换成2K，完成从540P到2k的超分辨率渲染；CPU也可以将图像的分辨率从1080P降低至270P，相应地NPU可以将207P转换成1080P。
在本实施例中，渲染方法对应的示意图如9所示，应用程序下发每帧图像的渲染命令，CPU截取应用程序下发的渲染命令，CPU基于渲染命令，确定当前渲染命令对应的图像的分辨率是高分辨率，CPU可以降低图像的分辨率至低分辨率；CPU向GPU发渲染指令，GPU基于渲染指令，生成低分辨率的图像数据；NPU基于人工智能(Artificial Intelligence，AI)超分辨率模型完成低分辨率到高分辨率的转换，得到高分辨率的图像数据；GPU基于高分辨率的图像数据进行UI渲染与后处理，并在显示屏上显示出图像。
图9所示渲染方法的流程图如图10所示，在图10中所示渲染方法中，NPU中运行有AI超分辨率模型，利用AI超分辨率模型完成超分辨率渲染，可以包括以下步骤：
(1)应用程序启动后，确定FB1为主FB，GPU在该FB1上执行的Drawcall数量最多。
(2)对AI超分辨率模型进行初始化，初始化包括运行时检查、模型加载、模型编译和内存配置；
(3)渲染方式是前向渲染时，在当前处理的FB为FB1时执行(4)；渲染方式时延迟渲染时，在当前处理的FB是除FB0之外的其他FB时执行(4)；
(4)降低图像的分辨率，图像的分辨率从高分辨率降低至低分辨率；
(5)生成低分辨率的图像数据；
(6)利用AI超分辨率模型对低分辨率的图像数据进行超分辨率渲染，得到高分辨率的图像数据；
(7)基于高分辨率的图像数据进行UI渲染与后处理，以完成一帧图像的绘制；
(8)在显示屏上显示图像；
(9)在应用程序运行过程中，可进行数据采集得到数据样本集，利用数据样本集对AI超分辨率模型进行模型训练，得到AI超分辨率模型，再利用转换工具(Conversion tools)将AI超分辨率模型转换成电子设备可识别的模型文件。
下面以在第N-1帧图像渲染过程中确定FB1为主FB以及在确定主FB后对第N帧图像的渲染过程，对本实施例提供的渲染方法的详细过程进行说明，其过程如图11所示，可以包括以下步骤：
S101、CPU在第N-1帧图像的渲染过程中确定FB1为主FB。主FB是渲染过程中执行渲染操作数量最多的FB，渲染操作数量最多的FB可以是执行Drawcall数量最多的FB，在本实施例中，CPU确定FB1为执行Drawcall数量最多的FB。
一帧图像对应多个FB，每个FB执行的Drawcall数量不同，本实施例可以对一帧图像中的所有FB进行计数，得到该帧图像中所有FB执行的Drawcall数量，然后将该帧图像中执行Drawcall数量最多的FB确定为主FB，该帧图像之后的其他图像的主FB可以与该帧图像的主FB相同。示例性的，以当前渲染的帧图像为第N帧图像为例。CPU可以根据上一帧(如第N-1帧图像)图像渲染过程中，在第N-1帧图像的不同FB上执行Drawcall的数量，通过不同FB上执行Drawcall的数量，确定FB1为主FB，将第N-1帧图像的主FB确定为第N帧图像、第N+1帧图像、第N+2帧图像等的主FB。
S102、CPU拦截应用程序下发的第N帧图像的渲染命令。
S103、CPU基于渲染命令，获取到第N帧图像的初始分辨率为高分辨率以及当前处理的FB。
CPU可以基于渲染命令，获取到第N帧图像的宽和高，一个分辨率下图像的宽和高是固定的，例如图像的分辨率是720P时，图像的宽是1280、图像的高是720；图像的分辨率是1080P时，图像的宽是1920、图像的高是1080，因此通过第N帧图像的宽和高能够确定第N帧图像的初始分辨率。一般1920×1080及以上的图像认为是高分辨率，本实施例可以在图像的宽和高满足1920×1080及以上时，确定图像的初始分辨率为高分辨率。CPU基于渲染命令，可以获取到当前处理的FB标识，如FB1。第N帧图像的初始分辨率和当前处理的FB可以从不同的渲染命令中得到。
S104、CPU对AI超分辨率模型进行初始化，通过初始化操作确定是否运行NPU中的AI超分辨率模型以及如果运行确保AI超分辨率模型可以正常运行。
AI超分辨率模型的初始化包括运行时检查、模型加载、模型编译以及内存配置，运行时检查是为了确定是否运行NPU中的AI超分辨率模型，模型加载、模型编译以及内存配置是为了确保AI超分辨率模型可以正常运行。
在一些实现中，运行时检查包括检查电子设备中是否配置有NPU和检查应用程序的分辨率；如果配置有NPU，在NPU中运行AI超分辨率模型；如果应用程序的分辨率为高分辨 率或者超高分辨率，运行AI超分辨率模型；如果电子设备中没有配置NPU或应用程序的分辨率为低分辨率，禁止运行AI超分辨率模型，由CPU向GPU下发渲染指令，GPU响应渲染指令进行渲染。
模型加载是将AI超分辨率模型转换成NPU可识别的模型文件，在初始化阶段将模型文件加载到内存中，模型编译是为了验证模型文件能够成功运行；内存配置是为AI超分辨率模型分配内存，所分配的内存用于存储AI超分辨率模型的输入数据和输出数据，在本实施例中为AI超分辨率模型分配的内存可以是CPU内存、Neural Processing API管理的内存(如SNPE ITensor buffer)、或共享内存。
CPU内存可以是为CPU分配的内存，CPU运行过程中使用的数据写入到CPU内存中，AI超分辨率模型占用CPU内存的部分空间作为AI超分辨率模型的内存，GPU和NPU之间的数据交互可以通过CPU内存，且GPU绘制图像时可以将CPU内存中的数据写入到GPU内存(为GPU分配的内存)中，然后从GPU内存中读取数据进行绘制。共享内存可以是CPU、GPU和NPU共用的内存，GPU可以直接从共享内存中读取数据进行绘制。
在这里说明一点，CPU确定第N帧图像的初始分辨率是高分辨率时触发AI超分辨率模型初始化，是AI超分辨率模型初始化的一种方式。AI超分辨率模型可以在其他时机下进行初始化，一种示例中，CPU监测到应用程序启动后对AI超分辨率模型进行初始化。一些应用程序的图像渲染次数少，对GPU的占用低，CPU可以设置一个应用程序的白名单，CPU监测到应用程序启动后确定应用程序是否在白名单中，如果在白名单中对AI超分辨率模型进行初始化。另一种示例中，CPU在渲染方式为前向渲染且当前处理的FB为FB1时对AI超分辨率模型进行初始化；在渲染方式为延迟渲染且当前处理的FB不是FB0时对AI超分辨率模型进行初始化。
S105、CPU从应用程序的配置文件中获取渲染方式。
渲染方式包括延迟渲染和前向渲染，渲染方式不同，执行超分辨率渲染的FB也不同。在延迟渲染中，可以对每帧图像对应的所有FB中除FB0之外的FB进行超分辨率渲染，即延迟渲染中，每帧图像对应的所有FB中除FB0之外的FB可以进行超分辨率渲染。CPU在确定当前执行渲染命令的FB不是FB0，对当前执行渲染命令的FB进行超分辨率渲染。
在前向渲染中，渲染过程中GPU的算力消耗集中在主FB渲染中。在主FB渲染中，GPU的算力集中在片元着色器(Fragment Shader，FS)的计算中，由此发现可推测出，对前向渲染中的主FB进行超分辨率渲染，主FB可以是执行Drawcall数量最多的FB。如本实施例中，CPU在第N-1帧图像渲染过程中确定出FB1为主FB，则当前处理的FB是FB1，CPU对FB1进行超分辨率渲染。
如果渲染方式为前向渲染且当前处理的FB不是FB1，CPU可以向GPU发送渲染指令，由GPU对当前FB进行渲染处理；如果渲染方式为延迟渲染且当前处理的FB是FB0，CPU可以向GPU发送渲染指令，由GPU对当前FB进行渲染处理。
S106、如果渲染方式为前向渲染且当前处理的FB是FB1，CPU基于超分倍数降低图像的初始分辨率至低分辨率；如果渲染方式为延迟渲染且FB不是FB0(如FB1和FB2)，CPU基于超分倍数降低图像的初始分辨率至低分辨率。
CPU可以通过降低图像的宽和高来降低图像的分辨率。CPU在降低图像的分辨率时可以 参照AI超分辨率模型的超分倍数而定，其中AI超分辨率模型可以对图像数据完成从低分辨率到高分辨率的转换，高分辨率大于低分辨率，例如高分辨率可以是低分辨率的2倍、3倍和4倍中的一个，即AI超分辨率模型的超分倍数是2倍、3倍和4倍中的一个。以540P为例，540P图像经过2倍超分辨率渲染后得到1080P图像，经过4倍超分辨率渲染后得到2k图像。
在本实施例中，CPU可以根据AI超分辨率模型的超分倍数，降低图像的分辨率。CPU可通过一个缩放倍数来降低图像的分辨率，降低图像的分辨率可通过降低图像的宽和高实现，如以宽和高表示图像：Res＝w×h，缩放倍数r的取值为r∈(0,1)，即缩放倍数r的取值在0至1之间，降低分辨率后的图像为Res L＝(r×w)×(r×h)，采用的缩放倍数与AI超分辨率模型的超分倍数相关。例如图像的初始分辨率为1080P，以宽和高表示为1920×1080，AI超分辨率模型的超分倍数为2，AI超分辨率模型能够完成从540P到1080P的转换，540P图像的宽和高为960×540，说明是对1080P图像缩小了2倍，相对应的缩放倍数为0.5。假如AI超分辨率模型的超分倍数为4，AI超分辨率模型能够完成从270P到1080P的转换，对1080P图像缩小4倍，相对应的缩放倍数为0.25。缩放倍数指的是对图像的边长的缩放，如缩放倍数为0.5指的是对图像的宽和高同时缩放0.5，在图像和宽和高同时缩放0.5时，图像的像素缩放了0.25。
超分倍数用于指示NPU的超分能力，指示NPU提升分辨率的倍数，CPU可以在AI超分辨率模型初始化过程中确定超分倍数；如果CPU没有在初始化过程中确定超分倍数，CPU可以向NPU发送图像的高分辨率和低分辨率，即降低之前的分辨率和降低之后的分辨率，NPU基于降低之前的分辨率和降低之后的分辨率，选择匹配的超分倍数。
S107、CPU通知GPU生成低分辨率的图像数据。在一些示例中，NPU的超分倍数可以是一个固定倍数，CPU中各应用程序所需图像的分辨率也可以是一个固定倍数，例如NPU的超分倍数是2倍，各应用程序所需图像的分辨率为1080P，那么CPU在降低分辨率时可以将分辨率降低至540P。因为超分倍数和图像的分辨率固定，GPU知道分辨率可以降低至540P，所以CPU通知GPU生成低分辨率的图像数据时，通知中可以不携带低分辨率。
在一些示例中，NPU的超分倍数可以是一个固定倍数，但CPU中各应用程序所需图像的分辨率不固定，例如一些应用程序所需图像的分辨率为1080P，一些应用程序所需图像的分辨率为2k，那么CPU降低后的低分辨率可能有所不同，CPU向GPU发送的通知中可以携带低分辨率。
在一些示例中，NPU的超分倍数可以不固定，即NPU的超分倍数是多个，CPU可以选择其中一个超分倍数来降低分辨率，相对应的CPU向GPU发送的通知中可以携带低分辨率。
CPU还可以向NPU发送通知，通知中可以携带第N帧图像降低前后的分辨率，如步骤S107’，步骤S107’是一个可选步骤，比如在NPU的超分倍数不固定时执行。
S107’、CPU通知NPU第N帧图像的高分辨率和低分辨率，NPU通过高分辨率和低分辨率可获知第N帧图像的分辨率变化情况，在利用NPU中的AI超分辨率模型进行超分辨率渲染时，AI超分辨率模块可确定超分倍数，以还原出与第N帧图像的高分辨率相符的图像数据。当然NPU也可以还原出分辨率大于第N帧图像的高分辨率的图像数据。
S108、GPU生成低分辨率的图像数据，如GPU生成低分辨率的RGB图像数据，RGB图像 数据包括R通道数据、G通道数据和B通道数据。
S109、GPU向NPU发送低分辨率的图像数据。
S110’、NPU确定AI超分辨率模型的超分倍数。
S110、NPU利用AI超分辨率模型，基于超分倍数对低分辨率的图像数据进行超分辨率渲染，得到高分辨率的图像数据。
AI超分辨率模型的超分倍数可以作为NPU的超分倍数。在一些示例中，AI超分辨率模型的超分倍数是一个固定倍数，省去步骤S110’。在一些示例中，AI超分辨率模型的超分倍数可以不固定，如AI超分辨率模型的超分倍数包括2倍和4倍，NPU执行步骤S110’，根据步骤S107’通知的第N帧图像的高分辨率和低分辨率，确定当前采用的超分倍数。
在本实施例中，AI超分辨率模型可以是以低分辨率的图像数据作为输入，以高分辨率的图像数据为输出，通过历史的低分辨率的图像数据进行训练得到，且在训练过程中以GPU渲染出的高分辨率的图像为参考对AI超分辨率模型进行校验。低分辨率和高分辨率是相对而言，高分辨率和低分辨率可以满足：高分辨率大于低分辨率的关系，例如低分辨率为540P，高分辨率为1080P；又例如低分辨率为720P，高分辨率为2k。
AI超分辨率模型可以使用TensorFlow、PyTorch等训练框架进行离线训练。但是一些电子设备不支持TensorFlow、PyTorch等训练框架，例如高通SNPE框架、NNAPI(Android Neural Networks API)等手机支持的模型格式与TensorFlow和PyTorch等不相同，高通SNPE框架支持的模型格式为DLC格式；NNAPI支持的模型格式为TensorFlow Lite，如果以TensorFlow、PyTorch等训练框架离线训练出AI超分辨率模型，可以将TensorFlow，PyTorch等格式的AI超分辨率模型转换为DLC或是TensorFlow Lite等格式，以便于在手机等设备中使用。
在训练出AI超分辨率模型后，AI超分辨率模型可以进行更新以适应图像变化，例如电子设备可以获得AI超分辨率模型的超分辨率渲染效果，根据超分辨率渲染效果对AI超分辨率模型进行参数调整，参数调整可以是离线调整也可以是在线调整。参数调整的设备方可以是使用AI超分辨模型进行超分辨率渲染的电子设备，也可以是其他电子设备，在其他电子设备完成参数调整后，发送给使用AI超分辨模型进行超分辨率渲染的电子设备，例如电脑对AI超分辨率模型的参数进行调整，电脑完成调整后发送给手机使用。在应用程序渲染过程中电子设备可以进行数据样本采集，得到一个数据样本集，利用数据样本集中的数据样本对AI超分辨率模型进行调整。
S111、NPU向GPU发送高分辨率的图像数据。
S112、GPU基于高分辨率的图像数据，对当前处理的FB进行渲染处理，得到渲染结果，如第N帧图像中元素的图像数据。
S113、GPU在得到所有FB的渲染结果后，基于渲染结果，将第N帧图像显示在显示屏上。
上述渲染方式可使用在GPU渲染图像的各个场景中，如可以使用在游戏应用程序、家居设计应用程序、建模应用程序、增强现实应用程序以及虚拟显示应用程序等应用程序中，利用运行在NPU中的AI超分辨率模型进行超分辨率渲染，以使用NPU的算力分担一部分GPU的计算量，从而使得电子设备的功耗更低，渲染时间更短。
在本实施例中，GPU和NPU之间的图像数据交互可借助GPU内存和CPU内存完成，GPU内存可以是GPU的私有内存，禁止CPU和NPU访问GPU内存，但是GPU可通过调用函数等方式从GPU内存中读取图像数据，GPU读取的图像数据可写入到CPU内存中、也可以在GPU绘制图像时使用，因此GPU内存可作为GPU的交互内存和运算内存使用。
CPU可以指定CPU内存的一块存储空间(如第一区域)作为NPU输入内存和NPU输出内存，NPU输入内存是写入NPU进行超分辨率渲染时使用的输入数据，NPU输出内存则是写入NPU完成超分辨率渲染时的输出数据，NPU输入内存和NPU输出内存可以是CPU内存中的同一块区域，CPU可根据NPU的输入数据的数据量和输出数据的数据量，指定CPU内存中被NPU占用的空间。无论是GPU还是NPU，GPU和NPU可以访问CPU内存。相对应的内存互访流程如图12所示，可以包括以下步骤：
S200、CPU将CPU内存中的第一区域指定为NPU输入内存和NPU输出内存，其中NPU输入内存和NPU输出内存可以是第一区域，在向NPU输入数据时作为NPU输入内存使用，在NPU输出数据时作为NPU输出内存使用，第一区域的大小可根据NPU处理的数据量而定，此处不再赘述。
S201、CPU通知NPU第一区域的指针地址，NPU可基于指针地址从NPU输入内存中读取图像数据，且基于指针地址向NPU输出内存中写入图像数据。
S202、GPU将低分辨率的图像数据写入到GPU内存中。
S203、GPU通知CPU低分辨率的图像数据已经写入到GPU内存。
S204、CPU通知GPU将低分辨率的图像数据写入到第一区域中。其中CPU发送的通知中携带有第一区域的指针地址。CPU向GPU通知第一区域的指针地址可以是在步骤S201中，即在步骤S201，CPU通知NPU和GPU第一区域的指针地址。
S205、GPU从GPU内存中读取低分辨率的图像数据。
S206、GPU基于指针地址，将低分辨率的图像数据写入到第一区域中。
S207、CPU通知NPU从第一区域中读取低分辨率的图像数据。
S208、NPU基于指针地址从第一区域中读取低分辨率的图像数据。
S209、NPU基于指针地址向第一区域中写入高分辨率的图像数据。
NPU在读取到低分辨率的图像数据后，将低分辨率的图像数据输入到运行在NPU中的AI超分辨率模型中，AI超分辨率模型对低分辨率的图像数据进行超分辨率渲染，输出高分辨率的图像数据。NPU再基于指针地址向第一区域中写入高分辨率的图像数据。
S210、CPU通知GPU从第一区域中读取高分辨率的图像数据。
S211、GPU基于指针地址，从第一区域中读取高分辨率的图像数据。
S212、GPU将高分辨率的图像数据写入到GPU内存中。
结合图12所示内存互访流程和图11所示渲染方法，图13示出了渲染方法的另一种时序图，可以包括以下步骤：
S301、CPU在第N-1帧图像的渲染过程中确定FB1为主FB。
S302、CPU拦截应用程序下发的第N帧图像的渲染命令。
S303、CPU基于渲染命令，获取到第N帧图像的初始分辨率为高分辨率以及当前处理的FB。
S304、CPU对AI超分辨率模型进行运行时检查、模型加载、模型编译。
S305、CPU将CPU内存中的第一区域指定为NPU输入内存和NPU输出内存，完成对AI超分辨率模型的内存配置，通过步骤S304和步骤S305完成对AI超分辨率模型的初始化。
S306、CPU通知NPU第一区域的指针地址。
S307、CPU从应用程序的配置文件中获取渲染方式。
S308、如果渲染方式为前向渲染且当前处理的FB是FB1，CPU基于超分倍数降低图像的初始分辨率至低分辨率；如果渲染方式为延迟渲染且FB不是FB0(如FB1和FB2)，CPU基于超分倍数降低图像的初始分辨率至低分辨率。
S309、CPU通知GPU生成低分辨率的图像数据。
S309’、CPU通知NPU第N帧图像的高分辨率和低分辨率。
S310、GPU生成低分辨率的图像数据。
S311、GPU将低分辨率的图像数据写入到GPU内存中。
S312、GPU通知CPU低分辨率的图像数据已经写入到GPU内存。
S313、CPU通知GPU将低分辨率的图像数据写入到第一区域中。CPU发送的通知中携带有第一区域的指针地址。
S314、GPU从GPU内存中读取低分辨率的图像数据。
S315、GPU基于指针地址，将低分辨率的图像数据写入到第一区域中。
S316、CPU通知NPU从第一区域中读取低分辨率的图像数据。
S317、NPU基于指针地址从第一区域中读取低分辨率的图像数据。
S318’、NPU确定AI超分辨率模型的超分倍数。
S318、NPU利用AI超分辨率模型，基于超分倍数对低分辨率的图像数据进行超分辨率渲染，得到高分辨率的图像数据。
S319、NPU基于指针地址向第一区域中写入高分辨率的图像数据。
S320、CPU通知GPU从第一区域中读取高分辨率的图像数据。
S321、GPU从第一区域中读取高分辨率的图像数据。
S322、GPU将高分辨率的图像数据写入到GPU内存中。
S323、GPU基于高分辨率的图像数据，对当前处理的FB进行渲染处理，得到渲染结果，如第N帧图像中元素的图像数据。
S324、GPU在得到所有FB的渲染结果后，基于渲染结果，将第N帧图像显示在显示屏上。
如果CPU从CPU内存中指定一块区域为NPU输入内存，指定另一块区域为NPU输出内存，则CPU将NPU输入内存的指针地址和NPU输出内存的指针地址发送给NPU，NPU基于NPU输入内存的指针地址读取低分辨率的图像数据，基于NPU输出内存的指针地址读取高分辨率的图像数据。同样的CPU将NPU输入内存的指针地址和NPU输出内存的指针地址发送给GPU，GPU基于NPU输入内存的指针地址写入低分辨率的图像数据，基于NPU输出内存的指针地址读取高分辨率的图像数据。
但是无论是低分辨率的图像数据还是高分辨率的图像数据，GPU和NPU可以借助GPU内存和CPU内存完成交互，存在图像数据在GPU内存和CPU内存中相互拷贝的问题，延长 了GPU和NPU获取数据的时长，从而增加渲染时间。
针对这一问题，本实施例中GPU、CPU和NPU可以以共享内存方式进行数据交互，其中共享内存是指GPU、CPU和NPU可以访问的内存，共享内存可以作为GPU的外部缓存，至少存储GPU向NPU输入的低分辨率的图像数据，并且共享内存可以作为NPU的处理缓存，至少存储NPU输出的高分辨率的图像数据。
GPU和NPU使用共享内存的内存互访流程如图14所示，可以包括以下步骤：
S401、CPU从硬件缓冲区(Hardware Buffer)中申请共享内存。CPU可以从随机存取存储器(Random Access Memory，RAM)的Hardware Buffer中申请共享内存。一种共享内存申请方式是，CPU从Hardware Buffer中申请GPU SSBO(Shader Storage Buffer Object)的存储空间，在GPU中的Fragment Shader或GPU中的Compute Shader得到低分辨率的图像数据后，低分辨率的图像数据写入到GPU SSBO中。NPU输出的高分辨率的图像数据也可以写入到共享内存中。
Hardware Buffer作为一种内存，CPU、GPU和NPU访问Hardware Buffer时，对Hardware Buffer的格式有一定的要求，如申请共享内存为GPU中的Fragment Shader或GPU中的Compute Shader可访问的存储空间时，可以指定Hardware Buffer的格式为AHARDWAREBUFFER_FORMAT_BLOB。
另一种共享内存申请方式是，CPU可以从Hardware Buffer中申请两块共享内存，一块共享内存用来存储GPU得到的低分辨率的图像数据，作为NPU的输入数据；另一块共享内存用来存储NPU的输出数据，如超分辨率渲染得到的高分辨率的图像数据，作为GPU在UI渲染时的输入数据。CPU所申请的两块共享内存的格式相同，如共享内存的格式可以是AHARDWAREBUFFER_FORMAT_BLOB。申请的共享内存大小可根据图像的分辨率而定，如申请的共享内存大小为宽×高×字节大小，例如图像数据的类型为float类型，float变量为4字节，则申请的共享内存大小为宽×高×3×4，宽和高由图像的分辨率而定。比如1080P图像，申请的共享内存大小为1920×1080×3×4。
S402、CPU将共享内存的指针地址通知给GPU，以将共享内存作为低分辨率的图像数据的存储内存。
S403、CPU将共享内存的指针地址通知给NPU，以将共享内存作为高分辨率的图像数据的存储内存。
S404、GPU基于共享内存的指针地址，将低分辨率的图像数据写入到共享内存中。
S405、CPU通知NPU从共享内存中读取低分辨率的图像数据。
S406、NPU基于共享内存的指针地址,从共享内存中读取低分辨率的图像数据。
S407、NPU基于共享内存的指针地址,向共享内存中写入高分辨率的图像数据。
S408、CPU通知GPU从共享内存中读取高分辨率的图像数据。
S409、GPU基于共享内存的指针地址,从共享内存中读取高分辨率的图像数据。
如果CPU从Hardware Buffer中申请两块共享内存，一块共享内存用来存储GPU得到的低分辨率的图像数据，作为NPU的输入数据；另一块共享内存用来存储NPU的输出数据，如超分辨率渲染得到的高分辨率的图像数据，作为GPU在UI渲染时的输入数据。CPU可以将这两块共享内存的指针地址发送给GPU和NPU，以使GPU和NPU可以在共享内存中完成 图像数据的读取和写入。
GPU和NPU使用上述图14所示内存互访流程，能够利用共享内存实现数据共享，实现GPU和NPU之间共享数据的零拷贝，提升处理效率。结合图14所示内存互访流程和图11所示渲染方法，图15示出了渲染方法的再一种时序图，可以包括以下步骤：
S501、CPU在第N-1帧图像的渲染过程中确定FB1为主FB。
S502、CPU拦截应用程序下发的第N帧图像的渲染命令。
S503、CPU基于渲染命令，获取到第N帧图像的初始分辨率为高分辨率以及当前处理的FB。
S504、CPU对AI超分辨率模型进行运行时检查、模型加载、模型编译。
S505、CPU从RAM的Hardware Buffer中申请共享内存，共享内存作为低分辨率的图像数据和高分辨率的图像数据的存储空间，通过申请共享内存完成对AI超分辨率模型的内存配置，通过步骤S504和步骤S505完成对AI超分辨率模型的初始化。
S506、CPU通知GPU和NPU共享内存的指针地址。
S507、CPU从应用程序的配置文件中获取渲染方式。
S508、如果渲染方式为前向渲染且当前处理的FB是FB1，CPU基于超分倍数降低图像的初始分辨率至低分辨率；如果渲染方式为延迟渲染且FB不是FB0(如FB1和FB2)，CPU基于超分倍数降低图像的初始分辨率至低分辨率。
S509、CPU通知GPU生成低分辨率的图像数据。
S509’、CPU通知NPU第N帧图像的高分辨率和低分辨率。
S510、GPU生成低分辨率的图像数据。
S511、GPU基于共享内存的指针地址，将低分辨率的图像数据写入到共享内存中。
S512、CPU通知NPU从共享内存中读取低分辨率的图像数据。
S513、NPU基于共享内存的指针地址，从共享内存中读取低分辨率的图像数据。
S514’、NPU确定AI超分辨率模型的超分倍数。
S514、NPU利用AI超分辨率模型，基于超分倍数对低分辨率的图像数据进行超分辨率渲染，得到高分辨率的图像数据。
S515、NPU基于共享内存的指针地址，将高分辨率的图像数据写入到共享内存中。
S516、CPU通知GPU从共享内存中读取高分辨率的图像数据。
S517、GPU基于共享内存的指针地址，从共享内存中读取高分辨率的图像数据.
S518、GPU基于高分辨率的图像数据，对当前处理的FB进行渲染处理，得到渲染结果，如第N帧图像中元素的图像数据。
S519、GPU在得到所有FB的渲染结果后，基于渲染结果，将第N帧图像显示在显示屏上。
本申请的一些实施例提供一种电子设备，电子设备包括：第一处理器、第二处理器和存储器；其中，存储器用于存储一个或多个计算机程序代码，计算机程序代码包括计算机指令，当第一处理器和第二处理器执行计算机指令时，第一处理器和第二处理器执行上述渲染方法。
本申请的一些实施例提供一种芯片系统，芯片系统包括程序代码，当程序代码在电子 设备上运行时，使得电子设备中的第一处理器和第二处理器执行上述渲染方法。
本申请的一些实施例提供一种处理器，处理器为第二处理器，第二处理器包括处理单元和存储器；其中，存储器用于存储一个或多个计算机程序代码，计算机程序代码包括计算机指令，当第二处理器执行计算机指令时，第二处理器执行上述渲染方法。
本申请的一些实施例提供一种计算机存储介质，计算机存储介质包括计算机指令，当计算机指令在电子设备上运行时，使得电子设备中的第二处理器执行上述渲染方法。
本实施例还提供了一种控制设备，所述控制设备包括一个或多个处理器、存储器，所述存储器用于存储一个或多个计算机程序代码，所述计算机程序代码包括计算机指令，当所述一个或多个处理器执行所述计算机指令时，所述控制设备执行上述方法。该控制设备可以是一个集成电路IC，也可以是一个片上系统SOC。其中集成电路可以是通用集成电路，也可以是一个现场可编程门阵列FPGA，也可以是一个专用集成电路ASIC。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。上述描述的系统，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本实施例所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本实施例各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器执行各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：快闪存储器、移动硬盘、只读存储器、随机存取存储器、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (32)

  1. 一种渲染方法，其特征在于，所述方法应用于电子设备中，所述电子设备运行有应用程序，所述电子设备包括第一处理器和第二处理器，所述方法包括：
    所述第一处理器接收所述应用程序下发的渲染命令，所述渲染命令用于指示所述第二处理器基于第一分辨率对第一图像进行渲染；
    所述第一处理器向所述第二处理器发送渲染指令，所述渲染指令用于指示所述第二处理器对所述第一图像进行渲染；
    所述第二处理器基于所述渲染指令，生成所述第一图像在第二分辨率下的图像数据，所述第二分辨率不大于所述第一分辨率；
    所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存；
    所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据，所述第三分辨率大于所述第二分辨率；
    所述第二处理器基于所述第一图像在所述第三分辨率下的图像数据，生成所述第一图像。
  2. 根据权利要求1所述的方法，其特征在于，若所述应用程序的渲染方式为前向渲染，所述渲染指令对应于第一帧缓冲，所述第一帧缓冲执行绘制指令的数量大于预设阈值；
    若所述应用程序的渲染方式为延迟渲染，所述渲染指令对应于所述应用程序下发的所有帧缓冲中除最后一个帧缓冲之外的帧缓冲。
  3. 根据权利要求2所述的方法，其特征在于，所述第一帧缓冲是所述所有帧缓冲中执行绘制指令的数量最多的帧缓冲。
  4. 根据权利要求2或3所述的方法，其特征在于，在所述第一处理器向所述第二处理器发送渲染指令之前，所述方法还包括：
    所述第一处理器从所述应用程序的配置文件中获取所述应用程序的渲染方式。
  5. 根据权利要求1所述的方法，其特征在于，所述渲染指令用于指示所述第二处理器基于所述第二分辨率对所述第一图像进行渲染，所述第二分辨率小于所述第一分辨率。
  6. 根据权利要求1至5中任一项所述的方法，其特征在于，所述第二分辨率小于所述第一分辨率；
    所述第三分辨率与所述第一分辨率相同，或者，所述第三分辨率大于所述第一分辨率。
  7. 根据权利要求1至4中任一项所述的方法，其特征在于，所述第二分辨率等于所述第一分辨率。
  8. 根据权利要求1至7中任一项所述的方法，其特征在于，所述电子设备还包括第三处理器，所述第一图像在所述第三分辨率下的图像数据由所述第三处理器生成。
  9. 根据权利要求8所述的方法，其特征在于，所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存之后，所述方法还包括：
    所述第三处理器从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据；
    所述第三处理器基于所述第一图像在所述第二分辨率下的图像数据，生成所述第一图像在所述第三分辨率下的图像数据；
    所述第三处理器将所述第一图像在所述第三分辨率下的图像数据写入到所述第一内存。
  10. 根据权利要求9所述的方法，其特征在于，所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存之前，所述方法还包括：
    所述第二处理器将所述第一图像在第二分辨率下的图像数据写入第二内存，所述第二处理器具有访问所述第二内存和所述第一内存的权限，所述第三处理器具有访问所述第一内存的权限；
    所述第二处理器向所述第一处理器发送第一通知，所述第一通知用于指示所述第一图像在所述第二分辨率下的图像数据成功写入所述第二内存；
    响应于接收所述第一通知，所述第一处理器向所述第二处理器发送第二通知，所述第二通知用于指示所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到所述第一内存中，所述第二通知中携带有所述第一内存的地址指针；
    响应于接收所述第二通知，所述第二处理器从所述第二内存中读取所述第一图像在第二分辨率下的图像数据。
  11. 根据权利要求9或10所述的方法，其特征在于，所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存之后，所述第三处理器从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据之前，所述方法还包括：
    所述第一处理器向所述第三处理器发送第三通知，所述第三通知用于指示所述第三处理器从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据。
  12. 根据权利要求9至11中任一项所述的方法，其特征在于，所述第三处理器将所述第一图像在所述第三分辨率下的图像数据写入到所述第一内存之后，所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据之前，所述方法还包括：
    所述第一处理器向所述第二处理器发送第四通知，所述第四通知用于指示所述第二处理器从所述第一内存中读取所述第一图像在所述第三分辨率下的图像数据，所述第四通知中携带有所述第一内存的地址指针。
  13. 根据权利要求1至12中任一项所述的方法，其特征在于，所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据之后，所述方法还包括：
    所述第二处理器将所述第一图像在第三分辨率下的图像数据写入第二内存，所述第二处理器具有访问所述第二内存和所述第一内存的权限，所述第三处理器具有访问所述第一内存的权限。
  14. 根据权利要求1至13中任一项所述的方法，其特征在于，所述电子设备还包括第三处理器，所述第一图像在所述第三分辨率下的图像数据由所述第三处理器生成；
    在所述第一处理器向所述第二处理器发送渲染指令之前，所述方法还包括：
    所述第一处理器从第三内存中分配出所述第一内存，所述第一处理器具备访问所述第三内存的权限；
    所述第一处理器向所述第三处理器发送所述第一内存的指针地址，所述第三处理器基于所述指针地址向所述第一内存中写入所述第一图像在所述第三分辨率下的图像数据、以及基于所述指针地址从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据。
  15. 根据权利要求1至7中任一项所述的方法，其特征在于，所述电子设备还包括第三处理器，所述第一图像在所述第三分辨率下的图像数据由所述第三处理器生成；
    在所述第一处理器向所述第二处理器发送渲染指令之前，所述方法还包括：
    所述第一处理器从硬件缓冲区中分配出所述第一内存；
    所述第一处理器向所述第三处理器和所述第二处理器发送所述第一内存的指针地址，所述第一处理器、所述第二处理器和所述第三处理器具有访问所述第一内存的权限，所述第三处理器和所述第二处理器基于所述指针地址，对所述第一内存执行图像数据的读写。
  16. 根据权利要求15所述的方法，其特征在于，所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存之后，所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据之前，所述方法还包括：
    所述第三处理器从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据；
    所述第三处理器基于所述第一图像在所述第二分辨率下的图像数据，生成所述第一图像在所述第三分辨率下的图像数据；
    所述第三处理器将所述第一图像在所述第三分辨率下的图像数据写入到所述第一内存。
  17. 根据权利要求16所述的方法，其特征在于，所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存之后，所述第三处理器从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据之前，所述方法还包括：
    所述第一处理器向所述第三处理器发送第五通知，所述第五通知用于指示所述第三处理器从所述第一内存中读取所述第一图像在所述第二分辨率下的图像数据。
  18. 根据权利要求16或17所述的方法，其特征在于，所述第三处理器将所述第一图像在所述第三分辨率下的图像数据写入到所述第一内存之后，所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据之前，所述方法还包括：
    所述第一处理器向所述第二处理器发送第六通知，所述第六通知用于指示所述第二处理器从所述第一内存中读取所述第一图像在所述第三分辨率下的图像数据。
  19. 根据权利要求1至18中任一项所述的方法，其特征在于，所述电子设备还包括第三处理器，所述第一图像在所述第三分辨率下的图像数据由所述第三处理器生成；
    所述第三处理器中运行有人工智能超分辨率模型，所述第三处理器利用所述人工智能超分辨率模型，对所述第一图像在所述第二分辨率下的图像数据进行超分辨率渲染，生成所述第一图像在所述第三分辨率下的图像数据。
  20. 根据权利要求19所述的方法，其特征在于，所述第三处理器利用所述人工智能超分辨率模型，对所述第一图像在所述第二分辨率下的图像数据进行超分辨率渲染之前，所述方法还包括：
    所述第一处理器向所述所述第三处理器发送所述第一分辨率和所述第二分辨率；
    所述第三处理器基于所述第一分辨率和所述第二分辨率，确定所述人工智能超分辨率模型的超分倍数，所述人工智能分辨率模型基于所述超分倍数，对所述第一图像在所述第二分辨率下的图像数据进行超分辨率渲染。
  21. 根据权利要求19或20所述的方法，其特征在于，所述第三处理器利用所述人工智能超分辨率模型，对所述第一图像在所述第二分辨率下的图像数据进行超分辨率渲染之前，所述方法还包括：所述第一处理器对所述人工智能超分辨率模型进行初始化，所述初始化用于确定运行所述人工智能超分辨率模型以及确定所述人工智能超分辨率模型的正常 运行；
    所述初始化包括运行时检测、模型加载、模型编译以及内存配置，所述运行时检查用于确定运行所述人工智能超分辨率模型，所述模型加载、所述模型编译以及所述内存配置用于确定所述人工智能超分辨率模型的正常运行。
  22. 根据权利要1至21中任一项所述的方法，其特征在于，在所述第一处理器向所述第二处理器发送所述渲染指令之前，所述方法还包括：
    所述第一处理器将所述第一图像的分辨率从所述第一分辨率降低至所述第二分辨率。
  23. 根据权利要求22所述的方法，其特征在于，所述电子设备还包括第三处理器，所述第一图像在所述第三分辨率下的图像数据由所述第三处理器生成；所述第三处理器具有超分倍数，所述超分倍数用于指示所述第二分辨率和所述第三分辨率之间的差异；
    所述第三分辨率和所述第一分辨率相同，所述第一处理器将所述第一图像的分辨率从所述第一分辨率降低至所述第二分辨率包括：所述第一处理器基于所述超分倍数，将所述第一图像的分辨率从所述第一分辨率降低至所述第二分辨率。
  24. 根据权利要求8、14、15、19和23中任一项所述的方法，其特征在于，所述第三处理器是神经网络处理器或数字信号处理器。
  25. 一种渲染方法，其特征在于，所述方法应用于电子设备的第二处理器中，所述电子设备运行有应用程序，所述电子设备还包括第一处理器，所述应用程序向所述第一处理器下发渲染命令，所述渲染命令用于指示所述第二处理器基于第一分辨率对第一图像进行渲染，所述方法包括：
    所述第二处理器接收所述第一处理器发送的渲染指令，所述渲染指令用于指示所述第二处理器对所述第一图像进行渲染；
    所述第二处理器基于所述渲染指令，生成所述第一图像在第二分辨率下的图像数据，所述第二分辨率不大于所述第一分辨率；
    所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存；
    所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据，所述第三分辨率大于所述第二分辨率；
    所述第二处理器基于所述第一图像在所述第三分辨率下的图像数据，生成所述第一图像。
  26. 根据权利要求25所述的方法，其特征在于，所述第二处理器将所述第一图像在所述第二分辨率下的图像数据写入到第一内存之前，所述方法还包括：
    所述第二处理器将所述第一图像在第二分辨率下的图像数据写入第二内存，所述第二处理器具有访问所述第二内存和所述第一内存的权限，所述第三处理器具有访问所述第一内存的权限；
    所述第二处理器向所述第一处理器发送第一通知，所述第一通知用于指示所述第一图像在第二分辨率下的图像数据成功写入所述第二内存；
    响应于所述第一处理器发送的第二通知，所述第二处理器从所述第二内存中读取所述第一图像在第二分辨率下的图像数据，所述第二通知在所述第一处理器接收到所述第一通知后发送，所述第二通知用于指示所述第二处理器将所述第一图像在第二分辨率下的图像 数据写入到所述第一内存中，所述第二通知中携带有所述第一内存的地址指针。
  27. 根据权利要求25或26所述的方法，其特征在于，所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据之后，所述方法还包括：
    所述第二处理器将所述第一图像在第三分辨率下的图像数据写入第二内存，所述第二处理器具有访问所述第二内存和所述第一内存的权限；所述第一图像在第三分辨率下的图像数据由所述电子设备中的第三处理器生成，所述第三处理器具有访问所述第一内存的权限。
  28. 根据权利要求25至27中任一项所述的方法，其特征在于，所述第二处理器从所述第一内存中读取所述第一图像在第三分辨率下的图像数据之前，所述方法还包括：
    所述第二处理器接收所述第一处理器发送的通知，所述通知用于指示所述第二处理器从所述第一内存中读取所述第一图像在所述第三分辨率下的图像数据，所述通知中携带有所述第一内存的地址指针。
  29. 一种电子设备，其特征在于，所述电子设备包括：第一处理器、第二处理器和存储器；其中，所述存储器用于存储一个或多个计算机程序代码，所述计算机程序代码包括计算机指令，当所述第一处理器和所述第二处理器执行所述计算机指令时，所述第一处理器和所述第二处理器执行如权利要求1至24中任意一项所述的渲染方法。
  30. 一种芯片系统，其特征在于，所述芯片系统包括程序代码，当所述程序代码在电子设备上运行时，使得所述电子设备中的第一处理器和第二处理器执行如权利要求1至24中任意一项所述的渲染方法。
  31. 一种处理器，其特征在于，所述处理器为第二处理器，所述第二处理器包括处理单元和存储器；其中，所述存储器用于存储一个或多个计算机程序代码，所述计算机程序代码包括计算机指令，当所述第二处理器执行所述计算机指令时，所述第二处理器执行如权利要求25至28中任意一项所述的渲染方法。
  32. 一种计算机存储介质，其特征在于，所述计算机存储介质包括计算机指令，当所述计算机指令在电子设备上运行时，使得所述电子设备中的第二处理器执行如权利要求25至28中任意一项所述的渲染方法。
PCT/CN2022/114771 2021-11-17 2022-08-25 渲染方法及装置 WO2023087827A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP22865904.1A EP4207047A4 (en) 2021-11-17 2022-08-25 DISPLAY METHOD AND APPARATUS

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111364418.7 2021-11-17
CN202111364418 2021-11-17
CN202111554611.7 2021-12-17
CN202111554611.7A CN116137015A (zh) 2021-11-17 2021-12-17 渲染方法及装置

Publications (3)

Publication Number Publication Date
WO2023087827A1 true WO2023087827A1 (zh) 2023-05-25
WO2023087827A9 WO2023087827A9 (zh) 2023-12-14
WO2023087827A8 WO2023087827A8 (zh) 2024-01-04

Family

ID=86334375

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/114771 WO2023087827A1 (zh) 2021-11-17 2022-08-25 渲染方法及装置

Country Status (3)

Country Link
EP (1) EP4207047A4 (zh)
CN (1) CN116137015A (zh)
WO (1) WO2023087827A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2574721A (en) * 2019-05-23 2019-12-18 Serif Europe Ltd Image rendering
CN111681167A (zh) * 2020-06-03 2020-09-18 腾讯科技（深圳）有限公司 画质调整方法和装置、存储介质及电子设备
CN113284054A (zh) * 2020-02-19 2021-08-20 华为技术有限公司 图像增强方法以及图像增强装置
CN114998087A (zh) * 2021-11-17 2022-09-02 荣耀终端有限公司 渲染方法及装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9082196B2 (en) * 2008-08-20 2015-07-14 Lucidlogix Technologies Ltd. Application-transparent resolution control by way of command stream interception

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2574721A (en) * 2019-05-23 2019-12-18 Serif Europe Ltd Image rendering
CN113284054A (zh) * 2020-02-19 2021-08-20 华为技术有限公司 图像增强方法以及图像增强装置
CN111681167A (zh) * 2020-06-03 2020-09-18 腾讯科技（深圳）有限公司 画质调整方法和装置、存储介质及电子设备
CN114998087A (zh) * 2021-11-17 2022-09-02 荣耀终端有限公司 渲染方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4207047A4

Also Published As

Publication number Publication date
WO2023087827A9 (zh) 2023-12-14
WO2023087827A8 (zh) 2024-01-04
CN116137015A (zh) 2023-05-19
EP4207047A1 (en) 2023-07-05
EP4207047A4 (en) 2024-06-12

Similar Documents

Publication Publication Date Title
US8982136B2 (en) Rendering mode selection in graphics processing units
US9990690B2 (en) Efficient display processing with pre-fetching
WO2024040815A1 (zh) 图形处理方法及系统
JP2018512644A (ja) 低品質タイルを使用してメモリ帯域幅を減らすためのシステムおよび方法
CN114998087B (zh) 渲染方法及装置
US9449585B2 (en) Systems and methods for compositing a display image from display planes using enhanced blending hardware
CN116821040B (zh) 基于gpu直接存储器访问的显示加速方法、装置及介质
CN111311478B (zh) 一种gpu渲染核数据的预读取方法、装置及计算机存储介质
CN115934383A (zh) Wayland合成器下多显卡渲染方法
CN113674132B (zh) 一种通过检测显卡能力切换窗口管理渲染后端的方法
CN108563519A (zh) 基于gpgpu中的2d桌面块拷贝填充操作的实现
WO2023087827A1 (zh) 渲染方法及装置
WO2017000605A1 (zh) 片上系统、图形绘制方法、中间层及嵌入式设备、介质
WO2023093779A1 (zh) 界面生成方法及电子设备
CN113994363A (zh) 用于波隙管理的方法和装置
US10110927B2 (en) Video processing mode switching
US9251557B2 (en) System, method, and computer program product for recovering from a memory underflow condition associated with generating video signals
CN116137675A (zh) 渲染方法及装置
US9472168B2 (en) Display pipe statistics calculation for video encoder
US10678553B2 (en) Pro-active GPU hardware bootup
CN116348904A (zh) 用simo方法优化gpu内核以用于利用gpu高速缓存进行缩小
US9317891B2 (en) Systems and methods for hardware-accelerated key color extraction
US20240070962A1 (en) Graphics processing method and system
WO2023225771A1 (en) Concurrent frame buffer composition scheme
WO2024044936A1 (en) Composition for layer roi processing

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 18246228

Country of ref document: US

ENP Entry into the national phase

Ref document number: 2022865904

Country of ref document: EP

Effective date: 20230314