WO2019071600A1

WO2019071600A1 - Image processing method and apparatus

Info

Publication number: WO2019071600A1
Application number: PCT/CN2017/106153
Authority: WO
Inventors: 党茂昌; 符玉襄; 周喜渝; 孟坤; 蒋铭辉; 陈国栋; 余先宇
Original assignee: 华为技术有限公司
Priority date: 2017-10-13
Filing date: 2017-10-13
Publication date: 2019-04-18
Also published as: CN109891388A

Abstract

Provided are an image processing method and apparatus, wherein same relate to the field of image processing, and are capable of solving the problem of the rate at which a terminal device processes an image frame being low. The method comprises: acquiring a first rendering command queue, wherein the first rendering command queue comprises multiple rendering commands corresponding to an Nth image frame, and N is greater than or equal to 1; acquiring a first calculation command queue, wherein the first calculation command queue comprises multiple calculation commands corresponding to an (N+1)th image frame; and according to the first rendering command queue, rendering the Nth image frame, and during the process of rendering the Nth image frame, calculating all pixels of the (N+1)th image frame according to a second calculation command queue.

Description

Image processing method and device

Technical field

The embodiments of the present invention relate to the field of image processing technologies, and in particular, to an image processing method and apparatus.

Background technique

During the display of the video, for each image frame in the video, the terminal device performs frame processing (ie, calculation), rendering, and display of these three tasks.

At present, the existing terminal device serially performs a computing task and a rendering task. Illustratively, after the terminal device performs a computing task and a rendering task on the Pth (P≥1) image frames, the P+1th image frame is further performed. Perform calculation tasks and render tasks. Thus, in a scenario where the video includes a large number of image frames, the rate at which the terminal device processes the image frames is lower.

Summary of the invention

The present application provides an image processing method and apparatus, which can solve the problem that a rate at which a terminal device processes an image frame is low.

To achieve the above objectives, the present application adopts the following technical solutions:

In a first aspect, an image processing method is provided, the image processing apparatus acquiring a first rendering command queue including a plurality of rendering commands corresponding to the Nth (N≥1) image frames, and including the N+1th image frame After the calculation command queue of the corresponding multiple calculation commands, the Nth image frame is rendered according to the first rendering command queue, and in the process of rendering the Nth image frame, the N+1 is calculated according to the second calculation command queue. All pixels of an image frame.

The image processing apparatus in the embodiment of the present application separates the rendering task from the computing task, and in the process of rendering the Nth image frame, calculates all the pixels of the N+1th image frame, and realizes the parallel of the rendering task and the computing task. Execution effectively increases the rate at which the image processing device processes image frames.

Optionally, in a possible implementation manner of the present application, in the method for processing an image frame by the image processing apparatus, the image processing apparatus further acquires a parallel processing message, where the parallel processing message is used to notify that the Nth image is being rendered. During the frame, all pixels of the N+1th image frame are calculated. In this way, the method of “retrieving the Nth image frame according to the first rendering command queue and calculating all the pixels of the N+1th image frame according to the second calculation command queue” is specifically: the image processing apparatus adopts the first thread. Executing a rendering command included in the first rendering command queue to render an Nth image frame; the image processing device uses a second thread to execute a calculation command response included in the first computing command queue, and calculates an N+1th image frame; and responds to the parallel processing message The image processing apparatus runs the first thread and the second thread in parallel.

The image processing device uses the first thread to complete the rendering task, and uses the second thread to complete the computing task. Under the action of the parallel processing message, the image processing device realizes the calculation of the N+1th image frame in the process of rendering the Nth image frame. All pixels.

Optionally, in another possible implementation manner of the present application, after acquiring the first rendering command queue, the image processing apparatus further calculates a feature value of the first rendering command queue according to a preset algorithm, and stores the first A feature value that renders the command queue. Correspondingly, when N>1, “the image processing apparatus according to the first rendering command queue, The method of rendering the Nth image frame is: when the image processing apparatus determines that the feature value of the first rendering command queue is equal to the feature value of the stored target rendering command queue, acquiring the stored Xth corresponding to the target rendering command queue ( N>X≥1) the rendering result of the image frames, and rendering the Nth image frame according to the rendering result of the Xth image frame.

In the case of N>1, when the image processing apparatus determines that the feature value of the target rendering command queue equal to the feature value of the first rendering command queue is locally stored, the target rendering command queue includes a plurality of corresponding to the Xth image frame. When the command is rendered, the Nth image frame is overlapped with the Xth image frame, so that the image processing apparatus directly renders the Nth image frame according to the rendering result of the Xth image frame, without further according to the first rendering command. Each rendering command in the queue renders the Nth image frame, reducing the hardware resources used in image rendering and reducing the power consumption of the image processing device.

The image processing apparatus may determine, according to the feature value of the rendering command queue, whether the image frames corresponding to the two rendering command queues are repeated. After calculating the feature value of the first rendering command queue, the image processing device stores the feature value of the first rendering command queue to facilitate determining whether the subsequent image frame is repeated with the Nth image frame.

Optionally, in another possible implementation manner of the present application, the image processing apparatus further stores a rendering result of the Nth image frame after the Nth image frame is rendered. Thus, when processing the image frame that is subsequently repeated with the Nth image frame, the image processing apparatus may render the image frame with reference to the rendering result of the stored Nth image frame.

In a second aspect, an image processing apparatus is provided, the image processing apparatus comprising an acquisition unit, a rendering unit, and a computing unit. Specifically, the acquiring unit is configured to acquire a first rendering command queue, where the first rendering command queue includes multiple rendering commands corresponding to the Nth image frame, N≥1, and is used to obtain a first computing command queue, A calculation command queue includes a plurality of calculation commands corresponding to the (N+1)th image frame. The rendering unit is configured to render the Nth image frame according to the first rendering command queue acquired by the acquiring unit. The calculating unit is configured to calculate, in the process of rendering the Nth image frame by the rendering unit, all the pixels of the (N+1)th image frame according to the second calculation command queue acquired by the acquiring unit.

Further, in a possible implementation manner of the present application, the acquiring unit is further configured to acquire a parallel processing message, where the parallel processing message is used to notify that, in the process of rendering the Nth image frame, calculate the N+1th All pixels of the image frame. The rendering unit is specifically configured to execute a rendering command included in the first rendering command queue by using the first thread to render the Nth image frame. The calculating unit is specifically configured to execute, by using the second thread, a calculation command response included in the first calculation command queue, calculate an N+1th image frame, and specifically, in response to the parallel processing message acquired by the acquiring unit, in the foregoing rendering In the process of rendering the Nth image frame by the unit, all pixels of the N+1th image frame are calculated.

Further, in another possible implementation manner of the present application, the calculating unit is specifically configured to calculate, according to a preset algorithm, a feature value of the first rendering command queue acquired by the acquiring unit. The image processing apparatus provided by the embodiment of the present application further includes a storage unit, configured to store a feature value of the first rendering command queue calculated by the calculating unit. The image processing apparatus provided by the embodiment of the present application further includes a determining unit, configured to determine that the feature value of the first rendering command queue calculated by the calculating unit is equal to the stored first value, and the first value is a target rendering command queue. The feature value, the target rendering command queue includes a plurality of rendering commands corresponding to the Xth image frame, N>X≥1. The obtaining unit is further configured to acquire a rendering result of the stored Xth image frame. The rendering unit is specifically configured to be used according to the rendering result of the Xth image frame acquired by the acquiring unit. Render the Nth image frame.

Further, in another possible implementation manner of the present application, the image processing apparatus provided by the embodiment of the present application further includes a storage unit, configured to store the Nth after the rendering unit renders the Nth image frame. The rendering result of the image frames.

In a third aspect, an image processing apparatus is provided, the image processing apparatus comprising: one or more processors, a memory, and a communication interface. The memory, communication interface is coupled to one or more processors via a system bus. The image processing device communicates with other devices through a communication interface. The memory is for storing computer program code, the computer program code comprising instructions, when the one or more processors execute the instructions stored in the memory, the image processing apparatus performs the image processing method as described in the first aspect above and any possible implementation thereof .

In a fourth aspect, a computer storage medium is provided having stored therein computer program code. When the processor of the image processing apparatus in the third aspect executes the computer program code, the image processing apparatus performs the image processing method as in the first aspect described above and any possible implementation thereof.

In a fifth aspect, there is also provided a computer program product comprising instructions which, when run on an image processing device, cause the image processing device to perform an image processing method as in the first aspect described above and any one of its possible implementations.

In the present application, the name of the above image processing apparatus is not limited to the device or the function module itself. In actual implementation, these devices or function modules may appear under other names. As long as the functions of the respective devices or functional modules are similar to the present application, they are within the scope of the claims and their equivalents.

For a detailed description of the second aspect, the third aspect, the fourth aspect, the fifth aspect, and various implementations thereof, reference may be made to the detailed description in the first aspect and various implementations thereof; and the second aspect, For the beneficial effects of the third aspect, the fourth aspect, the fifth aspect, and various implementations thereof, reference may be made to the beneficial effects in the first aspect and various implementation manners thereof, and details are not described herein again.

These and other aspects of the embodiments of the present application will be more apparent from the following description.

DRAWINGS

1 is a schematic flow chart of a method for processing an image frame by a GPU in the prior art;

2 is a schematic structural diagram of hardware of a mobile phone according to an embodiment of the present application;

FIG. 3 is a schematic flowchart diagram of an image processing method according to an embodiment of the present disclosure;

4 is a schematic flowchart of a method for processing an image frame by a GPU according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram 1 of an image processing apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram 2 of an image processing apparatus according to an embodiment of the present disclosure.

Detailed ways

The terms "first", "second", and "third" and the like in the specification and claims of the present application and the above drawings are used to distinguish different objects, and are not intended to limit the specific order.

In the embodiments of the present application, the words "exemplary" or "such as" are used to mean an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the words "exemplary" or "such as" is intended to present the concepts in a particular manner.

During the display of the video, for each image frame in the video, the terminal device performs three tasks of calculating, rendering, and displaying. Specifically, a graphics processing unit (Graphics Processing Unit, built in the terminal device) The GPU) performs computational tasks as well as rendering tasks and displays the rendered image frames on the display.

Currently, terminal devices serially perform computational tasks and rendering tasks. Exemplarily, as shown in FIG. 1 , after the GPU in the terminal device performs the calculation task of the Pth (P≥1) image frames, the rendering task of the Pth image frame is performed; and the rendering of the Pth image frame is performed. After the task, the GPU performs the calculation task of the P+1th image frame; after performing the calculation task of the P+1th image frame, the GPU performs the rendering task of the P+1th image frame, and so on, the GPU Perform the rendering task of the last image frame. It can be seen that in a scenario including a large number of image frames, the rate at which the terminal device in the method processes the image frame is relatively large.

In addition, in a scenario where the video includes a large number of image frames, there are multiple image frames that may be duplicated. For the repeated image frames, the terminal device in the above method needs to re-execute the computing task and the rendering task, further reducing the processed image frame. The rate, and the power consumption of the terminal device is large.

The image processing device separates the computing task from the rendering task, and the parallel computing processing task and the rendering task. The image processing apparatus calculates all the pixels of the (N+1)th image frame in the process of rendering the Nth image frame, effectively improving the rate of processing the image frame.

The image processing apparatus in the embodiment of the present application may be a mobile phone, an augmented reality (AR), a virtual reality (VR) device, a tablet computer, a notebook computer, or a super mobile personal computer (UMP). Any terminal device such as a netbook or a personal digital assistant (PDA). Of course, in the following embodiments, the specific form of the image processing device is not limited.

As shown in FIG. 2, the image processing apparatus in the embodiment of the present application may be the mobile phone 100. The embodiment will be specifically described below by taking the mobile phone 100 as an example. It should be understood that the illustrated mobile phone 100 is only one example of the image processing device described above, and the mobile phone 100 may have more or fewer components than those shown in the figures, and two or more components may be combined. Or it can have different component configurations.

As shown in FIG. 2, the mobile phone 100 may specifically include: a processor 101, a radio frequency (RF) circuit 102, a memory 103, a touch screen 104, a Bluetooth device 105, one or more sensors 106, a Wi-Fi device 107, and positioning. Components such as device 108, audio circuit 109, peripheral interface 110, and power system 111. These components can communicate over one or more communication buses or signal lines (not shown in Figure 2). It will be understood by those skilled in the art that the hardware structure shown in FIG. 2 does not constitute a limitation to the mobile phone, and the mobile phone 100 may include more or less components than those illustrated, or some components may be combined, or different component arrangements.

The various components of the mobile phone 100 will be specifically described below with reference to FIG. 2:

The processor 101 is a control center of the mobile phone 100, and connects various parts of the mobile phone 100 by using various interfaces and lines, and executes the mobile phone 100 by running or executing an application stored in the memory 103 and calling data stored in the memory 103. Various functions and processing data. In some embodiments, processor 101 can include one or more processing units. Optionally, the processor 101 can integrate an application processor and a modem processor. The application processor mainly processes an operating system, a user interface, an application, and the like; the modem processor mainly processes wireless communication. Optionally, the above modulation and demodulation processor and the application processor may also be independently set.

In the embodiment of the present application, the processor 101 may include a GPU 115 and a central processing unit (Central) The processing unit (CPU) 116 may also be a combination of the GPU 115, the CPU 116, the digital signal processing (DSP), and a control chip (for example, a baseband chip) in the communication unit. In the embodiment of the present application, both the GPU 115 and the CPU 116 may be a single operation core, and may also include multiple operation cores.

Among them, the GPU 115 is a microprocessor that performs image computing operations on personal computers, workstations, game consoles, and some mobile devices (such as tablets, smart phones, etc.). It can convert the display information required by the mobile phone 100 and provide a line scan signal to the display 104-2 to control the correct display of the display 104-2.

Specifically, during the display process, the mobile phone 100 may send a corresponding drawing command to the GPU 115. For example, the drawing command may be “drawing a rectangle having a length and a width of a×b at the coordinate position (x, y). Then, the GPU 115 can quickly calculate all the pixels of the graphic according to the drawing instruction, and draw corresponding graphics on the specified position on the display 104-2.

It should be noted that the GPU 115 may be integrated in the processor 101 in the form of a functional module, or may be disposed in the mobile phone 100 in a separate physical form (for example, a video card), which is not limited in this embodiment.

The GPU 115 performs computational tasks and rendering tasks for image frames. Specifically, the GPU 115 in the embodiment of the present application includes multiple Stream Processor (SM) units. The SM unit is mainly used to process data transmitted by the CPU and convert the processed data into a digital signal that the display 104-2 can recognize. That is to say, the SM unit in the GPU 115 completes the calculation task and the rendering task of the image frame.

The image processing apparatus in the embodiment of the present application uses an idle SM unit to render the Nth image frame, and in the process of rendering the Nth image frame, uses another idle SM unit to calculate the N+1th image frame. All pixels, separation between computational tasks and rendering tasks, and parallel processing, effectively increase the rate at which image frames are processed.

The radio frequency circuit 102 can be used to receive and transmit wireless signals during transmission or reception of information or calls. In particular, the radio frequency circuit 102 can process the downlink data of the base station and then process it to the processor 101; in addition, transmit the data related to the uplink to the base station. Generally, radio frequency circuits include, but are not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency circuit 102 can also communicate with other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global mobile communication systems, general packet radio services, code division multiple access, wideband code division multiple access, long term evolution, email, short message service, and the like.

The memory 103 is used to store applications and data, and the processor 101 executes various functions and data processing of the mobile phone 100 by running applications and data stored in the memory 103. The memory 103 mainly includes a storage program area and a storage data area, wherein the storage program area can store an operating system, an application required for at least one function (such as a sound playing function, an image playing function, etc.); the storage data area can be stored according to the use of the mobile phone. Data created at 100 o'clock (such as audio data, phone book, etc.). Further, the memory 103 may include high speed random access memory (RAM), and may also include nonvolatile memory such as a magnetic disk storage device, a flash memory device, or other volatile solid state storage device. The memory 103 can store various operating systems, for example, developed by Apple.

Operating system, developed by Google Inc.

Operating system, etc. The above memory 103 may be independent and connected to the processor 101 via the above communication bus; the memory 103 may also be integrated with the processor 101.

The touch screen 104 may specifically include a touch panel 104-1 and a display 104-2.

The touch panel 104-1 can collect touch events on or near the user of the mobile phone 100 (for example, the user uses any suitable object such as a finger, a stylus, or the like on the touch panel 104-1 or on the touchpad 104. The operation near -1), and the collected touch information is sent to other devices (for example, processor 101). The touch event of the user in the vicinity of the touch panel 104-1 may be referred to as a hovering touch; the hovering touch may mean that the user does not need to directly touch the touchpad in order to select, move or drag a target (eg, an icon, etc.) Instead, just the user is near the phone to perform the desired function. In addition, the touch panel 104-1 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves.

A display (also referred to as display) 104-2 can be used to display information entered by the user or information provided to the user as well as various menus of the mobile phone 100. The display 104-2 can be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The touchpad 104-1 can be overlaid on the display 104-2, and when the touchpad 104-1 detects a touch event on or near it, it is transmitted to the processor 101 to determine the type of touch event, and then the processor 101 may provide a corresponding visual output on display 104-2 depending on the type of touch event. Although in FIG. 2, the touchpad 104-1 and the display 104-2 are implemented as two separate components to implement the input and output functions of the handset 100, in some embodiments, the touchpad 104- 1 is integrated with the display screen 104-2 to implement the input and output functions of the mobile phone 100. It is to be understood that the touch screen 104 is formed by stacking a plurality of layers of materials. In the embodiment of the present application, only the touch panel (layer) and the display screen (layer) are shown, and other layers are not described in the embodiment of the present application. . In addition, the touch panel 104-1 may be disposed on the front surface of the mobile phone 100 in the form of a full-board, and the display screen 104-2 may also be disposed on the front surface of the mobile phone 100 in the form of a full-board, so that the front of the mobile phone can be borderless. Structure.

Optionally, the mobile phone 100 may also have a fingerprint recognition function. For example, the fingerprint recognition collector 112 can be configured on the back of the handset 100 (eg, below the rear camera) or the fingerprint collector 112 can be configured on the front side of the handset 100 (eg, below the touch screen 104). For another example, the fingerprint collector 112 can be configured in the touch screen 104 to implement the fingerprint recognition function, that is, the fingerprint collector 112 can be integrated with the touch screen 104 to implement the fingerprint recognition function of the mobile phone 100. In this case, the fingerprint collector 112 is disposed in the touch screen 104, may be part of the touch screen 104, or may be otherwise disposed in the touch screen 104. The main component of the fingerprint collector 112 in the embodiment of the present application is a fingerprint sensor, which can employ any type of sensing technology, including but not limited to optical, capacitive, piezoelectric or ultrasonic sensing technologies.

The mobile phone 100 can also include a Bluetooth device 105 for enabling data exchange between the handset 100 and other short-range terminal devices (eg, mobile phones, smart watches, etc.). The Bluetooth device in the embodiment of the present application may be an integrated circuit or a Bluetooth chip or the like.

The handset 100 can also include at least one type of sensor 106, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display of the touch screen 104 according to the brightness of the ambient light, and the proximity sensor may turn off the power of the display when the mobile phone 100 moves to the ear. . As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (usually three axes). When it is stationary, it can detect the magnitude and direction of gravity. It can be used to identify the gesture of the mobile phone (such as horizontal and vertical screen switching, related Game, magnetic (statue calibration), vibration recognition related functions (such as pedometer, tapping), etc.; other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which can be configured in the mobile phone 100, are not described here.

The Wi-Fi device 107 is configured to provide the mobile phone 100 with network access complying with the Wi-Fi related standard protocol, and the mobile phone 100 can access the Wi-Fi access point through the Wi-Fi device 107, thereby helping the user to send and receive emails, Browsing web pages and accessing streaming media, etc., it provides users with wireless broadband Internet access. In some other embodiments, the Wi-Fi device 107 can also function as a Wi-Fi wireless access point, and can provide Wi-Fi network access to other terminal devices.

The positioning device 108 is configured to provide a geographic location for the mobile phone 100. It can be understood that the positioning device 108 can be specifically a receiver of a positioning system such as a Global Positioning System (GPS) or a Beidou satellite navigation system, or a Russian GLONASS. After receiving the geographical location transmitted by the positioning system, the positioning device 108 sends the information to the processor 101 for processing, or sends it to the memory 103 for storage. In some other embodiments, the positioning device 108 can also be a receiver of an Assisted Global Positioning System (AGPS), which assists the positioning device 108 in performing ranging and positioning services by acting as an auxiliary server. In this case, the secondary location server provides location assistance over the wireless communication network in communication with the terminal device (e.g., the location device 108 of the handset 100, i.e., the GPS receiver). In still other embodiments, the positioning device 108 can also be a Wi-Fi access point based positioning technology. Since each Wi-Fi access point has a globally unique Media Access Control (MAC) address, the phone can scan and collect the surrounding Wi-Fi access points when Wi-Fi is turned on. The broadcast signal, so the MAC address broadcasted by the Wi-Fi access point can be obtained; the mobile phone sends the data (such as the MAC address) capable of indicating the Wi-Fi access point to the location server through the wireless communication network, and is retrieved by the location server. The geographic location of each Wi-Fi access point, combined with the strength of the Wi-Fi broadcast signal, calculates the geographic location of the mobile phone and sends it to the location device 108 of the handset.

The audio circuit 109, the speaker 113, and the microphone 114 can provide an audio interface between the user and the handset 100. The audio circuit 109 can transmit the converted electrical data of the received audio data to the speaker 113 for conversion to the sound signal output by the speaker 113; on the other hand, the microphone 114 converts the collected sound signal into an electrical signal by the audio circuit 109. After receiving, it is converted into audio data, and then the audio data is output to the RF circuit 102 for transmission to, for example, another mobile phone, or the audio data is output to the memory 103 for further processing.

The peripheral interface 110 is used to provide various interfaces for external input/output devices (such as a keyboard, a mouse, an external display, an external memory, a subscriber identity module card, etc.). For example, it is connected to the mouse through a Universal Serial Bus (USB) interface, and is connected to a Subscriber Identity Module (SIM) card provided by the service provider through a metal contact on the card slot of the subscriber identity module. . Peripheral interface 110 can be used to couple the external input/output peripherals described above to processor 101 and memory 103.

The mobile phone 100 may further include a power supply device 111 (such as a battery and a power management chip) that supplies power to the various components. The battery may be logically connected to the processor 101 through the power management chip to manage charging, discharging, and power management through the power supply device 111. And other functions.

Although not shown in FIG. 2, the mobile phone 100 may further include a camera (front camera and/or rear camera), A flash lamp, a micro projection device, a Near Field Communication (NFC) device, and the like are not described herein.

Hereinafter, an image processing method provided by an embodiment of the present application is described in detail in conjunction with a specific embodiment. As shown in FIG. 3, the image processing method includes:

S300. The image processing apparatus acquires a sequence of image frames to be processed during the running of the target application.

The target application may be any application installed in the image processing device that relies on the GPU for image calculation. The target application may be downloaded from the third-party application market, and may also be an application that is provided by the image processing device system, which is not specifically limited in this embodiment of the present application.

Exemplarily, the target application is AutoCAD (Auto Computer Aided Design), 3Ds Max (3D Studio Max), Pro/Engineer or video player.

The sequence of image frames to be processed includes a plurality of image frames to be processed. The image processing apparatus may directly acquire all the image frames to be processed included in the image frame sequence to be processed, and may also process each image frame to be processed in the image frame sequence in real time. This is not specifically limited.

Exemplarily, the target application is a live video player, and the content that the live video player needs to present to the user includes a character shape, an environment background, and the like. In the process of playing the live video, the image processing device renders different image frames by Express these elements. Since the live video is displayed in a frame manner, the image processing apparatus needs to acquire each image frame to be processed in real time.

S301. The image processing apparatus sequentially generates a calculation command queue and a rendering command queue of each image frame to be processed in the image sequence to be processed.

After acquiring the image frame to be processed, the image processing apparatus puts all the calculation commands of the image frame to be processed into the calculation command queue of the first preset type, and the image to be processed The frame is that all rendering commands are placed in the rendering command queue of the second preset type, so that the calculation command queue and the rendering command queue of the image frame to be processed can be generated.

It can be seen from the description of S300 that the image processing apparatus can acquire each image frame to be processed in real time, or directly acquire all the image frames to be processed included in the sequence of image frames to be processed. Regardless of which of the above manners the image processing apparatus acquires the image frame to be processed, the image processing apparatus generates a calculation command queue and a rendering command queue for each image frame to be processed in the order of the time at which the image frames to be processed are acquired.

Specifically, the CPU running the target application in the image processing apparatus generates a calculation command queue and a rendering command queue for each image frame to be processed.

For each image frame to be processed, after generating the calculation command queue and the rendering command queue of the image frame to be processed, the CPU sends the calculation command queue and the rendering command queue of the image frame to be processed to the GPU in the image processing apparatus. In this way, the GPU can calculate all the pixels of the image frame to be processed according to the obtained calculation command queue, and the GPU renders the image frame to be processed according to the obtained rendering command queue, and displays it on the display of the image processing apparatus. The image frame to be processed after rendering.

For convenience of description, the embodiment of the present application is described by taking a GPU to acquire a first calculation command queue and a first rendering command queue as an example. The first rendering command queue includes a plurality of rendering commands corresponding to the Nth image frame, N>1. The first calculation command queue includes a plurality of calculation commands corresponding to the (N+1)th image frame. Here, the Nth image frame and the (N+1)th image frame are both the image frames to be processed described above.

S302. The GPU of the image processing apparatus executes the rendering command included in the first rendering command queue by using the first thread to render the Nth image frame.

With reference to the above description, an idle SM unit (referred to as: the first SM unit) in the GPU of the image processing apparatus uses a first thread to execute a rendering command included in the first rendering command queue to complete a rendering task of the Nth image frame. Implement the rendering of the Nth image frame.

Specifically, the method for the first SM unit to render the Nth image frame is: the first SM unit first calculates a feature value of the first rendering command queue according to a preset algorithm; and then, the first SM unit determines whether the local storage unit stores The feature values of other render command queues whose feature values of the first render command queue are equal.

If the feature value of the first rendering command queue is equal to the feature value of the stored target rendering command queue (referred to as the first value), the target rendering command queue includes multiple corresponding to the Xth (N>X≥1) image frames. The rendering command indicates that the Nth image frame is overlapped with the Xth image frame, and the first SM unit can directly render the Nth image frame by using the rendering result of the X image frames, and the rendering result may include texture information and shadow information. Wait. Correspondingly, the first SM unit locally obtains a rendering result of the Xth (N>X≥1) image frames corresponding to the target rendering command queue, and renders the Nth image frame according to the rendering result of the X image frames. In this scenario, the process of rendering the Nth image frame does not need to occupy additional hardware resources, which can reduce the hardware resources used in image rendering and reduce the power consumption of the image processing device.

If no features of other rendering command queues equal to the feature values of the first rendering command queue are stored locally

The value indicates that the Nth image frame is changed compared to the previous image frame, and the first SM unit calls all rendering commands in the first rendering command queue to render the Nth image frame. In this scenario, the process of rendering the Nth image frame requires additional hardware resources such as memory, memory, and the like.

Optionally, the feature value of the rendering command queue in the embodiment of the present application may be a Secure Hash Algorithm (SHA) value of the rendering command queue. In this case, the image processing apparatus in the embodiment of the present application calculates the feature value of the first rendering command queue, that is, the image processing device calculates the SHA value of the first rendering command queue.

Specifically, the method for the image processing apparatus to calculate the SHA value of the first rendering command queue is: the first SM unit determines the first rendering command queue by calculating a stored SHA value of the plurality of rendering commands in the first rendering command queue. SHA value.

Optionally, the first SM unit may directly obtain the same target command queue as the first rendering command queue from the local storage, and directly compare the Xth image corresponding to the target command queue with the command in the first rendering command queue. The rendering result of the frame is the result of rendering the Nth image frame.

Specifically, after the image processing device renders the Nth image frame, the image processing device further stores the rendering result of the Nth image frame, which is convenient for rendering the subsequent repeated image frame.

S303. The GPU in the image processing apparatus uses a second thread to execute a calculation command response included in the first calculation command queue, and calculates an N+1th image frame.

Another idle SM unit (abbreviated as: the second SM unit) in the GPU of the image processing apparatus executes the calculation command response included in the first calculation command queue by using the second thread, and calculates the N+1th image frame.

S304. The image processing apparatus acquires a parallel processing message, where the line processing message is used to notify that all pixels of the (N+1)th image frame are calculated in the process of rendering the Nth image frame.

Specifically, in the process of generating the first rendering command queue and the first computing command queue, the CPU further generates and Line processing messages. After generating the parallel processing message, the CPU sends the parallel processing message to the GPU.

S305. The GPU in the image processing apparatus responds to the parallel processing message, and runs the first thread and the second thread in parallel.

Specifically, after receiving the parallel processing message sent by the CPU, the first SM unit and the second SM unit respond to the parallel processing message, so that the N+1th image frame is calculated during the process of rendering the Nth image frame. All pixels.

In summary, in the embodiment of the present application, two independent threads are used to complete the rendering task and the computing task, and in the process of rendering the Nth image frame, all pixels of the N+1th image frame are calculated. FIG. 4 shows a flow of processing an image frame by the image processing apparatus in the embodiment of the present application. As can be seen from FIG. 4, the GPU in the prior art performs the computing task and the rendering task serially, and the next image frame can be processed only after the current image frame is rendered. In the process of rendering the Nth image frame, the GPU in the embodiment of the present application can calculate all pixels of the N+1th image frame. Compared with the prior art, the image processing apparatus provided by the embodiment of the present application processes the image frame more efficiently.

In addition, for the repeated image frames, the GPU in the embodiment of the present application can directly render the repeated image frames according to the stored rendering results, reduce hardware resources occupied during image rendering, and reduce power consumption of the image processing apparatus.

An embodiment of the present application provides an image processing apparatus for performing the steps performed by the image processing apparatus in the above image processing method. The image processing apparatus provided by the embodiment of the present application may include a module corresponding to the corresponding step.

The embodiment of the present application may divide the function module into the image processing device according to the above method example. For example, each function module may be divided according to each function, or two or more functions may be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. The division of modules in the embodiments of the present application is schematic, and is only a logical function division, and may be further divided in actual implementation.

In the case where the respective functional modules are divided by corresponding functions, FIG. 5 shows a possible structural diagram of the image processing apparatus involved in the above embodiments. The image processing apparatus includes an acquisition unit 501, a calculation unit 502, a rendering unit 503, a storage unit 504, and a determination unit 505.

The obtaining unit 501 is configured to support the image processing apparatus to execute S300, S301, and S304 in FIG. 3; the calculating unit 502 is configured to support the image processing apparatus to execute S303 in FIG. 3; and the rendering unit 503 is configured to support the image processing apparatus to perform the image processing apparatus in FIG. S302; the storage unit 504 is configured to support the image processing device to store the rendering result of the rendered image frame, and is further configured to support the image processing device to store the SHA value of the rendering command queue of the rendered image frame; the determining unit 505 is configured to support the image processing device It is determined whether the feature value of the other rendering command queues having the same feature value as the first rendering command queue is stored locally. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.

In the case of employing an integrated unit, FIG. 6 shows a possible structural diagram of the image processing apparatus involved in the above embodiment. As shown in FIG. 6, the image processing apparatus includes a processing module 60 and a communication module 61. The processing module 60 is configured to control and manage the actions of the image processing apparatus. For example, the processing module 60 is configured to support the image processing apparatus to perform S301 to S305 in FIG. 3, and/or other techniques for the techniques described in the present application. process. The communication module 61 is for supporting communication of the image processing apparatus with an external device. For example, the communication module 61 is for supporting the image processing apparatus to execute S300 in FIG.

The processing module 60 in the embodiment of the present application may be a processor or a controller, and may be, for example, a CPU, a GPU, or a DSP. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the present disclosure.

The communication module 61 in the embodiment of the present application may be a Bluetooth module for interacting with external devices, and may further include an RF circuit corresponding to the Bluetooth module. The RF circuit is used to receive and transmit signals during the transmission or reception of information or during a call.

The communication module 61 in the embodiment of the present application may also be a communication interface for interacting with external devices, the full name of the English: Communication Interface. The communication module may include two communication interfaces, a transmission interface for transmitting data to an external device and a receiving interface for receiving data from the external device, that is, the image processing device may implement data through two different communication interfaces respectively. The receipt and transmission of data. Of course, the communication module 61 can integrate the data receiving function and the data transmitting function on a communication interface having a data receiving function and a data transmitting function. The communication interface can be integrated on a Bluetooth chip or an NFC chip.

The communication module 61 in the embodiment of the present application may also be a transceiver or a transceiver circuit or the like.

Optionally, as shown in FIG. 6, the image processing apparatus may further include an input module 62 for implementing interaction between the user and the image processing apparatus.

The input module 62 can receive numeric or character information input by the user to generate a signal input related to user setting or function control. In the specific implementation of the present application, the input module 62 may be a touch panel, or may be other human-computer interaction interfaces, such as physical input keys, microphones, etc., and may also be other external information capture devices, such as cameras.

In other implementations of the present application, the physical input keys employed by the input module 62 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, switch buttons, etc.). The input module 62 in the form of a microphone can collect the voice input by the user or the environment and convert it into a command executable by the processing module 60 in the form of an electrical signal.

Optionally, as shown in FIG. 6, the image processing apparatus may further include an output module 63 for implementing interaction between the user and the image processing apparatus.

The output module 63 includes, but is not limited to, an image output module and a sound output module. The image output module is used to output text, pictures and/or video. The image output module may include a display panel, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a field emission display (FED), or the like. Display panel. The image output module may include a single display panel or a plurality of display panels of different sizes. In the embodiment of the present application, the touch panel used by the input module 62 can also serve as the display panel of the output module 63. For example, when the touch panel detects a touch or proximity gesture operation thereon, it is transmitted to the processing module 60 to determine the type of the touch event, and then the processing module 60 provides a corresponding visual output on the display panel according to the type of the touch event. .

As shown in FIG. 6, the input module 62 and the output module 63 can function as two separate components to implement the input and output functions of the image processing apparatus, but in some embodiments, the input module 62 can be integrated with the output module 63. The input and output functions of the image processing apparatus are implemented (the input module 62 and the output module 63 are included in a broken line frame as shown in FIG. 6 to indicate that the input module 62 and the output module 63 are integrated into one body).

Optionally, the image processing apparatus in the embodiment of the present application further includes a storage module 64, configured to store the rendered The rendering result of the image frame is also used to support the image processing device to store the SHA value of the rendering command queue of the rendered image frame.

The processing module 60 in the embodiment of the present application may be the processor 101 in FIG. 2, the communication module 61 may be the antenna in FIG. 2, the input module 62 may be 112 and 104-1 in FIG. 2, and the output module 63 may be In the display 104-2 of FIG. 2, the storage module 64 may be the memory 103 of FIG.

When the image processing apparatus operates, the image processing apparatus executes the image processing method of the embodiment as shown in FIG. For a specific image processing method, refer to the related description in the foregoing embodiment shown in FIG. 3, and details are not described herein again.

Another embodiment of the present application also provides a computer readable storage medium including one or more program codes, the one or more programs including instructions, when a processor in an image processing apparatus is executing the At the time of the program code, the image processing apparatus executes the image processing method as shown in FIG.

In another embodiment of the present application, there is also provided a computer program product comprising computer executed instructions stored in a computer readable storage medium; at least one processor of the image processing apparatus is The computer readable storage medium reads the computer executable instructions, and the at least one processor executes the computer to execute the instructions such that the image processing apparatus implements the steps in performing the image processing method illustrated in FIG.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is illustrated. In practical applications, the above functions can be allocated according to needs. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the system, the device and the unit described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be used. Combinations can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application, in essence or the contribution to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. A number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to perform all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: flash memory, mobile A variety of media that can store program code, such as hard disks, read-only memories, random access memories, disks, or optical disks.

The foregoing is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. . Therefore, the scope of protection of the present application should be determined by the scope of the claims.

Claims

An image processing method, comprising:

Obtaining a first rendering command queue, where the first rendering command queue includes multiple rendering commands corresponding to the Nth image frame, N≥1;

Obtaining a first calculation command queue, where the first calculation command queue includes a plurality of calculation commands corresponding to the (N+1)th image frame;

And rendering the Nth image frame according to the first rendering command queue, and calculating the (N+1)th image according to the second calculation command queue in the process of rendering the Nth image frame All pixels of the frame.
The image processing method according to claim 1, wherein the image processing method further comprises:

Obtaining a parallel processing message, the parallel processing message is configured to notify, in the process of rendering the Nth image frame, all pixels of the (N+1)th image frame;

And the calculating, according to the first rendering command queue, the Nth image frame, and calculating, according to the second calculation command queue, all pixels of the (N+1)th image frame, specifically:

Performing, by using the first thread, a rendering command included in the first rendering command queue to render the Nth image frame;

Performing, by using a second thread, a calculation command response included in the first calculation command queue, and calculating the (N+1)th image frame;

The first thread and the second thread are run in parallel in response to the parallel processing message.
The image processing method according to claim 1 or 2, wherein the image processing method further comprises:

Calculating a feature value of the first rendering command queue according to a preset algorithm;

Storing a feature value of the first rendering command queue;

When N>1, the rendering the Nth image frame according to the first rendering command queue includes:

Determining that the feature value of the first rendering command queue is equal to the stored first value, the first value is a feature value of the target rendering command queue, and the target rendering command queue includes multiple corresponding to the Xth image frame Render command, N>X≥1;

Obtaining a stored rendering result of the Xth image frame;

And rendering the Nth image frame according to a rendering result of the Xth image frame.
The image processing method according to any one of claims 1 to 3, wherein after the rendering of the Nth image frame, the image processing method further comprises:

Stores the rendering result of the Nth image frame.
An image processing apparatus, comprising:

An acquiring unit, configured to acquire a first rendering command queue, where the first rendering command queue includes multiple rendering commands corresponding to the Nth image frame, N≥1, and is configured to acquire a first computing command queue, where a calculation command queue includes a plurality of calculation commands corresponding to the (N+1)th image frame;

a rendering unit, configured to render the Nth image frame according to the first rendering command queue acquired by the acquiring unit;

a calculating unit, configured to: according to the obtaining, in the process of rendering the Nth image frame by the rendering unit The second calculation command queue obtained by the unit calculates all pixels of the (N+1)th image frame.
The image processing device according to claim 5, wherein

The acquiring unit is further configured to acquire a parallel processing message, where the parallel processing message is used to notify, in the process of rendering the Nth image frame, all the pixels of the (N+1)th image frame;

The rendering unit is specifically configured to execute, by using a first thread, a rendering command included in the first rendering command queue to render the Nth image frame;

The calculating unit is configured to perform, by using a second thread, a calculation command response included in the first calculation command queue, calculate the (N+1)th image frame, and specifically, in response to the acquiring unit, Parallel processing of the message, in the process of rendering the Nth image frame by the rendering unit, calculating all pixels of the (N+1)th image frame.
The image processing device according to claim 5 or 6, wherein

The calculating unit is configured to calculate, according to a preset algorithm, a feature value of the first rendering command queue acquired by the acquiring unit;

The image processing device further includes a storage unit;

The storage unit is configured to store a feature value of the first rendering command queue calculated by the calculating unit;

When N>1, the image processing apparatus further includes a determining unit;

The determining unit is configured to determine that the feature value of the first rendering command queue calculated by the calculating unit is equal to the stored first value, where the first value is a feature value of the target rendering command queue, the target The rendering command queue includes a plurality of rendering commands corresponding to the Xth image frame, N>X≥1;

The obtaining unit is further configured to acquire a stored rendering result of the Xth image frame;

The rendering unit is specifically configured to render the Nth image frame according to a rendering result of the Xth image frame acquired by the acquiring unit.
The image processing device according to any one of claims 5-7, wherein the image processing device further comprises a storage unit;

The storage unit is configured to store a rendering result of the Nth image frame after the rendering unit renders the Nth image frame.
An image processing apparatus, comprising: one or more processors, a memory, and a communication interface;

The memory, the communication interface is coupled to the one or more processors; the image processing device is in communication with other devices via the communication interface, the memory is for storing computer program code, the computer program code comprising The image processing apparatus performs the image processing method according to any one of claims 1 to 4 when the one or more processors execute the instructions.
A computer storage medium having stored therein computer program code, wherein when the computer program code is run on an image processing device, the image processing device is caused to perform as in claims 1-4 The image processing method according to any one of the preceding claims.
A computer program product comprising instructions, wherein the image processing apparatus, when run on a communication device, causes the image processing apparatus to perform the image processing method of any one of claims 1-4.