CN116385611A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN116385611A CN116385611A CN202310145912.7A CN202310145912A CN116385611A CN 116385611 A CN116385611 A CN 116385611A CN 202310145912 A CN202310145912 A CN 202310145912A CN 116385611 A CN116385611 A CN 116385611A
- Authority
- CN
- China
- Prior art keywords
- node
- thread
- function
- frame
- frame buffer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 46
- 239000000872 buffer Substances 0.000 claims abstract description 331
- 238000000034 method Methods 0.000 claims abstract description 173
- 238000012545 processing Methods 0.000 claims abstract description 141
- 238000009877 rendering Methods 0.000 claims abstract description 136
- 238000003860 storage Methods 0.000 claims abstract description 43
- 238000004364 calculation method Methods 0.000 claims abstract description 35
- 230000006870 function Effects 0.000 claims description 191
- 230000008569 process Effects 0.000 claims description 104
- 238000013507 mapping Methods 0.000 claims description 22
- 230000014509 gene expression Effects 0.000 claims description 15
- 230000005540 biological transmission Effects 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 8
- 230000003139 buffering effect Effects 0.000 claims description 7
- 238000005070 sampling Methods 0.000 abstract description 27
- 238000010586 diagram Methods 0.000 description 27
- 230000000694 effects Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 13
- 239000002245 particle Substances 0.000 description 11
- 238000012805 post-processing Methods 0.000 description 10
- 238000004458 analytical method Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000012546 transfer Methods 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 238000010205 computational analysis Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 125000001475 halogen functional group Chemical group 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000006424 Flood reaction Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 125000002015 acyclic group Chemical group 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/60—Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4482—Procedural
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/50—Lighting effects
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/60—Methods for processing data by generating or executing the game program
- A63F2300/6009—Methods for processing data by generating or executing the game program for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A10/00—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE at coastal zones; at river basins
- Y02A10/40—Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
Abstract
The application discloses a data processing method, a data processing device and electronic equipment. The method is applied to a terminal running a target program, the target program comprises a rendering thread and a calculating thread, the rendering program loads a first link library file and a second link library file, and the first link library file comprises a first function and a second function; comprising the following steps: the rendering thread executes the second linked library file to: when each intra-frame monitors that the rendering thread calls the first function, executing Hook operation on the first function, and acquiring a plurality of frame buffers from a plurality of storage addresses; each frame monitors that the rendering thread calls a second function, then the Hook operation is executed on the second function, and a plurality of frame buffers are sent to the computing thread; the calculation thread determines the number of times of down-sampling processing performed to obtain the floodlight image according to the acquired plurality of frame buffers. The method is high in universality, does not need to be manually participated, and can determine the times of downsampling processing executed by obtaining the floodlight image in real time.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to a data processing method, a data processing device, and an electronic device.
Background
Flood is a common optical phenomenon, generally referred to as halation and overflow phenomenon that occurs when a physical camera photographs an object with high brightness. The contrast of the image can be visually improved by performing floodlight treatment on the image, the expressive force of the image is enhanced, and a better rendering effect is achieved. With the development of image processing technology, image floodlight processing has been widely applied to the fields of three-dimensional games, animation production, and the like. Currently, in a terminal game, a terminal performs floodlight processing on an image to be rendered to obtain a corresponding floodlight image, wherein the floodlight processing may include the following processes: downsampling, blurring, and upsampling. The floodlight effect exhibited by a floodlight image is mainly determined by the number of downsampling processes, so that the number of downsampling processes performed by the floodlight image, also called the floodlight layer number of the floodlight image, is obtained.
In the conventional art, there are the following problems in determining the number of times of down-sampling processing performed to obtain a floodlight image: the source codes and/or related configuration files of the application program need to be acquired, so that the universality is poor; the number of downsampling processes performed to obtain a floodlight image cannot be known in real time, requiring manual intervention statistics.
Therefore, there is a need for a data processing method that is highly versatile and that can determine in real time the number of downsampling processes performed to obtain a floodlight image without human intervention.
Disclosure of Invention
The method is high in universality, does not need to be participated manually, and can determine the times of downsampling processing executed by obtaining a floodlight image in real time.
The first aspect of the embodiment of the present application provides a data processing method, which is applied to a terminal running a target program, where the target program includes a rendering thread and a computing thread, the rendering program loads a first link library file and a second link library file, the first link library file includes a first function and a second function, the first function is used to draw any one of a plurality of frame buffers associated with each frame to obtain a floodlight image, the plurality of frame buffers are in one-to-one correspondence with a plurality of storage addresses of the terminal, and any one of the frame buffers is data to be rendered stored in the corresponding storage address; the second function indicates that the drawing of each frame is ended; the method comprises the following steps: the rendering thread executes the second linked library file to: executing Hook operation on the first function and executing data acquisition events for acquiring the plurality of frame buffers from the plurality of storage addresses under the condition that the rendering thread calls the first function in each frame; and executing the Hook operation on the second function and executing a data transmission event for transmitting the acquired frame buffers to the computing thread under the condition that the rendering thread calls the second function in each frame; in response to receiving the plurality of frame buffers sent by the rendering thread, the computing thread determines a number of downsampling processes performed to obtain the floodlight image in accordance with the plurality of frame buffers.
A second aspect of the present invention provides a data processing apparatus, applied to a terminal running a target program, where the target program includes a rendering thread and a computing thread, the rendering program loads a first link library file and a second link library file, the first link library file includes a first function and a second function, the first function is used to draw any one of a plurality of frame buffers associated with each frame to obtain a floodlight image, the plurality of frame buffers are in one-to-one correspondence with a plurality of storage addresses of the terminal, and any one of the frame buffers is data to be rendered stored in the corresponding storage address; the second function indicates that the drawing of each frame is ended; the device comprises: the first processing unit is configured to execute the second link library file to perform the following operations: executing Hook operation on the first function and executing data acquisition events for acquiring the plurality of frame buffers from the plurality of storage addresses under the condition that the rendering thread calls the first function in each frame; and executing the Hook operation on the second function and executing a data transmission event for transmitting the acquired frame buffers to the computing thread under the condition that the rendering thread calls the second function in each frame; the second processing unit is used for: in response to receiving the plurality of frame buffers sent by the rendering thread, determining a number of downsampling processes performed to obtain the floodlight image based on the plurality of frame buffers.
A third aspect of the embodiments of the present application further provides a computer readable storage medium having one or more computer instructions stored thereon, where the instructions are executed by a processor to implement a data processing method according to any one of the foregoing technical solutions.
The fourth aspect of the embodiments of the present application further provides an electronic device, including: a processor; and a memory for storing a data processing program, the server being powered on and executing the program by the processor, and executing the data processing method as described above.
It should be understood that the description in this section is not intended to identify key or critical features of the disclosed embodiments of the application, nor is it intended to be used to limit the scope of the disclosed embodiments of the application. Other features of the present disclosure will become apparent from the following specification.
In the technical scheme of the data processing method provided by the embodiment of the application, a rendering thread in a target program operated by a terminal loads a first dynamic link library file and a second dynamic link library file, and executes the second link library file in the rendering thread to execute the following operations: under the condition that the rendering thread calls the first function in each frame, hook operation is executed on the first function, and data acquisition events for acquiring a plurality of frame buffers from a plurality of storage addresses are executed; and under the condition that the rendering thread calls the second function in each frame, executing Hook operation on the second function, and executing a data transmission event for buffering and sending the acquired frames to the computing thread. The above-mentioned acquisition of multiple frame buffers associated with each frame is implemented by loading a custom second dynamic link library file by a rendering thread, and in the above-mentioned process, there is no need to modify the code of the target program, acquire the relevant configuration file of the target program or execute invasive analysis on the target program, i.e. the method has strong universality. In the implementation mode, the method is executed by taking each frame as a unit, so that the method is guaranteed to be strong in real-time performance. In the implementation manner of processing the acquired frame buffers by the computing thread to determine the number of times of downsampling processing performed to acquire the floodlight image, manual participation is not required. In conclusion, the data processing method provided by the application is high in universality, does not need to be manually participated, and can determine the times of downsampling processing executed by obtaining the floodlight image in real time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of an application scenario suitable for a data processing method provided in an embodiment of the present application.
Fig. 2 is a schematic diagram of a data processing method according to an embodiment of the present application.
Fig. 2A is a schematic diagram for performing S230 shown in fig. 2 described above.
Fig. 2B is a schematic diagram of any edge included in any path included in a directed acyclic graph according to an embodiment of the application.
Fig. 2C is a schematic diagram for performing S230-2 shown in fig. 2A described above.
Fig. 3 is a schematic diagram of another data processing method according to an embodiment of the present application.
Fig. 4A is a schematic diagram of a directed acyclic graph according to a method described in fig. 3, provided in an embodiment of the present application.
Fig. 4B is a schematic diagram of the types of each node included in the directed acyclic graph shown in fig. 4A, described above.
Fig. 5A is a schematic diagram of a directed acyclic graph according to a method described in fig. 3, provided in an embodiment of the present application.
Fig. 5B is a schematic diagram of the types of each node included in the directed acyclic graph shown in fig. 5A, described above.
Fig. 6 is a schematic diagram of a data processing apparatus according to an embodiment of the present application.
Fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.
Detailed Description
In order to enable those skilled in the art to better understand the technical solutions of the present application, the present application is clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. This application is intended to be limited to the details shown and described, and it is intended that the invention not be limited to the particular embodiment disclosed, but that the application will include all embodiments falling within the scope of the appended claims.
It should be noted that the terms "first," "second," "third," and the like in the claims, specification, and drawings herein are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. The data so used may be interchanged where appropriate to facilitate the embodiments of the present application described herein, and may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and their variants are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, technical terms that may be involved in embodiments of the present application are briefly described below.
1, image post-processing
The image post-processing refers to a process of optimizing an image, and is mainly used for enhancing characteristics of image antialiasing, high dynamic range image (high dynamic range, HDR), floodlight and the like. Image post-processing techniques include image processing techniques such as flood processing, antialiasing processing, motion blur processing, and depth of field processing. The object of the image post-processing may be an image rendered based on a three-dimensional scene.
2 floodlight (bloom)
Floodlight is a common optical phenomenon. Since a physical camera (i.e. a real video camera or a camera) cannot be perfectly focused when taking pictures, light rays can be diffracted at the edge of an object during imaging through a lens of the physical camera, and a halation overflow phenomenon occurs. Floodlight is not easily perceived in a scene with lower brightness (low light), but is more noticeable in a scene with higher brightness (high light). Thus, floodlight generally refers to a halo overflow phenomenon that occurs when a physical camera photographs an object with high brightness.
3 floodlight effect
In computer graphics, the floodlight effect, also known as highlights, is one type of computer graphics effect used in video games, presentation animations and high dynamic range imaging (High Dynamic Range Imaging, HDRI or HDR). The floodlight effect can generate stripe or feather-like light rays around the high-brightness object so as to blur image details, namely simulate floodlight phenomenon in the imaging process of the physical camera, and enable the image rendered by the electronic equipment to be more real.
4, open graphics library (open graphics library, openGL)
OpenGL, which is a specialized graphics program interface defining a cross-programming language and cross-platform programming interface specification, is applied to the industries including content creation, energy, entertainment, game development, manufacturing, pharmaceutical industry, virtual reality, and the like, and can help programmers to develop high-performance and high-visual-expressive graphics processing software on hardware devices such as personal computers (personal computer, PCs), workstations, supercomputers, and the like.
5, open graphic library of embedded system (openGL for embedded systems, openGL ES)
OpenGL ES is a subset of OpenGL three-dimensional graphics application program interfaces (application programming interface, APIs) designed for embedded devices such as cell phones and game hosts. Among them, openGL ES is based on OpenGL, and many non-absolutely necessary characteristics such as complex primitives including glBegin/glEnd, quadrangle (gl_quads), polygon (gl_polygens) and the like are removed. Over the years of development, there are now mainly 3 versions, openGL ES 1.X for fixed pipeline hardware, openGL ES 2.X for programmable pipeline hardware, and OpenGL ES 3.X. Wherein, openGL ES 1.0 is based on the OpenGL 1.3 specification, and OpenGL ES 1.1 is based on the OpenGL1.5 specification. OpenGL ES 2.0 is defined with reference to the OpenGL 2.0 specification. The OpenGL ES 3.X version adds more features (e.g., as a computer loader, etc.) to the OpenGL ES 2. X.
Enable scenarios for OpenGL ES include, but are not limited to: picture processing such as picture tone conversion, beauty, and the like; camera preview effect processing such as a face-beautifying camera or the like, video processing, 3D games, and the like.
6 directed acyclic graph (directed acyclic graph, DAG)
In mathematics, particularly graph theory and computer science, directed acyclic graphs refer to a loop-free directed graph. If there is a non-directed acyclic graph, and point A goes from point B back to point A via C, a ring is formed. Changing the side direction from C to A becomes a directed acyclic graph. The number of spanning trees of the directed acyclic graph is equal to the product of the degree of entry of the nodes with non-zero degrees of entry.
7 finite State automata (finite state machine, FSM)
Finite state automata is a computational model that is abstracted to study the computational processes of finite memory and certain language classes. Finite state automata possess a finite number of states, each of which can be migrated to zero or more states, and the input string determines which state to perform the migration. The finite state automaton may be represented as a directed graph. Finite state automata are the subject of investigation of automaton theory.
8, regular expression (regular expression)
A regular expression is a pattern that describes a set of string features that are used to match a particular string. The mode description is carried out through special characters and common characters, so that a tool for achieving the purpose of text matching is achieved. Regular expressions are currently integrated into a variety of text editors/text processing tools.
9, dynamic link library
A dynamically linked library is a library that contains code and data that can be used by multiple programs simultaneously.
The DLL file, the extension of the dynamic link library on the Windows platform, the extension of the dynamic link library on the Linux or Android platform and the like are SO files, and in the application, the DLL refers to the dynamic link library of a plurality of platforms instead of the dynamic link library of the Windows platform. That is, the dynamically linked library files described in the embodiments provided herein may be DLL files or SO files.
Floodlight (Bloom) is a common optical phenomenon, and generally refers to a halo overflow phenomenon occurring when a physical camera shoots an object with higher brightness. The contrast of the image can be visually improved by performing floodlight treatment on the image, the expressive force of the image is enhanced, and a better rendering effect is achieved. With the development of image processing technology, image floodlight processing has been widely applied to the fields of three-dimensional games, animation production, and the like. Currently, a terminal (e.g., a mobile phone end) performs a floodlight process on a floodlight image (e.g., a game floodlight image) associated with an application program displayed on the terminal, which mainly includes the following operations: downsampling processing, gaussian blur processing, and downsampling picture synthesizing processing. The number of times of performing the downsampling process on the floodlight image is also referred to as the floodlight layer number of the floodlight image, that is, the number of times of performing the downsampling process on the floodlight image is related to the implementation effect of floodlight corresponding to the floodlight image. More floodlight layers generally bring finer and more realistic floodlight effects. However, the graphics processor (graphics processing unit, GPU) of the terminal has problems of complex computation, high computation cost, and influence on the GPU performance of the terminal when performing the downsampling process a plurality of times. Therefore, in the process of testing the game performance, if the game has a bottleneck on the GPU side, the number of layers of floodlight is also a common consideration. If a tester can know the times of the downsampling processing corresponding to the floodlight image in real time, the bottleneck problem of reduced GPU performance caused by excessive times of the downsampling processing can be rapidly located.
In the conventional art, there is provided a method of determining the number of times of down-sampling processing performed to obtain a floodlight image associated with a game scene based on a game source code and an editor setting option. However, it is often difficult to obtain source code and editor settings for games due to rights, privacy, security, etc. In addition, problems such as coverage of setting priority, invalid information, useless information and the like may exist among the setting items in the codes and the configuration files, and analysis workload is large. In the conventional art, there is also provided an invasive analysis of a game scene to determine the number of downsampling processes performed to obtain a floodlight image associated with the game scene. The method comprises the following steps: cutting frames of the game scene by using a tool, obtaining a dump file of frame information and storing the dump file to a computer; processing the transfer storage file by using a tool to obtain a rendering instruction list; and manually checking a rendering instruction list, guessing the action of each instruction, positioning the instruction responsible for floodlight in the instruction, and acquiring the number of floodlight layers (namely, acquiring the number of times of downsampling processing executed by a floodlight image). In the above implementation, the project group is required to provide a bag for debugging, otherwise, it is difficult to perform invasive analysis. In addition, the data quantity corresponding to the data transferred and stored is large, the useful information is distributed in disorder and less, and automatic processing is difficult to realize. In summary, the conventional technique provides a determination of the number of times of downsampling processing performed to obtain a floodlight image, which has the following problems: (1) The source codes and/or related configuration files of the application program need to be acquired, so that the universality is poor; (2) The number of downsampling processes performed to obtain a floodlight image cannot be determined in real time, requiring manual intervention statistics.
In order to solve the problems, the application provides a data processing method, a data processing device and electronic equipment.
An application scenario and a data processing method applied to the data processing method in the embodiments of the present application are described in detail below with reference to the accompanying drawings. It is to be understood that the following embodiments and features thereof may be combined with each other without conflict between the embodiments provided in the present application. In addition, the sequence of steps in the method embodiments described below is only an example and is not strictly limited.
First, an application scenario applicable to the data processing method of the embodiment of the present application is described with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of an application scenario suitable for a data processing method provided in an embodiment of the present application. By way of example, the application scenario shown in fig. 1 includes a terminal 101 and a server 102. The terminal 101 communicates and data interacts with the server 102 via a network 103. The network 103 may be a wired network or a wireless network, which is not particularly limited in this application. The wireless or wired networks described above use standard communication techniques and/or protocols.
The terminal 101 is a client terminal with an application program installed, and the terminal 101 may render data to be rendered associated with the application program, and display a result after rendering to a user through a user interface of the terminal 101. The application may be an application that needs to be downloaded and installed, or may be a point-and-use application, which is not limited thereto. The application may be any application that is capable of providing a virtual environment in which virtual objects that are substituted and operated by a user are active. By way of example, the application may be a gaming application. Of course, other types of applications besides game applications may expose virtual objects to users and provide corresponding functionality to the virtual objects. That is, the application installed in the terminal 101 may also be another type of application other than the game application. For example, other types of applications may be, but are not limited to, any of the following: virtual Reality (VR) class applications, augmented reality (augmented reality, AR) class applications, three-dimensional map applications, military simulation applications, social class applications, or interactive entertainment class applications. The terminal 101 has at least the following processors installed therein: a central processor (central processing unit, CPU) for performing computational analysis operations, e.g., for performing computational analysis operations to obtain data to be rendered, and a graphics processor (graphics processing unit, GPU); the GPU is used for executing rendering operation on the data to be rendered obtained through calculation of the CPU so as to obtain a corresponding rendering image. By way of example, the terminal 101 may be, but is not limited to, any of the following: smart phones, tablet computers, palmtop computers, game consoles, electronic book readers, multimedia playing devices, vehicle-mounted devices, wearable devices (e.g., smart watches, smart bracelets, pedometers, etc.), smart home devices (e.g., refrigerators, televisions, air conditioners, electric meters, etc.), smart robots, personal computers (personal computer, PCs), and other electronic devices, as well as various forms of User Equipment (UE), mobile Stations (MS), terminal devices (terminal Equipment), and the like.
The server 102 is used to provide background services for applications in the terminal 101. The server 102 may be a server, a server cluster comprising a plurality of servers, or a cloud computing service center. Alternatively, the server 102 may provide background services for applications in a plurality of terminals 101 at the same time.
It should be understood that the application scenario shown in fig. 1 is merely schematic, and does not constitute any limitation on the application scenario applicable to the data processing method provided in the embodiment of the present application. Optionally, the application scenario may further include a greater number of terminals 101 or servers 102.
Next, a data processing method provided in an embodiment of the present application is described with reference to the accompanying drawings.
Fig. 2 is a schematic diagram of a data processing method according to an embodiment of the present application. By way of example, the execution subject of the data processing method shown in fig. 2 may be the terminal 101 shown in fig. 1 described above. As shown in fig. 2, the data processing method includes S210 and S220. Next, S210 and S220 are described in detail.
Before describing the data processing method provided by the embodiment of the present application, an execution body for executing the data processing method provided by the embodiment of the present application and related concepts are described. The data processing method provided by the embodiment of the application is applied to a terminal running a target program, the target program comprises a rendering thread and a computing thread, the rendering thread loads a first link library file and a second link library file, the first link library file comprises a first function and a second function, the first function is used for drawing any one of a plurality of frame buffers associated with each frame to obtain a floodlight image, the plurality of frame buffers are in one-to-one correspondence with a plurality of storage addresses of the terminal, and any one of the frame buffers is data to be rendered stored in the corresponding storage address; the second function indicates that the drawing for each frame is ended. In the above manner, the rendering thread loads the first link library file and the second link library file, so that the target program has the functions of the first link library file and the second link library file by setting the manner in which the rendering thread loads the first link library file and the second link library file.
The frame buffers associated with each frame are drawn according to the drawing instructions corresponding to each frame buffer, so that the corresponding floodlight image can be obtained in real time. The number of the plurality of frame buffers associated with each frame is not particularly limited and may be set according to actual demands. For example, the number of frame buffers associated with each frame may be 2, 5, 7, etc. Any one of the frame buffers has a unique, nonrepeatable, unalterable identification number (identity document, ID). That is, when the first function draws a different frame buffer associated with each frame, the frame buffer ID associated with the first function is different.
The plurality of storage addresses are in one-to-one correspondence with the plurality of frame buffers, and any one of the frame buffers is data to be rendered stored in the corresponding storage address. And the frame buffer attachment is used for buffering the data to be rendered stored in part of the memory addresses in the corresponding storage addresses of any frame buffer. Any one of the frame buffer attachments included may be a texture attachment or a render buffer object attachment. Specifically, when one frame buffer attachment is a texture attachment, in the case where a frame buffer corresponding to the one frame buffer attachment is bound by a frame buffer other than the frame buffer (bind), the rendering thread may draw the one frame buffer attachment to at least one frame buffer attachment included in the frame buffer other than the frame buffer through a drawing instruction. When one frame buffer attachment is a rendering buffer object attachment, in the case that the frame buffer corresponding to the one frame buffer attachment is bound by a frame buffer other than the one frame buffer (bind), the rendering thread can take the one frame buffer attachment as the output of the current drawing instruction through the drawing instruction. Any one of the frame buffer attachments also has a unique, non-repeatable, non-modifiable ID. The type of the memory address is not particularly limited, and for example, the memory address may be a memory address of the terminal. Any frame buffer may include one or more frame buffer appendages, and any frame buffer appendages include the following information: an ID of the frame buffer attachment, and an attribute of the frame buffer attachment. In some implementations, the frame buffer attachment is a texture attachment, the frame buffer attachment including the following information: an ID of the texture accessory, an attribute of the texture accessory (e.g., the texture accessory is one-dimensional, two-dimensional, or three-dimensional data, etc.). Optionally, in other implementations, the frame buffer attachment is a render buffer object attachment, and the frame buffer attachment includes the following information: an ID of the render buffer object attachment, an attribute of the render buffer object attachment (e.g., the texture attachment is one-dimensional, two-dimensional, or three-dimensional data, etc.). Optionally, any of the frame buffer attachments described above may also include other information, for example, other information may be, but is not limited to, the size of the frame buffer attachment.
Examples of the first function and the second function included in the first link library file provided in the embodiment of the present application are described below.
In some implementations, where the first linked library file is OpenGL ES, the first function may be a glBindFrameBuffer function, or a glBindTexture function. Specifically, in the case where the first function is a glbindframe buffer function, the first function is used to draw the first function, specifically, for any one of a plurality of frame buffers, and at least one frame buffer attachment included in the any one frame buffer is a rendering buffer object attachment. In the case where the first function is a glBindTexture function, the first function is used to render the first function specifically for any one of a plurality of frame buffers, and at least one frame buffer attachment included in the any one frame buffer is a texture attachment. Illustratively, the role of the first function is described below taking the first link library file as OpenGL ES and the first function as a glbindframe buffer function as an example. Under the condition that 2 frame buffers are associated with a floodlight image, when the glBindFrameBuffer function is associated with a frame buffer ID1, the rendering thread calls the glBindFrameBuffer function to be used for drawing data to be rendered which are associated with the frame buffer ID 1; when the glBindFrameBuffer function is associated with the frame buffer ID2, the rendering thread calls the glBindFrameBuffer function to be used for drawing the data to be rendered which is associated with the frame buffer ID 2; wherein, frame buffer ID1 is different from frame buffer ID2, and frame buffer corresponding to frame buffer ID1 is different from frame buffer corresponding to frame buffer ID 2.
In some implementations, where the first linked library file is OpenGL ES, the second function may be an EGLSwapBuffer function defined in OpenGL ES.
In the embodiment of the present application, the number of threads included in the target program is not specifically limited. Optionally, in other implementations, the target program may further include a logic thread, where the target program includes: rendering threads, the computing threads, and the logical threads. Among other things, logical threads are typically used to process logically related content, such as processing logic for communication with a game server, or calculation of enemy AI in a game, etc. The logic thread computationally determines an object to be rendered (e.g., a model position and a model structure of the object to be rendered, etc.) based on the logic described above for use by the rendering thread. Wherein, the rendering thread is composed of at least a CPU and a GPU; the above-mentioned calculation thread is formed by CPU at least; the logical thread is composed of at least a CPU. The relationship between the rendering thread and the computing thread is not particularly limited. In some implementations, the rendering thread and the computing thread are two different threads included by the target program. Alternatively, in other implementations, the computing thread is a child of the rendering thread. The above-described object program may provide a graphic user interface through which a floodlight image is displayed, and the type of the object program is not particularly limited. For example, the target program may be a game application, a Virtual Reality (VR) class application, or an augmented reality (augmented reality, AR) class application, or the like. The program form of the above terminal is not particularly limited. For example, the terminal may be a smart phone, a smart wearable device, a vehicle-mounted device or server, etc.
The rendering engines used for performing the rendering operation on the rendering thread running on the terminal are not particularly limited, and may use OpenGL or OpenES to draw images. For example, when the target program is a game application program, the rendering engine may be a UE game engine or a Unity game engine. Optionally, in some implementations, the object program is a game application program, and the first link library file is an open graphics library file OpenGL or an embedded system open graphics library OpenGL ES. Optionally, in other implementations, the target program is a program other than the game application program, and the first link library file is an open graphics library file OpenGL or an embedded system open graphics library OpenGL ES. For example, the other program may be, but is not limited to being, a virtual reality program.
In the embodiment of the present application, the types of the first link library file and the second link library file are not specifically limited. In practical applications, the type of the link library file is related to the platform on which the link library file is run. For example, when the platform is a Windows platform, the first and second link library files may be DLL type link library files. For another example, when the platform is a Linux or Android platform, the first link library file and the second link library file may be a link library file of a type of SO.
Next, S210 and S220 are described in detail.
S210, the rendering thread executes the second link library file to execute the following operations: under the condition that the rendering thread calls the first function in each frame, hook operation is executed on the first function, and data acquisition events for acquiring a plurality of frame buffers from a plurality of storage addresses are executed; and under the condition that the rendering thread calls the second function in each frame, executing Hook operation on the second function, and executing a data transmission event for buffering and sending the acquired frames to the computing thread.
Executing the step S210, namely, through the second link library file loaded by the rendering thread, to implement that, when the rendering thread calls the first function in each frame, the first function is not executed first, but a data acquisition event of a plurality of frame buffers is executed and acquired from a plurality of storage addresses, where the first function is used to draw any frame buffer of the plurality of frame buffers associated with each frame to obtain a floodlight image; and under the condition that the rendering thread calls the second function in each frame, executing the data transmission event for buffering and sending the acquired frames to the computing thread without executing the second function, wherein the second function indicates that drawing of each frame is finished. That is, the above S210 is performed, so that the rendering thread may send the plurality of frame buffers corresponding to each frame acquired in real time to the computing thread.
The second link library file comprises: a function for listening for events calling the first function and the third function within each frame, a function for performing data acquisition events, and a function for performing data transfer events. The number of functions included in the second link library file is not particularly limited. For example, the second link library file may include one function having the following functions: monitoring for events calling the first function and the third function within each frame, performing a data acquisition event, and performing a data transfer event. As another example, the second link library file may include a function 1 and a function 2, where the function 1 is used to snoop for events calling the first function and the third function within each frame, and the function 2 is used to: a data acquisition event is performed, and a data transmission event is performed. It can be appreciated that the second link library file is a custom link library file provided herein. Optionally, before performing S210 described above, the following operations may also be performed: a second link library file is generated.
The implementation of "in the case where the rendering thread is monitored to call the first function in each frame, the Hook operation is performed on the first function, and the data acquisition event of acquiring the plurality of frame buffers from the plurality of memory addresses" described in S210 above, and "in the case where the rendering thread is monitored to call the second function in each frame, the Hook operation is performed on the second function, and the data transmission event of transmitting the acquired plurality of frame buffers to the calculation thread" is performed is not particularly limited. In some implementations, the method described in S210 above may be implemented by setting the address of the function for performing the data acquisition event to be the same as the address of the function of the first function, and setting the address of the function for performing the data transfer event to be the same as the address of the second function.
Optionally, in some implementations, the computing thread is a child thread of the rendering thread, and before the rendering thread sends the plurality of frame buffer information associated with the acquired floodlight image to the computing thread, the method further includes: in response to listening to call the third function within each frame, the rendering thread creates a computing thread. That is, in implementations in which the computing thread is a child of the rendering thread, the rendering thread is to perform the following operations: under the condition that a third function calling the first link library file is monitored, a calculation thread is firstly created; the acquired plurality of frame buffers are then sent to the computing thread.
And executing the step S210, namely when the rendering thread monitors a third function for finishing drawing the floodlight image, the rendering thread sends a plurality of frame buffers associated with the obtained floodlight image to the computing thread so that the computing thread analyzes and processes the plurality of frame buffers associated with each obtained frame. In the embodiment of the present application, in the case where the rendering thread sends the obtained plurality of frame buffers to the computing thread, the rendering thread may also draw the plurality of frame buffers to obtain the floodlight image. That is, in some implementations, the method further includes: the rendering thread executes the second link library file to perform the following operations: and under the condition that the rendering thread is monitored to send the acquired frame buffers to the computing thread, finishing Hook operation on the first function and the second function, and calling the first function and the second function, so that any frame buffer in the frame buffers is drawn at each frame to obtain a floodlight image.
In this embodiment of the present application, after executing S210 above, the rendering thread sends the obtained multiple frame buffers associated with each frame to the computing thread, and correspondingly, the computing thread receives the multiple frame buffers associated with each frame sent by the rendering thread. In the process of acquiring the plurality of frame buffers associated with each frame by the computing thread, the rendering thread loads the customized second dynamic link library file, so that the code of the target program is not required to be modified or the related configuration file of the target program is not required to be acquired. In the implementation manner, the method is executed by taking each frame as a unit, so that the instantaneity of the acquired frame buffers is ensured, and the instantaneity of the method is high.
S220, in response to receiving a plurality of frame buffers sent by the rendering thread, the computing thread determines the number of times of downsampling processing executed by obtaining the floodlight image according to the plurality of frame buffers.
The method described in S220 above will be described below in connection with S220-1 and S220-2 shown in fig. 2A. That is, performing the above S220 includes at least S220-1 and S220-2.
S220-1, a computing thread generates a directed acyclic graph comprising a plurality of paths according to a plurality of frame buffers, wherein any one of the plurality of paths comprises at least one directed edge; at least one directed edge is an edge directed by the first node to the second node; binding at least one frame buffer attachment included in a first frame buffer corresponding to a first node to a second frame buffer corresponding to a second node as a texture attachment, so that the at least one frame buffer attachment included in the first frame buffer is drawn to the second frame buffer when the second frame buffer is drawn; the second frame buffer is obtained by executing preset processing on the first frame buffer, and the type of the second node is associated with the preset processing; the plurality of frame buffers includes a first frame buffer and a second frame buffer.
In the above S220-1, the directed acyclic graph includes a plurality of paths, wherein any one of the plurality of paths includes at least one directed edge; one directed edge is made up of a first node, also referred to as the parent node of the second node, and a second node. Illustratively, fig. 2B shows a schematic diagram of any edge included in any path included in the directed acyclic graph provided by an embodiment of the present application. The at least one frame buffer attachment included in the first frame buffer is drawn to the second frame buffer when the second frame buffer is drawn, that is, the at least one frame buffer attachment included in the first frame buffer is of a texture attachment type.
In the above step S220-1, the type of the second node is associated with the preset process. The preset processing is associated with rendering effects presented by the floodlight image. In some implementations, the floodlight image is a floodlight image, and in practical applications, obtaining the floodlight image at least needs to perform the following processing: an initialization stage process, a downsampling process, a blurring process (e.g., a gaussian blurring process), an upsampling process, and a post-process, wherein the number of times the downsampling process and the upsampling process are performed to obtain a floodlight image are the same. Based on this, the above-mentioned preset process may include at least: initialization processing, downsampling processing, blurring processing (e.g., gaussian blurring processing), upsampling processing, and post-processing. Wherein the initialization process may be, but is not limited to, adjusting the size of the frame buffer; the post-processing is processing for performing special effect synthesis on other processing included in the preset processing.
Alternatively, in other implementations, if other processing than the above processing may be performed to obtain the floodlight image, the above-described preset processing may further include the other processing. For example, other processes may be, but are not limited to: GPU particle computing processing, or shadow computing processing. Next, the type of the second node and the preset process are described by way of example. For example, if the preset process is an upsampling process, the second node is of the upsampling process type, in which way the second node is also referred to as an upsampling process node. For another example, if the preset process is a GPU particle computing process, the second node is of a GPU particle computing type, and in this manner, the second node is also referred to as a GPU particle computing node.
Optionally, in some implementations, the above object program further includes a logic thread, where the logic thread is configured to determine an object to be drawn corresponding to the floodlight image, the plurality of frame buffers is configured to indicate a manner of performing drawing on the object to be drawn, an ingress degree of a start node of any path is zero, an egress degree of a termination node of any path is zero, and the frame buffer corresponding to the start node of any path is determined according to the object to be drawn corresponding to the floodlight image. The type of the initial node of any path is associated with a frame buffer mode corresponding to the initial node determined according to an object to be drawn corresponding to the floodlight image. For example, if the frame buffer corresponding to the start node is obtained by performing a shadow map calculation on the object to be drawn, the type of the start node is the shadow map calculation type. For another example, if the frame buffer corresponding to the start node is obtained by performing GPU particle calculation on the object to be drawn, the type of the start node is the GPU particle calculation type. It should be noted that, in the case that any one of the paths includes only two nodes, the first node included in the any one of the paths is a start node of the any one of the paths, and the second node included in the any one of the paths is a stop node of the any one of the paths.
The directed acyclic graph DAG obtained after performing the above-described S220-1 is described below by way of example with reference to the accompanying drawings. Fig. 4A is a schematic diagram of a directed acyclic graph DAG corresponding to a plurality of frame buffers associated with a floodlight image according to an embodiment of the disclosure. Referring to fig. 4a, the dag graph includes 12 nodes (i.e., nodes 401 to 412), and the 12 nodes are in one-to-one correspondence with 12 different frame buffers, i.e., the floodlight image is obtained by rendering and drawing the 12 frame buffers. The DAG graph shown in fig. 4A includes multiple paths, e.g., the DAG shown in fig. 4A includes one path: node 401- > node 402- > node 404- > node 412, wherein the degree of entry of node 401 is zero, i.e. node 401 is the starting node of the one path; the degree of egress of node 412 is zero, i.e., node 412 is the termination node of the one path; node 402 and node 404 are intermediate nodes of the one path. It should be understood that the above-described directed acyclic graph corresponding to the floodlight image illustrated in fig. 4A is merely schematic and is not intended to be limiting in any way. The types of directed edges and nodes associated with the directed edges shown in fig. 4A are described below. For example, taking the edges 401-402 associated with node 401 and node 402 as examples, the degree of invasiveness of node 401 is zero, i.e., node 401 is the starting node. Wherein edges 401-402 are a directed edge directed by node 401 to node 402; at least one frame buffer attachment included in the frame buffer corresponding to node 401 is bound to the frame buffer corresponding to node 402 as a texture attachment; the frame buffer corresponding to the node 402 is obtained by performing a preset process on the frame buffer corresponding to the node 401, the node 402 is associated with the preset process, and the node 401 is also referred to as a parent node of the node 402. For example, referring to fig. 4B, in a case where the frame buffer corresponding to the node 402 is obtained by performing the GPU particle calculation process on the frame buffer corresponding to the node 401, the type of the node 402 is the GPU particle calculation type. The frame buffer corresponding to the node 401 is determined according to the object to be drawn, and the type of the node 402 is associated with the manner of determining the frame buffer corresponding to the starting node according to the object to be drawn corresponding to the floodlight image. For example, referring to fig. 4B, in a case where the frame buffer corresponding to the node 401 is rendering data of an object to be drawn obtained by performing GPU particle calculation, the type of the node 401 is a GPU particle calculation type. For another example, taking the edge 405-406 associated with the node 405 and the node 406 as an example, the edge 405-406 is a directed edge pointing from the node 405 to the node 406, at least one frame buffer attachment included in the frame buffer corresponding to the node 405 is bound to the frame buffer corresponding to the node 406 as a texture attachment, and the frame buffer corresponding to the node 406 is obtained by performing a preset process on the frame buffer corresponding to the node 405. That is, when the frame buffer corresponding to the node 406 is bound, at least one frame buffer attachment included in the frame buffer corresponding to the node 405 is drawn as a texture attachment to the frame buffer corresponding to the node 406. For example, referring to fig. 4B, in the case where the frame buffer corresponding to the node 406 is obtained by performing the down-sampling process and the blurring process on the frame buffer corresponding to the node 405, the type of the node 406 is a floodlight down-sampling blurring type. Illustratively, as shown in fig. 5A, the DAG graph includes 8 nodes (i.e., node 500 to node 507), and the 8 nodes are in one-to-one correspondence with 8 different frame buffers, i.e., the floodlight image is rendered and drawn from the 8 frame buffers. For an understanding of the directed acyclic graph shown in fig. 5A, reference may be made to the above understanding of the directed acyclic graph shown in fig. 4A, and detailed description thereof is omitted herein.
S220-2, the calculation thread determines the times of downsampling processing executed by obtaining the floodlight image according to the directed acyclic graph.
The method described in the above-described S220-2 will be described below with reference to S220-21 to S220-23 shown in fig. 2C. That is, performing the above-described S220-2 includes at least S220-21 to S220-23.
S220-21, determining a target area of any path by a calculation thread, wherein the type of any node in the target area of any path is associated with the first processing; the first treatment is at least any one of the following: downsampling, blurring, or upsampling, the preset process including a first process; the number of nodes of the first type in the target area of either path is equal to the number of nodes of the second type; the first type is associated with a downsampling process and the second type is associated with an upsampling process.
The calculation thread determines a target area of any path, and the method comprises the following steps: the computing thread determines the type of any node according to a preset mapping relation and attribute information of any node included in any path, wherein the preset mapping relation is the mapping relation between the attribute information of the node and the type of the node; the attribute information of the nodes comprises the output degree of the nodes, the input degree of the nodes and the types of the nodes except any node in the directed edges associated with any node; the method comprises the steps that a calculation thread determines a target area of any path according to sequence information of any path and a preset expression, wherein the sequence information of any path comprises a plurality of bits, an ith bit in the plurality of bits is a type of an ith node pointing to a termination node from a start node in any path, and i is a positive integer; the preset expression is used to identify a node of a type associated with the first process. An example of a computing thread determining a target region for any one path is described below by way of example. Illustratively, the directed acyclic graph shown in FIG. 4B includes one path: node 403- > node 404- > node 405- > node 406- > node 407- > node 408- > node 409- > node 410- > node 412, and the target area of the one path comprises the following nodes: node 406, node 408, node 409, and node 410. Illustratively, the directed acyclic graph shown in FIG. 5B includes one path: node 500- > node 501- > node 502- > node 506- > node 507, and the target area of the one path includes the following nodes: node 502 and node 506. Note that if one path included in the directed acyclic graph does not include a node of a type associated with the first process, the one path does not include the target area. Illustratively, the directed acyclic graph shown with reference to FIG. 4B includes one path: node 411- > node 412, neither node 411 nor node 412 are nodes associated with the first process, so it can be known that the one path does not include the target area according to the above determination rule.
Alternatively, in other implementations, the first process may be an initialization process, where the initialization process may, but is not limited to, adjusting the size of the frame buffer.
In the implementation manner, the preset mapping relationship is a mapping relationship between attribute information of the node and the type of the node; the attribute information of the node includes the degree of egress of the node, the degree of ingress of the node, and the type of the node other than any node in the directed edge associated with any node.
TABLE 1
For example, table 1 above shows a preset mapping relationship, and it should be understood that the preset mapping relationship shown in table 1 is only schematic and not limiting. It will be appreciated that in the preset mapping relationship shown in table 1, the down-sampling process and the blurring process are performed by two different nodes, respectively.
TABLE 2
By way of example, table 2 above shows another preset mapping relationship, and it should be understood that the preset mapping relationship shown in table 2 is only illustrative and not limiting. It will be appreciated that in the preset mapping relationship shown in table 2, the downsampling process and the blurring process are performed by the same node, and the same node is another node in the directional edges associated with the downsampling blurring process node, and the directional edges are one edge pointed to the downsampling blurring process node by the same node. In some implementations, the frame buffer corresponding to the intermediate node shown in table 2 above is the same as the frame buffer corresponding to the downsampled fuzzy node. In other implementations, the frame buffer corresponding to the intermediate node shown in the above table 2 is obtained by performing a downsampling blurring process on the frame buffer corresponding to the downsampling blurring node. Illustratively, based on the preset mapping relationship shown in the above table 2, according to the DAG shown in fig. 4A, a type of each node included in the DAG shown in fig. 4A may be obtained, and the type of each node included in the DAG shown in fig. 4A is specifically referred to the DAG shown in fig. 4B. It will be appreciated that in the DAG shown in fig. 4B, node 401, node 402, node 403, node 404, node 411, and node 412 are general nodes.
The presentation form of the preset mapping manner is not particularly limited. In some implementations, the preset mapping relationship may be implemented using a finite state automaton FSM. In this way, the state machine corresponding to the FSM is configured to implement the preset mapping relationship, where the input of the FSM is attribute information of one node, and the output of the FSM is the type of the one node. In the implementation mode, the function of determining the target area of any path is realized by utilizing the design of the finite state automaton, and the manual check of the stored data is avoided. Illustratively, one specific example of a computing thread utilizing the FSM to determine the node type in the path included in the directed acyclic graph is described with reference to S306-1, infra, and details not described in detail herein can be found in connection with S306-1, infra. Alternatively, in other implementations, the preset mapping relationship may also be presented in a data table stored in a database, that is, the preset mapping relationship is stored in a data table (for example, a two-dimensional data table). In this way, the computing thread can determine the type of any node corresponding to the attribute information of any node included in any path by searching the corresponding data table.
The preset expression is used to identify a node of a type associated with the first process. In some implementations, the preset expression may be a regular matching expression. The sequence information of any one path includes a plurality of bits, and an i-th bit in the plurality of bits is a type of an i-th node pointing from the start node to the end node in any one path, and i is a positive integer. For example, taking the DAG graph shown in fig. 5B as an example, one path shown in fig. 5B is: node 500- > node 507, the sequence information of the one path is: a basic rendering node and a post-processing node. Wherein the base rendering node performs rendering of the 3D scene, e.g. rendering opaque objects, semi-transparent objects, etc. onto the frame buffer. The post-processing node is a node that mixes and synthesizes the results of a plurality of post-processing processes.
Optionally, in some implementations, before the computing thread determines the target region for any path, the following steps may also be performed: and the computing thread performs path searching processing on the directed acyclic graph to obtain a plurality of paths. In the embodiment of the present application, a method for obtaining multiple paths included in the directed acyclic graph by a computing thread according to the generated directed acyclic graph is not specifically limited. For example, a computing thread may traverse a directed acyclic graph, finding all zero-degree-of-entries nodes that the directed acyclic graph includes, and all zero-degree-of-outputs nodes; and the computing thread finds all paths included in the directed acyclic graph in a parallelization depth-first traversal mode according to the nodes with zero degree of ingress and the nodes with zero degree of egress included in the directed acyclic graph.
S220-22, determining a target area according to a first path in the paths by a computing thread according to a plurality of target areas of the paths, and determining the times of downsampling processing executed by obtaining the floodlight image, wherein the number of nodes of a first type in the target area of the first path is larger than or equal to the number of nodes of the first type in the target area of a second path; the second path is any one of the paths other than the first path. S220-23, the computing thread determines the number of nodes of the first type in the target area of the first path as the number of downsampling processes performed to obtain the floodlight image.
Illustratively, taking the directed acyclic graph shown in fig. 4B as an example of a directed acyclic graph corresponding to a plurality of frame buffers associated with a floodlight image, and the intermediate node shown in fig. 4B is a downsampled fuzzy node, the first path of the plurality of paths included in the directed acyclic graph shown in fig. 4B may be the following paths: node 403- > node 404- > node 405- > node 406- > node 407- > node 408- > node 409- > node 410- > node 412, the target area of the one path comprising the following nodes: node 406, node 407, node 408, node 409, and node 410, and the target area of the one path includes 3 nodes associated with the downsampling process (i.e., node 406, node 407, and node 408). That is, the number of downsampling processes performed to obtain a floodlight image corresponding to the finite loop-free diagram shown in fig. 4B is 3.
The above-mentioned S220-21 to S220-23, namely, the method in which the computing thread determines the number of times of downsampling processing performed to obtain the floodlight image according to the directed acyclic graph corresponding to the plurality of frame buffers associated with the floodlight image, is as follows: that is, the computing thread first determines a target region for each of a plurality of paths included in the directed acyclic graph, and the target region is a region associated with the first process. Thereafter, the calculation thread determines the number of times of down-sampling processing performed to obtain the floodlight image corresponding to the directed acyclic graph, based on the number of nodes of the first type in the target area of each of the plurality of paths included in the directed acyclic graph. The implementation process is realized through a calculation thread, and manual participation is not needed. It should be understood that fig. 3 is merely illustrative and is not intended to limit the data processing method provided in the present application. For example, the above description is given by taking as an example that the rendering thread performs listening and acquiring data, and the computing thread performs analysis on the acquired data. Alternatively, in other implementations, the computing thread and the rendering thread may be the same thread, i.e., the same thread may perform the steps performed by the rendering thread and the steps performed by the computing thread.
In the embodiment of the application, a rendering thread in a target program operated by a terminal loads a first dynamic link library file and a second dynamic link library file, and executes the second link library file in the rendering thread to execute the following operations: under the condition that the rendering thread calls the first function in each frame, hook operation is executed on the first function, and data acquisition events for acquiring a plurality of frame buffers from a plurality of storage addresses are executed; and under the condition that the rendering thread calls the second function in each frame, executing Hook operation on the second function, and executing a data transmission event for buffering and sending the acquired frames to the computing thread. The above-mentioned acquisition of multiple frame buffers associated with each frame is implemented by loading a custom second dynamic link library file by a rendering thread, and in the above-mentioned process, there is no need to modify the code of the target program, acquire the relevant configuration file of the target program or execute invasive analysis on the target program, i.e. the method has strong universality. In the implementation mode, the method is executed by taking each frame as a unit, so that the method is guaranteed to be strong in real-time performance. Further, the computing thread processes the acquired frame buffers to generate a corresponding directed acyclic graph; and the directed acyclic graph is processed to determine the times of downsampling executed by obtaining the floodlight image, and in the implementation process, the calculation thread is used for realizing the purpose of not needing manual participation, thereby being beneficial to reducing the workload of manual analysis. The rendering thread and the calculating thread can be two different threads, and the logic calculation and the picture update of the game can be effectively prevented from being blocked by the calculation of the follow-up steps such as the reconstruction of the directed acyclic graph. In summary, the data processing method provided by the embodiment of the application has strong universality, does not need to be manually participated, and can determine the times of downsampling processing executed by obtaining the floodlight image in real time.
Next, another data processing method provided in the embodiment of the present application is described with reference to fig. 3. It will be appreciated that the data processing method described in fig. 3 is a specific example of the data processing method described in fig. 2 above. Specifically, the floodlight image described in fig. 3 below is one example of the floodlight image described in fig. 2 described above, and the floodlight image is drawn with the game engine UE. The floodlight region described below in fig. 3 is one example of the target region described in fig. 2 described above, and the floodlight region described in fig. 3 includes an initialization node, a downsampling blur node, an intermediate node, and an upsampling node.
Fig. 3 is a schematic diagram of another data processing method according to an embodiment of the present application. It should be understood that the example of fig. 3 is merely to aid one skilled in the art in understanding the present embodiments, and is not intended to limit the present embodiments to the specific values or particular scenarios illustrated. Various equivalent modifications and variations will be apparent to those skilled in the art from the example of fig. 3 given below, and such modifications and variations are intended to be within the scope of the embodiments of the present application. It will be appreciated that the execution subject of the data processing method shown in fig. 3 may be the terminal 101 shown in fig. 1 described above. As shown in fig. 3, the data processing method includes S301 to S306. Next, S301 to S306 will be described in detail.
In the embodiment of the present application, the method described in fig. 3 is applicable to a terminal running a game application. The game application may include, among other things, logical threads, rendering threads, and computing threads. The rendering thread in the game application loads the OpenGL ES dynamic link library file and the Hook dynamic link library file. The method depicted in fig. 3 is described with the game engine UE as rendering engine. Wherein data required for the game engine UE to implement the downsampling process and blurring process involved in the floodlight process is stored in one frame buffer. Alternatively, when the floodlight processing is implemented using Unity, the data required for the downsampling processing and blurring processing involved are stored in two different frame buffers in succession, respectively.
S301, determining an object to be rendered in the floodlight image by a logic thread in a game application program running in the terminal, and sending the determined object to be rendered in the floodlight image to a rendering thread.
The logical thread and rendering thread in S301 described above are two different threads in the game application. The implementation of determining the object to be rendered in the floodlight image by the logic thread is not particularly limited.
S302, a rendering thread in a game application program running in the terminal determines a plurality of frame buffer information to be rendered corresponding to the floodlight image, and stores the plurality of frame buffer information to be rendered corresponding to the floodlight image into a memory of the terminal.
The definition of the plurality of frame buffers described in S302 is the same as that of the plurality of frame buffers involved in the data processing method provided in fig. 2, and details not described herein are referred to in the related description. In the embodiment of the present application, the method principle of acquiring any one frame buffer information of a plurality of frame buffer information is the same. Next, description will be given taking as an example the acquisition of the target frame buffer information in connection with S303, wherein the target frame buffer information described below is any one of the above-mentioned plurality of frame buffer information.
S303, responding to a rendering thread in a game application program running in the terminal to call an OpenGL ES function #1 in an OpenGL ES dynamic link library file, executing a Hook function in the Hook dynamic link library file, so that after target frame buffer information associated with the OpenGL ES function #1 is acquired from a memory, executing the OpenGL ES function #1 to draw the target frame buffer information.
Wherein OpenGL ES function #1 is one example of the first function in the method described above in fig. 2; the Hook function is one example of a second function in the method described above in fig. 2.
In the embodiment of the application, when the game application runs, the rendering thread included in the game application already loads the Hook dynamic link library file, namely, the Hook function in the Hook dynamic link library file is always in a monitoring state in the running process of the game application. And executing the step S303, namely executing the event 1 corresponding to the Hook function by the rendering thread under the condition that the Hook function monitors that the rendering thread in the game application program calls the OpenGL ES function # 1. Wherein, event 1 corresponding to the Hook function sequentially comprises: acquiring target frame buffer attachment information included in a target frame buffer associated with the OpenGL ES function #1 from a memory; and loading an OpenGL ES function #1 in the OpenGL ES dynamic link library file so that the rendering thread executes the OpenGL ES function #1 to draw the target frame buffer attachment information after acquiring the target frame buffer attachment information.
The rendering thread and the logical thread may be two different threads comprised by the gaming application. The functions in OpenGL ES that render thread calls (e.g., openGL ES function # 1) are not implemented in game code associated with the game application, but rather in the system driver of the terminal.
The OpenGL ES function #1 in the OpenGL ES dynamic link library file is used for drawing the target frame buffer information, that is, the OpenGL ES function #1 is used for drawing the data stored in the storage address corresponding to the target frame buffer. In some implementations, the OpenGL ES function #1 is a glBindFrameBuffer function, or a glBindTexture function. When the OpenGL ES function #1 is a glbindframe buffer function, the OpenGL ES function #1 is specifically configured to use a frame buffer attachment included in the target frame buffer as an output of a current drawing instruction when the target frame buffer associated with the glbindframe buffer function is bound. When the OpenGL ES function #1 is a glBindTexture function, the OpenGL ES function #1 is specifically configured to use a frame buffer attachment included in a target frame buffer as a texture input when the target frame buffer associated with the glbindframe buffer is bound, and draw an attachment (or a part of an attachment) of another frame buffer other than the target frame buffer by a drawing instruction when the other frame buffer is bound (bind).
And executing the Hook function associated with the Hook dynamic link library in the step S303, namely when the rendering thread calls the OpenGL ES function #1, mapping the Hook dynamic link library into the address space of the game application program, and accessing the Hook function included in the Hook dynamic link library to execute the Hook event 1 corresponding to the Hook function.
And executing the step S303, namely when the rendering thread calls the OpenGL ES function #1, firstly acquiring frame buffer attachment information associated with the OpenGL ES function #1 by using a custom Hook function, wherein in the mode, game codes associated with a game application process are not modified, and only when the OpenGL ES dynamic link library is loaded into the process game application process in a system driving layer, the function address of the OpenGL ES#1 function associated with the OpenGL ES dynamic link library is modified into the function address of the Hook function associated with the Hook dynamic link library.
After the above S303 is performed, the target frame buffer information may be obtained, where the target frame buffer information includes all frame buffer attachment information included in the target frame buffer. Any one of the frame buffer attachment information includes at least one of the following: rendering buffer object attachment information, or texture attachment information, the rendering buffer object attachment information representing the arbitrary frame buffer attachment as information of the rendering buffer object, and the texture attachment information representing the arbitrary frame buffer attachment as information of the texture attachment.
In this embodiment of the present application, the floodlight image is an image obtained by drawing data stored in memory addresses associated with a plurality of frame buffers corresponding to a plurality of frame buffer information, where frame buffer information corresponding to any one of the plurality of frame buffers may be obtained by using the method described in S303, and the area is that frame buffer IDs corresponding to different frame buffers are different, and frame buffer attachment IDs corresponding to different frame buffer attachments are also different.
S304, responding to a rendering thread in a game application program running in a terminal to call an OpenGL ES function #2, and executing a Hook function in a Hook dynamic link library file, so that a plurality of frame buffer information corresponding to the obtained floodlight image is sent to a computing thread, wherein the OpenGL ES function #2 is used for indicating that the drawing of the floodlight image is finished.
Wherein OpenGL ES function #2 is an example of a third function in the method described above in fig. 2). In the embodiment of the application, the Hook dynamic link library file is already loaded when the game application program runs, namely, the Hook function in the Hook dynamic link library file is always in a monitoring state in the running process of the game application program. And S304, namely executing the event 2 corresponding to the Hook function by the rendering thread under the condition that the Hook function monitors that the rendering thread in the game application program calls the OpenGL ES function # 2. Wherein, event 2 corresponding to the Hook function includes: and sending the plurality of frame buffer information corresponding to the obtained floodlight images to a calculation thread.
The OpenGL ES function #2 is used to indicate that the rendering of the floodlight image is finished, that is, the rendering of the floodlight image is finished after the OpenGL ES function #2 is called, and the subsequent rendering of the floodlight image is not performed any more. Specifically, the OpenGL ES function #2 may be an eglswappuffer function in OpenGL ES, where the eglswappuffer function is used to display the frame buffer UI drawn in S303 on the screen of the terminal, so as to indicate that the drawing of the floodlight image is ended.
S305, a computing thread in a game application program running in the terminal generates a directed acyclic graph #1 according to a preset generation strategy and a plurality of frame buffer information corresponding to the floodlight image, and obtains all paths included in the directed acyclic graph # 1.
The preset generation strategy comprises the following steps: establishing a plurality of nodes corresponding to a plurality of frame buffers associated with a plurality of frame buffer information, wherein the plurality of frame buffers are in one-to-one correspondence with the plurality of nodes, and any one node represents the corresponding frame buffer; and determining directed edges corresponding to any two nodes in a plurality of nodes corresponding to the frame buffer information according to the frame buffer information, wherein the directed edges corresponding to any two nodes are edges pointing to a node 2 from a node 1, the any two nodes comprise the node 1 and the node 2, at least one frame buffer accessory included in the frame buffer corresponding to the node 1 is used as a texture accessory to be bound to the frame buffer corresponding to the node 2, and the frame buffer corresponding to the node 1 is different from the frame buffer corresponding to the node 2.
The directed acyclic graph #1 includes a plurality of nodes and a plurality of edges. Wherein, a plurality of nodes are in one-to-one correspondence with a plurality of frame buffers associated with a plurality of frame buffer information, and each node represents a frame buffer associated with the corresponding frame buffer information. Any one of the plurality of edges represents a dependency between two frame buffers associated with the any one edge. Wherein the dependency is a dependency between a frame buffer attachment included in one frame buffer as a rendering buffer object type binding to another frame buffer. Alternatively, the dependency is a dependency between a frame buffer attachment included as a texture attachment type in one frame buffer being bound to another frame buffer.
In the embodiment of the present application, the floodlight image is an image obtained through floodlight processing, and the number of a plurality of frame buffers associated with a plurality of frame buffer information corresponding to the floodlight image may be 5, 7 or more, which is not limited specifically. For example, when the floodlight image corresponds to 5 frame buffers, that is, the data stored in the storage addresses corresponding to the 5 frame buffers are drawn, a floodlight image with floodlight effect can be obtained. Next, the directed acyclic graph obtained by the method of S305 described above will be described by taking the example of the different data frame buffers corresponding to the floodlight image. Illustratively, FIG. 4A shows a schematic diagram of a directed acyclic graph generated according to the method described above in S305 of FIG. 3. Referring to the DAG diagram shown in fig. 4A, the DAG diagram includes 12 nodes, and the 12 nodes respectively correspond to 12 frame buffers corresponding to the floodlight image, that is, the floodlight image with the floodlight effect can be obtained by drawing data stored in the 12 frame buffers. The DAG graph shown in fig. 4A also includes 14 edges, each edge representing a dependency between the frame buffers corresponding to the two nodes associated with each edge. In the following, edges 405-406 are illustrated as examples. Edges 405-406 associate node 405 with node 406, wherein edges 405-406 are a directed edge directed by node 405 to node 406, i.e., at least one frame buffer attachment included in the frame buffer corresponding to node 405 is bound as a texture attachment to at least one frame buffer attachment included in the frame buffer corresponding to node 406, i.e., at least one frame buffer attachment included in the frame buffer corresponding to node 405 is drawn as a texture attachment to an attachment of the frame buffer corresponding to node 406 by a drawing instruction. By way of example, fig. 5A shows a schematic diagram of another directed acyclic graph generated according to the method described in S305 above. Referring to the DAG graph shown in fig. 5A, the DAG graph includes 8 nodes and 10 edges, in this manner, the floodlight image corresponds to 8 frame buffers, that is, the floodlight image with the floodlight effect can be obtained by drawing the data stored in the 8 frame buffers. It will be appreciated that the node and edge understanding shown in fig. 5A is the same as that described above for the node and edge shown in fig. 4A, and details not described here in detail may be found in the above-related description.
The directed acyclic graph #1 described in S305 can include multiple paths, each having an ingress of 0 at a start node and an egress of 0 at a stop node. The implementation of performing all paths included in the obtained directed acyclic graph #1 described in S305 above is not particularly limited. For example, the computing thread may traverse all nodes included in the directed acyclic graph #1 one time, obtaining all nodes included in the directed acyclic graph #1 having an ingress of 0, and all nodes having an egress of 0; and for all nodes with the ingress degree of 0 and all nodes with the egress degree of 0, finding all paths included in the directed acyclic graph #1 in a parallelized depth-first traversal mode.
After the above S305 is performed, the directed acyclic graph #1 and all paths included in the directed acyclic graph #1 can be obtained. Wherein the directed acyclic graph #1 depicts a plurality of frame buffers (represented by nodes in the directed acyclic graph # 1) corresponding to the floodlight image, and dependencies between the plurality of frame buffers (represented by directed edges in the directed acyclic graph # 1).
Next, the number of downsampling performed to obtain a floodlight image is determined from the finite state automaton #1 and the finite loop diagram #1 described above in connection with S306.
S306, the calculation thread in the game application program running in the terminal determines the number of times of downsampling executed by obtaining the floodlight image according to the directed acyclic graph #1 and the finite state automaton.
The determining, by a computing thread in a game application running in the terminal, the number of downsampling performed to obtain a floodlight image according to the directed acyclic graph #1 and the finite state automaton may include the steps of: the computing thread in the game application running in the terminal determines a floodlight section corresponding to each path in the directed acyclic graph #1 according to the directed acyclic graph #1 and the finite state automaton, wherein the type of the node included in the floodlight section can be any one of the following types: initializing a node, downsampling a fuzzy node, an intermediate node, or an upsampling node; the number of the down sampling fuzzy nodes included in the floodlight interval is the same as the number of the up sampling nodes; determining the number of downsampling fuzzy nodes included in a floodlight interval corresponding to each path in the directed acyclic graph #1 as the number of downsampling fuzzy nodes included in each path; and determining the number of downsampling fuzzy nodes included in a target path in the paths in the directed acyclic graph #1 as the number of downsampling executed to obtain the floodlight image, wherein the number of downsampling fuzzy nodes included in a floodlight section corresponding to the target path is greater than or equal to the number of downsampling fuzzy nodes included in a floodlight section corresponding to any path except the target path in the paths. Optionally, in other implementations, the node type included in the flooding interval may be any one of the following types: downsampling a fuzzy node, an intermediate node, or an upsampling node.
In the embodiment of the present application, the principle of determining the floodlight section corresponding to any one path in the directed acyclic graph #1 is the same, and the principle of determining the number of down-sampling fuzzy nodes included in any one path according to the floodlight section corresponding to the any one path is the same. In the following, taking the example of the calculation thread determining the path #1 in the directed acyclic graph #1, the number of downsampled fuzzy nodes included in the path #1 is determined by using the finite state automaton in combination with S306-1, S306-2 and S306-3.
S306-1, a computing thread in a game application running in the terminal determines the type of each node in the path #1 in the directed acyclic graph #1 according to the directed acyclic graph #1 and the finite state automaton # 1.
The step S306-1 is executed, namely, the computing thread determines the type of each node in the directed acyclic graph #1 according to the directed acyclic graph #1 and the finite state automaton #1, wherein the type of each node is associated with a processing operation executed on a frame buffer corresponding to another node in the directed edges associated with each node, the data associated with the frame buffer corresponding to each node is data obtained by executing the processing operation on the frame buffer corresponding to the other node, and the other node is a father node of the one node. Wherein the processing operations at least include the following operations: initialization processing, downsampling processing, blurring processing, upsampling processing, and post-processing. For example, in the case where the type of one node in the directed acyclic graph #1 is a down-sampling processing node, that is, the type of the one node is determined based on performing down-sampling processing on a frame buffer corresponding to another node in the directed edge associated with the one node, and the data associated with the frame buffer corresponding to the one node is data obtained by performing the down-sampling processing on the frame buffer corresponding to the other node, which is a parent node of the one node.
The method comprises the steps that a computing thread in a game application program running in a terminal determines the node type of each node in a path #1 included in a directed acyclic graph #1 according to the input of a finite state automaton #1, wherein the input of the finite state automaton #1 comprises the following steps: the path #1 includes an outbound degree of each node, an inbound degree of each node, and a type of a node having a dependency relationship with each node, and an output of the finite state automaton #1 is a node type of each node in the path # 1. Table 3 below shows a mapping relationship between the input of the finite state automaton #1 and the output of the finite state automaton #1 provided in the embodiment of the present application.
TABLE 3 Table 3
Referring to table 3 above, it may be uniquely determined whether the current node is an up-sampling node, a down-sampling fuzzy node, or a general node according to the ingress of the current node and the egress of the current node. The current node may be uniquely determined to be an initialization node or an intermediate node based on the ingress of the current node, the egress of the current node, and another node type in the edge associated with the current node. It should be noted that, if the type of one node is an initialization node, the processing operation associated with the one node is initialization processing; the type of one node is a down-sampling fuzzy node, and then the processing operation associated with the one node comprises down-sampling processing and fuzzy processing; the type of one node is an intermediate node, and then the processing operation associated with the one node comprises downsampling processing and blurring processing; one node is of the up-sampling type, and the processing operation associated with that one node is down-sampling. In the case where the processing operation associated with one node is another processing operation other than the initialization node, the downsampling process, the blurring process, and the upsampling process, the type of the one node is a general node. Illustratively, continuing with the DAG graph shown in fig. 4A and using the determination criteria shown in table 3, the type of each node included in the DAG graph shown in fig. 4A may be obtained. By way of example, FIG. 4B illustrates the type of each node included in the DAG graph illustrated in FIG. 4A described above. Referring to fig. 4B, wherein node 401, node 402, node 403, node 404, node 411, and node 412 are all general nodes. For example, taking the node 401 as an example, the ingress degree of the node 401 is 0, the egress degree of the node 401 is 1, and the frame buffer corresponding to the node 401 is obtained by performing GPU particle calculation on the object to be rendered determined by the logical thread. For another example, taking the node 406 as an example, the ingress degree of the node 406 is 1, the egress degree of the node 406 is 2, and the frame buffer corresponding to the node 406 is obtained by performing downsampling and blurring on the frame buffer corresponding to the parent node 405 serving as the node 406 in the associated directed edge of the node 406. Illustratively, continuing with the DAG graph shown in fig. 5A and using the determination criteria shown in table 3, the type of each node included in the DAG graph shown in fig. 5A may be obtained. By way of example, FIG. 5B illustrates the type of each node included in the DAG graph illustrated in FIG. 5A described above.
After executing S306-1 described above, the computing thread may obtain the types of the plurality of nodes included in the path #1. Illustratively, taking fig. 5B as an example, path #1 may be: node 500- > node 501- > node 502- > node 503- > node 504- > node 505- > node 507, and node 501 is an initialization node, node 502 is a downsampling fuzzy node, node 503 is a downsampling fuzzy node, node 504 is an intermediate node, node 505 is an upsampling node, node 506 is an upsampling node, node 500 and node 507 are general nodes.
S306-2, a computing thread in a game application program running in the terminal determines a floodlight section corresponding to the path #1 according to the node type of each node in the path #1 and the finite state automaton # 2.
The inputs to finite state automaton #2 are: including path #1 for each node type. Taking fig. 5B as an example, the path #1 is: when node 500- > node 501- > node 502- > node 503- > node 504- > node 505- > node 506- > node 507, the input of the finite state automaton #2 is the following sequence: node 500 is a general node, node 501 is an initialization node, node 502 is a downsampling fuzzy node, node 503 is a downsampling fuzzy node, node 504 is an intermediate node, node 505 is an upsampling node, node 506 is an upsampling node, and node 507 is a general node. The output of the finite state automaton #2 is the floodlight section corresponding to the path #1.
The flooding interval corresponding to the path #1 includes a plurality of nodes in the path # 1. In some implementations, the type of the plurality of nodes is any one of the following types: initializing a node, downsampling a fuzzy node, an intermediate node, or an upsampling node; the floodlight section corresponding to the path #1 includes the number of downsampling fuzzy nodes equal to the number of upsampling nodes. The frame buffer corresponding to the intermediate node may be obtained by performing a down-sampling process and a blurring process on the frame buffer corresponding to the parent node.
In the embodiment of the present application, the finite state automaton #2 may be implemented by using a regular matching expression, where the regular matching expression is used to screen out the following types of nodes from the multiple nodes included in the path # 1: an initialization node, a downsampling fuzzy node, an intermediate node, or an upsampling node. It will be appreciated that after screening out the above types of nodes using regular matching expressions, the computing thread may also perform the following steps: it is determined that the number of down-sampling fuzzy nodes screened from the plurality of nodes included in the path #1 is equal to the number of up-sampling nodes. Illustratively, the regular matching expression may be: pd+mu+, wherein pd represents screening out the initialization node and the downsampling fuzzy node, and mu represents screening out the intermediate node and the upsampling node.
S306-3, determining the number of the downsampled fuzzy nodes included in the floodlight section corresponding to the path #1 as the number of the downsampled fuzzy nodes included in the path #1 by a calculation thread in the game application program running in the terminal.
Illustratively, taking fig. 5B as an example, path #1 may be: node 500- > node 501- > node 502- > node 503- > node 504- > node 505- > node 506- > node 507, and the flooding interval corresponding to path #1 includes the following nodes: node 502, node 503, node 504, node 505, and node 506, and the floods interval includes a number of downsampled fuzzy nodes of 3 (i.e., node 502, node 503, and node 504).
After executing S306-1 to S306-3 described above, the calculation thread may obtain the number of downsampled fuzzy nodes included in each of the plurality of paths included in the directed acyclic graph # 1. Thereafter, the computing thread determines the number of downsampling fuzzy nodes included in a target path in the multiple paths in the directed acyclic graph #1 as the number of downsampling performed to obtain the floodlight image, wherein the number of downsampling fuzzy nodes included in a floodlight section corresponding to the target path is greater than or equal to the number of downsampling fuzzy nodes included in a floodlight section corresponding to any one path except the target path in the multiple paths. Illustratively, continuing with the DAG graph shown in fig. 5B, the DAG graph includes 4 paths. Specifically, path 1: node 500— node 501— node 507. Path 2: node 500- > node 501- > node 502- > node 506- > node 507. Path 3: node 500- > node 501- > node 502- > node 503- > node 505- > node 506- > node 507. Path 4: node 500- > node 501- > node 502- > node 503- > node 504- > node 505- > node 506- > node 507. Based on this, it can be determined that the number of down-sampling fuzzy nodes included in the floodlight section corresponding to the path 4 is greater than the number of down-sampling fuzzy nodes included in the floodlight section corresponding to any one of the path 3, the path 2, or the path 1. That is, the number of layers of floodlight corresponding to the floodlight image corresponding to the DAG map shown in fig. 5B (i.e., the number of downsampling performed to obtain the floodlight image) is 3.
Fig. 3 above describes an example of rendering with a UE engine, in which data required for the game engine UE to implement downsampling and blurring processes related to floodlight processing is stored in one frame buffer. That is, one node in the finite loop diagram #1 is of the down-sampling fuzzy processing node type, and the processing operation associated with the one node includes: downsampling processing and blurring processing. Alternatively, when the floodlight processing is implemented by using the Unity engine, the data required for the downsampling processing and blurring processing involved are stored in two consecutive different frame buffers, respectively. That is, the type of the first node of the two consecutive nodes in the finite loop diagram #1 is a down-sampling processing node, and the processing operation associated with the first node is a down-sampling processing, the type of the second node of the two consecutive nodes is a blurring processing node, and the processing operation associated with the second node is a blurring processing (e.g., a gaussian blurring processing).
It should be understood that the data processing method shown in fig. 3 is merely schematic, and does not limit the data processing method provided in the present application. For example, in other implementations, the rendering thread and the computing thread in the method described in fig. 3 may be the same thread.
In the embodiment of the application, the rendering thread in the game application process loads the OpenGL ES dynamic link library file and the Hook dynamic link library file. In the running process of the game application, under the condition that a rendering thread in the game application process calls a drawing function in an OpenGL ES to draw a floodlight image in each frame, the rendering thread executes a Hook function in a Hook dynamic link library file to acquire a plurality of frame buffer information corresponding to the floodlight image drawn in the OpenGL ES from a memory, until the rendering thread calls the drawing function ending the first frame image, the process of acquiring the frame buffer information from the memory is ended, and the acquired plurality of frame buffer information corresponding to the floodlight image is sent to a calculation thread. After all frame buffer information corresponding to the floodlight image is acquired, a Hook function in a rendering thread continuously calls a drawing function in an OpenGL ES to finish drawing operation of a plurality of frame buffer information. The above-described manner of obtaining the plurality of frame buffer information does not require modification of the game code associated with the game application process, nor does it require invasive analysis of the game code. Further, the computing thread processes all frame buffer information corresponding to the obtained floodlight image to generate a directed acyclic graph #1 corresponding to the floodlight image, wherein the directed acyclic graph #1 describes all frame buffers corresponding to the floodlight image, and sequence information of generation and use among frame buffer attachments included in all frame buffers. And then, the computing thread traverses all paths of the directed acyclic graph by utilizing the finite state automaton, and judges the type of the frame buffer corresponding to each node in all paths. Then, the computing thread uses the finite state automaton to analyze the sequence of a plurality of nodes in each path included in the directed acyclic graph #1 and the types corresponding to the plurality of nodes, and determines the number of downsampled fuzzy nodes included in the floodlight interval corresponding to each path. And finally, determining the times of downsampling executed by the computing thread to obtain the floodlight image corresponding to the directed acyclic graph #1 according to the number of downsampling fuzzy nodes included in the floodlight section corresponding to each path in the paths included in the directed acyclic graph # 1. In the implementation manner, the Hook function hijacking drawing function in the OpenGL ES is called by each frame, so that the calculation of the directed acyclic graph and the finite automatic state machine generated according to the acquired multiple frame buffer information is calculated by each frame, and the real-time performance of the data is ensured. The rendering thread and the computing thread may be two different threads to prevent the logic computation and the screen update of the game from being blocked by the computation of the subsequent steps such as DAG reconstruction. The design of the finite state automaton is utilized to realize the detection function of an automatic floodlight section, and the manual check of stored data is avoided. In summary, the data processing method provided by the embodiment of the application does not need to acquire game codes or related configuration files, namely has strong universality; the method does not need to be manually participated, and the workload of manual analysis can be reduced; the method can acquire the times of downsampling processing corresponding to the floodlight image in real time.
In the above, the application scenario and the data processing method applicable to the data processing method provided in the application are described in detail with reference to fig. 1 to 5B. Next, a data processing apparatus and a data processing device provided in the present application are described with reference to fig. 6 and 7. It will be appreciated that the above data processing method corresponds to the data processing apparatus and data processing device hereinafter. Details not described in detail below can be found in the relevant description of the method embodiments described above.
Fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 6, the target program includes a rendering thread and a computing thread, the rendering program loads a first link library file and a second link library file, the first link library file includes a first function and a second function, the first function is used for drawing any one of a plurality of frame buffers associated with each frame to obtain a floodlight image, the plurality of frame buffers are in one-to-one correspondence with a plurality of storage addresses of the terminal, and any one of the frame buffers is data to be rendered stored in the corresponding storage address; the second function indicates that the drawing of each frame is ended; the device comprises: a first processing unit 601 and a second processing unit 602.
The first processing unit 601 is configured to execute the second link library file, so as to perform the following operations: executing Hook operation on the first function and executing data acquisition events for acquiring the plurality of frame buffers from the plurality of storage addresses under the condition that the rendering thread calls the first function in each frame; and executing the Hook operation on the second function and executing a data transmission event for transmitting the acquired frame buffers to the computing thread under the condition that the rendering thread calls the second function in each frame; the second processing unit 602 is configured to: in response to receiving the plurality of frame buffers sent by the rendering thread, determining a number of downsampling processes performed to obtain the floodlight image based on the plurality of frame buffers.
Fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application. As shown in fig. 7, includes a memory 701, a processor 702, a communication interface 703, and a communication bus 704. The memory 701, the processor 702 and the communication interface 703 are in communication connection with each other through a communication bus 704.
The memory 701 may be a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a random access memory (random access memory, RAM). The memory 701 may store a program, and when the program stored in the memory 701 is executed by the processor 702, the processor 702 and the communication interface 703 are used to perform the respective steps of the data processing method of the embodiment of the present application.
The processor 702 may employ a general-purpose central processing unit (central processing unit, CPU), microprocessor, application specific integrated circuit (application specific integrated circuit, ASIC), graphics processor (graphics processing unit, GPU) or one or more integrated circuits for executing associated programs to perform functions required by the elements of the data processing apparatus of the embodiments of the present application or to perform the various steps of the data processing methods of the embodiments of the present application.
The processor 702 may also be an integrated circuit chip with signal processing capabilities. In implementation, various steps of the data processing method provided herein may be implemented by integrated logic circuitry of hardware in the processor 702 or instructions in the form of software. The processor 702 may also be a general purpose processor, a digital signal processor (digital signal processing, DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 701, and the processor 702 reads information in the memory 701, and in combination with hardware thereof, performs functions required to be executed by units included in the data processing apparatus of the embodiment of the present application, or executes the data processing method of the embodiment of the method of the present application.
The communication interface 703 enables communication between the device shown in fig. 7 and other devices or communication networks using transceiving means such as, but not limited to, a transceiver.
The embodiments of the present application provide a computer readable storage medium, where the computer readable storage medium includes computer instructions, which when executed by a processor, are configured to implement any one of the data processing solutions in the embodiments of the present application.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored on a computer readable medium, including several instructions to cause a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage media, or any other non-transmission media, which can be used to store information that can be accessed by a computing device. Computer readable media, as defined herein, does not include non-transitory computer readable media (transmission media), such as modulated data signals and carrier waves.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
While the preferred embodiment has been described, it is not intended to limit the invention thereto, and any person skilled in the art may make variations and modifications without departing from the spirit and scope of the present invention, so that the scope of the present invention shall be defined by the claims of the present application.
Claims (13)
1. The data processing method is characterized by being applied to a terminal running a target program, wherein the target program comprises a rendering thread and a computing thread, the rendering program loads a first link library file and a second link library file, the first link library file comprises a first function and a second function, the first function is used for drawing any one of a plurality of frame buffers associated with each frame to obtain a floodlight image, the plurality of frame buffers are in one-to-one correspondence with a plurality of storage addresses of the terminal, and any one of the frame buffers is data to be rendered stored in the corresponding storage address; the second function indicates that the drawing of each frame is ended; the method comprises the following steps:
The rendering thread executes the second linked library file to:
executing Hook operation on the first function and executing data acquisition events for acquiring the plurality of frame buffers from the plurality of storage addresses under the condition that the rendering thread calls the first function in each frame; the method comprises the steps of,
executing the Hook operation on the second function and executing the data transmission event of buffering and sending the acquired frames to the computing thread under the condition that the rendering thread calls the second function in each frame;
in response to receiving the plurality of frame buffers sent by the rendering thread, the computing thread determines a number of downsampling processes performed to obtain the floodlight image in accordance with the plurality of frame buffers.
2. The method of claim 1, wherein the computing thread determining the number of downsampling processes performed to obtain the floodlight image based on the plurality of frame buffers comprises:
the computing thread generates a directed acyclic graph comprising a plurality of paths according to the plurality of frame buffers, wherein any one of the plurality of paths comprises at least one directed edge; the at least one directed edge is an edge directed by the first node to the second node; binding at least one frame buffer attachment included in a first frame buffer corresponding to the first node to a second frame buffer corresponding to the second node as a texture attachment, so that the at least one frame buffer attachment included in the first frame buffer is drawn to the second frame buffer when the second frame buffer is drawn; the second frame buffer is obtained by performing preset processing on the first frame buffer, and the type of the second node is associated with the preset processing; the plurality of frame buffers includes the first frame buffer and the second frame buffer;
And the calculation thread determines the times of downsampling processing executed by obtaining the floodlight image according to the directed acyclic graph.
3. The method of claim 2, wherein the object program further comprises a logic thread, wherein the logic thread is configured to determine an object to be drawn corresponding to the floodlight image, wherein the plurality of frame buffers are configured to indicate a manner in which drawing is performed on the object to be drawn,
the ingress degree of the starting node of any path is zero, the egress degree of the ending node of any path is zero, and the frame buffer corresponding to the starting node of any path is determined according to the object to be drawn corresponding to the floodlight image.
4. A method according to claim 3, wherein the computing thread determining the number of downsampling processes performed to obtain the floodlight image from the directed acyclic graph comprises:
the computing thread determining a target area of the any one of the paths, wherein the type of any one of the nodes in the target area of the any one of the paths is associated with a first process; the first treatment is at least any one of the following: downsampling, blurring, or upsampling, the preset processing including the first processing; the number of nodes of the first type in the target area of any one path is equal to the number of nodes of the second type; the first type is associated with the downsampling process and the second type is associated with the upsampling process;
The computing thread determines a target area according to a first path in the paths according to a plurality of target areas of the paths, and determines the number of times of downsampling processing executed by obtaining the floodlight image, wherein the number of nodes of the first type in the target area of the first path is larger than or equal to the number of nodes of the first type in the target area of a second path; the second path is any one path except the first path in the paths;
the computing thread determines a number of nodes of the first type in a target area of the first path as a number of downsampling processes performed to obtain the floodlight image.
5. The method of claim 4, wherein the computing thread determining a target area for the any one path comprises:
the calculation thread determines the type of any node according to a preset mapping relation, attribute information of any node included in any path and the type of the node except any node in a directed edge associated with any node, wherein the preset mapping relation is a mapping relation between the attribute information of the node and the type of the node; the attribute information of the node comprises the outgoing degree of the node and the incoming degree of the node;
The calculation thread determines a target area of any path according to the sequence information of the any path and a preset expression, wherein the sequence information of the any path comprises a plurality of bits, the ith bit in the plurality of bits is the type of the ith node pointing to a termination node from a start node in the any path, and i is a positive integer; the preset expression is used to identify a node of a type associated with the first process.
6. The method of claim 4 or 5, wherein before the computing thread determines the target region for either path, the method further comprises:
and the calculation thread performs path search processing on the directed acyclic graph to obtain the paths.
7. The method according to any one of claims 1 to 6, further comprising:
the rendering thread executes the second linked library file to:
and ending the Hook operation executed on the first function and the second function and calling the first function and the second function when the rendering thread is monitored to send the acquired frame buffers to the computing thread, so that any frame buffer in the frame buffers is drawn at each frame to obtain the floodlight image.
8. The method of any of claims 1-7, wherein the computing thread is a child of the rendering thread, the method further comprising, before the rendering thread sends the plurality of frame buffers associated with the captured flood image to the computing thread:
in response to listening to invoking the second function within each frame, the rendering thread creates the computing thread.
9. The method of any of claims 1-7, wherein the rendering thread and the computing thread are two different threads included by the target program.
10. The method according to any one of claims 1 to 9, wherein,
the target program is a game application program, and the first link library file is an open graphic library file OpenGL or an embedded system open graphic library OpenGL ES.
11. The data processing device is characterized by being applied to a terminal running a target program, wherein the target program comprises a rendering thread and a calculating thread, the rendering program loads a first link library file and a second link library file, the first link library file comprises a first function and a second function, the first function is used for drawing any one of a plurality of frame buffers associated with each frame to obtain a floodlight image, the plurality of frame buffers are in one-to-one correspondence with a plurality of storage addresses of the terminal, and any one of the frame buffers is data to be rendered stored in the corresponding storage address; the second function indicates that the drawing of each frame is ended; the device comprises:
The first processing unit is used for:
executing Hook operation on the first function and executing data acquisition events for acquiring the plurality of frame buffers from the plurality of storage addresses under the condition that the rendering thread calls the first function in each frame; the method comprises the steps of,
executing the Hook operation on the second function and executing the data transmission event of buffering and sending the acquired frames to the computing thread under the condition that the rendering thread calls the second function in each frame;
the second processing unit is used for: in response to receiving the plurality of frame buffers sent by the rendering thread, determining a number of downsampling processes performed to obtain the floodlight image based on the plurality of frame buffers.
12. A data processing apparatus, comprising: a memory and a processor, the memory and the processor coupled;
the memory is used for storing one or more computer instructions;
the processor is configured to execute the one or more computer instructions to implement the method of any of claims 1 to 10.
13. A computer readable storage medium having stored thereon one or more computer instructions executable by a processor to implement the method of any of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310145912.7A CN116385611A (en) | 2023-02-14 | 2023-02-14 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310145912.7A CN116385611A (en) | 2023-02-14 | 2023-02-14 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116385611A true CN116385611A (en) | 2023-07-04 |
Family
ID=86960541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310145912.7A Pending CN116385611A (en) | 2023-02-14 | 2023-02-14 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116385611A (en) |
-
2023
- 2023-02-14 CN CN202310145912.7A patent/CN116385611A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113457160B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| US20230360337A1 (en) | Virtual image displaying method and apparatus, electronic device and storage medium | |
| KR101239029B1 (en) | Multi-buffer support for off-screen surfaces in a graphics processing system | |
| CN112933599A (en) | Three-dimensional model rendering method, device, equipment and storage medium | |
| US11475636B2 (en) | Augmented reality and virtual reality engine for virtual desktop infrastucture | |
| CN112652046B (en) | Game picture generation method, device, equipment and storage medium | |
| US11631216B2 (en) | Method and system for filtering shadow maps with sub-frame accumulation | |
| US11232623B2 (en) | Method and system for creating a neural net based lossy renderer | |
| CN116385611A (en) | Data processing method and device and electronic equipment | |
| CN116503529A (en) | Rendering, 3D picture control method, electronic device, and computer-readable storage medium | |
| CN114330675A (en) | Chip, accelerator card, electronic equipment and data processing method | |
| CN115311397A (en) | Method, apparatus, device and storage medium for image rendering | |
| CN113837918A (en) | Method and device for realizing rendering isolation by multiple processes | |
| CN114821001B (en) | AR-based interaction method and device and electronic equipment | |
| Fernández et al. | How mobile phones perform in collaborative augmented reality (CAR) applications | |
| CN115511886B (en) | Method, device and storage medium for realizing remote target statistics by using GPU | |
| CN111009025B (en) | Bone animation hiding method, device and equipment | |
| US11663768B2 (en) | Multi-process compositor | |
| WO2023168999A1 (en) | Rendering method and apparatus for virtual scene, and electronic device, computer-readable storage medium and computer program product | |
| Ohlenburg et al. | Morgan: A framework for realizing interactive real-time AR and VR applications | |
| CN115845363A (en) | Rendering method and device and electronic equipment | |
| CN110262798B (en) | WebGL-based graphic drawing method, device, operator and readable storage medium | |
| CN118015166A (en) | Rendering method and device for user interface special effects, electronic equipment and storage medium | |
| CN115525852A (en) | Webpage rendering method, device, equipment and storage medium | |
| CN117014689A (en) | Bullet screen display method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |