JP5281660B2 - Shader-based extension for declarative presentation framework - Google Patents

Description

  The present invention relates to a shader-based extension for a declarative presentation framework.

  [0001] A graphics processing unit (GPU) is a dedicated graphics rendering device for a computer. Today's GPUs are efficient at manipulating and displaying computer graphics, and their parallel structure makes them more efficient than the most general-purpose central processors for a series of complex algorithms. The CPU may also have multiple processor cores. Like the GPU, the multi-core CPU can execute a plurality of operations in parallel.

  [0002] Today, software developers who write software want to take advantage of this processing power in the software applications they write. Taking advantage of the power of GPUs or multi-core processors that exist in computers in applications that require a large amount of processing resources, such as desktop-based or web-based applications that utilize many graphic effects or multimedia Useful. However, while many platforms can take advantage of GPU or multi-core processor resources, end developers typically run applications on platforms that can take advantage of the extra processing power included in many computers today. Cannot be easily built.

  [0003] Various techniques and techniques for declaratively controlling a shader are disclosed. The declarative programming model allows the use of declarative statements that control the instantiation of shaders in a declarative presentation framework. A declaratively specified shader is later instantiated (instantiated, instantiated) by a presentation framework for rendering graphic effects for software applications (giving graphic effects to software applications) ). In one embodiment, the declarative programming model can be used as a single pass shader that controls and encapsulates the shader to be executed during a single pass operation. In another embodiment, a method is described for utilizing a declarative programming model as a multi-pass effect that controls and encapsulates a set of shaders to be executed during multiple pass operations. . A custom method is called for multi-pass effects. Custom methods are derived from multi-pass effect classes. The custom method has a context that allows the custom method to access shader context and control information to allow the custom method to control the behavior of the multi-pass effect.

  [0004] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. .

[0005] FIG. 1 is a diagram of a system for controlling the operation of a pixel shader. [0006] FIG. 2 is a diagram of a declarative programming model for one embodiment that enables declarative statements to control shader instantiation and behavior in a declarative presentation framework. [0007] FIG. 4 is a process flow diagram for one embodiment showing the stages involved in defining shader-based effects using a declarative programming model. [0008] FIG. 2 is a diagram for one embodiment showing a brush being used as an input to a shader. [0009] FIG. 2 is an exemplary source code for one embodiment illustrating the use of a brush as a secondary input to a shader and an implicit input to a shader. [0010] FIG. 4 is a diagram for one embodiment showing the binding of properties to shader inputs. [0011] FIG. 4 is an exemplary source code for one embodiment illustrating the combination of characteristics to shader inputs. [0012] FIG. 6 is a diagram for one embodiment illustrating the use of a shader in defining an effect brush. [0013] FIG. 3 is an example source code for one embodiment illustrating the use of shaders in defining an effect brush. [0014] FIG. 6 is a diagram for one embodiment showing hit testing by using a general transformation by a shader and / or by executing the shader on the pixel itself. [0015] FIG. 4 is an exemplary source code for one embodiment illustrating hit testing with the use of a generic transformation by a shader. [0016] FIG. 6 is a process flow diagram for one embodiment illustrating a multi-pass effect of controlling and encapsulating a set of shaders to be executed during multi-pass operation. [0017] FIG. 4 is a diagram for one embodiment illustrating the flow of control from a user interface thread to a configuration thread. [0018] FIG. 2 is a diagram of a computer system of one embodiment.

  [0019] Although techniques and techniques herein may be described in a general context as techniques and techniques for using shaders with a declarative programming model, the techniques and techniques may also be described herein. In addition it serves other purposes. In one embodiment, one or more techniques described herein are MICROSOFT®. Implemented as a function within a software development platform, such as the NET framework, or from other types of programs or services that provide a platform to allow developers to develop and / or customize software applications can do.

  [0020] A diagram of a system 100 for controlling the operation of a shader 106 using a declarative programming model 108 running on a declarative presentation framework is shown in FIG. As used herein, the terms “declarative statement” and “declarative” are text-based markup or instructions that entrust execution to the basic implementation of the relevant declarative framework. Intended to contain imperative code. Examples of declarative languages include XAML and XML, to name a few non-limiting examples. As used herein, the term “declarative programming model” is used by developers, other users, or programs to instantiate and use effects to be handled by the relevant declarative framework. It is intended to include a programming model that sets declaration statements by programming model. As used herein, the term “declarative presentation framework” is a more abstract description provided by a developer, another user, or program by declarative statements using a declarative programming model. It is intended to include a system that allows and displays interactions with. One non-limiting example of a declarative presentation framework is MICROSOFT Windows® Presentation Foundation (ie, WPF), which is MICROSOFT. It is a graphical subsystem function of the NET framework. WPF provides a consistent programming model for building applications and provides a clear separation between user interface logic and business logic. Another non-limiting example of a declarative presentation framework is MICROSOFT Silverlight.

  [0021] The term "shader" as used herein is intended to include a set of computer instructions that are used primarily by graphic resources to perform graphical rendering effects. There are different types of shaders that can be utilized in the system 100, such as pixel shaders, vertex shaders and / or shape (placement, geometry) shaders. Vertex shaders affect a series of vertices and can therefore change vertex properties such as position, color and texture coordinates. Vertices calculated by the vertex shader are usually passed to the shape shader. The shape shader can add and remove vertices from the mesh. A shape shader can be used to generate a shape (placement) according to a procedure or to add volumetric details to an existing mesh, which is usually used for processing on a processor. Is too expensive. Pixel shaders are more commonly known as fragment shaders and compute the color value of individual pixels when the polygons generated by the vertex and shape shaders are rasterized. Pixel shaders are typically used for scene lighting and related effects such as bump mapping and color adjustment.

  [0022] A shader 106 is executed on one or more processing units 104. A few non-limiting examples of processing unit 104 may include a graphics processing unit (GPU) or a multi-core processor. When the shader 106 is executed on one or more processing units 104, the term “shader processor” is used to refer to the execution of a shader on the processing unit 104. As previously mentioned, shader 106 is responsible for effecting (rendering) for a given software application. The declarative programming model 108 allows developers or other users to provide user-declared shader customization 110. As described in further detail herein in the flow and code example of FIGS. 2-13, shader customization 110 allows a user to control the instantiation and / or operation of shader 106.

  [0023] FIG. 2 illustrates a line of a declarative programming model 150 for one embodiment that allows declarative statements to control shader instantiation and behavior in a declarative presentation framework. FIG. The declarative programming model 150 supports various functions that can be specified declaratively for shader control. In one embodiment, the declarative programming model 150 can be used as a single pass shader that controls and encapsulates the shader to be executed during a single pass operation. In another embodiment, the declarative programming model 150 can be used as a multi-pass effect that controls and encapsulates a set of shaders to be executed during a multi-pass operation. The multi-pass effect is described in more detail in the discussion of multi-pass effect 164 herein and in the discussion of FIG.

  [0024] Using the declarative programming model 150, a brush can be specified as an input 152 to the shader. As used herein, the term “brush” is intended to represent an object that, when applied to fill a shape, can determine which color to place at which pixel location. More abstractly, a brush represents a 2D space to color mapping function. Exemplary brushes include simple raster images, linear and radial gradients, live video sources, sources from other vector graphic renderings, and the like. In addition, the brush serves as an input to the pixel shader and can be presented to the shader itself as a “sampler” to be processed by the shader. Further details about utilizing brushes as input to the shader are provided in the discussion of FIGS. 4-5.

  [0025] The declarative programming model 150 allows potential inputs to be provided to the shader processor and controls where the graphic effect gets its input 154, such as its primary input. As used herein, the term “implicit input” includes “a pixel-based bitmap resulting from rasterizing a user interface element to which an effect / shader is applied”. I intend to. Thus, for example, if an effect is applied to a button, the “potential input” is a bitmap representing that button. An example of a potential input is described in more detail in FIG.

  [0026] Property binding to the shader input 156 is also supported by the declarative programming model 150. As used herein, the term “characteristic binding” is a declarative type of object-oriented class definition that can be referenced to perform animation when performing data binding or otherwise performing flexible control. Intended to include properties. . In some embodiments, such properties are known as Dependency Properties. As used herein, the term “shader input” is intended to include both “shader constants” and “shader samplers” that are made available to shaders via a register index. The term “shader constant” as used herein is intended to include values provided to a shader program that remain constant throughout the execution of the shader program. A few non-limiting examples of shader constants include floating point values, colors, points, vectors, matrices, etc. As used herein, the term “shader sampler” is intended to include multidimensional arrays provided to a shader program that can be sampled within the shader program at specific coordinate points. A few non-limiting examples may include 2D bitmaps and 3D texture volumes. In other words, the declarative programming model 150 allows dependency properties to be declaratively connected to shader inputs. Characteristic coupling for shader input 156 is described in more detail in FIGS. In one embodiment, by supporting property binding for shader inputs, those dependent properties use shaders (via higher level objects that contain property declarations) that need to know more about the shader implementation. Can be passed to shaders without developers.

  [0027] The declarative programming model 150 also allows effect brushes to be defined 158 to act as brushes from shader execution. As used herein, the term “effect brush” includes any “brush” (as defined above) that defines its function from 2D space to color by the execution of a shader or series of shaders. Intended. Effect brushes can generally be used in the same manner as other brushes. Assuming a set of explicit inputs, the effect brush generates colors to meet the target to which the brush is applied. Using this feature, the system does not need to render anything in advance to execute the shader. In other words, the effects brush function allows the shader to run as a brush for user interface elements. The effect brush is described in more detail in FIGS.

  [0028] Another feature supported by the declarative programming model 150 is the hit test and coordinate space transformation using the generic transformation 160 to model how the effect changes the input. It is. The term “hit test” as used herein is intended to include the processing of a request for the position of a particular point in the coordinate space after the shader is applied. As used herein, the term “coordinate space transformation” includes the need to undergo shader-based effects and having code that transforms coordinates in the same way that a shader potentially transforms pixels. It is intended to include any mapping along the transformation hierarchy. An effect can move its content around so that when interacted the mouse position or input pointer position needs to be transformed via the inverse of the transformation to which the effect is applied. Hit testing using the generic transformation 160 function allows a way for certain effects to shift so that the input is modeled. For example, if the effect “swirl” a button, the input pointer is on the representation of the rotated button and the space where the button originally existed but no longer exists You should register as being “above” the button when it is not. The hit test is described in more detail in FIGS.

  [0029] The declarative programming model 150 also supports hit testing by executing shaders on the pixels themselves 162. This function allows the shader to run on a particular pixel to obtain the actual pixel coordinates. This represents one possible embodiment for performing the coordinate space mapping required for hit testing as described above. This is described in more detail in FIG.

  [0030] The multi-pass effect 164 is also supported by the declarative programming model 150. Multiple pass effects can have multiple levels of effect, embodied as a set of operations that occur on one shader at a time. Developers can use the declarative programming model 150 to control and extend multiple passes of shaders. The multi-pass effect 164 is described in more detail in FIG.

  [0031] Turning now to FIGS. 3-13, steps for implementing one or more embodiments of a system for controlling shader operation will be described in greater detail. In some embodiments, the process of FIGS. 3-13 is at least partially implemented in the operational logic of computing device 500 (of FIG. 14).

  [0032] FIG. 3 is a process flow diagram 200 for one embodiment showing the stages involved in defining shader-based effects using a declarative programming model. Declaratively referenced shader-based effects are programmatically instantiated (step 202) by a declarative presentation framework for rendering (rendering) graphic effects to software applications. A few non-limiting examples of shaders may include pixel shaders, vertex shaders, and / or shape shaders. The declaration statement is accessed for the software application (step 204). In other words, declaration statements that include graphic effect customizations for shaders are retrieved from one or more files that contain declaration statements. In one embodiment, declaration statements are accessed throughout the operation of the shader as they are needed by the shader (step 204). A declaration statement is sent to the shader processor to allow the shader to provide a graphic effect to the software application through graphic effects customization (step 206). In other words, the declaration statement is sent to the shader processor and used by the shader to perform the desired graphic effects customization specified by the developer, others or the user of the program in a declarative manner. In one embodiment, the declaration statement allows the graphics effects customization to utilize a processing unit such as a GPU or multi-core processor on which the shader is executed.

  [0033] FIG. 4 is a diagram 220 for one embodiment showing a brush 226 defined using one or more declaration statements 224 being used as input to the shader 222. Brushes allow features such as radial gradients, bitmap images, animations, vector graphics, etc. to serve as input to shaders (as opposed to legitimate bitmaps supported by some existing shaders) To. An example code to illustrate the brush concept in more detail is shown in FIG.

  [0034] FIG. 5 is an exemplary source code 250 for one embodiment that illustrates the use of a brush as a secondary input to a shader and illustrates the use of a potential input to the shader. . In the example shown, a button called myButton is declaratively specified. The button is declared with a button effect that includes a brush 254 as an input for the bottom stage characteristic (bottom characteristic) of the texture subtraction effect. The vertex characteristic of the texture subtraction effect includes a potential input characteristic 252 representing the bitmap rendering of the element to which the effect is applied—in this case, the rendering of myButton. The net result is that the button now looks like a rendering of the original button with the image from the brush 254 subtracted from it.

  [0035] FIG. 6 is a diagram 280 of one example illustrating the coupling of one or more dependency characteristics 288 to one or more shader inputs 284, which may be shader constants and / or samplers. As previously mentioned, shader inputs are made available to shaders by register index. Dependency characteristics can be specified using a declaration statement 286 to bind to the shader input 284 of a given shader 282. In one embodiment, by connecting the dependency characteristic with the shader input, the dependency characteristic can be defined and passed to the shader without the user having to worry about the details of the shader. An example of combining dependency characteristics to shader inputs (more specifically, to shader constants) in a declarative manner is shown in FIG. In the example source code shown in FIG. 7, two dependencies (304 and 306) on shader constants are specified declaratively. Those dependencies (304 and 306) are coupled to the actual registration of the shader 302, such as shader constants 308 and 310. In one embodiment, the binding is accomplished through the use of a callback, which updates the shader input with new values for the effect and thus the next time the shader is executed, whenever the dependency property is changed. This means that a callback function is called that tells the declarative framework to do so.

  [0036] FIG. 8 is a diagram 320 for one example illustrating the use of shaders in defining an effect brush. As described in FIG. 2, the effect brush 326 can be defined using a declaration statement 324 to act as a brush from the execution of the shader 322. Considering a set of explicit inputs, the effect brush 326 generates colors to meet the goal to which the brush is applied. The effect brush function allows shader 322 to be implemented as a brush for user interface elements. Exemplary source code 330 for declaratively specifying effect brush 326 is shown in FIG. In the example shown in FIG. 9, the effect brush is specified declaratively by the custom effect 334 (332). The custom result 334 in the example shown produces a fractal fill for the rectangle. In this example, the effect brush is used with a single parameter, Effect, which is an Effect type. In addition, MandelbrotEffect is a custom ShaderEffect with three dependency characteristics passed to the basic shader as shader input. The effect output is then used to fill the rectangle.

  [0037] Turning now to FIG. 10, the performance of hit testing and coordinate space transformation is shown by using a generic transformation 356 by the shader 352 and / or executing the shader 352 at a particular pixel 358. A diagram for one embodiment is shown. As previously mentioned, hit testing refers to the processing of a request for the position of a specified point in coordinate space after a shader is applied. A generic transformation 356 can be specified using the declaration statement 354, allowing hit testing and coordinate space transformations to be modeled. An example of a general transformation is shown in the example source code 380 of FIG. Alternatively or additionally, hit testing can be performed by executing a shader 352 on a particular pixel 358.

  [0038] FIG. 12 is a process flow diagram 440 for one embodiment illustrating the multi-pass effect of controlling and encapsulating a set of shaders to be executed during multi-pass operation. The term “multi-pass effect” as used herein is intended to include effects that potentially call more than one shader multiple times to achieve the desired rendering. As used herein, the term “multi-pass operation” is intended to include code that controls the execution of multiple passes and multiple shaders. The custom method is invoked for multi-pass effects (stage 442). In one embodiment, the custom method is derived from a multiple pass effect class. In one embodiment, a custom method is defined such that it overrides the method of applied shaders (eg, ApplyShaders). The applied shader method includes logic to programmatically select the respective shader for each pass of the multi-pass effect in the custom method.

  [0039] The custom method allows the custom method to access shader context and control information to allow the custom method to control the behavior of the multi-pass effect (stage 446). Step 444) Provide context. In other words, the custom method is invoked by the system to access relevant information such as shader index, current global scale, shader constant, shader sampler and / or ability to select and execute shaders and It has a “context” that allows control. This mechanism allows multi-pass effect developers in custom methods to control multi-pass effect rendering.

  [0040] In one embodiment, by using the multi-pass effect feature, a user of a particular subclass of a multi-shader effect base class can include the effect in a declarative graphic or user interface representation. It is only necessary to specify the effect declaratively.

  [0041] In one embodiment, graphic effects customization is invoked on the configuration thread for all passes in a given multi-pass effect. The configuration thread is described in more detail in FIG. 13, which will now be discussed next.

  [0042] FIG. 13 is a diagram 480 for one example illustrating the flow of control from a user interface thread to a configuration thread. As used herein, the term “user interface thread” is intended to include the main thread of execution to which the user interface elements (buttons, text boxes, lists) of the application respond. As used herein, the term “configuration thread” is intended to include a different thread than the user interface thread that controls the rendering of the display. Operations on the constituent thread are not blocked by operations on the UI thread.

  [0043] Returning to FIG. 13, control of the flow is shown between the user interface thread and the configuration thread. In the user interface thread, the declaration statement is parsed, effects are built (stage 482), and then a visual tree containing the effects is built (stage 484). In the composition thread, what corresponds to the visual tree on the composition side is constructed (step 486). Returning to the user interface thread, the effect dependent properties are combined and evaluated (stage 488). While present in the user interface thread, the user interface thread rendering loop evaluates the animation and invokes the configuration rendering (stage 490). The composition thread then renders the frame (stage 492) and renders the hit on the visual tree node with the effect (stage 494). In one embodiment, what happens next varies based on the type of effect being rendered (stage 496). In the case of a multi-pass effect, the ApplyShaders method is called from the configuration thread to control the execution of the effect. In the case of a single pass effect, there is already enough information on the configuration thread for the configuration thread to simply execute the effect without invoking any user code.

  [0044] As shown in FIG. 14, an exemplary computer system for use in implementing one or more portions of the system includes a computing device, such as computing device 500. In its most basic configuration, computing device 500 typically includes at least one processing unit 502 and memory 504. Depending on the exact configuration and type of computing device, the memory 504 may be volatile (such as RAM) or non-volatile (such as ROM, flash memory, etc.). Or some combination of the two. Most of this basic configuration is illustrated in FIG.

  [0045] Furthermore, the apparatus 500 may also have additional features / functions. For example, the device 500 may also include additional storage devices (removable and / or non-removable), including but not limited to magnetic or optical disks or tapes. Such additional storage devices are illustrated in FIG. 14 by removable storage device 508 and non-removable storage device 510. Computer storage media is volatile and non-volatile, removable and non-removable implemented in any method or technique for storing information such as computer readable instructions, data structures, program modules or other data. Includes media. Memory 504, removable storage 508 and non-removable storage 510 are all examples of computer storage media. Computer storage media can be RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or This includes, but is not limited to, other magnetic storage devices, or any other medium that can be used to store desired information and that can be accessed by device 500. Any such computer storage media may be part of device 500.

  [0046] The computing device 500 includes one or more communication connections 514 that allow the computing device 500 to communicate with other computers / applications 515. The device 500 may also include an input device 512 such as a keyboard, mouse, pen, voice input device, touch input device. An output device 511 such as a display, a speaker, or a printer may also be included. These devices are well known in the art and need not be discussed at length here.

  [0047] Although the subject matter has been described in language specific to structural features and / or methodological actions, it is understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. It should be. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. All equivalents, changes, and modifications within the spirit of the embodiments as described herein and / or as described by the following claims are desired to be protected.

  [0048] For example, one of ordinary skill in the computer software arts will appreciate that one example described herein includes fewer or additional options or features than those expressed in the examples. It will be appreciated that the above computers can be organized differently.

Claims (17)

A system for controlling the operation of a shader,
A declarative programming model that allows declarative language declaration statements to be used to control the instantiation and behavior of shaders in a declarative presentation framework, the shaders executing on a graphics processing unit The software application uses the declarative programming model because the declarative programming model utilizes the graphic processing device for graphic effects in the software application. And the declarative programming model is operable to allow a brush to be used as an input to the shader .   The system of claim 1, wherein the declarative language is an extensible markup language (XML).   The system of claim 1, wherein the declarative language is an extended application markup language (XAML).   The system of claim 1, wherein the shader is operable to execute on a computing device comprising one or more cores. The declarative programming model is Ri operable der to allow the potential input is designated to control or obtain the input where the effect of being rendered by the shader, potential The system of claim 1 , wherein the input includes a pixel-based bitmap that results from rasterizing a user interface element to which the effect is applied .   The system of claim 1, wherein the declarative programming model is operable to allow one or more dependency characteristics to be coupled to a shader input.   The system of claim 1, wherein the declarative programming model is operable to allow an effect brush to be defined that acts as a brush by executing the shader.   The system of claim 1, wherein the declarative programming model is operable to model how effects change input. Effects modeled on what kind alter the input system of claim 8 which is performed by executing the shader on a particular pixel.   The system of claim 1, wherein the declarative programming model is operable to allow multiple pass effects to be specified. A method for defining shader-based effects and using said shader-based effects in a declarative programming model, comprising:
Programmatically instantiating declaratively specified shader-based effects to render graphic effects for software applications;
Accessing a declaration statement for the software application, including graphic effects customization for a shader, wherein the declaration statement is written in a declarative language, the declarative language being an extended markup language or an extended application markup Including language, steps,
Sending the declaration statement to a shader processor to enable the shader to render a graphic effect for the software application with the graphic effects customization , the declaration statement including a brush Allows the input to be used as a secondary input to the shader, allows the potential input to be used as an input to the shader, and Comprising a pixel-based bitmap resulting from rasterizing the interface elements . The method of claim 11 , wherein the declaration statement is accessed throughout the operation of the shader when the declaration statement is required by the shader. The method of claim 11 , wherein the declaration statement enables the graphics effects customization to use a processing unit on which the shader is executed. The method of claim 11 , wherein the shader is a type of shader selected from the group consisting of a pixel shader, a vertex shader, and a shape shader. Using a declarative programming model as a multi-pass effect to control and encapsulate a set of shaders to be executed during multi-pass operation,
Invoking a custom method for multi-pass effects, derived from a multi-pass effect class, using a declaration statement written in a declarative language including an extended markup language or an extended application markup language;
Providing the custom method with a context that allows the custom method to access shader context and control information to allow the custom method to control the operation of the multi-pass effect. How to prepare. The method of claim 15 , wherein the graphic effects customization is invoked on a configuration thread for all paths in a multi-pass effect in the custom method. 16. The method of claim 15 , wherein the custom method overrides an apply shader method, and the apply shader method includes logic for programmatically selecting a respective shader for each pass of a multi-pass effect in the custom method.

Images