GB2564075A

GB2564075A - Multi-Threaded rendering system

Info

Publication number: GB2564075A
Application number: GB1704631.9A
Authority: GB
Inventors: li Adrian; Ng Roy
Original assignee: Pridefield Ltd
Current assignee: Pridefield Ltd
Priority date: 2017-03-23
Filing date: 2017-03-23
Publication date: 2019-01-09
Anticipated expiration: 2037-03-23
Also published as: GB2564075B; AU2018201757B2; GB201704631D0; AU2018201757A1

Abstract

Generating worker threads in a rendering system wherein a request to execute an application utilising a display is received 910, a layout time for providing a layout for frames of the application on the display and a time to render said frames are determined 920, a number of worker threads is generated based on this determination 930 such that the frames of the application can be displayed by executing the generated worker threads in parallel with a rendering thread which renders the frames of the application 940. The determination of the layout and render time may include a determination based on the actual work performed by the worker and render threads. If the layout time is greater than the render time, then the number of worker threads may be incremented, or conversely decrementing the number of worker threads if the render time is greater than the layout time. Historical data from previous executions of the application may be logged such that layout and rendering time can be determined based on the historical data.

Description

[0001] Some computer applications, such as simulations, games, and other applications, display animations. These animations can be drawn as a series of “frames”, or images to be displayed on a screen of a computing device, and then displayed on the screen. Each frame can be generated using a frame layout, or specification for arranging items within the display, and rendered, or drawn, by setting pixels of the frame to desired colour values as specified by the application and, in some cases, as specified by the layout.

[0002] The present disclosure relates to a method of implementing applications that display frames of imagery that takes use of multiple core processing power extant on many devices to increase animation smoothness and facilitate visual content with responsive design.

OVERVIEW [0003] Example embodiments are described herein. In a first respect, an example embodiment takes the form of a method. A computing device receives a request to execute an application that utilizes a display. The computing device determines a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display. The computing device generates a number of worker threads to provide the layout for the frames of the application on the display based on the layout time and the rendering time. The computing device displays the frames of the application on the display by executing the generated number of worker threads of the computing devi ce in parallel with a rendering thread of the computing device that renders the frames of the application.

[0004] In a second aspect, an article of manufacture is provided. The article of manufacture includes a computer-readable medium, having stored thereon instructions that, upon execution by a processor of a computing device, cause the computing device to perform operations. The operations include: receiving a request to execute an application that utilizes a display, determining a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display; generating a number of worker threads to provide the layout for the frames of the application on the display based on the layout time and the rendering time, and displaying the frames of the application on the display by executing the generated number of worker threads in parallel with a rendering thread that renders the frames of the application.

[0005] In a third respect, a computing device is provided. The computing device includes: one or more processors, memory; and program instructions, stored in the memory, that upon execution by the one or more processors cause the computing device to perform the operations. The operations include: receiving a request to execute an application that utilizes a display, determining a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display; generating a number of worker threads to provide the layout for the frames of the application on the display based on the layout time and the rendering time; and displaying the frames of the application on the display by executing the generated number of worker threads in parallel with a rendering thread that renders the frames of the application.

[0006] In a fourth respect, a computing device is provided. The computing device includes: means for receiving a request to execute an application that utilizes a display; means for determining a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display; means for generating a number of worker threads to provide the layout for the frames of the application on the display based on the layout time and the rendering time; and means for displaying the frames of the application on the display by executing the generated number of worker threads in parallel with a rendering thread that renders the frames of the application.

BRIEF DESCRIPTION OF THE DRAWINGS [0007] Example embodiments are described herein with reference to the drawings, in which: [0008] FIG. 1 is a diagram of a single-threaded application, in accordance with an example embodiment;

[0009] FIG. 2 is a diagram of a multi-threaded application, in accordance with an example embodiment;

[0010] FIG. 3 is a diagram of object representations for the multi-threaded application of FIG. 2, in accordance with an example embodiment;

[0011] FIG. 4 depicts an execution profile for the single-threaded application of FIG. 1 and an execution profile for the multi-threaded application of FIG. 2, in accordance with an example embodiment;

[0012] FIG. 5 is a flow chart il lustrating a method related to determining a number of threads for a multi-threaded application, in accordance with an example embodiment [0013] FIG. 6 depicts execution profiles for the multi-threaded application of FIG. 2, in accordance with an example embodiment;

J.

[0014] FIG. 7 depicts additional execution profiles for the multi-threaded application of FIG. 2, in accordance with an example embodiment, [0015] FIG. 8 is a block diagram of a computing device, in accordance with an example embodiment; and [0016] FIG. 9 is a flow chart of functions to cany out a method, in accordance with an example embodiment.

DETAILED DESCRIPTION

I. INTRODUCTION [0017] Herein are disclosed techniques for utilizing multiple threads, or independently managed sequences of computer instructions, for computer software applications that display animated imagery, such as a computer animation, game, video, or related applications. In this document, unless otherwise specifically stated, the term “application” refers to a computer software application.

[0018] To provide animated imagery, an application can divide the animated imagery into frames. The frames can be laid out and rendered by the multiple threads of the application. The multiple threads can include one or more worker threads that lays out each frame, and a rendering thread that renders each laid out frame.

[0019] The use of multiple threads allows sharing of rendering, layout, user interface, and other workloads between threads, or in some cases, “cores” or computing hardware units, of a computing device. By taking advantage of load sharing and utilization of multiple cores, the herein-described multi-threaded application can achieve better performance than a singlethreaded application based on performance metrics such as a higher number of frames per seconds (FPS). A higher FPS can lead to a smoother and more realistic experience of animated imagery provided to a user of the application.

[0020] As the application executes, some aspects of providing animated imagery can take more computer resources, such as computing time, than others. For example, some animated imagery takes a relatively-long time to render in comparison to the amount of time taken to layout the frames of the animated imagery; that is, rendering time is relatively long in comparison to layout time. In other examples, the reverse is true; that is, layout time is relatively long in comparison to rendering time. Further, for some applications, the relative amounts of rendering time and layout time can change during application execution.

[0021] The number and/or performance of the threads can be adjusted based on rendering time and layout time values. In some applications, there can only be one rendering thread. In these applications, if rendering time is relatively long in comparison to layout time, a number of worker threads can be reduced to one worker thread. Then, if rendering time is still relatively long in comparison to layout time even with one worker thread, the worker thread can be “controlled” or instructed to wait for an acknowledgment from the rendering thread before laying out another frame, thus ensuring frame layout does not get ahead of frame rendering. In these same one-rendering-thread applications, if layout time is relatively long in comparison to rendering time, the number of worker threads can be increased, and so provide additional work to the rendering thread, while reducing work on previously-executing worker threads. Thus, adjusting thread performance and numbers based on rendering time and layout time values can further enhance multi-threaded applications that provide animated imagery.

II. EXAMPLE MULTI-THREADED APPLICATIONS [0022] FIG. 1 is a diagram of a single-threaded application 102, in accordance with an example embodiment. Single-threaded application 102 can be executed on computing device 100 to carry out the procedures of method 112 to support a user interface that includes animated imagery that is displayed on a display, such as a screen of computing device 100. For example, singlethreaded application 102 can provide an animation, play a game, display videos, enable interactive communication, and/or provide other functionality while supporting a user interface that includes animated imagery. Single-threaded application 102 can be implemented using one or more computer languages, such as, but not limited to, JavaScript®.

[0023] Method 112 can begin at block 120, where single-threaded application 102 can check for and receive any user input available to the application. The user input can include, but is not limited to, user input related to controlling the application, user input providing information to the application, user input responding to the application (e.g., “click on the OK button to proceed”), and user input related to exiting and/or restarting the application.

[0024] At block 122, single-threaded application 102 can determine whether the user input of block 120 is user input related to exiting and/or restarting the application. If the user input of block 120 is user input related to exiting and/or restarting the application, single-threaded application 102 can proceed to block 124, where single-threaded application 102 can end method 112 and exit or restart, depending on the user input of block 120.

[0025] If the user input of block 120 is not user input related to exiting and/or restarting the application or of no user input was available at block 120, single-threaded application 102 can proceed to block 130.

[0026] At block 130, single-threaded application 102 can execute application logic to support the user interface and perhaps other functionality of the application 102. The execution of application logic can be based on (e.g., controlled by) the user input received at block 120. For example, the functionality of application 102 can include providing animated imagery on a frame-by-frame basis, and user input received at block 120 can control the animated imagery. In addition, single-threaded application 102 can update a display of the animated imagery based on the received user input; i.e., determine a response, if any, of the application to the user input; and the application logic can determine contents of a new display frame, or “frame” for short, and render the new frame on a screen of the computing device.

[0027] At block 140, single-threaded application 102 can determine contents of a new frame based on the execution of application logic carried out at block 130.

[0028] At block 150, single-threaded application 102 can layout and render the new frame.

[0029] At block 160, single-threaded application 102 can display the new frame. After displaying the new frame, single-threaded application 102 can proceed to block 120 to check for additional user input.

[0030] FIG. 2 is a diagram of multi-threaded application 202, in accordance with an example embodiment. The above-mentioned functionality of single-threaded application 102 can be implemented by multi-threaded application 202. Multi-threaded application 202 can be executed on computing device 200 to carry out the procedures of methods 290 and 292 to support a user interface that includes animated imagery that is displayed on a display, such as a screen of computing device 200. For example, multi-threaded application 202 can provide an animation, play a game, display videos, enable interactive communication, and/or provide other functionality while supporting a user interface that includes animated imagery.

[0031] In general, multi-threaded application 202 utilizes two or more threads to support a user interface that includes animated imagery that is displayed on a display. Multi-threaded application 202 utilizes one or more worker threads, such as one or more instances of worker thread 210, and one rendering thread, such as rendering thread 212. During execution, the number of worker threads can vary, as described in more detail below. In some examples, controls that impose an ordering of operations between the rendering thread and the worker thread can be enabled and disabled on the worker thread, as also described in more detail below. Multi-threaded application 202 can be implemented in one or more computer languages.

[0032] In some embodiments, responsive design principles can be embodied in multi-threaded application 202, where responsive design principles can include an approach that application design and development respond to the user behavior and environment based on screen size, platform and orientation. By reducing potentially-complex changes between frames to a minimum, multi-threaded application 202 can be more responsive to user behavior. In other embodiments, these procedures can enable application development based on a virtual screen or “canvas”, such as used in using HTML, and can allow JavaScript control of HTML objects, 0_NEW, and/or O__OLD. In particular embodiments, generating a frame for display in a responsive design application can include determining a layout for the frame and rendering the frame.

[0033] In other embodiments, computing device 200 can include one or more processors. The processor(s) of computing device 200 can include a plurality of cores. In particular embodiments, each core of computing device 200 can execute at least one thread of multithreaded application 202.

[0034] In single-threaded application 102, layout and rendering are executed by one thread sequentially. With multi-threaded application 202, layout of frames occurs in worker thread 210, which can execute in parallel with rendering performed by in rendering thread 212. This use of parallelism can reduce the amount of clock time utilized to generate and display frames of the application. Thus, multi-threaded application 202 can increase FPS and reduce user-interaction response time in comparison to single-threaded application.

[0035] In some embodiments, computing device 200 can receive a request to execute an application that utilizes a display, such as multi-threaded application 202. Computing device 200, upon receiving the request, can execute multi-threaded application 202. Multi-threaded application 202 and/or computing device 200, can then: determine a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display, generate a number of worker threads of the computing device to provide the layout for the frames of the application on the display based on the layout time and the rendering time; and display the frames of the application on the display by executing the generated number of worker threads of the computing device in parallel with a rendering thread that renders the frames of the application.

[0036] Multi-threaded application 202 can begin by creating and executing at least two threads: one worker thread and one rendering thread. Each worker thread can execute the procedures of method 290 to execute application logic, layout frame, and provide screen update messages that instruct the rendering thread what to draw while rendering a frame. Each rendering thread can execute the procedures of method 292 to process user input and render frames.

[0037] In the example shown in FIG. 2, multi-threaded application 202 executes two threads: worker thread 210 and rendering thread 212. Upon execution, rendering thread 212 can carry out the procedures of block 214 of method 292 to check for and receive any user input available to the application. User input is discussed in more detail in the context of block 120 of FIG. 1.

[0038] At block 216 of method 292, rendering thread 212 can determine whether the user input of block 214 is user input related to exiting and/or restarting the application. If the user input of block 214 is user input related to exiting and/or restarting the application, multi-threaded application 202 can proceed to block 218, where multi-threaded application 202 can end method 292 and exit or restart, depending on the user input of block 214.

[0039] If the user input of block 214 is not user input related to exiting and/or restarting the application or if no user input was available at block 214, rendering thread 212 can proceed with method 292.

[0040] Method 292 can continue with rendering thread 212 using thread communication component 220r to send user input message 222 to worker thread 210 to corresponding thread communication component 220w of worker thread 210. The thread communication components can bridge the threads of the application, including threads 210 and 212, to enable inter-thread communication. In some examples, no user input is received at block 214, in those examples, a user input message indicating no input was received can be sent as user input message 222. In still other examples, if no user input is received at block 214, no user input message 222 is sent by thread communication component 220r. Subsequently, user input message 222 (if sent) can be received at thread communication component 220w of worker thread 210.

[0041] Worker thread 210 can execute method 290. In particular, upon receiving user input message 222, worker thread 210 can carry out the procedures of block 230 to execute application logic to support the user interface and perhaps other functionality of the application 202. The execution of application logic can be based on (e.g., controlled by) user input in user input message 222. For example, the functionality of application 202 can include providing animated imagery on a frame-by-frame basis, and user input in user input message 222 can control the animated imagery. In addition, multi-threaded application 202 can update a display of the animated imagery based on the received user input; i.e., determine a response, if any, of the application to the user input; and the application logic can determine contents of a new frame, and render the new frame on a screen of the computing device.

[0042] At block 240, rendering thread 212 can layout a new frame. The layout of a new frame can include generating an object representation O NEW of the new frame. O NEW can be generated by merging data generated at block 230 and/or other data into an object representation of a previous frame OPREV. For example, OPREV can represent the frame immediately preceding the new frame represented by O NEW.

[0043] At block 250, worker thread 210 can compare object representation O PREV (for the frame preceding the new frame) to the object representation O NEW for the new frame to determine a minimum display change MDC to be rendered on the screen of the computing device. Worker thread 210 can generate drawing instructions DI that instruct rendering thread 210 to draw⁷ one or more pixels that represent the minimum display change MDC, as illustrated in both FIGS. 2 and 3. By performing this comparison to reduce potentially-complex changes between frames to a minimum, worker thread 210 can enable relatively-fast yet rich graphic processing, including frame layout and rendering.

[0044] FIG. 3 is a diagram of object representations 300, 310 for multi-threaded application 202, in accordance with an example embodiment. In the context of block 250, worker thread 210 includes OPREV object representation 300 and O NEW object representation 310. Both OPREV object representation 300 and O NEW object representation 310 have a tree structure. Then, worker thread 210 can perform a tree structure comparison O PREV object representation 300 and O NEW object representation 310 to determine minimum display change 320. By determining minimum display change 320 between O PREV object representation 300 and O NEW object representation 310, worker thread 210 can minimize the screen update workload, leading to shorter graphic rendering time.

[0045] In the particular example shown in FIG. 3, both O PREV object representation 300 and O NEW object representation 310 have six objects arranged using respective tree structures. O PREV object representation 300 has a root, or top-most, object A that refers to objects B and C. Object B of object representation 300 does not refer to any other objects, while object C of object representation 300 refers to objects D and E. Object D of object representation 300 does not refer to any other objects, while object E of object representation 300 refers to object F of object representation 300, which in turn does not refer to any other objects.

[0046] Also, 0_NEW object representation 310 has a root object A that refers to objects B and C. Object B of object representation 310 does not refer to any other objects, while object C of object representation 310 refers to objects D and E’. Object D of object representation 310 does not refer to any other objects, while object E’ of object representation 310 refers to object F’ of object representation 310, which in turn does not refer to any other objects.

[0047] The tree structure comparison of O PREV object representation 300 and O NEW object representation 310 can involve starting a root object of each object representation, comparing objects between object representations, and recursively moving down the tree structures of O PREV object representation 300 and O NEW object representation 310 to compare objects referred to by the root object and other objects in the tree structures of O PREV object representation 300 and O NEW object representation 310.

[0048] In the example shown in FIG. 3, the tree structure comparison can indicate that objects A, B, C, and D are the same in both OPREV object representation 300 and O NEW object representation 310, but object E of O PREV object representation 300 differs from object E’ of O NEW object representation 310 and object F of O PREV object representation 300 differs from object F’ of 0_NEW object representation 310. These differences can be recorded as minimum display changes 320 of objects E’ and F’. Then, worker thread 310 can convert minimum display changes 320 recording objects E’ and F’ into drawing instructions 330 that represent minimum display changes 320.

[0049] Returning to FIG. 2, at block 260, worker thread 212 can generate screen update message 262, which can include a copy of drawing instructions DI. In some embodiments, screen update message 262 can also include other information, such as object representation O NEW of the new frame. Drawing instructions DI can include one or more instructions that instruct the rendering thread to draw one or more pixels of the new frame.

[0050] Once generated, worker thread 212 can provide screen update message 262 to thread communication component 220w. Thread communication component 262w can send screen update message 262 to thread communication component 220r of rendering thread 212. Also at block 260, worker thread 212 can copy, rename, and/or otherwise designate the O NEW object representation as O PREV for the next iteration of method 290. Upon completion of block 260, method 290 can be completed.

[0051] Subsequently, screen update message 262 can be received at thread communication component 220r of rendering thread 212, which is executing method 292. In particular, upon receiving screen update message 262, rendering thread 212 can cany out the procedures of block 270 to unpack screen update message 262 and get drawing instructions DI from screen update message 262. In some embodiments, unpacking screen update message 262 can yield additional information, such as object representation 0_NEW of the new frame.

[0052] At block 280, rendering thread 212 can render the new frame as directed by drawing instructions DI. Rendering the new frame can include drawing a set of minimum changes, as specified by drawing instructions DI, onto a frame, frame buffer, canvas, screen, and/or other displayable representation of graphical information.

[0053] For example, drawing instructions DI can include one or more instructions that instruct the rendering thread to draw one or more pixels of the new frame, and rendering thread 212 can render by drawing the requested one or more pixels of the new frame. In some examples, rendering the new frame can include displaying the new frame on a display; e.g., a screen of computing device 200. In these examples, drawing one or more pixels of the new frame can, in effect, be drawing one or more pixels on the display In other examples, rendering frames of multi-threaded application 202, including rendering the new frame, can include displaying the new frame on a display using a web browser executing on computing device 200; e.g, using a web browser such as Safari®, Firefox®, Chrome™, Internet Explorer®, Microsoft Edge™, etc.

[0054] In some embodiments, upon rendering a new frame, rendering thread 212 can use thread communication component 220r to send an acknowledgment that the frame was rendered to worker thread 210. For example, rendering thread 212 can send an acknowledgment that a frame was rendered to worker thread 210 if controls are enabled on worker thread 210, where controls on worker threads are discussed in more detail at least in the context of FIGS. 5 and 7. Upon completion of block 280, rendering thread 212 can proceed to block 214.

[0055] In some embodiments, multi-threaded application 202 can display frames at a frame rate that provides smooth animation and responsive experience; e.g., a frame rate of at least 30 frames per second, and perhaps greater than 60 frames per second. In these embodiments, multithreaded application 202 can carry out the procedures of both methods 290 and 292 at least 30, and perhaps 60, times per second.

[0056] In some examples, data about the application, including data about layout of frames of the application, rendering of the frames of the application, can be provided in one or more style sheets or other files. The style sheet can include one or more name-value pairs that specify a name of a data item and a related value. For example, the following name value pair:

“backgroundColor” : 0, can specify that a data item named “backgroundColor” has a value of “0”.

[0057] In some examples, the value can be determined based on a calculation. For example, the following name-value pair can determine a value for a data-item named “width” “width” : “calc(min( 50 pw / 350 px , 50 ph / 200 px. ) * 200 px)” [0058] The value specification “calc(min( 50 pw / 350 px, 50 ph / 200 px ) * 300 px)” indicates that a calculation “calc” is to be performed. The calculation first determines a minimum “min” of two values: “50 pw / 350 px” and “50 ph / 200 px” and then that minimum is multiplied by “200 px”. In this example, “pw” stands for a percentage of a parent window’s width, “ph” stands for a percentage of a parent window’s height, and “px” stands for a number of effective pixels. Many other examples of name value pairs and calculations used to determine name-value pairs for applications, including multi-threaded applications used to provide animated imagery, are possible as well.

[0059] FIG. 4 depicts execution profile 410 for single-threaded application 102 and execution profile 420 for multi-threaded application 202, in accordance with an example embodiment.

Both execution profiles 410 and 420 are displayed using a common time axis, shown as a rightward-pointing arrow in FIG. 4.

[0060] A herein-disclosed execution profile graphically represents amounts of time taken during application execution. The execution profiles displayed in FIGS. 4, 6, and 7 represent both layout times and rendering times, where layout times indicate amounts of time taken during layout of frames by an application, and where rendering times indicate amounts of time taken while rendering frames by the application. In. each of FIGS. 4, 6, and 7, layout times are graphically represented using solid white rectangles, and rendering times are graphically represented using vertically-lined rectangles.

[0061] FIG. 4 shows single-threaded application execution profile 410 for single-threaded application 102 for a period of time while eight frames are displayed by single-threaded application 102. In this example, each of the eight frames takes approximately the same amount of layout time and rendering time. As shown in single-threaded application execution profile 410, the layout time for each frame directly precedes the rendering time for the frame, and so the total amount of time to display each frame by single-threaded application 102 is at least a sum of the layout time and the rendering time for a frame.

[0062] Multi-threaded application execution profile 420 includes worker thread execution profile 430 for one worker thread utilized by multi-threaded application 202, and rendering thread profile 440 for one rendering thread utilized by multi-threaded application 202. Both the worker thread and the rendering thread were utilized to render 13 frames in this example, even though the rendering time and the layout time taken for each frame is approximately equal between single-threaded application 102 and multi-threaded application 202.

[0063] Multi-threaded application execution profile 420 indicates that frame display can be increased by taking advantage of overlapping time between operations performed by the worker thread and operations performed by the rendering thread. For example, multi-threaded application execution profile 420 shows that for frame 9 rendered by multi-threaded application execution profile 420, overlapping time 452 where both threads were simultaneously executing is a relatively large portion of a total time 450 to display frame 9. Similar results can be observed in multi-threaded application execution profile 420 for rendering of other frames by multithreaded application 202. In other examples, a different, perhaps higher, degree of speedup (i.e., a ratio of a number of frames processed by multi-threaded application 202 to a number of frames processed by single-threaded application per unit time) can be achieved than shown in multithreaded application execution profile 420.

[0064] Layout computations and rendering computations can be complex. Interleaving of layout processing and rendering processing, such as shown above by multi-threaded application execution profile 420, can increase utilization of a computing device executing multi-threaded application 202. In some examples, the use of multipl e threads can increase use of multiple cores and other hardware of the computing device. Some computing devices have 2, 4, or even more cores to be utilized by application software. In cases where layout is relatively complex compared to rendering; that is, layout time is more than rendering time, multiple copies of the worker thread can be initiated and executed to layout frames that are rendered by the rendering thread, and so utilize multiple cores and other hardware of the computing device to increase the number of frames per second displayed by multi-threaded application 202.

[0065] In cases where rendering is relatively complex compared to layout; that is, rendering time is more than layout time, the rendering thread can be overworked by multiple worker threads, perhaps leading to decreased and/or inaccurate performance. In these cases, a worker thread can be “controlled” or instructed to wait for an acknowledgement from a rendering thread before sending a screen update message. The use of multiple worker threads and controlling worker threads can both balance application workloads across multiple cores and save processing powder by stopping and/or controlling threads. In some computing devices, such as smart phones and other mobile devices, saving processing power can significantly reduce battery usage, and thus prolong time between battery charges.

[0066] FIG. 5 is a flow chart illustrating method 500 related to determining a number of threads for a multi-threaded application, in accordance with an example embodiment. Method 500 can balance workload between rendering and worker threads, by determining a number of worker threads for multi-threaded application 202 and by determining whether a worker thread should be controlled.

[0067] M ethod 500 can be carried out by a computing device executing multi-threaded application 202, such as computing device 200. In some embodiments, method 500 can be embodied as software of multi-threaded application 202; then, execution of multi-threaded application 202 can involve execution of method 500.

[0068] Method 500 can begin at block 510, where computing device 200 can determine a number W of worker threads, where W > 0. W can be determined based on a hardcoded value (e.g., W is set to a pre-determined value, such as 1, 2, ...), based on aspects of the computing device executing multi-threaded application 202, based on historical data, and/or based on other information. W can be determined based on aspects of the computing device, such as, but not limited to, a number of cores of the computing device, a size of a screen used to provide a display for multi-threaded application 202, an amount of memory of the computing device, and available types of processors / cores of the computing device (e.g., availability of a graphics processing unit (GPU)). W can be determined based on historical data, such as, but not limited to, one or more values of W (that is, the number of worker threads) utilized by one or more previous executions of multi-threaded application 202 and one or more rendering time and/or layout time values observed during one or more previous executions of multi-threaded application 202.

[0069] To record historical data for later retrieval, computing device 200 and/or multi-threaded application 202 can maintain a log of historical data about the application, wherein the historical data comprises one or more of: layout-time data indicating one or more layout times observed execution of the application on the computing device, rendering-time data indicating one or more rendering times observed during execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during execution of the application on the computing device.

[0070] The log can be a file or other storage structure that include a number of entries, where each entry includes part or all of the above-mentioned historical data. For example, a log entry can include values for W, RT, LT, and/or a timestamp indicating when the log entry was recorded. Then, historical data can be read from the log entries in the log and processed to determine historical data for use by method 500, such as one or more values of W (that is, the number of worker threads ) utilized by one or more previous executions of multi-threaded application 202 and one or more rendering time and/or layout time values observed during one or more previous executions of multi-threaded application 202.

[0071] Other information, such as user-defined data, and/or combinations of these data can be used to determine W at block 510.

[0072] At block 520, the computing device can start multiple threads of multi-threaded application 202 to execute in parallel, including W worker threads and one rendering thread. In some examples, the computing device can include multiple cores; in these examples, each of the multiple cores can execute separate threads. In particular, suppose that the computing device has four cores and W == 3; then three of the four cores can execute the three worker threads and the fourth core can execute the rendering thread. Many other examples of computing device cores executing threads are possible as well.

[0073] At block 530, the computing device can execute the W worker threads and one rendering thread of multi-threaded application 202 to layout, render, and display one or more frames.

[0074] At block 540, the computing device can determine whether the worker threads and rendering thread should be balanced; e.g., change the number of worker threads and/or control worker threads. For example, the computing device can balance threads after a pre-determined amount of time has taken place since the threads were initially started and/or since the threads were previously balanced, after a pre-determined number of frames has been displayed the threads were initially started and/or since the threads were previously balanced, after user inputrelated to balancing threads is received. Other criteria for determining whether the worker threads and rendering thread should be balanced are possible as well.

[0075] If the computing device determines that the threads should be balanced, the computing device can proceed to block 550. Otherwise, the computing device determines that the threads should not be balanced, and the computing device can proceed to block 530.

[0076] At block 550, the computing device can determine an amount of rendering time RT and an amount of layout time LT. The amount of rendering time RT can represent an amount of time taken by the rendering thread to render frames for multi-threaded application 202; that is, RT can be based on an actual amount of work performed by rendering thread. The amount of layout time LT can represent an amount of time taken by the W worker threads to layout frames for multithreaded application 202; that is, LT can be based on an actual amount of work performed by the W worker threads.

[0077] In some examples, rendering time RT (and/or layout time LT) can represent an amount of time taken by the rendering thread (layout thread(s)) to render (layout) frames since a previous rebalancing of threads; i.e., since the last time block 550 was executed. In other examples, rendering time RT and/or layout time LT can be expressed as an amount of time per frame; that is, rendering time RT (and/or layout time LT) can be divided by a number of frames NF displayed during rendering time RT (layout time LT). For example, if rendering time RT == 1 second and NF = 200 frames, then the rendering time per frame is 1/200 = 0.005 seconds / frame. Similarly, if layout time LT = 1.2 second for the same NF = 200 frames, then the layout time per frame is 1.2/200 == 0.006 seconds / frame, [0078] At block 560, the computing device can determine whether rendering time RT is equal and/or approximately equal to layout time LT. A test of approximate equality between RT and LT can be used to determine whether a workload on the rendering thread as represented by the RT value is relatively balanced / approximately equal to a workload on the worker thread(s) as represented by the LT value.

[0079] To determine whether rendering time RT is approximately equal to layout time LT, an value D1FF can be determined as an absolute value of a difference between RT and LT, and the value DIFF can be compared to a threshold value; e.g., if DIFF is less than 0.002 seconds, 0.1 seconds, 0.5 seconds, 1 second or another threshold value, then RT can be considered to be approximately equal to LT; otherwise, RT is not approximately equal to LT. Another technique is to determine a ratio between LT and RT, e.g., a ratio RTIME - LT / RT. If RTIME is between a lower ratio threshold that is less than one and an upper ratio threshold that is greater than one; e.g,, RTIME is between a lower ratio threshold of 0.5, 0.66667, 0.8, 0.95, or another value less than one, and an upper ratio threshold of 1.1, 1.25, 1.3333, 2 or another value greater than one, then RT can be considered to be approximately equal to LT; otherwise, RT is not approximately equal to LT. Other techniques to determine approximate equality are possible as well.

[0080] After performing a comparison of relative equality between LT and RT, the computing device can determine which one of three cases exist: (1) LT and RT are approximately equal, (2) LT and RT are not approximately equal and LT is greater than RT, and (3) LT and RT are not approximately equal and RT is greater than LT. For case (1), since layout time LT and rendering time RT are approximately equal, it is likely that the workloads between worker thread(s) and the rendering thread are relatively balanced, and so rebalancing of workloads between the rendering thread and the worker thread(s) may not be necessary'.

[0081] In cases (2) and (3), since LT and RT are not approximately equal, it is likely that relative workloads between worker thread(s) and the rendering thread are out of balance, and so rebalancing of workloads may take place. For case (2), since layout time LT is greater than rendering time RT, the worker thread(s) likely have a relatively higher workload than the rendering thread, and so additional work can be provided and/or shifted to the rendering thread. For case (3), since rendering time RT is greater than layout time LT, the rendering thread likely has a relatively higher workload than the worker thread(s), and so additional work can be provided and/or shifted to the worker thread(s).

[0082] If rendering time RT is approximately equal to layout time LT, the computing device can proceed to block 530. However, if rendering time RT is not approximately equal to layout time LT, the computing device can proceed to block 562.

[0083] The procedures of blocks 562, 570, 572, 574, 576, 580, 582, 584, and 586 are designed to rebalance workloads between the rendering thread and the W worker threads. If the rendering thread is relatively overloaded, then RT should be greater than LT, and the procedures of blocks 570, 572, 574, and 576 can be used to reduce the relative workload on the rendering thread. If the W worker threads are relatively overloaded, then LT should be greater than RT, and the procedures of blocks 580, 582, 584, and 586 can be used to reduce the relative workload on the W worker threads.

[0084] At block 562, the computing device can compare the rendering time RT with the layout time LT. If RT is greater than LT, then the computing device can proceed to block 570; otherwise LT is greater than RT, and the computing device can proceed to block 580.

[0085] At block 570, the computing device can determine whether the number of worker threads W is greater than one. If W is greater than one, the computing device can proceed to block 572; otherwise, the computing device can proceed to block 576.

[0086] At block 572, the computing device can determine a number of threads NW for ceasing execution (i.e., killing), where NW is less than W; as at least one worker thread must be executing or else multi-threaded application will not be able to layout frames. The number NW can be determined based on: a pre-determined value (e.g, 1), a pre-determined percentage of W (e.g., 10%, 20%, 50%), a combination of values (e.g., a minimum or maximum of a predetermined value of 1 and a value of a pre-determined percentage of W), or another value that is less than W.

[0087] At block 574, the computing device can set W equal to W minus NW and cease execution of NW worker threads. As such, the value of W after carrying out the procedures of block 574 is the value of worker threads still executing after rebalancing. /After carrying out the procedures of block 574, the computing device can proceed to block 530.

[0088] At block 576, the number of worker threads W equals one and the rendering thread is overloaded. Then, at block 576, the computing device can enable controls on the worker thread in an attempt to reduce the workload on rendering thread. The controls on the worker thread, when enabled, can instruct the worker thread wait for an acknowledgement from a rendering thread before sending a screen update message. That is, a controlled worker thread waits for an acknowledgement from the rendering thread that a previous frame has been rendered before sending a screen update message requesting rendering a new frame by the rendering thread. In some embodiments, the controls are initially disabled for all worker threads, and so the controls may only be operable when explicitly enabled.

[0089] For example, suppose a controlled worker thread sends screen update message SUMI to the rendering thread to render a frame Fl. Then, after rendering a frame in accordance with screen update message SUMI, the rendering thread will send an acknowledgment ACK1 to the controlled worker thread. Only after the controlled worker thread has received acknowledgment ACK1 can the controlled worker thread send a second screen update message SUM2 to request the rendering thread render a new frame F2. Note that if the controls on the worker thread were disabled, the uncontrolled worker thread could send SUMI and then send SUM2 to render respective frames Fl and F2 without waiting for an acknowledgement (or any other message) from the rendering thread.

[0090] A controlled worker thread can be put into an idle mode while awaiting an acknowledgement, which can save power while the controlled thread is idle. That is, enabling controls on the rendering thread can save power and reduce workload on the rendering thread.

[0091] In some examples, upon reaching block 576, controls can already be enabled on the worker thread. In some of these examples, controls may be re-enabled on the rendering thread. In other of these examples, enabling the controls on the rendering thread can be omitted as redundant.

[0092] After carrying out the procedures of block 576, the computing device can proceed to block 530.

[0093] At block 580, the computing device can determine whether controls are enabled on the W worker threads. Controls on a worker thread are discussed above in more detail in the context of block 576. If controls on worker thread(s) are enabled, the computing device can proceed to block 582; otherwise, controls on worker thread(s) are not enabled and the computing device can proceed to block 584.

[0094] At block 582, the computing device can remove / disable controls on the W worker threads. After disabling controls, the W worker threads (presumably W == 1, if controls were enabled) are no longer instructed to wait for an acknowledgement from the rendering thread before sending screen update messages to the rendering thread. In some examples, the rendering thread only sends acknowledgments to the worker thread(s) when controls on the worker thread(s) are enabled to save inter-thread communication costs, that is, the rendering thread does not send acknowledgments to the worker thread(s) when controls are disabled. After carrying out the procedures of block 582, the computing device can proceed to block 530.

[0095] At block 584, the computing device can determine a number of threads NW to start, where NW is greater than 0. The number NW can be determined based on: a pre-determined value (e.g\, I), a pre-determined percentage of W (e.g:, 10%, 20%, 50%), a combination of values (e.g., a minimum or maximum of a pre-determined value of 1 and a value of a predetermined percentage of W), or another value that is less than W.

[0096] At block 586, the computing device can set W equal to W plus NW and start NW new worker threads. As such, the value of W after carrying out the procedures of block 586 is the value of worker threads still executing after rebalancing. After carrying out the procedures of block 586, the computing device can proceed to block 530.

[0097] In some embodiments, multi-threaded application 202 can be instructed to exit and/or restart, such as discussed above in the context of blocks 216 and 218 of method 292. In those embodiments, when multi-threaded application 202 is instructed to exit and/or restart, method 500 can be completed as well.

[0098] Method 500 assumes that only one rendering thread can be executed at one time, e.g, due to limitations on a computing environment for multi-threaded application 202. In other embodiments where multiple rendering threads can be executed, a number R, R > 0, of rendering threads can be determined using techniques similar to those discussed above at block 510 for determining the number W of worker threads. In these embodiments, in some cases when rendering time RT is less than worker time WT, the number R of rendering threads can be reduced and some but not all rendering threads can be ceased (i.e., killed) using techniques similar to those discussed above at blocks 572 and 574 for reducing the number W of worker threads and ceasing worker threads. Also in these embodiments, in some cases when layout time LT is less than rendering time RT, the number R of rendering threads can be increased and more rendering threads started using techniques similar to those discussed above at blocks 584 and 586 for increasing the number W of worker threads and staring more worker threads.

[0099] In other embodiments, rendering time RT can be compared to one or more thresholds. Based on these comparisons, the number of worker threads can be increased, the number of worker threads can be reduced, and/or a worker thread can be controlled. For example, suppose two threshold amounts of rendering time THR_MIN_RENDER and THR_MAX_RENDER were defined, with THR_MIN_RENDER < THR_MAX_RENDER. Then, if the amount of rendering time RT was less than THR MIN RENDER, the rendering thread can take less than a minimum threshold amount of time to render frames, and so be relatively underworked; thus, more worker threads can be started, such as discussed above at blocks 584 and 586. If the amount of rendering time RT was less than TEORMAXRENDER, the rendering thread can take more than a maximum threshold amount of time to render frames, and so be relatively overworked: thus, execution of worker can cease (i.e., the worker threads can be killed) and/or controlled, such as discussed above at blocks 570, 572, 574, and 576. In other examples, thresholds on layout time LT can be used as well or instead of thresholds on rendering time RT. Other techniques for starting, ceasing / killing and controlling threads of multi-threaded application 202 are possible as well.

[0100] If the rendering thread is relatively underworked, additional worker threads can be added, as discussed above in the context of blocks 584 and 586 of FIG. 5 to further utilize the rendering thread. In some examples, adding additional worker threads can increase the number of frames rendered per second.

[0101] FIG. 6 depicts execution profiles for multi-threaded application 202, in accordance with an example embodiment. In an upper portion of FIG. 6, one worker thread application execution profile 610 shows render thread execution profile 620 and worker thread execution profile 630, where render thread execution profile 620 shows that a relatively small amount of rendering time is used in comparison to layout time, shown in worker thread execution profile 630. Thus, these workloads can be balanced by adding more worker threads to provide additional work to the rendering thread.

[0102] In a lower portion of FIG. 6, two worker thread application execution profile 640 shows render thread execution profile 650 and worker thread execution profiles 660 and 670. In the example illustrated by two worker thread application execution profile 640, another worker thread has been added to multi-threaded application 202 after the time represented by one worker thread application execution profile 610, so that two worker threads and one rendering thread are now being utilized by multi-threaded application 202.

[0103] Two worker thread application execution profile 640 show's that both worker threads, depicted using worker thread execution profiles 660 and 670, are contributing work to the rendering thread, and that the rendering thread at the time of render thread execution profile 650 is now has more work in comparison to the rendering thread at the time of render thread execution profile 620. Further, render thread execution profile 650 shows that approximately double the number of frames are rendered in comparison to render thread execution profile 620 while the time frames for both render thread execution profiles 620 and 650 are the same, indicating that the frame rate for multi-threaded application 202 has approximately doubled since the second worker thread was started. Thus load balancing techniques, such as those described in the context of FIG. 5, can be used to increase a number of frames per second when computing resources, such as time available to a rendering thread, allow more work to be performance by a multi-threaded application, such as application 202.

[0104] If the rendering thread is relatively overworked, worker threads can be removed and/or controlled, as discussed above in the context of blocks 572, 574, and 576 to reduce utilization of the rendering thread. In particular, a controlled worker thread can wait for an acknowledgement from the rendering thread before sending screen update messages to the rendering thread.

[0105] FIG. 7 depicts execution profiles for multi-threaded application 202, in accordance with an example embodiment. In an upper portion of FIG. 7, uncontrolled worker thread application execution profile 710 shows render thread execution profile 720 and uncontrolled worker thread execution profile 730. Render thread execution profile 720 shows that a relatively large amount of rendering time is used in comparison to layout time, shown in uncontrolled worker thread execution profile 730. Uncontrolled worker thread application execution profile 710 also shows that, in some examples, the rendering thread can receive multiple screen update messages 732 during rendering time for just one frame. In this case, the rendering thread cannot keep up with the worker thread when multiple screen update messages are provided for any duration of time, and thus screen update messages may be ignored, executed out of order, cause burdensome overhead to the rendering frame, and/or be executed so slowly as to lead to display of out-of-date information.

[0106] Thus, as mentioned above, these workloads can be balanced by ceasing execution of (/.e., killing) worker threads and/or controlling the worker thread(s). In a lower portion of FIG. 6, controlled application execution profile 740 shows render thread execution profile 750 and worker thread execution profiles 760. In the example illustrated by controlled application execution profile 740, the worker thread has been controlled after the time represented by uncontrolled thread application execution profile 710.

[0107] Controlled application execution profile 740 shows that the worker thread, depicted using controlled worker thread execution profile 760, waits for acknowledgements, such as acknowledgment 752, from the rendering thread before sending screen update messages. As such, while the relative workloads between the rendering thread and the worker thread have not changed after the worker thread has been controlled, the two threads are coordinated. In some examples, the controlled worker thread can, after sending a screen update message, switch to idle mode to save power until receiving an acknowledgement from the rendering thread, such as acknowledgment 752.

[0108] Thus, screen update messages are only sent from the worker thread when the rendering thread can render frames. And, the screen update messages sent to the rendering thread can be generated based on current information from the application logic. Thus load balancing techniques, such as those described in the context of FIG. 5, can be used to reduce the likelihood that screen update messages are ignored, executed out of order, and/or lead to display of out-ofdate information by a multi-threaded application, such as application 202.

111. EXAMPLE ARCHITECTURE [0109] FIG. 8 is a block diagram of computing device 800, in accordance with an example embodiment. In particular, computing device 800 can be configured to perform at least one function related: to one or more of: computing device 100, 200, single-threaded application 102, methods 112, 290, 292, 500, 900, multi-threaded application 202, worker thread 210, rendering thread 212, single-threaded application execution profile 410, multi-threaded application execution profile 420, one worker thread application execution profile 610, two worker thread application execution profile 640, uncontrolled worker thread application execution profile 710, and controlled worker thread application execution profile 740.

[0110] Computing device 800 can include one or more processors 802, user interface module 804, data storage 806, and network-communication interface module 820, all of which can be linked together via a system bus, network, or other connection mechanism.

[0111] Processors 802 can include one or more general purpose processors and/or one or more special purpose processors. Processors 802 can be configured to execute computer-readable program instructions 810 that are contained in data storage 806, software applications, and/or other instructions as described herein. For example, processors 802 can comprise one or more general purpose processors (e.g., INTEL single core microprocessors or INTEL multicore microprocessors) and/or one or more special puipose processors (e.g., application-specific integrated circuits (ASICs), graphics processing units (GPUs), field-programmable gate array (FPGAs), and/or digital signal processors (DSPs)).

[0112] User interface module 804 can be operable to send data to and/or receive data from external user input/output devices. For example, user interface module 804 can be configured to send and/or receive data to and/or from user input devices such as a keyboard, a keypad, a touch screen, a computer mouse, a track ball, a joystick, a camera, a voice recognition module, and/or other similar devices. User interface module 804 can also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRT), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and/or other similar devices. User interface module 804 can also include one or more devices configured to generate audible output(s), such as a speaker, speaker jack, audio output port, audio output device, earphones, and/or other similar devices. Further, in some embodiments, user interface module 804 can also include one or more dewices configured to generate haptic output(s) such as forces, vibrations, and/or motions detectable at least by a user’s sense of touch.

[0113] Data storage 806 can include one or more computer-readable storage media that can be read and/or accessed by at least one of processors 802. The one or more computer-readable storage media can include volatile and/or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with at least one of processors 802. In some embodiments, data storage 806 can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc

22.

storage unit), while in other embodiments, data storage 806 can be implemented using two or more physical devices.

[0114] Data storage 806 can include computer-readable program instructions 810 and perhaps additional data. In some embodiments, data storage 806 can additionally include storage required to perform at least part of the herein-described methods and techniques and/or at least part of the functionality of the herein-described devices.

[0115] Network-communications interface module 820 can include one or more wireless interfaces 822 and/or one or more wireline interfaces 824 that are configurable to communicate via a network. Wireless interfaces 822 can include one or more wireless transmitters, receivers, and/or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, and/or other similar type of wireless transceiver configurable to communicate via a wireless network. Wireline interfaces 824 can include one or more wireline transmitters, receivers, and/or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or similar transceiver configurable to communicate via a twisted pair wire, a coaxial cable, a fiber-optic link, or a similar physical connection to a wireline network.

[0116] In some embodiments, network communications interface module 820 can be configured to provide reliable, secured, and/or authenticated communications. For each communication described herein, information for ensuring reliable communications (/.e., guaranteed message delivery) can be provided, perhaps as part of a message header and/or footer (e.g., packet/message sequencing information, encapsulation header(s) and/or footer(s), size/time information, and transmission verification information such as CRC and/or parity check values). Communications can be made secure (e.g., be encoded or encrypted) and/or decrypted/decoded using one or more cryptographic protocols and/or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), an Rivest-Shamir-Adelman (RSA) algorithm, a Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport. Layer Security (TLS), and/or Digital Signature Algorithm (DSA). Other cryptographic protocols and/or algorithms can be used as well or in addition to those listed herein to secure (and then decrypt/decode) communications.

[0117] In some embodiments, computing device 800 can include one or more sensors. The sensor(s) can be configured to measure conditions in an environment for computing device 800 and provide data about that environment. For example, the sensor/s) can include one or more of (i) an identification sensor to identify other objects and/or devices, such as, but not limited to, an RFID reader, proximity sensor, one-dimensional barcode reader, two-dimensional barcode (e.g,

Quick Response (QR) code) reader, and a laser tracker, where the identification sensor(s) can be configured to read identifiers, such as REID tags, barcodes, QR codes, and/or other devices and/or object configured to be read and provide at least identifying information; (ii) a location sensor to measure locations and/or movements of the computing device 800, such as, but not limited to, a gyroscope, an accelerometer, a Doppler sensor, a Global Positioning System (GPS) device, a sonar sensor, a radar device, a laser-displacement sensor, and a compass; (iii) an environmental sensor to obtain data indicative of an environment of computing device 800, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a camera, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a movement sensor, a microphone, a sound sensor, an ultrasound sensor, and/or a smoke sensor. Many other examples of sensor(s) are possible as well.

[0118] Each computer-readable storage medium (or, more simply “readable medium”) described in this disclosure can include a non-transitory computer-readable medium that includes volatile and/or non-volatile storage components such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with a processor. Additionally or alternatively, each computer-readable medium described in this disclosure can include a transitory computer-readable medium. The transitory computer-readable medium can include, but is not limited to, a communications medium such as a digital or analogue communications medium (e.g., a fibre optic cable, a waveguide, a wired communication link, or a wireless communication line).

[0119] A network interface, such as network interface communication interface module 820 or any other network interface disclosed herein, can include an interface to one or more networks and/or communication channels. For example, the network interface can include one or more transmitters configured for transmitting data using the one or more networks and/or communication channels, one or more receivers configured for receiving data using the one or more networks and/or communication channels, and/or one or more transceivers configured to both transmit and receive data using the one or more networks and/or communication channels. In particular, the network interface can enable communications of content, sendees and/or other information between one or more computing devices and one or more servers.

[0120] The network interface can further include one or more receivers configured to receive data transmitted over the network or communication channel from another device within or on the network or communication channel. Any of the network interfaces disclosed herein can include circuitry, for example electronic circuitry, for converting data received from the network or communication channel to data that can be provided to a processor for processing the received data. For example, the circuitry of the network interfaces can include a modulator and/or demodulator (modem). Any of the network interfaces disclosed herein can include circuitry, for example electronic circuitry, for converting data received from another device, such as a processor or a computer-readable medium, to data in a form that can be transmitted over a network or communication channel.

IV. EXAMPLE OPERATION [0121] FIG. 9 is a flow chart of functions to carry out method 900, in accordance with an example embodiment. The functions are shown within blocks 910, 920, 930, and 940. The functions of method 900 can be carried out by a computing device, such as computing device 800 described above in the context of at least FIG. 8.

[0122] Method 900 can begin at block 910, where the computing device can receive a request to execute an application that utilizes a display, such as discussed above at least in the context of FIG. 2.

[0123] At block 920, the computing device can determine a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display, such as discussed above at least in the context of FIG. 5. In some embodiments, determining the layout time and the rendering time includes one or more of: determining the layout time based on an actual amount of work performed by the number of worker threads and determining the rendering time based on an actual amount of work performed by rendering thread, such as discussed above at least in the context of FIG. 5.

[0124] In other embodiments, determining the layout time can include determining the layout time based on one or more of: a size of a screen used to provide the display, a processor of the computing device, and layout-time data indicating one or more layout times observed during a previous execution of the application on the computing device, such as discussed above at least in the context of FIG. 5. In even other embodiments, determining the rendering time can include determining the layout time based on one or more of: a size of a screen used to provide the display, a processor of the computing device, and render) ng-time data indicating one or more rendering times observed during a previous execution of the application on the computing device, such as discussed above at least in the context of FIG. 5.

[0125] At block 930, the computing device can generate a number of worker threads of the computing device to provide the layout for the frames of the application on the display based on the layout time and the rendering time, such as discussed above at least in the context of FIG. 5. In some embodiments, generating the number of worker threads can include determining the number of worker threads based on one or more of: a size of a screen used to provide the display, a processor of the computing device, layout-time data indicating one or more layout times observed during a previous execution of the application on the computing device, render] ng-time data indicating one or more rendering times observed during a previous execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during a previous execution of the application on the computing device, such as discussed above at least in the context of FIG. 5.

[0126] In other embodiments, a worker thread can be configured to send a screen update message to instruct the rendering thread to draw one or more pixels on the display, and the rendering thread can be configured to draw the one or more pixels on the display, such as discussed above at least in the context of FIG. 2.

[0127] In still other embodiments, a worker thread can be configured to maintain a plurality of object representations that represent respective frames of the display, such as discussed above at least in the context of FIGS. 2 and 3. In some of these embodiments, the worker thread can be configured to instruct the rendering thread to draw one or more pixels in the display based on comparing a first object representation and a second object representation of the plurality of object representations, such as discussed above at least in the context of FIGS. 2 and 3. In some other of these embodiments, the first object representation can represent a first frame that precedes a second frame that is represented by the second object representation; then, comparing the first object representation and the second object representation can include determining a difference between the first object representation and the second object representation, such as discussed above at least in the context of FIGS. 2 and 3. In even other of these embodiments, the worker thread can be configured to instruct the rendering thread to draw one or more pixels that represent the difference between the first object representation and the second object representation, such as discussed above at least in the context of FIG S. 2 and 3 [0128] At block 940, the computing device can display the frames of the application on the display by executing the generated number of worker threads of the computing device in parallel with a rendering thread of the computing device that renders the frames of the application, such as discussed above at least in the context of FIGS. 2 and 5.

[0129] In some embodiments, displaying the frames of the application on the display can include displaying the frames of the application on the display at a non-zero frame rate, such as discussed above at least in the context of FIG. 2. In some of these embodiments, the non-zero frame rate can be at least 30 frames per second, such as discussed above at least in the context of FIG. 2. In other embodiments, displaying the frames of the application on the di splay can include displaying the frames of the application on the display using a \veb browser executing on the computing device, such as discussed above at least in the context of FIG. 2. In even other embodiments, the rendering thread can be configured to receive one or more user inputs and to communicate data about the one or more user inputs to the worker thread, such as discussed above at least in the context of FIG. 2.

[0130] In some embodiments, method 900 can further include: determining whether the layout time is greater than the rendering time; and after determining that the layout time is greater than the rendering time: incrementing the number of worker threads by a. worker-thread count of at least one, and executing the worker-thread count of additional worker threads, such as discussed above at least in the context of FIG. 5.

[0131] In other embodiments, method 900 can further include: determining whether the layout time is greater than the rendering time; and after determining that the layout time is greater than the rendering time: determining whether a control of a worker thread to wait for an acknowledgement from the rendering thread has been enabled, and after determining that the control of the worker thread to wait for an acknowledgement from the rendering thread has been enabled, disabling the control of the worker thread to wait for an acknowledgement from the rendering thread, such as discussed above at least in the context of FIG. 5.

[0132] In still other embodiments, method 900 can further include: determining whether the rendering time is greater than the layout time; and after determining that the rendering time is greater than the layout time: determining whether the number of worker threads is greater than one, and after determining that the number of worker threads is greater than one, decrementing the number of worker threads by a worker-thread count that is at least one and less than the number of worker threads; and ceasing execution of the worker-thread count number of worker threads, such as discussed above at least in the context of FIG. 5. In particular of these other embodiments, method 900 can further include: after determining that the number of worker threads is not greater than one: enabling a control of a worker thread to wait for an acknowledgement from the rendering thread, such as discussed above at least in the context of FIG. 5.

[0133] In even other embodiments, method 900 can further include: after enabling the control of the worker thread to wait for an acknowledgement: sending a first screen update message from the worker thread to the rendering thread; receiving an acknowledgment of the first screen update message from the rendering thread at the worker thread, and after receiving the acknowledgment of the first screen update message, sending a second screen update message from the worker thread to the rendering thread, such as discussed above at least in the context of FIGS. 5 and 7. In particular of these embodiments, the worker thread can be configured to operate in a power-saving mode while waiting for the acknowledgment of the first screen update message, such as discussed above at least in the context of FIGS. 5 and 7.

[0134] In yet other embodiments, method 900 can further include: maintaining a log of historical data about the application, where the historical data includes one or more of: layout-time data indicating one or more layout times observed execution of the application on the computing device, rendering-time data indicating one or more rendering times observed during execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during execution of the application on the computing device, such as discussed above at least in the context of FIG. 5. In particular of these embodiments, determining the layout time and the rendering time can include determining the layout time and/or the rendering time based on historical data from the log of historical data, such as discussed above at least in the context of FIG. 5. In other particular of these embodiments, generating the number of worker threads can include determining the number of worker threads based on historical data from the log of historical data such as discussed above at least in the context of FIG. 5

V. ADDITIONAL EXAMPLE EMBODIMENTS [0135] Therefore, from one perspective, there have been described methods and apparatus related to displaying frames of animated imagery. A computing device can receive a request to execute an application that utilizes a display. The computing device can determine a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames on the display. The computing device can generate a number of worker threads to provide the layout for the frames of the application on the display based on the layout time and the rendering time. The computing device can display the frames of the application on the display by executing the generated number of worker threads in parallel with a rendering thread of the computing device that renders the frames of the application.

[0136] The following clauses are offered as further description of the disclosure.

[0137] Clause 1 - A method, including: receiving, at a computing device, a request to execute an application that utilizes a display, determining a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display using the computing device; generating a number of worker threads of the computing device to provide the layout for the frames of the application on the display based on the layout time and the rendering time, and displaying the frames of the application on the display by executing the generated number of worker threads of the computing device in parallel with a rendering thread of the computing device that renders the frames of the application.

[0138] Clause 2 - The method of Clause 1, where determining the layout time and the rendering time includes one or more of: determining the layout time based on an actual amount of work performed by the number of worker threads and determining the rendering time based on an actual amount of work performed by rendering thread.

[0139] Clause 3 - The method of Clause 1 or Clause 2, where the method further includes: determining whether the layout time is greater than the rendering time; and after determining that the layout time is greater than the rendering time: incrementing the number of worker threads by a worker-thread count of at least one, and executing the worker-thread count of additional worker threads.

[0140] Clause 4 - The method of any one of Clauses 1-3, where the method further includes: determining whether the layout time is greater than the rendering time; and after determining that the layout time is greater than the rendering time: determining whether a control of a worker thread to wait for an acknowledgement from the rendering thread has been enabled, and after determining that the control of the worker thread to wait for an acknowledgement from the rendering thread has been enabled, disabling the control of the worker thread to wait for an acknowledgement from the rendering thread.

[0141] Clause 5 - The method of any one of Clauses 1-4, where the method further includes: determining whether the rendering time is greater than the layout time; and after determining that the rendering time is greater than the layout time: determining whether the number of worker threads is greater than one, and after determining that the number of worker threads is greater than one, decrementing the number of worker threads by a worker-thread count that is at least one and less than the number of worker threads; and ceasing execution of the worker-thread count number of worker threads.

[0142] Clause 6 - The method of Clause 5, where the method further includes: after determining that the number of worker threads is not greater than one: enabling a control of a worker thread to wait for an acknowledgement from the rendering thread.

[0143] Clause 7 - The method of Clause 6, where the method further includes: after enabling the control of the worker thread to wait for an acknowledgement: sending a first screen update message from the worker thread to the rendering thread, receiving an acknowledgment of the first screen update message from the rendering thread at the worker thread; and after receiving the acknowledgment of the first screen update message, sending a second screen update message from the worker thread to the rendering thread.

[0144] Clause 8 - The method of Clause 6 or Clause 7, where the worker thread is configured to operate in a power-saving mode while waiting for the acknowledgment of the first screen update message.

[0145] Clause 9 - The method of any one of Clauses 1-8, where determining the layout time includes determining the layout time based on one or more of: a size of a screen used to provide the display, a processor of the computing device, and layout-time data indicating one or more layout times observed during a previous execution of the application on the computing device.

[0146] Clause 10 - The method of any one of Clauses 1-9, where determining the rendering time includes determining the layout time based on one or more of: a size of a screen used to provide the display, a processor of the computing device, and rendering-time data indicating one or more rendering times observed during a previous execution of the application on the computing device.

[0147] Clause 11 - The method of any one of Clauses 1-10, where generating the number of worker threads includes determining the number of worker threads based on one or more of: a size of a screen used to provide the display, a processor of the computing device, layout-time data indicating one or more layout times observed during a previous execution of the application on the computing device, rendering-time data indicating one or more rendering times observed during a previous execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during a previous execution of the application on the computing device.

[0148] Clause 12 - The method of any one of Clauses 1-11, further including: maintaining a log of historical data about the application, where the historical data includes one or more of: layouttime data indicating one or more layout times observed execution of the application on the computing device, rendering-time data indicating one or more rendering times observed during execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during execution of the application on the computing device.

[0149] Clause 13 - The method of Clause 12, where determining the layout time and the rendering time includes determining the layout time and/or the rendering time based on historical data from the log of historical data.

[0150] Clause 14 - The method of Claim 12 or Claim 13, where generating the number of worker threads includes determining the number of worker threads based on historical data from the log of historical data.

[0151] Clause 15 - The method of any one of Clauses 1-14, where a worker thread is configured to send a screen update message to instruct the rendering thread to draw one or more pixels on the display, and where the rendering thread is configured to draw the one or more pixels on the display.

[0152] Clause 16 - The method of any one of Clauses 1-15, where a worker thread is configured to maintain a plurality of object representations that represent respective frames of the display.

[0153] Clause 17 - The method of Clause 16, where the worker thread is configured to instruct the rendering thread to draw one or more pixels in the display based on comparing a first object representation and a. second object representation of the plurality of object representations.

[0154] Clause 18 - The method of Clause 17, where the first object representation represents a first frame that precedes a second frame that is represented by the second object representation, and where comparing the first object representation and the second object representation includes determining a difference between the first object representation and the second object representation.

[0155] Clause 19 - The method of Clause 18, where the worker thread is configured to instruct the rendering thread to draw one or more pixels that represent the difference between the first object representation and the second object representation.

[0156] Clause 20 - The method of any one of Clauses 1-19, where displaying the frames of the application on the display includes displaying the frames of the application on the display at a non-zero frame rate.

[0157] Clause 21 - The method of Clause 20, where the non-zero frame rate is at least 30 frames per second.

[0158] Clause 22 - The method of any one of Clauses 1-21, where displaying the frames of the application on the display includes displaying the frames of the application on the display using a web browser executing on the computing device.

[0159] Clause 23 - The method of any one of Clauses 1-22, where the rendering thread is configured to receive one or more user inputs and to communicate data about the one or more user inputs to the number of worker threads.

[0160] Clause 24 - An article of manufacture including a computer-readable medium, having stored thereon instructions that, upon execution by a processor of a computing device, cause the computing device to perform operations of any one of Clauses 1-23.

[0161] Clause 25 - The article of manufacture of Clause 24, where the computer-readable medium includes a non-transitory computer-readable medium.

[0162] Clause 26 - A computing device, including: one or more processors; memory’; and program instructions, stored in the memory', that upon execution by the one or more processors cause the computing device to perform the operations of any one of Clauses 1-23.

[0163] Clause 27 - The computing device of Clause 26, where the memory' includes a nontransitory computer-readable medium.

[0164] Clause 28 - The computing device of Clause 26 or Clause 27, where the one or more processors includes a plurality of cores, and where each core can execute at least one thread of the application.

[0165] Clause 29 - A computing device including means for performing the operations of any one of Clauses 1-23.

[0166] Clause 30 - A system, including the computing device of any one of Clauses 26-29.

VI. CONCLUSION [0167] Example embodiments have been described above. Those skilled in the art will understand that changes and modifications can be made to the described embodiments without departing from the true scope of the described embodiments as claimed.

[0168] This detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be used, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0169] With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block and/or communication can represent a processing of information and/or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as steps, blocks, transmissions, communications, requests, responses, and/or messages can be executed out of order from that shown or discussed, including in substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer steps, blocks and/or functions can be used with any of the message flow diagrams, scenarios, and flow charts

32.

discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.

[0170] A step or block that represents a processing of information can correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or 5 technique. Alternatively or additionally, a step or block that represents a processing of information can correspond to a module, a segment, or a portion of program code (including related data). The program code can include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data can be stored on any type of computer-readable medium such as a 10 storage device including a disk or hard drive or other storage media.

[0171] The computer-readable medium can include non-transitory computer-readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and/or random access memory (RAM). The computer-readable media can include non-transitory computer-readable media that stores program code and/or data for longer 15 periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, and/or compact-disc read only memory (CD-ROM), for example. The computer-readable media can be any other volatile or non-volatile storage systems. A computer-readable medium can be considered a computer-readable storage medium, for example, or a tangible storage device.

[0172] Software for use in carrying out the herein-described embodiments can also be in transitory form, for example in the form of signals transmitted over a network such as the Internet. Moreover, a step or block that represents one or more information transmissions can correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions can be between software modules 25 and/or hardware modules in different physical devices.

[0173] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting.

Claims

1. A method, comprising:

receiving, at a computing device, a request to execute an application that utilizes a display;

determining a layout time to provide a layout for frames of the application on the display and a rendering time to render the frames of the application on the display using the computing device;

generating a number of worker threads of the computing device to provide the layout for the frames of the application on the display based on the layout time and the rendering time; and displaying the frames of the application on the display by executing the generated number of worker threads of the computing device in parallel with a rendering thread of the computing device that renders the frames of the application.

2. The method of claim 1, wherein determining the layout time and the rendering time comprises one or more of: determining the layout time based on an actual amount of work performed by the number of worker threads and determining the rendering time based on an actual amount of work performed by rendering thread.

3. The method of claim I or claim 2, wherein the method further comprises:

determining whether the layout time is greater than the rendering time; and after determining that the layout time is greater than the rendering time:

incrementing the number of worker threads by a worker-thread count of at least one, and executing the worker-thread count of additional worker threads.

4. The method of any one of claims 1-3, wherein the method further comprises:

determining whether a control of a worker thread to wait for an acknowledgement from the rendering thread has been enabled, and after determining that the control of the worker thread to wait for an acknowledgement from the rendering thread has been enabled, disabling the control of the worker thread to wait for an acknowledgement from the rendering thread.

5. The method of any one of claims 1-4, wherein the method further comprises:

determining whether the rendering time is greater than the layout time; and after determining that the rendering time is greater than the layout time:

determining whether the number of worker threads is greater than one, and after determining that the number of worker threads is greater than one, decrementing the number of worker threads by a worker-thread count that is at least one and less than the number of worker threads; and ceasing execution of the worker-thread count number of worker threads.

6. The method of claim 5, wherein the method further comprises:

after determining that the number of worker threads is not greater than one:

enabling a control of a worker thread to wait for an acknowledgement from the rendering thread.

7. The method of claim 6, wherein the method further comprises:

after enabling the control of the worker thread to wait for an acknowledgement:

sending a first screen update message from the worker thread to the rendering thread;

receiving an acknowledgment of the first screen update message from the rendering thread at the worker thread; and after receiving the acknowledgment of the first screen update message, sending a second screen update message from the worker thread to the rendering thread.

8. The method of claim 6 or claim 7, wherein the worker thread is configured to operate in a power-saving mode while waiting for the acknowledgment of the first screen update message.

9. The method of any one of claims 1-8, wherein determining the layout time comprises determining the layout time based on one or more of: a. size of a screen used to provide the display, a processor of the computing device, and layout-time data indicating one or more layout times observed during a previous execution of the application on the computing device.

10. The method of any one of claims 1-9, wherein determining the rendering time comprises determining the layout time based on one or more of: a size of a screen used to provide the display, a processor of the computing device, and rendering-time data indicating one or more rendering times observed during a previous execution of the application on the computing device.

11. The method of any one of claims 1-10, wherein generating the number of worker threads comprises determining the number of worker threads based on one or more of: a size of a screen used to provide the display, a processor of the computing device, layout-time data indicating one or more layout times observed during a previous execution of the application on the computing device, rendering-time data indicating one or more rendering times observed during a previous execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during a previous execution of the application on the computing device.

12. The method of any one of claims 1-11, further comprising:

maintaining a log of historical data about the application, wherein the historical data comprises one or more of: layout-time data indicating one or more layout times observed execution of the application on the computing device, rendering-time data indicating one or more rendering times observed during execution of the application on the computing device, and worker-thread data indicating one or more numbers of worker threads used during execution of the application on the computing device.

13. The method of claim 12, wherein determining the layout time and the rendering time comprises determining the layout time and/or the rendering time based on historical data from the log of historical data.

14. The method of claim 12 or claim 13, wherein generating the number of worker threads comprises determining the number of worker threads based on historical data from the log of historical data.

15. The method of any one of claims 1-14, wherein a worker thread is configured to send a screen update message to instruct the rendering thread to draw one or more pixels, and wherein the rendering thread is configured to draw7 the one or more pixels.

16. The method of any one of claims 1-15, wherein a worker thread is configured to maintain a plurality of object representations that represent respective frames of the display.

17. The method of claim 16, wherein the worker thread is configured to instinct the rendering thread to draw one or more pixels in the display based on comparing a first object representation and a second object representation of the plurality of object representations.

18. The method of claim 17, wherein the first object representation represents a first frame that precedes a second frame that is represented by the second object representation, and wherein comparing the first object representation and the second object representation comprises determining a difference between the first object representation and the second object representation.

19. The method of claim 18, wherein the worker thread is configured to instruct the rendering thread to draw one or more pixels that represent the difference between the first object representation and the second object representation.

20. The method of any one of claims 1-19, wherein displaying the frames of the application on the display comprises displaying the frames of the application on the display at a noil-zero frame rate.

21. The method of claim 20, wherein the non-zero frame rate is at least 30 frames per second.

22. The method of any one of claims 1-21, wherein displaying the frames of the application on the display comprises displaying the frames of the application on the display using a web browser executing on the computing device.

23. The method of any one of claims 1-22, wherein the rendering thread is configured to receive one or more user inputs and to communicate data about the one or more user inputs to the number of worker threads.

24. An article of manufacture including a computer-readable medium, having stored thereon instructions that, upon execution by a processor of a computing device, cause the computing device to perform operations of any one of claims 1-23.

25. The article of manufacture of claim 24, wherein the computer-readable medium comprises a non-transitory computer-readable medium.

26. A computing device, comprising:

one or more processors;

memory; and program instructions, stored in the memory, that upon execution by the one or more processors cause the computing device to perform the operations of any one of claims 1-23.

27. The computing device of claim 26, wherein the memory comprises a non-transitory computer-readable medium.

28. The computing device of claim 26 or claim 27, wherein the one or more processors comprises a plurality of cores, and wherein each core can execute at least one thread of the application.

29. A computing device comprising means for performing the operations of any one of claims 123.

30. A system, comprising the computing device of any one of claims 26-29.

Intellectual Property Office