KR102640054B1 - A minimization method for rendering by extending a frame, and computer-readable recording medium recorded with computer program executing the same - Google Patents

Abstract

In order to minimize the number of renderings due to scrolling, the present invention simultaneously performs rendering on frames included in content played on the screen and frames included in content to be newly played when scrolling, and stores them in advance when the user scrolls. We propose a method of minimizing rendering through frame expansion, in which the rendered and saved frame is played by moving it by the scroll distance. The rendering minimization method through frame expansion simultaneously renders frames included in UI content to be played directly on the screen and frames included in UI content to be additionally played after playback of the frames, and renders them as target frames and expansion frames, respectively. It includes a step of saving and when the user applies a scroll command to the screen, replacing some or all of the target frames currently playing on the screen with the extended frames and playing them on the screen.

Description

A computer-readable recording medium on which a method for minimizing rendering through frame expansion and a computer program for executing the method on a computer are recorded }

The present invention relates to a rendering method. In particular, in order to minimize the number of renderings due to scrolling, the frames included in the content played on the screen and the frames included in the content to be newly played when scrolling are simultaneously rendered and stored. It relates to a method and program to minimize rendering through frame expansion that plays the pre-rendered and stored frame by moving it by the scroll distance when the user scrolls, and a computer-readable recording medium on which the program is recorded.

In 3D computer graphics, a graphics pipeline or rendering pipeline refers to a step-by-step method for expressing a 3D image as a 2D raster image. Here, raster is a method of expressing image information on a computer. It consists of an image in two-dimensional pixels and combines their shapes to express one image information with pixels at regular intervals. A set of consecutive pixels in one line is also called a raster.

The graphics pipeline, an important module of the Android operating system, consists of rendering, adjustment, composition, and output operations of UI (User Interface) content. Rendering is the process of converting two-dimensional or three-dimensional data described in numbers and equations into images that can be perceived by humans. If modeling is the process of creating a basic skeleton, then rendering can be said to be the process of processing the surface of the skeleton.

The application renders frames for UI content, SurfaceFlinger adjusts the size, transparency, rotation, and position of the frames, and the Hardware Composer (HWC) composites and synthesizes frames received from SurfaceFlinger. Perform output.

The user app, status bar, and navigation bar each have a window, a rectangular area composed of multiple views (buttons, text, and images) that are UI elements. The UI thread manages the user's interaction with the window and the multiple views that make up the window. The render thread is responsible for rendering frames of content to be played on the screen. Buffer Queue manages three graphic buffers to support triple buffering, which reduces screen tearing.

Figure 1 illustrates the frame rendering operation of a conventional Android graphics pipeline when a user scrolls the screen.

Referring to FIG. 1, the frame rendering operation begins when the user touches the screen 110 and performs a scroll operation to change the view constituting the UI content.

The UI thread 120 searches for a view to be redrawn in the view tree to obtain a display list, and each display list has a series of drawing commands required to render one view. The render thread 120 receives the display list from the UI thread 120 and renders the corresponding frame. The GPU 130 generates a rendered frame by executing OpenGL ES instructions corresponding to each of the display lists, stores the rendered frame in an empty graphics buffer (150, B ₀ ), and stores the rendered frame in the corresponding graphics buffer (B ₀ ) into the buffer queue (140). For convenience of explanation, the member numbers of the UI thread and render thread were combined to 120, and the number of graphic buffers 150 was limited to 3 (B ₀ , B ₁ , B ₂ ).

Screen output information, including the graphics buffer number (ex, 0) where the rendered frame is stored, is delivered to the surface flinger (170) in a buffer item message (160). At this time, when the frame rendering operation is repeatedly executed, the graphic buffer in which to store the frame is alternately selected among the three graphic buffers (B ₀ , B ₁ , B ₂ ).

The surface flinger 170 determines the graphic buffer address that stores the rendered frame in the table 180 (mImages), which manages the address (or handle) of the graphic buffer, from the graphic buffer number included in the received buffer item message 160. obtain. When the surface flinger 170 transmits the acquired graphics buffer address to the HWC (not shown), the HWC outputs the frame stored in the graphics buffer (B ₀ ) on the screen.

The Android graphics pipeline, which performs the above-described process, encounters the following problems in scrolling operations.

First, even if the scroll distance is very small, rendering must be performed to redraw all or a large portion of the UI content on the screen, not just part of it.

Second, the continuous change of UI content during the scroll movement from when the user touches the screen to when he lifts his finger causes a large number of frames to be rendered, with the frame rate reaching up to 60fps (frames per second). cause. A high number of frames rendered per second means increased CPU and GPU usage, which ultimately leads to an increase in power consumption.

Republic of Korea Patent Publication No. 10-2017-0040698 (Method and device for performing graphics pipeline, April 13, 2017)

The technical problem to be solved by the present invention is to simultaneously perform rendering and store frames included in content played on the screen and frames included in content to be newly played when scrolling, in order to minimize the number of renderings due to scrolling operations. , a rendering minimization method and program through frame expansion that plays the pre-rendered and stored frame by moving it by the scroll distance when the user scrolls, and a computer-readable recording medium on which the program is recorded.

A method of minimizing rendering through frame expansion according to an aspect of the present invention for achieving the above technical problem includes frames included in UI content to be played immediately on the screen and UI content to be additionally played after playback of the frame. Simultaneously rendering the frames and storing them as target frames and extended frames, respectively; And when the user applies a scroll command to the screen, replacing some or all of the target frames currently playing on the screen with the extended frames and playing them on the screen.

Additionally, the step of playing on the screen can be performed by arranging the extended frame as much as the scroll distance authorized by the user.

Additionally, the extended frame may include a frame to be played on the screen by replacing the target frame when the scroll command is up scroll.

Additionally, when the scroll command is a down scroll, the frame to be played according to the down scroll command can be rendered and then played.

Additionally, when rendering a frame to be played according to the down scroll command, pre-rendering may not be performed on a frame corresponding to an additional down scroll command.

A method of minimizing rendering through frame expansion according to another aspect of the present invention for achieving the above technical problem includes frames included in UI content to be played immediately on the screen and UI content to be played when the user scrolls the screen. Rendering the frames together and storing them as target frames and extended frames, respectively, in a graphics buffer; determining whether a scroll command is being applied to the screen; determining whether the scroll operation authorized by the user is an up-scroll operation; accumulating an up-scroll distance approved by the user; determining whether the accumulated up-scroll distance is in an overflow state; When the accumulated up-scroll distance is not in an overflow state, moving and playing the extended frame stored in the graphic buffer by the up-scroll distance approved by the user; resetting the accumulated scroll distance to 0 (zero) when it is determined that the accumulated up-scroll distance is overflow; When the scroll operation approved by the user is not an up scroll, determining it to be a down scroll operation, rendering and playing the target frame included in the UI content to be played when the scroll operation is moved by the down scroll distance; and playing the target frame when the user does not authorize a scroll operation.

In addition, after performing the steps of moving and playing the extended frame by the up-scroll distance approved by the user, resetting the accumulated scroll distance to 0, and rendering and playing the target frame, the target The step of storing a frame and the extended frame may be performed, and after the step of playing the target frame, the step of determining whether a scroll command is applied to the screen may be performed.

Additionally, when the accumulated up-scroll distance is in an overflow state, the accumulated up-scroll distance may be longer than the distance of the extended frame.

Additionally, the extended frame may include a frame to be played on the screen by replacing the target frame when the scroll command is up scroll.

Additionally, in the step of playing after rendering the target frame, pre-rendering may not be performed on a frame corresponding to an additional down-scroll command expected when rendering the target frame.

Meanwhile, a program according to another aspect of the present invention for achieving the above technical problem may be a program for executing the above-described rendering minimization method through frame expansion.

In addition, a recording medium according to another aspect of the present invention for achieving the above technical problem may be a computer-readable recording medium on which the above-described program is recorded.

As described above, the method and program for minimizing rendering through frame expansion according to the present invention and the computer-readable recording medium on which the program is recorded can extend battery life by optimizing the Android operating system without adding hardware, Since users can reduce the purchase of auxiliary batteries, they have the advantage of significantly reducing power consumption in homes and industries, thereby reducing power production and environmental costs.

Figure 1 illustrates the frame rendering operation of a conventional Android graphics pipeline when a user scrolls the screen.
Figure 2 is an example of a scroll recognition rendering method according to the present invention.
Figure 3 explains the process of performing the rendering minimization method through frame expansion according to the present invention in any device.
Figure 4 shows the process of creating, reusing, and displaying an extended frame according to a scroll operation.

In order to fully understand the present invention, its operational advantages, and the objectives achieved by practicing the present invention, reference should be made to the accompanying drawings illustrating exemplary embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings. The same reference numerals in each drawing indicate the same member.

Figure 2 is an example of a scroll recognition rendering method according to the present invention.

Referring to FIG. 2, the scroll recognition rendering method 200 according to the present invention is performed according to a pre-written program, and includes an extended frame rendering step 210, a step of determining whether the user scrolls (220), and a scroll operation. Step 230 of determining whether the up-scroll is approved, step 240 of calculating the accumulated scroll distance, step 250 of determining whether the up-scroll is in an overflow state, step 260 of playing the extended frame, step 260 of accumulating It includes a step of resetting the scroll distance (270), a step of rendering and playing the target frame (280), and a step of playing the target frame (290). Here, pre-written programs include programs applied to Google's Android operating system or Apple's iOS operating system applied to smartphones.

The extended frame rendering step 210 renders frames included in UI content to be played immediately on the screen and frames included in UI content to be played when the user scrolls the screen together and stores them in a graphics buffer. For convenience of explanation, it is assumed that the frame included in the UI content to be played immediately is called a target frame, and the frame included in the UI content to be played when the user scrolls the screen is called an extended frame. Therefore, it is desirable that the target frame and the extended frame are stored in the graphics buffer in series.

Here, the expansion frame or frame expansion refers to the size of the frame containing the content to be played immediately on the screen, which is the basic frame size, but in the present invention, the frame and graphic buffer are used to include not only the content to be played but also content to be viewed in the near future by scrolling. It includes the meaning of increasing the size of . In other words, while the existing object of rendering was the “content of the target frame,” in the present invention, the content of the frame that is the object of rendering was expanded and the “content of the extended frame” was added to the “content of the target frame.” Target frame It is not a distinction between a frame and an extended frame, but one frame to be rendered at a time. For convenience of explanation, it belongs to one frame but is classified as a frame to be played currently or a frame to be played in the near future. The following explanation is: It was written based on these assumptions.

In the step 220 of determining whether the user scrolls, it is determined whether the scroll recognition rendering method 200 according to the present invention is applying a scroll command to the screen of the activated device.

In the step 230 of determining whether the scroll operation is an up-scroll operation, it is determined whether the scroll operation 220 (yes) authorized by the user is an up-scroll operation.

In step 240 of calculating the accumulated scroll distance, the scroll distance is accumulated when the user approves the up-scroll operation (230, Yes).

In step 250 of determining whether the up-scroll is in an overflow state, it is determined whether the accumulated up-scroll distance is in an overflow state. Scroll overflow will be described later.

In the step 260 of playing an extended frame, when the accumulated up-scroll distance does not overflow (250, No), the extended frame stored in the graphic buffer is moved by the up-scroll distance approved by the user and played. As will be described later, some or all of the currently playing target frames will be replaced with extended frames depending on the scroll distance authorized by the user.

In step 270 of resetting the accumulated scroll distance, if it is determined that the accumulated up-scroll distance is overflow (250, Yes), the accumulated scroll distance is reset to 0 (zero).

In the step 280 of rendering and playing the target frame, when the scroll operation approved by the user is not an up scroll (230, No), the target frame included in the UI content to be played when moved by the scroll distance is rendered and then played. do.

In the step 290 of playing the target frame, the target frame is played when the user does not authorize a scroll operation (220, No).

After performing the step 260 of playing the extended frame, the step 270 of resetting the accumulated scroll distance, and the step 280 of playing the target frame after rendering, the extended frame rendering step 210 is performed again, and the target frame rendering step 210 is performed again. After performing the step 290 of playing the frame, the step 220 of determining whether the user scrolls.

As described above, one of the core ideas of the present invention is the frame (target frame) included in the UI content to be played directly on the screen in the extended frame rendering step 210 and the UI content to be played when the user scrolls the screen. The frames included (extended frames) are rendered together.

Another key idea of the present invention is that when the user performs a scroll operation and a new frame (extended frame) needs to be played, in the step 260 of playing the extended frame, the extended frame that has been pre-rendered and stored is equal to the scroll distance. Since it moves and plays (step 260), there is no need to add a rendering task within the scope of the scroll overflow of the scroll operation authorized by the user, thereby optimizing the number of rendering operations.

Figure 3 explains the process of performing the rendering minimization method through frame expansion according to the present invention in any device.

FIG. 3A illustrates the rendering and image playback process when the user does not perform a scroll operation, and FIG. 3B illustrates the rendering and image playback process when the user performs a scroll operation.

Referring to FIG. 3A, when rendering UI content to be displayed on the screen 110 using an existing frame, that is, a target frame (1, 2, 3) rendering operation, an extended frame including content that is likely to be viewed in the near future through a scroll operation is also used. (4, 5, 6) are rendered and stored sequentially in the graphics buffer (150, B _o ). The area (4, 5, 6) of UI content that will be visible in the near future in the expansion frame is defined as the pre-rendering area. Surface Flinger 170 outputs only the UI content areas (1 to 3) visible on the screen, excluding the pre-rendering area.

Referring to Figure 3b, in the user's scrolling motion, a previously rendered extended frame is reused without rendering a new frame and output in the following operation. The previous operation of rendering a frame is the same as the existing graphics pipeline, with the difference being that the graphics buffer (B ₁ ) is obtained from the buffer queue 140. However, since previously rendered extended frames are reused, the operations required for frame rendering, that is, executing OpenGL functions, are not performed (Skip frame rendering). Even though the graphic buffer (B ₁ ) does not store the frame, it is put into the buffer queue 140 to request screen output.

Like the existing graphics pipeline, screen output information, including the number (B ₁ ) of the graphics buffer returned to the buffer queue 140, is sent to the surface flinger 170 in the buffer item message 160. At this time, when inserting the graphic buffer number (1) into the buffer item message 160 so that the surface flinger 170 can output the extended frame, the graphic buffer number (0) that stores the extended frame and the scroll distance d are added. do.

After receiving the buffer item message 160, the surface flinger 170 checks the number (1) of the graphic buffer storing the extended frame, positions the extended frame stored in the graphic buffer (B ₁ ) as much as d, and outputs it on the screen. do.

In the end, it can be seen that reusing extended frames when scrolling reduces the number of frame rendering operations, thereby reducing GPU and CPU load, thereby reducing power consumption.

Figure 4 shows the process of creating, reusing, and displaying an extended frame according to a scroll operation.

On the left side of Figure 4(a), the UI contents (1, 2, and 3) of the target frame are played on the screen, and the extended frames (4, 5, and 6) are stored. When the user scrolls by the scroll distance (d ₁ ), the expansion frame (4) moves upward and the target frame (1) is excluded from the screen. The scrolling operation in this direction is called up scrolling. The total scroll distance is d ₁ , which is the starting position indicating where the UI content to be displayed at the top of the screen after scrolling is located in the expanded frame. This starting position plus the screen height H is the ending position indicating where the UI content to be displayed at the bottom of the screen is located in the expanded frame. Finally, the UI content within these start and end positions is output to the screen (see step 260).

As shown on the left of FIG. 4(b), if the user scrolls an additional distance (d ₂ ), the total scroll distance is d ₁ + d ₂ , and from the position d ₁ + d ₂ in the extended frame, the screen height H is Output the area (see step 260).

As shown on the left side of FIG. 4(c), if the user scrolls an additional distance (d ₃ ), the total scroll distance becomes d ₁ + d ₂ + d ₃ . In this case, as shown in the middle of Figure 4(c), the total scroll distance plus the screen height H exceeded the end position of the extended frame. This scroll attempted to output unrendered UI content (ex, 7) outside the extended frame to the screen, and this phenomenon is defined as up-scroll overflow (or scroll overflow). When an up-scroll overflow occurs, an expansion frame containing UI content (ex, 6 to 10) equal to the height of the expansion frame, starting with the UI content (ex, 5) that will appear newly, as shown on the right side of FIG. 4(c). Render . When a new extended frame is created, the total scroll distance is initialized (see step 270).

Figure 4(d) shows the output process when scrolling in the opposite direction to Figure 4(c). The scrolling operation in this direction is defined as downscrolling. Since the UI content (ex, 4) that will newly appear at the top of the screen when scrolling is not included in the frame, a new extended frame with the corresponding UI content must be created. This phenomenon is defined as downscroll overflow. As shown in the middle of FIG. 4(d), it can be expected that the UI content (ex, 1 to 3) in the upper area will be pre-rendered from the UI content that will be newly visible due to the scroll direction. However, depending on the embodiment, for example, due to the policy of the Android graphics pipeline, the UI content (ex, 4 to 6) to be newly shown as shown on the right side of FIG. 4(d) is not pre-rendered to the upper area. First, pre-render the bottom UI content (ex, 7 to 9).

Ultimately, even though pre-rendered UI content of extended frames cannot be reused when scrolling down, rendering non-reusable extended frames unnecessarily increases CPU and GPU load. To prevent this problem, this patent proposes to render only the UI content visible on the screen, excluding the pre-rendering area in the extended frame, as shown in FIG. 4(e) when down-scrolling occurs (see step 280).

An experiment on the effectiveness of this patent was conducted by modifying the source code of the Android version of Pie. As a result of measuring performance on a Google Pixel 3XL smartphone equipped with this, the average number of frames per second rendered was lowered to less than 1fps through the reuse of extended frames in continuous up-scrolling operations, and the CPU and GPU loads were reduced by up to 62% each compared to before. and reduced by 100%. These improvements improved power consumption by up to 30%.

In the above explanation, it is stated that pre-rendering is performed when the user scrolls up, but pre-rendering is not performed when the user scrolls down. However, expanding on the above, the frame that is the target of pre-rendering is An embodiment that allows application not only when up-scrolling but also when down-scrolling may also be possible.

Since the plurality of functional blocks that perform the rendering minimization method 200 through frame expansion according to the present invention use the conventional functional blocks 110 to 180 shown in FIG. 1, they have not been added to the above description. However, as described above, each functional block 110 to 180 has a different function when performing the rendering minimization method 200 through frame expansion according to the present invention, and the difference can be easily inferred from the above description. Because it can be done, it is not explained in detail here.

Smartphones are equipped with high-performance hardware, but the increase in battery capacity is not significant. Since the present invention optimizes the Android operating system without adding hardware, battery life can be extended and users can reduce the purchase of auxiliary batteries. Therefore, by significantly reducing the power consumption of households and industries, costs for power production and the environment can be reduced.

This patent can be applied to Google's Android operating system and Apple's iOS operating system, and applicable products are as follows.

- Samsung Electronics Android smartphones: Galaxy S21, Galaxy Note, Galaxy Z Fold 2

- Google Android smartphones: Pixel 4 and 4XL, Pixel 5 and 5XL

- Apple's iPhone series smartphones: iPhone 12, iPhone 13

Additionally, products and services where this technology may be partially applied will include smartphone web browsers: Chrome, Opera Mini, and Microsoft Edge. In smartphones, the graphics pipeline and the web browser's rendering pipeline are run separately. In the rendering pipeline of a web browser, continuous rendering may occur during scrolling, which may increase power consumption. This patent can be applied to rendering pipelines such as Chrome, Opera Mini, and Microsoft Edge.

The Android graphics pipeline is an important module included in the Android operating system. In the first quarter of 2021, Android accounts for 71.81% and iOS accounts for 27.43% of the world's smartphone operating system share. In Korea, Android accounts for 74.03% of smartphone operating systems and iOS accounts for 25.63%. In the end, it can be seen that the market size of Android smart phones at home and abroad is very high, so it can be seen that the industrial applicability of the present invention is sufficient.

The present invention described above can be implemented as computer-readable code on a program-recorded medium. Computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is.

In the above, the technical idea of the present invention has been described along with the accompanying drawings, but this is an exemplary description of a preferred embodiment of the present invention and does not limit the present invention. In addition, it is clear that anyone skilled in the art can make various modifications and imitations without departing from the scope of the technical idea of the present invention.

110: screen
120: UI thread and render thread
130: GPU
140: buffer queue
150: Graphics buffer
160: buffer item or buffer item messages
170: SurfaceFlinger
180: Table that manages the address of the graphics buffer

Claims (14)

Simultaneously rendering a frame included in UI content to be played immediately on the screen and a frame included in UI content to be additionally played after playing the frame and storing them as a target frame and an extended frame, respectively; and
When a user applies a scroll command to the screen, replacing some or all of the target frames currently playing on the screen with the extended frames and playing them on the screen,
The steps to play on the screen are,
Playing the extended frame by placing it according to the scroll distance approved by the user,
When the scroll command is downscroll,
Rendering is performed on the frame to be played according to the down scroll command and then played,
A method of minimizing rendering through frame expansion, in which pre-rendering is not performed on frames corresponding to additional down-scroll commands when rendering frames to be played according to the down-scroll command. delete According to claim 1,
The extended frame is,
A method of minimizing rendering through frame expansion, including a frame to be played on the screen by replacing the target frame when the scroll command is an up scroll. delete delete A computer-readable recording medium on which a computer program for executing the method of minimizing rendering through frame expansion according to any one of claims 1 and 3 on a computer is recorded. Rendering frames included in UI content to be played immediately on the screen and frames included in UI content to be played when the user scrolls the screen together and storing them as target frames and extended frames, respectively, in a graphics buffer;
determining whether a scroll command is being applied to the screen;
determining whether the scroll operation authorized by the user is an up-scroll operation;
accumulating an up-scroll distance approved by the user;
determining whether the accumulated up-scroll distance is in an overflow state;
When the accumulated up-scroll distance is not in an overflow state, moving and playing the extended frame stored in the graphic buffer by the up-scroll distance approved by the user;
resetting the accumulated scroll distance to 0 (zero) when it is determined that the accumulated up-scroll distance is overflow;
When the scroll operation approved by the user is not an up scroll, determining it to be a down scroll operation, rendering and playing the target frame included in the UI content to be played when the scroll operation is moved by the down scroll distance; and
A method of minimizing rendering through frame expansion, comprising the step of playing the target frame when the user does not authorize a scroll operation. According to claim 7,
After performing the steps of moving and playing the extended frame by the up-scroll distance approved by the user, resetting the accumulated scroll distance to 0, and rendering and playing the target frame, the target frame and Perform the step of saving as the extended frame,
A method of minimizing rendering through frame expansion, wherein after performing the step of playing the target frame, a step of determining whether a scroll command is applied to the screen is performed. According to claim 8,
When the accumulated up-scroll distance is in an overflow state,
A method of minimizing rendering through frame expansion, when the accumulated up-scroll distance is longer than the distance of the expanded frame. According to claim 7,
The extended frame is,
A method of minimizing rendering through frame expansion, including a frame to be played on the screen by replacing the target frame when the scroll command is an up scroll. According to claim 7,
In the step of rendering and playing the target frame,
A method of minimizing rendering through frame expansion, in which pre-rendering is not performed on frames corresponding to additional down-scroll commands expected when rendering on the target frame. A computer-readable recording medium on which a computer program for executing the method of minimizing rendering through frame expansion according to any one of claims 7 to 11 on a computer is recorded. delete delete