WO2024066926A1 - 显示方法及装置 - Google Patents

显示方法及装置 Download PDF

Info

Publication number
WO2024066926A1
WO2024066926A1 PCT/CN2023/116602 CN2023116602W WO2024066926A1 WO 2024066926 A1 WO2024066926 A1 WO 2024066926A1 CN 2023116602 W CN2023116602 W CN 2023116602W WO 2024066926 A1 WO2024066926 A1 WO 2024066926A1
Authority
WO
WIPO (PCT)
Prior art keywords
refresh rate
vsync
signal
electronic device
image frame
Prior art date
Application number
PCT/CN2023/116602
Other languages
English (en)
French (fr)
Inventor
徐涛
蔡立峰
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Publication of WO2024066926A1 publication Critical patent/WO2024066926A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/20—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/39—Control of the bit-mapped memory
    • G09G5/393—Arrangements for updating the contents of the bit-mapped memory

Definitions

  • the present application relates to the field of image processing technology, and in particular to a display method and device.
  • Electronic devices can install multiple applications (Application, APP), and when running the APP, the display of image frames is controlled by the refresh rate corresponding to the APP. Some of the multiple APPs have different corresponding refresh rates. When the electronic device switches APPs, the refresh rate also switches synchronously. However, when the electronic device switches from a high refresh rate to a low refresh rate, frame loss occurs, causing the electronic device to have a noticeable sense of lag, reducing the user experience.
  • Application Application
  • APP Application
  • the display of image frames is controlled by the refresh rate corresponding to the APP.
  • Some of the multiple APPs have different corresponding refresh rates.
  • the refresh rate also switches synchronously. However, when the electronic device switches from a high refresh rate to a low refresh rate, frame loss occurs, causing the electronic device to have a noticeable sense of lag, reducing the user experience.
  • the present application provides a display method and device, the purpose of which is to solve the frame loss problem caused by switching from a high refresh rate to a low refresh rate, so as to reduce the lag of electronic devices and improve user experience.
  • the present application provides the following technical solutions:
  • the present application provides a display method for application in an electronic device, the method comprising: determining that the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate, the first refresh rate being greater than the second refresh rate; in response to the refresh rate switching from the first refresh rate to the second refresh rate, generating a second image frame after the electronic device completes display of the first image frame; and displaying the second image frame.
  • an electronic device Normally, an electronic device generates an image frame if two conditions are met: one condition is that the level change of the first signal (i.e., Vsync-sf) meets the preset condition, which is that the level of the first signal changes from a low level to a high level (corresponding to a rising edge), or the level of the first signal changes from a high level to a low level (corresponding to a falling edge); the other condition is that the electronic device completes the display of the previous image frame.
  • the time consumption of the second signal i.e., Vsync-hw
  • the time-consuming second signal can cover at least one first signal.
  • the electronic device cannot determine whether the display of the image frame is completed, so that the electronic device cannot process other image frames, resulting in the loss of image frames.
  • the electronic device can determine that the display of an image frame is completed, but the electronic device still has to wait for the rising edge or falling edge of the first signal, which may also cause the electronic device to lose image frames during the waiting process.
  • the display method provided in the present application can immediately generate and display the second image frame after determining that the refresh rate switches from the first refresh rate to the second refresh rate and after the electronic device completes the display of the first image frame.
  • the electronic device determines that the display of one image frame is completed, the electronic device does not need to wait for the rising edge or the falling edge of the first signal and can immediately generate and display another image frame, thereby reducing the loss of image frames, reducing the lag of the electronic device, and improving the user experience.
  • the method further includes: after the refresh rate is switched to the second refresh rate, controlling the first signal to align with the second signal, the first signal being used to indicate the generation of an image frame, and the second signal being used to indicate the display of an image frame; when the electronic device completes the display of the second image frame, and the level change of the first signal satisfies a preset condition, generating a third image frame, wherein the preset condition is that the level of the first signal changes from a low level to a high level, or the level of the first signal changes from a high level to a low level; when the second signal Alignment means that the periods of the first signal and the second signal are the same, the rising edges of the first signal and the second signal are aligned, and the falling edges of the first signal and the second signal are also aligned, that is, the waveforms of the first signal and the second signal are the same, and the signal amplitudes can be the same or different.
  • the image frame can be generated once every first signal, so that the electronic device synthesizes one frame and loses one frame. Synthesizing one frame can refer to generating one image frame and further displaying the image frame.
  • the electronic device can control the alignment of the first signal and the second signal, so that the electronic device can generate one image frame under each first signal, preventing the phenomenon of synthesizing one frame and losing one frame, reducing the loss of image frames, thereby reducing the sense of freeze of the electronic device and improving the user experience.
  • the method before determining that the refresh rate of the electronic device switches from the first refresh rate to the second refresh rate, the method further includes: in response to the refresh rate being the first refresh rate, generating the (i+n)th first signal according to the i-th second signal, the duration of the interval between the (i+n)th first signal and the i-th second signal being the sum of the periods of the n first signals, and n being a natural number greater than 1; after the refresh rate is switched to the second refresh rate, controlling the alignment of the first signal with the second signal includes: after the refresh rate is switched to the second refresh rate, sampling the first second signal after the refresh rate is switched, and using the sampling result of the first second signal to generate the first signal, the duration of the interval between the first signal and the first second signal being the sum of the periods of the (n-1) second signals.
  • the electronic device After the refresh rate is switched, the reason why the first signal and the second signal are not aligned is that in the process of generating the first signal, a second signal with a period increase is used. Therefore, in the present application, after determining that the refresh rate is switched, the electronic device adjusts the generation mechanism of the first signal after the switch, and uses the first first signal after the refresh rate is switched to generate the second signal, so that the first signal and the second signal can be aligned after the refresh rate is switched.
  • the sampling result of the second signal can be stored in a result sequence, and when the first signal is generated, the sampling result of the second signal is read from the result sequence.
  • the sampling result of the second signal can be a plurality of timestamps of the second signal, and the result sequence can be a timestamp sequence storing the plurality of timestamps of the second signal.
  • the method also includes: discarding a special signal, where the special signal is a second signal generated during the process of switching the refresh rate from a first refresh rate to a second refresh rate, and the special signal is a previous signal of the first second signal, thereby reducing the amount of data processed by the electronic device.
  • the special signal is a second signal generated during the process of switching the refresh rate from a first refresh rate to a second refresh rate, and the special signal is a previous signal of the first second signal, thereby reducing the amount of data processed by the electronic device.
  • discarding the special signal includes: prohibiting sampling of the special signal; or sampling the special signal, and the sampling result of the special signal is not stored in the result sequence.
  • the second image frame is an image frame lost during the process of switching the refresh rate from the first refresh rate to the second refresh rate.
  • the electronic device can immediately generate the first image frame of the lost multiple image frames to ensure the continuity of the image; in some examples, after determining that the display of the first image frame is completed, the electronic device can immediately synthesize the last image frame of the lost multiple image frames to shorten the delay time of the last image frame, and the image frame displayed by the electronic device next time is the next image frame of the lost last image frame. Immediately generating and displaying the lost last image frame can make the image frame displayed next time and the lost last image frame continuous, thereby ensuring continuity.
  • the electronic device completing display of the first image frame includes: after the electronic device calls the kernel thread crtc_commit to release fence resources, determining that the electronic device completes display of the first image frame.
  • determining that the refresh rate of the electronic device switches from the first refresh rate to the second refresh rate includes: determining an end time of the first signal and a start time of the first signal when the refresh rate of the electronic device is the first refresh rate; determining the end time and the start time of the first signal; The difference between the start times; if the difference satisfies the preset switching condition, it is determined that the refresh rate of the electronic device is switched from the first refresh rate to the second refresh rate.
  • the preset switching condition can be used to indicate that the period of the first signal increases.
  • the preset switching condition can be a preset threshold or a preset value range. The values of the preset threshold and the preset value range are determined according to the refresh rate. For example, when switching from 90Hz to 60Hz, the preset threshold can be a value less than 19.4ms, and the preset value range can be (11.1, 19.4].
  • the application framework layer of the electronic device includes: a refresh rate processing unit and a data reading unit; the hardware abstraction layer of the electronic device includes a hardware hybrid renderer; the kernel layer of the electronic device includes a display driver; a refresh rate processing unit, used to determine that the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate; a data reading unit, used to respond to the refresh rate switching from the first refresh rate to the second refresh rate, and read the first image data after the electronic device completes display of the first image frame; a hardware hybrid renderer, used to synthesize the first image data to generate a second image frame; and a display driver, used to display the second image frame.
  • the application framework layer also includes: a signal generating unit, the signal generating unit is used to control the alignment of the first signal and the second signal after the refresh rate is switched to the second refresh rate, the first signal is used to indicate the generation of an image frame, and the second signal is used to indicate the display of an image frame; a data reading unit is also used to complete the display of the second image frame on the electronic device, and the level change of the first signal meets a preset condition, and read the second image data, wherein the preset condition is that the level of the first signal changes from a low level to a high level, or the level of the first signal changes from a high level to a low level; a hardware hybrid renderer is also used to synthesize the second image data to generate a third image frame; a display driver is also used to display the third image frame under the action of the second signal.
  • a signal generating unit is used to control the alignment of the first signal and the second signal after the refresh rate is switched to the second refresh rate, the first signal is used to indicate the generation of an
  • the present application provides an electronic device, which includes: one or more processors; one or more memories; the memories store one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the above-mentioned display method.
  • the present application provides a computer-readable storage medium, in which a computer program is stored.
  • the computer program is executed by a processor, the processor executes the above-mentioned display method.
  • FIG1 is a schematic diagram of rendering and displaying an image frame using Vsync-app, Vsync-sf, and Vsync-hw provided by the present application;
  • FIG2 is a schematic diagram of a display driver generating Vsync-hw provided by the present application.
  • FIG3 is a schematic diagram of tracking the frame loss problem provided by the present application.
  • FIG4 is an enlarged schematic diagram of the frame loss problem provided by the present application.
  • FIG5 is a schematic diagram of software/hardware Vsync after the refresh rate is switched provided by the present application
  • FIG6 is a schematic diagram of rendering an image frame to display using Vsync-app, Vsync-sf, and Vsync-hw after a refresh rate switch provided by the present application;
  • FIG7 is a schematic diagram of tracking for optimizing frame loss problem provided by the present application.
  • FIG8 is an enlarged schematic diagram of the optimization frame loss problem provided by the present application.
  • FIG9 is a schematic diagram of optimizing software/hardware Vsync after the refresh rate is switched provided by the present application.
  • FIG. 10 is a diagram of rendering an image frame using optimized Vsync-app, Vsync-sf, and Vsync-hw provided by the present application.
  • FIG11 is a hardware architecture diagram of an electronic device provided by the present application.
  • FIG12 is a software architecture diagram of the electronic device provided by the present application.
  • FIG13 is a signaling diagram of the display method provided by the present application.
  • FIG. 14 and FIG. 15 are schematic diagrams of a timestamp sequence and a predicted Vsync-sf provided by the present application.
  • one or more refers to one, two or more; “and/or” describes the association relationship of the associated objects, indicating that three relationships may exist; for example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.
  • the character “/” generally indicates that the objects associated before and after are in an "or” relationship.
  • references to "one embodiment” or “some embodiments” etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application.
  • the phrases “in one embodiment”, “in some embodiments”, “in some other embodiments”, “in some other embodiments”, etc. appearing in different places in this specification do not necessarily all refer to the same embodiment, but mean “one or more but not all embodiments", unless otherwise specifically emphasized in other ways.
  • the terms “including”, “comprising”, “having” and their variations all mean “including but not limited to”, unless otherwise specifically emphasized in other ways.
  • Vsync signals are divided into hardware Vsync signals and software Vsync signals.
  • Hardware Vsync signals are abbreviated as Vsync-hw signals, and Vsync-hw signals can be understood as tearing effect (TE) signals.
  • Software Vsync signals include Vsync-app signals and Vsync-sf signals.
  • Vsync-hw signals, Vsync-app signals, and Vsync-sf signals are abbreviated as Vsync-hw, Vsync-app, and Vsync-sf.
  • Vsync-app and Vsync-sf are generated according to Vsync-hw.
  • DispSyncSource.cpp in SF surfaceflinger
  • DispSyncSource.cpp samples Vsync-hw, and the sampling result is input into the software Vsync model.
  • DispSyncSource.cpp runs the software Vsync model to output Vsync-app and Vsync-sf through the software Vsync model.
  • Vsync-hw can be generated by the display driver according to the refresh rate.
  • FIG. 1 The process of electronic devices using Vsync-app, Vsync-sf and Vsync-hw to render image frames to display is shown in Figure 1.
  • APP Under the action of Vsync-app, APP generates image data (this process is the APP rendering process, and render in Figure 1 represents image data) and stores the image data in a buffer.
  • SF Under the action of Vsync-sf, SF reads the image data from the buffer and sends the image data to the hardware hybrid renderer (HWChwcomposer, HWC), which synthesizes the image data to generate a frame of image frames (this process is called SF synthesis). HWC sends the image frame to the display driver.
  • HWChwcomposer HWC
  • the display driver sends the image frame to the screen for display (this process is called HWC display sending).
  • HWC display sending After the screen completes the image frame display, the display driver calls the kernel thread crtc_commit, and crtc_commit releases the fence resources. Releasing the fence resources can indicate that the electronic device has completed the display of one image frame.
  • an image frame goes through three cycles from rendering to display. In the first cycle of the three cycles, APP rendering is completed, the second cycle completes SF synthesis, and the third cycle completes HWC display.
  • the periods of the three signals Vsync-app, Vsync-sf and Vsync-hw can be determined according to the refresh rate corresponding to the APP. For example, if the refresh rate corresponding to the APP is 90Hz, the periods of the three signals Vsync-app, Vsync-sf and Vsync-hw are 11.1 milliseconds (ms); if the refresh rate corresponding to the APP is 60Hz, the periods of the three signals Vsync-app, Vsync-sf and Vsync-hw are 16.6ms. When the electronic device switches the APP, the refresh rate also switches synchronously.
  • the synthesized frame can be image data synthesized into one frame of image frames by SF.
  • Dropping a frame may mean delaying SF synthesis, such as performing SF synthesis at the rising edge or falling edge of the next Vsync-sf to synthesize image data of the image frame at the next Vsync-sf, or dropping a frame may mean dropping image data of one image frame.
  • the display driver in the electronic device can generate Vsync-hw, and when the refresh rate is switched, the display driver can simulate the refresh rate switching to adjust the Vsync-hw according to the refresh rate after switching.
  • the display driver when the refresh rate is 90Hz, the display driver generates a Vsync-hw with a duration of 11.1ms, which includes a low level with a duration of 8.3ms and a high level with a duration of 2.8ms.
  • the display driver determines that the refresh rate switch occurs on the rising edge of Vsync-hw, and the refresh rate switches from 90Hz to 60Hz.
  • the display driver uses the rising edge as the start of a Vsync-hw.
  • the Vsync-hw includes a high level with a duration of 2.8ms and a low level with a duration of 16.6ms.
  • the duration of the Vsync-hw is 19.4ms. Therefore, the display driver simulates the mechanism of Vsync-hw at 90Hz, resulting in a Vsync-hw with a duration of 19.4ms when the refresh chip switches from 90Hz to 60Hz.
  • This phenomenon not only causes SF to lose 1 to 2 frames of image frames due to the lack of fence resources, but also causes the rising and falling edges of Vsync-sf and Vsync-hw to be continuously misaligned, resulting in the alternation of synthesizing one frame and dropping one frame, which brings an obvious sense of stuttering.
  • SF waits until crtc_commit releases the fence resource, SF can read the image data. If SF does not wait until crtc_commit releases the fence resource, SF cannot read the image data. After 19.4ms, SF waits for a fence resource once every Vsync-sf, which means SF can The image data is read once, resulting in a situation where one frame is synthesized and one frame is dropped alternately. As shown in 3 in Figure 3, the large box pointed to by 3 shows that one frame is synthesized and one frame is dropped alternately, and the small box in the large box indicates that one frame is dropped.
  • the refresh rate switches from 90Hz to 60Hz.
  • the electronic device first loses two frames of image frames. After the switch is completed, one frame is refreshed and one frame is lost, causing a sense of lag in the whole process, which reduces the user experience.
  • Figure 5 shows a schematic diagram of software/hardware Vsync when switching from 90Hz to 60Hz.
  • the vertical line can be used as a benchmark for software/hardware Vsync, pointing to the start time of software/hardware Vsync.
  • the cycles of the three signals Vsync-app, Vsync-sf and Vsync-hw are the same, and at least the references of Vsync-sf and Vsync-hw are the same in each cycle, that is, Vsync-sf and Vsync-hw are aligned in each cycle, and the alignment can be the same cycle and the rising edges of Vsync-sf and Vsync-hw are aligned, and the falling edges of Vsync-sf and Vsync-hw are aligned.
  • Vsync-app the cycle is the same as Vsync-sf and Vsync-hw, but the rising edge and falling edge may be different from those of Vsync-sf and Vsync-hw.
  • Vsync-app is aligned with Vsync-sf and Vsync-hw. However, when the refresh rate switches from 90Hz to 60Hz, a longer Vsync-hw appears.
  • the Vsync-hw duration is 19.4ms, that is, a longer Vsync-hw appears in the fourth cycle, and the cycle of Vsync-hw increases from 11.1ms to 19.4ms, but the cycle of Vsync-app and Vsync-sf in the fourth cycle is still 11.1ms. Therefore, the cycle of Vsync-hw in the fourth cycle is different from the cycle of Vsync-app and Vsync-sf in the fourth cycle, resulting in the benchmark of Vsync-hw in the fifth cycle being different from the benchmark of Vsync-app and Vsync-sf in the fourth cycle. Starting from the fifth cycle, Vsync-app, Vsync-sf and Vsync-hw are not aligned.
  • Vsync-app As shown in FIG5 , although the cycles of Vsync-app, Vsync-sf, and Vsync-hw are 16.6 ms when the refresh rate is switched to 60 Hz, Vsync-hw takes a long time in the fourth cycle, so that Vsync-hw, which originally ends at (1), ends at (2), resulting in the cycle of Vsync-hw in the fourth cycle being different from the cycle of Vsync-app and Vsync-sf in the fourth cycle, and the end of Vsync-hw in the fourth cycle being different from the end of Vsync-app and Vsync-sf in the fourth cycle, which leads to the fact that starting from the fifth cycle, Vsync-app, Vsync-sf, and Vsync-hw in actual situations continue to be misaligned up and down.
  • the time consumed by crtc_commit to release the fence resources increases.
  • the fence resources that should have been released at the end of the fourth cycle are released after the start of the fifth cycle, so that the SF synthesis of the fifth cycle is delayed to the sixth cycle, that is, the fence resources are missed at (1), and the SF synthesis that should have been executed at (1) is actually executed at (2), and the delay in SF synthesis causes a frame to be lost; similarly, the fence resources are missed at (3), and the SF synthesis that should have been executed at (3) is actually executed at (4), resulting in a frame loss, that is, relative to the generation of one image frame under each Vsync-app, after the refresh rate is switched, one image frame is generated every Vsync-app interval.
  • the present application provides a display method, which synthesizes image data of one frame of image frames in response to switching from a high refresh rate to a low refresh rate, when receiving an instruction from crtc_commit to release fence resources but without receiving Vsync-sf.
  • SF determines that crtc_commit has released the fence resources (i.e., the display of one frame of image frames is completed), then SF can read the image data immediately after crtc_commit releases the fence resources to immediately perform SF synthesis, and synthesizes image data of one frame of image frames in advance compared to waiting to receive Vsync-sf to read the image data, so that when the high refresh rate is switched to the low refresh rate, the image data of one frame of image frames can be quickly synthesized, reducing the number of frame drops.
  • Vsync-app, Vsync-sf, and Vsync-hw are synchronized, that is, after switching to a low refresh rate, Vsync-app, Vsync-sf, and Vsync-hw are aligned to solve the situation where one frame is synthesized and one frame is dropped alternately due to the fact that Vsync-app, Vsync-sf, and Vsync-hw are not aligned after switching.
  • FIG7 a trace diagram of the optimization of the frame loss problem after the electronic device implements the display method of the present application.
  • the refresh rate is switched from 90 Hz to 60 Hz.
  • the Vsync-hw that takes 19.4 ms covers a Vsync-sf.
  • crtc_commit completes the release of fence resources at the end position of Vsync-hw.
  • SF cannot wait for crtc_commit to release the fence resources, resulting in the loss of two frames under the Vsync-hw that takes 19.4 ms. After the Vsync-hw that takes 19.4 ms ends, SF determines that crtc_commit has released the fence resources, and immediately performs SF synthesis to immediately force the synthesis of image data of one frame of image frames. That is, although SF is not at the rising edge or falling edge of Vsync-sf, SF determines that crtc_commit releases the fence resources and immediately performs SF synthesis to forcibly synthesize image data of one frame.
  • SF synthesis must meet two conditions: one condition is to be at the rising edge or falling edge of Vsync-sf, and the other condition is to determine that crtc_commit has released the fence resources; but when switching from a high refresh rate to a low refresh rate, a time-consuming Vsync-hw occurs, and SF synthesis only needs to meet one condition, which can be to determine that crtc_commit has released the fence resources. In this way, SF does not need to wait for the next rising edge or falling edge of Vsync-sf, and the SF synthesis at the next Vsync-sf will be advanced, so that the image data of one frame can be synthesized in advance.
  • the image data synthesized in advance may be the image data of any one of the two image frames that are lost. Two frames are lost under the Vsync-hw that takes 19.4ms, but the image data of any one of the two lost frames can be synthesized immediately after the Vsync-hw that takes 19.4ms ends, which is equivalent to losing one frame, reducing the number of lost frames.
  • the image data of the first frame of the two lost frames can be synthesized immediately after the Vsync-hw that takes 19.4ms ends to ensure the continuity of the image; in some examples, the image data of the second frame of the two lost frames can be synthesized immediately after the Vsync-hw that takes 19.4ms ends to shorten the delayed synthesis time of the second frame.
  • Vsync-hw may cause the subsequent Vsync-app, Vsync-sf and Vsync-hw to be misaligned, and the misalignment of Vsync-app, Vsync-sf and Vsync-hw may cause the electronic device to synthesize one frame and lose one frame. To address this problem, after switching to a low refresh rate, the electronic device can align Vsync-app, Vsync-sf and Vsync-hw.
  • SF can read image data from the buffer at the rising and falling edges of Vsync-sf respectively, and the image data is synthesized by HWC, thereby performing SF synthesis at the rising and falling edges of Vsync-sf respectively, so that the electronic device can perform SF synthesis at the rising and falling edges of Vsync-sf at 60Hz after switching to 60Hz, avoiding the loss of image frames.
  • the Vsync-hw time in the fourth cycle in FIG9 is 19.4 ms, that is, a long Vsync-hw appears in the fourth cycle.
  • crtc_commit originally releases the fence resource before the end of (1), but because the refresh rate is switched from 90 Hz to 60 Hz, crtc_commit releases the fence resource at (2) and completes the release of the fence resource at (5) (i.e., the end of the fourth cycle), resulting in a frame loss at (3) because the fence resource is missed and SF synthesis is not performed.
  • (4) is the end of Vsync-hw in the fourth cycle.
  • the process waits until crtc_commit releases the fence resources, and then immediately performs SF synthesis at (4) to synthesize the image data of one frame, thereby forcing the refresh of one frame, that is, forcing the display of one frame in the fifth cycle.
  • the end time of the three signals Vsync-app, Vsync-sf and Vsync-hw is the same, so in the sixth cycle, the start time of the three signals Vsync-app, Vsync-sf and Vsync-hw is the same, and because the cycles of the three signals Vsync-app, Vsync-sf and Vsync-hw are the same, Vsync-app, Vsync-sf and Vsync-hw are aligned up and down in the sixth cycle, as shown in Figure 9, starting from the sixth cycle, Vsync-app, Vsync-sf and Vsync-hw are aligned up and down.
  • the sixth cycle can be based on the sixth vertical line in Figure 9 as the start time and the seventh vertical line as the end time, so that the subsequent software/hardware Vsync can be aligned, and the electronic device can stably refresh (display) the image at a refresh rate of 60Hz.
  • the electronic device can be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable Electronic devices, smart watches, etc.
  • UMPC ultra-mobile personal computer
  • PDA personal digital assistant
  • This application does not specifically limit the specific form of the electronic device.
  • the electronic device may include: a processor, an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, a sensor module, a button, a motor, an indicator, a camera, a display screen (also referred to as a screen), and a Subscriber Identity Module (SIM) card interface, etc.
  • a processor an external memory interface
  • an internal memory a Universal Serial Bus (USB) interface
  • USB Universal Serial Bus
  • the audio module may include a speaker, a receiver, a microphone, an earphone interface, etc.
  • the sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.
  • the processor may include one or more processing units, for example: the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural network processor (NPU).
  • AP application processor
  • GPU graphics processor
  • ISP image signal processor
  • DSP digital signal processor
  • NPU neural network processor
  • different processing units can be independent devices or integrated into one or more processors.
  • the processor is the nerve center and command center of the electronic device.
  • the controller can generate operation control signals according to the instruction opcode and timing signal to complete the control of fetching and executing instructions.
  • the display screen is used to display images, videos, a series of graphical user interfaces (GUI), etc.
  • GUI graphical user interfaces
  • the external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device.
  • the external memory card communicates with the processor through the external memory interface to implement the data storage function. For example, music, video and other files are saved in the external memory card.
  • the internal memory can be used to store computer executable program code, which includes instructions.
  • the processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. For example, in the present application, the processor enables the electronic device to execute the display method provided in the present application by running the instructions stored in the internal memory.
  • the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device.
  • the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently.
  • the components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
  • operating systems are running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft.
  • the operating system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture.
  • the embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device.
  • Figure 12 is a software architecture diagram of the electronic device.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor.
  • the layers communicate with each other through software interfaces.
  • the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer (Framework), the hardware abstraction layer (HAL) and the kernel layer (Kernel).
  • the application layer can include a series of application packages.
  • Application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other APPs.
  • the application framework layer provides application programming interface (API) and programming framework for the applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.
  • the application framework layer can also The HAL may include SF, which includes DispSyncSource.cpp, SF.cpp and VsyncReactor.cpp.
  • DispSyncSource.cpp, SF.cpp and VsyncReactor.cpp can be regarded as three units in SF, such as DispSyncSource.cpp can be called a signal generation unit, SF.cpp can be called a data reading unit, and VsyncReactor.cpp can be called a refresh rate processing unit.
  • HAL may include HWC.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer contains at least display driver, camera driver, audio driver and sensor driver. HWC can be used to synthesize image data, and functions such as setCallback, registerCallback and onComposerhalVsync can be called in the process of synthesizing image data.
  • VsyncReactor.cpp can sample Vsync-hw, and DispSyncSource.cpp generates Vsync-app and Vsync-sf according to the sampling results.
  • VsyncReactor.cpp can call addResyncSample to sample Vsync-hw and obtain the timestamp of Vsync-hw.
  • the timestamp of Vsync-hw is an example of the sampling result.
  • the timestamp of Vsync-hw is stored in the timestamp sequence through addHwVsyncTimestamp.
  • VsyncReactor.cpp can also call periodConfirmed for scene recognition.
  • VsyncReactor.cpp After VsyncReactor.cpp recognizes that a switch from high refresh rate to low refresh rate has occurred and a long Vsync-hw occurs, it can send instructions to SF.cpp to instruct SF.cpp to perform SF synthesis with HWC after determining that crtc_commit has released the fence resources, and adjust the timestamp sequence through VsyncReactor.cpp to ensure that Vsync-app, Vsync-sf and Vsync-hw can be aligned after the refresh rate is switched.
  • the signaling diagram is shown in FIG13, which may include the following steps:
  • VsyncReactor.cpp calls addResyncSample to sample Vsync-hw(i) to obtain the timestamp of Vsync-hw(i), and the timestamp of Vsync-hw(i) is stored in the timestamp sequence.
  • VsyncReactor.cpp calls addHwVsyncTimestamp to store the timestamp of Vsync-hw(i) in the timestamp sequence.
  • Vsync-sf(i+2) is the second Vsync-sf after Vsync-sf(i).
  • Vsync-hw(i) and Vsync-sf(i) correspond to the same cycle.
  • Vsync-hw(i) and Vsync-sf(i) are Vsync-hw and Vsync-sf in the i-th cycle.
  • SF can also generate Vsync-app(i+2) when generating Vsync-sf(i+2).
  • APP generates image data under the action of Vsync-app(i+2), which will not be described in detail here.
  • generating Vsync-sf(i+2) according to the timestamp of Vsync-hw(i) is only an example and is not limited to this embodiment.
  • SF.cpp determines that crtc_commit releases the fence resource and is at the rising edge or falling edge of Vsync-sf(i+2), and SF.cpp reads the image data.
  • the rising edge of Vsync-sf(i+2) indicates that the level of Vsync-sf(i+2) changes from a low level to a high level
  • the falling edge of Vsync-sf(i+2) indicates that the level of Vsync-sf(i+2) changes from a high level to a low level.
  • HWC synthesizes the image data to complete SF synthesis through SF.cpp and HWC to generate image frames.
  • the HWC may send the synthesized image data to a display driver, and the display driver may drive the screen to display the image frame.
  • VsyncReactor.cpp calls periodConfirmed to identify that the time taken for Vsync-hw(i+1) is 19.4ms, so as to determine that the refresh rate has been switched, and VsyncReactor.cpp sends a refresh rate switching instruction to SF.cpp, which not only indicates that the refresh rate has been switched, but also carries the time taken for Vsync-hw(i+1). Vsync-hw(i+1) is the next Vsync-hw of Vsync-hw(i).
  • a preset threshold is output to identify whether a preset scenario occurs.
  • the preset scenario is a scenario in which Vsync-hw takes a long time when the refresh rate is switched.
  • the preset threshold may be, but is not limited to, a value less than 19.4ms. For example, a value between 11.1 and 19.4 may be taken.
  • periodConfirmed may identify whether a preset scenario occurs by determining whether the difference is within a preset value range.
  • the preset value range may be 11.1 to 19.4.
  • the preset value range may not include 11.1, but may include 19.4.
  • the preset threshold is not limited to being determined based on 19.4ms, and this embodiment does not limit the preset threshold.
  • HWC synthesizes the image data to generate image frames.
  • the HWC may send the synthesized image data to a display driver, and the display driver may drive the screen to display the image frame.
  • SF.cpp receives the instruction sent by VsyncReactor.cpp, which indicates the refresh rate switching and the Vsync-hw(i) time consumption is 19.4ms. Then SF.cpp does not need to wait for the rising and falling edges of Vsync-sf. After determining that crtc_commit releases the fence resources, SF can read the image data, so that SF reads the image data in advance, and similarly HWC synthesizes the image data in advance.
  • VsyncReactor.cpp prohibits calling addResyncSample to sample Vsync-hw(i+1), so that the timestamp of Vsync-hw(i+1) is not stored in the timestamp sequence.
  • VsyncReactor.cpp calls addResyncSample to sample Vsync-hw(i+1), but prohibits calling addHwVsyncTimestamp to store the timestamp of Vsync-hw(i+1) in the timestamp sequence, thereby achieving the purpose of not storing the timestamp of Vsync-hw(i+1) in the timestamp sequence.
  • VsyncReactor.cpp calls addResyncSample to sample Vsync-hw(i+2) to obtain the timestamp of Vsync-hw(i+2), and calls addHwVsyncTimestamp to store the timestamp of Vsync-hw(i+2) into the timestamp sequence.
  • DispSyncSource.cpp generates Vsync-sf(i+3) according to the timestamp of Vsync-hw(i+2).
  • the generated Vsync-sf(i+3) is aligned with Vsync-hw(i+3).
  • SF.cpp After SF.cpp determines that crtc_commit has released the fence resources and is at the rising edge or falling edge of Vsync-sf(i+3), SF.cpp reads the image data.
  • the HWC may send the synthesized image data to a display driver, and the display driver may drive the screen to display the image frame.
  • Vsync-sf(i+3) and Vsync-hw(i+3) are aligned to ensure that Vsync-app, Vsync-sf and Vsync-hw can be aligned after the refresh rate is switched.
  • SF reads the image data and HWC synthesizes the image data to complete SF synthesis through SF and HWC to generate a frame of image frames.
  • the electronic device can generate image frames normally to avoid the phenomenon of synthesizing one frame and losing one frame.
  • FIG14 shows an example of a timestamp sequence and a predicted Vsync-sf.
  • Vsync-hw When sampling Vsync-hw, multiple time points of Vsync-hw can be sampled to obtain multiple timestamps, and Vsync-sf is generated using multiple timestamps of Vsync-hw.
  • FIG14 takes the use of multiple timestamps of the i-th Vsync-hw to generate the (i+2)th Vsync-sf as an example. This embodiment does not limit which Vsync-sf is generated using which Vsync-hw.
  • Multiple timestamps of Vsync-hw can be stored in a timestamp queue as a timestamp set.
  • FIG14 takes the use of one timestamp (such as the first timestamp) of multiple timestamps of Vsync-hw to store in a timestamp queue as an example for
  • the first timestamp TS1 of Vsync-hw (abbreviated as Vsync-hw2) in the second cycle is stored in The timestamp of Vsync-hw2 is used to generate Vsync-sf4; the first timestamp TS2 of Vsync-hw3 is stored in the timestamp queue, and Vsync-sf5 can be generated according to the timestamp of Vsync-hw3; the first timestamp TS3 of Vsync-hw4 is stored in the timestamp queue, and Vsync-sf6 can be generated according to the timestamp of Vsync-hw4.
  • Vsync-hw3 takes a long time, the sampling time of Vsync-hw3 is increased, so that Vsync-sf5 is generated with a delay, and before the refresh rate is switched, the period of Vsync-hw should be the same as that of Vsync-sf, but the time consumption of Vsync-hw3 increases, so that the periods of Vsync-hw3 and Vsync-sf3 are different, resulting in the end time of Vsync-sf3 being different from the end time of Vsync-hw3, and the end of Vsync-hw3 is later than the end of Vsync-sf3.
  • Vsync-sf3 After Vsync-sf3 ends, Vsync-sf4 starts, and after Vsync-hw3 ends, Vsync-hw4 starts. If the end time of Vsync-sf3 is different from the end time of Vsync-hw3, the start time of Vsync-sf4 is different from that of Vsync-hw4. Starting from Vsync-hw4, the software/hardware Vsync is not aligned.
  • Figure 15 shows the optimization of the timestamp sequence and predicted Vsync-sf.
  • addHwVsyncTimestamp skips the timestamp of Vsync-hw3 and generates Vsync-sf5 using the timestamp of Vsync-hw4, where Vsync-sf5 is separated from Vsync-hw3 by a Vsync-hw cycle, and Vsync-hw5 is separated from Vsync-hw3 by a Vsync cycle.
  • the cycle of c-hw indicates that the start time of Vsync-hw5 and Vsync-sf5 is separated from the end time of Vsync-hw3 by a cycle of Vsync-hw (i.e., one Vsync-hw4).
  • Vsync-hw5 and Vsync-sf5 have the same refresh rate, indicating that the cycle of Vsync-hw5 is the same as that of Vsync-sf5.
  • Vsync-hw5 and Vsync-sf5 have not only the same cycle, but also the same benchmark, i.e., Vsync-hw5 and Vsync-sf5 are aligned.
  • addHwVsyncTimestamp skips the timestamp of Vsync-hw3 and generates Vsync-sf5 using the timestamp of Vsync-hw4, and Vsync-hw4 is the next Vsync-hw of Vsync-hw3, then using the timestamp of Vsync-hw4 to generate Vsync-sf5 means that the generation of Vsync-sf5 is delayed, and the end time of Vsync-sf4 is also delayed, which increases the period of Vsync-sf4.
  • the period of Vsync-sf4 is 19.4ms, and the dotted line indicates that Vsync-sf5 is generated using Vsync-hw3, and when Vsync-sf5 is generated using the timestamp of Vsync-hw4, the period of Vsync-sf4 is increased from 16.6ms to 19.4ms.
  • the period of Vsync-sf4 increases, Vsync-hw5 and Vsync-sf5 are aligned, so that starting from Vsync-sf5, the electronic device can generate image frames normally, avoiding the phenomenon of synthesizing one frame and losing one frame.
  • the present application provides an electronic device, which includes: one or more processors; one or more memories; the memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the above-mentioned display method.
  • the present application provides a computer-readable storage medium, in which a computer program is stored.
  • the computer program is executed by a processor, the processor executes the above-mentioned display method.

Abstract

提供了一种显示方法及装置。显示方法包括：确定电子设备的刷新率从第一刷新率切换至第二刷新率，第一刷新率大于第二刷新率；响应刷新率从第一刷新率切换至第二刷新率，在电子设备完成第一图像帧的显示后，生成第二图像帧；显示第二图像帧，目的在于解决从高刷新率切换至低刷新率出现的丢帧问题，以降低电子设备的卡顿感，提高用户体验。

Description

显示方法及装置
本申请要求于2022年09月30日提交中国国家知识产权局、申请号为202211230396.X、发明名称为“显示方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种显示方法及装置。
背景技术
电子设备可以安装多种应用程序(Application，APP)，在运行APP过程中，以APP对应的刷新率控制图像帧的显示。多种APP中部分APP对应的刷新率不同，电子设备切换APP时刷新率也同步切换，但是电子设备从一个高刷新率切换至低刷新率时出现丢帧问题，使得电子设备出现明显的卡顿感，降低用户体验。
发明内容
本申请提供了一种显示方法及装置，目的在于解决从高刷新率切换至低刷新率出现的丢帧问题，以降低电子设备的卡顿感，提高用户体验。为了实现上述目的，本申请提供了以下技术方案：
第一方面，本申请提供一种显示方法，应用于电子设备中，方法包括：确定电子设备的刷新率从第一刷新率切换至第二刷新率，第一刷新率大于第二刷新率；响应刷新率从第一刷新率切换至第二刷新率，在电子设备完成第一图像帧的显示后，生成第二图像帧；显示第二图像帧。
通常情况下，电子设备生成一帧图像帧，需要满足两个条件：一个条件是第一信号(即Vsync-sf)的电平变化满足预设条件，预设条件是第一信号的电平从低电平变为高电平(对应上升沿)，或者第一信号的电平从高电平变为低电平(对应下降沿)；另一个条件是电子设备完成前一帧图像帧的显示。但是电子设备从第一刷新率切换至第二刷新率时第二信号(即Vsync-hw)的耗时增大(即周期增大)。耗时增大的第二信号可以覆盖至少一个第一信号，在覆盖的第一信号下，电子设备不能确定是否完成图像帧的显示，使得电子设备无法处理其他图像帧，导致图像帧丢失。在第二信号的结束位置，电子设备可以确定完成一帧图像帧的显示，但是电子设备还要等待第一信号的上升沿或下降沿，在等待过程中也可能导致电子设备丢失图像帧。而本申请提供的显示方法，在确定刷新率从第一刷新率切换至第二刷新率，且在电子设备完成第一图像帧的显示后，电子设备可以立即生成并显示第二图像帧，由此电子设备在确定完成一帧图像帧的显示后，电子设备不需要等待第一信号的上升沿或下降沿就可以立即生成并显示另一帧图像帧，从而减少图像帧的丢失，降低了电子设备的卡顿感，提高了用户体验。
可选的，方法还包括：在刷新率切换至第二刷新率后，控制第一信号与第二信号对齐，第一信号用于指示生成图像帧，第二信号用于指示显示图像帧；在电子设备完成第二图像帧的显示，且第一信号的电平变化满足预设条件，生成第三图像帧，其中预设条件是第一信号的电平从低电平变为高电平，或者第一信号的电平从高电平变为低电平；在第二信号 作用下，显示第三图像帧。对齐指的是第一信号和第二信号的周期相同，且第一信号和第二信号的上升沿对齐、第一信号和第二信号的下降沿也对齐，即第一信号和第二信号的波形相同，信号幅值可以相同也可以不同。
通常情况下，在刷新率切换至第二刷新率过程中，电子设备生成的第二信号的周期增大，使得切换前能够对齐的第一信号和第二信号，在刷新率切换后无法保持对齐，在这种情况下，图像帧的生成可以每间隔一个第一信号生成一次，从而电子设备出现合成一帧、丢失一帧的现象，合成一帧可以是指生成一帧图像帧，进一步的显示该图像帧。而本申请提供的显示方法中，在刷新率切换后电子设备可以控制第一信号与第二信号对齐，这样电子设备可以在每个第一信号下生成一帧图像帧，防止出现合成一帧、丢失一帧的现象，减少图像帧的丢失，从而降低电子设备的卡顿感，提高了用户体验。
可选的，在确定电子设备的刷新率从第一刷新率切换至第二刷新率之前，方法还包括：响应刷新率为第一刷新率，根据第i个第二信号，生成第(i+n)个第一信号，第(i+n)个第一信号与第i个第二信号之间间隔的时长为n个第一信号的周期之和，n为大于1的自然数；在刷新率切换至第二刷新率后，控制第一信号与第二信号对齐包括：在刷新率切换至第二刷新率后，对刷新率切换后的第一个第二信号进行采样，利用第一个第二信号的采样结果，生成第一信号，第一信号与第一个第二信号之间间隔的时长为(n-1)个第二信号的周期之和。在刷新率切换后，第一信号与第二信号没有对齐的原因是因为在生成第一信号过程中，采用了周期增长的第二信号，所以本申请中，电子设备在确定刷新率切换后，调整了切换后第一信号的生成机制，利用刷新率切换后的第一个第一信号来生成第二信号，以在刷新率切换后第一信号和第二信号是可以对齐的。第二信号的采样结果可以存储在结果序列中，在生成第一信号时，从结果序列中读取第二信号的采样结果。其中第二信号的采样结果可以是第二信号的多个时间戳，结果序列可以是存储第二信号的多个时间戳的时间戳序列。
可选的，方法还包括：丢弃特殊信号，特殊信号是在刷新率从第一刷新率切换至第二刷新率的过程中生成的第二信号，且特殊信号为第一个第二信号的前一个信号，减少电子设备处理的数据量。
可选的，丢弃特殊信号包括：禁止对特殊信号进行采样；或者，对特殊信号进行采样，特殊信号的采样结果没有存储到结果序列中。
可选的，第二图像帧是刷新率从第一刷新率切换至第二刷新率的过程中丢失的一帧图像帧。在一些示例中，在确定完成第一图像帧显示后，电子设备可以立即生成丢失的多帧图像帧的第一帧图像帧，保证图像的连续性；在一些示例中，在确定完成第一图像帧显示后，电子设备可以立即合成丢失的多帧图像帧的最后一帧图像帧，缩短最后一帧图像帧的延后时长，电子设备下一次显示的图像帧是丢失的最后一帧图像帧的下一帧图像帧，立即生成并显示丢失的最后一帧图像帧，可以使得下一次显示的图像帧和丢失的最后一帧图像帧是连续的，保证了连续性。
可选的，电子设备完成第一图像帧的显示包括：电子设备调用内核线程crtc_commit释放fence资源后，确定电子设备完成第一图像帧的显示。
可选的，确定电子设备的刷新率从第一刷新率切换至第二刷新率包括：确定电子设备的刷新率为第一刷新率时第一信号的结束时间以及第一信号的开始时间；确定结束时间和 开始时间之间的差值；如果差值满足预设切换条件，确定电子设备的刷新率从第一刷新率切换至第二刷新率。预设切换条件可以用于指示第一信号的周期增大，如预设切换条件可以是一个预设阈值或者一个预设取值范围，预设阈值和预设取值范围的取值根据刷新率确定，如从90Hz切换至60Hz时，预设阈值可以是小于19.4ms的一个数值，预设取值范围可以是(11.1，19.4]。
可选的，电子设备的应用程序框架层包括：刷新率处理单元和数据读取单元；电子设备的硬件抽象层包括硬件混合渲染器；电子设备的内核层包括显示驱动；刷新率处理单元，用于确定电子设备的刷新率从第一刷新率切换至第二刷新率；数据读取单元，用于响应刷新率从第一刷新率切换至第二刷新率，在电子设备完成第一图像帧的显示后，读取第一图像数据；硬件混合渲染器，用于合成第一图像数据，以生成第二图像帧；显示驱动，用于显示第二图像帧。
可选的，应用程序框架层还包括：信号生成单元，信号生成单元，用于在刷新率切换至第二刷新率后，控制第一信号与第二信号对齐，第一信号用于指示生成图像帧，第二信号用于指示显示图像帧；数据读取单元，还用于在电子设备完成第二图像帧的显示，且第一信号的电平变化满足预设条件，读取第二图像数据，其中预设条件是第一信号的电平从低电平变为高电平，或者第一信号的电平从高电平变为低电平；硬件混合渲染器，还用于合成第二图像数据，以生成第三图像帧；显示驱动，还用于在第二信号作用下，显示第三图像帧。
第二方面，本申请提供一种电子设备，电子设备包括：一个或多个处理器；一个或多个存储器；存储器存储有一个或多个程序，当一个或者多个程序被处理器执行时，使得电子设备执行上述显示方法。
第三方面，本申请提供一种计算机可读存储介质，计算机可读存储介质中存储了计算机程序，当计算机程序被处理器执行时，使得处理器执行上述显示方法。
本申请在上述各方面提供的实现方式的基础上，还可以进行进一步组合以提供更多实现方式。
附图说明
图1为本申请提供的利用Vsync-app、Vsync-sf和Vsync-hw进行图像帧的渲染至显示的一种示意图；
图2为本申请提供的显示驱动生成Vsync-hw的示意图；
图3为本申请提供的丢帧问题的跟踪示意图；
图4为本申请提供的丢帧问题的放大示意图；
图5为本申请提供的刷新率发生切换后软/硬件Vsync的示意图；
图6为本申请提供的在刷新率切换后利用Vsync-app、Vsync-sf和Vsync-hw进行图像帧的渲染至显示的一种示意图；
图7为本申请提供的优化丢帧问题的跟踪示意图；
图8为本申请提供的优化丢帧问题的放大示意图；
图9为本申请提供的刷新率发生切换后优化软/硬件Vsync的示意图；
图10为本申请提供的利用优化后的Vsync-app、Vsync-sf和Vsync-hw进行图像帧的渲 染至显示的一种示意图；
图11为本申请提供的电子设备的硬件架构图；
图12为本申请提供的电子设备的软件架构图；
图13为本申请提供的显示方法的信令图；
图14和图15为本申请提供的时间戳序列和预测Vsync-sf的示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述。以下实施例中所使用的术语只是为了描述特定实施例的目的，而并非旨在作为对本申请的限制。如在本申请的说明书和所附权利要求书中所使用的那样，单数表达形式“一个”、“一种”、“所述”、“上述”、“该”和“这一”旨在也包括例如“一个或多个”这种表达形式，除非其上下文中明确地有相反指示。还应当理解，在本申请实施例中，“一个或多个”是指一个、两个或两个以上；“和/或”，描述关联对象的关联关系，表示可以存在三种关系；例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B的情况，其中A、B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。
在本说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。
本申请实施例涉及的多个，是指大于或等于两个。需要说明的是，在本申请实施例的描述中，“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。
首先，介绍本申请涉及的技术术语：
垂直同步(Vertical Synchronization，Vsync)信号分为硬件Vsync信号和软件Vsync信号，硬件Vsync信号简写为Vsync-hw信号，Vsync-hw信号可以理解为撕裂效应(Tearing Effect，TE)信号。软件Vsync信号包括Vsync-app信号和Vsync-sf信号。为了便于描述，Vsync-hw信号、Vsync-app信号和Vsync-sf信号简写为Vsync-hw、Vsync-app和Vsync-sf。
Vsync-app和Vsync-sf是根据Vsync-hw生成，具体是由SF(surfaceflinger)中的DispSyncSource.cpp将Vsync-hw虚拟化成Vsync-app和Vsync-sf，其中DispSyncSource.cpp对Vsync-hw进行采样，采样结果输入到软件Vsync模型中，DispSyncSource.cpp运行软件Vsync模型，以通过软件Vsync模型输出Vsync-app和Vsync-sf。Vsync-hw可以由显示驱动根据刷新率生成。
电子设备利用Vsync-app、Vsync-sf和Vsync-hw进行图像帧的渲染至显示的过程如图1所示。在Vsync-app作用下，APP生成图像数据(这一过程为APP渲染过程，图1中的render表示图像数据)，将图像数据存储在缓冲区(buffer)中。在Vsync-sf作用下SF从buffer中读取图像数据，将图像数据发送给硬件混合渲染器(HWChwcomposer，HWC)，由HWC合成图像数据，以生成一帧图像帧(这一过程称为SF合成)。HWC将图像帧发送给显示驱 动，在Vsync-hw作用下显示驱动将图像帧发送至屏幕上显示(这一过程称为HWC送显)，在屏幕完成图像帧显示后，显示驱动调用内核线程crtc_commit，crtc_commit释放fence资源，释放fence资源可以表示电子设备完成一帧图像帧的显示。
然而，一帧图像帧的显示不完全独立于前一帧图像帧，在前一帧图像帧显示完成且crtc_commit释放了fence资源后，SF才可以在Vsync-sf作用下读取下一帧图像帧的图像数据，然后通过HWC和显示驱动完成下一帧图像帧的显示。在图1中，一帧图像帧从渲染至显示经历了三个周期，在三个周期的第一个周期完成APP渲染，第二个周期完成SF合成，第三个周期完成HWC送显。
其中Vsync-app、Vsync-sf和Vsync-hw这三个信号的周期可以根据APP对应的刷新率确定，如APP对应的刷新率为90Hz，则Vsync-app、Vsync-sf和Vsync-hw这三个信号的周期是11.1毫秒(ms)；APP对应的刷新率为60Hz，则Vsync-app、Vsync-sf和Vsync-hw这三个信号的周期是16.6ms。电子设备切换APP时刷新率也同步切换，但是电子设备从一个高刷新率切换至低刷新率时，Vsync-hw的周期过长导致在APP切换过程中会出现丢帧问题。此外在切换完成后Vsync-app和Vsync-sf与Vsync-hw没有对齐，导致电子设备在切换完成后出现合成一帧、丢一帧交替出现的情况，使得电子设备出现明显的卡顿感，降低用户体验。其中合成一帧可以是通过SF合成一帧图像帧的图像数据。丢一帧可以是延后执行SF合成，如在下一个Vsync-sf的上升沿或下降沿执行SF合成，以在下一个Vsync-sf合成图像帧的图像数据，又或者丢一帧可以是丢掉一帧图像帧的图像数据。
以电子设备的刷新率从90Hz切换到60Hz为例。电子设备中的显示驱动可以生成Vsync-hw，并在刷新率切换时，显示驱动可以模拟刷新率切换，以根据切换后的刷新率，调整Vsync-hw。示意图如图2所示，在刷新率为90Hz时，显示驱动生成一个时长为11.1ms的Vsync-hw，该Vsync-hw包括一个时长为8.3ms的低电平和一个时长为2.8ms的高电平。显示驱动确定在Vsync-hw的上升沿发生刷新率切换，且刷新率从90Hz切换到60Hz，显示驱动以上升沿作为一个Vsync-hw的开始，该Vsync-hw包括一个时长为2.8ms的高电平和一个时长为16.6ms的低电平，该Vsync-hw的耗时为19.4ms，因此显示驱动模拟90Hz下Vsync-hw的机制导致在刷芯片从90Hz切换至60Hz时出现一个耗时为19.4ms的Vsync-hw，该现象不但导致SF因等不到fence资源而丢失1到2帧图像帧，还导致Vsync-sf与Vsync-hw上升沿、下降沿持续对不齐而出现合成一帧、丢一帧交替出现的情况，带来明显的卡顿感。
丢帧问题的跟踪(trace)如图3所示，在90Hz切换到60Hz时，存在一个Vsync-hw耗时长达19.4ms，图3中以TE＝19.4ms表示，在19.4ms时长下，SF没有等到crtc_commit释放fence资源，SF合成过程无法执行，导致在19.4ms时长下丢失两帧图像帧，如图3中的1和2所示。其中图3中的1和2的放大示意图如图4所示，从onMessageReceived至HwcPresentOrValidateDisplay表示正常执行SF合成；如果SF一直在等待(wait)，说明SF没有等到crtc_commit释放fence资源，存在丢帧的可能性，在图4中，SF丢失两帧后crtc_commit才释放了fence资源。
在Vsync-sf的上升沿和下降沿，如果SF等到crtc_commit释放了fence资源，SF可以读取图像数据。如果SF没有等到crtc_commit释放fence资源，SF无法读取图像数据。在19.4ms之后，每间隔一个Vsync-sf，SF等到一次fence资源，说明SF可以每间隔一个Vsync-sf 读取一次图像数据，从而出现合成一帧、丢一帧交替出现的情况。如图3中的3所示，3指向的大方框示出了合成一帧、丢一帧交替出现，大方框中的小方块指示丢一帧。对于3指向大方框的第一个小方框来说，因为在Vsync-sf的上升沿SF没有等到crtc_commit释放fence资源，SF没有读取图像数据，所以丢失一帧图像帧。接着在该Vsync-sf的下降沿SF等到crtc_commit释放fence资源，SF可以读取图像数据，并由HWC合成图像数据，因此在该Vsync-sf的下降沿合成一帧图像帧的图像数据。在SF丢失5帧图像帧后，电子设备开始以60Hz正常合成。
例如电子设备侧滑退出APP过程中，刷新率从90Hz切换至60Hz，在这一切换过程中，电子设备先丢失2帧图像帧。在完成切换后刷新一帧丢失一帧，整个过程出现卡顿感，降低用户体验。
刷新率发生切换后，软/硬件Vsync的示意如图5所示，其示出了90Hz切换至60Hz下软/硬件Vsync的示意图，在图5中竖线可以作为软/硬件Vsync的基准，指向软/硬件Vsync的开始时间。
在刷新率为90Hz时，Vsync-app、Vsync-sf和Vsync-hw这三个信号的周期相同，且至少每个周期下Vsync-sf和Vsync-hw的基准也是相同的，即在每个周期下Vsync-sf和Vsync-hw是对齐的，对齐可以是周期相同且Vsync-sf和Vsync-hw的上升沿对齐、Vsync-sf和Vsync-hw的下降沿对齐。对于Vsync-app来说，周期与Vsync-sf和Vsync-hw，但是上升沿和下降沿可以与Vsync-sf和Vsync-hw存在一定的差异，在图5中Vsync-app是与Vsync-sf和Vsync-hw对齐的。但是刷新率从90Hz向60Hz切换时，出现一个耗时较长的Vsync-hw，如图5中第四个周期下Vsync-hw耗时为19.4ms，即第四个周期出现耗时较长的Vsync-hw，Vsync-hw的周期从11.1ms增加至19.4ms，但是第四个周期下Vsync-app、Vsync-sf的周期仍是11.1ms，因此第四个周期下Vsync-hw的周期与第四个周期下Vsync-app、Vsync-sf的周期不同，导致第五个周期下Vsync-hw的基准与第四个周期下Vsync-app、Vsync-sf的基准不同，从第五个周期开始，Vsync-app、Vsync-sf与Vsync-hw没有对齐。
如图5所示，虽然在刷新率切换至60Hz，Vsync-app、Vsync-sf和Vsync-hw的周期为16.6ms，但是因为第四个周期下Vsync-hw的耗时较长，使得原本在(1)处结束的Vsync-hw在(2)处结束，导致第四个周期下Vsync-hw的周期与第四个周期下Vsync-app、Vsync-sf的周期不同，第四个周期下Vsync-hw的结束与第四个周期下Vsync-app、Vsync-sf的结束不同，进而导致从第五个周期开始，实际情况下的Vsync-app、Vsync-sf与Vsync-hw持续上下不对齐。
并且因为第四个周期下Vsync-hw在(2)处结束，原本crtc_commit在(1)处释放fence资源，延后至在(2)处释放fence资源。因为在(2)处不是Vsync-sf的上升沿或下降沿，SF合成延后一个Vsync-sf，即如图5中所示，在(3)处因SF错过fence资源丢失一帧，原本在(3)处执行的SF合成实际在(4)处执行。
在图5所示软/硬件Vsync作用下，利用Vsync-app、Vsync-sf和Vsync-hw完成图像帧的渲染至显示的过程如图6所示，从图6可以看出，在刷新率从90Hz切换至60Hz之前，APP在一个周期完成渲染，且在该周期下crtc_commit可以释放fence资源，意味着SF在下一个周期可以等到fence资源，在下一个周期执行SF合成。在SF合成后显示驱动在下 一个周期送显，使得屏幕在下一个周期显示图像帧，以在三个周期内完成图像帧的渲染至显示。但是在刷新率从90Hz向60Hz切换时，crtc_commit释放fence资源的耗时增加，如图6中的(1)所示，本该在第四个周期结束时释放的fence资源在第五个周期开始后完成释放，使得第五个周期的SF合成延后至第六个周期，即在(1)处错过fence资源，本该在(1)处执行的SF合成实际在(2)处执行，SF合成延后导致丢失一帧；同样的，在(3)处错过fence资源，本该在(3)处执行的SF合成实际在(4)处执行，丢失一帧，即相对于每个Vsync-app下生成一帧图像帧，在刷新率切换后变为每间隔一个Vsync-app生成一帧图像帧。
从上述图3至图6可知，电子设备从一个高刷新率的APP切换至低刷新率的APP时，Vsync-hw的周期过长导致在APP切换过程中会出现丢帧问题。在切换完成后Vsync-app和Vsync-sf与Vsync-hw没有对齐，导致在切换完成后出现合成一帧、丢一帧交替出现的情况，使得电子设备出现明显的卡顿感，降低用户体验。
针对这一问题，本申请提供一种显示方法，该显示方法响应于从高刷新率切换至低刷新率，在接收到crtc_commit释放fence资源的指令，但没有接收到Vsync-sf的情况下，合成一帧图像帧的图像数据。也就是说，在从高刷新率切换至低刷新率时，虽然SF没有处于Vsync-sf的上升沿或下降沿，但是SF确定crtc_commit释放了fence资源(即完成了一帧图像帧的显示)，那么SF可以在crtc_commit释放了fence资源后立即读取图像数据以立即执行SF合成，相对于等待接收到Vsync-sf读取图像数据来说，提前合成一帧图像帧的图像数据，从而在高刷新率切换至低刷新率时，能够快速合成一帧图像帧的图像数据，减少了丢帧数量。
在一些示例中，响应于切换至低刷新率，同步Vsync-app、Vsync-sf和Vsync-hw，即在切换至低刷新率后，将对齐Vsync-app、Vsync-sf和Vsync-hw，以解决因切换后Vsync-app、Vsync-sf和Vsync-hw没有对齐引起的合成一帧、丢一帧交替出现的情况。
如图7示出了电子设备实施本申请的显示方法后，丢帧问题优化的trace示意。在图7中刷新率从90Hz切换至60Hz，在从90Hz切换至60Hz时，一帧Vsync-hw耗时从11.1ms延长至19.4ms(如图7中的TE＝19.4ms所示)。耗时19.4ms的Vsync-hw覆盖一个Vsync-sf，一般crtc_commit在Vsync-hw的结束位置完成fence资源的释放，因此在覆盖的Vsync-sf下，SF等不到crtc_commit释放fence资源，导致在耗时19.4ms的Vsync-hw下丢失两帧。在耗时19.4ms的Vsync-hw结束后，SF确定crtc_commit释放了fence资源，立即执行SF合成，以立即强制合成一帧图像帧的图像数据。也就是说，虽然SF没有处于Vsync-sf的上升沿或下降沿，但是SF确定了crtc_commit释放了fence资源，立即执行SF合成，以强制合成一帧图像帧的图像数据。
即，一般情况下，SF合成要满足两个条件：一个条件是处于Vsync-sf的上升沿或下降沿，另一个条件是确定crtc_commit释放了fence资源；但是在从高刷新率切换至低刷新率时，出现一个耗时较长的Vsync-hw，SF合成满足一个条件即可，该条件可以是确定crtc_commit释放了fence资源，这样SF不需要等待下一个Vsync-sf的上升沿或下降沿，将在下一个Vsync-sf的SF合成被提前，从而可以提前合成一帧图像帧的图像数据。
其中，提前合成的图像数据可以是丢失两帧图像帧中的任意一帧的图像数据，虽然在 耗时19.4ms的Vsync-hw下丢失两帧，但是在耗时19.4的Vsync-hw结束后可以立即合成出丢失的两帧图像帧中的任意一帧的图像数据，相当于丢失一帧，减少丢帧数量。在一些示例中，在耗时19.4ms的Vsync-hw结束后可以立即合成丢失的两帧图像帧的第一帧的图像数据，保证图像的连续性；在一些示例中，在耗时19.4ms的Vsync-hw结束后可以立即合成丢失的两帧图像帧的第二帧的图像数据，缩短第二帧图像帧的延后合成时长。
因为耗时19.4ms的Vsync-hw可能导致后面的Vsync-app、Vsync-sf和Vsync-hw没有对齐，而Vsync-app、Vsync-sf和Vsync-hw没有对齐可以导致电子设备出现合成一帧、丢一帧的情况。针对这个问题，在切换至低刷新率后，电子设备可以对齐Vsync-app、Vsync-sf和Vsync-hw。Vsync-app、Vsync-sf和Vsync-hw对齐后，SF可以分别在Vsync-sf的上升沿和下降沿从buffer中读取图像数据，由HWC合成图像数据，从而分别在Vsync-sf的上升沿和下降沿执行SF合成，使得电子设备可以在切换至60Hz后，按照60Hz分别在Vsync-sf的上升沿和下降沿执行SF合成，避免图像帧的丢失。
下面结合附图说明，刷新率发生切换后，软/硬件Vsync的优化示意图如图9所示。在图9中，在刷新率为90Hz时，Vsync-app、Vsync-sf和Vsync-hw这三个信号的周期相同，且每个周期下Vsync-app、Vsync-sf和Vsync-hw的基准也是相同的。但是刷新率从90Hz向60Hz切换时，出现耗时较长的Vsync-hw，如图9中第四个周期下Vsync-hw耗时为19.4ms，即在第四个周期出现耗时较长的Vsync-hw，那么原本crtc_commit在(1)结束前释放fence资源，但是因为发生刷新率从90Hz切换至60Hz，crtc_commit在(2)处释放fence资源，在(5)处(即第四个周期的结束)完成fence资源的释放，导致在(3)处因为错过fence资源没有执行SF合成，丢一帧。(4)处为第四个周期下Vsync-hw的结束，在(4)处等到crtc_commit释放了fence资源，在(4)处立即执行SF合成，合成一帧图像帧的图像数据，从而可以强制刷新一帧图像帧，即在第五个周期下强制显示一帧图像帧。
在第五个周期下，Vsync-app、Vsync-sf和Vsync-hw这三个信号的结束时间相同，那么第六个周期下，Vsync-app、Vsync-sf和Vsync-hw这三个信号的开始时间相同，又因为Vsync-app、Vsync-sf和Vsync-hw这三个信号的周期相同，所以第六个周期下Vsync-app、Vsync-sf与Vsync-hw是上下对齐的，如图9所示，从第六个周期开始Vsync-app、Vsync-sf与Vsync-hw是上下对齐的。以Vsync-hw为例，第六个周期可以以图9中第六个竖线为开始时间，第七个竖线为结束时间，从而后续软/硬件Vsync是可以对齐的，电子设备可以以60Hz的刷新率稳定刷新(显示)图像。
软/硬件Vsync优化后，利用Vsync-app、Vsync-sf和Vsync-hw完成图像帧的渲染至显示的过程如图10所示，在(1)处错过fence资源本应该丢一帧，但是(2)处等到fence资源时深灰色帧立即刷新，而不是等到下一个Vsync_sf(具体是等到Vsync_sf的上升沿或下降沿)到来，避免浅灰色帧丢帧；(3)处正常等到浅灰色帧释放fence资源，正常SF合成、HWC送显；(4)处等到深灰色帧释放fence资源，正常SF合成、HWC送显，以此类推，后续无合成一帧、丢一帧交替出现的现象，降低卡顿感，提高用户体验。
上述显示方法可以应用于电子设备中，在一些实施例中，该电子设备可以是手机、平板电脑、桌面型、膝上型、笔记本电脑、超级移动个人计算机(Ultra-mobile Personal Computer，UMPC)、手持计算机、上网本、个人数字助理(Personal Digital Assistant，PDA)、可穿戴 电子设备、智能手表等设备。本申请对电子设备的具体形式不做特殊限定。
如图11所示，该电子设备可以包括：处理器，外部存储器接口，内部存储器，通用串行总线(Universal Serial Bus，USB)接口，充电管理模块，电源管理模块，电池，天线1，天线2，移动通信模块，无线通信模块，传感器模块，按键，马达，指示器，摄像头，显示屏(也可以称为屏幕)，以及用户标识模块(Subscriber Identity Module，SIM)卡接口等。其中音频模块可以包括扬声器，受话器，麦克风，耳机接口等，传感器模块可以包括压力传感器，陀螺仪传感器，气压传感器，磁传感器，加速度传感器，距离传感器，接近光传感器，指纹传感器，温度传感器，触摸传感器，环境光传感器，骨传导传感器等。
处理器可以包括一个或多个处理单元，例如：处理器可以包括应用处理器(Application Processor，AP)，调制解调处理器，图形处理器(Graphics Processing Unit，GPU)，图像信号处理器(Image Signal Processor，ISP)，控制器，视频编解码器，数字信号处理器(Digital Signal Processor，DSP)，基带处理器，和/或神经网络处理器(Neural-network Processing Unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。处理器是电子设备的神经中枢和指挥中心，控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
显示屏用于显示图像，视频、一系列图形用户界面(Graphical User Interface，GUI)等。
外部存储器接口可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备的存储能力。外部存储卡通过外部存储器接口与处理器通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。内部存储器可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器通过运行存储在内部存储器的指令，从而执行电子设备的各种功能应用以及数据处理。例如，在本申请中，处理器通过运行内部存储器中存储的指令，使得电子设备执行本申请提供的显示方法。
可以理解的是，本实施例示意的结构并不构成对电子设备的具体限定。在另一些实施例中，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
另外，在上述部件之上，运行有操作系统。例如苹果公司所开发的iOS操作系统，谷歌公司所开发的Android开源操作系统，微软公司所开发的Windows操作系统等。
电子设备的操作系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本申请实施例以分层架构的Android系统为例，示例性说明电子设备的软件结构。图12是电子设备的软件架构图。分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。以Android系统为例，在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层(Framework)，硬件抽象层(HAL)以及内核层(Kernel)。
其中，应用程序层可以包括一系列应用程序包。应用程序包可以包括相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等APP。应用程序框架层为应用程序层的应用程序提供应用编程接口(Application Programming Interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。例如应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。应用程序框架层还可 以包括SF，SF包括DispSyncSource.cpp、SF.cpp和VsyncReactor.cpp，DispSyncSource.cpp、SF.cpp和VsyncReactor.cpp可以视为SF中的三个单元，如DispSyncSource.cpp可以称为信号生成单元，SF.cpp可以称为数据读取单元，VsyncReactor.cpp可以称为刷新率处理单元。HAL可以包括HWC。内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动。HWC可以用于合成图像数据，在合成图像数据过程中可以调用setCallback、registerCallback和onComposerhalVsync等函数。
其中，VsyncReactor.cpp可以对Vsync-hw进行采样，DispSyncSource.cpp根据采样结果生成Vsync-app、Vsync-sf。VsyncReactor.cpp可以调用addResyncSample对Vsync-hw进行采样，得到Vsync-hw的时间戳，Vsync-hw的时间戳为采样结果的一种示例。Vsync-hw的时间戳通过addHwVsyncTimestamp存储到时间戳序列中。VsyncReactor.cpp还可以调用periodConfirmed进行场景识别。在VsyncReactor.cpp识别到发生高刷新率到低刷新率的切换，且出现耗时较长的Vsync-hw后，可以向SF.cpp发送指令，以指示SF.cpp在确定crtc_commit释放了fence资源后，与HWC执行SF合成，并通过VsyncReactor.cpp调整时间戳序列，以保证刷新率切换后Vsync-app、Vsync-sf和Vsync-hw是可以对齐的。
以电子设备的刷新率为90Hz，刷新率从90Hz切换至60Hz为例，对上述DispSyncSource.cpp、SF.cpp、VsyncReactor.app和HWC之间的交互过程进行说明，其信令图如图13所示，可以包括以下步骤：
S101、VsyncReactor.cpp调用addResyncSample对Vsync-hw(i)进行采样，得到Vsync-hw(i)的时间戳，Vsync-hw(i)的时间戳存储到时间戳序列中。如VsyncReactor.cpp调用addHwVsyncTimestamp将Vsync-hw(i)的时间戳存储到时间戳序列中。
S102、DispSyncSource.cpp根据Vsync-hw(i)的时间戳，生成Vsync-sf(i+2)。Vsync-sf(i+2)是Vsync-sf(i)之后的第二个Vsync-sf，Vsync-hw(i)和Vsync-sf(i)对应同一个周期，在本实施例中，Vsync-hw(i)和Vsync-sf(i)是第i个周期下的Vsync-hw和Vsync-sf。SF在生成Vsync-sf(i+2)时还可以生成Vsync-app(i+2)，在Vsync-app(i+2)作用下APP生成图像数据，此处不再详述。其中，根据Vsync-hw(i)的时间戳，生成Vsync-sf(i+2)，仅是示例，本实施例不限定。
S103、SF.cpp在确定crtc_commit释放了fence资源，且处于Vsync-sf(i+2)的上升沿或下降沿，SF.cpp读取图像数据。处于Vsync-sf(i+2)的上升沿表示Vsync-sf(i+2)的电平从低电平变为高电平，处于Vsync-sf(i+2)的下降沿表示Vsync-sf(i+2)的电平从高电平变为低电平。
S104、HWC合成图像数据，以通过SF.cpp和HWC完成SF合成，生成一帧图像帧。
S105、HWC可以将合成的图像数据发送至显示驱动，由显示驱动驱动屏幕显示图像帧。
S106、VsyncReactor.cpp调用periodConfirmed识别到Vsync-hw(i+1)的耗时为19.4ms，以确定出刷新率发生了切换，VsyncReactor.cpp向SF.cpp发送刷新率发生切换的指令，指令除了指示切换了刷新率，还可以携带Vsync-hw(i+1)的耗时。Vsync-hw(i+1)是Vsync-hw(i)的下一个Vsync-hw。
理想情况下，当前时间下一个Vsync-hw信号结束，那么当前时间与该Vsync-hw的开始时间之间的差值等于该Vsync-hw的周期，因此periodConfirmed可以通过该差值是否超 出预设阈值，识别是否出现预设场景，预设场景为刷新率切换出现了Vsync-hw耗时较长的场景。在90Hz切换至60Hz下，预设阈值可以是但不限于是小于19.4ms的一个数值，例如可以在11.1至19.4之间取一个值。又或者，periodConfirmed可以通过该差值是否在预设取值范围内，识别是否出现预设场景，预设取值范围可以是11.1至19.4，预设取值范围可以不包括11.1，但可以包括19.4；在其他刷新率切换场景下，预设阈值不限于根据19.4ms确定，对于预设阈值，本实施例不进行限定。
S107、SF.cpp响应指令，在确定crtc_commit释放了fence资源后立即读取图像数据。
S108、HWC合成图像数据，以生成一帧图像帧。
S109、HWC可以将合成的图像数据发送至显示驱动，由显示驱动驱动屏幕显示图像帧。
也就是说，SF.cpp接收VsyncReactor.cpp发送的指令，该指令指示刷新率切换且Vsync-hw(i)的耗时为19.4ms的指令，则SF.cpp不需要等待Vsync-sf的上升沿和下降沿，在确定crtc_commit释放了fence资源后，SF可以读取图像数据，使得SF提前读取图像数据，同样的HWC合成图像数据也提前。
S110、VsyncReactor.cpp禁止调用addResyncSample对Vsync-hw(i+1)进行采样，这样时间戳序列中没有存储Vsync-hw(i+1)的时间戳。
S110’、VsyncReactor.cpp调用addResyncSample对Vsync-hw(i+1)进行采样，但是禁止调用addHwVsyncTimestamp将Vsync-hw(i+1)的时间戳存储到时间戳序列中，同样达到时间戳序列中没有存储Vsync-hw(i+1)的时间戳的目的。
S111、VsyncReactor.cpp调用addResyncSample对Vsync-hw(i+2)进行采样，得到Vsync-hw(i+2)的时间戳，并调用addHwVsyncTimestamp将Vsync-hw(i+2)的时间戳存储到时间戳序列中。
S112、DispSyncSource.cpp根据Vsync-hw(i+2)的时间戳，生成Vsync-sf(i+3)。生成的Vsync-sf(i+3)与Vsync-hw(i+3)是对齐的。
S113、SF.cpp在确定crtc_commit释放了fence资源，且处于Vsync-sf(i+3)的上升沿或下降沿，SF.cpp读取图像数据。
S114、HWC合成图像数据，以通过SF.cpp和HWC完成SF合成，生成一帧图像帧。
S115、HWC可以将合成的图像数据发送至显示驱动，由显示驱动驱动屏幕显示图像帧。
Vsync-sf(i+3)和Vsync-hw(i+3)对齐，保证在刷新率切换后，Vsync-app、Vsync-sf和Vsync-hw是可以对齐的。在后续Vsync-sf的上升沿或下降沿作用下，SF读取图像数据，并由HWC合成图像数据，以通过SF和HWC完成SF合成，生成一帧图像帧，这样从Vsync-sf(i+3)开始，电子设备可以正常生成图像帧，避免出现合成一帧、丢一帧的现象。
图14示出了时间戳序列和预测Vsync-sf的一种示例，在对Vsync-hw进行采样时可以对Vsync-hw的多个时间点进行采样，得到多个时间戳，利用Vsync-hw的多个时间戳生成Vsync-sf。其中图14是以利用第i个Vsync-hw的多个时间戳，生成第(i+2)个Vsync-sf为例，本实施例对利用第几个Vsync-hw生成第几个Vsync-sf，不进行限定。Vsync-hw的多个时间戳可以作为一个时间戳集合存储到时间戳队列中，在图14中以Vsync-hw的多个时间戳中的一个时间戳(如第一个时间戳)存储到时间戳队列中为例进行说明。
如在图14中，第二个周期下Vsync-hw(简写为Vsync-hw2)的第一个时间戳TS1存 储到时间戳队列中，Vsync-hw2的时间戳用于生成Vsync-sf4；Vsync-hw3的第一个时间戳TS2存储到时间戳队列中，Vsync-sf5可以根据Vsync-hw3的时间戳生成；Vsync-hw4的第一个时间戳TS3存储到时间戳队列中，Vsync-sf6可以根据Vsync-hw4的时间戳生成。因Vsync-hw3耗时较长，增加了Vsync-hw3的采样耗时，使得Vsync-sf5延时生成，并且在刷新率切换前，Vsync-hw的周期应该与Vsync-sf的周期相同，但是Vsync-hw3的耗时增加，使得Vsync-hw3与Vsync-sf3的周期不同，导致Vsync-sf3的结束时间与Vsync-hw3的结束时间不同，Vsync-hw3的结束晚于Vsync-sf3的结束。Vsync-sf3结束后，Vsync-sf4开始，Vsync-hw3结束后，Vsync-hw4开始，那么在Vsync-sf3的结束时间与Vsync-hw3的结束时间不同的情况下，Vsync-sf4与Vsync-hw4的开始不同，从Vsync-hw4开始，软/硬件Vsync没有对齐。
图15示出了时间戳序列和预测Vsync-sf的优化，在识别到刷新率发生切换，且刷新率切换之前的Vsync-hw3的耗时为19.4ms，则addHwVsyncTimestamp跳过Vsync-hw3的时间戳，利用Vsync-hw4的时间戳生成Vsync-sf5，其中Vsync-sf5与Vsync-hw3之间间隔一个Vsync-hw的周期，Vsync-hw5与Vsync-hw3之间也间隔一个Vsync-hw的周期，说明Vsync-hw5和Vsync-sf5的开始时间都与Vsync-hw3的结束时间间隔一个Vsync-hw的周期(即间隔一个Vsync-hw4)，Vsync-hw5与Vsync-sf5对应相同的刷新率，说明Vsync-hw5的周期与Vsync-sf5的周期相同，因此Vsync-hw5和Vsync-sf5不但周期相同，基准也是相同的，即Vsync-hw5和Vsync-sf5对齐。
因为addHwVsyncTimestamp跳过Vsync-hw3的时间戳，利用Vsync-hw4的时间戳生成Vsync-sf5，Vsync-hw4是Vsync-hw3的下一个Vsync-hw，那么利用Vsync-hw4的时间戳生成Vsync-sf5，意味着Vsync-sf5的生成延后，Vsync-sf4的结束时间也被延后，使得Vsync-sf4的周期增大。如图15中Vsync-sf4的周期为19.4ms，虚线表示利用Vsync-hw3生成Vsync-sf5，在利用Vsync-hw4的时间戳生成Vsync-sf5时使得Vsync-sf4的周期从16.6ms增大到19.4ms。虽然Vsync-sf4的周期增大，但是Vsync-hw5和Vsync-sf5是对齐的，这样从Vsync-sf5开始，电子设备可以正常生成图像帧，避免出现合成一帧、丢一帧的现象。
此外，本申请提供一种电子设备，电子设备包括：一个或多个处理器；一个或多个存储器；存储器存储有一个或多个程序，当一个或者多个程序被处理器执行时，使得电子设备执行上述显示方法。
本申请提供一种计算机可读存储介质，计算机可读存储介质中存储了计算机程序，当计算机程序被处理器执行时，使得处理器执行上述显示方法。

Claims (12)

  1. 一种显示方法，其特征在于，应用于电子设备中，所述方法包括：
    确定所述电子设备的刷新率从第一刷新率切换至第二刷新率，所述第一刷新率大于所述第二刷新率；
    响应所述刷新率从所述第一刷新率切换至所述第二刷新率，在所述电子设备完成第一图像帧的显示后，生成第二图像帧；
    显示所述第二图像帧。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    在所述刷新率切换至所述第二刷新率后，控制第一信号与第二信号对齐，所述第一信号用于指示生成图像帧，所述第二信号用于指示显示图像帧；
    在所述电子设备完成所述第二图像帧的显示，且所述第一信号的电平变化满足预设条件，生成第三图像帧，其中所述预设条件是所述第一信号的电平从低电平变为高电平，或者所述第一信号的电平从所述高电平变为所述低电平；
    在所述第二信号作用下，显示所述第三图像帧。
  3. 根据权利要求2所述的方法，其特征在于，在所述确定所述电子设备的刷新率从第一刷新率切换至第二刷新率之前，所述方法还包括：响应所述刷新率为所述第一刷新率，根据第i个所述第二信号，生成第(i+n)个所述第一信号，所述第(i+n)个所述第一信号与所述第i个所述第二信号之间间隔的时长为n个所述第一信号的周期之和，n为大于1的自然数；
    所述在所述刷新率切换至所述第二刷新率后，控制第一信号与第二信号对齐包括：在所述刷新率切换至所述第二刷新率后，对刷新率切换后的第一个所述第二信号进行采样，利用所述第一个所述第二信号的采样结果，生成所述第一信号，所述第一信号与所述第一个所述第二信号之间间隔的时长为(n-1)个所述第二信号的周期之和。
  4. 根据权利要求3所述的方法，其特征在于，所述方法还包括：丢弃特殊信号，所述特殊信号是在所述刷新率从所述第一刷新率切换至所述第二刷新率的过程中生成的所述第二信号，且所述特殊信号为所述第一个所述第二信号的前一个信号。
  5. 根据权利要求4所述的方法，其特征在于，所述丢弃特殊信号包括：禁止对所述特殊信号进行采样；
    或者，
    对所述特殊信号进行采样，所述特殊信号的采样结果没有存储到结果序列中。
  6. 根据权利要求1至5中任意一项所述的方法，其特征在于，所述第二图像帧是所述刷新率从所述第一刷新率切换至所述第二刷新率的过程中丢失的一帧图像帧。
  7. 根据权利要求1至6中任意一项所述的方法，其特征在于，所述电子设备完成第一图像帧的显示包括：所述电子设备调用内核线程crtc_commit释放fence资源后，确定所述电子设备完成所述第一图像帧的显示。
  8. 根据权利要求1至7中任意一项所述的方法，其特征在于，所述确定所述电子设备的刷新率从第一刷新率切换至第二刷新率包括：确定所述电子设备的刷新率为所述第一刷新率时第一信号的结束时间以及所述第一信号的开始时间；
    确定所述结束时间和所述开始时间之间的差值；
    如果所述差值满足预设切换条件，确定所述电子设备的刷新率从第一刷新率切换至第二刷新率。
  9. 根据权利要求1至8中任意一项所述的方法，其特征在于，所述电子设备的应用程序框架层包括：刷新率处理单元和数据读取单元；所述电子设备的硬件抽象层包括硬件混合渲染器；所述电子设备的内核层包括显示驱动；
    所述刷新率处理单元，用于确定所述电子设备的刷新率从第一刷新率切换至第二刷新率；
    所述数据读取单元，用于响应所述刷新率从所述第一刷新率切换至所述第二刷新率，在所述电子设备完成第一图像帧的显示后，读取第一图像数据；
    所述硬件混合渲染器，用于合成所述第一图像数据，以生成第二图像帧；
    所述显示驱动，用于显示所述第二图像帧。
  10. 根据权利要求9所述的方法，其特征在于，所述应用程序框架层还包括：信号生成单元，所述信号生成单元，用于在所述刷新率切换至所述第二刷新率后，控制第一信号与第二信号对齐，所述第一信号用于指示生成图像帧，所述第二信号用于指示显示图像帧；
    所述数据读取单元，还用于在所述电子设备完成所述第二图像帧的显示，且所述第一信号的电平变化满足预设条件，读取第二图像数据，其中所述预设条件是所述第一信号的电平从低电平变为高电平，或者所述第一信号的电平从所述高电平变为所述低电平；
    所述硬件混合渲染器，还用于合成所述第二图像数据，以生成第三图像帧；
    所述显示驱动，还用于在所述第二信号作用下，显示所述第三图像帧。
  11. 一种电子设备，其特征在于，所述电子设备包括：
    一个或多个处理器；
    一个或多个存储器；
    所述存储器存储有一个或多个程序，当所述一个或者多个程序被所述处理器执行时，使得所述电子设备执行如权利要求1至10中任意一项所述的显示方法。
  12. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储了计算机程序，当所述计算机程序被处理器执行时，使得所述处理器执行如权利要求1至10中任意一项所述的显示方法。
PCT/CN2023/116602 2022-09-30 2023-09-01 显示方法及装置 WO2024066926A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211230396.XA CN117854455A (zh) 2022-09-30 2022-09-30 显示方法及装置
CN202211230396.X 2022-09-30

Publications (1)

Publication Number Publication Date
WO2024066926A1 true WO2024066926A1 (zh) 2024-04-04

Family

ID=90476081

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/116602 WO2024066926A1 (zh) 2022-09-30 2023-09-01 显示方法及装置

Country Status (2)

Country Link
CN (1) CN117854455A (zh)
WO (1) WO2024066926A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180262815A1 (en) * 2016-03-24 2018-09-13 Tencent Technology (Shenzhen) Company Limited Video processing method and apparatus, and computer storage medium
CN109104638A (zh) * 2018-08-03 2018-12-28 Oppo广东移动通信有限公司 帧率优化方法、装置、终端及存储介质
WO2019242488A1 (zh) * 2018-06-22 2019-12-26 Oppo广东移动通信有限公司 丢帧补偿方法、装置、存储介质及终端
CN110933315A (zh) * 2019-12-10 2020-03-27 Oppo广东移动通信有限公司 图像数据处理方法及相关设备
CN112256219A (zh) * 2020-10-13 2021-01-22 北京小米移动软件有限公司 显示方法及装置、终端及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180262815A1 (en) * 2016-03-24 2018-09-13 Tencent Technology (Shenzhen) Company Limited Video processing method and apparatus, and computer storage medium
WO2019242488A1 (zh) * 2018-06-22 2019-12-26 Oppo广东移动通信有限公司 丢帧补偿方法、装置、存储介质及终端
CN109104638A (zh) * 2018-08-03 2018-12-28 Oppo广东移动通信有限公司 帧率优化方法、装置、终端及存储介质
CN110933315A (zh) * 2019-12-10 2020-03-27 Oppo广东移动通信有限公司 图像数据处理方法及相关设备
CN112256219A (zh) * 2020-10-13 2021-01-22 北京小米移动软件有限公司 显示方法及装置、终端及存储介质

Also Published As

Publication number Publication date
CN117854455A (zh) 2024-04-09

Similar Documents

Publication Publication Date Title
AU2020233755B2 (en) Controlling display performance
WO2019174474A1 (zh) 卡顿信息获取方法、装置及终端
JP6894976B2 (ja) 画像円滑性向上方法および装置
WO2020207354A1 (zh) 任务处理方法、装置、终端以及计算机可读存储介质
CN115631258B (zh) 一种图像处理方法及电子设备
CN109343902B (zh) 音频处理组件的运行方法、装置、终端及存储介质
WO2021013055A1 (zh) 数据处理的方法、装置及电子设备
CN114518817B (zh) 一种显示方法、电子设备及存储介质
CN114911336B (zh) 调整频率的方法、装置、电子设备及可读存储介质
CN111813490A (zh) 插帧处理方法及装置
CN116052618B (zh) 一种屏幕刷新率切换方法及电子设备
CN115097994B (zh) 数据处理方法和相关装置
CN113132526A (zh) 一种页面绘制方法及相关装置
CN114189732A (zh) 图像数据读写控制的方法及相关装置
WO2024066926A1 (zh) 显示方法及装置
WO2024098871A1 (zh) 数据处理方法、设备及存储介质
WO2024007970A1 (zh) 线程调度方法及电子设备
WO2023124225A1 (zh) 帧率切换方法及装置
CN116069187B (zh) 一种显示方法及电子设备
CN115831074B (zh) 基于单缓存模式的帧率转换方法及装置
CN115841804B (zh) 分辨率实时切换控制方法及装置
CN111258415B (zh) 基于视频的肢体动作检测方法、装置、终端和介质
CN117991961A (zh) 数据处理方法、设备及存储介质
CN115269485A (zh) 数据处理方法、多核多系统模块和电子设备
CN115604540A (zh) 视频获取方法及装置