TW202046741A - A method for performing client side latency enhancement, a host processor and a processing circuit thereof - Google Patents

Abstract

A method and apparatus for performing client side latency enhancement with aid of cloud game server side image orientation control. The method may include: utilizing at least one orientation-related sensor in a client device to detect a client device orientation of the client device; utilizing a network interface circuit in the client device to notify a cloud game server of the client device orientation; utilizing the cloud game server to rotate video contents of a cloud game in advance for the client device according to the client device orientation to generate pre-rotated video contents, for being output to the client device through video streaming; and utilizing the network interface circuit to receive the pre-rotated video contents through the video streaming and buffering the pre-rotated video contents in a video buffer, for being output to a display panel and displayed on the display panel.

Description

Client side delay improvement method, main processor and processing circuit

The present invention generally relates to image display, and more specifically, to a method and device for performing client-side delay enhancement by means of cloud game server-side image orientation control, wherein the device may include At least a part (for example, part or all) of the electronic device, for example, the main processor, processing circuit, etc. in the electronic device.

Modern games usually require the huge GPU computing power of an image processing unit (GPU) to obtain excellent and complete scene rendering. More specifically, due to specific types of rendering tasks on the GPU, a heavier load on the GPU will be introduced. When the game is designed as a cloud game, the cloud game server may prepare the video content of the cloud game and send the video content to the client device (for example, a multi-function mobile phone) through the video stream, so as to be played by the client device. However, this can cause several problems. For example, for game playing in landscape mode, the client device may need to rotate the cached video data to match the display direction, and thus may increase more delay and bandwidth (BW) consumption. Therefore, a novel method and associated architecture is needed to improve the overall performance of electronic devices running cloud games.

In order to solve the above-mentioned problems, the objective of the present invention is to provide a method for improving client-side delay through image direction control on a cloud game server, and to provide related devices such as electronic devices or main processors and processing circuits in electronic devices. , And cloud game server.

Another object of the present invention is to provide a method for performing client-side delay improvement by means of cloud game server-side image direction control, and to provide related devices such as electronic devices or main processors and processing circuits in electronic devices, and cloud Game server to improve the overall performance of electronic devices.

At least one embodiment of the present invention provides a method for improving client-side delay, using cloud game server-side image direction control. The method includes: using at least one direction-related sensor of the client device to detect the direction of the client device of the client device; using the network interface circuit in the client device to notify the cloud game server of the direction of the client device; using the cloud According to the direction of the client device, the game server generates pre-rotation video content of the cloud game for the client device in advance for output to the client device through a video stream, wherein the pre-rotation video content corresponds to the direction of the client device; and The network interface circuit receives the pre-rotated video content through the video stream, and caches the pre-rotated video content in the video register for output to and display on the display panel.

At least one embodiment of the present invention provides a main processor that is located in an electronic device and performs client-side delay improvement by means of cloud game server-side image direction control. The main processor includes a core circuit, a display interface circuit coupled to the core circuit, and a bus interface circuit. The core circuit is used to control the main processor for controlling the operation of the electronic device, wherein the electronic device is used as a client device, and under the control of the core circuit, the main processor performs interactive control of the client device, With the help of cloud game server-side image direction control to implement client-side delay improvement. In addition, the display interface circuit is used to couple the display panel to the main processor; and the bus interface circuit is used to couple at least one element to the main processor through the bus, wherein the at least one element includes the client device In at least one direction related sensor. For example, the main processor uses the at least one direction-related sensor to detect the direction of the client device of the client device; the main processor uses the network interface circuit in the client device to notify the cloud game server of the direction of the client device ; The main processor uses the cloud game server to generate pre-rotation video content of the cloud game for the client device in advance according to the direction of the client device for output to the client device through a video stream, wherein the pre-rotation video content pair The direction of the client device; and the host processor uses the network interface circuit to receive the pre-rotated video content through the video stream, and caches the pre-rotated video content in the video register for output to the display panel and Display on this display panel.

According to many embodiments, the present invention further provides a processing circuit including the above-mentioned main processor, wherein the processing circuit further includes: the at least one direction-related sensor; and the network interface circuit. For example, the processing circuit may include the video register, and the video register is used for buffering frame information, wherein the frame information includes the pre-rotated video content. In addition, the video register is implemented as an internal register or an external register of the main processor.

According to many embodiments, the present invention further provides an electronic device including the above-mentioned main processor, wherein the electronic device may include a processing circuit and a display panel. The processing circuit may include the main processor, the at least one direction-related sensor, and the network interface circuit. The display panel is coupled to the main processor for displaying information. For example, the processing circuit may include the video register, and the video register is used for buffering frame information, wherein the frame information includes the pre-rotated video content. In addition, the video register is implemented as an internal register or an external register of the main processor.

At least one embodiment of the present invention provides a cloud game server for performing client-side delay improvement by means of image direction control on the cloud game server side. The cloud game server may include at least one processor (for example, one or more processors) and a network interface circuit coupled to the at least one processor. At least one processor may be arranged to control the operation of the cloud game server, wherein, under the control of the at least one processor, the cloud game server performs game control and selectively executes image rotation of the client device to use the cloud The game server side image direction control implements the client side delay improvement. In addition, the network interface circuit can be arranged to couple the cloud game server to the client device through at least one network. For example, at least one direction-related sensor in the client device is configured to detect the direction of the client device of the client device; the client device is configured to notify the cloud game server of the direction of the client device; the cloud game server is configured according to The direction of the client device generates pre-rotation video content of the cloud game for the client device in advance for output to the client device through a video stream, wherein the pre-rotation video content corresponds to the direction of the client device; and the client device is configured to pass through The video stream receives the pre-rotated video content, and caches the pre-rotated video content in a video register for output to and display on the display panel.

The method and related devices of the present invention (for example, the main processor, processing circuit, etc. in the electronic device) can appropriately control the operation of the electronic device, and more particularly, can request the cloud game server to execute the video content of the cloud game for the client device in advance Image rotation, and offload the image rotation task from the client device to the cloud game server, thereby improving the overall performance of the electronic device. For example, for a game played in landscape mode, the client device does not need to rotate the cached video data to match the display direction, and thus can reduce latency and bandwidth (BW) consumption. In addition, when solving the problems of the prior art, the implementation of the embodiments of the present invention will not greatly increase additional costs. Compared with the traditional structure, the present invention can achieve the best performance of the electronic device without introducing any side effects or being unlikely to introduce side effects.

Other embodiments and advantages are described in the detailed description below. This summary is not intended to limit the invention. The present invention is limited by the scope of patent application.

Certain words are used in the specification and the scope of the patent application to refer to specific elements. Those skilled in the art should understand that hardware manufacturers may use different terms to refer to the same component. This specification and the scope of the patent application do not use differences in names as a way to distinguish elements, but use differences in functions of elements as a criterion. The "include" mentioned in the entire specification and the scope of patent application is an open term, so it should be interpreted as "include but not limited to". In addition, the term "coupling" here includes any direct and indirect electrical connection means. Therefore, if a device is described in the text as being coupled to another device, it means that the device can be directly electrically connected to the other device, or indirectly electrically connected to the other device through other devices or connecting means.

Fig. 1 is a schematic diagram of an electronic device 100 according to a first embodiment of the present invention, in which a cloud game server 10 is also described for better understanding. Examples of the electronic device 100 may include, but are not limited to, multifunctional mobile phones, tablet computers, wearable devices, all-in-one computers, and notebook computers. The electronic device 100 can be used as a client device of a server (for example, the cloud game server 10) to provide one or more services to users, and in particular, the electronic device 100 can be used as a game client of the cloud game server 10 Device to allow users to play one or more cloud games. As shown in the upper right corner of Figure 1, the cloud game server 10 may include a bus 10B and at least one processor (for example, one or more processors) that may be collectively referred to as a processor 11, and the cloud game server 10 Can include image processing unit (GPU) 12, random access memory (RAM) (for example, dynamic RAM (DRAM) 13), network interface circuit 14 (marked as "Network IF CKT" in the first figure for simplicity) , And the video codec 15 (marked as "VENC" in the first figure for simplicity), in which the processor 11, GPU 12, DRAM 13, network interface circuit 140 and video codec 15 can communicate with each other through the bus 10B Coupling. For example, at least one storage device (not shown in Figure 1) of the cloud game server 10 can be arranged to store some program codes 11P. Examples of the code 11P may include, but are not limited to, an operating system (OS), one or more drives, and one or more applications. The processor 11 can load the program code 11P from the storage device, and run the program code 11P to control the operation of the cloud game server 10. Under the control of the processor 11 running the program code 11P, the cloud game server 10 can perform game control, and more particularly, can perform image fixation control for the client device (for example, the electronic device 100) to use the cloud game server Image fixation control on the device side to perform client-side delay optimization (or delay reduction).

As shown in Figure 1, the electronic device 100 may include a processing circuit 105 and a display panel 150 coupled to the processing circuit 105, and the processing circuit 105 may include a bus 105B and a main processor 110 (for example, a central processing unit (CPU)) And at least one direction-related sensor (for example, one or more direction-related sensors) that can be collectively referred to as the direction-related sensor 120, and may include RAM (for example, DRAM 130) and a network interface circuit 140 (for For simplicity, it is labeled "Network IF CKT" in Figure 1). In particular, the main processor 110 may include a core circuit 112, a display interface circuit 114 (labeled as "Display IF CKT" in Figure 1 for simplicity), and a bus The interface circuit 116, and the display panel 150 may include a display interface circuit 152 (labeled as "Network IF CKT" in Figure 1 for simplicity), a display controller 154, and a display module (for example, a liquid crystal display (LCD) module 156) ), where the main processor 110, the direction-related sensor 120, the DRAM 130, and the network interface circuit 140 can be coupled to each other through the bus 105B, and the display interface circuit 114 and the bus interface circuit 116 can be as shown in Figure 1. The illustrated method is coupled to the core circuit 112, but the present invention is not limited to this. According to some embodiments, the structure of the electronic device 100 shown in FIG. 1 can be varied. For example, a display module (for example, LCD module 156) and a touch sensing module (not shown) can be integrated in the same module to form a touch sensing display device (for example, touch screen), and the touch sensing display The device may include a touch controller for performing touch control to detect user input through the touch sensing module. In addition, the network interface circuit 14 of the cloud game server 10 and the network interface circuit 140 of the electronic device 100 can be arranged to couple the cloud game server 10 and the electronic device 100 to each other through at least one network. For example, the network interface circuit 140 may include a plurality of wireless communication circuits (for example, Wi-Fi circuits, mobile network circuits, etc.) that comply with several communication standards for performing wireless communication of the electronic device 100.

The processing circuit 105 (for example, the main processor 110) and the display panel 150 can be coupled to each other through the display interface circuits 114 and 152, and the display interface circuits 114 and 152 can be implemented as interface circuits compliant with specific standards. For example, the specific standard may be the MIPI Display Sequence Interface (DSI) standard of the Mobile Industry Processor Interface (MIPI) Alliance, and the display interface circuits 114 and 152 may be implemented as DSI circuits. Therefore, the main processor 110 (for example, the core circuit 112) can communicate with the display panel 150 for performing associated control of the electronic device 100. In addition, the electronic device 100 may further include additional circuits, such as a power management circuit, a storage interface circuit, etc. (not shown), to provide the electronic device 100 with the ability to perform associated operations (eg, power management, storage interaction, etc.) . In addition, the main processor 110 (for example, the core circuit 112) may control various operations of the electronic device 100. For example, many program codes 112P running on the main processor 110 (for example, the core circuit 112) can control the electronic device 100 so that the electronic device 100 is equipped with various functions. Examples of the program code 112P may include, but are not limited to, the OS, one or more drivers, and one or more applications.

According to this embodiment, the main processor 110 can be arranged to perform interactive control on cloud games. In particular, the core circuit 112 may be arranged to control the main processor 110 for controlling the operation of the electronic device 100. Under the control of the core circuit 112, the main processor 110 can perform display control of the electronic device 100. For example, the main processor 110 (for example, the core circuit 112) may set the refresh rate of the display interface circuit 114 as the target refresh rate in advance, for controlling the main processor 110 to output images to the display panel 150 according to the preset target refresh rate. And when there is demand, the refresh rate can be adjusted dynamically. The display interface circuit 114 can be arranged to control the output of the image from the frame register (for example, the video register 132 in the DRAM 130) to the display panel 150 The timing of the data, but the invention is not limited to this. In addition, the display interface circuits 114 and 152 can be arranged to couple the display panel 150 and the main processor 110, and send one or more commands and image data from the main processor 110 to the display panel 150. Therefore, the display panel 150 can display information (for example, image data) of the electronic device 100.

One or more applications of the program code 112P may include a client application (APP) of a cloud game. The main processor 110 (for example, the core circuit 112) may run a client APP of the cloud game to perform interactive control of the cloud game. For example, the cloud game server 10 may send at least one video stream of the cloud game to the client device (for example, the electronic device 100). Under the control of the main processor 110 (for example, the core circuit 112) running the client APP, the electronic device 100 can receive and play the video stream of the cloud game for the user. The video stream can carry the image data of the cloud game (for example, , Image frame) and associated audio data. The main processor 110 (for example, the core circuit 112) running the client APP can use the touch sensor module or the touch screen of the electronic device 100 to detect user control, such as user touch behavior (for example, the user uses his finger in the touch sensor module Click, drag, etc. on the group or touch screen), and use the network interface circuit 140 to send the user control information controlled by the user to the cloud game server 10 through the network. Therefore, in response to user control, the cloud game server 10 can change the image data (eg, image frame) and associated audio data of the cloud game. In addition, the main processor 110 (for example, the core circuit 112) running the client APP can use the direction-related sensor 120 to perform direction-related sensing of the electronic device 100, thereby generating a plurality of direction detection results at a plurality of time points, and using The network interface circuit 140 sends the detection results of various directions at a plurality of time points to the cloud game server 10 as real-time feedback of the direction of the game client. Among them, the detection results of the plurality of directions can indicate the client device (for example, electronic Device 100) client device orientation.

As shown in Figure 1, a display interface circuit 114 can be arranged to couple the display panel 150 to the main processor 110, and a bus interface circuit 116 can be arranged to couple at least one component to the main processor 110 through the bus 105B. At least one component may include a direction-dependent sensor 120, a DRAM 130, and a network interface circuit 140. The DRAM 130 may include a video register 132, and therefore may be regarded as an external register of the main processor 110, but The present invention is not limited to this. According to many embodiments, the video register 132 may be integrated into the main processor 110, and in particular, may be implemented as an internal register of the main processor 110. In addition, the bus interface circuit 116 may comply with a specific communication standard to allow the main processor 110 and at least one of the above-mentioned components (for example, the direction-dependent sensor 120 and the DRAM 130 shown in FIG. 1 and the video register 132, Used for communication when the video register 132 is used as an external register of the main processor 110). For example, the direction-related sensor 120, the DRAM 130, and the network interface circuit 140 can work under the control of the main processor 110, respectively. In addition, the video register 132 can be one of a plurality of cache areas in the RAM (for example, the DRAM 130). For example, the video register 132 can be arranged to buffer frame information.

According to many embodiments, when there is a demand, the circuits in the main processor 110 shown in FIG. 1 can be varied. For example, the display interface circuit 114 and the bus interface circuit 116 can be coupled to each other through direct connection. In another example, in the main processor 110, one or more other components (for example, a frame register for temporarily storing image data to be output to the display panel 150) can be inserted into the display interface circuit 114 and The bus interface circuit 116, and the display interface circuit 114 and the bus interface circuit 116 can be coupled to each other through one or more other components.

According to many embodiments, the direction-related sensor 120 may include one or a combination of a gyroscope, a magnetometer, and a camera. For example, according to one or more sensing results (for example, the latest sensing result and historical sensing result) of the direction-related sensor 120, the main processor 110 (for example, the core circuit 112) running the client APP can perform sensor fusion, To generate a plurality of direction detection results, but the present invention is not limited to this. For another example, the main processor 110 (for example, the core circuit 112) running the client APP can perform image detection with the help of a camera. In particular, it can obtain the user's face image from the camera and execute the use based on the face image. The face of the person is detected to generate multiple direction detection results.

Figure 2 is a flowchart of a method for performing client-side delay optimization by means of cloud game server-side image direction control according to an embodiment of the present invention. This method can be applied to the electronic device 100 and the cloud game server 10 shown in FIG. 1, and in particular, it can be applied to the main processor 110 in the structure of the electronic device 100 (for example, the core running the program code 112P The circuit 112) and related components (for example, the direction-related sensor 120, the DRAM 130 and the network interface circuit 140) are also applied to the processor 11 and related components running the code 11P in the cloud game server 10 architecture (For example, GPU 12, DRAM 13, network interface circuit 14, and video transcoder 15).

In step S10, the main processor 110 may use the direction-related sensor 120 to detect the direction of the client device (for example, the electronic device 100). According to this embodiment, the main processor 110 can obtain sensing information from the direction-related sensor 120, and in particular, can collect various information from the direction-related sensor 120 (for example, motion, position, location, and environment sensors). Sensor information fragments, and can perform calculations (for example, sensor fusion) at least based on sensor information to determine the direction of the client device, where the sensor information can include motion sensor information, position sensor information, location sensor information, and One or a combination of environmental sensing information. For example, the user may hold the client device (for example, the electronic device 100) according to one of a plurality of predetermined client device directions (for example, one of a plurality of horizontal directions or one of a plurality of vertical directions), and the main processor 110 may detect The latest direction of the electronic device 100.

In step S12, the main processor 110 may use the network interface circuit 140 in the client device (for example, the electronic device 100) to notify the cloud game server 10 of the direction of the client device. According to this embodiment, the main processor 110 may use the network interface circuit 140 to send a direction index to the cloud game server 10 to notify the cloud game server 10 of the direction of the client device, wherein the direction index may indicate the direction of the client device .

In step S13, according to the direction of the client device (for example, the direction index received from the electronic device 100), the cloud game server 10 (for example, the processor 11 running the code 11P) may determine whether to rotate one or more cloud games Image frame. If the answer is yes, go to step S14; if the answer is no, go to step S24.

In step S14, the main processor 110 may use the cloud game server 10 to rotate the video content (for example, one or more image frames) of the cloud game for the client device (for example, the electronic device 100) in advance according to the direction of the client device. Thus, the pre-rotation video content of the cloud game is generated, for example, the pre-rotation video content corresponding to the direction of the client device is output to the client device through the video stream. For example, during the video streaming, the cloud game server 10 can encode the pre-rotated video content by using the video transcoder 15 to generate the encoded result of the pre-rotated video content.

In step S16, the main processor 110 may use the network interface circuit 140 to receive the pre-rotated video content through the video stream, and cache the pre-rotated video content in the video register 132 without rotating the video register 132 The cached pre-rotated video content is used for output to and display on the display panel 150. For example, during the video stream, the client device (for example, the electronic device 100) can decode the encoded result (for example, by using the main processor 110) to obtain the pre-rotated video content.

In step S24, without rotating the video content of the cloud game for the client device (for example, the electronic device 100) in advance, the main processor 110 can use the cloud game server 10 to prepare (for example, generate) cloud game content according to the direction of the client device. The video content is used to output to the client device through the video stream. For example, during the video streaming, the cloud game server 10 may encode the video content by using the video codec 15 to generate the encoded result of the video content.

In step S26, the main processor 110 may use the network interface circuit 140 to receive the video content through the video stream, and buffer the video content in the video register 132 for output to the display panel 150 and display on the display panel 150 . For example, during the video streaming, the client device (for example, the electronic device 100) can decode the encoded result (for example, by using the main processor 110) to obtain the video content.

According to this embodiment, some operations in the process shown in Fig. 2 can be repeated, wherein the latest decision result of step S13 can guide the execution of the associated operation corresponding to the latest decision result. As time passes, the user can change the way of holding the electronic device 100, and the main processor 110 can generate the latest direction detection result (for example, the latest direction index) indicating the latest direction of the electronic device 100 in response to the latest direction detection result (For example, the latest direction index) makes the electronic device 100 enter a sub-process of the process, for example, the sub-process of steps S13, S14, and S16 or the sub-process of steps S13, S24, and S26.

For better understanding, this method can be described as the process shown in Figure 2, but the present invention is not limited to this. According to some embodiments, one or more steps can be added, deleted, or changed in the process shown in Figure 2.

According to some embodiments, regardless of whether the video content (for example, one or more video frames) mentioned in step S14 is first generated, the cloud game server 10 can provide the client device (for example, the electronic device 100) in advance according to the direction of the client device. ) Generate pre-rotation video content for cloud games, and complete the rotation on the server side to generate pre-rotation video content corresponding to the direction of the client device. The above content can be achieved in the following ways: 1) Use the GPU 12 of the cloud game server 10 to directly draw the pre-rotated video content corresponding to the direction of the client device, or 2) First generate the video content of the cloud game (for example, one or more images) Image frame) and then rotate the video content, where the rotator (ROT) in the cloud game server 10 (for example, a specific processor of the cloud game server 10 (for example, the main processor 110 or the GPU 12) runs on The program module (for example, ROT module) or the dedicated hardware circuit (for example, ROT circuit) of the cloud game server 10 can perform the operation of rotating these video contents, but the present invention is not limited to this. In addition, regarding the flow shown in FIG. 2, the operation of step S14 can be various. For example, during the period when the cloud game server 10 is used to generate pre-rotation video content of the cloud game for the client device (for example, the electronic device 100) in advance according to the direction of the client device, the main processor 110 may use the cloud game server 10 (for example, the GPU 12). ) Directly generate the pre-rotated video content corresponding to the direction of the client device, without first generating the video content of the cloud game (for example, one or more image frames), and then rotating the video content.

Fig. 3 describes the control scheme of the method shown in Fig. 2 according to an embodiment of the present invention. For better understanding, it also describes a situation with extra rotation to increase more delay and bandwidth (BW) consumption. As shown in the left half of Figure 3, the source points (simplified as "O") of the coordinate planes of the image frame prepared on the cloud game server side and the image frame displayed on the client side are different from each other , Where the eye symbol can indicate the direction the user looks at the screen of the game client device. For games played in landscape mode, the game client device usually needs to rotate the register (especially, the cached video content of the register, such as all the image frames cached in the register) to match the display direction, and As a result, more delay and bandwidth consumption are added (simplified labeled "additional rotation adds more delay and BW"). For example, when rotating the register (especially the cached video content of the register), the game client device must perform more operations, for example, more register access (for example, read and write) operations, This causes more delays in game control and more bandwidth consumption of one or more internal data paths of the game client device (for example, one or more input/output (I/O) paths of the register). Therefore, there may be many game problems, such as video lagging behind audio, user-controlled delayed image display, etc.

As shown in the right half of Figure 3, based on the control scheme of this embodiment, the source points of the image frames prepared on the cloud game server side and the image frames displayed on the client side (simplified Marked as "O") is the same, where the eye symbol can indicate the direction in which the user views the screen operation of the game client device (working according to this method (for example, the electronic device 100)). For games played in landscape mode, the game client device does not need to rotate the register (for example, the video register 132) (especially, the buffered video content of the register, such as any image information buffered by the register) Box), and therefore reduces latency and bandwidth consumption (simplified labelled as "reduced latency and BW"). For example, in step S31, the game client device may send real-time feedback about the game client's direction (for example, the latest direction detection result (for example, the latest direction index)) to the cloud game server 10 (labeled as “direction feedback” for simplicity). During game rendering and encoding, the cloud game server 10 can use the GPU 12 to perform rendering to generate cloud game video in the game rendering register dedicated to game rendering of the cloud game (for example, the cache area in DRAM 13) Content, and more specifically, in step S32, for example, the processor 11 or GPU 12 may be used to pre-rotate the video content in the game rendering register (for example, the above at least one video stream according to the feedback of the game client device). Video stream content) (labeled as "pre-rotation" for simplicity), and the video codec 15 can be further used to encode the video content to generate the encoded result of the video content. Since the video content should match the direction of the game client device, in step S33, the game client device can directly play the video content without additional rotation (for simplicity, it is labeled "Play without additional rotation"). Therefore, the delay and bandwidth consumption can be reduced.

According to some embodiments, during the game rendering of the cloud game, the cloud game server 10 can use the GPU 12 to directly generate the pre-rotated video content of the cloud game in the game rendering register in response to the game client's direction (for example, the latest Real-time feedback of orientation detection results (for example, the latest orientation index), where the pre-rotated video content corresponds to the latest client device orientation, but the invention is not limited to this.

Fig. 4 describes many implementation details of the control scheme shown in Fig. 3 according to an embodiment of the present invention, where the 90-degree direction can be taken as a plurality of lateral examples. Taking the Android OS as an example of the OS with the code 112P, the low-level software module in the multi-layer software module in the code 112P can provide the function of obtaining the current screen orientation, as shown below:

//Get current screen orientation Display display = ((WindowManager) getSystemService(WINDOW_SERVICE)).getDefaultDisplay(); int orientation = display.getOrientation();

The present invention is not limited to this. For example, for other types of OS, the functions can be diverse.

In step S40, the client APP can obtain the current screen orientation from the lower layer of the software module (marked as "Android" in Figure 4 for simplicity). Under the control of the main processor 110 (for example, the core circuit 112) running the client APP, in step S41, the game client device (for example, a client device similar to the electronic device 100) may generate a direction hint to indicate that the current screen direction is a game Real-time feedback of the client's direction (for example, the latest direction detection result similar to the latest direction index), and a direction hint is sent to the cloud game server 10 in step S42. In addition, in step S43, the cloud game server 10 may generate rotated game content such as pre-rotated video content according to the direction hint (for example, by using the GPU 12 to generate the video content and then rotate it to generate the rotated game content), To generate pre-rotated video content (labeled "ROT" for better understanding), used for encoding by the video transcoder 15 (labeled "VENC" for simplicity), and can send at least one of the above to the game client device in step S44 The video stream (for example, the video stream carrying the rotated game content in this embodiment) is used for direct playback on the client side.

According to many embodiments, the operation of step S43 can be varied. For example, during the generation of the pre-rotated video content based on the direction hint, the cloud game server 10 can use the GPU 12 to directly generate the pre-rotated video content corresponding to the current screen orientation.

Fig. 5 describes several implementation details of the control scheme shown in Fig. 3 according to another embodiment of the present invention, in which the 270 degree direction can be taken as another example of plural lateral directions. Except that the 270-degree direction is used instead of the 90-degree direction, the operations of steps S40-S44 are almost the same as those of the embodiment shown in FIG. 4. For simplicity, similar description details of this embodiment are not repeated.

According to many embodiments, at the first moment, the client device direction mentioned in step S10 may represent the client device's first client device direction (for example, a 90-degree direction), and therefore respond to the latest direction detection result (for example, the latest direction index). ), the operations of steps S14 and S16 can be performed. At the second moment, in step S10, the main processor 110 may use the direction-related sensor 120 to detect the second client device direction (for example, a 270 degree direction) of the client device, wherein, for example, the client device of the electronic device 100 may have been The rotation of the first client device direction is the second client device direction. Then, in step S12, the main processor 110 may use the network interface circuit 140 to notify the cloud game server 10 of the direction of the second client device. Therefore, the sub-flow of steps S13, S14 and S16 is entered. More specifically, the main processor 110 may use the cloud game server 10 to rotate other video content of the cloud game (for example, corresponding to the second moment or from the second client device) in advance for the client device (for example, the electronic device 100) according to the direction of the second client device. The second video content of the second time period starting from the time, instead of the first video content corresponding to the first time or the first time period between the first time and the second time), to generate other pre-rotated video content of the cloud game (For example, the second pre-rotation video content corresponding to the second time or the second time period, rather than the first pre-rotation video content corresponding to the first time or the first time period), used to output to the client device through the video stream, Wherein, other pre-rotated video content corresponds to the direction of the second client device. In addition, the main processor 110 can use the network interface circuit 140 to receive other pre-rotated video content through the video stream, and cache the other pre-rotated video content in the video buffer 132 for output to the display panel 150 and on the display panel 150 for display. Taking the embodiments shown in Figures 4 and 5 as examples, the first client device direction and the second client device direction of these embodiments may belong to a plurality of lateral directions of the client device, and may be at different times (for example, the first One moment and second moment) are detected.

According to many embodiments, the above operations of using the cloud game server 10 to rotate other video content of the cloud game for the client device (for example, the electronic device 100) in advance according to the direction of the second client device to generate other pre-rotated video content of the cloud game can be multiple Diverse. For example, the cloud game server 10 is used to generate other pre-rotation video content of the cloud game for the client device (for example, the electronic device 100) in advance according to the direction of the second client device (for example, the second time corresponding to the second time or the second time period). During the pre-rotation video content), the main processor 110 can use the cloud game server 10 (for example, GPU 12) to directly generate other pre-rotated video content corresponding to the direction of the second client device without first generating other video content of the cloud game (for example, , Corresponding to the second time or the second video content of the time period starting at the second time) and rotate it.

Fig. 6 describes many implementation details of the control scheme shown in Fig. 3 according to another embodiment of the present invention, in which the 0 degree direction can be taken as an example of a plurality of longitudinal directions. Except that the 0-degree direction is used instead of the 90-degree direction and the cloud game can be changed to run in the portrait mode, the operations of steps S40-S42 and S44 are almost the same as those of the embodiment shown in FIG. 4. Since the source points of the coordinate planes of the image frame prepared on the cloud game server side and the image frame displayed on the client side (simplified as "O") are the same as each other, there is no need to install it on the cloud game side. Prepare the image frame, and therefore use step S43' instead of step S43 to correspond to this situation. For example, as shown in step S24, the cloud game server 10 can use the GPU 12 to generate unrotated game content such as video content according to the direction hint in step S43, and use the video codec 15 for encoding (simply labeled "VENC" ), and the above at least one video stream (for example, a video stream carrying non-rotating game content in this embodiment) may be sent to the game client device in step S44 for direct playback on the client side. For simplicity, similar description details of this embodiment are not repeated.

Fig. 7 describes many implementation details of the control scheme shown in Fig. 3 according to another embodiment of the present invention, in which the 180-degree direction can be taken as another example of plural longitudinal directions. Except that the 180-degree direction is used instead of the 90-degree direction and the cloud game can be changed to run in the portrait mode, the operations of steps S40-S44 are almost the same as those of the embodiment shown in FIG. 4. For simplicity, similar description details of this embodiment are not repeated.

According to many embodiments, at the first moment, the direction of the client device mentioned in step S10 may represent the first client device direction of the client device (for example, a 180 degree direction), and therefore respond to the latest direction detection result (for example, the latest direction index) ), the operations of steps S14 and S16 can be performed. At the second moment, in step S10, the main processor 110 may use the direction-related sensor 120 to detect the second client device direction (for example, 0 degree direction) of the client device, wherein, for example, the client device of the electronic device 100 may have been The rotation of the first client device direction is the second client device direction. Then, in step S12, the main processor 110 may use the network interface circuit 140 to notify the cloud game server 10 of the direction of the second client device. Therefore, the sub-flow of steps S13, S24 and S26 is entered. More specifically, the main processor 110 can use the cloud game server 10 to prepare (for example, generate) other video content of the cloud game according to the direction of the second client device (for example, corresponding to the second time or the second time from the second time). Segment of the second video content, instead of the first video content corresponding to the first time or the first time period between the first time and the second time), there is no need to rotate the cloud game for the client device (for example, the electronic device 100) in advance Other video content is used to output to client devices through video streams. In addition, the main processor 110 can use the network interface circuit 140 to receive other video content through the video stream, and buffer the other video content in the video buffer 132 for output to the display panel 150 and display on the display panel 150 . Taking the embodiments shown in FIGS. 6 and 7 as examples, the first client device orientation and the second client device orientation of these embodiments may belong to a plurality of longitudinal directions of the client device, and may be at different times (for example, the first One moment and second moment) are detected.

FIG. 8 is a schematic diagram of an electronic device according to another embodiment of the invention. Compared with the structure shown in FIG. 1, the internal memory 118 is inserted between the display interface circuit 114 and the bus interface circuit 116 to replace the aforementioned frame buffer to temporarily store image data to be output to the display panel 150. In response to structural changes, many labels were changed accordingly. For example, the main processor 210, the processing circuit 205, and the electronic device 200 may be used to replace the above-mentioned main processor 110, the processing circuit 105, and the electronic device 100, respectively. For simplicity, similar description details of this embodiment are not repeated.

Although the present invention has been described through examples and according to preferred embodiments, it is to be understood that the present invention is not limited thereto. Those familiar with the technology can still make various changes and modifications without departing from the scope and spirit of the present invention. Therefore, the scope of the present invention should be limited and protected by the scope of the attached patent application and its equivalents. The foregoing descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention shall fall within the scope of the present invention.

100,200: electronic device 105,205: processing circuit 105B: Bus 110, 210: main processor 112: core circuit 112P: Code 114: display interface circuit 116: bus interface circuit 120: Orientation related sensor 130: DRAM 132: Video buffer 140: network interface circuit 150: display panel 152: display interface circuit 154: display controller 156: LCD module 10: Cloud game server 11: processor 12: GPU 13: DRAM 14: Network interface circuit 15: Video codec 10B: Bus 11P: Code 118: Internal memory S31, S32, S33, S40, S41, S42, S43, S44, S43’, S10, S12, S13, S14, S16, S24, S26: steps

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is described by way of example rather than limitation, wherein the same reference numerals identify similar elements. It should be noted that different references to one embodiment in the present invention are not necessarily the same embodiment, and such references mean at least one. In addition, when describing a particular feature, structure, or characteristic in combination with an embodiment, whether or not it is explicitly described, it is recommended that those skilled in the art know the feature, structure, or characteristic in combination with other embodiments. Figure 1 is a schematic diagram of an electronic device according to a first embodiment of the present invention, in which a cloud game server is also described for better understanding. Figure 2 is a flowchart of a method for performing client-side delay optimization by means of cloud game server-side image direction control according to an embodiment of the present invention. Figure 3 illustrates the control scheme of the method shown in Figure 2 according to an embodiment of the present invention. Figure 4 describes many implementation details of the control scheme shown in Figure 3 according to an embodiment of the present invention. Figure 5 describes several implementation details of the control scheme shown in Figure 3 according to another embodiment of the present invention. Figure 6 illustrates many implementation details of the control scheme shown in Figure 3 according to another embodiment of the present invention. Figure 7 depicts many implementation details of the control scheme shown in Figure 3 according to another embodiment of the present invention. FIG. 8 is a schematic diagram of an electronic device according to another embodiment of the invention.

S10, S12, S13, S14, S16, S24, S26: steps

Claims (10)

A client-side delay improvement method, with the help of cloud game server-side image direction control, the method includes: Using at least one direction-related sensor of the client device to detect the direction of the client device of the client device; Use the network interface circuit in the client device to notify the cloud game server of the direction of the client device; Using the cloud game server to generate pre-rotation video content of the cloud game for the client device in advance according to the direction of the client device for output to the client device through a video stream, wherein the pre-rotation video content corresponds to the direction of the client device; as well as The network interface circuit is used to receive the pre-rotated video content through the video stream, and the pre-rotated video content is cached in a video register for output to and display on the display panel. According to the client-side delay improvement method described in claim 1, wherein the step of using the at least one direction-related sensor of the client device to detect the direction of the client device of the client device further includes: Obtain sensing information from the at least one direction-related sensor, where the sensing information includes one or a combination of motion sensing information, location sensing information, position sensing information, and environment sensing information; and At least according to the sensing information, a calculation is performed to determine the direction of the client device. According to the client-side delay improvement method described in claim 1, wherein the step of using the network interface circuit in the client device to notify the cloud game server of the direction of the client device further includes: The network interface circuit is used to send a direction index to the cloud game server to notify the cloud game server of the direction of the client device, wherein the direction index indicates the direction of the client device. According to the client-side delay improvement method described in claim 1, wherein, during the video stream, the cloud game server encodes the pre-rotated video content to generate an encoded result of the pre-rotated video content, and the The client device decodes the encoded result to obtain the pre-rotated video content. According to the client-side delay improvement method described in claim 1, wherein the step of using the cloud game server to generate the pre-rotated video content of the cloud game for the client device in advance according to the direction of the client device further includes : Use the cloud game server to rotate the video content of the cloud game for the client device in advance according to the direction of the client device, thereby generating the pre-rotated video content to output to the client device through the video stream; or use the cloud game server The pre-rotation video content corresponding to the direction of the client device is directly generated to output to the client device through the video stream. The client-side delay improvement method described in claim 1, wherein the network interface circuit is used to receive the pre-rotated video content through the video stream and cache the pre-rotated video content in the video register The steps further include: Use the network interface circuit to receive the pre-rotated video content through the video stream, and cache the pre-rotated video content in the video register without rotating the pre-rotated video content cached in the video register. It is output to the display panel and displayed on the display panel. According to the client-side delay improvement method described in claim 1, wherein the client device direction represents the first client device direction of the client device, and the method further includes: Using the at least one direction-related sensor of the client device to detect the second client device direction of the client device, wherein the client device has rotated from the first client device direction to the second client device direction; Using the network interface circuit in the client device to notify the cloud game server of the direction of the second client device; The cloud game server is used to generate other pre-rotated video content of the cloud game for the client device in advance according to the direction of the second client device for outputting to the client device through the video stream, wherein the other pre-rotated video content Correspond to the direction of the second client device; and The network interface circuit is used to receive the other pre-rotated video content through the video stream, and the other pre-rotated video content is cached in the video register for output to and display on the display panel. According to the client-side delay improvement method described in claim 1, wherein the client device direction represents the first client device direction of the client device, and the method further includes: Using the at least one direction-related sensor of the client device to detect the second client device direction of the client device, wherein the client device has rotated from the first client device direction to the second client device direction; Using the network interface circuit in the client device to notify the cloud game server of the direction of the second client device; Use the cloud game server to prepare other video content of the cloud game according to the direction of the second client device, and do not rotate the other video content of the cloud game for the client device in advance, for outputting to the client device through the video stream ;as well as The network interface circuit is used to receive the other video content through the video stream, and the other video content is cached in the video register for output to and display on the display panel. A main processor is located in an electronic device and performs client-side delay improvement with the help of cloud game server-side image direction control. The main processor includes: The core circuit is used to control the main processor and control the operation of the electronic device, wherein the electronic device is used as a client device, and under the control of the core circuit, the main processor performs interactive control of the client device to Use cloud game server-side image direction control to implement client-side delay improvement; A display interface circuit, coupled to the core circuit, for coupling the display panel to the main processor; and A bus interface circuit, coupled to the core circuit, for coupling at least one element to the host processor through the bus, wherein the at least one element includes at least one direction-related sensor in the client device; among them: The main processor uses the at least one direction-related sensor to detect the direction of the client device of the client device; The main processor uses the network interface circuit in the client device to notify the cloud game server of the direction of the client device; The main processor uses the cloud game server to generate pre-rotation video content of the cloud game for the client device in advance according to the direction of the client device for output to the client device through a video stream, wherein the pre-rotation video content corresponds to Client device orientation; and The main processor uses the network interface circuit to receive the pre-rotated video content through the video stream, and caches the pre-rotated video content in a video register for output to and display on the display panel . A processing circuit for improving client-side delay, including the main processor in the scope of patent application 9, and further including: The at least one direction-related sensor; and The network interface circuit.

Images