CN113672293B - Media data processing method based on cloud mobile phone and terminal equipment - Google Patents
Media data processing method based on cloud mobile phone and terminal equipment Download PDFInfo
- Publication number
- CN113672293B CN113672293B CN202010444415.3A CN202010444415A CN113672293B CN 113672293 B CN113672293 B CN 113672293B CN 202010444415 A CN202010444415 A CN 202010444415A CN 113672293 B CN113672293 B CN 113672293B
- Authority
- CN
- China
- Prior art keywords
- media data
- browser
- mobile phone
- cloud
- cloud mobile
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 8
- 238000013515 script Methods 0.000 claims abstract description 27
- 238000012545 processing Methods 0.000 claims description 42
- 238000000034 method Methods 0.000 claims description 36
- 230000001419 dependent effect Effects 0.000 claims description 6
- 230000015654 memory Effects 0.000 description 23
- 238000007726 management method Methods 0.000 description 20
- 230000006854 communication Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 7
- 239000004973 liquid crystal related substance Substances 0.000 description 7
- 230000002093 peripheral effect Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 4
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 238000009877 rendering Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000027455 binding Effects 0.000 description 2
- 238000009739 binding Methods 0.000 description 2
- 238000009792 diffusion process Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007667 floating Methods 0.000 description 2
- 101100364301 Schizosaccharomyces pombe (strain 972 / ATCC 24843) rsv1 gene Proteins 0.000 description 1
- 101100364302 Schizosaccharomyces pombe (strain 972 / ATCC 24843) rsv2 gene Proteins 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 239000003990 capacitor Substances 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000012528 membrane Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000002985 plastic film Substances 0.000 description 1
- 229920006255 plastic film Polymers 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
- 239000000565 sealant Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 125000006850 spacer group Chemical group 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/70—Media network packetisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/764—Media network packet handling at the destination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Abstract
The application provides a media data processing method based on a cloud mobile phone, which is applied to terminal equipment, wherein the terminal equipment comprises a browser and a processor, the browser is loaded with a script, and the processor runs the script to execute the following steps: receiving first media data sent by a cloud mobile phone, wherein the first media data is generated when the cloud mobile phone runs a cloud application; decoding the first media data to obtain first audio and first video; playing the first audio and the first video in the browser; and receiving an operation command for the cloud application, which is input by a user in the browser, and sending the operation command to the cloud mobile phone. The decoding of the media data of the game sent by the cloud mobile phone is realized by loading the script in the browser of the terminal equipment, the decoding is not limited by a personal computer, and the audio and video display of the cross-platform cloud mobile phone game can be realized.
Description
Technical Field
The application relates to the field of cloud technology, in particular to a media data processing method based on a cloud mobile phone and terminal equipment.
Background
In the process of running the cloud application, the cloud mobile phone can send media data to a browser of the personal computer through a network, a flash plug-in is installed in the browser of the personal computer, and the flash plug-in is used for processing the media data to generate audio and video so as to present images and sounds of the cloud application to a user in the browser.
The Flash plug-in needs to be installed in a browser of the personal computer, and the Flash plug-in only supports an operating system of the personal computer and is not suitable for the operating system of a mobile phone.
Disclosure of Invention
The application provides a media data processing method based on a cloud mobile phone and terminal equipment, wherein decoding of media data of cloud applications sent by the cloud mobile phone is achieved through loading scripts in a browser of the terminal equipment, and audio and video display of the cloud mobile phone applications with cross platforms can be achieved without being limited by a personal computer.
In a first aspect, a media data processing method based on a cloud mobile phone is provided, the method is applied to a terminal device, the terminal device comprises a browser and a processor, the browser is loaded with a script, and the processor runs the script to execute the following steps:
receiving first media data sent by a cloud mobile phone, wherein the first media data is generated when the cloud mobile phone runs a cloud application;
Decoding the first media data to obtain first audio and first video;
playing the first audio and the first video in the browser;
and receiving an operation command for the cloud application, which is input by a user in the browser, and sending the operation command to the cloud mobile phone.
In the scheme, the decoding of the media data of the application sent by the cloud mobile phone is realized by loading the script in the browser of the terminal equipment, the application audio and video display of the cross-platform cloud mobile phone can be realized without being limited by a personal computer.
In some possible designs, after sending the operation command to the cloud handset, the processor runs the script to perform the steps of:
receiving second media data sent by the cloud mobile phone, wherein the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command;
decoding the second media data to obtain second audio and second video;
and playing the second audio and the second video in the browser.
In the scheme, the operation command of the user on the terminal equipment is also sent to the cloud mobile phone to run, the media data generated by the cloud mobile phone is decoded by loading the script in the browser of the terminal equipment, so that the resource consumption of the terminal equipment can be further reduced, meanwhile, the terminal equipment is not limited by a personal computer, and the cross-platform cloud mobile phone application audio and video display can be realized.
In some possible designs, the processor runs the script to spawn a first browser thread, a second browser thread, and a third browser thread;
a first processing core of the processor runs the first browser thread to receive the first media data;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain first audio and first video;
a first processing core of the processor runs the second browser thread to play the first audio and first video in the browser; and receiving an operation command for the game, which is input by a user in the browser, and sending the operation command to the cloud mobile phone.
In the above scheme, the third browser thread for decoding can occupy the second processing core alone, so that the decoding speed is improved.
In some possible designs, decoding the first media data includes:
and when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcedly decode the first media data, wherein the media consumption queue is used for storing the first media data.
In the above scheme, the third browser thread can forcedly decode the first media data when the data amount of the media data in the media consumption queue is smaller than the threshold value, and does not need to slowly wait for the decoding when the data amount of the media data in the media consumption queue is larger than or equal to the threshold value, thereby accelerating the decoding process of the third browser thread and effectively reducing the time delay.
In some possible designs, decoding the first media data includes:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data amount of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data amount of the media data is larger than or equal to the threshold value.
In the scheme, the increase of the data quantity in the media data in the media consumption queue is quickened by adding the dummy frame so as to reach the threshold value as soon as possible for decoding, thereby quickening the decoding process of the third browser thread and effectively reducing the time delay.
In some possible designs, the second processing core of the processor invokes binary code written in a low-level language to decode the first media data during execution of the third browser thread.
In the above scheme, decoding of the first media data is expedited by invoking binary code written in a low-level language.
In a second aspect, there is provided a terminal device comprising a browser loaded with a script and a processor running the script to: receiving first media data sent by a cloud mobile phone; decoding the first media data to obtain first audio and first video; playing the first audio and the first video in the browser; and receiving an operation command for cloud application, which is input by a user at the browser, and sending the operation command to the cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs the cloud application.
In some possible designs, the processor runs the script to: receiving second media data sent by the cloud mobile phone; decoding the second media data to obtain second audio and second video; and playing the second audio and the second video in the browser, wherein the second media data are generated when the cloud mobile phone runs the cloud application according to the operation command.
In some possible designs, the processor runs the script to spawn a first browser thread, a second browser thread, and a third browser thread;
the first processing core of the processor runs the first browser thread to receive first media data sent by a cloud mobile phone;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video;
a first processing core of the processor runs the second browser thread to play the first audio and first video in the browser; and receiving an operation command for the cloud application, which is input by a user in the browser, and sending the operation command to the cloud mobile phone.
In some possible designs, decoding the first media data includes:
and when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcedly decode the first media data, wherein the media consumption queue is used for storing the first media data.
In some possible designs, decoding the first media data includes:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data amount of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data amount of the media data is larger than or equal to the threshold value.
In some possible designs, the second processing core of the processor invokes binary code written in a low-level language to decode the first media data during execution of the third browser thread.
In a third aspect, there is provided a computer readable storage medium comprising instructions which, when run on a terminal device, cause the terminal device to perform the method of any of the first aspects.
In a fourth aspect, there is provided a terminal device comprising a processor and a memory, the processor executing code in the memory to perform the method of any of the first aspects.
In a fifth aspect, there is provided a computer program product comprising a computer program which, when read and executed by a computing device, causes the computing device to perform the method of any of the first aspects.
In the scheme, the decoding of the media data of the application sent by the cloud mobile phone is realized by loading the script in the browser of the terminal equipment, the application audio and video display of the cross-platform cloud mobile phone can be realized without being limited by a personal computer.
Drawings
Fig. 1 is a schematic structural diagram of a media data processing system of a cloud mobile phone according to the present application;
FIGS. 2A-2B illustrate two different exemplary user interfaces for a terminal device to open a game page in a browser to play a game;
3A-3B illustrate a comparison of an original browser structure and a modified browser structure;
fig. 4 is a flow chart interaction diagram of a media data processing method based on a cloud mobile phone provided by the application;
fig. 5 is a schematic structural diagram of a WebSocket protocol packet related to the present application;
fig. 6 is a schematic structural diagram of a terminal device provided in the present application;
fig. 7 is a schematic structural diagram of another terminal device provided in the present application.
Detailed Description
Referring to fig. 1, fig. 1 is a schematic structural diagram of a media data processing system of a cloud mobile phone according to the present application. The media data processing system of the cloud mobile phone of the present embodiment includes a terminal device 110 and a cloud data center 120. Wherein the terminal device 110 may communicate with the cloud data center 120 through a network.
The terminal device 110 may be a smart phone, a cloud terminal, a thin terminal, or the like. In a specific embodiment, the terminal device 110 may be a terminal device used with a cloud mobile phone, i.e. the terminal device 110 has a strong network communication capability, but the computing capability and the storage capability may be weakened. The terminal device 110 has a browser installed therein, and a browser engine used by the browser may be Trident, gecko, presto or Webkit, etc., which is not particularly limited herein. The operating system supported by the terminal device 110 may be an IOS system, an Android system (Android), a macOS system, a Windows system, a Linux system, and the like. In the game system, different terminal devices 110 may support different operating systems, for example, terminal device 1 supports an android system, terminal device 2 supports a macOS system, terminal device 3 supports an IOS system, terminal device 4 supports a Windows system, and terminal device 5 supports a Linux system. Of course, in the game system, different terminal devices 110 may also support the same operating system, which is not particularly limited herein.
Cloud data center 120 may include a service node 121 and a cloud handset management node 125. The service node 121 may be a general physical server, for example, a physical server, such as an X86 server or an ARM server. The cloud mobile phone management node 125 may be a server for managing cloud mobile phones in public cloud, specifically may be a general physical server, or may be a Virtual Machine (VM) implemented based on a general physical server and combined with NFV technology, where the Virtual Machine refers to a complete computer system that has a complete hardware system function and operates in a completely isolated environment through software simulation, and the application is not limited specifically.
In a specific embodiment, the service node 121 includes a cloud mobile management agent node 124, hardware resources, an Operating System (OS) 122, and at least one cloud mobile 123, where each cloud mobile 123 shares the hardware resources and the Operating System with other cloud mobile on the server in a Container (Container). The cloud server with the operating system and the virtual mobile phone function is used for transferring the application on the mobile phone to the virtual mobile phone on the public cloud to run.
The hardware resources may include computing resources, storage resources, and network resources, such as processors, memories, network cards, routes, databases, etc., and may also include hardware resources that may be needed by other cloud handsets, which are not specifically limited in this application.
The operating system 122 may be an operating system applicable to a cloud mobile phone, such as an Android operating system, which is not specifically limited in this application. It should be noted that, the operating system 122 may be an official complete operating system, or may be an operating system that is modified by an individual driver module of the official complete operating system to adapt to the operation mode of the service node 121, which is not specifically limited in this application.
The cloud phone 123 shares hardware resources and an operating system with other cloud phones on the server in the form of a container, and each container may include a cloud application required by the cloud phone and a dependent resource required by each cloud application, and the dependent resource required by the cloud application may be an operating system core library and a system library, such as a function library, a three-dimensional graphics processing library (such as OpenGLES), a Media library (Media Libraries), an Input Manager (Input Manager), and the like. In one embodiment, the container may include the gaming application and the dependent resources required by the gaming application. The container where each cloud mobile phone 123 is located uses the hardware resource and the operating system 122, but does not monopolize the hardware resource, but shares the hardware resource and the operating system 122 with the containers where other cloud mobile phones are located. In other words, each container has no own kernel, and the application processes within the container run directly on the kernel of the service node 121.
The cloud mobile phone management proxy node 124 may be configured to monitor and collect service information of the service node 121, report the service information to the cloud mobile phone management node 125 in real time, and further, create a container on the service node 121 according to the received specification information of the cloud mobile phone, and divide resources from hardware resources into hardware and software required by the container to set the specification information of the cloud mobile phone.
The process of creating a cloud handset at cloud data center 120 may be: the terminal device 110 sends cloud handset creation information to the cloud handset management node 125. The cloud mobile phone management node 125 determines that the cloud mobile phone 1 is created by the server 1 and the cloud mobile phone 2 is created by the server 2 according to the specification information of the cloud mobile phone 123 in the cloud mobile phone creation information and the service information sent by each cloud mobile phone management proxy node 124, and then the cloud mobile phone management proxy node 1 on the server 1 can receive the specification information of the cloud mobile phone 1 sent by the cloud mobile phone management node 125, and the cloud mobile phone management proxy node 2 on the server 2 can receive the specification information of the cloud mobile phone 2 sent by the cloud mobile phone management node 125 and complete the creation of the cloud mobile phone 2 according to the specification information. It may be appreciated that the cloud mobile management node 125 and the cloud mobile management proxy node 124 deployed on each service node 121 may be implemented based on a container management system such as Kubernetes (K8S), docker, etc., which is not specifically limited in this application.
On the basis of the media data processing system of the cloud mobile phone shown in fig. 1, a user can play a game with low delay through a browser in the terminal equipment without installing the cloud application in the terminal equipment. The following will describe in detail the case of the terminal devices respectively being a smart phone and a thin terminal. For simplicity, the following examples will be described using a cloud application as an example of a game application, but in other examples, the cloud application may be another application, which should not be limited specifically herein.
The terminal equipment is a smart phone. FIG. 2A illustrates an exemplary user interface for opening a game page in a browser to play a game. As shown in the upper left corner of fig. 2A: the user interface may include: status bars, calendar indicators, weather indicators, trays with commonly used application icons, navigation bars, and other application icons. Wherein:
the status bar may include: one or more signal strength indicators of a mobile communication signal (also may be referred to as a cellular signal), one or more signal strength indicators of a wireless high-fidelity (wireless fidelity, wi-Fi) signal, a battery status indicator, a time indicator.
Calendar indicators may be used to indicate the current time, such as date, day of the week, time division information, and so forth.
Weather indicators may be used to indicate weather types, such as cloudy, rainy, etc., and also information such as air temperature.
Trays with common application icons may show: telephone icons, contact icons, text message drawings, and camera icons.
The navigation bar may include: a return key, a home screen key, a multi-tasking key, and the like. When detecting that the user clicks the return key, the smart phone may display the last page of the current page. When the user is detected to click the home screen key, the smart phone displays the home interface. When it is detected that the user clicks the multitasking key, the smartphone may display the task that the user recently opened. The names of the navigation keys may be other, and the application is not limited thereto. Not limited to virtual keys, each navigation key in the navigation bar may also be implemented as a physical key.
Other application icons may be, for example: inter-transferred icons, gallery icons, music icons, application icons, mailbox icons, browser icons 212, memo icons, set icons. The user interface 210 may also include page indicators. Other application icons may be distributed across multiple pages, and page indicators may be used to indicate which page the user is currently browsing for applications in. The user may slide the area of the other application icons left and right to view the application icons in the other pages.
In some embodiments, the user interface 210 exemplarily shown in fig. 2A may be a Home screen (Home screen). It will be appreciated that fig. 2A illustrates a user interface on a smartphone by way of example only, and should not be construed as limiting embodiments of the present application.
Illustratively, the user may click on an icon 212 of the browser on the user interface 210, the smartphone detects the user operation, and in response to the user operation, the smartphone opens the browser 220.
As shown in the upper right corner of fig. 2A: browser 220 may include: search box 222, scan controls, tab bars, multiple recommended catalogs, and trays of commonly used controls, wherein:
the search box 222 may be used to search for game web pages that match the character based on the character entered by the user.
The scanning control can be used for scanning the two-dimensional code or the bar code, so that the game webpage corresponding to the two-dimensional code or the bar code can be opened quickly.
The title bar may be used to indicate various types of topics, such as recommendations, games, hotspots, entertainment, and so forth. The presentation form of the title bar may be text information, icons, or other forms.
The control 302 may receive a user operation (e.g., a touch operation), and in response to detecting the user operation, the anchor terminal 120 may display a page to log in or switch the cloud live account.
The tray of common controls may show: a previous page control icon, a next page control icon, a home page control icon, an opened web page icon, and the like. The control icons can all accept user operations (such as touch operations), and the smart phone can display a responsive page corresponding to the detected user operations, specifically, the control icon of the previous page can be used for displaying the web page displayed by the previous page, the control icon of the previous page can be used for displaying the web page displayed by the next page, the control icon of the first page can be used for the first page of the browser, the first line of the browser can be user-defined, or the navigation page preset by the browser, and the opened web page icon can be used for displaying a plurality of pages which are opened currently.
The plurality of recommendation directories may be, for example: a display title and a display picture 224, the display title displaying information related to the game, the display picture 224 displaying pictures related to the game. Display picture 224 is associated with a hyperlink to a game page, and when the user clicks on display picture 224, the browser may jump to game page 230.
As shown in the lower diagram of fig. 2A, the game page 230 is used to display the screen content of the game and play the sound of the game. The game page may include a game title, a game screen, and a game control.
The game title is selectable, i.e., the user may choose to display the game title or not display the game title, or the game page may display the game tab by default, or not display the game title by default.
The image quality, definition, resolution, size and color of the game picture can be set according to the needs of the user, for example, the user can set the picture of the game as full screen or not. In addition, the game screen can be arranged on other applications in a floating mode so that a user can play a game and process other matters. The game screen may be a two-dimensional screen or a three-dimensional screen.
The specific process of the user opening the game page in the browser to play the game is as follows: the User clicks a browser icon 212 on a User Interface 210 (UI) of the smartphone with a finger, thereby opening a browser 220. The user enters a uniform resource locator (Uniform Resource Locator, URL) of the game web page 230 in the search field 222 of the browser 120, or clicks on a hyperlink 224 recommended by the browser 220 (the hyperlink points to the game web page 230 and includes the URL of the game web page 230), thereby opening the game web page 230. The user may then play the game on the game page 230.
It will be appreciated that fig. 2A illustrates only the various user interfaces on a smartphone that open a game page, and should not be construed as limiting embodiments of the present application.
The terminal device is a thin terminal. FIG. 2B illustrates an exemplary desktop that opens a game page in a browser to play a game. As shown in the upper left corner of fig. 2B: the desktop may include: a calendar indicator, an application tray has been opened, a start button.
Wherein:
calendar indicators, located in the lower right hand corner of the desktop, may be used to indicate the current time, such as date, day of week, time division information, etc.
An opened application tray is located at the bottom of the desktop and is used for indicating the opened application, for example, when the browser is opened, the icon of the browser is displayed in the opened application tray, and when the browser is closed, the icon of the browser disappears from the opened application tray. When a browser is opened multiple times, icons of multiple overlapping browsers may be displayed to indicate that the browser is opened multiple times.
A start button, located in the lower left corner of the desktop, for displaying one or more levels of menus, each of which may display entries for a plurality of applications, such as control panels, devices and printers, default programs, security and help, and the like.
A plurality of application icons may be displayed on the desktop, for example: an icon of a cloud application, an icon of a camera, an icon of a time, an icon of a calendar, an icon of an application, an icon 310 of a browser, an icon of a setting, and the like.
Illustratively, as shown in the upper left diagram of fig. 2B, a user may click on an icon 310 of a browser on a desktop, and the thin terminal detects the above user operation, and in response to the above user operation, the thin terminal opens the browser 320.
As shown in the upper right corner of fig. 2B: browser 320 may include: search box 322, previous page control, next page control, reload control, title bar, and a plurality of recommendation directories, wherein:
the search box 322 may be used to search for game web pages that match the character based on the character entered by the user.
The previous page control may be used to display the web page displayed by the previous page, the previous page control icon may be used to display the web page displayed by the next page, and the reload control may be used to refresh the current page to reload the current page.
The title bar may be used to indicate various types of topics, such as recommendations, games, hotspots, entertainment, and so forth. The presentation form of the title bar may be text information, icons, or other forms.
The plurality of recommendation directories may be, for example: a display title and a display picture 324, the display title displaying information related to the game, the display picture 324 displaying pictures related to the game. Display picture 324 is associated with a hyperlink to a game page, and when the user clicks on display picture 324, the browser may jump to game page 330.
As shown in the lower diagram of fig. 2B, the game page 330 is used to display the screen content of the game and play the sound of the game. The game page may include a game title, a game screen, and a game control.
The game title is selectable, i.e., the user may choose to display the game title or not display the game title, or the game page may display the game tab by default, or not display the game title by default.
The image quality, definition, resolution, size and color of the game picture can be set according to the needs of the user, for example, the user can set the picture of the game as full screen or not. In addition, the game screen can be arranged on other applications in a floating mode so that a user can play a game and process other matters. The game screen may be a two-dimensional screen or a three-dimensional screen.
The specific process of the user opening the game page in the browser to play the game is as follows: the user clicks on the browser icon 312 on the display's interface 310 with a mouse, thereby opening the browser 320. The user enters a uniform resource locator (Uniform Resource Locator, URL) of the game web page 330 in the search field 322 of the browser 320, or clicks on a hyperlink 324 recommended by the browser 320 (the hyperlink points to the game web page 330 and includes the URL of the game web page 330), thereby opening the game web page 330. The user may then play the game on the game page 330. The interface 310 may be used to display a music application icon, a mail application icon, a gallery application icon, a setting icon, etc., in addition to the browser icon, which is not particularly limited herein.
It will be appreciated that fig. 2B is merely illustrative of various user interfaces on a thin terminal for opening a game page, and should not be construed as limiting embodiments of the present application.
In addition to the above examples, in practical applications, the terminal device may be a virtual device or the like, and is not particularly limited herein.
The internal structure of the browser according to the present application will be specifically described below.
As shown in fig. 3A, taking a browser as WebKit as an example, the browser includes a rendering engine and a JavaScript (JS) engine, where the rendering engine is configured to convert a resource file (HTML/CSS/JavaScript text and corresponding resource file) into a result visible to a user. The rendering engine typically includes CSS, SVG, layout, rendering tree, HTML, DOM, insactor, and so forth. The JS engine is used for reading the JavaScript in the file, processing and executing the JavaScript, and is single-threaded. In addition, the Webkit further comprises a Webkit embedded interface and corresponding Webkit bindings, a Webkit2 embedded interface and corresponding Webkit2 bindings, and Webkit ports. WebKitPorts may include, among other things, grid segments, video, text, hardware acceleration, picture decoding, and so forth. In addition, webKit dependent databases and operating systems may be included. The databases may include, among other things, 2D graphics libraries, 3D graphics libraries, network libraries, storage, audio libraries, and video libraries, among others.
As shown in the dashed box portion of fig. 3B, multiple browser threads, including a first browser thread, a second browser thread, and a third browser thread, may be added to the JS engine of the existing browser. Wherein the browser thread may be WebWorker. The WebWorker is used for enabling the browser to run a script operation in a background thread (worker) separated from a main execution thread, so that a multithreaded running environment is created for the JS engine, namely, the main thread of the browser creates a browser thread and distributes tasks to the browser thread, the browser thread runs simultaneously with the main thread running, the browser threads do not interfere with each other, and a result is returned to the main thread after the running of the browser thread is finished. The first browser thread may also be called a network thread, and the network thread adopts WebSocket protocol, so that bidirectional communication of game data (including media data and control data) between the terminal device and the cloud mobile phone can be realized. The second browser thread, which may also be referred to as a master thread, is responsible for the overall operation of the game. The third browser thread, which may also be referred to as a decode thread, is responsible for decoding the media data of the game.
The implementation of low-latency game play by a user through a browser in a terminal device will be described in detail below in connection with the browser shown in fig. 3B. As shown in fig. 4, fig. 4 is a flow chart of a game running method provided in the present application. The game running method in the present embodiment includes the steps of:
s101: in case that the user inputs the URL of the game web page in the search field of the browser or clicks on the hyperlink recommended by the browser, the browser of the terminal device generates a media data request, wherein the media data request is used for generating the media data of the game web page, and the media data request contains the URL, and the URL corresponds to the game.
In a specific embodiment, the browser engine employed by the browser may be Trident, gecko, presto or Webkit, etc., without limitation.
In a specific embodiment, the terminal device detects, through a main control thread in the browser, whether the user inputs the URL of the game webpage in the search bar, or whether the user clicks the hyperlink recommended by the browser. In the case of inputting the URL of the game web page in the search bar or clicking the hyperlink recommended by the browser, the main control thread generates a media data request and sends the media data request to the network thread through a shared memory or pipeline communication method and the like.
S102: and the browser of the terminal equipment sends the media data request to the cloud mobile phone. Accordingly, the cloud mobile phone receives a media data request sent by the terminal equipment.
In a specific embodiment, the web thread in the browser sends the media data request to the cloud mobile phone through WebSocket protocol. The network thread is a uniparted Webwork, so that the network thread can realize data receiving and transmitting between the browser and the cloud mobile phone.
S103: the cloud mobile phone runs a game application to generate first media data.
In a specific embodiment, after receiving a media data request sent by a browser of a terminal device, the cloud mobile phone obtains a URL of a game webpage from the media data request, determines a corresponding game application according to the URL of the game webpage, runs the game application, and generates first media data. Then, the cloud mobile phone encodes the generated first media data through an encoding method, so that the number of data transmission is reduced.
S104: and the cloud mobile phone sends the first media data to the terminal equipment. Correspondingly, the terminal equipment receives the first media data sent by the cloud mobile phone.
In a specific embodiment, a network thread in a browser of the terminal device receives first media data sent by the cloud mobile phone through WebSocket protocol. After a master control thread in the browser monitors first media data sent by a network thread, the first media data is pushed to a media data queue. The main control thread in the browser sends the data in the media data queue to the decoding thread for decoding in an instant mode through a shared memory or pipeline communication method and the like, so that the time delay of the game is effectively reduced.
S105: the terminal device decodes the first media data to obtain first audio and/or first video.
In a specific embodiment, the decoding thread in the browser in the terminal device decodes the first media data, thereby obtaining the first audio and/or the first video. The decoding thread in the browser then sends the first media data to the main control thread by means of shared memory or pipe communication, etc. The decoding method adopted by the decoding thread corresponds to the encoding method adopted by the cloud mobile phone, when the encoding method adopted by the cloud mobile phone is lossless compression, the decoding method adopted by the terminal equipment is also lossless compression, and when the encoding method adopted by the cloud mobile phone is lossy compression, the decoding method adopted by the terminal equipment is also lossy compression.
S106: the terminal device plays the first audio and/or the first video in the first game page generated by the browser.
S107: and under the condition that the user operates the game on the game page of the browser, receiving an operation command of the game input by the user on the game page of the browser, and converting the operation command into game operation which can be identified by the cloud mobile phone.
In a specific embodiment, a main control thread in a browser of the terminal device receives an operation command of a game input by a user on a game page of the browser through an external input device. The external input device may be a touch screen, a mouse, a keyboard, a camera, a scanner, a light pen, a handwriting input board, a joystick, a microphone, and the like. The operational commands for the game may include forward, backward, left turn, right turn, left move, right move, jump, sit down, stand up, receive/pick up weapons, walk/run, etc. The terminal device and the cloud mobile phone have different definitions on the operation, so that the browser needs to convert an operation command input by a user into a game operation which can be identified by the cloud mobile phone. The master thread in the browser then sends the game operations to the network thread by means of shared memory or pipe communication, etc.
S108: and the terminal equipment sends game operation to the cloud mobile phone. Accordingly, the cloud mobile phone receives the game operation sent by the terminal equipment.
In a specific embodiment, the terminal device sends the game operation to the cloud mobile phone through WebSocket protocol.
S109: and the cloud mobile phone runs the game application according to the game operation, so that second media data are generated.
In a specific embodiment, the cloud mobile phone continues to run the game application and generates the second media data according to the game operation. Then, the cloud mobile phone encodes the generated second media data through an encoding method, so that the number of data transmission is reduced.
S110: and the cloud mobile phone sends the second media data to the terminal equipment. Correspondingly, the terminal equipment receives second media data sent by the cloud mobile phone.
In a specific embodiment, the network thread in the browser of the terminal device receives the second media data sent by the cloud mobile phone through WebSocket protocol. After the master control thread in the browser monitors the second media data sent by the network thread, the second media data is pushed to the media data queue. The main control thread in the browser sends the data in the media data queue to the decoding thread for decoding in an instant mode through a shared memory or pipeline communication method and the like, so that the time delay of the game is effectively reduced.
S111: the terminal device decodes the second media data to obtain second audio and/or second video.
In a specific embodiment, the decoding thread in the browser in the terminal device decodes the second media data, thereby obtaining the second audio and/or the second video. The decoding thread in the browser then sends the second media data to the main control thread by means of shared memory or pipe communication, etc.
S112: and the terminal equipment plays the second audio and/or the second video in the second game page generated by the browser.
In a specific embodiment, the media data request, the first audio and/or first video, the game operation, and the second audio and/or second video are all encapsulated in WebSocket protocol format. As shown in fig. 5, the WebSocket protocol format includes FIN of 1 bit, opcodes of 1 bit, RSV1, RSV2, RSV3,4 bits each, MASK of 1 bit, data payload length of 7 bits (payload len), MASK key of 0 to 4 bytes, and payload data of multiple bytes (payload). Here, the media data request, the first audio and/or first video, the game operation, and the second audio and/or second video payload are in payload data (payload).
In a specific embodiment, the terminal device receives the first media data sent by the cloud mobile phone, and needs to decode the first media data into the first audio and/or the first video as soon as possible, and play the first audio and/or the first video in the first game page generated by the browser. The terminal device can be divided into at least two modes according to different browser capabilities:
in a first way, the terminal device decodes the first media data into the first audio and/or the first video as soon as possible through a video (video) component, and plays the first audio and/or the first video in a first game page generated by a browser. Here, the video (video) component may be implemented using modules in WebKitPorts, e.g., video, hardware acceleration, etc. The terminal device typically plays video in the browser through a video (video) component, which is therefore not delay sensitive. After the network thread receives the media data, the video (video) component firstly caches the media data, and then decodes the media data until the number of the cached media data exceeds a threshold value, and plays the media data in the browser to ensure the smoothness of playing, which, however, causes a very large time delay. However, games are very sensitive to latency, and such a large latency is unacceptable. To solve the problem of large delay, a video (video) component may be improved so as to reduce delay. In particular, improvements to video (video) components include at least the following: (1) The video (video) component issues a force command that instructs the decoding thread to immediately decode the received first media data to obtain the first video and the first audio, and the host thread immediately plays the first video and the first audio in the browser without buffering the first media data. (2) The video (video) component instructs the master thread to add dummy frames in the media consumption queue such that media data in the buffered media consumption queue reaches a threshold as soon as possible, when the threshold is reached, the video component instructs the decoding thread to decode the first media data to obtain a first video and a first audio, and the video component instructs the master thread to play the first video and the first audio in the browser. The dummy frame may be a data frame completely unrelated to the first media data, and may also be referred to as a transparent frame. (3) After decoding the multi-frame video and audio of the first media data, a video (video) component instructs the master thread to forcibly play the video and audio of the last frame of the decoded first video and first audio to reduce latency. To further reduce latency, a video (video) component may also process media data in a frame-skip manner, where media data may be sampled as it is processed in a frame-skip manner, and more closely sampled data frames may be more densely sampled and more sparsely sampled data frames may be more widely sampled.
In a second mode, the terminal device decodes the first media data into the first audio and/or the first video as soon as possible through the Canvas component, and plays the first audio and/or the first video in the first game page generated by the browser. Here, the Canvas component may be implemented using modules in WebKitPorts, e.g., video, hardware acceleration, etc. The decoding thread decodes through the file written by JavaScript semantics, the decoding efficiency is very low, and for this purpose, the Canvas component can instruct the decoding thread to call the binary code written by the low-level language, for example, webAssemble for decoding, so that the decoding rate is improved, and in addition, the decoding thread can exclusively decode by using the kernel of one processor, and the decoding rate can also be obviously improved.
The foregoing details of the method according to the embodiments of the present application are provided for better implementation of the foregoing aspects of the embodiments of the present application, and accordingly, related devices for implementing the foregoing aspects in cooperation are also provided below.
The terminal device provided by the application may include a processor and a browser as shown in fig. 3B, where the browser includes a JS engine, and the browser is loaded with a script, and the processor runs the script to make the JS engine generate a first browser thread, a second browser thread, and a third browser thread.
The first processing core of the processor runs the first browser thread to receive first media data sent by a cloud mobile phone, and the first media data is generated when the cloud mobile phone runs a game;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain first audio and first video;
a first processing core of the processor runs the second browser thread to play the first audio and first video in the browser;
and the first processing core of the processor runs the second browser thread to receive an operation command for the game, which is input by a user at the browser, and sends the operation command to the cloud mobile phone.
In some possible designs, the first processing core of the processor runs the first browser thread to receive second media data sent by a cloud mobile phone, the second media data being generated when the cloud mobile phone runs a game according to the operation command;
a second processing core of the processor runs the third browser thread to decode the second media data to obtain second audio and second video;
The second processing core of the processor runs the third browser thread to play the second audio and second video in the browser.
In some possible designs, the second processing core of the processor runs the third browser thread to force decoding of the first media data when an amount of media data in a media consumption queue for storing the first media data is less than a threshold.
In some possible designs, the second processing core of the processor runs the third browser thread to add dummy frames in the media consumption queue when the amount of media data in the media consumption queue is less than a threshold, such that the amount of media data in the media consumption queue including the dummy frames is greater than or equal to the threshold, and decodes the first media data if the amount of media data is greater than or equal to the threshold.
In some possible designs, the second processing core of the processor invokes binary code written in a low-level language to decode the first media data during execution of the third browser thread.
It should be understood that the unit modules inside the terminal device shown in fig. 3B may also have various divisions, and each module may be a software module, a hardware module, or a part of a software module and a part of a hardware module, which is not limited in this application. Fig. 3B is an exemplary division, which is not specifically limited in this application.
An implementation manner of the terminal device according to the embodiment of the present invention is described below. The terminal device supports a browser running a plurality of web labels. The browser may use different input and output devices, respectively, for example, the input device may be a keyboard and a mouse, and the output device may be a display screen. Alternatively, the browser uses a common input-output device: a touch screen. The touch screen is used for receiving touch operation of a user and displaying output content of the browser.
Fig. 6 is a block diagram of the architecture of one implementation of a terminal device. As shown in fig. 6, the terminal device may be a thin terminal, including: host 410, output device 420, and input device 430.
Host 410 may integrally include: one or more processors, a clock module, and a power management module. The clock module integrated in the host 410 is mainly used to generate clocks for the processor that are needed for data transmission and timing control. The power management module integrated in the host 410 is mainly used to provide stable, high-precision voltage to the processor, the output device 420, and the input device 430. Host 410 also integrates memory for storing various software programs and/or sets of instructions. In particular implementations, the memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory may not need to have an operating system and a large number of applications installed, but rather the memory need only store some of the necessary applications, such as input and output programs that may be used to communicate with one or more input devices or output devices. And the communication program can be used for communicating with the cloud mobile phone. The memory can also store a user interface program, the user interface program can vividly display the content image of the browser through a graphical operation interface, and control operation of a user on the browser is received through input controls such as menus, dialog boxes, buttons and the like.
The output device 420 mainly includes a display, which may include a Cathode Ray Tube (CRT) display, a plasma display (Plasma Display Panel, PDP), a liquid crystal display (Liquid Crystal Display, LCD), and the like. Taking a display as an example of an LCD, the liquid crystal display includes a liquid crystal panel and a backlight module, wherein the liquid crystal display panel includes a polarizing film, a glass substrate, a black matrix, a color filter, a protective film, a common electrode, a calibration layer, a liquid crystal layer (liquid crystal, spacers, sealant), a capacitor, a display electrode, a prism layer, and a light diffusion layer. The backlight module comprises: an illumination light source, a reflection plate, a light guide plate, a diffusion sheet, a brightness enhancement film (prism sheet), a frame, and the like.
The input device 430 may include a keyboard and a mouse. The keyboard and the mouse are the most commonly used and main input devices, and English letters, numbers, punctuation marks and the like can be input into a computer through the keyboard, so that commands, input data and the like are sent to the computer, and the longitudinal and transverse coordinate positioning can be rapidly performed through the mouse, so that the operation is simplified. The keyboard may include a Mechanical keyboard, a plastic film type keyboard (Mechanical), a conductive rubber type keyboard (Membrane), a contactless electrostatic capacitance type keyboard (capacitance), etc., and the mouse may include a ball mouse, an optical mouse, a wireless mouse, etc.
The input device 430 is configured to receive first media data sent by a cloud mobile phone, where the first media data is generated when the cloud mobile phone runs a cloud application;
the host 410 is configured to decode the first media data to obtain first audio and first video;
the output device 420 is used for playing the first audio and the first video in the browser;
the input device 430 is configured to receive an operation command for the cloud application input by the user at the browser, so as to send the operation command to the cloud mobile phone through the communication module.
For simplicity, the terminal device shown in fig. 6 is not described in detail, and reference is made specifically to fig. 1 to 5 and the related description, which will not be described again.
Fig. 7 is a block diagram of another implementation of a terminal device. As shown in fig. 7, the terminal device may be a smart phone, including: baseband chip 510, memory 515, including one or more computer-readable storage media, radio Frequency (RF) module 516, peripheral system 517. These components may communicate over one or more communication buses 514.
The peripheral system 517 is mainly used for implementing an interactive function between the terminal device 510 and a user/external environment, and mainly includes input and output devices of the terminal device. In particular implementations, the peripheral system 517 may include: a touch screen controller 518, a camera controller 519, an audio controller 520, and a sensor management module 521. Wherein each controller may be coupled to a respective corresponding peripheral device, such as a touch screen 523, a camera 524, audio circuitry 525, and a sensor 526. In some embodiments, gesture sensors in the sensors 526 may be used to receive gesture control operations entered by a user. The pressure sensor in the sensor 526 may be disposed below the touch screen 523, and may be used to collect touch pressure acting on the touch screen 523 when a user inputs a touch operation through the touch screen 523. It should be noted that the peripheral system 517 may also include other I/O peripherals.
The baseband chip 510 may be integrated to include: one or more processors 511, a clock module 512, and a power management module 513. The clock module 512 integrated in the baseband chip 510 is mainly used to generate clocks required for data transmission and timing control for the processor 511. The power management module 513 integrated in the baseband chip 510 is mainly used to provide a stable, high-precision voltage for the processor 511, the rf module 516, and the peripheral system.
Memory 515 is used to store various software programs and/or sets of instructions. In particular implementations, the memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory may not need to have an operating system and a large number of applications installed, but rather the memory need only store some of the necessary applications, such as input and output programs that may be used to communicate with one or more input devices or output devices. And the communication program can be used for communicating with the cloud mobile phone. The memory may also store desktop programs.
A Radio Frequency (RF) module 516 is used to receive and transmit radio frequency signals, principally integrating the receiver and transmitter of the terminal device. Radio Frequency (RF) module 516 communicates with communication networks and other communication devices via radio frequency signals. In particular implementations, radio Frequency (RF) module 516 may include, but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card 5161, a storage medium, and the like. In addition, the radio frequency module may further include a WIFI 5162, bluetooth, and other short range wireless communication modules. In some embodiments, the Radio Frequency (RF) module 516 may be implemented on a separate chip.
The Radio Frequency (RF) module 516 is configured to receive first media data sent by a cloud mobile phone, where the first media data is generated when the cloud mobile phone runs a cloud application;
the baseband chip 510 is configured to decode the first media data to obtain a first audio and a first video;
the touch screen 523 is used for playing a first video in the browser, and the audio circuit 525 is used for playing a first audio;
the touch screen 523 is used for receiving an operation command for the cloud application input by a user at the browser, so as to send the operation command to the cloud mobile phone through a Radio Frequency (RF) module 516.
For simplicity, the terminal device shown in fig. 7 is not described in detail, and reference is made specifically to fig. 1 to 5 and the related description, which will not be described again.
In the scheme, decoding of the media data of the game sent by the cloud mobile phone is realized by loading the script in the browser of the terminal equipment, and audio and video display of the cross-platform cloud mobile phone game can be realized without being limited by a personal computer.
From the above description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented by means of software plus necessary general purpose hardware, or of course may be implemented by dedicated hardware including application specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components and the like. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions can be varied, such as analog circuits, digital circuits, or dedicated circuits. However, a software program implementation is a preferred embodiment in many cases for the present application. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk or an optical disk of a computer, etc., including several instructions for causing a computer device (which may be a personal computer, a training device, or a network device, etc.) to perform the method described in the embodiments of the present application.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with embodiments of the present application are generated, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via a wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, or the like that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.
Claims (13)
1. The media data processing method based on the cloud mobile phone is characterized by being applied to terminal equipment, wherein the terminal equipment comprises a browser and a processor, the browser is loaded with a script, and the processor runs the script to execute the following steps:
receiving first media data sent by a cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs a cloud application, the cloud application and dependent resources required by the cloud application are arranged in containers corresponding to the cloud mobile phone, and the containers are arranged in a server;
decoding the first media data to obtain first audio and first video;
playing the first audio and the first video in the browser;
and receiving an operation command for the cloud application, which is input by a user in the browser, and sending the operation command to the cloud mobile phone.
2. The method of claim 1, wherein after sending the operation command to the cloud handset, the processor runs the script to perform the steps of:
receiving second media data sent by the cloud mobile phone, wherein the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command;
Decoding the second media data to obtain second audio and second video;
and playing the second audio and the second video in the browser.
3. The method of claim 1 or 2, wherein the processor runs the script to spawn a first browser thread, a second browser thread, and a third browser thread;
a first processing core of the processor runs the first browser thread to receive the first media data;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain first audio and first video;
a first processing core of the processor runs the second browser thread to play the first audio and first video in the browser; and receiving an operation command for games input by a user in the browser, and sending the operation command to the cloud mobile phone.
4. The method of claim 3, wherein decoding the first media data comprises:
and when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcedly decode the first media data, wherein the media consumption queue is used for storing the first media data.
5. The method of claim 3, wherein decoding the first media data comprises:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data amount of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data amount of the media data is larger than or equal to the threshold value.
6. The method of any one of claims 3 to 5, wherein the second processing core of the processor invokes binary code written in a low-level language to decode the first media data during execution of the third browser thread.
7. A terminal device comprising a browser loaded with a script and a processor running the script to: receiving first media data sent by a cloud mobile phone; decoding the first media data to obtain first audio and first video; playing the first audio and the first video in the browser; receiving an operation command for a cloud application, which is input by a user at the browser, and sending the operation command to the cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs the cloud application, the cloud application and dependent resources required by the cloud application are arranged in containers corresponding to the cloud mobile phone, and the containers are arranged in a server.
8. The terminal device of claim 7, wherein the processor runs the script to: receiving second media data sent by the cloud mobile phone; decoding the second media data to obtain second audio and second video; and playing the second audio and the second video in the browser, wherein the second media data are generated when the cloud mobile phone runs the cloud application according to the operation command.
9. The terminal device of claim 7 or 8, wherein the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;
the first processing core of the processor runs the first browser thread to receive first media data sent by a cloud mobile phone;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video;
a first processing core of the processor runs the second browser thread to play the first audio and first video in the browser; and receiving an operation command for the cloud application, which is input by a user in the browser, and sending the operation command to the cloud mobile phone.
10. The terminal device of claim 9, wherein decoding the first media data comprises:
and when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcedly decode the first media data, wherein the media consumption queue is used for storing the first media data.
11. The terminal device of claim 9, wherein decoding the first media data comprises:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data amount of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data amount of the media data is larger than or equal to the threshold value.
12. A terminal device according to any of claims 9 to 11, wherein the second processing core of the processor, during execution of the third browser thread, invokes binary code written in a low-level language to decode the first media data.
13. A computer readable storage medium comprising instructions which, when run on a terminal device, cause the terminal device to perform the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2021/088845 WO2021218751A1 (en) | 2020-04-30 | 2021-04-22 | Cloud phone-based media data processing method and terminal device |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010367883 | 2020-04-30 | ||
| CN2020103678835 | 2020-04-30 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113672293A CN113672293A (en) | 2021-11-19 |
| CN113672293B true CN113672293B (en) | 2024-04-09 |
Family
ID=78537903
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010444415.3A Active CN113672293B (en) | 2020-04-30 | 2020-05-22 | Media data processing method based on cloud mobile phone and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113672293B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116233520B (en) * | 2023-05-06 | 2023-07-25 | 海马云（天津）信息技术有限公司 | Method and device for transmitting and acquiring video data, server device and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932676A (en) * | 2012-11-14 | 2013-02-13 | 武汉烽火众智数字技术有限责任公司 | Self-adaptive bandwidth transmitting and playing method based on audio and video frequency synchronization |
| CN103546802A (en) * | 2013-10-15 | 2014-01-29 | 深圳Tcl新技术有限公司 | Method for dynamically adjusting cache and display terminal |
| CN105163148A (en) * | 2015-08-31 | 2015-12-16 | 腾讯科技（北京）有限公司 | Video data operating method, mobile phone server, smart watch and smart equipment |
| CN106604042A (en) * | 2016-12-22 | 2017-04-26 | Tcl集团股份有限公司 | Panorama webcasting system and panorama webcasting method based on cloud server |
| CN107295084A (en) * | 2017-06-26 | 2017-10-24 | 深圳水晶石数字科技有限公司 | A kind of video editing system and method based on high in the clouds |
| CN108337560A (en) * | 2017-01-20 | 2018-07-27 | 韩华泰科株式会社 | Media playback and media serving device for playing media in web browser |
| CN109005245A (en) * | 2018-09-07 | 2018-12-14 | 广州微算互联信息技术有限公司 | The use management method and system of cloud mobile phone |
| CN109906107A (en) * | 2016-06-13 | 2019-06-18 | 索尼互动娱乐有限责任公司 | Cloud game based on browser |
| CN110213308A (en) * | 2018-02-28 | 2019-09-06 | 杭州海康威视数字技术股份有限公司 | A kind of method and device of decoding video data |
| CN110780889A (en) * | 2019-10-14 | 2020-02-11 | 广州微算互联信息技术有限公司 | Cloud mobile phone data cloning method and cloud mobile phone data restoring method |
| CN110798702A (en) * | 2019-10-15 | 2020-02-14 | 平安科技（深圳）有限公司 | Video decoding method, device, equipment and computer readable storage medium |
-
2020
- 2020-05-22 CN CN202010444415.3A patent/CN113672293B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932676A (en) * | 2012-11-14 | 2013-02-13 | 武汉烽火众智数字技术有限责任公司 | Self-adaptive bandwidth transmitting and playing method based on audio and video frequency synchronization |
| CN103546802A (en) * | 2013-10-15 | 2014-01-29 | 深圳Tcl新技术有限公司 | Method for dynamically adjusting cache and display terminal |
| CN105163148A (en) * | 2015-08-31 | 2015-12-16 | 腾讯科技（北京）有限公司 | Video data operating method, mobile phone server, smart watch and smart equipment |
| CN109906107A (en) * | 2016-06-13 | 2019-06-18 | 索尼互动娱乐有限责任公司 | Cloud game based on browser |
| CN106604042A (en) * | 2016-12-22 | 2017-04-26 | Tcl集团股份有限公司 | Panorama webcasting system and panorama webcasting method based on cloud server |
| CN108337560A (en) * | 2017-01-20 | 2018-07-27 | 韩华泰科株式会社 | Media playback and media serving device for playing media in web browser |
| CN107295084A (en) * | 2017-06-26 | 2017-10-24 | 深圳水晶石数字科技有限公司 | A kind of video editing system and method based on high in the clouds |
| CN110213308A (en) * | 2018-02-28 | 2019-09-06 | 杭州海康威视数字技术股份有限公司 | A kind of method and device of decoding video data |
| CN109005245A (en) * | 2018-09-07 | 2018-12-14 | 广州微算互联信息技术有限公司 | The use management method and system of cloud mobile phone |
| CN110780889A (en) * | 2019-10-14 | 2020-02-11 | 广州微算互联信息技术有限公司 | Cloud mobile phone data cloning method and cloud mobile phone data restoring method |
| CN110798702A (en) * | 2019-10-15 | 2020-02-14 | 平安科技（深圳）有限公司 | Video decoding method, device, equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113672293A (en) | 2021-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021218751A1 (en) | Cloud phone-based media data processing method and terminal device | |
| US8650501B2 (en) | User interface with preview transitions | |
| US8775844B1 (en) | Dynamic information adaptation for a computing device having multiple power modes | |
| US10839137B2 (en) | Method and apparatus for playing video in independent window by browser, and storage medium | |
| EP3835931A1 (en) | Icon display method and apparatus, and terminal and storage medium | |
| US20110252302A1 (en) | Fitting network content onto a reduced-size screen | |
| RU2602791C2 (en) | Typesetting method, device and system | |
| CN109917995B (en) | Object processing method and terminal equipment | |
| WO2018223558A1 (en) | Data processing method and electronic device | |
| CN111602107B (en) | Display method and terminal during application quitting | |
| US20120131446A1 (en) | Method for displaying web page in a portable terminal | |
| US11314391B2 (en) | Navigation bar controlling method and terminal | |
| US20230297222A1 (en) | Page display switching method, apparatus, storage medium, and electronic device | |
| CN113613064B (en) | Video processing method, device, storage medium and terminal | |
| CN112437318A (en) | Content display method, device and system and storage medium | |
| CN108780400B (en) | Data processing method and electronic equipment | |
| CN113672293B (en) | Media data processing method based on cloud mobile phone and terminal equipment | |
| CN111046265B (en) | Card data display method, device, equipment and storage medium | |
| CN111949150B (en) | Method and device for controlling peripheral switching, storage medium and electronic equipment | |
| CN114090938A (en) | Page processing method and equipment | |
| CN110147555B (en) | Content translation method and terminal equipment | |
| US20150074530A1 (en) | Method for controlling content in mobile electronic device | |
| CN108563712B (en) | Webpage picture viewing method and device and storage equipment | |
| CN109032379B (en) | Language option display method and terminal | |
| CN112528177A (en) | Method for connecting waiting page switching content pages, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220211 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technology Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| GR01 | Patent grant |