CN113672293A - Media data processing method based on cloud mobile phone and terminal equipment - Google Patents
Media data processing method based on cloud mobile phone and terminal equipment Download PDFInfo
- Publication number
- CN113672293A CN113672293A CN202010444415.3A CN202010444415A CN113672293A CN 113672293 A CN113672293 A CN 113672293A CN 202010444415 A CN202010444415 A CN 202010444415A CN 113672293 A CN113672293 A CN 113672293A
- Authority
- CN
- China
- Prior art keywords
- media data
- browser
- mobile phone
- cloud
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 8
- 238000013515 script Methods 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims description 42
- 238000012545 processing Methods 0.000 claims description 42
- 230000015654 memory Effects 0.000 description 23
- 238000007726 management method Methods 0.000 description 20
- 230000006854 communication Effects 0.000 description 15
- 230000008569 process Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 10
- 230000002093 peripheral effect Effects 0.000 description 7
- 238000004590 computer program Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 239000004973 liquid crystal related substance Substances 0.000 description 6
- 239000003795 chemical substances by application Substances 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000006835 compression Effects 0.000 description 4
- 238000007906 compression Methods 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000001133 acceleration Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000009877 rendering Methods 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000007667 floating Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 101100364301 Schizosaccharomyces pombe (strain 972 / ATCC 24843) rsv1 gene Proteins 0.000 description 1
- 101100364302 Schizosaccharomyces pombe (strain 972 / ATCC 24843) rsv2 gene Proteins 0.000 description 1
- 238000000149 argon plasma sintering Methods 0.000 description 1
- 230000007175 bidirectional communication Effects 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 239000003990 capacitor Substances 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000012528 membrane Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 239000002985 plastic film Substances 0.000 description 1
- 229920006255 plastic film Polymers 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 239000000565 sealant Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 125000006850 spacer group Chemical group 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/70—Media network packetisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/764—Media network packet handling at the destination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
- Telephonic Communication Services (AREA)
Abstract
The application provides a media data processing method based on a cloud mobile phone, which is applied to terminal equipment, wherein the terminal equipment comprises a browser and a processor, the browser is loaded with a script, and the processor runs the script to execute the following steps: receiving first media data sent by a cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs a cloud application; decoding the first media data to obtain a first audio and a first video; playing the first audio and the first video in the browser; and receiving an operation command aiming at the cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone. The media data of the game sent by the cloud mobile phone are decoded by loading the script in the browser of the terminal device, and the cross-platform audio and video display of the cloud mobile phone game can be realized without being limited by a personal computer.
Description
Technical Field
The application relates to the technical field of cloud, in particular to a media data processing method based on a cloud mobile phone and a terminal device.
Background
The cloud mobile phone sends the media data to a browser of the personal computer through a network in the process of running the cloud application, a flash plug-in is installed in the browser of the personal computer and used for processing the media data to generate audio and video, and therefore pictures and sound of the cloud application are presented to a user in the browser.
The Flash plug-in is required to be installed in a browser of a personal computer, and the Flash plug-in only supports an operating system of the personal computer and is not suitable for an operating system of a mobile phone.
Disclosure of Invention
The application provides a media data processing method based on a cloud mobile phone and a terminal device, decoding of media data of cloud application sent by the cloud mobile phone is achieved by loading scripts in a browser of the terminal device, the decoding is not limited by a personal computer, and cross-platform audio and video display of the cloud mobile phone application can be achieved.
In a first aspect, a media data processing method based on a cloud mobile phone is provided, where the method is applied to a terminal device, the terminal device includes a browser and a processor, the browser is loaded with a script, and the processor runs the script to perform the following steps:
receiving first media data sent by a cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs a cloud application;
decoding the first media data to obtain a first audio and a first video;
playing the first audio and the first video in the browser;
and receiving an operation command aiming at the cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone.
In the scheme, the media data of the application sent by the cloud mobile phone are decoded by loading the script in the browser of the terminal device, the application is not limited by a personal computer, and the cross-platform audio and video display of the cloud mobile phone application can be realized.
In some possible designs, after sending the operation command to the cloud handset, the processor runs the script to perform the following steps:
receiving second media data sent by the cloud mobile phone, wherein the second media data are generated when the cloud mobile phone runs the cloud application according to the operation command;
decoding the second media data to obtain a second audio and a second video;
and playing the second audio and the second video in the browser.
In the scheme, the operation command of the user on the terminal equipment is also sent to the cloud mobile phone to run, the media data generated by the cloud mobile phone is decoded by loading the script in the browser of the terminal equipment, the resource consumption of the terminal equipment can be further reduced, and meanwhile, the application of the audio and video display of the cross-platform cloud mobile phone can be realized without being limited by a personal computer.
In some possible designs, the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;
a first processing core of the processor runs the first browser thread to receive the first media data;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain a first audio and a first video;
a first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser; and receiving an operation command aiming at the game input by the user in the browser, and sending the operation command to the cloud mobile phone.
In the above scheme, the third browser thread for decoding may occupy the second processing core separately, thereby increasing the decoding speed.
In some possible designs, decoding the first media data includes:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcibly decode the first media data, wherein the media consumption queue is used for storing the first media data.
In the above scheme, the third browser thread may forcibly decode the first media data when the data amount of the media data in the media consumption queue is less than the threshold, and does not need to wait for decoding when the data amount of the media data in the media consumption queue is greater than or equal to the threshold, so that the process of decoding by the third browser thread is accelerated, and the time delay is effectively reduced.
In some possible designs, decoding the first media data includes:
when the data volume of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data volume of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data volume of the media data is larger than or equal to the threshold value.
In the scheme, the data quantity in the media data in the media consumption queue is increased by adding the dummy frame so as to reach the threshold value for decoding as soon as possible, the process of decoding by the third browser thread is accelerated, and the time delay is effectively reduced.
In some possible designs, during the process of the second processing core of the processor running the third browser thread, binary code written in a low-level language is called to decode the first media data.
In the above scheme, decoding of the first media data is expedited by calling binary code written in a low-level language.
In a second aspect, a terminal device is provided, where the terminal device includes a browser and a processor, the browser is loaded with a script, and the processor executes the script to: receiving first media data sent by a cloud mobile phone; decoding the first media data to obtain a first audio and a first video; playing the first audio and the first video in the browser; receiving an operation command aiming at cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs the cloud application.
In some possible designs, the processor executes the script to: receiving second media data sent by the cloud mobile phone; decoding the second media data to obtain a second audio and a second video; and playing the second audio and the second video in the browser, wherein the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command.
In some possible designs, the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;
a first processing core of the processor runs the first browser thread to receive first media data sent by a cloud mobile phone;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video;
a first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser; and receiving an operation command aiming at the cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone.
In some possible designs, decoding the first media data includes:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcibly decode the first media data, wherein the media consumption queue is used for storing the first media data.
In some possible designs, decoding the first media data includes:
when the data volume of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data volume of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data volume of the media data is larger than or equal to the threshold value.
In some possible designs, during the process of the second processing core of the processor running the third browser thread, binary code written in a low-level language is called to decode the first media data.
In a third aspect, a computer-readable storage medium is provided, comprising instructions which, when run on a terminal device, cause the terminal device to perform the method according to any of the first aspect.
In a fourth aspect, there is provided a terminal device comprising a processor and a memory, the processor executing code in the memory to perform the method according to any one of the first aspect.
In a fifth aspect, there is provided a computer program product comprising a computer program that, when read and executed by a computing device, causes the computing device to perform the method of any of the first aspects.
In the scheme, the media data of the application sent by the cloud mobile phone are decoded by loading the script in the browser of the terminal device, the application is not limited by a personal computer, and the cross-platform audio and video display of the cloud mobile phone application can be realized.
Drawings
Fig. 1 is a schematic structural diagram of a media data processing system of a cloud handset to which the present application relates;
2A-2B illustrate exemplary user interfaces for two different terminal devices to open a game page in a browser to play a game;
3A-3B illustrate a comparison of an original browser structure and a modified browser structure;
fig. 4 is a flowchart of a media data processing method based on a cloud mobile phone according to the present application;
fig. 5 is a schematic structural diagram of a WebSocket protocol packet related to the present application;
fig. 6 is a schematic structural diagram of a terminal device provided in the present application;
fig. 7 is a schematic structural diagram of another terminal device provided in the present application.
Detailed Description
Referring to fig. 1, fig. 1 is a schematic structural diagram of a media data processing system of a cloud handset according to the present application. The media data processing system of the cloud mobile phone of the present embodiment includes a terminal device 110 and a cloud data center 120. Among other things, the terminal device 110 may communicate with the cloud data center 120 through a network.
The terminal device 110 may be a smartphone, a cloud terminal, or a thin terminal, etc. In a specific embodiment, the terminal device 110 may be a terminal device used with a cloud mobile phone, that is, the terminal device 110 has a strong network communication capability, but may weaken the computing capability and the storage capability. The terminal device 110 is installed with a browser, and a browser engine used by the browser may be Trident, Gecko, Presto, Webkit, or the like, which is not limited herein. The operating system supported by terminal device 110 may be an IOS system, an Android system (Android), a macOS system, a Windows system, a Linux system, and the like. In the game system, different terminal devices 110 may respectively support different operating systems, for example, terminal device 1 supports an android system, terminal device 2 supports a macOS system, terminal device 3 supports an IOS system, terminal device 4 supports a Windows system, and terminal device 5 supports a Linux system. Of course, in the game system, different terminal devices 110 may support the same operating system, and are not limited in particular.
Cloud data center 120 may include a service node 121 and a cloud handset management node 125. The service node 121 may be a general physical server, for example, a physical server, such as an X86 server or an ARM server. The cloud phone management node 125 may be a server used for managing a cloud phone in a public cloud, specifically, a general physical server, or a Virtual Machine (VM) implemented based on a general physical server and combining with an NFV technology, where the VM refers to a complete computer system that has a complete hardware system function and operates in a completely isolated environment through software simulation, and the present application is not limited specifically.
In a specific embodiment, the service node 121 includes a cloud handset management agent node 124, hardware resources, an Operating System (OS) 122, and at least one cloud handset 123, and each cloud handset 123 shares the hardware resources and the Operating System with other cloud handsets on the server in the form of a Container (Container). The cloud server is essentially used for transferring an application on a mobile phone to a virtual mobile phone on a public cloud to run.
The hardware resources may include computing resources, storage resources, and network resources, such as a processor, a memory, a network card, a router, a database, and the like, and may also include hardware resources that may be needed by other cloud handsets, which is not specifically limited in this application.
The operating system 122 may be an operating system suitable for a cloud mobile phone, such as an Android (Android) operating system, which is not specifically limited in this application. It should be noted that the operating system 122 may be a complete operating system, or may be an operating system obtained by modifying individual driver modules of the complete operating system in order to adapt to the operation mode of the service node 121, and the application is not limited in particular.
The cloud phone 123 shares hardware resources and an operating system with other cloud phones on the server in the form of a container, each container may include a cloud application required by the cloud phone and a dependent resource required by each cloud application, and the dependent resource required by the cloud application may be an operating system core library and a system library, such as a function library, a three-dimensional graphics processing library (e.g., OpenGLES), a Media library (Media Libraries), an Input Manager (Input Manager), and the like. In one embodiment, the container may include the gaming application and the dependent resources required by the gaming application. Each container in which the cloud handsets 123 are located uses hardware resources and the operating system 122, but the hardware resources are not monopolized, and the hardware resources and the operating system 122 are shared with containers in which other cloud handsets are located. In other words, each container does not have its own kernel, and the application processes within the container run directly on the kernel of the service node 121.
The cloud mobile phone management agent node 124 may be configured to monitor and collect service information of the service node 121, report the service information to the cloud mobile phone management node 125 in real time, and further configured to create a container on the service node 121 according to the received specification information of the cloud mobile phone, and divide resources from hardware resources to set hardware and software required by the specification information of the cloud mobile phone for the container.
The process of creating a cloud handset at the cloud data center 120 may be: the terminal device 110 transmits cloud handset creation information to the cloud handset management node 125. The cloud mobile phone management node 125 determines that the cloud mobile phone 1 is created by the server 1 and the cloud mobile phone 2 is created by the server 2 according to the specification information of the cloud mobile phone 123 in the cloud mobile phone creation information and the service information sent by each cloud mobile phone management agent node 124, so that the cloud mobile phone management agent node 1 on the server 1 can receive the specification information of the cloud mobile phone 1 sent by the cloud mobile phone management node 125 and complete creation of the cloud mobile phone 1 according to the specification information, and the cloud mobile phone management agent node 2 on the server 2 can receive the specification information of the cloud mobile phone 2 sent by the cloud mobile phone management node 125 and complete creation of the cloud mobile phone 2 according to the specification information. It can be understood that the cloud mobile phone management node 125 and the cloud mobile phone management proxy node 124 disposed on each service node 121 may be implemented based on a container management system such as Kubernetes (K8S), Docker, and the like, and the present application is not limited specifically.
On the basis of the media data processing system of the cloud mobile phone shown in fig. 1, a user can play a game with low delay through a browser in a terminal device without installing the cloud application in the terminal device. The following description will take the terminal devices as a smart phone and a thin terminal, respectively, as examples. For the sake of simplicity, the following examples are described by taking the cloud application as the game application, but in other examples, the cloud application may be other applications, and should not be limited specifically here.
The terminal equipment is a smart phone. FIG. 2A illustrates an exemplary user interface for opening a game page in a browser to play a game. As shown in the upper left corner of fig. 2A: the user interface may include: status bar, calendar indicator, weather indicator, tray with common application icons, navigation bar, and other application icons. Wherein:
the status bar may include: one or more signal strength indicators for mobile communication signals (which may also be referred to as cellular signals), one or more signal strength indicators for wireless fidelity (Wi-Fi) signals, a battery status indicator, and a time indicator.
The calendar indicator may be used to indicate the current time, such as the date, day of the week, time division information, and the like.
The weather indicator can be used for indicating weather types, such as cloudy sunny weather, light rain and the like, and can also be used for indicating information such as temperature and the like.
A tray with common application icons may show: phone icon, contact icon, short message image, camera icon.
The navigation bar may include: a return key, a home screen key, a multitasking key, etc. When it is detected that the user clicks a return key, the smart phone may display a page previous to the current page. When detecting that the user clicks a main screen key, the smart phone displays a main interface. When it is detected that the user clicks a multi-task key, the smartphone may display the task that the user has recently opened. The names of the navigation keys can be other keys, and the application does not limit the names. Not limited to virtual keys, each navigation key in the navigation bar may also be implemented as a physical key.
Other application icons may be, for example: an icon for mutual communication, an icon for gallery, an icon for music, an icon for application, an icon for mailbox, an icon 212 for browser, an icon for memo, and an icon for setting. The user interface 210 may also include a page indicator. Other application icons may be distributed across multiple pages, and a page indicator may be used to indicate which page the user is currently browsing for applications in. The user may slide the area of the other application icons from side to browse the application icons in the other pages.
In some embodiments, the user interface 210 illustratively shown in FIG. 2A may be a Home screen (Home Screen). It is understood that fig. 2A is only an exemplary illustration of a user interface on a smartphone and should not be construed as limiting the embodiments of the present application.
Illustratively, the user may click on the browser icon 212 on the user interface 210, the smartphone detects the user action, and in response to the user action, the smartphone opens the browser 220.
As shown in the upper right hand corner of fig. 2A: browser 220 may include: a search box 222, a scan control, a tab bar, a plurality of recommended categories, and a tray of common controls, wherein:
the search box 222 may be used to search for game pages that match the characters entered by the user.
The scanning control can be used for scanning the two-dimensional code or the bar code, so that the game webpage corresponding to the two-dimensional code or the bar code is opened quickly.
Title bars may be used to indicate various types of subject matter, such as recommendations, games, hotspots, and entertainment, among others. The presentation of the subtitle bar may be in the form of text information, icons, or other forms.
The control 302 can receive a user operation (e.g., a touch operation), and in response to detecting the user operation, the anchor terminal 120 can display a page to log in or switch a cloud live account.
The tray of common controls may show: a previous page control icon, a next page control icon, a home page control icon, an open web page icon, and the like. The control icons can all accept user operations (for example, touch operations), and corresponding to the detected user operations, the smart phone can display a page of response, specifically, the previous control icon can be used for displaying a previous webpage, the previous control icon can be used for displaying a next webpage, the home page control icon can be used for a home page of a browser, the first priority of the browser can be user-defined, or a navigation page preset by the browser, and the opened webpage icon can be used for displaying multiple pages which are opened currently.
The plurality of recommendation directories may for example: a title is displayed with a display picture 224, the display title showing information related to the game, and the display picture 224 showing a picture related to the game. The display graphic 224 is associated with a hyperlink on the game page, and when the user clicks on the display graphic 224, the browser can jump to the game page 230.
As shown in the lower diagram of fig. 2A, the game page 230 is used to display screen contents of a game and play a sound of the game. The game page may include a game title, a game screen, and game controls.
The game title is optional, i.e., the user may choose to show the game title or not, or the game page may show the game tab by default, or not show the game title by default.
The image quality, definition, resolution, size and color of the game picture can be set according to the needs of the user, for example, the user can set the picture of the game to be full screen or not. In addition, the game screen can be arranged on other applications in a floating manner, so that the user can play the game and process other things. The game screen may be a two-dimensional screen or a three-dimensional screen.
The specific process that a user opens a game page in a browser to play a game is as follows: a User clicks with a hand point on a browser icon 212 on a User Interface 210 (UI) of the smartphone, opening a browser 220. The user inputs a Uniform Resource Locator (URL) of the game web page 230 in the search bar 222 of the browser 120, or clicks on a hyperlink 224 recommended by the browser 220, which points to the game web page 230 and includes the URL of the game web page 230, thereby opening the game web page 230. The user may then play the game on the game page 230.
It is understood that fig. 2A is only an exemplary illustration of various user interfaces for opening a game page on a smart phone, and should not be construed as a limitation to the embodiments of the present application.
The terminal equipment is a thin terminal. FIG. 2B illustrates an exemplary desktop with a game page open in a browser for playing a game. As shown in the upper left corner of fig. 2B: the desktop may include: calendar indicator, application tray opened, start button.
Wherein:
a calendar indicator, located in the lower right corner of the desktop, may be used to indicate the current time, e.g., date, day of week, time division information, etc.
An opened application tray is located at the bottom of the desktop and is used for indicating the opened application, for example, when the browser is opened, the icon of the browser is displayed in the opened application tray, and when the browser is closed, the icon of the browser disappears from the opened application tray. When the browser is opened multiple times, icons of multiple overlapping browsers may be displayed to represent that the browser is opened multiple times.
A start button, located in the lower left corner of the desktop, displays one or more levels of menus, each of which may display the entries for multiple applications, e.g., control panel, devices and printers, default programs, security and help, etc.
A plurality of application icons may be displayed on the desktop, such as: an icon of a cloud application, an icon of a camera, an icon of a time, an icon of a calendar, an icon of an application, an icon of a browser 310, an icon of a setting, and the like.
Illustratively, as shown in the upper left diagram of fig. 2B, a user may click on an icon 310 of a browser on the desktop, the thin terminal detects the user operation, and in response to the user operation, the thin terminal opens the browser 320.
As shown in the upper right hand corner of fig. 2B: browser 320 may include: a search box 322, a previous page control, a next page control, a reload control, a title bar, and a plurality of recommendation categories, wherein:
the search box 322 may be used to search for game pages that match the characters entered by the user.
The previous page control may be configured to display a previous page of a displayed web page, the previous page control icon may be configured to display a next page of the displayed web page, and the reload control may be configured to refresh the current page to reload the current page.
Title bars may be used to indicate various types of subject matter, such as recommendations, games, hotspots, and entertainment, among others. The presentation of the subtitle bar may be in the form of text information, icons, or other forms.
The plurality of recommendation directories may for example: a title is displayed with a display picture 324, the display title showing information related to the game, and the display picture 324 showing a picture related to the game. The display icon 324 is associated with a hyperlink on the game page, and when the user clicks on the display icon 324, the browser can jump to the game page 330.
As shown in the lower diagram of fig. 2B, the game page 330 is used to display screen contents of a game and play a sound of the game. The game page may include a game title, a game screen, and game controls.
The game title is optional, i.e., the user may choose to show the game title or not, or the game page may show the game tab by default, or not show the game title by default.
The image quality, definition, resolution, size and color of the game picture can be set according to the needs of the user, for example, the user can set the picture of the game to be full screen or not. In addition, the game screen can be arranged on other applications in a floating manner, so that the user can play the game and process other things. The game screen may be a two-dimensional screen or a three-dimensional screen.
The specific process that a user opens a game page in a browser to play a game is as follows: the user mouse clicks on a browser icon 312 on the interface 310 of the display, thereby opening the browser 320. The user inputs a Uniform Resource Locator (URL) of the game web page 330 in the search bar 322 of the browser 320, or clicks on a hyperlink 324 recommended by the browser 320, which points to the game web page 330 and includes the URL of the game web page 330, thereby opening the game web page 330. The user may then play the game on the game page 330. The interface 310 may be configured to display a music application icon, a mail application icon, a gallery application icon, a setting icon, and the like, in addition to the browser icon, and is not limited in detail here.
It is to be understood that fig. 2B is merely exemplary of various user interfaces for opening a game page on a thin terminal and should not be construed as limiting embodiments of the present application.
In addition to the above examples, in practical applications, the terminal device may also be a virtual device or the like, and is not specifically limited herein.
The internal structure of the browser related to the present application will be specifically described below.
As shown in fig. 3A, taking the browser as WebKit as an example, the browser includes a rendering engine and a JavaScript (js) engine, where the rendering engine is configured to convert the resource files (HTML/CSS/JavaScript text and corresponding resource files) into a result visible to the user. Rendering engines typically include CSS, SVG, layout, render tree, HTML, DOM, and instector, among others. The JS engine is used for reading JavaScript in the file, processing and executing the JavaScript, and the JS engine is single-threaded. In addition, the Webkit also comprises a WebKit embedded interface and corresponding WebKit binding, a WebKit2 embedded interface and corresponding WebKit2 binding, and WebKitPorts. Among them, WebKitPorts may include mesh segments, video, text, hardware acceleration, picture decoding, and so on. In addition, a WebKit-dependent database and operating system may also be included. The database may include a 2D graphics library, a 3D graphics library, a web library, a storage, an audio library, a video library, and so on.
As shown in the dashed box portion of fig. 3B, a plurality of browser threads including a first browser thread, a second browser thread, and a third browser thread may be added to the JS engine of the existing browser. The browser thread may be WebWorker. The WebWorker is used for enabling the browser to run a script operation in a background thread (worker) separated from the main execution thread, so that a multi-thread running environment is created for the JS engine, namely, the main thread of the browser creates the browser thread, tasks are distributed to the browser thread, the browser thread runs while the main thread runs, the browser thread does not interfere with each other, and a result is returned to the main thread after the browser thread runs. The first browser thread can also be called a network thread, the network thread adopts a WebSocket protocol, and bidirectional communication of game data (including media data and control data) between the terminal device and the cloud mobile phone can be achieved. The second browser thread may also be referred to as a master thread, which is responsible for the overall running of the game. The third browser thread, which may also be referred to as a decode thread, is responsible for decoding the media data for the game.
The following describes in detail an implementation process of playing a game with low latency by a user through a browser in a terminal device, with reference to the browser shown in fig. 3B. As shown in fig. 4, fig. 4 is a flowchart of a game running method provided by the present application. The game running method in the present embodiment includes the steps of:
s101: in the case that a user inputs a URL of a game webpage in a search bar of a browser or clicks a hyperlink recommended by the browser, the browser of the terminal device generates a media data request, wherein the media data request is used for generating media data of the game webpage, the media data request includes the URL, and the URL corresponds to a game.
In a specific embodiment, the browser engine used by the browser may be Trident, Gecko, Presto, Webkit, or the like, and is not limited herein.
In a specific embodiment, the terminal device detects whether the user inputs the URL of the game webpage in the search bar or whether the user clicks the hyperlink recommended by the browser through a main control thread in the browser. And when the URL of the game webpage is input in the search bar or a hyperlink recommended by the browser is clicked, the main control thread generates a media data request and sends the media data request to the network thread by a method of sharing a memory or pipeline communication and the like.
S102: and the browser of the terminal equipment sends the media data request to the cloud mobile phone. Correspondingly, the cloud mobile phone receives a media data request sent by the terminal equipment.
In a specific embodiment, a network thread in the browser sends a media data request to the cloud mobile phone through a WebSocket protocol. The network thread is a shared Webwork, so that the network thread can realize data transceiving between the browser and the cloud mobile phone.
S103: the cloud mobile phone runs the game application to generate first media data.
In a specific embodiment, after receiving a media data request sent by a browser of a terminal device, a cloud mobile phone obtains a URL of a game webpage from the media data request, determines a corresponding game application according to the URL of the game webpage, runs the game application, and generates first media data. Then, the cloud mobile phone encodes the generated first media data by an encoding method, so that the number of data transmission is reduced.
S104: and the cloud mobile phone sends the first media data to the terminal equipment. Correspondingly, the terminal equipment receives first media data sent by the cloud mobile phone.
In a specific embodiment, a network thread in a browser of a terminal device receives first media data sent by a cloud mobile phone through a WebSocket protocol. After monitoring first media data sent by a network thread, a main control thread in the browser pushes the first media data to a media data queue. The main control thread in the browser sends the data in the media data queue to the decoding thread for decoding in an instant mode through methods of shared memory or pipeline communication and the like, so that the time delay of the game is effectively reduced.
S105: the terminal device decodes the first media data to obtain a first audio and/or a first video.
In a specific embodiment, a decoding thread in a browser in the terminal device decodes the first media data to obtain the first audio and/or the first video. Then, the decoding thread in the browser sends the first media data to the main control thread through a shared memory or pipeline communication or the like. The decoding method adopted by the decoding thread corresponds to the encoding method adopted by the cloud mobile phone, when the encoding method adopted by the cloud mobile phone is lossless compression, the decoding method adopted by the terminal device is lossless compression, and when the encoding method adopted by the cloud mobile phone is lossy compression, the decoding method adopted by the terminal device is lossy compression.
S106: and the terminal equipment plays the first audio and/or the first video in the first game page generated by the browser.
S107: and receiving an operation command of the game input by the user on the game page of the browser under the condition that the user operates the game on the game page of the browser, and converting the operation command into game operation which can be identified by the cloud mobile phone.
In a specific embodiment, a main control thread in a browser of a terminal device receives an operation command of a game input by a user on a game page of the browser through an external input device. The external input device may be a touch screen, a mouse, a keyboard, a camera, a scanner, a light pen, a handwriting input board, a joystick, a microphone, or the like. The operation commands of the game may include forward, backward, left turn, right turn, left move, right move, jump, sit down, stand up, weapon reception/retrieval, walk/run, and the like. The definition of the operation by the terminal device and the cloud mobile phone is different, so that the browser needs to convert an operation command input by a user into a game operation which can be recognized by the cloud mobile phone. Then, the main control thread in the browser sends the game operation to the network thread through a method of sharing a memory or pipeline communication, and the like.
S108: and the terminal equipment sends game operation to the cloud mobile phone. Correspondingly, the cloud mobile phone receives the game operation sent by the terminal equipment.
In a specific embodiment, the terminal device sends the game operation to the cloud mobile phone through a WebSocket protocol.
S109: the cloud mobile phone runs the game application according to the game operation, and therefore second media data are generated.
In a specific embodiment, the cloud mobile phone continues to run the game application, and generates the second media data according to the game operation. Then, the cloud mobile phone encodes the generated second media data by an encoding method, thereby reducing the number of data transmission.
S110: and the cloud mobile phone sends the second media data to the terminal equipment. Correspondingly, the terminal equipment receives second media data sent by the cloud mobile phone.
In a specific embodiment, a network thread in a browser of the terminal device receives second media data sent by the cloud mobile phone through a WebSocket protocol. And after monitoring second media data sent by the network thread, a main control thread in the browser pushes the second media data to a media data queue. The main control thread in the browser sends the data in the media data queue to the decoding thread for decoding in an instant mode through methods of shared memory or pipeline communication and the like, so that the time delay of the game is effectively reduced.
S111: the terminal device decodes the second media data to obtain second audio and/or second video.
In a specific embodiment, a decoding thread in a browser in the terminal device decodes the second media data to obtain the second audio and/or the second video. Then, the decoding thread in the browser sends the second media data to the main control thread through a shared memory or pipeline communication or the like.
S112: and the terminal equipment plays the second audio and/or the second video in the second game page generated by the browser.
In a specific embodiment, the media data request, the first audio and/or the first video, the game operation, and the second audio and/or the second video are encapsulated in a WebSocket protocol format. As shown in fig. 5, the WebSocket protocol format includes a FIN of 1 bit, RSV1, RSV2, RSV3, Opcode of 4 bits, MASK (MASK) of 1 bit, data payload length (payload len) of 7 bits, MASK-key value (Masking-key) of 0 to 4 bytes, and payload data (payload) of multiple bytes. Here, the media data request, the first audio and/or first video, the game operation, and the second audio and/or second video payload are in payload data (payload).
In a specific embodiment, the terminal device receives first media data sent by the cloud mobile phone, and needs to decode the first media data as soon as possible into a first audio and/or a first video, and play the first audio and/or the first video in a first game page generated by the browser. The terminal device can be divided into at least the following two modes according to different browser capabilities:
in the first way, the terminal device decodes the first media data into the first audio and/or the first video as soon as possible through a video (video) component, and plays the first audio and/or the first video in the first game page generated by the browser. Here, the video (video) component may be implemented using modules in WebKitPorts, e.g., video, hardware acceleration, and so on. The terminal device generally plays the video in the browser through a video (video) component, and therefore, the video (video) component is not sensitive to the time delay. After the network thread receives the media data, the video (video) component buffers the media data, and decodes the media data until the amount of buffered media data exceeds a threshold value, and plays the media data in the browser to ensure the smoothness of playing, but this causes a very large delay. However, the sensitivity of the game to latency is very high and cannot accept such a large latency. To solve the problem of large latency, video (video) components may be improved to reduce latency. Specifically, improving a video (video) component includes at least the following aspects: (1) the video (video) component issues a forcing command instructing the decoding thread to immediately decode the received first media data to obtain a first video and a first audio, and the main control thread immediately plays the first video and the first audio in the browser without buffering the first media data. (2) And the video (video) component instructs the main control thread to add a pseudo frame in the media consumption queue so that the media data in the buffered media consumption queue reaches a threshold value as soon as possible, and when the threshold value is reached, the video component instructs the decoding thread to decode the first media data so as to obtain a first video and a first audio, and instructs the main control thread to play the first video and the first audio in the browser. The dummy frame may be pre-stored, may be a data frame completely unrelated to the first media data, and may also be referred to as a transparent frame. (3) After decoding the plurality of frames of video and audio of the first media data, a video (video) component instructs a main control thread to force the video and audio of the last frame of the decoded first video and first audio to be played, so as to reduce latency. To further reduce latency, a video component may also process media data in a frame-skipping manner, which may be sampled when processed in a frame-skipping manner, and closer data frames may be sampled more densely and farther data frames may be sampled more sparsely.
In the second way, the terminal device decodes the first media data into the first audio and/or the first video as soon as possible through the Canvas component, and plays the first audio and/or the first video in the first game page generated by the browser. Here, the Canvas component may be implemented using modules in WebKitPorts, e.g., video, hardware acceleration, and the like. The decoding thread is decoded by a file written by JavaScript semantics, the decoding efficiency is very low, for this reason, the Canvas component can instruct the decoding thread to call binary codes written by a low-level language, for example, WebAssembly, for decoding, so as to improve the decoding rate, and in addition, the decoding thread can independently occupy the kernel of one processor for decoding, and the decoding rate can also be obviously improved.
The method of the embodiments of the present application is described above in detail, and in order to better implement the above-mentioned aspects of the embodiments of the present application, the following also provides related apparatuses for implementing the above-mentioned aspects.
The terminal device provided by the application can include a processor and a browser as shown in fig. 3B the browser includes a JS engine, the browser is loaded with a script, the processor runs the script to make the JS engine generate a first browser thread, a second browser thread and a third browser thread.
A first processing core of the processor runs the first browser thread to receive first media data sent by a cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs a game;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain a first audio and a first video;
a first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser;
and the first processing core of the processor runs the second browser thread to receive an operation command aiming at the game and input by a user in the browser, and sends the operation command to the cloud mobile phone.
In some possible designs, the first processing core of the processor runs the first browser thread to receive second media data sent by a cloud mobile phone, wherein the second media data is generated when the cloud mobile phone runs a game according to the operation command;
a second processing core of the processor runs the third browser thread to decode the second media data to obtain a second audio and a second video;
a second processing core of the processor runs the third browser thread to play the second audio and second video in the browser.
In some possible designs, the second processing core of the processor runs the third browser thread to force decoding of the first media data when a data amount of the media data in a media consumption queue is less than a threshold, wherein the media consumption queue is to store the first media data.
In some possible designs, when the data amount of the media data in the media consumption queue is less than a threshold, the second processing core of the processor executes the third browser thread to add a dummy frame in the media consumption queue, so that the data amount of the media data including the dummy frame in the media consumption queue is greater than or equal to the threshold, and decodes the first media data if the data amount of the media data is greater than or equal to the threshold.
In some possible designs, during the process of the second processing core of the processor running the third browser thread, binary code written in a low-level language is called to decode the first media data.
It should be understood that the internal unit modules of the terminal device shown in fig. 3B may also have various partitions, and each module may be a software module, a hardware module, or a part of a software module and a part of a hardware module, which is not limited in this application. Fig. 3B is an exemplary division, which is not limited in this application.
An implementation manner of the terminal device according to the embodiment of the present invention is described below. The terminal device supports a browser running a plurality of web tags. The browser may use different input and output devices, respectively, for example, the input device may be a keyboard and a mouse, and the output device may be a display screen. Alternatively, the browser uses a common input-output device: a touch screen. The touch screen is used for receiving touch operation of a user and displaying output content of the browser.
Fig. 6 is a block diagram of an implementation of a terminal device. As shown in fig. 6, the terminal device may be a thin terminal, including: a host 410, an output device 420, and an input device 430.
The host 410 may be integrated including: one or more processors, a clock module, and a power management module. The clock module integrated in the host 410 is mainly used to generate the clocks required for data transmission and timing control for the processor. The power management module integrated in the host 410 is mainly used to provide stable, high-precision voltages to the processor, the output device 420, and the input device 430. The host 410 also incorporates memory for storing various software programs and/or sets of instructions. In particular implementations, the memory may include high speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory may not need to have an operating system installed and a large number of applications, but rather the memory need only store some of the necessary applications, such as input and output programs that may be used to communicate with one or more input devices or output devices. And the communication program can be used for communicating with the cloud mobile phone. The memory can also store a user interface program, and the user interface program can vividly display the content image of the browser through a graphical operation interface and receive the control operation of a user on the browser through input controls such as a menu, a dialog box and a key.
The output device 420 mainly includes a Display, which may include a Cathode Ray Tube (CRT) Display, a Plasma Display Panel (PDP), a Liquid Crystal Display (LCD), and the like. Taking the display as an LCD as an example, the liquid crystal display includes a liquid crystal panel and a backlight module, wherein the liquid crystal panel includes a polarizing film, a glass substrate, a black matrix, a color filter, a protective film, a common electrode, an alignment layer, a liquid crystal layer (liquid crystal, spacer, sealant), a capacitor, a display electrode, a prism layer, and a light scattering layer. The backlight module includes: an illumination light source, a reflection plate, a light guide plate, a diffusion sheet, a brightness enhancement film (prism sheet), a frame, and the like.
The input device 430 may include a keyboard and a mouse. The keyboard and the mouse are the most common and the most main input devices, English letters, numbers, punctuations and the like can be input into a computer through the keyboard, so that commands, input data and the like are sent to the computer, and the vertical and horizontal coordinate positioning can be quickly carried out through the mouse, so that the operation is simplified. The keyboard may include a Mechanical keyboard, a plastic film keyboard (Mechanical), a conductive rubber keyboard (Membrane), a contactless electrostatic capacitance keyboard (capacities), and the like, and the mouse may include a rolling ball mouse, an optical mouse, a wireless mouse, and the like.
The input device 430 is configured to receive first media data sent by a cloud mobile phone, where the first media data is generated when the cloud mobile phone runs a cloud application;
the host 410 is configured to decode the first media data to obtain a first audio and a first video;
an output device 420 for playing the first audio and first video in the browser;
the input device 430 is used for receiving an operation command for the cloud application, which is input by a user in the browser, and sending the operation command to the cloud mobile phone through the communication module.
For the sake of simplicity, the terminal device shown in fig. 6 is not described in detail, and please refer to fig. 1 to fig. 5 and the related description, which will not be described herein.
Fig. 7 is a block diagram of another implementation of a terminal device. As shown in fig. 7, the terminal device may be a smart phone, including: baseband chip 510, memory 515, including one or more computer-readable storage media, Radio Frequency (RF) module 516, peripheral system 517. These components may communicate over one or more communication buses 514.
The peripheral system 517 is mainly used to implement the interaction function between the terminal device 510 and the user/external environment, and mainly includes an input/output device of the terminal device. In a specific implementation, the peripheral system 517 may include: a touch screen controller 518, a camera controller 519, an audio controller 520, and a sensor management module 521. Wherein each controller may be coupled to a respective peripheral device, such as a touch screen 523, a camera 524, audio circuitry 525, and sensors 526. In some embodiments, a gesture sensor of sensors 526 may be used to receive gesture control operations of user input. The pressure sensor in the sensor 526 may be disposed below the touch screen 523 and may be configured to collect a touch pressure applied to the touch screen 523 when a user inputs a touch operation through the touch screen 523. It should be noted that the peripheral system 517 may also include other I/O peripherals.
The baseband chip 510 may integrally include: one or more processors 511, a clock module 512, and a power management module 513. The clock module 512 integrated in the baseband chip 510 is mainly used for generating clocks required for data transmission and timing control for the processor 511. The power management module 513 integrated in the baseband chip 510 is mainly used to provide stable and high-precision voltages for the processor 511, the rf module 516 and peripheral systems.
The memory 515 is used to store various software programs and/or sets of instructions. In particular implementations, the memory may include high speed random access memory and may also include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory may not need to have an operating system installed and a large number of applications, but rather the memory need only store some of the necessary applications, such as input and output programs that may be used to communicate with one or more input devices or output devices. And the communication program can be used for communicating with the cloud mobile phone. The memory may also store a desktop program.
A Radio Frequency (RF) module 516 is used for receiving and transmitting radio frequency signals, and mainly integrates a receiver and a transmitter of a terminal device. Radio Frequency (RF) module 516 communicates with a communication network and other communication devices via radio frequency signals. In particular implementations, the Radio Frequency (RF) module 516 may include, but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card 5161, a storage medium, and the like. In addition, the radio frequency module can also comprise a WIFI 5162, a Bluetooth and other short-distance wireless communication module. In some embodiments, the Radio Frequency (RF) module 516 may be implemented on a separate chip.
A Radio Frequency (RF) module 516 is configured to receive first media data sent by a cloud handset, where the first media data is generated when the cloud handset runs a cloud application;
the baseband chip 510 is configured to decode the first media data to obtain a first audio and a first video;
the touch screen 523 is used for playing a first video in the browser, and the audio circuit 525 is used for playing a first audio;
the touch screen 523 is configured to receive an operation command for the cloud application, which is input by a user in the browser, and send the operation command to the cloud handset through the Radio Frequency (RF) module 516.
For the sake of simplicity, the terminal device shown in fig. 7 is not described in detail, and please refer to fig. 1 to fig. 5 and the related description, which will not be described herein.
In the scheme, the media data of the game sent by the cloud mobile phone are decoded by loading the script in the browser of the terminal device, the method is not limited by a personal computer, and the cross-platform audio and video display of the cloud mobile phone game can be realized.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present application can be implemented by software plus necessary general-purpose hardware, and certainly can also be implemented by special-purpose hardware including special-purpose integrated circuits, special-purpose CPUs, special-purpose memories, special-purpose components and the like. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions may be various, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, the implementation of a software program is more preferable. Based on such understanding, the technical solutions of the present application may be substantially embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, an exercise device, or a network device) to execute the method according to the embodiments of the present application.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, training device, or data center to another website site, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a training device, a data center, etc., that incorporates one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
Claims (13)
1. The media data processing method based on the cloud mobile phone is applied to terminal equipment, the terminal equipment comprises a browser and a processor, the browser is loaded with a script, and the processor runs the script to execute the following steps:
receiving first media data sent by a cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs a cloud application;
decoding the first media data to obtain a first audio and a first video;
playing the first audio and the first video in the browser;
and receiving an operation command aiming at the cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone.
2. The method of claim 1, wherein after sending the operation command to the cloud handset, the processor runs the script to perform the steps of:
receiving second media data sent by the cloud mobile phone, wherein the second media data are generated when the cloud mobile phone runs the cloud application according to the operation command;
decoding the second media data to obtain a second audio and a second video;
and playing the second audio and the second video in the browser.
3. The method of claim 1 or 2, wherein the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;
a first processing core of the processor runs the first browser thread to receive the first media data;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain a first audio and a first video;
a first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser; and receiving an operation command aiming at the game input by the user in the browser, and sending the operation command to the cloud mobile phone.
4. The method of claim 3, wherein decoding the first media data comprises:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcibly decode the first media data, wherein the media consumption queue is used for storing the first media data.
5. The method of claim 3, wherein decoding the first media data comprises:
when the data volume of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data volume of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data volume of the media data is larger than or equal to the threshold value.
6. The method of any of claims 3 to 5, wherein during execution of the third browser thread by the second processing core of the processor, binary code written in a low-level language is invoked to decode the first media data.
7. A terminal device, comprising a browser and a processor, wherein the browser is loaded with a script, and the processor executes the script to: receiving first media data sent by a cloud mobile phone; decoding the first media data to obtain a first audio and a first video; playing the first audio and the first video in the browser; receiving an operation command aiming at cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone, wherein the first media data are generated when the cloud mobile phone runs the cloud application.
8. The terminal device of claim 7, wherein the processor executes the script to: receiving second media data sent by the cloud mobile phone; decoding the second media data to obtain a second audio and a second video; and playing the second audio and the second video in the browser, wherein the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command.
9. The terminal device of claim 7 or 8, wherein the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;
a first processing core of the processor runs the first browser thread to receive first media data sent by a cloud mobile phone;
a second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video;
a first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser; and receiving an operation command aiming at the cloud application and input by a user in the browser, and sending the operation command to the cloud mobile phone.
10. The terminal device of claim 9, wherein decoding the first media data comprises:
when the data amount of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to forcibly decode the first media data, wherein the media consumption queue is used for storing the first media data.
11. The terminal device of claim 9, wherein decoding the first media data comprises:
when the data volume of the media data in the media consumption queue is smaller than a threshold value, the second processing core of the processor runs the third browser thread to add a dummy frame in the media consumption queue, so that the data volume of the media data including the dummy frame in the media consumption queue is larger than or equal to the threshold value, and the first media data is decoded when the data volume of the media data is larger than or equal to the threshold value.
12. The terminal device according to any one of claims 9 to 11, wherein during the execution of the third browser thread by the second processing core of the processor, binary code written in a low-level language is called to decode the first media data.
13. A computer-readable storage medium comprising instructions that, when executed on a terminal device, cause the terminal device to perform the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2021/088845 WO2021218751A1 (en) | 2020-04-30 | 2021-04-22 | Cloud phone-based media data processing method and terminal device |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2020103678835 | 2020-04-30 | ||
| CN202010367883 | 2020-04-30 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113672293A true CN113672293A (en) | 2021-11-19 |
| CN113672293B CN113672293B (en) | 2024-04-09 |
Family
ID=78537903
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010444415.3A Active CN113672293B (en) | 2020-04-30 | 2020-05-22 | Media data processing method based on cloud mobile phone and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113672293B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116233520A (en) * | 2023-05-06 | 2023-06-06 | 海马云（天津）信息技术有限公司 | Method and device for transmitting and acquiring video data, server device and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932676A (en) * | 2012-11-14 | 2013-02-13 | 武汉烽火众智数字技术有限责任公司 | Self-adaptive bandwidth transmitting and playing method based on audio and video frequency synchronization |
| CN103546802A (en) * | 2013-10-15 | 2014-01-29 | 深圳Tcl新技术有限公司 | Method for dynamically adjusting cache and display terminal |
| CN105163148A (en) * | 2015-08-31 | 2015-12-16 | 腾讯科技（北京）有限公司 | Video data operating method, mobile phone server, smart watch and smart equipment |
| CN106604042A (en) * | 2016-12-22 | 2017-04-26 | Tcl集团股份有限公司 | Panorama webcasting system and panorama webcasting method based on cloud server |
| CN107295084A (en) * | 2017-06-26 | 2017-10-24 | 深圳水晶石数字科技有限公司 | A kind of video editing system and method based on high in the clouds |
| CN108337560A (en) * | 2017-01-20 | 2018-07-27 | 韩华泰科株式会社 | Media playback and media serving device for playing media in web browser |
| CN109005245A (en) * | 2018-09-07 | 2018-12-14 | 广州微算互联信息技术有限公司 | The use management method and system of cloud mobile phone |
| CN109906107A (en) * | 2016-06-13 | 2019-06-18 | 索尼互动娱乐有限责任公司 | Cloud game based on browser |
| CN110213308A (en) * | 2018-02-28 | 2019-09-06 | 杭州海康威视数字技术股份有限公司 | A kind of method and device of decoding video data |
| CN110780889A (en) * | 2019-10-14 | 2020-02-11 | 广州微算互联信息技术有限公司 | Cloud mobile phone data cloning method and cloud mobile phone data restoring method |
| CN110798702A (en) * | 2019-10-15 | 2020-02-14 | 平安科技（深圳）有限公司 | Video decoding method, device, equipment and computer readable storage medium |
-
2020
- 2020-05-22 CN CN202010444415.3A patent/CN113672293B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102932676A (en) * | 2012-11-14 | 2013-02-13 | 武汉烽火众智数字技术有限责任公司 | Self-adaptive bandwidth transmitting and playing method based on audio and video frequency synchronization |
| CN103546802A (en) * | 2013-10-15 | 2014-01-29 | 深圳Tcl新技术有限公司 | Method for dynamically adjusting cache and display terminal |
| CN105163148A (en) * | 2015-08-31 | 2015-12-16 | 腾讯科技（北京）有限公司 | Video data operating method, mobile phone server, smart watch and smart equipment |
| CN109906107A (en) * | 2016-06-13 | 2019-06-18 | 索尼互动娱乐有限责任公司 | Cloud game based on browser |
| CN106604042A (en) * | 2016-12-22 | 2017-04-26 | Tcl集团股份有限公司 | Panorama webcasting system and panorama webcasting method based on cloud server |
| CN108337560A (en) * | 2017-01-20 | 2018-07-27 | 韩华泰科株式会社 | Media playback and media serving device for playing media in web browser |
| CN107295084A (en) * | 2017-06-26 | 2017-10-24 | 深圳水晶石数字科技有限公司 | A kind of video editing system and method based on high in the clouds |
| CN110213308A (en) * | 2018-02-28 | 2019-09-06 | 杭州海康威视数字技术股份有限公司 | A kind of method and device of decoding video data |
| CN109005245A (en) * | 2018-09-07 | 2018-12-14 | 广州微算互联信息技术有限公司 | The use management method and system of cloud mobile phone |
| CN110780889A (en) * | 2019-10-14 | 2020-02-11 | 广州微算互联信息技术有限公司 | Cloud mobile phone data cloning method and cloud mobile phone data restoring method |
| CN110798702A (en) * | 2019-10-15 | 2020-02-14 | 平安科技（深圳）有限公司 | Video decoding method, device, equipment and computer readable storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116233520A (en) * | 2023-05-06 | 2023-06-06 | 海马云（天津）信息技术有限公司 | Method and device for transmitting and acquiring video data, server device and storage medium |
| CN116233520B (en) * | 2023-05-06 | 2023-07-25 | 海马云（天津）信息技术有限公司 | Method and device for transmitting and acquiring video data, server device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113672293B (en) | 2024-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021218751A1 (en) | Cloud phone-based media data processing method and terminal device | |
| RU2616536C2 (en) | Method, device and terminal device to display messages | |
| US8650501B2 (en) | User interface with preview transitions | |
| US8775844B1 (en) | Dynamic information adaptation for a computing device having multiple power modes | |
| RU2602791C2 (en) | Typesetting method, device and system | |
| US11144195B2 (en) | Fast data copying method and electronic device | |
| US11455075B2 (en) | Display method when application is exited and terminal | |
| US20120131446A1 (en) | Method for displaying web page in a portable terminal | |
| US20190213241A1 (en) | Web page display method and apparatus, mobile terminal, and storage medium | |
| US11314391B2 (en) | Navigation bar controlling method and terminal | |
| US20200005737A1 (en) | Method for Controlling Display of Terminal, Storage Medium, and Electronic Device | |
| US10768947B2 (en) | Method for interface refresh synchronization,terminal device, and non-transitory computer-readable storage medium | |
| CN113613064B (en) | Video processing method, device, storage medium and terminal | |
| CN112437318A (en) | Content display method, device and system and storage medium | |
| US9754391B2 (en) | Webpage display method and apparatus | |
| CN108780400B (en) | Data processing method and electronic equipment | |
| CN112507259A (en) | Webpage loading method and device, electronic equipment and storage medium | |
| CN113672293B (en) | Media data processing method based on cloud mobile phone and terminal equipment | |
| CN110717126A (en) | Page browsing method and device, electronic equipment and computer readable storage medium | |
| CN114090938A (en) | Page processing method and equipment | |
| CN114489697A (en) | Application program installation method and device | |
| US20150074530A1 (en) | Method for controlling content in mobile electronic device | |
| EP2234019A1 (en) | Graphic instruction transmitter and method for graphic instruction transmission | |
| CN115175002A (en) | Video playing method and device | |
| CN116149488A (en) | Input method starting method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220211 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technologies Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |