WO2021164532A1 - 云游戏直播方法和装置 - Google Patents

云游戏直播方法和装置 Download PDF

Info

Publication number
WO2021164532A1
WO2021164532A1 PCT/CN2021/074692 CN2021074692W WO2021164532A1 WO 2021164532 A1 WO2021164532 A1 WO 2021164532A1 CN 2021074692 W CN2021074692 W CN 2021074692W WO 2021164532 A1 WO2021164532 A1 WO 2021164532A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
video
server
live broadcast
cloud game
Prior art date
Application number
PCT/CN2021/074692
Other languages
English (en)
French (fr)
Inventor
周晓鹏
张希文
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021164532A1 publication Critical patent/WO2021164532A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/30—Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
    • A63F13/35—Details of game servers
    • A—HUMAN NECESSITIES
    • A63—SPORTS; GAMES; AMUSEMENTS
    • A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/85—Providing additional services to players
    • A63F13/86—Watching games played by other players
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4781—Games
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/854—Content authoring
    • H04N21/8547—Content authoring involving timestamps for synchronizing content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85—Assembly of content; Generation of multimedia applications
    • H04N21/858—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
    • H04N21/8586—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot by using a URL

Definitions

  • This application relates to electronic technology, in particular to a cloud game live broadcast method and device.
  • Game live broadcast refers to the player's game operation on the mobile phone at a certain time, which is uploaded to the server of the live broadcast platform in the form of video in real time. Other viewers can watch the player's game interface and operations at the same time through the live broadcast platform.
  • game live broadcasting has become a trend, which requires a terminal device to record the current game interface while the player is playing the game, and record the player's voice and speaker's voice through a microphone.
  • the embodiments of the present application provide a cloud game live broadcast method and device, so as to realize clear game live broadcast and noise-free sound reproduction, and can also reduce the power consumption of terminal equipment and provide game services without jams.
  • an embodiment of the present application provides a cloud game live broadcast method, including: generating a video recording the running process of a cloud game application APP; receiving audio from a terminal device, the audio being used to record the user running the cloud game Voice information in APP; obtain the audio and video synthesized from the video and the audio; encode the audio and video and send it to the server of the live broadcast platform.
  • the cloud game live broadcast method of this application multiplexes cloud video processing capabilities.
  • the cloud game server directly generates game video.
  • the terminal device only records the player’s voice to generate audio.
  • the cloud game server combines the game video and the player’s voice and audio and transmits it to
  • the server of the live broadcast platform reduces the upload operation of terminal equipment. On the one hand, it can realize clear game screen live broadcast and noise-free sound reproduction. On the other hand, the player’s terminal equipment does not need to be responsible for screen recording operations and can provide no jamming. Game services to reduce the power consumption of terminal equipment.
  • the generating a video used to record the running process of the APP includes: rendering the screen of the cloud game APP according to an operation instruction received during the running process of the cloud game APP, and the operation The instruction comes from the terminal device and is triggered by a user's operation; the video is generated according to the screen of the cloud game APP.
  • the obtaining the audio and video synthesized from the video and the audio includes: synchronizing the video and the audio according to the time stamp of the video and the time stamp of the audio And merge to obtain the audio and video.
  • the encoding the audio and video and sending it to the server of the live broadcast platform includes: encoding the audio and video according to the stream requirements of the live broadcast platform; and encoding the encoded code The stream is sent to the server of the live broadcast platform.
  • an embodiment of the present application provides a cloud game live broadcast method, including: turning on a microphone during the running of the cloud game application APP; using the microphone to record the user's voice when running the cloud game APP to obtain audio; The audio is sent to the server of the cloud game APP.
  • the method further includes: acquiring an input operation of the user through an input device; generating an operation instruction according to the input operation; and sending the operation instruction to the server of the cloud game APP.
  • the method further includes: receiving picture data sent by the server of the cloud game APP; and displaying the interface of the cloud game APP on the screen according to the picture data.
  • an embodiment of the present application provides a cloud game live broadcast method, including: receiving a stream of audio and video sent by a server of a cloud game application APP, where the audio and video includes a process for recording the running process of the cloud game APP The video and the voice used to record the user when the user runs the cloud game APP; decode the audio and video stream to obtain the audio and video; provide the audio and video live broadcast service to the terminal device requesting the live broadcast.
  • an embodiment of the present application provides a cloud game server, including:
  • the processing module is used to generate a video recording the running process of the cloud game application APP; the receiving module is used to receive audio from the terminal device, the audio is used to record the voice information when the user runs the cloud game APP; The processing module is also used to obtain the audio and video synthesized by the video and the audio; the sending module is used to encode the audio and video and send it to the server of the live broadcast platform.
  • the processing module is specifically configured to render the screen of the cloud game APP according to the operation instructions received during the operation of the cloud game APP, and the operation instructions come from the terminal device and The generation is triggered by the user's operation; the video is generated according to the screen of the cloud game APP.
  • the processing module is specifically configured to synchronize and merge the video and the audio according to the time stamp of the video and the time stamp of the audio to obtain the audio and video.
  • the sending module is specifically configured to encode the audio and video according to the stream requirements of the live broadcast platform; and send the encoded stream to the server of the live broadcast platform.
  • an embodiment of the present application provides a cloud game client, including:
  • the processing module is used to turn on the microphone during the running of the cloud game application APP; the audio is collected by the microphone when the user runs the cloud game APP; the sending module is used to send the audio to the cloud game APP Server.
  • the processing module is also used to obtain user input operations through an input device; to generate an operation instruction according to the input operation; and the sending module is also used to send the operation instruction to The server of the cloud game APP.
  • it further includes: a receiving module, configured to receive screen data sent by the server of the cloud game APP; and the processing module, further configured to display all images on the screen according to the screen data.
  • a receiving module configured to receive screen data sent by the server of the cloud game APP
  • the processing module further configured to display all images on the screen according to the screen data.
  • an embodiment of the present application provides a server end of a live broadcast platform, including:
  • the receiving module is used to receive the stream of audio and video sent by the server of the cloud game application APP, the audio and video including a video for recording the running process of the cloud game APP and for recording the user running the cloud game APP Time voice; processing module, used to decode the audio and video stream to obtain the audio and video; sending module, used to provide the live broadcast service of the audio and video to the terminal device requesting the live broadcast.
  • an embodiment of the present application provides a server, including:
  • One or more processors are One or more processors;
  • Memory used to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the first or third aspects.
  • an embodiment of the present application provides a terminal device, including:
  • One or more processors are One or more processors;
  • Memory used to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the above-mentioned second aspects.
  • an embodiment of the present application provides a computer-readable storage medium, including a computer program, which when executed on a computer, causes the computer to execute any one of the first to third aspects described above method.
  • an embodiment of the present application provides a computer program, when the computer program is executed by a computer, it is used to execute the method described in any one of the first to third aspects.
  • Fig. 1 shows an exemplary structural diagram of a communication system
  • FIG. 2 shows an exemplary structural diagram of the server 200
  • FIG. 3 shows an exemplary structural diagram of the terminal device 300
  • FIG. 4 shows an exemplary structural diagram of the software layer of the terminal device 300
  • FIG. 5 is a flowchart of an embodiment of an application for a cloud game cloud game live broadcast method
  • Figures 6-9 exemplarily show a schematic diagram of the interface of the cloud game live broadcast method of the present application.
  • Fig. 10 is a schematic structural diagram of an embodiment of a device of this application.
  • At least one (item) refers to one or more, and “multiple” refers to two or more.
  • “And/or” is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, “A and/or B” can mean: only A, only B, and both A and B , Where A and B can be singular or plural.
  • the character “/” generally indicates that the associated objects before and after are in an “or” relationship.
  • the following at least one item (a) or similar expressions refers to any combination of these items, including any combination of a single item (a) or a plurality of items (a).
  • At least one of a, b, or c can mean: a, b, c, "a and b", “a and c", “b and c", or "a and b and c" ", where a, b, and c can be single or multiple.
  • the process of live broadcasting the game in the prior art includes: at a certain time, the player plays the game through the terminal device with the game APP installed, and the player’s terminal device needs to process the game data based on the user's operation while recording the screen of the current game interface. And through the microphone to record the player's voice and the speaker's voice, and then upload the player's game operation (recorded screen content) and surrounding sounds on the mobile phone through the network to the server of the live broadcast platform in the form of audio and video in real time.
  • Other viewers can connect to the live broadcast platform through their respective terminal devices (clients installed with the live broadcast platform), and multiple people can watch the player's game interface and operations at the same time.
  • Fig. 1 shows an exemplary structural diagram of a communication system.
  • the communication system includes a server and a terminal device (client for installing applications).
  • the communication system may include a Or multiple servers, and the coverage of each server may include one or more terminal devices, which is not limited in this application.
  • the communication system may also include other network entities such as a network controller and a switching device, and the present application is not limited thereto.
  • the black arrow in Figure 1 indicates that there is a communication connection between the server and the terminal device, that is, data transmission can be realized between the server and the terminal device through a communication network.
  • the above-mentioned communication network may be a local area network, or a wide area network switched through a relay (relay) device, or include a local area network and a wide area network.
  • the communication network may be a short-distance communication network such as a wifi hotspot network, a wifi P2P network, a Bluetooth network, a zigbee network, or a near field communication (NFC) network.
  • the communication network is a wide area network, for example, the communication network may be a 3rd-generation wireless telephone technology (3G) network, or the 4th generation mobile communication technology (4G). ) Network, fifth-generation mobile communication technology (5G) network, future evolution of public land mobile network (PLMN) or the Internet, etc. This application does not limit this.
  • the client of the cloud game APP sends a request to the server of the cloud game to start the game and live broadcast according to the user operation.
  • the server of the cloud game determines and renders the game screen according to the request of the client of the cloud game APP, and sends it The rendered game screen.
  • the client of the cloud game APP translates the user's operations on the cloud game APP into operation instructions and uploads it to the server of the cloud game, and at the same time uploads the recorded audio data of the client.
  • the cloud game server also records the video of the running process of the cloud game APP operated by the user, synthesizes the video and the audio received from the client to obtain audio and video, and sends the audio and video to the server of the live broadcast platform.
  • the server of the live broadcast platform sends the audio and video to the client on the live broadcast platform that requests to watch, so that multiple users of the live broadcast platform can play/watch the audio and video at the same time.
  • FIG. 1 is only for ease of understanding, and schematically shows a communication system, but this should not constitute any limitation to this application.
  • the communication system may also include a larger number of servers, or a larger number of servers.
  • the servers communicating with different terminal devices may be the same server or different servers, and the number of servers communicating with different terminal devices may be the same or different, which is not limited in this application.
  • the server in the communication system can be any device with a transceiver function or a chip that can be installed in the device.
  • FIG. 2 shows an exemplary structural diagram of the server 200, and the structure of the server 200 can refer to the structure shown in FIG. 2.
  • the server includes at least one processor 201, at least one memory 202, and at least one network interface 203.
  • the processor 201, the memory 202, and the network interface 203 are connected, for example, by a bus. In the present application, the connection may include various interfaces, transmission lines, or buses, which are not limited in this embodiment.
  • the network interface 203 is used to connect the server to other communication devices through a communication link, such as an Ethernet interface.
  • the processor 201 is mainly used for processing communication data, controlling the entire server, executing software programs, and processing data of the software programs, for example, for supporting the server to perform the actions described in the embodiments.
  • the processor 201 is mainly used to control the entire server, execute software programs, and process data of the software programs.
  • the server may include multiple processors to enhance its processing capability, and various components of the server may be connected through various buses.
  • the processor 201 may also be expressed as a processing circuit or a processor chip.
  • the memory 202 is mainly used to store software programs and data.
  • the memory 202 may exist independently and is connected to the processor 201.
  • the memory 202 may be integrated with the processor 201, for example, integrated in one chip.
  • the memory 202 can store program codes for executing the technical solutions of the present application, and the processor 201 controls the execution.
  • Various types of computer program codes that are executed can also be regarded as drivers of the processor 201.
  • Figure 2 shows only one memory and one processor. In an actual server, there may be multiple processors and multiple memories.
  • the memory may also be referred to as a storage medium or storage device.
  • the memory may be a storage element on the same chip as the processor, that is, an on-chip storage element, or an independent storage element, which is not limited in this application.
  • terminal equipment in the communication system can also be referred to as user equipment (UE), which can be deployed on land, including indoor or outdoor, handheld or vehicle-mounted; and can also be deployed on water (such as ships, etc.) ); It can also be deployed in the air (such as airplanes, balloons, and satellites, etc.).
  • Terminal devices can be mobile phones, tablets, wearable devices with wireless communication functions (such as smart watches), location trackers with positioning functions, computers with wireless transceiver functions, virtual reality (virtual reality) , VR) devices, augmented reality (AR) devices, wireless devices in smart homes (smart home), etc., this application does not limit this.
  • the aforementioned terminal equipment and the chips that can be installed in the aforementioned terminal equipment are collectively referred to as terminal equipment.
  • FIG. 3 shows an exemplary structural diagram of the terminal device 300.
  • the terminal device 300 includes: an application processor 301, a microcontroller unit (MCU) 302, a memory 303, a modem (modem) 304, a radio frequency (RF) module 305, and wireless fidelity (Wireless-Fidelity, Wi-Fi for short) module 306, Bluetooth module 307, sensor 308, input/output (input/output, I/O) device 309, positioning module 310 and other components.
  • These components can communicate through one or more communication buses or signal lines.
  • the aforementioned communication bus or signal line may be the CAN bus provided in this application.
  • the terminal device 300 may include more or fewer components than those shown in the figure, or a combination of certain components, or a different component arrangement.
  • the application processor 301 is the control center of the terminal device 300, and uses various interfaces and buses to connect various components of the terminal device 300.
  • the processor 301 may include one or more processing units.
  • the memory 303 stores computer programs, such as the operating system 311 and the application program 312 shown in FIG. 3.
  • the application processor 301 is configured to execute the computer program in the memory 303, so as to realize the functions defined by the computer program.
  • the application processor 301 executes the operating system 311 to implement various functions of the operating system on the terminal device 300.
  • the memory 303 also stores other data besides the computer program, such as data generated during the running of the operating system 311 and the application program 312.
  • the memory 303 is a non-volatile storage medium, and generally includes a memory and an external memory. Memory includes but is not limited to random access memory (RAM), read-only memory (ROM), or cache (cache).
  • External storage includes, but is not limited to, flash memory (flash memory), hard disks, optical disks, universal serial bus (USB) disks, etc.
  • Computer programs are usually stored in external memory, and the processor loads the program from external memory to memory before executing the computer program.
  • the memory 303 may be independent and connected to the application processor 301 through a bus; the memory 303 may also be integrated with the application processor 301 into a chip subsystem.
  • MCU 302 is a co-processor used to acquire and process data from sensor 308.
  • the processing power and power consumption of MCU 302 are less than that of application processor 301, but it has the feature of "always on” and can be used in application processor 301 continuously collects and processes sensor data when it is in sleep mode to ensure the normal operation of the sensor with extremely low power consumption.
  • the MCU 302 may be a sensor hub chip.
  • the sensor 308 may include a light sensor and a motion sensor.
  • the light sensor may include an ambient light sensor and a proximity sensor, where the ambient light sensor can adjust the brightness of the display 3091 according to the brightness of the ambient light, and the proximity sensor can turn off the power of the display when the terminal device 300 is moved to the ear .
  • the accelerometer sensor can detect the magnitude of acceleration in various directions (usually three axes), and can detect the magnitude and direction of gravity when stationary; the sensor 308 can also include a gyroscope, barometer, hygrometer, Other sensors such as thermometers and infrared sensors will not be described here.
  • the MCU 302 and the sensor 308 may be integrated on the same chip, or may be separate components, connected by a bus.
  • the modem 304 and the radio frequency module 305 constitute the communication subsystem of the terminal device 300, and are used to implement the main functions of the wireless communication standard protocol. Among them, modem 304 is used for encoding and decoding, signal modulation and demodulation, and equalization.
  • the radio frequency module 305 is used for receiving and transmitting wireless signals.
  • the radio frequency module 305 includes but is not limited to an antenna, at least one amplifier, a coupler, a duplexer, and the like.
  • the radio frequency module 305 cooperates with the modem 304 to realize the wireless communication function.
  • the modem 304 can be used as a separate chip, or can be combined with other chips or circuits to form a system-level chip or integrated circuit. These chips or integrated circuits can be applied to all terminal devices that implement wireless communication functions, including: mobile phones, computers, notebooks, tablets, routers, wearable devices, automobiles, home appliances, etc.
  • the terminal device 300 may also use the Wi-Fi module 306, the Bluetooth module 307, etc. to perform wireless communication.
  • the Wi-Fi module 306 is used to provide the terminal device 300 with network access that complies with Wi-Fi related standard protocols.
  • the terminal device 300 can access the Wi-Fi access point through the Wi-Fi module 306, and then access the Internet.
  • the Wi-Fi module 306 can also be used as a Wi-Fi wireless access point, which can provide Wi-Fi network access for other terminal devices.
  • the Bluetooth module 307 is used to implement short-distance communication between the terminal device 300 and other terminal devices (such as mobile phones, smart watches, etc.).
  • the Wi-Fi module 306 in this application may be an integrated circuit or a Wi-Fi chip, and the Bluetooth module 307 may be an integrated circuit or a Bluetooth chip.
  • the positioning module 310 is used to determine the geographic location of the terminal device 300. It is understandable that the positioning module 310 may specifically be a receiver of a positioning system such as a global positioning system (GPS), Beidou satellite navigation system, and Russian GLONASS.
  • GPS global positioning system
  • Beidou satellite navigation system Beidou satellite navigation system
  • Russian GLONASS Russian GLONASS
  • the Wi-Fi module 306, the Bluetooth module 307, and the positioning module 310 may be separate chips or integrated circuits, respectively, or they may be integrated together.
  • the Wi-Fi module 306, the Bluetooth module 307 and the positioning module 310 may be integrated on the same chip.
  • the Wi-Fi module 306, the Bluetooth module 307, the positioning module 310, and the MCU 302 can also be integrated into the same chip.
  • the input/output device 309 includes, but is not limited to: a display 3091, a touch screen 3092, an audio circuit 3093, and so on.
  • the touch screen 3092 can collect touch events on or near the user of the terminal device 300 (for example, the user uses any suitable object such as a finger or a stylus to operate on the touch screen 3092 or near the touch screen 3092), and Send the collected touch events to other devices (for example, the application processor 301).
  • the user's operation near the touch screen 3092 can be called floating touch; through the floating touch, the user can select, move, or drag objects (such as icons, etc.) without directly touching the touch screen 3092.
  • multiple types of resistive, capacitive, infrared, and surface acoustic waves can be used to implement the touch screen 3092.
  • the display (also called a display screen) 3091 is used to display information input by the user or information presented to the user.
  • the display can be configured in the form of liquid crystal display, organic light emitting diode, etc.
  • the touch screen 3092 can be overlaid on the display 3091. When the touch screen 3092 detects a touch event, it is sent to the application processor 301 to determine the type of the touch event, and then the application processor 301 can provide corresponding information on the display 3091 according to the type of the touch event. Visual output.
  • the touch screen 3092 and the display 3091 are used as two independent components to realize the input and output functions of the terminal device 300, in some embodiments, the touch screen 3092 and the display 3091 can be integrated to realize the terminal device 300. Input and output functions.
  • the touch screen 3092 and the display 3091 may be configured on the front of the terminal device 300 in the form of a full panel to realize a frameless structure.
  • the audio circuit 3093, the speaker 3094, and the microphone 3095 can provide an audio interface between the user and the terminal device 300.
  • the audio circuit 3093 can transmit the electrical signal converted from the received audio data to the speaker 3094, which is converted into a sound signal by the speaker 3094 for output; on the other hand, the microphone 3095 converts the collected sound signal into an electrical signal, which is then output by the audio circuit 3093
  • the audio data is converted into audio data, and then the audio data is sent to, for example, another terminal device through the modem 304 and the radio frequency module 305, or the audio data is output to the memory 303 for further processing.
  • the terminal device 300 may also have a fingerprint recognition function.
  • a fingerprint collection device may be configured on the back of the terminal device 300 (for example, below the rear camera), or a fingerprint collection device may be configured on the front of the terminal device 300 (for example, below the touch screen 3092).
  • a fingerprint collection device can be configured in the touch screen 3092 to realize the fingerprint identification function, that is, the fingerprint collection device can be integrated with the touch screen 3092 to realize the fingerprint identification function of the terminal device 300.
  • the fingerprint collection device is configured in the touch screen 3092, may be a part of the touch screen 3092, or may be configured in the touch screen 3092 in other ways.
  • the main component of the fingerprint acquisition device in this application is a fingerprint sensor, which can use any type of sensing technology, including but not limited to optical, capacitive, piezoelectric or ultrasonic sensing technology.
  • the terminal device 300 can be logically divided into a hardware layer, an operating system 311, and an application program layer.
  • the hardware layer includes hardware resources such as the application processor 301, the MCU 302, the memory 303, the modem 304, the Wi-Fi module 306, the sensor 308, and the positioning module 310 as described above.
  • the operating system 311 carried by the terminal device 300 may be Or other operating systems, this application does not impose any restrictions on this.
  • the operating system 311 and the application program layer may be collectively referred to as the software layer of the terminal device 300.
  • FIG. 4 shows an exemplary structural diagram of the software layer of the terminal device 300. As shown in Figure 4, Take the operating system as an example. As a software middleware between the hardware layer and the application layer, the operating system is a computer program that manages and controls hardware and software resources.
  • the application layer includes one or more applications, which can be any type of application such as social applications, e-commerce applications, and browsers. For example, desktop launcher, settings, calendar, camera, photos, calls and SMS, etc.
  • the operating system includes the kernel layer, the Android runtime and system libraries, and the application framework layer.
  • the kernel layer is used to provide underlying system components and services, such as: power management, memory management, thread management, hardware drivers, etc.; hardware drivers include display drivers, camera drivers, audio drivers, and touch drivers.
  • the kernel layer encapsulates the kernel driver, provides an interface to the application framework layer, and shields low-level implementation details.
  • the Android runtime and system library provide the required library files and execution environment for the executable program at runtime.
  • a virtual machine or virtual machine instance capable of converting the bytecode of an application into a machine code.
  • System library is a program library that provides support for executable programs at runtime, including two-dimensional graphics engine, three-dimensional graphics engine, media library, surface manager, condition monitoring services, etc.
  • the application framework layer is used to provide various basic common components and services for applications in the application layer, including window managers, activity managers, package managers, resource managers, display policy services, and so on.
  • the terminal device 300 may include fewer or more components than those shown in FIG. 3, and the terminal device shown in FIG. 3 only includes components that are more relevant to the multiple implementations disclosed in this application. .
  • the cloud game live broadcast method provided by this application is suitable for the communication system shown in FIG. 1.
  • the server includes a server of a provider of cloud computing-based application programs (application, APP), and the APP may include a live broadcast APP and other APPs.
  • APP cloud computing-based application programs
  • all the above-mentioned cloud computing-based APPs can adopt a server-client (client-server, C/S) structure.
  • the client of the APP installed on the terminal device is responsible for interacting with the user, and sending the operation instructions generated by the user on the operation interface of the APP to the server of the APP; the server of the APP is responsible for the management of APP data and responding to the customers from the APP Terminal operation instructions and rendering of the screen displayed on the client.
  • the APP in this application may be, for example, a cloud game APP.
  • the game runs on a server, and the rendered game screen is video compressed and transmitted to the client through the network.
  • the user watches the game screen and operates the game on the client.
  • the APP in this application can be a cloud game, or a short video platform that supports cloud game trial play, or an APP application market that supports cloud game trial play.
  • Cloud gaming is a way of gaming based on cloud computing. In the operating mode of cloud gaming, all games are run on the server.
  • the server compresses the rendered game screen and transmits it to the network via the network.
  • Client On the client side, the terminal device does not need to have high-end processors and graphics cards, only basic video decompression capabilities.
  • Cloud computing is a computing method based on the Internet. In this way, shared software and hardware resources and information can be provided to terminal devices on demand.
  • the network that provides resources is called "cloud”. Cloud games get rid of the dependence on hardware. For the server, it only needs to improve the performance of the server without the need to develop a new host. For the client, it can get higher picture quality without having to equip high-performance terminal equipment.
  • the process of cloud games is as follows. First, the user operates the terminal device to connect to the transfer server and selects the game, and then the transfer server sends the information of the selected game to the game server. At this time, the user's terminal device can obtain the uniform resource locator ( uniform resource locator, URL), connect to the game server through the URL to start playing the game.
  • uniform resource locator uniform resource locator
  • the client of the live broadcast platform installed on the terminal device is responsible for interacting with the user, and sends the operation instructions generated by the user performing operations on the operation interface of the live broadcast platform to the server of the live broadcast platform. Or by setting up an independent signal acquisition device (audio + video) on site, importing the collected audio and video data into the client of the live broadcast platform, and the terminal device installed with the client uploads the audio and video data to the server through the communication network; the server of the live platform responsible for the management of live platform data, respond to operating instructions from the client of the live platform, and render the screen displayed on the client.
  • the live broadcast platform in this application can be, for example, Wait.
  • FIG. 5 is a flowchart of an embodiment of a cloud game cloud game live broadcast method according to this application. As shown in FIG. 5, the method of this embodiment can be applied to the communication system shown in FIG. 1 above.
  • the cloud game live broadcast method may include:
  • Step 501 During the running of the application, the cloud game server generates a cloud game running video.
  • cloud games based on cloud computing usually adopt a C/S structure.
  • Players need to install the client of the cloud game on the terminal device first, and then click the icon of the cloud game to open the client.
  • the client connects to the server of the cloud game through the communication function of the terminal device and starts to run.
  • the client can store a large amount of resources in the cloud game.
  • the user inputs operating instructions through the client, and the client translates it into data and sends it to the server. After the server processes the instructions according to the operating instructions, the processing results are obtained and sent to the client.
  • the terminal graphically displays it on the screen of the terminal device. It can be said that the client is an intermediary between the user and the server.
  • the client will translate according to the user's operation (that is, generate what the server can recognize Operating instructions).
  • the user's operations can include operations such as clicking, dragging, sliding, and long pressing on the touch screen of the smart device. It can also include operations such as clicking and dragging on the mouse of the computer, and input operations on the keyboard. Including the operation of the voice command input through the microphone, the operation such as somatosensory input, hovering input, etc. realized through the sensor, and related operations on other input devices, etc., which are not specifically limited in this application.
  • the running process of the cloud game triggered by the user's operation has processing, screen rendering, and recording on the server. Therefore, the server can integrate the above information to obtain a video recording the running process of the cloud game.
  • the cloud game server records and generates the cloud game running video, which can be based on a user's instruction, for example, the user starts a live broadcast of the cloud game, or an instruction based on the server of the live broadcast platform, or it can be executed automatically when the cloud game starts.
  • Step 502 The client of the cloud game collects surrounding sounds to generate audio data.
  • the client will activate the microphone of the terminal device to record surrounding sounds, especially the sounds made by the user when operating the cloud game.
  • the client terminal generates audio data that can be recognized by the server after collecting the sound.
  • the client's activation of the microphone of the terminal device can be based on the user's activation of the cloud game live broadcast, or based on the user's initiative to turn on the microphone, or based on the instructions of the live broadcast platform server to automatically or manually turn on the microphone with the user's permission.
  • the activated microphone is not limited to being located on the terminal device where the cloud game client is running, but may also be other terminal devices, such as other microphones around the terminal device where the client is based on the Internet of Things.
  • Step 503 The client of the cloud game sends the audio data to the server of the cloud game.
  • the client sends audio data to the server through the communication network between the terminal device and the server.
  • the sending of the audio data can be real-time streaming media, or can be sent at a fixed time that can ensure the real-time performance of the live broadcast.
  • Step 504 The server of the cloud game merges the running video and audio of the cloud game to obtain audio and video, and encodes and generates a code stream matching the live broadcast platform.
  • the cloud game server After the cloud game server receives the audio data from the client, it synchronizes the two according to the time stamp of the audio and the locally generated cloud game running video, and then synthesizes the two into audio and video including audio and video. It reflects the operation of the cloud game by the user during the running process of the cloud game. Regarding the synchronization of audio and video, you can refer to the following process: video is usually represented by the frame rate (the number of frames displayed in one second of the video), and the audio is usually represented by the sampling rate (the number of samples played in one second of the audio). speed. Both the sound card and the graphics card use one frame of data as the playback unit.
  • audio and video are completely synchronized. However, due to the characteristics of the audio and video itself (for example, the audio output is linear, and the video output may be non-linear), it may slowly appear that the audio and video are out of sync.
  • a time stamp is introduced.
  • a reference clock (requires that the time on the reference clock is linearly increasing), corresponding to each time node of the reference clock, divide the audio and video into data blocks one by one; when encoding, give each time node the corresponding audio data Blocks and video data blocks are time stamped; during playback, the playback is adjusted according to the audio and video time stamps and the reference clock. So the time stamp is the key to the synchronization of audio and video.
  • the client and the cloud game server are time synchronized.
  • the reference time of the audio and video timestamps are the same, so it can be ensured that the audio and video are synchronized.
  • the user can select the platform to be broadcasted before running the cloud game. Therefore, the server of the cloud game can encode the audio and video according to the requirements of selecting the live broadcast platform to obtain a matching code stream.
  • Step 505 The server of the cloud game sends the code stream to the server of the live broadcast platform.
  • the server of the cloud game sends the code stream to the server of the live broadcast platform through the communication network between the servers.
  • Step 506 The server of the live broadcast platform plays audio and video.
  • the server of the live broadcast platform may be responsible for the management of the live broadcast platform data, respond to operation instructions from the client of the live broadcast platform, and render the screen displayed on the client.
  • the cloud game live broadcast method of this application multiplexes the cloud video processing capabilities.
  • the server directly generates the game video.
  • the terminal device only records the player’s voice to generate audio.
  • the server combines the game video and the player’s voice and audio and transmits it to the server of the live broadcast platform.
  • the player’s terminal device does not need to be responsible for the screen recording operation. It can provide game services without jams and reduce the number of terminals. The power consumption of the device.
  • the cloud game live broadcast method provided in this application will be explained by taking a live broadcast player playing a certain cloud game APP 1 as an example.
  • the player installs a certain cloud game APP 1 on the terminal device, and the player opens the game APP 1 to enter the game interface, and plays the game by clicking, dragging, zooming, long pressing, etc.
  • the server of the cloud game APP 1 responds to the operation from the client Instruct and render the screen displayed on the client.
  • the server generates a game video based on the game data.
  • the game video includes the game screen and the sounds and sound effects of the game itself.
  • the terminal device starts the microphone to record the player’s voice to generate audio.
  • the audio is transmitted to the server of the cloud game APP 1 through the network.
  • the server synchronizes the two according to the timestamp of the game video and the player’s voice and then merges into one
  • the server encodes the audio and video to generate a code stream matching the live broadcast platform and sends it to the server of the live broadcast platform, which is then played by the live broadcast platform to other users who watch the game live broadcast.
  • FIGs 6-9 exemplarily show the interface schematic diagrams of the cloud game live broadcast method of the present application.
  • the cloud game APP 1 provides a live broadcast setting interface, which provides several live broadcast platform options (such as live broadcast platforms 1-4).
  • Players can click the corresponding controls to select the live broadcast platform (such as live broadcast). Platform 1).
  • the player chooses not to broadcast the game, they can click the "skip" control.
  • the operating instructions generated by the player's operation on the live broadcast setting interface are transmitted to the server via the network.
  • the server processes game data and renders the game screen according to the live broadcast platform indicated by the live broadcast platform information ( For example, the code stream of the live broadcast platform 1) needs to generate game video.
  • the operation instruction does not include the live broadcast platform information, or the operation instruction indicates "skip"
  • the server only processes game data and renders game screens, and does not generate game video.
  • the server of the cloud game will process the game data and render the game screen when the player is playing the game. Therefore, the server of the cloud game can generate the game video according to the game data, the screen and the game operation of the player in the process.
  • the game video includes The screen of the game and the sounds and sound effects of the game itself.
  • the cloud game server After the cloud game server receives the player’s voice and audio, it aligns the player’s voice, audio and the game video’s respective timestamps, and merges the two to generate the player’s game audio and video.
  • This audio and video includes the game’s screen and game The sound and sound effects and the sound of the player explaining the game.
  • the cloud game server encodes the game audio and video to obtain a code stream matching the live broadcast platform 1, and sends the code stream to the live broadcast platform 1 server. In this way, the player's terminal device does not need to record the screen while the player is playing the game, which reduces the consumption of the terminal device, and a small amount of audio upload does not interfere with the uplink transmission of the game itself.
  • the server of the live broadcast platform 1 receives the code stream from the cloud game server, decodes the code stream to obtain game video and audio, and broadcasts the code stream to multiple users who watch the game live broadcast through their respective terminals.
  • the device is connected to the server of the live broadcast platform 1 to watch the screen of the player playing the game and hear the player's explanation.
  • the cloud game live broadcast method of this application multiplexes the cloud video processing capabilities.
  • the server directly generates the game video.
  • the terminal device only records the player’s voice to generate audio.
  • the server combines the game video and the player’s voice and audio and transmits it to the server of the live broadcast platform.
  • the player’s terminal device does not need to be responsible for the screen recording operation. It can provide game services without jams and reduce the number of terminals. The power consumption of the device.
  • FIG. 10 is a schematic structural diagram of an embodiment of an apparatus of this application.
  • the apparatus of this embodiment can be applied to the server of the cloud game, can also be applied to the terminal device, and can also be applied to the server of the live broadcast platform.
  • the device may include: a processing module 1001, a receiving module 1002, and a sending module 1003.
  • the processing module 1001 When the device is applied to the above-mentioned cloud game server, the processing module 1001 is used to generate a video recording the running process of the cloud game application APP; the receiving module 1002 is used to receive audio from the terminal device, and the audio is used to record the user Voice information when running the cloud game APP; the processing module 1001 is also used to obtain the audio and video synthesized by the video and the audio; the sending module 1003 is used to encode the audio and video and send it to the live broadcast platform server.
  • the processing module 1001 is specifically configured to render the screen of the cloud game APP according to the operation instructions received during the operation of the cloud game APP, and the operation instructions come from the terminal device And the generation is triggered by the user's operation; the video is generated according to the screen of the cloud game APP.
  • the processing module 1001 is specifically configured to synchronize and merge the video and the audio according to the time stamp of the video and the time stamp of the audio to obtain the audio and video.
  • the sending module 1003 is specifically configured to encode the audio and video according to the stream requirements of the live broadcast platform; and send the encoded stream to the server of the live broadcast platform .
  • the processing module 1001 is used to turn on the microphone during the running of the cloud game application APP; use the microphone to record the voice information when the user runs the cloud game APP to obtain audio; the sending module 1003 , Used to send the audio to the server of the cloud game APP.
  • the processing module 1001 is further configured to obtain user input operations through an input device; generate operation instructions according to the input operations; and the sending module 1003 is further configured to transmit the operation instructions Sent to the server of the cloud game APP.
  • the receiving module 1002 is used to receive the screen data sent by the server of the cloud game APP; the processing module 1101 is also used to display the screen data on the screen according to the screen data.
  • the interface of cloud game APP is used to display the screen data on the screen according to the screen data.
  • the receiving module 1002 is used to receive the audio and video stream sent by the server of the cloud game application APP. Video and the voice used to record the user running the cloud game APP; the processing module 1001 is used to decode the audio and video stream to obtain the audio and video; the sending module 1003 is used to send the live broadcast to the terminal device requesting Provide live broadcast services of the audio and video.
  • the steps of the foregoing method embodiments may be completed by hardware integrated logic circuits in the processor or instructions in the form of software.
  • the processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other Programming logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware encoding processor, or executed and completed by a combination of hardware and software modules in the encoding processor.
  • the software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers.
  • the storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
  • the memory mentioned in the above embodiments may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electrically available Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • the volatile memory may be random access memory (RAM), which is used as an external cache.
  • RAM random access memory
  • static random access memory static random access memory
  • dynamic RAM dynamic RAM
  • DRAM dynamic random access memory
  • synchronous dynamic random access memory synchronous DRAM, SDRAM
  • double data rate synchronous dynamic random access memory double data rate SDRAM, DDR SDRAM
  • enhanced synchronous dynamic random access memory enhanced SDRAM, ESDRAM
  • synchronous connection dynamic random access memory serial DRAM, SLDRAM
  • direct rambus RAM direct rambus RAM
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional units in the various embodiments of the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solutions of the embodiments of the present application are essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium.
  • Including several instructions to make a computer device personal computer, server, or network device, etc.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

Abstract

本申请提供一种云游戏直播方法和装置。本申请云游戏直播方法，包括：生成记录云游戏应用程序APP的运行过程的视频；接收来自终端设备的音频，所述音频用于记录用户运行所述云游戏APP时的语音信息；获得所视频和所述音频合成的音视频；对所述音视频编码后发送给直播平台的服务器。本申请可以实现清晰的游戏画面直播，以及无杂音的声音再现，还可以减少终端设备的功耗，提供无卡顿的游戏服务。

Description

云游戏直播方法和装置
本申请要求于2020年2月18日提交中国专利局、申请号为202010100145.4、申请名称为“云游戏直播方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及电子技术，尤其涉及一种云游戏直播方法和装置。
背景技术
游戏直播是指在某一时间通过网络将玩家在手机上的游戏操作，以视频的形式实时上传到直播平台的服务器，其他观众可以通过该直播平台同时观看玩家的游戏界面和操作等。近年来游戏直播成为一种趋势，其需要终端设备在玩家玩游戏的同时对当前游戏界面进行录屏，并通过麦克风录制玩家声音和扬声器声音。
但是，一边供玩家玩游戏一边录屏会增加终端设备的功耗，上传游戏视频又会对游戏本身的上行传输造成干扰，而大量的数据上传会进一步加剧终端设备的功耗，并造成游戏上行传输的卡顿。
发明内容
本申请实施例提供一种云游戏直播方法和装置，以实现清晰的游戏画面直播，以及无杂音的声音再现，还可以减少终端设备的功耗，提供无卡顿的游戏服务。
第一方面，本申请实施例提供一种云游戏直播方法，包括：生成记录云游戏应用程序APP的运行过程的视频；接收来自终端设备的音频，所述音频用于记录用户运行所述云游戏APP时的语音信息；获得所述视频和所述音频合成的音视频；对所述音视频编码后发送给直播平台的服务器。
本申请的云游戏直播方法复用了云端视频处理能力，由云游戏的服务器直接生成游戏视频，终端设备只收录玩家的声音生成音频，云游戏的服务器将游戏视频和玩家声音音频合并后传输给直播平台的服务器，减少了终端设备的上传操作，一方面可以实现清晰的游戏画面直播，以及无杂音的声音再现，另一方面玩家的终端设备不需要再负责录屏操作，可以提供无卡顿的游戏服务，减少终端设备的功耗。
在一种可能的实现方式中，所述生成用于记录APP的运行过程的视频，包括：根据所述云游戏APP运行过程中接收到的操作指令渲染所述云游戏APP的画面，所述操作指令来自所述终端设备且由用户的操作触发生成；根据所述云游戏APP的画面生成所述视频。
在一种可能的实现方式中，所述获得所述视频和所述音频合成的音视频，包括：根据所述视频的时间戳和所述音频的时间戳对所述视频和所述音频进行同步并合并得到所述音视频。
在一种可能的实现方式中，所述对所述音视频编码后发送给直播平台的服务器，包括：根据所述直播平台的码流要求对所述音视频进行编码；将编码后得到的码流发送给所述直 播平台的服务器。
第二方面，本申请实施例提供一种云游戏直播方法，包括：在云游戏应用程序APP运行的过程中开启麦克风；通过所述麦克风收录用户运行所述云游戏APP时的语音得到音频；将所述音频发送给云游戏APP的服务器。
在一种可能的实现方式中，还包括：通过输入装置获取用户的输入操作；根据所述输入操作生成操作指令；将所述操作指令发送给所述云游戏APP的服务器。
在一种可能的实现方式中，还包括：接收所述云游戏APP的服务器发送的画面数据；根据所述画面数据在所述屏幕上显示所述云游戏APP的界面。
第三方面，本申请实施例提供一种云游戏直播方法，包括：接收云游戏应用程序APP的服务器发送的音视频的码流，所述音视频包括用于记录所述云游戏APP的运行过程的视频和用于记录用户运行所述云游戏APP时的语音；对所述音视频的码流进行解码得到所述音视频；向请求直播的终端设备提供所述音视频的直播服务。
第四方面，本申请实施例提供一种云游戏的服务端，包括：
处理模块，用于生成记录云游戏应用程序APP的运行过程的视频；接收模块，用于接收来自终端设备的音频，所述音频用于记录用户运行所述云游戏APP时的语音信息；所述处理模块，还用于获得所述视频和所述音频合成的音视频；发送模块，用于对所述音视频编码后发送给直播平台的服务器。
在一种可能的实现方式中，所述处理模块，具体用于根据所述云游戏APP运行过程中接收到的操作指令渲染所述云游戏APP的画面，所述操作指令来自所述终端设备且由用户的操作触发生成；根据所述云游戏APP的画面生成所述视频。
在一种可能的实现方式中，所述处理模块，具体用于根据所述视频的时间戳和所述音频的时间戳对所述视频和所述音频进行同步并合并得到所述音视频。
在一种可能的实现方式中，所述发送模块，具体用于根据所述直播平台的码流要求对所述音视频进行编码；将编码后得到的码流发送给所述直播平台的服务器。
第五方面，本申请实施例提供一种云游戏的客户端，包括：
处理模块，用于在云游戏应用程序APP运行的过程中开启麦克风；通过所述麦克风收录用户运行所述云游戏APP时的语音得到音频；发送模块，用于将所述音频发送给云游戏APP的服务器。
在一种可能的实现方式中，所述处理模块，还用于通过输入装置获取用户的输入操作；根据所述输入操作生成操作指令；所述发送模块，还用于将所述操作指令发送给所述云游戏APP的服务器。
在一种可能的实现方式中，还包括：接收模块，用于接收所述云游戏APP的服务器发送的画面数据；所述处理模块，还用于根据所述画面数据在所述屏幕上显示所述云游戏APP的界面。
第六方面，本申请实施例提供一种直播平台的服务端，包括：
接收模块，用于接收云游戏应用程序APP的服务器发送的音视频的码流，所述音视频包括用于记录所述云游戏APP的运行过程的视频和用于记录用户运行所述云游戏APP时的语音；处理模块，用于对所述音视频的码流进行解码得到所述音视频；发送模块，用于向请求直播的终端设备提供所述音视频的直播服务。
第七方面，本申请实施例提供一种服务器，包括：
一个或多个处理器；
存储器，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如上述第一或三方面中任一项所述的方法。
第八方面，本申请实施例提供一种终端设备，包括：
一个或多个处理器；
存储器，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如上述第二方面中任一项所述的方法。
第九方面，本申请实施例提供一种计算机可读存储介质，包括计算机程序，所述计算机程序在计算机上被执行时，使得所述计算机执行上述第一至三方面中任一项所述的方法。
第十方面，本申请实施例提供一种计算机程序，当所述计算机程序被计算机执行时，用于执行上述第一至三方面中任一项所述的方法。
附图说明
图1示出了通信系统的一个示例性的结构示意图；
图2示出了服务器200的一个示例性的结构示意图；
图3示出了终端设备300的一个示例性的结构示意图；
图4示出了终端设备300的软件层的一个示例性的结构示意图；
图5为本申请云游戏云游戏直播方法实施例的流程图；
图6-9示例性的示出了本申请云游戏直播方法的界面示意图；
图10为本申请装置实施例的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合本申请中的附图，对本申请中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书实施例和权利要求书及附图中的术语“第一”、“第二”等仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元。方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
应当理解，在本申请中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和/或B”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项(个)” 或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。
现有技术中对游戏进行直播过程包括：在某一时间，玩家通过安装了游戏APP的终端设备玩游戏，玩家的终端设备需要一边基于用户的操作处理游戏数据，一边对当前游戏界面进行录屏和通过麦克风录制玩家声音和扬声器声音，再通过网络将玩家在手机上的游戏操作(录屏内容)和周围的声音，以音视频的形式实时上传到直播平台的服务器。其他观众可以通过各自的终端设备(安装了直播平台的客户端)连接至该直播平台，多人同时观看玩家的游戏界面和操作等。这样一边供玩家玩游戏一边录屏会增加终端设备的功耗，而上传游戏音视频又会对游戏本身的上行传输造成干扰，会进一步加剧终端设备的功耗，并造成游戏上行传输的卡顿。
基于目前流行的云游戏，本申请提供了一种用于云游戏直播的通信系统。图1示出了通信系统的一个示例性的结构示意图，如图1所示，该通信系统包括服务器和终端设备(用于安装应用程序的客户端)，可选地，该通信系统可以包括一个或多个服务器并且每个服务器的覆盖范围内可以包括一个或多个终端设备，本申请对此不做限定。可选地，该通信系统还可以包括网络控制器、交换设备等其他网络实体，本申请不限于此。图1中黑色箭头表示服务器与终端设备存在通信连接，即服务器和终端设备之间可以通过通信网络实现数据传输。
需要说明的是，上述通信网络可以是局域网，也可以是通过中继(relay)设备转接的广域网，或者包括局域网和广域网。当该通信网络为局域网时，示例性的，该通信网络可以是wifi热点网络、wifi P2P网络、蓝牙网络、zigbee网络或近场通信(near field communication，NFC)网络等近距离通信网络。当该通信网络为广域网时，示例性的，该通信网络可以是第三代移动通信技术(3rd-generation wireless telephone technology，3G)网络、第四代移动通信技术(the 4th generation mobile communication technology，4G)网络、第五代移动通信技术(5th-generation mobile communication technology，5G)网络、未来演进的公共陆地移动网络(public land mobile network，PLMN)或因特网等，本申请对此不作限定。
云游戏APP的客户端根据用户操作向云游戏的服务器发送请求以开始游戏和直播，如图1所示，云游戏的服务器根据云游戏APP的客户端的请求，确定和渲染游戏画面，并下发渲染后的游戏画面。云游戏APP的客户端将用户对云游戏APP的操作翻译成操作指令上传给云游戏的服务器，同时上传录得的客户端的音频数据。云游戏的服务器同时还记录该用户操作的云游戏APP的运行过程的视频，将该视频和从该客户端接收到的音频合成得到音视频，并将该音视频发送给直播平台的服务器。直播平台的服务器收到后将该音视频发送给直播平台上请求观看的客户端，以供该直播平台的多个用户能同时播放/观看该音视频。
应理解，图1中仅为便于理解，示意性地示出了一个通信系统，但这不应对本申请构成任何限定，该通信系统中还可以包括更多数量的服务器，也可以包括更多数量的终端设备，与不同的终端设备通信的服务器可以是相同的服务器，也可以是不同的服务器，与不同的终端设备通信的服务器的数量可以相同，也可以不同，本申请对此不做限定。还应理 解，该通信系统中的服务器可以是任意一种具有收发功能的设备或可设置于该设备的芯片。图2示出了服务器200的一个示例性的结构示意图，服务器200的结构可以参考图2所示的结构。
服务器包括至少一个处理器201、至少一个存储器202和至少一个网络接口203。处理器201、存储器202和网络接口203相连，例如通过总线相连，在本申请中，所述连接可包括各类接口、传输线或总线等，本实施例对此不做限定。网络接口203用于使得服务器通过通信链路，与其它通信设备相连，例如以太网接口。
处理器201主要用于对通信数据进行处理，以及对整个服务器进行控制，执行软件程序，处理软件程序的数据，例如用于支持服务器执行实施例中所描述的动作。处理器201主要用于对整个服务器进行控制，执行软件程序，处理软件程序的数据。本领域技术人员可以理解，服务器可以包括多个处理器以增强其处理能力，服务器的各个部件可以通过各种总线连接。处理器201也可以表述为处理电路或者处理器芯片。
存储器202主要用于存储软件程序和数据。存储器202可以是独立存在，与处理器201相连。可选的，存储器202可以和处理器201集成在一起，例如集成在一个芯片之内。其中，存储器202能够存储执行本申请的技术方案的程序代码，并由处理器201来控制执行，被执行的各类计算机程序代码也可被视为是处理器201的驱动程序。
图2仅示出了一个存储器和一个处理器。在实际的服务器中，可以存在多个处理器和多个存储器。存储器也可以称为存储介质或者存储设备等。存储器可以为与处理器处于同一芯片上的存储元件，即片内存储元件，或者为独立的存储元件，本申请对此不做限定。
还应理解，该通信系统中的终端设备又可称之为用户设备(user equipment，UE)，可以部署在陆地上，包括室内或室外、手持或车载；也可以部署在水面上(如轮船等)；还可以部署在空中(例如飞机、气球和卫星上等)。终端设备可以是手机(mobile phone)、平板电脑(pad)、具备无线通讯功能的可穿戴设备(如智能手表)、具有定位功能的位置追踪器、带无线收发功能的电脑、虚拟现实(virtual reality，VR)设备、增强现实(augmented reality，AR)设备、智慧家庭(smart home)中的无线设备等，本申请对此不作限定。本申请中将前述终端设备及可设置于前述终端设备的芯片统称为终端设备。
图3示出了终端设备300的一个示例性的结构示意图。如图3所示，终端设备300包括：应用处理器301、微控制器单元(microcontroller unit，MCU)302、存储器303、调制解调器(modem)304、射频(radio frequency，RF)模块305、无线保真(Wireless-Fidelity，简称Wi-Fi)模块306、蓝牙模块307、传感器308、输入/输出(input/output，I/O)设备309、定位模块310等部件。这些部件可通过一根或多根通信总线或信号线进行通信。前述通信总线或信号线可以是本申请提供的CAN总线。本领域技术人员可以理解，终端设备300可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
下面结合图3对终端设备300的各个部件进行具体的介绍：
应用处理器301是终端设备300的控制中心，利用各种接口和总线连接终端设备300的各个部件。在一些实施例中，处理器301可包括一个或多个处理单元。
存储器303中存储有计算机程序，诸如图3所示的操作系统311和应用程序312。应用处理器301被配置用于执行存储器303中的计算机程序，从而实现该计算机程序定义的功能，例如应用处理器301执行操作系统311从而在终端设备300上实现操作系统的各种 功能。存储器303还存储有除计算机程序之外的其他数据，诸如操作系统311和应用程序312运行过程中产生的数据。存储器303为非易失性存储介质，一般包括内存和外存。内存包括但不限于随机存取存储器(random access memory，RAM)，只读存储器(read-only memory，ROM)，或高速缓存(cache)等。外存包括但不限于闪存(flash memory)、硬盘、光盘、通用串行总线(universal serial bus，USB)盘等。计算机程序通常被存储在外存上，处理器在执行计算机程序前会将该程序从外存加载到内存。
存储器303可以是独立的，通过总线与应用处理器301相连接；存储器303也可以和应用处理器301集成到一个芯片子系统。
MCU 302是用于获取并处理来自传感器308的数据的协处理器，MCU 302的处理能力和功耗小于应用处理器301，但具有“永久开启(always on)”的特点，可以在应用处理器301处于休眠模式时持续收集以及处理传感器数据，以极低的功耗保障传感器的正常运行。在一个实施例中，MCU 302可以为sensor hub芯片。传感器308可以包括光传感器、运动传感器。具体地，光传感器可包括环境光传感器及接近传感器，其中，环境光传感器可根据环境光线的明暗来调节显示器3091的亮度，接近传感器可在终端设备300移动到耳边时，关闭显示屏的电源。作为运动传感器的一种，加速计传感器可检测各个方向上(一般为三轴)加速度的大小，静止时可检测出重力的大小及方向；传感器308还可以包括陀螺仪、气压计、湿度计、温度计、红外线传感器等其它传感器，在此不再赘述。MCU 302和传感器308可以集成到同一块芯片上，也可以是分离的元件，通过总线连接。
modem 304以及射频模块305构成了终端设备300通信子系统，用于实现无线通信标准协议的主要功能。其中，modem 304用于编解码、信号的调制解调、均衡等。射频模块305用于无线信号的接收和发送，射频模块305包括但不限于天线、至少一个放大器、耦合器、双工器等。射频模块305配合modem 304实现无线通信功能。modem 304可以作为单独的芯片，也可以与其他芯片或电路在一起形成系统级芯片或集成电路。这些芯片或集成电路可应用于所有实现无线通信功能的终端设备，包括：手机、电脑、笔记本、平板、路由器、可穿戴设备、汽车、家电设备等。
终端设备300还可以使用Wi-Fi模块306，蓝牙模块307等来进行无线通信。Wi-Fi模块306用于为终端设备300提供遵循Wi-Fi相关标准协议的网络接入，终端设备300可以通过Wi-Fi模块306接入到Wi-Fi接入点，进而访问互联网。在其他一些实施例中，Wi-Fi模块306也可以作为Wi-Fi无线接入点，可以为其他终端设备提供Wi-Fi网络接入。蓝牙模块307用于实现终端设备300与其他终端设备(例如手机、智能手表等)之间的短距离通信。本申请中的Wi-Fi模块306可以是集成电路或Wi-Fi芯片等，蓝牙模块307可以是集成电路或者蓝牙芯片等。
定位模块310用于确定终端设备300的地理位置。可以理解的是，定位模块310具体可以是全球定位系统(global position system，GPS)或北斗卫星导航系统、俄罗斯GLONASS等定位系统的接收器。
Wi-Fi模块306，蓝牙模块307和定位模块310分别可以是单独的芯片或集成电路，也可以集成到一起。例如，在一个实施例中，Wi-Fi模块306，蓝牙模块307和定位模块310可以集成到同一芯片上。在另一个实施例中，Wi-Fi模块306，蓝牙模块307、定位模块310以及MCU 302也可以集成到同一芯片中。
输入/输出设备309包括但不限于：显示器3091、触摸屏3092，以及音频电路3093等等。
其中，触摸屏3092可采集终端设备300的用户在其上或附近的触摸事件(比如用户使用手指、触控笔等任何适合的物体在触摸屏3092上或在触控屏触摸屏3092附近的操作)，并将采集到的触摸事件发送给其他器件(例如应用处理器301)。其中，用户在触摸屏3092附近的操作可以称之为悬浮触控；通过悬浮触控，用户可以在不直接接触触摸屏3092的情况下选择、移动或拖动目的(例如图标等)。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型来实现触摸屏3092。
显示器(也称为显示屏)3091用于显示用户输入的信息或展示给用户的信息。可以采用液晶显示屏、有机发光二极管等形式来配置显示器。触摸屏3092可以覆盖在显示器3091之上，当触摸屏3092检测到触摸事件后，传送给应用处理器301以确定触摸事件的类型，随后应用处理器301可以根据触摸事件的类型在显示器3091上提供相应的视觉输出。虽然在图3中，触摸屏3092与显示器3091是作为两个独立的部件来实现终端设备300的输入和输出功能，但是在某些实施例中，可以将触摸屏3092与显示器3091集成而实现终端设备300的输入和输出功能。另外，触摸屏3092和显示器3091可以以全面板的形式配置在终端设备300的正面，以实现无边框的结构。
音频电路3093、扬声器3094、麦克风3095可提供用户与终端设备300之间的音频接口。音频电路3093可将接收到的音频数据转换后的电信号，传输到扬声器3094，由扬声器3094转换为声音信号输出；另一方面，麦克风3095将收集的声音信号转换为电信号，由音频电路3093接收后转换为音频数据，再通过modem 304和射频模块305将音频数据发送给比如另一终端设备，或者将音频数据输出至存储器303以便进一步处理。
另外，终端设备300还可以具有指纹识别功能。例如，可以在终端设备300的背面(例如后置摄像头的下方)配置指纹采集器件，或者在终端设备300的正面(例如触摸屏3092的下方)配置指纹采集器件。又例如，可以在触摸屏3092中配置指纹采集器件来实现指纹识别功能，即指纹采集器件可以与触摸屏3092集成在一起来实现终端设备300的指纹识别功能。在这种情况下，该指纹采集器件配置在触摸屏3092中，可以是触摸屏3092的一部分，也可以以其他方式配置在触摸屏3092中。本申请中的指纹采集器件的主要部件是指纹传感器，该指纹传感器可以采用任何类型的感测技术，包括但不限于光学式、电容式、压电式或超声波传感技术等。
终端设备300从逻辑上可划分为硬件层、操作系统311，以及应用程序层。硬件层包括如上所述的应用处理器301、MCU 302、存储器303、modem 304、Wi-Fi模块306、传感器308、定位模块310等硬件资源。终端设备300搭载的操作系统311可以为
Figure PCTCN2021074692-appb-000001
Figure PCTCN2021074692-appb-000002
或者其它操作系统，本申请对此不作任何限制。
操作系统311和应用程序层可以统称为终端设备300的软件层，图4示出了终端设备300的软件层的一个示例性的结构示意图。如图4所示，以
Figure PCTCN2021074692-appb-000003
操作系统为例，操作系统作为硬件层和应用程序层之间的软件中间件，是管理和控制硬件与软件资源的计算机程序。
应用程序层包括一个或多个应用程序，应用程序可以为社交类应用、电子商务类应用、浏览器等任意类型的应用程序。例如，桌面启动器、设置、日历、相机、照片、通话和短 信等等。
Figure PCTCN2021074692-appb-000004
操作系统包括内核层、安卓运行时和系统库以及应用程序框架层。其中，内核层用于提供底层系统组件和服务，例如：电源管理、内存管理、线程管理、硬件驱动程序等；硬件驱动程序包括显示驱动、摄像头驱动、音频驱动和触控驱动等。内核层对内核驱动程序的封装，向应用程序框架层提供接口，屏蔽低层的实现细节。
安卓运行时和系统库为可执行程序在运行时提供所需要的库文件和执行环境。能够把应用程序的字节码转换为机器码的虚拟机或虚拟机实例。系统库是为可执行程序在运行时提供支持的程序库，包括二维图像引擎、三维图形引擎、媒体库、表面管理器、状态监测服务等。
应用程序框架层用于为应用程序层中的应用程序提供各种基础的公共组件和服务，包括窗口管理器、活动管理器、包管理器、资源管理器、显示策略服务等等。
以上描述的操作系统的各个组件的功能均可以由应用处理器301执行存储器303中存储的程序来实现。
所属领域的技术人员可以理解终端设备300可包括比图3所示的更少或更多的部件，图3所示的该终端设备仅包括与本申请所公开的多个实现方式更加相关的部件。
本申请提供的云游戏直播方法适用于图1所示的通信系统。其中，服务器包括基于云计算的应用程序(application，APP)的提供商的服务器，该APP可以包括直播APP和其他APP。需要说明的是，上述基于云计算的APP均可以采用服务器-客户端(client-server，C/S)结构。
安装于终端设备上的APP的客户端负责与用户交互，将用户在APP的操作界面上执行操作所产生的操作指令发送给APP的服务器；APP的服务器负责APP数据的管理、响应来自APP的客户端的操作指令以及渲染显示于客户端的画面。
例如，本申请中的APP例如可以是云游戏APP，该游戏在服务器上运行，渲染后的游戏画面经视频压缩后通过网络传送至客户端，用户在客户端观看游戏画面和操作游戏。除此之外，本申请中的APP可以是云游戏，或者支持云游戏试玩的短视频平台，或者云游戏试玩的APP应用市场。云游戏是一种以云计算(cloud computing)为基础的游戏方式，在云游戏的运行模式下，所有游戏都在服务器上运行，由服务器将渲染后的游戏画面进行视频压缩后通过网络传送给客户端。在客户端，终端设备不需要有高端处理器和显卡，只需要基本的视频解压能力即可。云计算是一种基于互联网的计算方式，通过这种方式，共享的软硬件资源和信息可以按需被提供给终端设备。而提供资源的网络被称为“云”。云游戏摆脱了对硬件的依赖，对服务器来说，仅仅需要提高服务器性能而不需要研发新主机，对客户端来说，可以得到更高的画质而不用配备高性能的终端设备。通常云游戏的流程为，首先用户操作终端设备连接到传送服务器并选择游戏，然后传送服务器把选择的游戏的信息发送给游戏服务器，此时用户的终端设备可以得到游戏服务器的统一资源定位符(uniform resource locator，URL)，通过该URL连接到游戏服务器开始玩游戏。
安装于终端设备上的直播平台的客户端负责与用户交互，将用户在直播平台的操作界面上执行操作所产生的操作指令发送给直播平台的服务器。或者通过在现场架设独立的信号采集设备(音频+视频)，将采集到的音视频数据导入直播平台的客户端，安装客户端的终端设备通过通信网络将音视频数据上传至服务器；直播平台的服务器负责直播平台数 据的管理、响应来自直播平台的客户端的操作指令以及渲染显示于客户端的画面。本申请中的直播平台例如可以是
Figure PCTCN2021074692-appb-000005
等。
图5为本申请云游戏云游戏直播方法实施例的流程图，如图5所示，本实施例的方法可以应用于上述图1所示的通信系统。该云游戏直播方法可以包括：
步骤501、在应用程序运行的过程中，云游戏的服务器生成云游戏运行视频。
如上所述，基于云计算的云游戏通常采用C/S结构，玩家需要在终端设备上先安装云游戏的客户端，然后点击云游戏的图标打开客户端。该客户端通过终端设备的通信功能连接到云游戏的服务器并开始运行。客户端可以将云游戏中的大量资源储存起来，用户通过客户端输入操作指令，客户端将其翻译成数据发送给服务器，服务器根据操作指令处理完后得到处理结果并发送给客户端，由客户端将其图形化后显示于终端设备的屏幕上。可以说，客户端是一个用户和服务器之间中介。由此可见，在云游戏运行的过程中，无论用户在客户端上进行了什么操作，根据上述云计算类的云游戏的原理，客户端均会根据用户操作进行翻译(即产生服务器可以识别的操作指令)。通常用户的操作可以包括在智能设备的触摸屏上的点击、拖动、滑动、长按等操作，也可以包括在计算机的鼠标上的点击、拖动等操作，键盘上的输入操作等，也可以包括通过麦克风输入的语音指令的操作，也可以包括通过传感器实现例如体感输入，悬浮输入等的操作，还可以包括其他输入设备上的相关操作等，本申请对此不做具体限定。而用户的操作所触发的云游戏的运行过程，在服务器上均有处理过程、画面渲染和记录等，因此服务器可以整合上述信息得到记录云游戏运行过程的视频。云游戏的服务器记录和生成云游戏运行视频，可以是基于用户的指示，例如用户启动云游戏的直播，或者是基于直播平台的服务器的指示，也可以是在云游戏开始时自动执行。
步骤502、云游戏的客户端收录周围的声音生成音频数据。
在应用程序运行的过程中，客户端会启动终端设备的麦克风，收录周围的声音，尤其是用户在操作云游戏时发出的声音。客户端采集到声音后生成服务器可识别的音频数据。客户端启动终端设备的麦克风可以是基于用户启动云游戏直播，也可以是基于用户主动打开麦克风，还可以是基于直播平台服务器的指示，经用户许可自动或手动打开麦克风。另外，启动的麦克风不限于位于运行云游戏客户端所在的终端设备上，还可以是其他终端设备，例如基于物联网，在该客户端所在终端设备周围的其他麦克风。
步骤503、云游戏的客户端将音频数据发送给云游戏的服务器。
客户端通过终端设备和服务器之间的通信网络将音频数据发送给服务器。该音频数据的发送可以是实时的流媒体，也可以是能保证直播实时性的定时发送。
步骤504、云游戏的服务器合并云游戏运行视频和音频得到音视频，并编码生成和直播平台匹配的码流。
云游戏的服务器在收到来自客户端的音频数据后，根据音频和本地生成的云游戏运行视频的时间戳对二者进行同步，然后将二者合成为包含音频和视频的音视频，该音视频便是反映了云游戏在运行过程中，用户对该云游戏的操作。关于音频和视频的同步，可参照如下过程：视频通常以帧率(视频在一秒内显示的帧数)表示播放速度，音频通常以采样率(音频一秒播放的样本的个数)表示播放速度。声卡和显卡均是以一帧数据来作为播放单位，如果单纯依赖帧率及采样率来进行播放，在理想条件下，应该是同步的，不会出现 偏差。以一个44.1KHz的音频流和24FPS的视频流为例，音频帧每个声道包含1024个采样点，则一个帧的播放时长为：(1024/44100)×1000ms＝23.22ms，视频帧播放时长为：1000ms/24＝41.67ms。理想情况下，音视频完全同步。但由于音频和视频本身的特性(例如音频输出是线性的，而视频输出可能是非线性)可能慢慢的就会出现音视频不同步的情况。为了解决音视频同步问题，引入了时间戳。首先选择一个参考时钟(要求参考时钟上的时间是线性递增的)，对应参考时钟的每个时间节点，将音频和视频分别分成一个一个的数据块；编码时给每个时间节点对应的音频数据块和视频数据块打上时间戳；播放时，根据音视频时间戳及参考时钟，来调整播放。所以时间戳就是音频和视频进行同步的关键。云游戏运行时客户端和云游戏服务器时间同步，本实施例中的音频和视频的时间戳的参考时间是相同的，因此能保证音频和视频两者同步。
可选的，用户在运行云游戏之前可以选择要直播的平台，因此云游戏的服务器可以根据选择直播平台的需求，对音视频进行编码得到与其匹配的码流。
步骤505、云游戏的服务器将码流发送给直播平台的服务器。
云游戏的服务器通过服务器之间的通信网络将码流发送给直播平台的服务器。
步骤506、直播平台的服务器播放音视频。
直播平台的服务器可以负责直播平台数据的管理、响应来自直播平台的客户端的操作指令以及渲染显示于客户端的画面。
本申请的云游戏直播方法复用了云端视频处理能力，由服务器直接生成游戏视频，终端设备只收录玩家的声音生成音频，服务器将游戏视频和玩家声音音频合并后传输给直播平台的服务器，减少了终端设备的上传操作，一方面可以实现清晰的游戏画面直播，以及无杂音的声音再现，另一方面玩家的终端设备不需要再负责录屏操作，可以提供无卡顿的游戏服务，减少终端设备的功耗。
示例性的，以直播玩家玩某款云游戏APP 1为例对本申请提供的云游戏直播方法进行说明。假设玩家在终端设备上安装了某云游戏APP 1，玩家打开该游戏APP 1进入游戏界面，通过点击、拖动、缩放、长按等操作玩游戏，云游戏APP 1的服务器响应来自客户端的操作指令以及渲染显示于客户端的画面，该服务器基于游戏数据生成游戏视频，该游戏视频包括了游戏的画面和游戏本身的声音和音效。终端设备在玩家玩游戏的同时启动麦克风录制玩家的声音生成音频，该音频通过网络传输给云游戏APP 1的服务器，该服务器根据游戏视频和玩家声音音频的时间戳对二者同步后合并成为一个音视频，服务器对该音视频进行编码后生成和直播平台相匹配的码流发送给直播平台的服务器，再由该直播平台播放给其他观看游戏直播的用户。
图6-9示例性的示出了本申请云游戏直播方法的界面示意图。如图6和7所示，云游戏APP 1提供了直播设置界面，该界面上提供了几个直播平台的选项(例如直播平台1-4)，玩家可以点击相应的控件选择直播平台(例如直播平台1)。如果玩家选择不对游戏过程进行直播，则可以点击“跳过”控件。玩家在直播设置界面的操作所产生的操作指令通过网络传输给服务器，如果操作指令中包括直播平台信息，则服务器在处理游戏数据、渲染游戏画面的同时根据该直播平台信息所指示的直播平台(例如直播平台1)的码流需求生成游戏视频，如果操作指令中不包括直播平台信息，或者操作指令指示“跳过”，则服务器只处理游戏数据、渲染游戏画面，并不生成游戏视频。
如图8所示，在直播平台选择界面之后，进入云游戏的操作界面。假设玩家选择了直播平台1，此时终端设备开启麦克风，收录周围的声音。直播游戏的玩家通常会在玩游戏的同时讲解游戏，因此麦克风收录玩家的声音生成音频。终端设备将玩家的声音音频通过网络传输给云游戏的服务器。云游戏的服务器在玩家玩游戏的过程中，会处理游戏数据、渲染游戏画面等，因此云游戏的服务器可以在该过程中根据游戏数据、画面以及玩家的游戏操作生成游戏视频，该游戏视频包括了游戏的画面和游戏本身的声音和音效。云游戏的服务器在收到玩家的声音音频后，根据玩家的声音音频和游戏视频各自的时间戳进行对齐，将二者进行合并生成玩家玩游戏的音视频，该音视频包括游戏的画面、游戏本身的声音和音效以及玩家讲解游戏的声音。根据玩家在直播设置界面上选择的直播平台1，云游戏的服务器将游戏音视频编码处理得到匹配直播平台1的码流，并将该码流发送给直播平台1的服务器。这样玩家的终端设备不需要在玩家玩游戏的同时录屏，减少了终端设备的消耗，而少量的音频上传并不会对游戏本身的上行传输造成干扰。
如图9所示，直播平台1的服务器接收来自云游戏的服务器的码流，解码该码流得到游戏视频和音频，通过网络播放给观看该游戏直播的多个用户，用户可以通过各自的终端设备连接至直播平台1的服务器，以观看玩家玩游戏的画面并听到玩家的讲解声音。
本申请的云游戏直播方法复用了云端视频处理能力，由服务器直接生成游戏视频，终端设备只收录玩家的声音生成音频，服务器将游戏视频和玩家声音音频合并后传输给直播平台的服务器，减少了终端设备的上传操作，一方面可以实现清晰的游戏画面直播，以及无杂音的声音再现，另一方面玩家的终端设备不需要再负责录屏操作，可以提供无卡顿的游戏服务，减少终端设备的功耗。
图10为本申请装置实施例的结构示意图，如图10所示，本实施例的装置可以应用于上述云游戏的服务器，也可以应用于上述终端设备，还可以应用于上述直播平台的服务器。该装置可以包括：处理模块1001、接收模块1002和发送模块1003。
当装置应用于上述云游戏的服务器时，处理模块1001，用于生成记录云游戏应用程序APP的运行过程的视频；接收模块1002，用于接收来自终端设备的音频，所述音频用于记录用户运行所述云游戏APP时的语音信息；所述处理模块1001，还用于获得所视频和所述音频合成的音视频；发送模块1003，用于对所述音视频编码后发送给直播平台的服务器。
在一种可能的实现方式中，所述处理模块1001，具体用于根据所述云游戏APP运行过程中接收到的操作指令渲染所述云游戏APP的画面，所述操作指令来自所述终端设备且由用户的操作触发生成；根据所述云游戏APP的画面生成所述视频。
在一种可能的实现方式中，所述处理模块1001，具体用于根据所述视频的时间戳和所述音频的时间戳对所述视频和所述音频进行同步并合并得到所述音视频。
在一种可能的实现方式中，所述发送模块1003，具体用于根据所述直播平台的码流要求对所述音视频进行编码；将编码后得到的码流发送给所述直播平台的服务器。
当装置应用于上述终端设备时，处理模块1001，用于在云游戏应用程序APP运行的过程中开启麦克风；通过所述麦克风收录用户运行所述云游戏APP时的语音信息得到音频；发送模块1003，用于将所述音频发送给云游戏APP的服务器。
在一种可能的实现方式中，所述处理模块1001，还用于通过输入装置获取用户的输 入操作；根据所述输入操作生成操作指令；所述发送模块1003，还用于将所述操作指令发送给所述云游戏APP的服务器。
在一种可能的实现方式中，接收模块1002，用于接收所述云游戏APP的服务器发送的画面数据；所述处理模块1101，还用于根据所述画面数据在所述屏幕上显示所述云游戏APP的界面。
当装置应用于上述直播平台的服务器时，接收模块1002，用于接收云游戏应用程序APP的服务器发送的音视频的码流，所述音视频包括用于记录所述云游戏APP的运行过程的视频和用于记录用户运行所述云游戏APP时的语音；处理模块1001，用于对所述音视频的码流进行解码得到所述音视频；发送模块1003，用于向请求直播的终端设备提供所述音视频的直播服务。
在实现过程中，上述方法实施例的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。处理器可以是通用处理器、数字信号处理器(digital signal processor,DSP)、特定应用集成电路(application-specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array,FPGA)或其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。本申请实施例公开的方法的步骤可以直接体现为硬件编码处理器执行完成，或者用编码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。
上述各实施例中提及的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(dynamic RAM，DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。应注意，本文描述的系统和方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请实施例的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请实施例所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请实施例各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(个人计算机，服务器，或者网络设备等)执行本申请实施例各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请实施例的具体实施方式，但本申请实施例的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请实施例揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请实施例的保护范围之内。因此，本申请实施例的保护范围应以所述权利要求的保护范围为准。

Claims (11)

  1. 一种云游戏直播方法，其特征在于，由服务器执行，包括：
    生成记录云游戏应用程序APP的运行过程的视频；
    接收来自终端设备的音频，所述音频记录用户运行所述云游戏APP时的语音信息；
    获得所述视频和所述音频合成的音视频；
    对所述音视频编码后发送给直播平台的服务器。
  2. 根据权利要求1所述的方法，其特征在于，所述生成用于记录APP的运行过程的视频，包括：
    根据所述云游戏APP运行过程中接收到的操作指令渲染所述云游戏APP的画面，所述操作指令来自所述终端设备且由用户的操作触发生成；
    根据所述云游戏APP的画面生成所述视频。
  3. 根据权利要求1或2所述的方法，其特征在于，所述获取所述视频和所述音频合成的音视频，包括：
    根据所述视频的时间戳和所述音频的时间戳对所述视频和所述音频进行同步并合并得到所述音视频。
  4. 根据权利要求1-3中任一项所述的方法，其特征在于，所述对所述音视频编码后发送给直播平台的服务器，包括：
    根据所述直播平台的码流要求对所述音视频进行编码；
    将编码后得到的码流发送给所述直播平台的服务器。
  5. 一种云游戏的服务端，其特征在于，包括：
    处理模块，用于生成记录云游戏应用程序APP的运行过程的视频；
    接收模块，用于接收来自终端设备的音频，所述音频用于记录用户运行所述云游戏APP时的语音信息；
    所述处理模块，还用于获得所述视频和所述音频合成的音视频；
    发送模块，用于对所述音视频编码后发送给直播平台的服务器。
  6. 根据权利要求5所述的装置，其特征在于，所述处理模块，具体用于根据所述云游戏APP运行过程中接收到的操作指令渲染所述云游戏APP的画面，所述操作指令来自所述终端设备且由用户的操作触发生成；根据所述云游戏APP的画面生成所述视频。
  7. 根据权利要求5或6所述的装置，其特征在于，所述处理模块，具体用于根据所述视频的时间戳和所述音频的时间戳对所述视频和所述音频进行同步并合并得到所述音视频。
  8. 根据权利要求5-7中任一项所述的装置，其特征在于，所述发送模块，具体用于根据所述直播平台的码流要求对所述音视频进行编码；将编码后得到的码流发送给所述直播平台的服务器。
  9. 一种服务器，其特征在于，包括：
    一个或多个处理器；
    存储器，用于存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实 现如权利要求1-4中任一项所述的方法。
  10. 一种计算机可读存储介质，其特征在于，包括计算机程序，所述计算机程序在计算机上被执行时，使得所述计算机执行权利要求1-4中任一项所述的方法。
  11. 一种计算机程序，其特征在于，当所述计算机程序被计算机执行时，用于执行权利要求1-4中任一项所述的方法。
PCT/CN2021/074692 2020-02-18 2021-02-01 云游戏直播方法和装置 WO2021164532A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010100145.4A CN111314724B (zh) 2020-02-18 2020-02-18 云游戏直播方法和装置
CN202010100145.4 2020-02-18

Publications (1)

Publication Number Publication Date
WO2021164532A1 true WO2021164532A1 (zh) 2021-08-26

Family

ID=71161681

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/074692 WO2021164532A1 (zh) 2020-02-18 2021-02-01 云游戏直播方法和装置

Country Status (2)

Country Link
CN (1) CN111314724B (zh)
WO (1) WO2021164532A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113973215A (zh) * 2021-10-25 2022-01-25 北京字节跳动网络技术有限公司 数据去重方法、装置及存储介质
CN114025116A (zh) * 2021-11-25 2022-02-08 北京字节跳动网络技术有限公司 视频生成方法、装置、可读介质和电子设备
CN114071111A (zh) * 2021-12-27 2022-02-18 北京百度网讯科技有限公司 视频播放方法和装置
CN114125536A (zh) * 2021-11-26 2022-03-01 北京字跳网络技术有限公司 一种游戏内直播交互方法、装置、计算机设备及存储介质
CN114339444A (zh) * 2021-12-10 2022-04-12 北京达佳互联信息技术有限公司 视频帧的播放时长调整方法、装置、设备及存储介质
CN114554229A (zh) * 2022-02-28 2022-05-27 广州点云科技有限公司 麦上用户语音状态标识方法、装置、终端设备及介质

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111314724B (zh) * 2020-02-18 2022-03-25 华为技术有限公司 云游戏直播方法和装置
CN111711832B (zh) * 2020-06-29 2022-09-27 腾讯科技（深圳）有限公司 视频直播方法、装置、设备及存储介质
CN111796794B (zh) * 2020-07-09 2023-12-22 西安万像电子科技有限公司 语音数据的处理方法、系统及虚拟机
CN113301359A (zh) * 2020-07-30 2021-08-24 阿里巴巴集团控股有限公司 音视频处理方法、装置及电子设备
CN111918082A (zh) * 2020-07-31 2020-11-10 北京视博云信息技术有限公司 云游戏直播方法、装置、存储介质及设备
CN113747181A (zh) * 2020-08-06 2021-12-03 上海达龙信息科技有限公司 基于远程桌面的网络直播方法、直播系统及电子设备
CN112055072B (zh) * 2020-08-31 2023-06-30 北京爱奇艺科技有限公司 云端音频输入方法、装置、云系统、电子设备与存储介质
CN112423028A (zh) * 2020-10-26 2021-02-26 深圳Tcl新技术有限公司 多媒体文件传输方法、装置、多媒体终端以及存储介质
CN114501040B (zh) * 2020-11-13 2024-01-30 腾讯科技（深圳）有限公司 直播视频流的获取方法、装置、设备及存储介质
CN112565911B (zh) * 2020-11-30 2021-10-08 腾讯科技（深圳）有限公司 弹幕显示方法、弹幕生成方法、装置、设备及存储介质
CN114666613B (zh) * 2020-12-22 2024-01-05 华为终端有限公司 一种网络直播方法及电子设备
CN113423018B (zh) * 2021-08-24 2021-11-02 腾讯科技（深圳）有限公司 一种游戏数据处理方法、装置及存储介质
CN112995692B (zh) * 2021-03-04 2023-05-02 广州虎牙科技有限公司 互动数据处理方法、装置、设备及介质
CN113457123B (zh) * 2021-07-21 2024-03-08 腾讯科技（深圳）有限公司 基于云游戏的互动方法、装置、电子设备及可读存储介质
CN116614650A (zh) * 2023-06-16 2023-08-18 上海随幻智能科技有限公司 音画同步的私域直播方法、系统、设备、芯片及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120184373A1 (en) * 2010-12-24 2012-07-19 Kim I-Gil Apparatus and method for providing a game service in cloud computing environment
CN105791958A (zh) * 2016-04-22 2016-07-20 北京小米移动软件有限公司 游戏直播方法及装置
CN109304031A (zh) * 2018-09-19 2019-02-05 电子科技大学 一种基于异构智能终端的虚拟化云游戏平台
CN110213601A (zh) * 2019-04-30 2019-09-06 大鱼互联科技(深圳)有限公司 一种基于云游戏的直播系统及直播方法、直播互动方法
CN111314724A (zh) * 2020-02-18 2020-06-19 华为技术有限公司 云游戏直播方法和装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10525343B2 (en) * 2012-10-05 2020-01-07 Aaa Internet Publishing Inc. Method of using cloud infrastructure and virtualization for bandwidth-efficient remote video streaming of software application operation by executing computer-executable instructions stored on a non-transitory computer-readable medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120184373A1 (en) * 2010-12-24 2012-07-19 Kim I-Gil Apparatus and method for providing a game service in cloud computing environment
CN105791958A (zh) * 2016-04-22 2016-07-20 北京小米移动软件有限公司 游戏直播方法及装置
US20170304735A1 (en) * 2016-04-22 2017-10-26 Beijing Xiaomi Mobile Software Co., Ltd. Method and Apparatus for Performing Live Broadcast on Game
CN109304031A (zh) * 2018-09-19 2019-02-05 电子科技大学 一种基于异构智能终端的虚拟化云游戏平台
CN110213601A (zh) * 2019-04-30 2019-09-06 大鱼互联科技(深圳)有限公司 一种基于云游戏的直播系统及直播方法、直播互动方法
CN111314724A (zh) * 2020-02-18 2020-06-19 华为技术有限公司 云游戏直播方法和装置

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113973215A (zh) * 2021-10-25 2022-01-25 北京字节跳动网络技术有限公司 数据去重方法、装置及存储介质
CN114025116A (zh) * 2021-11-25 2022-02-08 北京字节跳动网络技术有限公司 视频生成方法、装置、可读介质和电子设备
CN114025116B (zh) * 2021-11-25 2023-08-04 北京字节跳动网络技术有限公司 视频生成方法、装置、可读介质和电子设备
CN114125536A (zh) * 2021-11-26 2022-03-01 北京字跳网络技术有限公司 一种游戏内直播交互方法、装置、计算机设备及存储介质
CN114339444A (zh) * 2021-12-10 2022-04-12 北京达佳互联信息技术有限公司 视频帧的播放时长调整方法、装置、设备及存储介质
CN114071111A (zh) * 2021-12-27 2022-02-18 北京百度网讯科技有限公司 视频播放方法和装置
CN114554229A (zh) * 2022-02-28 2022-05-27 广州点云科技有限公司 麦上用户语音状态标识方法、装置、终端设备及介质
CN114554229B (zh) * 2022-02-28 2023-08-29 广州点云科技有限公司 麦上用户语音状态标识方法、装置、终端设备及介质

Also Published As

Publication number Publication date
CN111314724A (zh) 2020-06-19
CN111314724B (zh) 2022-03-25

Similar Documents

Publication Publication Date Title
WO2021164532A1 (zh) 云游戏直播方法和装置
TWI558146B (zh) Screen sharing methods and related equipment and communication systems
US8825468B2 (en) Mobile wireless display providing speech to speech translation and avatar simulating human attributes
WO2018059352A1 (zh) 直播视频流远程控制方法及装置
CN112394895B (zh) 画面跨设备显示方法与装置、电子设备
EP4362441A2 (en) Mobile wireless display providing speech to speech translation and avatar simulating human attributes
KR20170097414A (ko) 전자 장치 및 그의 동작 방법
WO2021218493A1 (zh) 终端控制方法、装置及终端设备
KR20140047232A (ko) 사용자 단말 장치, sns 제공 서버 및 그들의 컨텐츠 제공 방법
CN113225592B (zh) 基于Wi-Fi P2P的投屏方法和装置
CN112995759A (zh) 互动业务处理方法、系统、装置、设备及存储介质
WO2022135527A1 (zh) 一种视频录制方法及电子设备
KR20170043324A (ko) 전자 장치 및 전자 장치의 영상 인코딩 방법
CN112969093A (zh) 互动业务处理方法、装置、设备及存储介质
WO2022160959A1 (zh) 息屏显示方法、终端设备及芯片
WO2021164533A1 (zh) 渲染方法和装置
CN112506859B (zh) 一种维护硬盘数据的方法及显示设备
US20160085348A1 (en) Application host with distributed remote input and output interfaces
CN111742556A (zh) 用于共享实时内容数据的电子设备
CN116185438A (zh) 云服务的控制方法及相关装置
CN115250357A (zh) 终端设备、视频处理方法和电子设备
WO2022179530A1 (zh) 一种视频配音的方法、相关设备以及计算机可读存储介质
CN111770373B (zh) 基于直播的内容同步方法、装置、设备及存储介质
WO2022161128A1 (zh) 一种组合显示方法、移动设备及电子设备
WO2023185636A1 (zh) 图像显示方法及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21756559

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21756559

Country of ref document: EP

Kind code of ref document: A1