WO2023030057A1

WO2023030057A1 - Screen recording method, electronic device, and computer readable storage medium

Info

Publication number: WO2023030057A1
Application number: PCT/CN2022/113756
Authority: WO
Inventors: 罗芊
Original assignee: 华为技术有限公司
Priority date: 2021-08-30
Filing date: 2022-08-19
Publication date: 2023-03-09
Also published as: CN115734021A

Abstract

Embodiments of the present application relate to the field of terminals, and provide a screen recording method, comprising: in response to a first operation of starting a screen recording function, an electronic device obtaining identification information of an application associated with the first operation, the application associated with the first operation comprising an application running in the background of the electronic device; the electronic device obtaining a media stream of the application according to the identification information; and generating, according to the media stream of the application, a screen recording file corresponding to the application. The embodiments of the present application further provide an electronic device and a computer-readable storage medium. According to the present application, screen recording can be performed on an application running in the background of the electronic device, thereby improving screen recording experience of a user.

Description

A screen recording method, electronic equipment, and computer-readable storage medium

This application claims the priority of the Chinese patent application with the application number 202111006119.6 and the application name "A Screen Recording Method, Electronic Equipment, and Computer-Readable Storage Medium" submitted to the China Patent Office on August 30, 2021, and its entire content Incorporated in this application by reference.

technical field

The present application relates to the field of terminals, and in particular to a screen recording method, electronic equipment, and a computer-readable storage medium.

Background technique

The screen recording function is a commonly used function of electronic devices (such as mobile phones, tablet computers, etc.). The existing screen recording method is to record the interface displayed on the current screen, and save it as a screenshot or as a video file. However, the existing screen recording method cannot record the interface of the application program not displayed on the screen, which reduces the user experience.

Contents of the invention

In view of this, it is necessary to provide a screen recording method, which can realize the screen recording of the interface of the application program running in the background or the local device side to record the screen of the application window displayed on other electronic devices.

The first aspect of the embodiment of the present application discloses a screen recording method, which is applied to a first electronic device. The screen recording method includes: responding to the first operation of starting the screen recording function, acquiring the identification information of the application program associated with the first operation , wherein the application program associated with the first operation includes an application program running in the background of the first electronic device; acquiring the media stream of the application program according to the identification information; generating a recording corresponding to the application program according to the media stream of the application program screen file.

Using this technical solution, by obtaining the identification information of the application that needs to be recorded, and then generating a screen recording file based on the media stream obtained from the identification information, compared with the existing interface that can only record the front-end display, this application can realize electronic equipment to record the screen of one or more background applications.

In some embodiments, the identification information includes a process identification number (Identity Document, ID) of the application program or a package name of the application program.

In some embodiments, acquiring the media stream of the application program according to the identification information includes: determining a screen data buffer corresponding to the application program according to the identification information; acquiring the media stream of the application program from the screen data buffer.

With this technical solution, by extracting the media stream from the screen data buffer instead of recording the interface displayed on the current screen, it is possible to record the interface of one or more application programs of the electronic device, and the recorded screen The application program can run in the foreground of the electronic device, and can also run in the background of the electronic device.

In some embodiments, generating the screen recording file corresponding to the application program according to the media stream of the application program includes: performing image synthesis on the media stream of the application program to generate the screen recording file corresponding to the application program.

Using this technical solution, the electronic device implements image synthesis on the media stream of the application program by calling SurfaceFlinger or Direct Digital Synthesizer (DDS) algorithm to generate a screen recording file.

In some embodiments, the application corresponds to the first screen data buffer and the second screen data buffer, and generates a screen recording file corresponding to the application according to the media stream of the application, including: acquiring from the first screen data buffer The first media stream of the first media stream is image synthesized to generate the first screen recording sub-file; the second media stream obtained from the second screen data buffer is image synthesized to generate the second screen recording sub-file; the first screen recording sub-file is generated The sub-file is spliced with the second screen-recording sub-file to generate a screen-recording file corresponding to the application.

With this technical solution, two or more screen display contents in a distributed display scenario can be recorded on the electronic device side, and multiple screen display contents can be recorded by the same device to ensure that the recording clocks are synchronized. The electronic device side splices multiple screen recording sub-files to generate a screen recording file, so that the user can watch the contents of multiple display interfaces in one screen recording file.

In some embodiments, splicing the first screen recording sub-file and the second screen recording sub-file includes: combining the first screen recording sub-file and the second screen recording sub-file according to the time stamps of the first screen recording sub-file and the second screen recording sub-file The second screen recording sub-file is time-aligned; and the time-aligned first screen recording sub-file is spliced with the second screen recording sub-file.

Using this technical solution, the recording of multiple screen display contents by the same device can ensure the synchronization of recording clocks, and at the same time, multiple screen recording sub-files can be spliced on the electronic device side based on the timestamps of the recording sub-files to obtain a screen recording file. It enables users to watch the content of multiple display interfaces with time synchronization in one screen recording file.

In some embodiments, the application corresponds to the first screen data buffer and the second screen data buffer, and generating a screen recording file corresponding to the application according to the media stream of the application includes: obtaining the first screen data buffer from the first screen data buffer A media stream, and obtain the second media stream from the second screen data buffer; perform image synthesis and splicing on the first media stream and the second media stream, and generate a screen recording file corresponding to the application program.

With this technical solution, two or more screen display contents in a distributed display scenario can be recorded on the electronic device side, and multiple screen display contents can be recorded by the same device to ensure that the recording clocks are synchronized. The electronic device side aligns multiple media streams in time, and then performs image synthesis and splicing to generate a screen recording file, so that users can watch the contents of multiple display interfaces in one screen recording file.

In some embodiments, the image synthesis and splicing of the first media stream and the second media stream includes: time-aligning the first media stream and the second media stream according to the time stamps of the first media stream and the second media stream Processing: performing image synthesis and stitching on the first media stream and the second media stream that have undergone time-alignment processing.

Using this technical solution, recording multiple screen display contents by the same device can ensure the synchronization of recording clocks. At the same time, multiple media streams can be time-aligned based on the timestamp of the media stream on the electronic device side, and then image synthesis and splicing are performed to obtain a Screen recording files allow users to watch the content of multiple display interfaces synchronized in time in one screen recording file.

In some embodiments, the screen recording file includes a first interface and a second interface, and the first interface and the second interface are displayed in left and right split screens, or the first interface and the second interface are displayed in up and down split screens, or the first interface A part of the area or the entire area is suspended and displayed on the second interface.

By adopting this technical solution, multiple interfaces in the screen recording file can be set to be displayed in the upper and lower split screens, the left and right split screens, or displayed in the form of a floating window according to the actual needs of the user, so as to improve the user experience.

In some embodiments, the screen recording method further includes: in response to the screen recording instruction sent by the second electronic device, acquiring identification information of an application program associated with the screen recording instruction.

With this technical solution, for screen projection scenarios, the user can select the screen recording operation on the source device side or the destination device side, and designate the source device for screen recording applications.

In some embodiments, the application program associated with the first operation further includes screen projection to an application program on the second electronic device.

By adopting this technical solution, it is possible to realize screen recording of an application projected to a second electronic device on the side of the first electronic device, and the screen projected application may run in the foreground or background of the first electronic device.

In the second aspect, the embodiment of the present application provides a screen recording method, using the second electronic device, the first electronic device projects one or more application program windows on the second electronic device for display, the screen recording method includes: receiving the second electronic device A media stream sent by an electronic device; in response to the first operation of starting the screen recording function, obtaining the identification information of the application window associated with the first operation; obtaining the application window associated with the first operation from the received media stream The media stream corresponding to the identification information; generate a screen recording file according to the obtained media stream.

With this technical solution, by obtaining the identification information of the application window to be recorded, and then extracting the media stream corresponding to the identification information from the received media stream based on the identification information to generate a screen recording file, it can be realized on the second electronic device side. Record the screen of the application interface displayed on the screen.

In some embodiments, the identification information includes a process ID of the application window and a device ID of the first electronic device.

With this technical solution, the process ID of the application window and the device ID of the first electronic device can be used as the unique identifier of the application window for screen recording on the second electronic device side.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the screen recording method as described in the first aspect or the second aspect.

In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the instructions described in the first aspect or the second aspect. screen recording method described above.

In the fifth aspect, the embodiment of the present application provides a computer program product, which, when the computer program product is run on a computer, causes the computer to execute the screen recording method as described in the first aspect or the second aspect.

According to a sixth aspect, an apparatus is provided, and the apparatus has a function of realizing the behavior of the first electronic device in the method provided in the first aspect or the second aspect. The functions may be implemented by hardware, or may be implemented by executing corresponding software through hardware. Hardware or software includes one or more modules corresponding to the above-mentioned functions.

It can be understood that the computer-readable storage medium described in the third aspect, the electronic device described in the fourth aspect, the computer program product described in the fifth aspect, and the device described in the sixth aspect are all related to the above-mentioned first aspect. The method of the aspect or the second aspect is corresponding, therefore, the beneficial effect that it can achieve can refer to the beneficial effect in the corresponding method provided above, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a software structure of an electronic device provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of an application scenario of a screen recording method provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of an application scenario of a screen recording method provided by another embodiment of the present application;

FIG. 5 is a schematic diagram of an application scenario of a screen recording method provided by another embodiment of the present application;

FIG. 6 is a schematic flowchart of a screen recording method provided by an embodiment of the present application;

FIG. 7 is a schematic flowchart of a screen recording method provided in another embodiment of the present application;

FIG. 8 is a schematic structural diagram of a possible first electronic device provided by an embodiment of the present application.

Detailed ways

It should be noted that in this application, "at least one" means one or more, and "multiple" means two or more than two. "And/or" describes the association relationship of associated objects, indicating that there can be three types of relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B Can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of this application and the drawings are used to distinguish similar objects, not to Describe a specific order or sequence.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In order to facilitate understanding, some descriptions of concepts related to the embodiments of the present application are provided as examples for reference.

Multiple electronic devices can form a hyper terminal through a communication network. A hyper terminal can be defined as presenting multiple electronic devices as a unified whole, organically integrating multiple electronic devices in terms of user experience, and improving the user experience of multiple electronic devices in the environment. user experience. Each electronic device is a part of the HyperTerminal, and the background application of a certain electronic device may be used on other electronic devices in the HyperTerminal, or other electronic devices in the HyperTerminal can be viewed on a certain electronic device some running information.

Electronic devices can be mobile phones, foldable electronic devices, tablet computers, personal computers (PC), laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPC), netbooks, cellular Telephone, personal digital assistant (PDA), augmented reality (augmented reality, AR) device, virtual reality (virtual reality, VR) device, artificial intelligence (artificial intelligence, AI) device, wearable device, vehicle-mounted device , smart home equipment, or at least one of smart city equipment, the embodiment of the present application does not specifically limit the specific type of electronic equipment. The communication network may be a wired network or a wireless network. For example, the communication network may be a local area network (local area networks, LAN), or a wide area network (wide area networks, WAN), such as the Internet. When the communication network is a local area network, exemplary, the communication network may be a short distance communication network such as a wifi hotspot network, a wifi P2P network, a bluetooth network, a zigbee network or a near field communication (near field communication, NFC) network. When the communication network is a wide area network, exemplary, the communication network may be a third-generation mobile communication technology (3rd-generation wireless telephone technology, 3G) network, a fourth-generation mobile communication technology (the 4th generation mobile communication technology, 4G ) network, the fifth-generation mobile communication technology (5th-generation mobile communication technology, 5G) network, the future evolution of the public land mobile network (public land mobile network, PLMN) or the Internet, etc.

An electronic device may have one or more application programs installed. An application program may be referred to as an application, and is a software program capable of realizing one or more specific functions. For example, instant messaging applications, video applications, audio applications, image capture applications, cloud desktop applications, and so on. Among them, instant messaging applications, for example, may include SMS applications,

wait. The image capturing application may include, for example, a camera application (system camera or third-party camera application). Video applications, for example, can include

etc. Audio applications, for example may include

etc. The applications mentioned in the following embodiments may be system applications installed on the electronic device when it leaves the factory, or third-party applications downloaded by the user from the Internet or obtained from other electronic devices during the use of the electronic device.

Electronic equipment including but not limited to carrying

or other operating systems.

FIG. 1 shows a schematic structural diagram of an electronic device 10 .

The electronic device 10 may include a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 130, a wireless communication module 140, an audio module 150, a sensor module 160, a camera module 170, and a display screen 180 wait.

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10 . In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown, or combine some components, or separate some components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

The processor can generate an operation control signal according to the instruction opcode and the timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 may be a cache memory. The memory may store instructions or data used by the processor 110 or used frequently. If the processor 110 needs to use the instruction or data, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and /or universal serial bus (universal serial bus, USB) interface, etc. The processor 110 may be connected to modules such as an audio module, a wireless communication module, a display, and a camera through at least one of the above interfaces.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present application is only a schematic illustration, and does not constitute a structural limitation of the electronic device 10 . In other embodiments of the present application, the electronic device 10 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The wireless communication function of the electronic device 10 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 130 , the wireless communication module 140 , a modem processor, a baseband processor, and the like.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 10 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 130 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 10 . The mobile communication module 130 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 130 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 130 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 130 and at least part of the modules of the processor 110 may be set in the same device.

A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 180 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 130 or other functional modules.

The wireless communication module 140 can provide applications on the electronic device 10 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), Bluetooth low power consumption (bluetooth low energy, BLE), ultra wide band (UWB), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 140 may be one or more devices integrating at least one communication processing module. The wireless communication module 140 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 140 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 10 is coupled to the mobile communication module 130, and the antenna 2 is coupled to the wireless communication module 140, so that the electronic device 10 can communicate with the network and other electronic devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and/or IR technology, etc. The GNSS can include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou satellite navigation system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi- zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 10 may implement a display function through a GPU, a display screen 180, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 180 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

Sensor modules include touch sensors, pressure sensors, fingerprint sensors, etc. The camera module 170 includes a camera. The display screen 180 is used to display images, videos and the like. The display screen 180 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 10 may include one or more display screens 180 .

The electronic device 10 can realize the camera function through the camera module 170, ISP, video codec, GPU, display screen 180, application processor AP, neural network processor NPU, and the like.

Digital signal processors are used to process digital signals and can also process other digital signals. For example, when the electronic device 10 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. Electronic device 10 may support one or more video codecs. In this way, the electronic device 10 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 10 can be implemented through the NPU, such as image recognition, face recognition, voice recognition, text understanding, and the like.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card. Or transfer music, video and other files from electronic devices to external memory cards.

The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 10 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional methods or data processing of the electronic device 10 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 150 may also be used to encode and decode audio signals. In some embodiments, the audio module 150 may be set in the processor 110 , or some functional modules of the audio module 150 may be set in the processor 110 .

The software system of the electronic device 10 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 10 .

FIG. 2 is a block diagram of the software structure of the electronic device 10 according to the embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into five layers, which are application program layer, application program framework layer, Android runtime (Android runtime, ART) and native C/C++ library, hardware abstraction layer (Hardware Abstract Layer, HAL) and the kernel layer.

The application layer can consist of a series of application packages.

As shown in Figure 2, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer may include window managers, content providers, view systems, resource managers, notification managers, activity managers, input managers, and so on.

The window manager provides window management service (Window Manager Service, WMS). WMS can be used for window management, window animation management, surface management and as a transfer station for input systems.

Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

The activity manager can provide activity management service (Activity Manager Service, AMS), AMS can be used for system components (such as activities, services, content providers, broadcast receivers) to start, switch, schedule, and manage and schedule application processes .

The input manager can provide input management service (Input Manager Service, IMS), and IMS can be used to manage the input of the system, such as touch screen input, key input, sensor input, etc. IMS fetches events from input device nodes, and distributes events to appropriate windows through interaction with WMS.

The Android runtime includes the core library and the Android runtime. The Android runtime is responsible for converting source code into machine code. The Android runtime mainly includes the use of ahead of time (ahead or time, AOT) compilation technology and just in time (just in time, JIT) compilation technology.

The core library is mainly used to provide basic Java class library functions, such as basic data structure, mathematics, IO, tools, database, network and other libraries. The core library provides APIs for users to develop Android applications.

A native C/C++ library can include multiple functional modules. For example: surface manager (surface manager), media framework (Media Framework), libc, OpenGL ES, SQLite, Webkit, etc.

Among them, the surface manager is used to manage the display subsystem, and provides the fusion of 2D and 3D layers for multiple applications. The media framework supports playback and recording of various commonly used audio and video formats, as well as still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides the drawing and manipulation of 2D graphics and 3D graphics in applications. SQLite provides a lightweight relational database for applications of the electronic device 10 .

The hardware abstraction layer runs in user space, encapsulates the kernel layer driver, and provides a call interface to the upper layer.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

The workflow of the software and hardware of the electronic device 10 will be exemplarily described below in conjunction with capturing and photographing scenes.

When the display screen receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, and other information). Raw input events are stored at the kernel level. The application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Take the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example. The camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. Cameras capture still images or video.

A schematic diagram of an application scenario of a screen recording method provided by an embodiment of the present invention is exemplarily introduced below with reference to FIG. 3 .

This embodiment may include two electronic devices (such as the electronic device 10 shown in FIG. 1 ). For ease of description, the two electronic devices are referred to as the first electronic device 100 and the second electronic device 200 hereinafter. The first electronic device 100 and the second electronic device 200 may be under the same local area network. For example, the first electronic device 100 and the second electronic device 200 establish a communication connection through Wi-Fi. The first electronic device 100 includes a first display screen 1001. The second electronic device 200 includes a second display screen 2001 . The first electronic device 100 is the source device, and the second electronic device 200 is the destination device. An application running on the first electronic device 100 can be projected and displayed on the second electronic device 200. The program can be a desktop application, a gallery apps, calling apps, maps apps, music apps, video apps, etc.

The first electronic device 100 and the second electronic device 200 include but are not limited to devices such as mobile phones, tablet computers, PCs, and notebook computers. The following description will be made by taking the first electronic device 100 as a mobile phone and the second electronic device 200 as a notebook computer as an example.

The first electronic device 100 can open n applications APP ₁ -APP _n , where n is a positive integer. When the first electronic device 100 projects m applications APP ₁ to APP _m to the second electronic device 200, the second electronic device 200 can use the existing distributed display technology to generate images corresponding to the m applications APP ₁ to APP _m . m application windows, m is a positive integer. For example, when multiple applications APP ₁ ~ APP _m are running, the frame data output by SurfaceFlinger can be encoded and compressed by the video encoder respectively to generate encoded data corresponding to the application windows of multiple applications APP ₁ ~ APP _m , for example, in YUV format The data. The first electronic device 100 can send the encoded data of multiple applications APP ₁ to APP _m output by the video encoder in real time to the second electronic device 200, and the second electronic device 200 can decode the encoded data and draw multiple application APPs. ₁ ~ Application window of APP _m .

The user can control and operate one or more applications APP ₁ -APP _m displayed on the projected screen at the side of the second electronic device, for example, the user can control and operate the applications APP ₁ -APP _m through the mouse and keyboard of the second electronic device 200 .

For example, the first electronic device 100 is opened with a calculator application APP ₁ , a gallery application APP ₂ and a chat application APP ₃ . The user can perform touch control on the interface of the gallery application APP ₂ to evoke the screen projection function icon. When the user clicks the screen projection function icon, a list of devices that can perform screen projection is popped up, and the device list may include information such as device identity (identity document, ID) or device name. When the user selects the second electronic device 200 as the destination device in the device list, the first electronic device 100 can project the application window of the gallery application APP ₂ to the second electronic device 200, and the second electronic device 200 can generate a ₂ corresponding to the application window. The user can project the application window of the chat application APP ₃ to the second electronic device 200 by adopting a screen projection operation method similar to that of the gallery application APP ₂ on the side of the first electronic device 100 .

As shown in FIG. 3 , the calculator application APP ₁ is displayed in the foreground of the first electronic device 100, the gallery application APP ₂ and the chat application APP ₃ are running in the background of the first electronic device 100, and the gallery application APP ₂ and the chat application APP ₃ are running in the background of the first electronic device 100. The projected screen is displayed on the second electronic device 200 .

When it is necessary to record the screen on the first electronic device 100 , the user can select an application that needs to record the screen on the first electronic device 100 or the second electronic device 200 . For example, the operating system of the first electronic device 100 has a built-in screen recording function, and the user can click the screen recording icon on the first electronic device 100 to start the screen recording function. When the user clicks on the screen recording icon, the operating system can obtain the application information currently opened by the first electronic device 100, and pop up a list of applications that can be selected for screen recording. The application list can display the name or thumbnail of the application. When the user selects one or more applications in the application list, the first electronic device 100 may obtain the identification information of the one or more applications according to the user's operation, the identification information may refer to the process ID of the application or the package name of the application, The first electronic device 100 may start recording the screen of the one or more applications according to the identification information of the applications. The process ID is a numerical value used by the kernel of the operating system of the device to uniquely identify the process of the application.

In some embodiments, each application running on the electronic device has a unique process ID and package name, and each electronic device has a unique device ID, so a certain application in a certain electronic device can be uniquely determined through the process ID and device ID. process as the unique identification information of the application. For example, the device ID of the electronic device is the International Mobile Equipment Identity, and the applications installed on the electronic device include

The device ID of the electronic device, the process ID and package name of each application are shown in Table 1 below:

Table 1

In some embodiments, when the screen recording scene only includes the first electronic device 100, when the first electronic device 100 records the screen of the foreground application or the background application, only the process ID or package name of the application can be used as the unique Identification information. When the screen recording scene includes multiple electronic devices, such as the scene where the application screen of the first electronic device 100 is projected on the second electronic device 200, and the first electronic device 100 records the screen of the foreground application or the background application, it can also only use The process ID or package name of the application is used as the unique identification information of the application. When the second electronic device 200 records the screen of the screen projection application, it may use the process ID and the device ID of the first electronic device 100 as the unique identification information of the screen projection application.

The first electronic device 100 may also be installed with a third-party screen recording application or pre-installed with a screen recording application when leaving the factory, and the first electronic device 100 may start the screen recording application. The screen recording application can obtain the application information currently opened by the first electronic device 100. When the user selects one or more applications in the screen recording application of the first electronic device 100 to record the screen, the first electronic device 100 can record the screen according to the user's operation. After acquiring the identification information of the one or more applications, the first electronic device 100 may start recording the screen of the one or more applications according to the identification information of the applications.

In some embodiments, the user can also designate a screen-casting application (such as a gallery application APP ₂ and a chat application APP ₃ ) that needs to be recorded on the second electronic device 200 , and the specific screen recording operation is performed by the first electronic device 100 . For example, the second electronic device 200 can generate a screen recording instruction in response to the user's operation, and the second electronic device 200 can send the screen recording instruction to the first electronic device 100 through a Wi-Fi channel, so that the first electronic device 100 can record the screen instruction specified by the user. app to record screen. For example, the second electronic device 200 is installed with a screen recording application, and the screen recording application of the second electronic device 200 may obtain the application information of the first electronic device 100 currently displayed on the screen of the second electronic device 200 . When the user selects one or more applications in the screen recording application of the second electronic device 200 to record the screen, the second electronic device 200 may generate a screen recording instruction associated with the one or more applications in response to the user operation, the screen recording instruction Can include identifying information for the application. The second electronic device 200 may send a screen recording instruction to the first electronic device 100, so as to control the first electronic device 100 to start recording the screen of one or more applications selected by the user.

When the first electronic device 100 starts an application, the process of the application can request the WMS to create a Surface for the application window. Multiple application processes can correspond to multiple Surfaces. Surface can refer to a data buffer provided by screen data consumers to screen data producers. Screen data producers can produce image content on Surface, and screen data consumers can draw the produced data on the screen. Or convert to the data it needs. For example, in the screen recording scene of the first electronic device 100, a screen data producer, such as a virtual display (VirtualDisplay), can generate data on Surface, and a screen data consumer, such as a multimedia recorder (MediaRecoder), can obtain data from Surface. After the recorder acquires the data, it can encode the data into a video using a coding method such as fast forward moving picture expert group (FFMPEG).

Assuming that the first electronic device 100 has opened the calculator application APP ₁ , the gallery application APP ₂ and the chat application APP ₃ , the calculator application APP ₁ is displayed in the foreground of the first electronic device 100, and the gallery application APP ₂ and the chat application APP ₃ are in the first display. An electronic device 100 is running in the background, and the user chooses to record the screen of the gallery application APP ₂ and the chat application APP ₃ . The first electronic device 100 may acquire the identification information of the gallery application APP ₂ and the identification information of the chat application APP ₃ according to the user's screen recording operation. The first electronic device 100 obtains the media stream of the gallery application APP ₂ from the corresponding Surface according to the identification information of the gallery application APP ₂ , and the first electronic device 100 may generate a recording corresponding to the gallery application APP ₂ according to the media stream of the gallery application APP ₂ . screen file f1. For example, the first electronic device 100 may call SurfaceFlinger or a direct digital synthesis (Direct Digital Synthesizer, DDS) algorithm to perform image synthesis on the media stream of the gallery application APP ₂ to generate the screen recording file f1.

Similarly, the first electronic device 100 can obtain the media stream of the chat application APP ₃ from the corresponding Surface according to the identification information of the chat application APP ₃ , and the first electronic device 100 can generate and communicate with the chat application APP ₃ according to the media stream of the chat application APP 3. ₃ corresponds to the screen recording file f2.

In some embodiments, the first electronic device 100 may encode the media stream on the Surface to form a screen recording file during or before the media stream on the Surface is sent to the second electronic device 200 .

Assume that the user chooses to record the screen of the calculator application APP ₁ . The first electronic device 100 may acquire the identification information of the calculator application APP ₁ according to the user's screen recording operation. The first electronic device 100 can obtain the media stream of the calculator application APP ₁ from the corresponding Surface according to the identification information of the calculator application APP ₁ , and the first electronic device 100 can generate and communicate with the calculator application APP ₁ according to the media stream of the calculator application APP 1. Screen recording file f3 corresponding to APP ₁ . For example, the first electronic device 100 invokes the SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the computer application APP ₁ to generate the screen recording file f3.

In some embodiments, the second electronic device 200 can also record the screen of the screen-casting application (such as the gallery application APP ₂ and the chat application APP ₃ ). For example, the operating system of the second electronic device 200 has a built-in screen recording function, and the user can click the screen recording icon on the second electronic device 200 to start the screen recording function. When the user clicks the screen recording icon, the operating system can obtain the application information currently displayed on the screen of the second electronic device 200, and pop up a list of applications that can be selected for screen recording. The application list can display the name or thumbnail of the application. When the user selects one or more applications in the application list, the second electronic device 200 can obtain the identification information of the one or more applications according to the user's operation, and the second electronic device 200 can implement the identification information of one or more applications according to the identification information of the applications. or multiple apps for screen recording.

In the screen projection display scenario, the first electronic device 100 sends the media stream of the gallery application APP ₂ and the media stream of the chat application APP ₃ to the second electronic device 200, and the second electronic device 200 decodes the received media stream and sends it to the second electronic device 200. display, realizing displaying the gallery application APP2 and the chat application _APP3 on the second display screen 2001.

The second electronic device 200 can obtain the media stream of the chat application APP ₃ from the received media stream according to the identification information of the chat application APP ₃ , and the identification information can include a process ID and a device ID (determined by the device ID that the media stream is generated by the first sent by the electronic device 100), the second electronic device 200 can call SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the chat application APP ₃ , and generate a screen recording file corresponding to the chat application APP ₃ . For example, the second electronic device 200 may perform image synthesis on the media stream of the chat application APP ₃ before sending the media stream of the chat application APP ₃ for display.

The second electronic device 200 can also obtain the media stream of the picture application APP ₂ from the received media stream according to the identification information of the picture application APP ₂ , and the second electronic device 200 can invoke the SurfaceFlinger or DDS algorithm to apply the media stream of the picture application APP ₂ Perform image synthesis to generate a screen recording file corresponding to the image application APP ₂ .

A schematic diagram of an application scenario of a screen recording method provided by another embodiment of the present invention is exemplarily introduced below with reference to FIG. 4 .

This embodiment includes a first electronic device 100 and a second electronic device 200 . The following description will be made by taking the first electronic device 100 as a mobile phone and the second electronic device 200 as an in-vehicle device as an example. The first electronic device 100 can establish a communication connection with the second electronic device 200 through Bluetooth, and the first electronic device 100 and the second electronic device 200 can establish a communication channel between the mobile phone and the car through the vehicle-mounted mobile phone mapping scheme to realize the application of the mobile phone. And the service extends to cars. Car phone mapping solutions include but are not limited to HiCar, CarPlay, or CarLife, etc.

As shown in FIG. 4 , the first electronic device 100 projects the map application APP _{11 ,} the phone application APP ₁₂ , and the music application APP ₁₃ on the second electronic device 200 for display. In the existing in-vehicle projection scenario, when an application on the mobile phone is projected on the screen of the in-vehicle device, the application is running in the background of the mobile phone, and the mobile phone can no longer open the application, and the mobile phone cannot switch the application to Screen recording in the foreground. Since the car equipment generally does not support the screen recording function, users cannot record the screen projection application on the mobile phone side or the car side.

In this application, in order to realize the screen recording of the screen projection application in the vehicle screen projection scene, the first electronic device 100 can obtain the screen recording information to be recorded according to the user's screen recording operation (operation on the first electronic device 100 side). Identification information of the application, where the identification information may refer to a process ID of the application or a package name of the application. The first electronic device 100 may acquire the media stream of the application from the corresponding Surface according to the identification information of the application, and the first electronic device 100 may generate a screen recording file corresponding to the application according to the media stream of the application. For example, the first electronic device 100 invokes the SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the application to generate a screen recording file.

When the first electronic device 100 starts the map application APP _11, the phone application APP ₁₂ and the music application APP ₁₃ , the process of the map application APP ₁₁ can request the WMS to create a Surface for its application window, and the process of the phone application APP ₁₂ can request the WMS to create a surface for the application window. Its application window creates a Surface, and the process of the music application APP ₁₃ can request the WMS to create a Surface for its application window. The following takes the screen recording of the map application APP ₁₁ and the phone application APP ₁₂ as an example. Assuming that the operating system of the first electronic device 100 has a built-in screen recording function, the user can click the screen recording icon on the first electronic device 100 to start the screen recording function. When the user clicks the screen recording icon, the operating system can obtain information about applications currently opened by the first electronic device 100, and pop up a list of applications that can be selected for screen recording. For example, the application list can display the name of the application. When the user selects the names of the map application APP ₁₁ and the phone application APP ₁₂ in the application list, the first electronic device 100 can obtain the identification information of the map application APP ₁₁ and the phone application APP ₁₂ , and the first electronic device 100 can obtain the identification information of the map application APP 11 and the phone application APP ₁₂ . The identification information of the map application APP ₁₁ is obtained from the corresponding Surface, and the first electronic device 100 performs image synthesis on the media stream of the map application APP ₁₁ to generate a screen recording file corresponding to the map application APP ₁₁ . The first electronic device 100 can also obtain the media stream of the phone application APP ₁₂ from the corresponding Surface according to the identification information of the phone application APP ₁₂ , and the first electronic device 100 performs image synthesis on the media stream of the phone application APP ₁₂ to generate a The screen recording file corresponding to APP ₁₂ .

In some embodiments, the first electronic device 100 may also be installed with a third-party screen recording application or pre-installed with a screen recording application at the factory, and the user may also start the screen recording application in the vehicle screen projection scene to record the screen. After the first electronic device 100 starts the screen recording application, the screen recording application can obtain the application information currently opened by the first electronic device 100, when the user chooses to record the map application APP ₁₁ and the phone application APP ₁₂ in the screen recording application The first electronic device 100 can obtain the identification information of the map application APP ₁₁ and the phone application APP ₁₂ according to the user's operation, and the first electronic device 100 can obtain the identification information of the map application APP ₁₁ and the phone application APP ₁₂ according to the identification information of the map application APP 11 and the phone application APP ₁₂ . For the media stream of the phone application APP ₁₂ , the first electronic device 100 performs image synthesis on the media stream of the map application APP ₁₁ and the media stream of the phone application APP ₁₂ respectively, and generates a screen recording file corresponding to the map application APP ₁₁ and a screen recording file corresponding to the phone application APP 11. The screen recording file corresponding to APP ₁₂ .

A schematic diagram of an application scenario of a screen recording method provided by another embodiment of the present invention is exemplarily introduced below with reference to FIG. 5 .

This embodiment includes a first electronic device 100 and a second electronic device 200 . The first electronic device 100 may be a small-screen terminal device such as a mobile phone or a tablet, and the second electronic device 200 may be a large-screen display device (such as a TV, a smart screen, etc.). Hereinafter, the first electronic device 100 is a mobile phone and the second electronic device 200 is a television as an example for description. The first electronic device 100 can establish a communication connection with the second electronic device 200 through Wi-Fi. In a distributed game scenario, the first electronic device 100 can be used as an operating handle, and the second electronic device 200 can be used as a display terminal to display the game screen projected by the first electronic device 100 .

For the screen recording in the existing distributed game scene, the screen display content of the first electronic device 100 is recorded on the first electronic device 100 side, and the screen display content of the second electronic device 200 is recorded on the second electronic device 200 side. For screen recording, it is impossible to record the screen display content on both sides at the same time. Currently, you can only record on both sides separately, and then splicing the screen recording files on both sides. Due to the existence of network delay, the clocks of the devices on both sides may not be accurately synchronized, and there is a time difference in the spliced screen recording files.

In this application, in order to achieve synchronous (clock synchronization) recording of screen display content on both sides of the screen in a distributed game scene, the screen display of the first electronic device 100 and the second electronic device 200 on the side of the first electronic device 100 screen recording content. In the distributed game scene shown in Figure 5, the mobile phone is used as the game control terminal, and the TV is used as the game display terminal. The device IDs of the two devices (mobile phone and TV) are different, because the game applications run by the two devices belong to the same game developer. , use the same package name, that is, the package names of the game applications on the two devices are the same. The kernels of the operating systems of different devices generate different values for uniquely identifying the processes of the applications running on them, that is, the process IDs of the game applications of the two devices are different.

In a distributed game scenario, when the first electronic device 100 projects the game application to the second electronic device 200, the first electronic device 100 can automatically enable the game control control, and at this time the first electronic device 100 displays the game control interface, The second electronic device 200 displays a game screen. The user can customize the position and size of the buttons in the game control interface. The WMS creates two mutually independent Surfaces for the process of the game application in the first electronic device 100 . Two mutually independent Surfaces can correspond to the same process ID, and screen data producers can produce game control interface content on one Surface, and produce game screen content on the other Surface.

In some embodiments, when the first electronic device 100 casts the screen of the game application to the second electronic device 200, the first electronic device 100 may display a handle function icon. When the user clicks on the handle function icon, the first electronic device 100 enters a "handle mode", and the first electronic device 100 displays a game control interface.

The first electronic device 100 can respond to the user's screen recording operation and obtain media streams from the two Surfaces. The first electronic device 100 can time-align the two media streams according to the timestamps of the two media streams, and then align the elapsed time. The processed two media streams are image synthesized and spliced to generate a screen recording file including the game operation interface and the game screen interface, and then the user can watch the screen recording of the time-synchronized game operation interface and the game screen interface, improving the user experience . For example, splicing processing may mean that the game operation interface and the game screen interface in the screen recording file are arranged in a top-bottom split screen, or in a left-right split screen arrangement.

In some embodiments, the game operation interface in the screen recording file can also be suspended and displayed in a designated area of the game screen interface, and the position of this area can be customized by the user. For example, when the user performs a screen recording operation on the first electronic device 100 , the screen recording operation includes setting the game operation interface to be displayed in a floating area in the lower left corner or the lower right corner of the game screen interface.

In some embodiments, the first electronic device 100 can also obtain media streams from the two Surfaces in response to the user's screen recording operation, and the first electronic device 100 can respectively perform image synthesis on the two media streams to generate an image corresponding to the game operation. The first screen recording sub-file of the interface and the second screen recording sub-file corresponding to the game screen interface. The first electronic device 100 may then perform time alignment and splicing processing on the first screen recording sub-file and the second screen recording sub-file according to the timestamp of the first screen recording sub-file and the time stamp of the second screen recording sub-file to generate Including the screen recording files of the game operation interface and the game screen interface, and then the user can watch the time-synchronized screen recording of the game operation interface and the game screen interface.

Assuming that the operating system of the first electronic device 100 has a built-in screen recording function, the user can click the screen recording icon on the first electronic device 100 to start the screen recording function. When the user clicks the screen recording icon, the operating system can obtain information about the game application currently opened by the first electronic device 100, and the first electronic device 100 obtains the media stream of the game application from the corresponding Surface according to the identification information of the game application, such as the game application The identification information of the game application includes the process ID of the game application, and the first electronic device 100 can determine the two corresponding Surfaces according to the process ID of the game application. The first electronic device 100 can obtain media streams from the two Surfaces, and the first electronic device 100 can time-align the two media streams according to the time stamps of the two media streams, and then perform time-alignment on the two media streams that have undergone time-aligned processing. Image synthesis and splicing processing, generating screen recording files including game operation interface and game screen interface.

Referring to FIG. 6 , the embodiment of the present application provides a method for recording a screen, which is applied to the first electronic device 100 . In this embodiment, the screen recording method may include:

61. In response to the first operation of starting the screen recording function, acquire identification information of an application associated with the first operation.

In some embodiments, when the user wishes to record the screen, the user may start the screen recording function of the first electronic device 100 , for example, perform a first operation to start the screen recording function of the first electronic device 100 . Wherein, the first operation may be, for example, an operation in which the user presses a physical button (for example: press and hold the power button and the volume up button at the same time), the user performs a predefined gesture (for example: slide three fingers down on the screen, tap the knuckle on the screen, Shake the mobile phone), the user operates the corresponding button on the screen (click the screen recording control in the notification bar), the user enters a voice command, the user performs any operation on the screen recording application, etc. The embodiment of the present application does not specifically limit the screen recording operation.

In some embodiments, the applications associated with the first operation may include one or more applications, and the one or more application programs may run in the foreground of the first electronic device 100, or run in the background, or run partly in the foreground, partly running in the background. The identification information may refer to information that can uniquely identify the application, for example, the identification information may include a process ID of the application or a package name of the application. In other embodiments, the identification information may also include the process ID of the application and the ID of the device running the application, or the package name of the application and the ID of the device running the application.

In some embodiments, in the screen projection scenario, when one or more applications running on the first electronic device 100 are projected to be displayed on the second electronic device 200, the user can also specify on the second electronic device 200 that the screen needs to be recorded. The screen-casting application of the screen, and the specific screen-recording operation is performed by the first electronic device 100 . The first electronic device 100 may also, in response to the screen recording instruction sent by the second electronic device 200, obtain identification information of an application associated with the screen recording instruction. For example, the second electronic device 200 may generate a screen recording instruction associated with one or more applications in response to a user operation, and the screen recording instruction may include identification information of the application. The second electronic device 200 may send the screen recording instruction to the first electronic device 100, and then the first electronic device 100 may acquire the identification information of the application that needs to record the screen.

62. Acquire the media stream of the application according to the identification information.

In some embodiments, when the first electronic device 100 starts an application, the process of the application may request the WMS to create a Surface for the application window. Each application process can correspond to at least one Surface. Surface can refer to a data buffer provided by screen data consumers to screen data producers. Screen data producers can produce image content on Surface, and screen data consumers can consume the produced data to the screen ( drawn) or converted into the required data. The first electronic device 100 may determine the Surface corresponding to the application according to the identification information of the application, and the first electronic device 100 may acquire the media stream of the application from the Surface corresponding to the application.

63. Generate a screen recording file corresponding to the application according to the media stream of the application.

In some embodiments, when the first electronic device 100 obtains the media stream of the application, the first electronic device 100 can invoke the SurfaceFlinger or DDS algorithm to perform image synthesis on the media stream of the application, and generate a screen recording file corresponding to the application .

In some embodiments, in some application scenarios, the first electronic device 100 may be able to obtain two media streams according to the identification information of a certain application. For example, in a distributed game scenario, when the first electronic device 100 uses the game When the application screen is projected to the second electronic device 200, the first electronic device 100 can automatically turn on the game control control, at this time the first electronic device 100 displays the game control interface, and the second electronic device 200 displays the game screen. The WMS creates two mutually independent Surfaces (for example, the first Surface and the second Surface) for the process of the game application in the first electronic device 100 . Two mutually independent Surfaces can correspond to the same process ID, and screen data producers can produce game control interface content on the first Surface, and produce game screen content on the second Surface.

The first electronic device 100 can respond to the user's screen recording operation to obtain media streams from the first Surface and the second Surface, and the first electronic device 100 can time-align the two media streams according to the time stamps of the two media streams, and then Perform image synthesis and splicing processing on the two media streams that have been time-aligned to generate a screen recording file including the game operation interface and the game screen interface, and then the user can watch the time-synchronized screen recording of the game operation interface and the game screen interface, Improve user experience. For example, splicing processing may mean that the game operation interface and the game screen interface in the screen recording file are arranged in a vertical split screen, or in a left and right split screen arrangement.

In some embodiments, the first electronic device 100 can also obtain media streams from the first Surface and the second Surface in response to the user's screen recording operation, and the first electronic device 100 can respectively perform image synthesis on the two media streams to generate The first screen recording sub-file corresponding to the game operation interface and the second screen recording sub-file corresponding to the game screen interface. The first electronic device 100 may then perform time alignment and splicing processing on the first screen recording sub-file and the second screen recording sub-file according to the timestamp of the first screen recording sub-file and the time stamp of the second screen recording sub-file to generate Including the screen recording files of the game operation interface and the game screen interface, and then the user can watch the time-synchronized screen recording of the game operation interface and the game screen interface.

The above screen recording method obtains the identification information of the application to be recorded, and then generates a screen recording file based on the media stream obtained from the identification information. Compared with the existing screen recording of the local desktop, this application can realize only one or Multiple application interfaces are used to record the screen, and the recorded application can run in the foreground of the electronic device or in the background of the electronic device. At the same time, two or two applications in the distributed display scenario can be viewed on the local electronic device side. More than one screen display content can be recorded. Recording multiple screen display contents by the same device can ensure that the recording clock is synchronized, and allows users to watch the content of multiple display interfaces with time synchronization in one screen recording file.

Referring to FIG. 7 , the embodiment of the present application provides a screen recording method, which is applied to the first electronic device 100 . The first electronic device 100 and the second electronic device 200 may be under the same local area network, for example, the first electronic device 100 and the second electronic device 200 establish a communication connection through Wi-Fi or Bluetooth, and the first electronic device 100 includes a first display screen 1001 , the second electronic device 200 includes a second display screen 2001 . The first electronic device 100 is a source device, and the second electronic device 200 is a destination device. One or more applications running on the first electronic device 100 can be displayed on the second electronic device 200 through projection. In this embodiment, the screen recording method may include:

71. When the first electronic device 100 detects the first operation for starting the screen recording function, the first electronic device 100 acquires the identification information of the application associated with the first operation, wherein the application associated with the first operation includes screen projection to the application of the second electronic device.

In some embodiments, in the screen casting scenario, when the user wishes to record the screen, the user can start the screen recording function of the first electronic device 100 , for example, perform a first operation to start the screen recording function of the first electronic device 100 . Wherein, the first operation may be, for example, an operation of pressing a physical button by the user, performing a predefined gesture by the user, operating a corresponding button on the screen by the user, inputting a voice command by the user, or performing an operation by the user on a screen recording application. The embodiment of the present application does not specifically limit the screen recording operation.

In some embodiments, the applications associated with the first operation may include one or more applications, and the one or more applications may run in the foreground of the first electronic device 100, or run in the background (such as projecting a screen on the second electronic device 100). device 200), or partly running in the foreground and partly running in the background (for example, projecting a screen on the second electronic device 200). The identification information may refer to information that can uniquely identify the application, for example, the identification information may include a process ID of the application or a package name of the application. In other embodiments, the identification information may also include the ID of the device running the application.

72. When the first electronic device 100 receives the screen recording instruction sent by the second electronic device 200, the first electronic device 100 obtains the identification information of the application associated with the screen recording instruction, wherein the application associated with the screen recording instruction includes screen projection to the application of the second electronic device.

In some embodiments, the user can also designate a screen-casting application that needs to be recorded on the second electronic device 200 , and the specific screen recording operation is performed by the first electronic device 100 . When the first electronic device 100 receives the screen recording instruction sent by the second electronic device 200, the first electronic device 100 acquires the identification information of the application associated with the screen recording instruction. For example, the second electronic device 200 may generate a screen recording instruction associated with one or more applications in response to a user operation, and the screen recording instruction may include identification information of the application. The second electronic device 200 may send the screen recording instruction to the first electronic device 100, and then the first electronic device 100 may acquire the identification information of the application that needs to record the screen.

In some embodiments, the first electronic device 100 may choose to execute step 71 or choose to execute step 72 .

73. The first electronic device 100 acquires the media stream of the application according to the identification information.

In some embodiments, the first electronic device 100 may determine the Surface corresponding to the application according to the identification information of the application, and the first electronic device 100 may acquire the media stream of the application from the Surface corresponding to the application.

74. The first electronic device 100 generates a screen recording file corresponding to the application according to the media stream of the application.

In some embodiments, in some application scenarios, the first electronic device 100 may be able to obtain two media streams according to the identification information of a certain application. For example, in a distributed game scenario, the first electronic device 100 displays the game control interface, the second electronic device 200 displays a game screen. The WMS creates two mutually independent Surfaces (for example, the first Surface and the second Surface) for the process of the game application in the first electronic device 100 . Two mutually independent Surfaces can correspond to the same process ID, and screen data producers can produce game control interface content on the first Surface, and produce game screen content on the second Surface.

In some embodiments, the second electronic device 200 may also record the screen of the screen-casting application. In an existing screen projection display scenario, the second electronic device 200 may receive the media stream sent by the first electronic device 100, decode the received media stream and send it for display. When the user wishes to record the screen on the side of the second electronic device 200 , the user can start the screen recording function of the second electronic device 200 , for example, perform a second operation to start the screen recording function of the second electronic device 200 . Wherein, the second operation may be, for example, an operation in which the user presses a physical button, the user performs a predefined gesture, the user operates a corresponding button on the screen, the user enters a voice command, or the user performs an operation on a screen recording application. The embodiment of the present application does not specifically limit the screen recording operation.

When the second electronic device 200 detects the second operation for starting the screen recording function, the second electronic device 200 obtains the identification information of the application program associated with the second operation, wherein the identification information of the application program associated with the second operation It may include the process ID of the application program and the device ID of the first electronic device 100 (the device ID of the first electronic device 100 determines that the media stream really originates from the first electronic device 100). The second electronic device 200 may acquire the media stream corresponding to the identification information of the application program associated with the second operation from the received media stream, and the second electronic device 200 generates a screen recording file according to the acquired media stream. For example, the second electronic device 200 may perform image synthesis on the media stream to generate a screen recording file before sending the received media stream for display.

The above-mentioned screen recording method obtains the identification information of the application to be recorded, and then generates a screen recording file based on the media stream obtained from the identification information. Compared with the existing inability to record the screen of the screen projection application, this application can realize the backend of the electronic device The application or screen projection to other electronic devices can be used to record the screen, and at the same time, the local electronic device can record the screen content of two or more screens in the distributed display scene, and the same device can record multiple The content displayed on one screen can ensure the synchronization of the recording clock, and enable the user to watch the content of multiple display interfaces with time synchronization in one screen recording file.

Referring to FIG. 8 , it is a schematic diagram of a hardware structure of the first electronic device 100 provided in the embodiment of the present application. As shown in FIG. 8 , the first electronic device 100 may include a first display screen 1001 , a first processor 1002 , a first memory 1003 and a first communication bus 1004 . The first memory 1003 is used to store one or more first computer programs 1005 . One or more first computer programs 1005 are configured to be executed by the first processor 1002 . The one or more first computer programs 1005 include instructions, and the above instructions can be used to implement the screen recording method as described in FIG. 6 or FIG. 7 in the first electronic device 100 .

It can be understood that, the structure illustrated in this embodiment does not constitute a specific limitation on the first electronic device 100 . In other embodiments, the first electronic device 100 may include more or fewer components than shown in the illustration, or combine some components, or separate some components, or arrange different components.

This embodiment also provides a computer storage medium, in which computer instructions are stored, and when the computer instructions are run on the electronic device, the electronic device is made to perform the above related method steps to implement the screen recording method in the above embodiment.

This embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the above related steps, so as to realize the screen recording method in the above embodiment.

In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, The processor can execute the computer-executable instructions stored in the memory, so that the chip executes the screen recording methods in the above method embodiments.

Wherein, the first electronic device, computer storage medium, computer program product or chip provided in this embodiment is all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the above provided The beneficial effects of the corresponding method will not be repeated here.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be combined or may be integrated into another device, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and a component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only a specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application should be covered within the protection scope of the application .

Claims

A screen recording method applied to a first electronic device, characterized in that the screen recording method comprises:

Responding to the first operation of starting the screen recording function, acquiring identification information of an application associated with the first operation, wherein the application associated with the first operation includes an application running in the background of the first electronic device program;

Acquire the media stream of the application program according to the identification information;

A screen recording file corresponding to the application is generated according to the media stream of the application.
The screen recording method according to claim 1, wherein the identification information includes a process ID number (ID) of the application program or a package name of the application program.
The screen recording method according to claim 1 or 2, wherein said obtaining the media stream of said application program according to said identification information comprises:

determining a screen data buffer corresponding to the application program according to the identification information;

Obtain the media stream of the application program from the screen data buffer.
The screen recording method according to any one of claims 1 to 3, wherein the generating a screen recording file corresponding to the application program according to the media stream of the application program includes:

Image synthesis is performed on the media stream of the application program to generate a screen recording file corresponding to the application program.
The screen recording method according to any one of claims 1 to 3, wherein the application corresponds to the first screen data buffer and the second screen data buffer, and the media stream according to the application Generate a screen recording file corresponding to the application, including:

Performing image synthesis on the first media stream obtained from the first screen data buffer to generate the first screen recording sub-file;

performing image synthesis on the second media stream obtained from the second screen data buffer to generate a second screen recording sub-file;

Splicing the first screen recording sub-file and the second screen recording sub-file to generate a screen recording file corresponding to the application.
The screen recording method according to claim 5, wherein the splicing the first screen recording sub-file and the second screen recording sub-file comprises:

performing time alignment processing on the first screen recording subfile and the second screen recording subfile according to the timestamps of the first screen recording subfile and the second screen recording subfile;

Splicing the first screen-recording sub-file and the second screen-recording sub-file that have been time-aligned.
The screen recording method according to any one of claims 1 to 3, wherein the application corresponds to the first screen data buffer and the second screen data buffer, and the media stream according to the application Generate a screen recording file corresponding to the application, including:

obtaining a first media stream from the first screen data buffer, and obtaining a second media stream from the second screen data buffer;

Image synthesis and splicing are performed on the first media stream and the second media stream to generate a screen recording file corresponding to the application program.
The screen recording method according to claim 7, wherein said performing image synthesis and splicing of said first media stream and said second media stream comprises:

performing time alignment processing on the first media stream and the second media stream according to the timestamps of the first media stream and the second media stream;

Image synthesis and splicing are performed on the first media stream and the second media stream that have been time-aligned.
The screen recording method according to any one of claims 5 to 8, wherein the screen recording file includes a first interface and a second interface, and the first interface and the second interface are left and right split screens display, or the first interface and the second interface are displayed in a split screen up and down, or part or all of the first interface is displayed in a floating manner on the second interface.
The screen recording method according to any one of claims 1 to 9, wherein the screen recording method further comprises:

In response to the screen recording instruction sent by the second electronic device, the identification information of the application program associated with the screen recording instruction is acquired.
The screen recording method according to any one of claims 1 to 9, wherein the application program associated with the first operation further includes an application program for projecting a screen display to a second electronic device.
A screen recording method, applied to a second electronic device, the first electronic device projects one or more application program windows on the second electronic device for display, wherein the screen recording method includes:

receiving the media stream sent by the first electronic device;

Responding to a first operation of starting a screen recording function, acquiring identification information of an application window associated with the first operation;

Acquire a media stream corresponding to the identification information of the application window associated with the first operation from the received media stream;

Generate a screen recording file based on the obtained media stream.
The screen recording method according to claim 12, wherein the identification information includes a process ID of the application window and a device ID of the first electronic device.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and when the computer instructions are run on an electronic device, the electronic device executes the electronic device according to claims 1 to 13. The screen recording method described in any one.
An electronic device, characterized in that the electronic device includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the method described in claim 1. To the screen recording method described in any one of claim 13.