WO2024037421A1

WO2024037421A1 - Operation method, electronic device, and medium

Info

Publication number: WO2024037421A1
Application number: PCT/CN2023/112297
Authority: WO
Inventors: 胡怡洁; 严凯凯
Original assignee: 华为技术有限公司
Priority date: 2022-08-18
Filing date: 2023-08-10
Publication date: 2024-02-22
Also published as: CN117631919A

Abstract

The present application relates to the technical field of communications, and discloses an operation method, an electronic device, and a medium. The operation method comprises: a first electronic device acquiring a first video and first information associated with the first video, wherein the first information comprises interface information of at least one display interface recorded in a screen recording process and identification information of a target control operated by a user on the at least one display interface; playing back the first video, and executing an operation on the basis of the first information in the playback process of the first video. According to the solution, in a video playback process, the first electronic device can execute the same operation as the operation executed by a user on an electronic device during video recording, such that the user does not need to operate by himself/herself, thereby improving the operation convenience and the user experience.

Description

An operating method, electronic device and medium

This application claims priority to the Chinese patent application submitted to the China Patent Office on August 18, 2022, with application number 202210995317.8 and the application title "An operating method, electronic device and medium", the entire content of which is incorporated herein by reference. Applying.

Technical field

The present application relates to the field of communication technology, and in particular to an operating method, electronic equipment and media.

Background technique

At present, electronic devices such as smartphones have been widely used in various customer groups. For example, electronic devices such as smartphones are also widely used in middle-aged and elderly people. However, most users in the middle-aged and elderly people do some operations while using mobile phones. Not very familiar with the problem, or unable to perform some operations at all, such as not being able to connect to Bluetooth and WIFI, not being able to adjust the font display size of the mobile phone, etc. At this time, family members are not around and cannot provide on-site guidance. In addition, in addition to middle-aged and elderly users, other users may not be able to use some operations, resulting in users being unable to perform certain operations.

In the existing technology, some solutions are shown in Figure 1a. User A generally clicks the screen recording control 101 of the mobile phone 100 to implement the process of performing the target operation, such as the process of performing some Bluetooth connection or WIFI connection operations in the settings application 102. , recorded as video 103, and then as shown in Figure 1b, the video 103 is sent to the mobile phone 200 of user B who needs to learn the operation through communication software. The user who needs to learn the operation can learn how to perform the target operation according to the video content. However, in the above solution, user B easily forgets how to operate after watching the video, so he needs to watch the video repeatedly to learn. In this way, it is more cumbersome for users to implement the above operations and reduces the user experience.

Contents of the invention

In order to solve the above-mentioned technical problem of relatively complicated operations for users to record videos, embodiments of the present application provide an operating method, electronic equipment, and media.

In a first aspect, embodiments of the present application provide an operating method applied to a first electronic device. The method includes: obtaining a first video and first information associated with the first video, wherein the first information includes a screen The interface information of at least one display interface recorded during the recording process and the identification information of the target control operated by the user on the at least one display interface; play the first video, and during the process of playing the first video, based on the perform the operation on the first information.

It can be understood that in the embodiment of this application, the first video may refer to the recorded video mentioned in the embodiment of this application.

Based on the above solution, during the process of playing the video, the first electronic device can perform the same operation as the operation performed by the user on the electronic device in the recorded video, which can eliminate the need for the user to operate by himself and improve the convenience of operation and user experience.

In addition, the first information includes the identification information of the target control operated by the user on the display interface, where the identification information of the target space may be the layout ID information of the target space, since the identification information of each operation target is unique. Even if the interface settings of the first electronic device and the electronic device in the recorded video are different, the first electronic device can still accurately find the corresponding control according to the identification of the target control and perform the corresponding operation, effectively improving the accuracy of the operation.

In a possible implementation, the operation is consistent with the user's operation on the at least one display interface in the first video.

In a possible implementation, the first video and the first information are stored as the same file, or the first video and the first information are stored as different files.

In a possible implementation, the method includes: the first information also includes the user's operation mode of the target control on the at least one display interface.

In a possible implementation, performing an operation based on the first information includes: obtaining interface information of a first display interface in the first information, and determining the interface information of the first display interface based on the interface information of the first display interface. The first to-be-operated interface of the electronic device; obtains identification information of a target control operated by the user on the first display interface, and determines the first operation based on the identification information of the target control operated by the user on the first display interface. The to-be-operated control on the to-be-operated interface; obtains the user's operating mode for the target control on the first display interface, and controls the to-be-operated control based on the user's operating mode for the target control on the first display interface. Controls perform corresponding operations; Obtain the interface information of the second display interface in the first information, and determine the second to-be-operated interface of the electronic device based on the interface information of the second display interface.

In a possible implementation manner, when the operation mode of the target control is a multi-state operation mode, the first information further includes auxiliary information corresponding to the operation mode.

In a possible implementation, the multi-state operation mode includes a typing operation and an adjustment operation, the auxiliary information corresponding to the typing operation includes input content, and the auxiliary information corresponding to the adjustment operation includes an adjustment state.

It can be understood that the adjustment may include sliding operations, such as sliding progress bars, etc.

It can be understood that in the embodiment of the present application, when the second electronic device performs operations with multiple states such as sliding and typing, it can accurately obtain the operation that should be performed on the target control based on the auxiliary information, thereby improving the accuracy of the operation.

In a possible implementation, the interface information includes an interface package name, an interface application name, and an interface activity name.

In a possible implementation, the first electronic device includes a first display area and a second display area;

When the first electronic device plays the first video, the first display area displays the playback screen of the first video, and the second display area displays the first electronic device performing the operation. Operation process.

In a possible implementation, the first display area overlaps a partial area of the second display area, or the second display area overlaps a partial area of the first display area.

In a possible implementation, the second display area and the first display area are displayed in separate screens.

In a possible implementation, the third electronic device is connected to the first electronic device, the third electronic device displays the playback screen of the first video, and the first electronic device displays the first video. An electronic device performs the operation process of the operation.

In a possible implementation, the third electronic device is connected to the first electronic device, the third electronic device displays the operation process of the first electronic device performing the operation, and the first electronic device The device displays the playback screen of the first video.

In a possible implementation, the third electronic device is connected to the first electronic device, and the first electronic device displays the operation process and/or the first video of the first electronic device performing the operation. the playback screen; the third electronic device includes a third display area and a fourth display area, the third display area displays the display content of the first electronic device, and the fourth display area displays the operation process or The playback screen of the first video.

In a possible implementation, the method further includes: upon receiving an instruction to exit playback and detecting that the operation is not completed, sending prompt information to a display interface of the first electronic device.

In a possible implementation, the first video is marked with multiple time points, and each time point corresponds to part of the user's operation during the screen recording process.

In a possible implementation, the method further includes: fixing associated partial time points among the plurality of time points, so that videos corresponding to the associated partial time points are played continuously.

It can be understood that in the embodiment of the present application, the associated time point may be a time point during the operation of executing any function, such as multiple time points during the operation of turning on the WIFI function in the embodiment of the present application. Fixing the associated time points prevents the electronic device from dragging the progress bar back and forth between the associated time points when playing a video, preventing the user from dragging the progress bar when recording a video that includes the execution of multiple functions. The user can perform subsequent viewing during the intermediate process of performing a certain function, causing the problem that the electronic device performs redundant operations or performs less operations.

In a possible implementation, when the first video is played to a target time point and the operation corresponding to the target time point is not completed, the playback of the first video is stopped; After the operation corresponding to the target time point is completed, the first video continues to be played.

In a possible implementation, the method further includes: editing the first video to obtain a second video.

In a possible implementation, the editing processing includes cutting processing and/or splitting processing and/or reorganization processing.

In a possible implementation, the first electronic device stores the first video and the first information in MOV format, EXIF format or compressed package format.

In a possible implementation, during the process of playing the first video, performing an operation based on the first information includes: detecting a play instruction, playing the first video, and performing an operation on the first video. When a video is played to the first time point, the operation corresponding to the first time point is added to the task queue, and the operation in the task queue is executed based on the first information; when the first video is played to At the second time point, add the operation corresponding to the second time point to the task queue.

In a second aspect, this application provides an operating method, applied to a second electronic device, which is characterized by including: detecting a screen recording instruction; starting screen recording, and generating a first video and first information, where the first information includes The interface information of at least one display interface recorded during the screen recording process and the identification information of the target control of the user's operation of the at least one display interface, wherein the first information can be used: when the electronic device plays the first video In the process, the corresponding operation is performed based on the first information.

In a possible implementation, the second electronic device sends the first video and the first information to the first electronic device.

In a third aspect, the present application provides an electronic device, which is characterized in that it includes: a memory for storing instructions executed by one or more processors of the electronic device, and the processor is a processor of the electronic device. One of one or more processors, configured to perform the operation methods mentioned in this application.

In a fourth aspect, the present application provides a readable storage medium. Instructions are stored on the readable medium. When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the operation method mentioned in the present application.

In a fifth aspect, the present application provides a computer program product, including: execution instructions, the execution instructions are stored in a readable storage medium, and at least one processor of an electronic device can read the execution instructions from the readable storage medium. Instructions, the at least one processor executes the execution instructions so that the electronic device implements the operation method mentioned in this application.

Description of drawings

Figure 1a shows a schematic diagram of a screen recording process according to some embodiments of the prior art.

Figure 1b shows a schematic diagram of playing a recorded video according to some embodiments of the prior art.

Figure 2a shows a schematic diagram of a screen recording process according to some embodiments of the present application.

Figure 2b shows a schematic diagram of a screen recording process according to some embodiments of the present application.

Figure 3 shows a schematic diagram of a screen recording process according to some embodiments of the present application.

Figures 4a-4h show a schematic diagram of a screen recording process according to some embodiments of the present application.

Figure 5 shows a schematic structural diagram of an electronic device according to some embodiments of the present application.

Figure 6 shows a schematic flowchart of an operating method according to some embodiments of the present application.

Figures 7a-7c respectively show a schematic diagram of recording a video according to some embodiments of the present application.

Figure 7d shows a storage schematic diagram of recorded video and operation information respectively according to some embodiments of the present application.

Figure 8 shows a schematic diagram of playback of recorded video according to some embodiments of the present application.

Figures 9a-9b respectively show a schematic diagram of playback of a recorded video according to some embodiments of the present application.

Figure 10 shows a schematic diagram of playback of recorded video according to some embodiments of the present application.

Figure 11 shows a schematic diagram of playback of recorded video according to some embodiments of the present application.

Figure 12 shows a schematic diagram of playback of recorded video according to some embodiments of the present application.

Figure 13 shows a schematic diagram of a scene of exiting playback during playback of a recorded video according to some embodiments of the present application.

Figure 14 shows a schematic diagram of setting time points in a recorded video according to some embodiments of the present application.

Figure 15 shows a schematic diagram of associated fixation of time points in a recorded video according to some embodiments of the present application.

Figure 16 shows a schematic diagram of the generation of a task queue during the playback process of a recorded video according to some embodiments of the present application.

17a-17b illustrate a schematic diagram of a video clip according to some embodiments of the present application.

Figure 18 shows a schematic diagram of the hardware structure of an electronic device according to some embodiments of the present application.

Detailed ways

Illustrative embodiments of the present application include, but are not limited to, a method of operation, electronic equipment, and media.

In order to solve the above problem, an embodiment of the present application provides an operation method, which includes: after detecting the recording triggering operation, the first electronic device starts screen recording, and records the user's actions on the target controls triggered by each interface during the recording process. Location information, for example, as shown in Figure 2a, during the recording process, the user clicks on the WIFI control 4031 on the setting interface 403 of the mobile phone 100, then the electronic device can record the location information of the WIFI control 4031. For example, the location information of the WIFI control 4031 is "Set the second control row of the interface".

And the first electronic device can send the recorded video and the corresponding position information of the target control of each interface to the second electronic device. While playing the video, the second electronic device can automatically trigger the target control based on the position information of the target control on each interface, that is, to achieve the same Record actions consistent with the actions in the video. For example, during the recording process of the first electronic device, the user clicks on the WIFI control 4031 on the setting interface 403 of the mobile phone 100 . Then, while playing the video, the second electronic device will also trigger the WIFI control on the second electronic device based on the location information of the WIFI control 4031.

The above solution can automatically trigger corresponding operations while playing videos, simplifying the operation process. However, in some embodiments, the positions of different controls on different devices may change, for example, as shown in Figure 2a. The location information of the WIFI control 4031 on the mobile phone 100 is "the second control row of the setting interface", and the location of the WIFI control 2011 on the mobile phone 200 is the third control row of the setting interface 201. When the mobile phone 200 plays the above recorded video, it will still trigger the second control row of the setting interface 201 of the mobile phone 200 based on the above location information, resulting in an incorrect operation.

In addition, in the above solution, the first electronic device can only record the location of the target control. Therefore, if there is a control with multiple state operations, such as multiple adjustment states, multiple typing states, etc., the second electronic device cannot determine the target control. The specific status makes it impossible to perform specific operations. For example, as shown in Figure 2b, during the recording process, the user slides the brightness adjustment control 1001 on the shortcut interface 101 of the mobile phone 100. In the above solution, the first electronic device can only record the location of the brightness adjustment control 1001. For example, in The last control row of the shortcut interface. However, the degree of brightness adjustment is not recorded. Therefore, based on the above solution, the second electronic device cannot determine the degree of brightness adjustment based on the position of the brightness adjustment control. Therefore, the second electronic device cannot adjust the brightness to the same level as the brightness in the recorded video.

In order to solve the above problem, an embodiment of the present application provides an operation method, which includes: the first electronic device detects a recording trigger operation. For example, as shown in Figure 3, the user clicks the operation recording control 4011 on the mobile phone 100, then Start screen recording, that is, record the change process of screen display caused by user operation, obtain the recorded video and the first information, and store the recorded video and the first information. The first information may be the user's operation information on the first electronic device during the recording process. The first information includes the interface information of each interface recorded during the screen recording process and the user's operation information on each interface. The user's operation information on each interface may include the user's operation style, operation target and operation target. The layout ID.

It can be understood that in the embodiment of the present application, the user's operation information on each interface may also include information such as the coordinates of the operation target, the position of the control row where the operation target is located, the size of the control, and other information. Among them, the operation target may refer to the target control operated by the user.

It can be understood that in the embodiment of the present application, the recorded video and the first information can be stored as two separate files or as one file.

In the embodiment of the present application, the first electronic device can send the recorded video and the corresponding first information to the second electronic device. The second electronic device detects the user's playback operation of the recorded video, then plays the recorded video, and plays the recorded video. During the video process, a corresponding operation is performed on the second electronic device based on the first information, so that the second electronic device can perform the same operation as the user performed on the first electronic device in the video recording. For example, in the recorded video played by the second electronic device, the user's operation on the first electronic device is to open the settings application, then the second electronic device can open the settings application based on the received first information.

Among them, the interface information of each interface may include the package name, application name, activity name, etc. of the interface.

The user's operation information on each interface may include the user's operation style (such as clicking, typing, filling information, sliding, selecting, typing, etc.), operation target, layout ID of the operation target, etc. Among them, the layout ID of each operation target is unique. For example, if the layout ID of the WIFI control in the settings interface is "12345", then on any device, the layout ID of the WIFI control will be "12345". In this way, even if the interface settings of the second electronic device and the first electronic device are different, the electronic device can still accurately find the corresponding control according to the layout ID of the target control and perform the corresponding operation.

For example, during the recording process, if the user clicks on the WIFI control on the setting interface of the first electronic device, the first electronic device will record the interface information of the setting interface, including, for example, the application name "settings" and the package name "com. setting" and the activity name "setting interface", and can record the user's operation information on the setting interface. For example, it can include: the user's operation style "click", the operation target "WIFI control", and the layout ID of "WIFI control" "12345" .

It can be understood that when some operation styles are operations with multiple states such as sliding and typing, the operation information may also include auxiliary information corresponding to the operation style. For example, when the operation style is a typing operation, there can be multiple types of typing content, so the typing auxiliary information can also include the typing content; when the operation style is a sliding operation, there are many types of sliding positions, so the sliding auxiliary information can also include the sliding position. , or adjust the position of the progress bar, etc.

For example, as shown in Figure 2b, during the recording process, the user slid the brightness adjustment control 1001 on the shortcut interface 101 of the mobile phone 100. In the embodiment of the present application, in addition to recording the user's operation style "sliding", the operation target "brightness adjustment control" ”, in addition to the layout ID “12347” of the brightness adjustment control 1001, the brightness adjustment degree can also be recorded, that is, the adjustment position of the brightness bar, for example, the percentage of the overall brightness bar. The position of thirty. In this way, when the second electronic device performs a brightness adjustment operation based on the operation information, the brightness can be accurately adjusted according to the adjustment position of the brightness bar.

Based on the above solution, when the second electronic device performs operations with multiple states such as sliding and typing, it can accurately obtain the operation that should be performed on the target control based on the auxiliary information, thereby improving the accuracy of the operation.

It can be understood that in some embodiments, when the operation target is content in a list form such as a table, the index ID of the corresponding position of the content in the list can be used instead of the layout ID.

It can be understood that in some embodiments, the first electronic device can send the recorded video and the corresponding first information to the second electronic device through any means such as communication software or Bluetooth transmission.

It can be understood that in some embodiments, the way in which the second electronic device performs corresponding operations on the second electronic device based on the first information during the video playing process may be:

The second electronic device obtains the interface information and user operation information of the first interface in the received first information, enters the interface to be operated according to the interface information, and enters the interface to be operated according to the layout ID of the corresponding control in the user's operation information. Traverse the layout IDs of all controls to find the target control that is consistent with the layout ID of the operation target in the operation information of the first interface received, and perform corresponding operations on the target control according to the operation style in the operation information. After the operation on the first interface is completed, the interface information and user operation information of the next interface are obtained, and the above steps are performed to complete the corresponding operation on the next interface.

In summary, in the embodiments of the present application, the user can obtain the recorded video and the user's first information about the first electronic device during the recording process by performing screen recording on the first electronic device, and record the video and the user's first information about the first electronic device during the recording process. First information from an electronic device is sent to a second electronic device. In this way, during the process of playing the video, the second electronic device can perform the same operation as the operation performed by the user on the first electronic device in the recorded video, so that the user who receives the screen recording video and the first information does not need to do the operation himself. The electronic device can perform corresponding operations based on the received recorded video and first information, thereby improving operation convenience and user experience.

At the same time, since the user's first information includes detailed interface information and operation information corresponding to the interface, such as the layout ID of the target space, and auxiliary information, the second electronic device can accurately identify the interface to be operated based on the first information. and corresponding controls to be operated to improve the accuracy of operations.

It can be understood that in some embodiments, when the second electronic device is playing a video, the operation process performed by the second electronic device can be displayed in the form of a floating window to facilitate viewing by the user.

The solution in the embodiment of the present application is explained below by taking a scenario in which the user performs screen recording of the process of turning on the WIFI function through the first electronic device and sends the recorded video and operation information to the second electronic device as an example. The first electronic device may be a mobile phone 100, and the second electronic device may be a mobile phone 200.

For example, as shown in Figure 4a, user A clicks the operation recording control 4011 on the shortcut interface 401 on the mobile phone 100. The mobile phone 100 will record the change process of the display state of the screen caused by the user A's operation, and record the corresponding actions of the user's operation. first information, and the mobile phone 100 can display the recording box 400 as shown in Figure 4b during the entire recording process. For example, as shown in Figure 4b, the user records the WIFI connection operation on the mobile phone 100. The user first clicks the setting application 4021 on the main interface 402. The mobile phone 100 can record the interface information of the main interface 402 while recording the video, such as It includes the activity name "main interface" and can record the user's operation information on the main interface 402. For example, it can include: the user's operation style "click", the operation target "setting application", and the layout ID "1234" of the setting application 4021.

After the user clicks the setting control 4021, as shown in Figure 4c, the mobile phone 100 can display the setting interface 403. The user clicks the WIFI control 4031 on the setting interface 403. The mobile phone 100 can record the setting interface 403 while recording the video. Information, for example, includes the application name "Settings", the package name "com.setting" and the activity name "Settings Interface", as well as the user's operation information in the settings interface 403, for example, it may include: the user's operation style "click", operation target " WIFI Control", the layout ID of WIFI control 4031 is "12345".

After the user clicks the WIFI control 4031, as shown in Figure 4d, the mobile phone 100 can display the WIFI interface 404. The user clicks the switch control 4041 on the WIFI. The mobile phone 100 can record the interface information of the WIFI interface 404 while recording the video. For example, it includes the application name "Settings", the package name "com.setting" and the activity name "WIFI interface", as well as the user's operation information on the WIFI interface 404. For example, it can include: the user's operation style "click", the operation target "switch control" ", the layout ID of switch control 4041 is "12346". After the user clicks the switch control 4041, as shown in Figure 4e, the WIFI interface 404 of the mobile phone 100 can display a list of available WLANs. After the user finishes the operation, he can click the recording stop control 4001 in the recording box 400 to stop recording. At this time, the mobile phone 100 obtains the recorded video 500 of the entire operation process and the corresponding first information and stores it.

If user A wants to send the recorded video 500 to user B's mobile phone 200, the recorded video can be sent through communication software as shown in Figure 4f. Video 500 is sent to user B's mobile phone 200. It can be understood that while the recorded video 500 is being sent, the first information will also be sent to user B's mobile phone 200 at the same time. As shown in Figure 4g, user B can receive and display the recorded video 500 through communication software.

When user B clicks on the recorded video 500 for video playback, as shown in Figure 4h, while the recorded video 500 is playing, user B's electronic device can operate the mobile phone 200 based on the recorded information, and display the operation in the operation floating window 405 process. The specific operation process may include: first triggering the setting control on the main interface and displaying the setting interface, and then triggering the WIFI control on the setting interface and displaying the WIFI interface. And trigger the switch control on the WIFI interface to turn on the WIFI function.

Before introducing the operation method in the embodiment of the present application in detail below, the electronic device provided in the embodiment of the present application will be introduced in detail first. Wherein, the electronic device is a first electronic device or a second electronic device.

Software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The embodiment of the present invention takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.

Figure 5 is a software structure block diagram of the electronic device according to the embodiment of the present invention. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system libraries, and kernel layer.

The application layer can include a series of application packages. As shown in Figure 5, the application package can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 5, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

A window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.

It can be understood that in some embodiments, the window manager can respond to the user's playback operation of the recorded video and send a playback instruction to the corresponding application. For example, if the user clicks play in the photo album application, the window manager can send a playback instruction to the photo album. The application sends playback instructions. In addition, the window manager can obtain the first information in response to the user's playback operation of the recorded video, and control the corresponding application program to perform corresponding operations on the second electronic device based on the first information.

For example, the window manager can obtain the interface information and user operation information of the first interface in the received first information, determine the interface to be operated based on the interface information, and determine the interface to be operated based on the layout ID of the corresponding control in the user's operation information. The interface traverses the layout IDs of all controls to find the target control that is consistent with the layout ID of the operation target in the operation information received from the first interface, and controls the corresponding application to perform operations on the target control according to the operation style in the operation information. Act accordingly. After the operation on the first interface is completed, the interface information and user operation information of the next interface are obtained, and the above steps are performed to complete the corresponding operation on the next interface.

Content providers are used to store and retrieve data and make this data accessible to applications. Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

The view system includes visual controls, such as controls that display text, controls that display pictures, etc. A view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

Telephone managers are used to provide communication functions of electronic devices. For example, call status management (including connected, hung up, etc.).

The resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc.

The notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text information is prompted in the status bar, a beep sounds, the electronic device vibrates, the indicator light flashes, etc.

Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.

The core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.

The application layer and application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and application framework layer into binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

System libraries can include multiple functional modules. For example: surface manager (surface manager), media libraries (Media Libraries), 3D graphics processing libraries (for example: OpenGL ES), 2D graphics engines (for example: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

It can be understood that in the embodiment of the present application, in response to the user's recording operation, the media library can call a dedicated recording interface to perform screen recording, and store the obtained recorded video and the user's first information on the first electronic device during the recording process in Set a location, such as in the Photos app.

For example, in some embodiments, the media library may store the file containing the recorded video and the first information in the form of a compressed package in the photo album application of the electronic device.

In some embodiments, the media library may store the entire file corresponding to the recorded video and the first information in a multi-track format, such as an Exchangeable Image File Format (EXIF) or MOV format.

Wherein, when the video is recorded and the first information is saved in the photo album or other folder of the electronic device through EXIF format. Then the recorded video can be stored in the photo album or other folders of the first electronic device in the form of dynamic images through the video track, and the first information can be stored in the photo album or other folders of the first electronic device through the data track in the form of text. It is understandable that files in EXIF format do not support operations such as cropping and editing.

When the video is recorded and the first information is saved in the photo album or other folders of the electronic device via MOV format. The media library will record the video and store it in the video track according to the format of the video. The media library can output each interface information in the first information into a corresponding text file through the command line, and the operation information of each interface can also be output into the corresponding text file through the command line, and the interface information and the operation of each interface can be output The text files corresponding to the information can be stored in the form of text through data tracks.

It can be understood that in the embodiment of the present application, the video in MOV format supports arbitrary cropping and editing operations. Therefore, using the MOV format to store the recorded video and the first information can make the recorded video in an editable form, which facilitates post-processing.

In some embodiments, the media library may bundle the recorded video and the first information and save them in the photo album application of the electronic device in MP4 format.

And in some embodiments, the media library can receive recorded videos and first information sent by other electronic devices.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing.

2D Graphics Engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

Based on the above-mentioned electronic device, the operation method provided by the embodiment of the present application is introduced in detail below. Figure 6 shows the interaction process between the first electronic device and the second electronic device according to the embodiment of the present application. As shown in Figure 6, the operation method includes:

601: The first electronic device detects the recording trigger operation and starts screen recording.

It can be understood that, as shown in FIG. 4a , the recording triggering operation may be an operation in which the user clicks on the recording control 4011 on the mobile phone 100 .

It can be understood that after the first electronic device starts screen recording, during the recording process, a recording box can be displayed on the display interface to remind the user that the recording is in progress and to prompt the user for the recording time. In some embodiments, a recording stop control may also be present in the recording box.

For example, as shown in Figure 4b, a recording box 400 may be displayed on the display interface of the mobile phone 100, and the recording box 400 may include a recording time and a recording stop control 4001.

602: The first electronic device obtains the recorded video and the first information.

It can be understood that the recorded video may include a video of changes in the screen display caused by user operations during the recording process.

The first information may include interface information of each interface recorded during the screen recording process and user operation information on each interface.

For example, during the recording process, the user clicked the WIFI control on the setting interface of the first electronic device, and the first electronic device The device will record the interface information of the setting interface, such as the application name "Settings", the package name "com.setting" and the activity name "Settings Interface", and can record the user's operation information in the setting interface, for example, it can include: the user's The operation style is "click", the operation target is "WIFI control", and the layout ID of "WIFI control" is "12345".

For example, as shown in Figure 2b, during the recording process, the user slid the brightness adjustment control 1001 on the shortcut interface 101 of the mobile phone 100. In the embodiment of the present application, in addition to recording the user's operation style "sliding", the operation target "brightness adjustment control" ", in addition to the layout ID "12347" of the brightness adjustment control 1001, the brightness adjustment degree can also be recorded, that is, the adjustment position of the brightness bar, for example, the position of 30% of the overall brightness bar. In this way, when the second electronic device performs a brightness adjustment operation based on the operation information, the brightness can be accurately adjusted according to the adjustment position of the brightness bar.

It can be understood that the first electronic device can store the recorded video and the first information. There are various storage forms of the recorded video and the first information in the first electronic device.

In some embodiments, the recorded video and the first information may be bundled and saved in a compressed package file format in a photo album application or other folder of the first-end electronic device.

For example, in some embodiments, the compressed package file containing the recorded video and the first information may be displayed in the photo album application in the form of a video plus an operation mark. For example, the presentation form of the compressed package file can be a video style as shown in Figure 7a, and a border highlighting operation mark 501 is added. The presentation form of the compressed package file can also be a video style as shown in Figure 7b, and a corner mark is added. Mark operation mark 502, the presentation form of the compressed package file can also be a video style as shown in Figure 7c, and add a masking layer prompt operation mark 503.

In some embodiments, the recorded video and the first information may be saved in the photo album application of the first electronic device or in other folders in MP4 format.

In some embodiments, the entire file corresponding to the recorded video and the first information can also be stored in a multi-track format, such as an exchangeable image file format (EXIF) or MOV format.

As shown in Figure 7d, when the recorded video and the first information are saved in the photo album or other folder of the electronic device through the MOV format as shown in Figure 7d. The recorded video can be stored in the video format, such as MP4 format, through the video track. Each interface information in the first information can be output as a corresponding text file through the command line, and the operation information of each interface can also be output as a corresponding text file through the command line. The interface information and the operation information of each interface correspond to Text files can be stored as text via data tracks.

603: The first electronic device sends the recorded video and the corresponding first information to the second electronic device.

In some embodiments, the first electronic device can send the recorded video and the file corresponding to the corresponding first information, such as a compressed package file, to the second electronic device through the communication software through the network.

In some embodiments, the first electronic device may send the recorded video and the file corresponding to the corresponding first information to the second electronic device through Bluetooth transmission.

In some embodiments, the first electronic device can also upload files corresponding to the recorded video and the corresponding first information to the cloud space, and the second electronic device can download the files corresponding to the recorded video and the corresponding first information from the cloud space. .

In some embodiments, the first electronic device will share the recorded video and the file corresponding to the corresponding first information through a link, password, etc. Share to the second electronic device, and the second electronic device obtains the recorded video and the corresponding first information by opening the sharing link or entering a password.

It can be understood that the above-mentioned ways in which the first electronic device transmits the recorded video and the corresponding first information to the second electronic device are just examples. In the embodiment of the present application, the first electronic device transmits the recorded video and the corresponding first information to the second electronic device. The mode of the second electronic device may be any implementable mode.

For example, as shown in Figure 4f, if user A wants to send the recorded video 500 to user B's mobile phone 200, the recorded video 500 can be sent to user B's mobile phone through the network and communication software as shown in Figure 4f. 200 for mobile phone. It can be understood that while the recorded video is being sent, the first information will also be sent to user B's mobile phone 200 at the same time.

604: The second electronic device receives the recorded video and the corresponding first information.

It can be understood that while receiving the recorded video, the second electronic device will also receive the first information at the same time.

As shown in Figure 4g, user B can receive the recorded video 500 through communication software and display the recorded video in a video style.

It can be understood that after receiving the recorded video and the corresponding first information, the second electronic device can store the recorded video and the corresponding first information.

The recorded video and the first information may be stored in the second electronic device in a variety of storage forms. In some embodiments, the recorded video and the first information may be bundled and saved in a compressed package file format in a photo album application or other folder of the second electronic device.

For example, in some embodiments, the compressed package file containing the recorded video and the first information may be displayed in the photo album application in the form of a video plus an operation mark. For example, the presentation form of the compressed package file can be a video style as shown in the aforementioned Figure 7a, and a border highlighting operation mark 501 is added. The presentation form of the compressed package file can also be a video style as shown in Figure 7b, with the addition The corner mark operation mark 502, the presentation form of the compressed package file can also be a video style as shown in Figure 7c, and a masking layer prompt operation mark 503 is added.

In some embodiments, the recorded video and the first information may be bundled and saved in MP4 format in a photo album application of the second electronic device or in other folders.

605: The second electronic device responds to the user's playback operation and performs a corresponding operation based on the first information.

It can be understood that in the embodiment of the present application, the operation performed by the second electronic device based on the first information is consistent with the user's operation on the display interface in the recorded video.

It can be understood that in this embodiment of the present application, the way in which the second electronic device performs the corresponding operation based on the first information may be:

The second electronic device obtains the information of the first interface and user operation information in the received first information, enters the interface to be operated according to the interface information, and displays the interface to be operated according to the layout ID of the corresponding control in the user's operation information. The layout IDs of all controls are traversed to find the target control that is consistent with the layout ID of the operation target in the operation information of the first interface received, and corresponding operations are performed on the target control according to the operation style in the operation information. After the operation on the first interface is completed, the interface information and user operation information of the next interface are obtained, and the above steps are performed to complete the corresponding operation on the next interface.

For example, taking the second electronic device as the mobile phone 200 and the content in the recorded video as the operation process of turning on the WIFI function shown in Figure 4, it is explained that the second electronic device performs the second electronic device on the second electronic device based on the first information during the video playback process. How electronic equipment performs corresponding operations:

The mobile phone 200 first obtains the interface information 402 of the main interface in the first information, including, for example, the activity name "main interface", and Obtaining the user's operation information on the main interface 402 may include, for example: the user's operation style "click", the operation target "setting application", and the layout ID "1234" of the setting application 4021.

Then, according to the interface information of 402 of the main interface, the main interface of the mobile phone 200 is found as the interface to be operated. According to the layout ID "1234" corresponding to the setting application 4021 in the user's operation information, the layout IDs of all controls are traversed on the main interface of the mobile phone 200 to find the target control consistent with the layout ID "1234" of the setting application 4021. According to The operation style "click" in the first information performs a click operation on the target control. After the click operation on the target control is completed, the interface information and user operation information of the setting interface 403 are obtained, and the above steps are performed to complete the corresponding operation on the setting interface of the mobile phone 200 .

In some embodiments, as shown in Figure 8, in response to the user's playback operation, the second electronic device can play the recorded video in full screen or in a large window, and perform the operation in the background without displaying the operation process.

In some embodiments, in response to the user's play operation, the second electronic device can play the recorded video in full screen or in a large window, and display the operation process in the operation floating window.

For example, as shown in Figure 4h, when the user clicks on the recorded video 500 received by the mobile phone 200 for video playback, as shown in Figure 4h, the recorded video can be played in full screen on the main interface of the mobile phone 200, and in the operation floating window 405 Show the operation process.

In some embodiments, in response to the user's playback operation, the second electronic device can display the recorded video in a video operation floating window, and place the operation process in a large window for playback.

For example, as shown in Figure 9a, when the user clicks on the recorded video 500 received by the mobile phone 200 to play the video, the operation process can be displayed on the main interface of the mobile phone 200, and the recorded video can be played in the video floating window 406.

In some embodiments, as shown in Figure 9b, the second electronic device may include a first display area and a second display area. The first display area displays the recorded video, and the second display area displays the operation process, that is, the recorded video and the operation process. Split screen display.

In some embodiments, the second electronic device may have any number of other devices connected to each other. Then, the video can be played through any device among the multiple devices connected to each other, and any device among the other devices can control the second electronic device to perform the operation corresponding to the first information. Among them, the above-mentioned mutual connection method can be any method such as Bluetooth connection, mobile phone NFC function pairing connection, scan code connection, wired connection, etc.

For example, as shown in Figure 10, the third electronic device is a tablet 300, and the second electronic device is a mobile phone 200. Then the tablet 300 and the mobile phone 200 are paired and connected through the NFC function of the mobile phone, and the third electronic device can be used to play the recorded video 500, The second electronic device may be used to perform corresponding operations according to the first information.

For another example, as shown in Figure 11, the mobile phone 200 can be used to play the recorded video 500, and the tablet 300 can be used to control the second device to perform corresponding operations based on the first information, and display the operation process.

In some embodiments, as shown in Figure 12, the tablet 300 can display the virtual interface 302 corresponding to the mobile phone 200. The mobile phone 200 can display the operation process on the main interface of the mobile phone 200. The virtual interface 302 of the tablet 300 can be consistent with the mobile phone. Show the operation process. And the play window 301 of the tablet 300 can play the recorded video at the same time.

In some embodiments, when the user exits playback, if the second electronic device still has unfinished operations, the user may be reminded in the form of a pop-up window or other forms.

For example, as shown in FIG. 13, when the user exits playback, a prompt box 600 may be prompted that "the operation is not completed, do you want to continue?" When the user clicks the Yes control 601 in the prompt box 600, the video will continue to be played. When the user clicks the No control 602 in the prompt box 600, the video playback will be stopped.

It can be understood that in some embodiments, the recorded video can maintain any functions of conventional video such as fast forward, rewind, window size adjustment, and pause. And the recorded video can be played multiple times.

It can be understood that in some embodiments, the first electronic device can mark multiple time points in the recorded video. For example, in some embodiments, during the recording process, after all operations on each interface are completed, a time point can be marked before the next interface is displayed.

The following description takes the first electronic device as a mobile phone 100 and the mobile phone 100 recording the operation process of user A turning on the WIFI function as an example to describe the acquisition of the recorded information and the first information in the embodiment of the present application. As shown in Figure 4b, the user records the WIFI opening operation on the mobile phone 100. The user first clicks the settings application 4021 on the main interface 402. The mobile phone 100 records the interface information of the main interface 402 while recording the video, such as the activity name. "Main interface", and can record the user's operation information on the main interface 402, which may include, for example: the user's operation style "click", the operation target "setting application", and the layout ID "1234" of the setting application 4021. After the user clicks the setting control 4021, the mobile phone 100 can mark the first time point in the recorded video.

After the user clicks on the setting control 4021, as shown in Figure 4c, the mobile phone 100 can display the setting interface 403. The user clicks the WIFI control 401 on the setting interface 403. The mobile phone 100 can also record the interface information of the setting interface 403, for example, including The application name "Settings", the package name "com.setting" and the activity name "Settings Interface", as well as the user's operation information on the settings interface 403, may include, for example: the user's operation style "click", the operation target "WIFI control", After the user clicks the layout ID "12345" of the WIFI control 401, the mobile phone 100 can mark the second time point in the recorded video.

After the user clicks the WIFI control 401, as shown in Figure 4d, the mobile phone 100 can display the WIFI interface 404. The user clicks the switch control 4041 on the WIFI at this time, and the mobile phone 100 can simultaneously record the interface information of the WIFI interface 404, including the application name. "Settings", the package name "com.setting" and the activity name "WIFI interface", as well as the user's operation information on the WIFI interface 404, for example, may include: the user's operation style "click", the operation target "switch control", switch control 4041 has layout ID "12346". After the user clicks the switch control 4041, as shown in Figure 4e, the WIFI interface 404 of the mobile phone 100 can display a list of available WLANs. After the user finishes the operation, he can click the recording stop control 4001 in the recording box 400 to stop recording. At this time, the mobile phone 100 marks the third time point, which is the end time point. The mobile phone 100 obtains and stores the recorded video of the entire operation process and the corresponding first information.

For example, the obtained recorded video 500 may include three time points as shown in FIG. 14 , where each time point may correspond to a subtask in the WIFI activation function task.

It can be understood that in some embodiments, several associated time points among multiple time points can be fixed to each other. It can be understood that the associated time points can be time points during the operation of performing any function, such as performing the above Multiple time points during the operation of turning on the WIFI function. This prevents the first electronic device or the second electronic device from dragging the progress bar between associated time points back and forth when playing a video, thereby preventing the user from dragging the video when the video includes the execution of multiple functions. The progress bar is followed up until the execution of a certain function, causing the problem that the electronic device performs unnecessary operations or performs less operations.

For example, in some scenarios, in addition to recording the WIFI process, users also record the transfer process. For example, as shown in Figure 15, the first time point in the recorded video 500 is the time point after the user clicks on the WIFI control 401 in the scene shown in Figure 4, and the second time point is after the user clicks on the WIFI control 401. The third time point is the time point after the user clicks the switch control 4041. The operation between the third time point and the fourth time point is the operation of returning to the main interface. The fifth time point, The sixth time point and the seventh time point are respectively the time points in the recording process when the user performs the transfer function. At this time, the first electronic device can transfer the first time point, the second time point and the third time point. Click to fix the association. The fifth time point, the sixth time point and the seventh time point are also related and fixed. That is, when playing a video, the first electronic device or the second electronic device cannot drag the progress bar back and forth before the third time point between the starting times.

It can be understood that the content between the fixed time point and the previous time point is the operation content corresponding to the fixed time point. Therefore, when any time point is fixed, the content between the fixed time point and the previous time point will The content is also fixed. Because for the recorded video shown in Figure 15, when the second electronic device or the first electronic device plays the recorded video 500, the user cannot drag the progress bar between the starting time and the third time point. Nor can you drag the progress bar between the fourth time point and the seventh time point back and forth. That is, if the user wants to drag the progress bar, he can only drag it directly from the initial time point to the third time point, and directly play the video after the third time point. He cannot drag directly from the initial time point to the first time point. Or the second point in time. In this way, it can be effectively avoided that if the user wants to perform the operation of turning on the WIFI function, the electronic device will perform less functions. If the user wants to perform the operation of the transfer function, the electronic device will perform more operations of turning on the WIFI function.

In some embodiments, the second electronic device can perform corresponding operations on the second electronic device based on the first information while starting to play the video. In the process of playing the video, when the second electronic device reaches any point in time, if the operation before the point in time is not completed, it can automatically pause the playback to wait for the operation before the point in time to be completed before playing.

For example, while the second electronic device is playing the recorded video 500, if the recorded video 500 has been played to the second time point, but the second electronic device has not completed the operation of clicking the WIFI control on the setting interface, the second electronic device Video playback will stop and the video will be played again after the second electronic device completes the operation of clicking the WIFI control on the setting interface.

In some embodiments, the second electronic device may not trigger the execution of the operation when starting to play the video. Instead, when the video is played to each time point, the second electronic device may trigger the execution of the operation task before that time point, and each operation task may be added. into the task queue, and the second device sequentially executes the above operation tasks in the order in which the operation tasks are added. That is, the second electronic device triggers the execution of the operation when the playback reaches the first time point.

For example, as shown in Figure 16, when the received video is the operation process of turning on the WIFI function shown in Figure 4, when the second electronic device plays to the first time point, the first operation task is triggered, that is, the first operation task is triggered. Operations performed before a certain point in time, such as clicking settings on the main interface Set the operation of the application and add the first operation task to the task queue; when the second electronic device plays to the second point in time, the second task is triggered, that is, the operation before the second point in time, for example, to implement the Click the operation of the WIFI control on the setting interface, and add the second operation task to the task queue.

It can be understood that in the implementation shown in Figure 16, it is allowed that the recorded video has been played to the second point in time, but the operation task corresponding to the first point in time has not been completed. At this time, the video can continue to play and continue in the background. Execute each operation task in sequence according to the above task queue.

In some embodiments, both the first electronic device and the second electronic device can edit the recorded video, such as cropping by mistake, splitting the operation to obtain multiple video files, etc. In this way, incorrect operations or redundant operations during the recording process can be effectively avoided. It can be understood that during the editing process of the recorded video by the electronic device, the first information will also be adjusted accordingly.

For example, in the video recording shown in Figure 17a, after the electronic device performed the operation of clicking the WIFI control on the setting interface, it marked the second time point, and then accidentally pressed the volume button, the electronic device marked the third time point, so The operation corresponding to the third time point is a redundant operation. At this time, the user can crop out the video content between the second time point and the third time point as shown in Figure 17b. At the same time, the electronic device will automatically delete the second time point. and the first information corresponding to the video between the third time point.

The following describes the hardware structure of the electronic device provided by the embodiment of the present application, taking the mobile phone 10 as an example.

As shown in Figure 18, the mobile phone 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, a button 101 and a display. Screen 102 etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 10 . In other embodiments of the present application, the mobile phone 10 may include more or less components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, may include a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a digital signal processor DSP, a microprocessor (Micro- Processing modules or processing circuits such as programmed Control Unit (MCU), artificial intelligence (Artificial Intelligence, AI) processor or programmable logic device (Field Programmable Gate Array, FPGA). Among them, different processing units can be independent devices or integrated in one or more processors. A storage unit may be provided in the processor 110 for storing instructions and data. In some embodiments, the storage unit in the processor 110 is a cache memory 180, and the cache memory 180 can save instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can be called directly from the memory.

It can be understood that the operation methods in the embodiments of the present application can be executed by the processor 110 of the corresponding electronic device.

Power module 140 may include a power supply, power management components, and the like. The power source can be a battery. The power management component is used to manage the charging of the power supply and the power supply from the power supply to other modules. In some embodiments, the power management component includes a charge management module and a power management module. The charging management module is used to receive charging input from the charger; the power management module is used to connect the power supply, the charging management module and the processor 110 . The power management module receives input from the power supply and/or charging management module and supplies power to the processor 110, the display screen 102, the camera 170, the wireless communication module 120, etc.

The mobile communication module 130 may include, but is not limited to, an antenna, a power amplifier, a filter, an LNA (Low noise amplifier), etc. The mobile communication module 130 can provide wireless communication solutions including 2G/3G/4G/5G applied on the mobile phone 10 . The mobile communication module 130 can receive electromagnetic waves through an antenna, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 130 may be disposed in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 130 may be provided in the same device as at least part of the modules of the processor 110 .

A modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 102 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processing The processor 110 is provided in the same device as the mobile communication module 130 or other functional modules.

The wireless communication module 120 may include an antenna, and implements the transmission and reception of electromagnetic waves via the antenna. The wireless communication module 120 can provide applications on the mobile phone 10 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (bluetooth, BT), and global navigation satellite systems. (global navigation satellite system, GNSS), frequency modulation (FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The mobile phone 10 can communicate with the network and other devices through wireless communication technology.

In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the mobile phone 10 may also be located in the same module.

The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. Display 102 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode). emitting diode (AMOLED), flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diode (QLED), etc.

The sensor module 190 may include a proximity light sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

The audio module 150 is used to convert digital audio information into an analog audio signal output, or convert an analog audio input into a digital audio signal. Audio module 150 may also be used to encode and decode audio signals. In some embodiments, the audio module 150 may be disposed in the processor 110 , or some functional modules of the audio module 150 may be disposed in the processor 110 . In some embodiments, audio module 150 may include speakers, earpieces, microphones, and headphone jacks.

Speakers, also called "horns," are used to convert audio electrical signals into sound signals.

A handset, also called a "receiver," is used to convert audio electrical signals into sound signals.

Microphone, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak through the mouth close to the microphone and input the sound signal into the microphone. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the mobile phone 10 can also be equipped with three, four or more microphones to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions, etc.

The headphone jack is used to connect wired headphones. The headphone interface can be a USB interface, or it can be a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

Camera 170 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to ISP (Image Signal Processing) to convert it into a digital image signal. The mobile phone 10 can realize the shooting function through the ISP, camera 170, video codec, GPU (Graphic Processing Unit, graphics processor), display screen 102 and application processor.

Specifically, the ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: MOVing picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

The interface module 160 includes an external memory interface, a universal serial bus (universal serial bus, USB) interface, a subscriber identification module (subscriber identification module, SIM) card interface, and so on. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 10. The external memory card communicates with the processor 110 through the external memory interface to implement data storage functions. The Universal Serial Bus interface is used by mobile phones 10 and other electronic devices to communicate. The user identity module card interface is used to communicate with the SIM card installed in the mobile phone 1010, such as reading the phone number stored in the SIM card, or writing the phone number into the SIM card.

In some embodiments, the mobile phone 10 also includes buttons 101, motors, indicators, etc. Among them, the keys 101 may include volume keys, on/off keys, etc. The motor can produce vibration cues. The motor can be used to vibrate for incoming calls or for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. The motor can also respond to different vibration feedback effects for touch operations in different areas of the display screen 102 . Different application scenarios (such as time reminders, receiving information, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized. Indicators may include laser pointers, radio frequency indicators, LED indicators, etc.

Embodiments of the present application provide an electronic device, including: a memory, used to store instructions executed by one or more processors of the electronic device; and a processor, which is one of the one or more processors of the electronic device, used to execute The above operation method.

Embodiments of the present application provide a readable storage medium, in which instructions are stored on the readable storage medium. When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the above operation method.

Embodiments of the present application provide a computer program product, including: execution instructions. The execution instructions are stored in a readable storage medium. At least one processor of an electronic device can read the execution instructions from the readable storage medium. The at least one processor executes the execution instructions. The instructions enable the electronic device to implement the above operation method.

Various embodiments disclosed in this application may be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of the present application may be implemented as a computer program or program code executing on a programmable system including at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements) , at least one input device and at least one output device.

Program code may be applied to input instructions to perform the functions described herein and to generate output information. Output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

Program code may be implemented in a high-level procedural language or an object-oriented programming language to communicate with the processing system. When necessary, assembly language or machine language can also be used to implement program code. In fact, the mechanisms described in this application are not limited to the scope of any particular programming language. In either case, the language may be a compiled or interpreted language.

In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried on or stored on one or more transitory or non-transitory machine-readable (e.g., computer-readable) storage media, which may be operated by one or more processors Read and execute. For example, instructions may be distributed over a network or through other computer-readable media. Thus, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magnetic Optical disk, read-only memory (ROM), random-access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical card, flash memory, or Tangible machine-readable storage used to transmit information (e.g., carrier waves, infrared signals, digital signals, etc.) using electrical, optical, acoustic, or other forms of propagated signals over the Internet. Thus, machine-readable media includes any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, computer).

In the drawings, some structural or methodological features may be shown in specific arrangements and/or orders. However, it should be understood that such specific arrangement and/or ordering may not be required. Rather, in some embodiments, the features may be arranged in a manner and/or order different than that shown in the illustrative figures. Additionally, the inclusion of structural or methodological features in a particular figure is not meant to imply that such features are required in all embodiments, and in some embodiments these features may not be included or may be combined with other features.

It should be noted that each unit/module mentioned in each device embodiment of this application is a logical unit/module. Physically, a logical unit/module can be a physical unit/module, or it can be a physical unit/module. Part of the module can also be implemented as a combination of multiple physical units/modules. The physical implementation of these logical units/modules is not the most important. The combination of functions implemented by these logical units/modules is what solves the problem of this application. Key technical issues raised. In addition, in order to highlight the innovative part of this application, the above-mentioned equipment embodiments of this application do not introduce units/modules that are not closely related to solving the technical problems raised by this application. This does not mean that the above-mentioned equipment embodiments do not exist. Other units/modules.

It should be noted that in the examples and descriptions of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply There is no such actual relationship or sequence between these entities or operations. Furthermore, the terms "comprises,""comprises," or any other variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, by The statement "comprises a" a qualified element does not exclude the presence of additional identical elements in a process, method, article, or device that includes the stated element.

Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes may be made in form and detail without departing from the present invention. Scope of application.

Claims

An operating method, applied to a first electronic device, characterized by including:

Obtain a first video and first information associated with the first video, wherein the first information includes interface information of at least one display interface recorded during the screen recording process and the target of the user's operation on the at least one display interface. The identification information of the control;

The first video is played, and during the playing of the first video, an operation is performed based on the first information.
The method of claim 1, wherein the operation is consistent with the user's operation on the at least one display interface in the first video.
The method of claim 1, wherein the first video and the first information are stored as the same file, or the first video and the first information are stored as different files.
The method according to any one of claims 1 to 3, characterized in that the first information also includes a user's operation mode of the target control on the at least one display interface.
The method of claim 4, wherein the performing an operation based on the first information includes:

Obtain the interface information of the first display interface in the first information, and determine the first to-be-operated interface of the first electronic device based on the interface information of the first display interface;

Obtain the identification information of the target control operated by the user on the first display interface, and determine the control to be operated on the first interface to be operated based on the identification information of the target control operated by the user on the first display interface;

Obtain the user's operation mode on the target control on the first display interface, and perform corresponding operations on the to-be-operated control based on the user's operation mode on the target control on the first display interface;

Obtain the interface information of the second display interface in the first information, and determine the second to-be-operated interface of the first electronic device based on the interface information of the second display interface.
The method of claim 5, wherein when the operation mode of the target control is a multi-state operation mode, the first information further includes auxiliary information corresponding to the operation mode.
The method according to claim 6, wherein the multi-state operation mode includes a typing operation and an adjustment operation, the auxiliary information corresponding to the typing operation includes input content, and the auxiliary information corresponding to the adjustment operation includes an adjustment state.
The method according to claim 1, characterized in that the interface information includes interface package name, interface application name and interface activity name.
The method according to any one of claims 1-3, 6-8, wherein the first electronic device includes a first display area and a second display area;

When the first electronic device plays the first video, the first display area displays the playback screen of the first video, and the second display area displays the first electronic device performing the operation. Operation process.
The method of claim 9, wherein the first display area overlaps a partial area of the second display area, or the second display area overlaps a partial area of the first display area. .
The method according to claim 9, characterized in that the second display area and the first display area are displayed in separate screens.
The method according to any one of claims 1-3, 6-8, 10-11, characterized in that a third electronic device is connected to the first electronic device, and the third electronic device displays the first In the playback screen of the video, the first electronic device displays the operation process of the first electronic device performing the operation.
The method according to any one of claims 1-3, 6-8, 10-11, characterized in that a third electronic device is connected to the first electronic device, and the third electronic device displays the first The electronic device performs the operation process of the operation, and the first electronic device displays the playback screen of the first video.
The method according to any one of claims 1-3, 6-8, 10-11, characterized in that a third electronic device is connected to the first electronic device, and the first electronic device displays the first The electronic device performs the operation process of the operation and/or the playback screen of the first video;

The third electronic device includes a third display area and a fourth display area, the third display area displays the display content of the first electronic device, and the fourth display area displays the operation process or the first The video playback screen.
The method according to any one of claims 1-3, 6-8, 10-11, further comprising: sending prompt information when receiving an exit playback instruction and detecting that the operation is not completed. to the display interface of the first electronic device.
The method according to any one of claims 1-3, 6-8, 10-11, characterized in that the first video is marked with multiple time points, and each of the time points corresponds to the screen recording. Part of the user's operations during the process.
The method according to claim 16, further comprising: fixing associated partial time points among the plurality of time points, so that videos corresponding to the associated partial time points are played continuously.
The method according to claim 17, characterized in that when the first video is played to a target time point and the operation corresponding to the target time point is not completed, the playback of the first video is stopped. ;

After the operation corresponding to the target time point is completed, the first video continues to be played.
The method according to any one of claims 1-3, 6-8, 10-11, 17-18, further comprising: editing the first video to obtain a second video.
The method according to claim 19, characterized in that the editing processing includes cutting processing and/or splitting processing and/or reorganization processing.
The method according to any one of claims 1-3, 6-8, 10-11, 17-18, wherein the first electronic device stores the first electronic device in MOV format, EXIF format or compressed package format. a video and said first message.
The method according to claim 1, characterized in that, during the process of playing the first video, performing an operation based on the first information includes:

Detect the play instruction, play the first video, and when the first video is played to the first time point, add the operation corresponding to the first time point to the task queue, and based on the first information Perform operations in the task queue;

When the first video is played to a second time point, the operation corresponding to the second time point is added to the task queue.
An operating method, applied to a second electronic device, characterized by including:

Screen recording command detected;

Turn on screen recording and generate a first video and first information, where the first information includes interface information of at least one display interface recorded during the screen recording process and identification information of target controls operated by the user on the at least one display interface, Wherein, the first information can be used for the electronic device to perform corresponding operations based on the first information during the process of playing the first video.
The method of claim 23, wherein the second electronic device sends the first video and the first information to the first electronic device.
An electronic device, characterized in that it includes: a memory for storing instructions executed by one or more processors of the electronic device, and the processor is one of the one or more processors of the electronic device. 1. Used to perform the operating method described in any one of claims 1 to 24.
A readable storage medium, characterized in that instructions are stored on the readable medium, and when the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the operation method described in any one of claims 1-24. .
A computer program product comprising: execution instructions stored in a readable storage medium from which at least one processor of an electronic device can read the execution instructions, the at least one process The processor executes the execution instructions so that the electronic device implements the operating method described in any one of claims 1-24.