WO2020244500A1

WO2020244500A1 - Method for touch control in screen casting scenario, and electronic apparatus

Info

Publication number: WO2020244500A1
Application number: PCT/CN2020/093908
Authority: WO
Inventors: 魏曦; 曹原; 范振华
Original assignee: 华为技术有限公司
Priority date: 2019-06-05
Filing date: 2020-06-02
Publication date: 2020-12-10
Also published as: CN110377250A; CN110377250B

Abstract

The present invention relates to the technical field of terminals. Provided are a method for touch control in a screen casting scenario, and an electronic apparatus allowing a destination apparatus to receive and respond to a control operation performed by a user with respect to a screen casting interface, thereby improving the touch control experience of a user in a screen casting scenario. The method comprises: a source apparatus displaying a first display interface; in response to a screen casting instruction input by a user, the source apparatus projecting N controls on the first display interface to a screen casting interface displayed by a first destination apparatus, where N is an integer greater than 0; the source apparatus receiving a first touch event sent by the first destination apparatus; and the source apparatus executing an operation instruction corresponding to the first touch event.

Description

Touch control method and electronic equipment in screen projection scene

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China with application number 201910487623.9 on June 5, 2019, and a Chinese patent with the title of “a touch method and electronic device in a projection screen scenario” The priority of the application, the entire content of which is incorporated in this application by reference.

Technical field

This application relates to the field of terminal technology, and in particular to a touch method and electronic equipment in a screen projection scenario.

Background technique

With the development of smart home technology, a user or family often has multiple electronic devices that can communicate with each other. Various electronic devices generally have their own device characteristics. For example, mobile phones have better portability, TV screens have better display effects, and speakers have better sound quality. In order to give full play to the device characteristics of different electronic devices, the electronic device can realize the switching and display of multimedia data among multiple devices by means of screen projection.

Exemplarily, when a user uses a video application in a mobile phone to watch a video, the mobile phone can be set as the source device, and the display interface in the source device can be sent to other destination devices that support the screen projection function for display. When the user needs to operate the current display interface of the video application, he still needs to perform corresponding operations in the mobile phone (ie, the source device) to update the display data of the mobile phone, and then the mobile phone projects the updated display data to the destination device for display.

Then, when the source device is not around the user or the user is inconvenient to operate the source device, the user cannot perform related control on the display interface that is being projected, resulting in a poor user experience when the projected screen is displayed.

Summary of the invention

The present application provides a touch method and electronic device in a screen projection scene. The target device can receive and respond to the control operation performed by the user on the screen projection interface, thereby improving the user's touch experience in the screen projection scene.

In order to achieve the above objectives, this application adopts the following technical solutions:

In the first aspect, this application provides a touch method in a screen projection scenario, including: a source device displays a first display interface; in response to a screen projection instruction input by a user, the source device sets N in the first display interface (N is (Integer greater than 0) controls are projected to the projection interface displayed by the first destination device; subsequently, if the source device receives the first touch event sent by the first destination device, the source device can execute the corresponding Operating instructions. In this way, in the screen projection scenario, the destination device can generate a touch event in response to a touch operation input by the user, and send the touch event to the source device to implement corresponding functions, and realize the control function of reversely controlling the source device, thereby improving the user's casting Touch experience in the screen scene.

Exemplarily, the aforementioned first touch event may include the coordinates of the touch point and the type of the touch event (for example, an event type such as a single click, a double tap, or a sliding).

In a possible implementation manner, after the source device receives the first touch event sent by the first destination device, the method further includes: the source device determines a target control corresponding to the first touch event, and the target control is among the aforementioned N controls At this time, the operation instruction executed by the source device is the corresponding operation instruction when the target control is triggered on the source device. For example, when the target control corresponding to the first touch event is a play button, the operation instruction corresponding to the first touch event is an operation instruction when the play button is triggered.

Exemplarily, the aforementioned first touch event may be: a touch event generated by the first destination device when the user inputs a first touch operation on the aforementioned projection interface. Wherein, the first touch operation is a touch operation actually input by the user when the screen is projected.

For another example, the above-mentioned first touch event may also be: after the first destination device generates a touch event (such as a fifth touch event) in response to a touch operation input by the user on the projection interface, the first destination device sends the fifth touch event The mapping is a touch event in the first display interface.

In a possible implementation manner, the source device may store a configuration file corresponding to the above-mentioned first display interface. The configuration file records the display positions of the above-mentioned N controls in the first display interface, and where the N controls are The display position in the projection interface; at this time, the source device determines the target control corresponding to the first touch event, which specifically includes: the source device can determine the display position of the above N controls in the projection interface recorded in the configuration file The target control corresponding to the first touch event.

For example, when the coordinates of the touch point in the first touch event (ie, the first coordinates) fall within the display position of the first control in the configuration file, the source device may determine the first control as the target control.

For another example, when the coordinates of the touch point in the first touch event (that is, the first coordinates) both fall within the display position of the first control in the above configuration file, and at the same time fall within the display position of the second control in the configuration file , The source device can determine the first control at the top level as the target control.

In a possible implementation manner, after the source device determines the target control corresponding to the first touch event, the method further includes: the source device maps the first touch event to the second touch event according to the above configuration file, and the second touch The event is: a touch event that the source device will generate when the user inputs a second touch operation to the target control in the first display interface; it should be noted that the user does not actually input the second touch operation in the first display interface. The source device maps the second touch event corresponding to the second touch operation according to the first touch event.

At this time, the execution of the operation instruction corresponding to the first touch event by the source device means: the source device executes and reports the mapped second touch event to the first application (the first display interface being displayed by the source device is the interface of the first application) ), so that the first application executes the operation instruction corresponding to the second touch event. For example, the coordinate of the touch point in the first touch event is A, and the coordinate of the touch point in the second touch event after mapping is B. Then, the source device executes the operation instruction corresponding to the first touch event, which is actually the first application responding to the coordinate This is the second touch event of B. In this way, the user's touch operation on the projection interface can also reversely control the related application in the source device to implement the corresponding function.

In a possible implementation manner, the source device maps the first touch event to the second touch event according to the above configuration file, including: the source device displays the first display on the first display interface according to the target control recorded in the configuration file The corresponding relationship between the position and the second display position in the projection interface is converted from the first coordinate in the first touch event to the second coordinate to obtain the second touch event. For example, the source device can reversely calculate the second coordinate corresponding to the above-mentioned first coordinate in the first display interface according to changes in the translation, scaling, or rotation of the target control before and after the screen projection recorded in the configuration file.

In a possible implementation, after the source device determines the target control corresponding to the first touch event, it can also report the identification of the target control and the event type of the first touch event to the first application, so that the first The application executes the first function, that is, the function corresponding to the first application when the target control is triggered by the operation indicated by the event type, so as to realize the function of the user to reversely control the related application in the source device in the projection interface.

In a possible implementation manner, after the source device determines the target control corresponding to the first touch event, the method further includes: the source device generates the target control at the first display position in the first display interface according to the target control recorded in the configuration file. The third touch event, the event type of the third touch event is the same as the event type of the first touch event, and the third coordinate in the third touch event is located in the first display position; at this time, the source device executes corresponding to the first touch event The operation instructions include: the source device reports the third touch event to the first application. That is to say, the source device converts the user’s first touch event on the screen projection interface into a third touch event on the first display interface. Then, the process of the source device responding to the third touch event is actually the source device responding to the third touch event. The process of the first touch event.

It should be noted that in the foregoing implementation manner, the source device determines the target control of the first touch event, and maps the first touch event to the second touch event. It is understandable that after the target device generates the first touch event, it can also determine the target control of the first touch event according to the above method, and map the first touch event to the second touch event. Furthermore, the destination device can send the mapped second touch event to the source device, and the source device reports the second touch event to the first application, so that the first application executes the operation instruction corresponding to the second touch event, that is, The operation instruction corresponding to the first touch event.

In a possible implementation manner, after the source device displays the first display interface, it further includes: in response to the second screen-casting instruction input by the user, the source device sets M in the first display interface (M is an integer greater than 0). ) A control is projected to the second destination device for display; the source device receives the fourth touch event sent by the second destination device; the source device executes an operation instruction corresponding to the fourth touch event. In other words, when the source device simultaneously projects the display content in its display interface to multiple destination devices for display, the user can input a touch operation in any destination device to reversely control the source device to realize the control function corresponding to the touch operation. So as to improve the user's touch experience in the projection scene.

In a second aspect, this application provides a touch method in a projection scene, including: a destination device receives a first message sent by a first source device, the first message includes a drawing instruction of a first target control, and the first target control One or more controls in the first display interface displayed for the first source device; the destination device invokes the drawing instruction of the first target control to draw the screen projection interface, the screen projection interface includes the first target control; in response to the user casting For the first touch operation input to the first target control on the screen interface, the destination device generates a first touch event; the destination device instructs the first source device to execute an operation instruction corresponding to the first touch event.

In a possible implementation manner, the destination device instructs the first source device to execute the operation instruction corresponding to the first touch event, including: the destination device sends the first touch event to the first source device, so that the first source device executes The operation instruction corresponding to the first touch event. After receiving the first touch event, the first source device can execute the operation instruction corresponding to the first touch event according to the method in the first aspect.

In a possible implementation manner, after the target device generates the first touch event, it further includes: the target device maps the first touch event to a second touch event, and the second touch event is: The touch event that the first source device will generate when the first target control inputs the second touch operation. The method for the destination device to map the first touch event to the second touch event is the same as the method for the source device to map the first touch event to the second touch event in the first aspect.

At this time, the destination device instructs the first source device to execute the operation instruction corresponding to the first touch event, including: the destination device sends the mapped second touch event to the first source device, so that the first source device executes the operation instruction corresponding to the second touch event. The operation instruction corresponding to the touch event.

In a possible implementation, the above method further includes: the destination device receives a second message sent by the second source device, the second message includes a drawing instruction of the second target control, and the second target control is displayed by the second source device One or more controls in the second display interface; the destination device calls the drawing instruction of the second target control to draw the second target control in the projection interface; in response to the user input to the second target control in the projection interface In the third touch operation, the destination device generates a third touch event; the destination device instructs the second source device to execute an operation instruction corresponding to the third touch event. In other words, when multiple source devices simultaneously project the display content in their display interface to the same target device for display, the user can input touch operations on the controls projected from different source devices in the target device to control the corresponding The source device implements the function corresponding to the touch operation, thereby improving the user's touch experience in the screen projection scene.

In a third aspect, the present application provides an electronic device, including: a touch screen, one or more processors, one or more memories, and one or more computer programs; wherein the processor is coupled with the touch screen and the memory, and the above one One or more computer programs are stored in the memory, and when the electronic device is running, the processor executes one or more computer programs stored in the memory, so that the electronic device executes any one of the contact recommendation methods described above.

In a fourth aspect, the present application provides a computer storage medium including computer instructions, which when the computer instructions run on an electronic device, cause the electronic device to execute the contact recommendation method described in any one of the first aspect.

In a fifth aspect, this application provides a computer program product, which when the computer program product runs on an electronic device, causes the electronic device to execute the contact recommendation method described in any one of the first aspect.

In a sixth aspect, the present application provides a touch control system, which may include at least one source device and at least one destination device; the source device may be used to perform touch control in the projection scene as described in any one of the first aspect. Control method, the target device can be used to execute the touch method in the projection scene as described in any one of the second aspect.

Understandably, the electronic equipment described in the third aspect, the computer storage medium described in the fourth aspect, the computer program product described in the fifth aspect, and the touch control system described in the sixth aspect provided above are all used for execution. For the corresponding method provided in the text, therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above, which will not be repeated here.

Description of the drawings

FIG. 1 is an architecture diagram 1 of a communication system provided by an embodiment of this application;

FIG. 2 is a second structural diagram of a communication system provided by an embodiment of this application;

FIG. 3 is a first structural diagram of an electronic device according to an embodiment of the application;

4 is a structural diagram of an operating system in an electronic device provided by an embodiment of the application;

FIG. 5 is a first schematic diagram of a touch method in a projection scene provided by an embodiment of the application;

6 is a schematic diagram of a second scene of a touch method in a projection scene provided by an embodiment of the application;

FIG. 7 is a scene schematic diagram 3 of a touch method in a projection scene provided by an embodiment of the application;

FIG. 8 is a scene schematic diagram 4 of a touch method in a projection scene provided by an embodiment of the application;

9 is a schematic diagram five of a touch method in a projection scene provided by an embodiment of the application;

FIG. 10 is a scene schematic diagram 6 of a touch method in a projection scene provided by an embodiment of the application;

FIG. 11 is a scene schematic diagram 7 of a touch method in a projection scene provided by an embodiment of this application;

FIG. 12 is a scene schematic diagram eight of a touch method in a projection scene provided by an embodiment of this application;

FIG. 13 is a scene schematic diagram 9 of a touch method in a projection scene provided by an embodiment of the application;

14 is a schematic diagram ten of a touch method in a projection scene provided by an embodiment of the application;

15 is a schematic eleventh scene of a touch method in a projection scene provided by an embodiment of this application;

16 is a schematic diagram 12 of a touch method in a projection scene provided by an embodiment of this application;

FIG. 17 is a schematic diagram 13 of a touch method in a projection scene provided by an embodiment of the application;

FIG. 18 is a second structural diagram of an electronic device provided by an embodiment of this application.

Detailed ways

The implementation of this embodiment will be described in detail below with reference to the accompanying drawings.

As shown in FIG. 1, a touch method in a projection scene provided by an embodiment of the present application can be applied to a communication system 100, and the communication system 100 may include N (N>1) electronic devices. For example, the communication system 100 may include an electronic device 101 and an electronic device 102.

Exemplarily, the electronic device 101 may be connected to the electronic device 102 through one or more communication networks 104.

The communication network 104 may be a wired network or a wireless network). For example, the aforementioned communication network 104 may be a local area network (local area networks, LAN), or a wide area network (wide area networks, WAN), such as the Internet. The communication network 104 can be implemented using any known network communication protocol. The above-mentioned network communication protocol can be various wired or wireless communication protocols, such as Ethernet, universal serial bus (USB), and Firewire (FIREWIRE). , Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Broadband Code Division Multiple Access (wideband) code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth, wireless fidelity (Wi-Fi) , NFC, voice over Internet protocol (VoIP) based on Internet protocol, communication protocol supporting network slicing architecture or any other suitable communication protocol. Exemplarily, in some embodiments, the electronic device 101 may establish a Wi-Fi connection with the electronic device 102 through a Wi-Fi protocol.

Exemplarily, the electronic device 101 may be used as a source device, the electronic device 102 may be used as a destination device, and the electronic device 101 (ie, the source device) may project the display content in its display interface to the electronic device 102 (ie, the destination device) for display. Of course, the electronic device 102 can also be used as a source device, and the electronic device 102 projects the display content in its display interface to the electronic device 101 (that is, the destination device) for display.

As still shown in FIG. 1, the communication system 100 described above may further include one or more other electronic devices such as an electronic device 103. For example, the electronic device 103 may be a wearable device. Exemplarily, the electronic device 103 may also be used as a source device or a destination device for projection display.

Exemplarily, as shown in (a) of FIG. 2, taking the electronic device 101 as an example of the source device, both the electronic device 102 and the electronic device 103 can be used as the destination device of the electronic device 101. The electronic device 101 can project the display content in its display interface to the electronic device 102 and the electronic device 103 for display at the same time. In other words, one source device can simultaneously project to multiple destination devices.

Or, as shown in (b) in FIG. 2, taking the electronic device 101 as an example of the destination device, both the electronic device 102 and the electronic device 103 can be used as the source device of the electronic device 101. At this time, the electronic device 102 and the electronic device 103 can simultaneously project the display content in their display interfaces to the electronic device 101 for display. In other words, a destination device can simultaneously receive and display the display content sent by multiple source devices.

It should be noted that when the source device projects the display content to the destination device, all the controls in its display interface can be projected to the destination device for display, or some controls in its display interface can be projected to the destination device for display. The application embodiment does not impose any restriction on this. In addition, any electronic device in the aforementioned communication system 100 can be used as a source device or a destination device, which is not limited in the embodiment of the present application.

In some embodiments, the specific structures of the electronic device 101, the electronic device 102, and the electronic device 103 may be the same or different.

For example, each of the above electronic devices may specifically be mobile phones, tablet computers, smart TVs, wearable electronic devices, car machines, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks, and personal digital assistants. (personal digital assistant, PDA), virtual reality equipment, etc. The embodiments of the present application do not make any restrictions on this.

Taking the electronic device 101 as an example, FIG. 3 shows a schematic structural diagram of the electronic device 101.

The electronic device 101 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , Mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, camera 193, display screen 194, etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc. Among them, the different processing units may be independent devices or integrated in one or more processors.

A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

In some embodiments, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (PCM) interface, and a universal asynchronous transmitter receiver/transmitter, UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and / Or Universal Serial Bus (USB) interface, etc.

The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 may receive the wireless charging input through the wireless charging coil of the electronic device 101. While the charging management module 140 charges the battery 142, it can also supply power to the electronic device through the power management module 141.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 101 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.

The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 101 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 150 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the electronic device 101. The mobile communication module 150 may include one or more filters, switches, power amplifiers, low noise amplifiers (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering and amplifying the received electromagnetic waves, and then transmitting them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor and then passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 101, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), and global navigation satellites. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating one or more communication processing modules. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic wave radiation via the antenna 2.

In some embodiments, the antenna 1 of the electronic device 101 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 101 can communicate with the network and other devices through wireless communication technology. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).

The electronic device 101 implements a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 101 may include one or N display screens 194, and N is a positive integer greater than one.

The electronic device 101 can realize a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

The ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye. ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other formats. In some embodiments, the electronic device 101 may include 1 or N cameras 193, and N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 101 selects the frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 101 may support one or more video codecs. In this way, the electronic device 101 can play or record videos in a variety of encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 101. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.

The internal memory 121 may be used to store one or more computer programs, and the one or more computer programs include instructions. The processor 110 can run the above-mentioned instructions stored in the internal memory 121 to enable the electronic device 101 to execute the methods provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store the operating system; the storage program area can also store one or more application programs (such as a gallery, contacts, etc.) and so on. The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 101. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, universal flash storage (UFS), etc. In other embodiments, the processor 110 executes the instructions stored in the internal memory 121 and/or the instructions stored in the memory provided in the processor to cause the electronic device 101 to execute the method provided in the embodiments of the present application. And various functional applications and data processing.

The electronic device 101 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.

The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 101 can listen to music through the speaker 170A, or listen to a hands-free call.

The receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 101 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear.

The microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can approach the microphone 170C through the mouth to make a sound, and input the sound signal to the microphone 170C. The electronic device 101 may be provided with one or more microphones 170C. In other embodiments, the electronic device 101 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals. In some other embodiments, the electronic device 101 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.

The earphone interface 170D is used to connect wired earphones. The earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, and a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

In addition, the above electronic equipment may also include one or more components such as buttons, motors, indicators, and SIM card interfaces, which are not limited in the embodiment of the present application.

The above-mentioned software system of the electronic device 101 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes a layered Android system as an example to illustrate the software structure of the electronic device 101.

4 is a block diagram of the software structure of the electronic device 101 according to an embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

1. Application layer

The application layer can include a series of applications.

As shown in FIG. 4, the above-mentioned applications may include APPs (applications) such as call, contact, camera, gallery, calendar, map, navigation, Bluetooth, music, video, short message, etc.

2. Application framework layer

The application framework layer provides application programming interfaces (application programming interface, API) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 4, the application framework layer may include a view system, a notification manager, an activity manager, a window manager, a content provider, a resource manager, an input manager, and so on.

Among them, the view system can be used to construct the display interface of the application. Each display interface can consist of one or more controls. Generally speaking, controls can include interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets. Multiple controls in the display interface can be organized hierarchically according to a tree structure to form a complete ViewTree (view tree). The view system can draw the display interface according to the ViewTree of the display interface, and each control in the display interface corresponds to a set of drawing instructions, such as DrawLine, DrawPoint, DrawBitmap, etc.

For example, (a) in Figure 5 shows the chat interface 401 of the WeChat APP. The bottommost control in the chat interface 401 is the root node. The root node is provided with a basemap 402 control. In the basemap 402 The following controls are also included: a title bar 403, a chat background 404, and an input bar 405. Wherein, the title bar 403 further includes a return button 406 and a title 407, the chat background 404 further includes an avatar 408 and a bubble 409, and the input bar 405 further includes a voice input button icon 410, an input box 411, and a send button 412.

The above controls are layered in order to form a view tree A as shown in Figure 5(b). Among them, the base map 402 is a child node of the root node, and the title bar 403, the chat background 404, and the input field 405 are all child nodes of the base map 402. Both the return button 406 and the title 407 are child nodes of the title bar 403. Both the avatar 408 and the bubble 409 are child nodes of the chat background 404. The voice input button icon 410, the input box 411, and the send button 412 are all child nodes of the input field 405. When displaying the chat interface 401, the view system can call the drawing instructions of the corresponding controls layer by layer to draw each control according to the layer relationship between the controls in the view tree A, and finally form the chat interface 401.

In the embodiment of the present application, if the electronic device 101 is the source device, when the electronic device 101 projects a screen to the destination device, the view system can split, delete, or reorganize the controls in the view tree of the current display interface to determine This time, it needs to be projected to one or more target controls displayed on the target device. Furthermore, the electronic device 101 can project the determined target control to the target device to form a screen projection interface, thereby adapting to device characteristics such as the display size of the target device, and improving the display effect and user experience of the target device in the screen projection scene.

In addition, in the embodiment of the present application, after the electronic device 101 projects the target control in the display interface to the projection interface of the target device (for example, the electronic device 102), the user can display the control in the projection interface in the electronic device 102. The target control inputs a corresponding touch operation to control the electronic device 101 to implement a function corresponding to the touch operation.

Exemplarily, the APP running in the application layer (taking application A as an example) obtains the touch operation input by the user on the touch screen is a process of distributing messages from the bottom layer to the top layer.

When the user's finger is in contact with the touch screen, the touch screen can obtain the relevant information of the touch operation (for example, the coordinates of the touch point, etc.), and then the touch screen can report the original touch generated by the touch operation to the kernel layer in the form of an interrupt through the corresponding driver event. As shown in Figure 4, after the kernel layer obtains the above-mentioned original touch event, it can encapsulate the touch event into a high-level touch event (for example, action down event, action move event, action up event, etc.) that can be read by the upper layer, and combine the Advanced touch events are sent to the Framework layer. Furthermore, the Framework layer can report the above-mentioned advanced touch events to the application process of the running application A in the application layer. The application process of application A calls the corresponding library function to determine the specific control acted by the advanced touch event and the event type of the advanced touch event. For example, the event type may include single click, double tap, sliding, etc. Take the user clicking the play button as an example. After application A determines that the control for this advanced touch event is the play button and the event type is click, the process of application A can call the callback function corresponding to the touch event of clicking the play button Realize the application function corresponding to this touch operation.

In the embodiment of the present application, as shown in FIG. 4, a coordinate conversion module may be set in the application framework layer of the target device. The original touch event reported by the touch screen of the target device to the kernel layer includes the coordinates (x, y) of the touch point, and the coordinates (x, y) are the user's touch position in the projection interface after the projection. Similarly, the touch point (x, y) in the advanced touch event reported by the kernel layer to the Framework layer is also the user's touch position in the projection interface. After the Framework layer receives the advanced touch event, its coordinate conversion module can map the coordinate (x, y) to the corresponding coordinate (x', y') in the display interface of the source device. Furthermore, the destination device can send an advanced touch event carrying coordinates (x', y') to the source device, and the Framework layer of the source device reports the advanced touch event to the screen-projecting application. After the application receives an advanced touch event with a touch point of (x', y'), it is equivalent to receiving a touch event generated by the user on the touch coordinates (x', y') in the source device, then the application can Respond to the touch event carrying the coordinates (x', y') to realize the corresponding application function.

That is to say, after the user inputs a first touch operation with a touch point of (x, y) into the projection interface on the target device, the target device can generate a first touch event that carries coordinates (x, y). Furthermore, the destination device may map the first touch event to a second touch event whose touch point is (x', y') on the display interface of the source device. In this way, after the source device receives the second touch event sent by the target device, it can execute the corresponding application function in response to the second touch event, so as to realize the reverse control function of the target device on the display interface of the source device after the screen is projected.

Of course, the aforementioned coordinate conversion module can also be set in the Framework layer of the source device. At this time, the destination device can send the first touch event with a touch point of (x, y) to the source device, and the coordinate conversion module of the source device maps the first touch event to a touch point of (x', y') In response to the second touch event and execute the corresponding application function in response to the second touch event, the embodiment of the present application does not impose any limitation on this.

It should be noted that, in the above embodiment, a touch event is detected and generated by the touch screen, and the coordinates of the touch point in the touch event are coordinated as an example. It is understandable that when the user inputs a single click, long press or sliding touch operation on the touch screen, the touch screen can detect a series of touch events. For each touch event, the destination device (or source device) can convert the coordinates of the touch point in the touch event according to the above-mentioned method, and the embodiment of the present application does not impose any limitation on this.

In addition, the aforementioned activity manager can be used to manage the life cycle of each application. Applications usually run in the operating system in the form of activity. The activity manager can schedule the activity process of the application to manage the life cycle of each application. The window manager is used to manage window programs. The window manager can obtain the size of the display, determine whether there is a status bar, lock the screen, take a screenshot, etc. The content provider is used to store and retrieve data and make these data accessible to applications. The data may include video, image, audio, phone calls made and received, browsing history and bookmarks, phone book, etc. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

3. Android runtime and system libraries

Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

The system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.

Among them, the surface manager is used to manage the display subsystem, and provides a combination of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to realize 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

4. The kernel layer

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, a sensor driver, etc., which are not limited in the embodiment of the present application.

The following describes in detail a touch method in a projection scene provided by an embodiment of the present application with reference to the accompanying drawings.

Exemplarily, taking a mobile phone as a source device and a smart watch as an example of a destination device, the mobile phone may project one or more controls in its display interface to the smart watch for display.

As shown in FIG. 6, if the mobile phone turns on the screen projection function to the smart watch and displays the play interface 600 of the music APP, it means that the display content in the play interface 600 needs to be projected to the smart watch for display at this time. Exemplarily, the playback interface 600 includes the following controls: a base map 601, a status bar 602, a title bar 603, an album cover 604, lyrics 605, and a control bar 606. Among them, the status bar 602 includes controls such as time, signal strength, and battery capacity. The title bar 603 includes controls such as the song name 6031 and the singer 6032. The control bar 606 includes controls such as a progress bar 6061, a pause button 6062, a previous button 6063, and a next button 6064.

Furthermore, the mobile phone can obtain the corresponding view tree when the view system draws the aforementioned playback interface 600, and the drawing instructions and drawing resources of each control in the view tree. For example, as shown in FIG. 7, it is the view tree 701 of the above-mentioned playing interface 600. The view tree 701 records the layer relationship between the various controls in the aforementioned playback interface 600. In the view tree 701, the root node of the playback interface 600 includes a sub-node of the base map 601. The status bar 602, the title bar 603, the album cover 604, the lyrics 605, and the control bar 606 are all sub-nodes of the base map 601. The song title 6031 and the singer 6041 are child nodes of the title column 603. The progress bar 6061, the pause button 6062, the previous button 6063, and the next button 6064 are child nodes of the control bar 606.

Based on the view tree 701 of the above-mentioned playback interface 600, the mobile phone can further determine one or more controls (ie, target controls) in the playback interface 600 that need to be projected to the display in the mobile phone 500.

Exemplarily, a configuration file corresponding to the aforementioned playback interface 600 may be preset in the mobile phone. Alternatively, the mobile phone may obtain the configuration file corresponding to the playing interface 600 from the server. The configuration file records one or more controls (ie, target controls) that need to be projected onto the smart watch in the playback interface 600 of the mobile phone.

The above configuration file may be stored in a mobile phone or server in a format such as JSON (JavaScript Object Notation) format, XML (Extensible Markup Language) format, or text format, and the embodiment of the present application does not impose any limitation on this.

Exemplarily, the configuration file 1 corresponding to the playing interface 600 may be:

It can be seen that the configuration file 1 contains multiple "src" fields (for example, the aforementioned "src1" field and "src2" field). Each "src" field records the specific position of a control in the playback interface 600. For example, the position of each control can be uniquely determined by the values of the 4 parameters: left, top, widht, and height. Among them, left is the size of the top left corner of the control on the x axis, top is the size of the top left corner of the control on the y axis, width is the width of the control, and height is the height of the control. The one or more controls recorded in the configuration file 1 are the target controls that the mobile phone needs to project to the smart watch.

Then, according to the position of each control recorded in the "src" field in the configuration file 1, the mobile phone can identify the target control in the playback interface 600 that needs to be projected to the mobile phone 500 based on the view tree 701. For example, the target control includes: the song name 6031 and artist 6041 in the title bar 603, the pause button 6062 in the control bar 606, the previous button 6063 and the next button 6064, and the album cover 604.

In some embodiments, the configuration file 1 may also record the specific display position of the target control on the screen projection interface after the screen is projected. For example, a "dest1" field corresponding to the "src1" field can be set in the above configuration file 1, and the "dest1" field is used to indicate the display position of the control 1 in the target device. Exemplarily, the "dest1" field is as follows:

Then, the mobile phone can determine the display position of each target control in the playback interface 600 on the screen projection interface of the smart watch according to the respective "dest" fields in the configuration file 1. Exemplarily, as shown in Figure 8(a), the playback interface 600 of the mobile phone (ie the source device) is located in the first coordinate system, and the control 1 recorded in the "src1" field is located in the first coordinate system before the screen is projected. In area 801. As shown in Figure 8(b), the screen projection interface of the smart watch (ie, the destination device) is located in the second coordinate system, and the control 1 recorded in the "dest1" field is located in the area 802 of the second coordinate system before and after the projection. . Any position in the area 801 uniquely corresponds to a position in the area 802.

In other embodiments, the configuration file 1 can also record the change of the display position of the target control before and after the screen is projected. For example, the following fields are also set for control 1 in configuration file 1:

Among them, the "translationx" field and the "translationy" field are used to indicate the translation distance of the control 1 on the x-axis and the y-axis after the projection; the "scalex" field and the "scaley" field are respectively used to indicate that the control 1 is in the The zoom ratio on the x-axis and y-axis; the "rotatedegree" field is used to indicate the rotation angle of control 1 after projection; the "order" field is used to indicate the layer position of control 1 after projection (for example, in the bottom layer Still in the top layer).

Similarly, the mobile phone can also determine the display position of the control 1 on the screen projection interface of the smart watch after the screen is projected based on the change relationship of the display position of the control 1 before and after the screen is projected recorded in the above field. That is, the position of the control 1 in the first coordinate system and the position of the control 1 in the second coordinate system are determined.

Exemplarily, after the mobile phone recognizes the target control in the playback interface 600 and the specific display position of each target control after being screened, the view tree 701 of the playback interface 600 can be split, cropped, and reorganized to generate a projection screen. The view tree 901 of the projection interface displayed on the smart watch. As shown in Figure 9(a), in the view tree 901, the mobile phone deletes nodes in the view tree 701 that are not target controls, such as the above-mentioned base map 601, status bar 602, various controls in the status bar 602, and the control bar 606 in the progress bar 6061. Also, if the target controls in the title bar 603 and the control bar 606 after the projection are recorded in the profile 1 are located on the layer of the album cover 604, then in the view tree 901, the phone can change the song name in the title bar 603 6031 and singer 6041 are set as the child nodes of the album cover 604, and the pause button 6062, the previous button 6063 and the next button 6064 in the control bar 606 are also set as the child nodes of the album cover 604.

Furthermore, the mobile phone (that is, the source device) can send a UI message to the smart watch (that is, the destination device) through the communication network 104. The UI message includes the view tree 901 and the drawing instructions and drawing resources related to each control in the view tree 901. .

After the smart watch receives the UI message corresponding to the above-mentioned playback interface 600, it can call the drawing instructions of each target control in the view tree 901 in turn according to the level and order in the view tree 901, and draw the target at the position specified in the configuration file 1. Control. Finally, as shown in (b) of FIG. 9, the smart watch can draw the screen projection interface 902 after the above-mentioned playback interface 600 is projected. Each control in the projection interface 902 corresponds to each control in the view tree 901 one-to-one.

It can be seen that when the mobile phone displays the above-mentioned playback interface 600 on the smart watch, it can split, delete, and reorganize the controls in the playback interface 600, so that the final screen projection is in the smart watch. The interface 902 can be adapted to the display size of the display screen in the smart watch and the user's usage requirements, so as to improve the display effect and user experience during screen projection between multiple devices.

Still taking the mobile phone to project the playback interface 600 to the smart watch to display the above-mentioned screen projection interface 902 as an example, the user can input the corresponding touch operation on each target control after the projection in the screen projection interface 902, and the smart watch can respond to the touch operation A corresponding touch event is generated, and further, the smart watch can control the mobile phone to implement a function corresponding to the touch event. For example, the touch event may include the coordinates of the touch point and the time type of the touch event (for example, single click, double tap, slide, etc.).

For example, when the user wants the music APP to pause the song being played, he can click the pause button 6062 in the screen projection interface 902. For another example, when the user wants the music APP to play the previous song, he can click the previous button 6063 in the screen projection interface 902. For another example, when the user wants the music APP to play the next song, he can click the next button 6064 in the screen projection interface 902.

Taking the user clicking the pause button 6062 in the screen projection interface 902 as an example, as shown in FIG. 10, the screen projection interface 902 displayed by the smart watch is located in the second coordinate system. The touch sensor of the smart watch can detect the touch operation input by the user on the screen projection interface 902 in real time. When it is detected that the user's finger touches the projection interface 902, the touch sensor of the smart watch can encapsulate the detected touch information (for example, the coordinate information of touch point A, the touch time, etc.) as a first touch event, and the first touch The touch event is reported to the core layer of the smart watch. The first touch event is generated by the smart watch in response to the first touch operation of the user clicking the pause button 6062 in the projection interface 902.

Taking the coordinates of the touch point as A(x, y) for example, the touch sensor can encapsulate the detected touch operation as the first original touch event and report it to the kernel layer through the driver, and then the kernel layer encapsulates the original touch event as the upper layer. The first high-level touch event read is reported to the application framework layer. After the application framework layer receives the first advanced touch event carrying the coordinates A (x, y), it can determine the user's current touch target according to the display position of each control recorded in the configuration file 1 in the projection interface 902 Which one is the control?

For example, the “dest1” field in the aforementioned configuration file 1 records that the pause button 6062 is located in area 1 of the projection interface 902. Then, when the coordinates A(x, y) fall into area 1, the smart watch can determine that the target control acted on by the user this touch operation is the pause button 6062.

In some embodiments, the coordinates A(x, y) may fall into two controls at the same time. For example, the coordinates A(x, y) are located in the area where the pause button 6062 is located, and are also located in the area where the album cover 604 is located. At this time, the smart watch can determine the uppermost control as the target control for the user's current touch operation according to the "order" field recorded in the configuration file 1.

Take the user clicking the pause button 6062 in the screen projection interface 902 as an example. After the application framework layer determines that the first touch event this time is a touch event on the pause button 6062, the screen can be projected according to the pause button 6062 recorded in the configuration file 1. The front and back position relationship restores the touch point A'corresponding to the touch point A in the first coordinate system of the mobile phone (that is, the source device).

Exemplarily, when the mobile phone projects the pause button 6062 to the smart watch, after performing one or more operations such as translation, zooming, or rotation on the pause button 6062 in the first coordinate system, a pause in the second coordinate system is formed Button 6062. Then, when the smart watch restores the touch point A'corresponding to the touch point A, it can perform the corresponding reverse translation, reverse zoom, or reverse rotation on the above coordinates A(x, y), so as to restore the phone The point A'(x', y') corresponding to the coordinates A (x, y) in the playback interface 600 of.

For example, as shown in (a) of FIG. 11, when the playback interface 600 is displayed in the first coordinate system where the mobile phone (i.e., the source device) is located, the coordinates of point A'in the pause button 6062 are A'(100, 20). When projecting the pause button 6062, as shown in (b) in Figure 11, the pause button 6062 is translated 20 units in the negative direction on the x-axis, and the pause button 6062 is translated in the positive direction on the y-axis 30 units. And, the pause button 6062 is enlarged by 1.5 times. After the pause button 6062 is projected to the second coordinate system where the projection interface 902 is located, the coordinates of point A corresponding to point A'on the pause button 6062 are A((100-20)*1.5, (20+30*1.5)) , Which is A(120,75). Then, after the smart watch detects that the user inputs a touch event to point A of the pause button 6062, the smart watch can reduce the coordinates of point A on the x-axis and y-axis by 1.5 times, and then reverse the x-axis coordinates by 20 Units, and translate the coordinates of point A on the y axis by 30 units in the opposite direction to obtain the coordinates A'(100, 20) corresponding to the coordinates A (120, 75) in the first coordinate system.

Exemplarily, when the translation distance of the pause button 6062 on the x-axis and the y-axis is recorded in the configuration file 1, the smart watch can reversely calculate the translation distance of the touch point A this time according to the translation distance. When the zoom ratio of the pause button 6062 on the x-axis and the y-axis is recorded in the configuration file 1, the smart watch can reversely calculate the zoom ratio of the touched point A according to the zoom ratio. When the rotation angle of the pause button 6062 is recorded in the configuration file 1, the smart watch can reversely calculate the rotation angle of the touched point A according to the rotation angle.

Alternatively, the smart watch may also preset a coordinate mapping formula between the first coordinate system and the second coordinate system. In this way, after the smart watch obtains the touch point A of this touch event, it can calculate the touch point A'corresponding to the touch point A in the first coordinate system of the mobile phone according to the coordinate mapping formula.

After the smart watch restores the touch point A'in the playback interface 600 corresponding to the touch point A on the screen projection interface 902, the coordinates A'(x', y') of the touch point A'can be replaced in the first touch event The coordinates A(x, y) of the touch point A form a second touch event. At this time, the second touch event refers to a touch event generated by the mobile phone when the user inputs a second touch operation of clicking the pause button 6062 in the playback interface 600. It is understandable that, in this embodiment of the application, the user does not actually click the pause button 6062 in the playback interface 600, but the smart watch simulates the user's presence in the playback interface 600 by converting touch point A to touch point A' The second touch operation of clicking the pause button 6062.

Furthermore, the smart watch can send the aforementioned second touch event to the mobile phone. The application framework layer of the mobile phone can report the second touch event to the music APP running in the application layer, so that the music APP can pause the audio being played in response to the second touch event at point A'. It is understandable that the music APP can respond to the second touch event of point A', which is equivalent to the music APP responding to the user's first touch event of point A on the screen projection interface 902.

That is, the user inputs the first touch operation at point A(x, y) in the projection interface 902 of the target device, and the target device generates the first touch event corresponding to the first touch operation. The destination device (or source device) performs coordinate conversion on the coordinates of the touch point in the first touch event and generates a second touch event, so that the music APP in the source device thinks that the user is on the A'( of the playback interface 600 based on the second touch event). x', y') point performs the second touch operation. Furthermore, the music APP can execute the corresponding application function in response to the second touch event, so as to realize reverse control of the source device by the destination device during screen projection.

In other embodiments, the smart watch (that is, the destination device) can also send touch events carrying A(x, y) points to the mobile phone (that is, the source device), and then the application framework layer in the mobile phone sends the touch event according to the above method. The A(x,y) point is restored to the A'(x',y') point in the playback interface 600, and the touch event with the touch point being A'(x',y') is reported to the music APP in the phone , To achieve the function of pausing audio playback.

In other embodiments, the identification of each control may also be recorded in the aforementioned configuration file 1. For example, the control corresponding to the "dest1" field is the pause button 6062, and the identification of the pause button 6062 is 001. Then, the smart watch (ie, the target device) can determine that the user has performed a click operation on the pause button 6062 in the screen projection interface 902 based on the coordinates of the touch point and the touch time in the series of touch events detected. Furthermore, the smart watch may send the identification of the pause button 6062 (for example, 001) and the determined type of touch event (for example, a click operation) to the mobile phone (that is, the source device). Furthermore, the mobile phone can determine that the user has performed the event of clicking the pause button 6062. Then, the application framework layer in the mobile phone can report the event of the user clicking the pause button 6062 to the running music APP, so that the music APP can be called and clicked. The function corresponding to the pause button 6062 realizes the function of pausing the audio playback, that is, executing the operation instruction corresponding to the first touch event.

Or, after the smart watch (that is, the destination device) determines that the user has performed a click operation on the pause button 6062 in the screen projection interface 902, it can also be based on the specific position of the pause button 6062 recorded in the configuration file 1 in the playback interface 600 , Generate a corresponding touch event (such as a third touch event). The event type of the third touch event is the same as the event type of the first touch event, and both are click events. The coordinate B of the touch point in the third touch event can be located at any position of the pause button 6062 in the play interface 600. In this way, after the smart watch sends the generated third touch event to the mobile phone (that is, the source device), the mobile phone can also report the third touch event to the music APP running in the application layer, so that the music APP can respond to the first point of point B. The three-touch event pauses the audio being played. Similarly, the music APP responding to the third touch event of point B is equivalent to the music APP responding to the user's first touch event of point A on the screen projection interface 902.

Of course, the user can also input the corresponding touch operation in the playback interface 600 displayed on the mobile phone (ie, the source device). After the mobile phone detects the touch event corresponding to the touch operation, the coordinate of the touch point can be converted. The touch event is reported to the music APP to realize the corresponding application function.

That is to say, in the screen projection scenario, the user can input touch operations in the source device to control the source device to achieve corresponding functions, or input touch operations in the destination device to control the source device to achieve corresponding functions, thereby improving the user’s ability to cast Touch experience in the screen scene.

In addition, if the source device updates its display screen in response to the touch operation input by the user on the source device or the destination device, the source device can continue to use the above screen projection method to project the updated display interface to the destination device for display. The embodiment does not impose any limitation on this.

In some usage scenarios, the user can project the display content in one source device to multiple different destination devices for display. Then, according to the aforementioned touch control method, the user can input a corresponding touch operation in each target device to control the source device to implement related application functions.

Exemplarily, as shown in FIG. 12, after the mobile phone is used as the source device to turn on the screen projection function, the playback interface 1201 being displayed by the video APP can be simultaneously screened to two destination devices. Among them, one destination device is a smart watch, and the other destination device is a smart TV.

Similar to the above-mentioned projection method, the mobile phone can recognize that the playback interface 1201 needs to be projected to the first target control displayed on the smart watch according to the configuration file 1 corresponding to the smart watch: the control bar 1205 and the controls 1206 in the control bar 1205. Control 1207 and control 1208. Furthermore, as shown in FIG. 12, the mobile phone can project the control bar 1205 and various controls in the control bar 1205 to the smart watch to form a first projection interface 1301.

At the same time, according to the configuration file 2 corresponding to the smart TV, the mobile phone can recognize that the second target control that the playback interface 1201 needs to be projected to the smart TV display is: the video screen 1202 and the text control 1203 and the progress bar 1204 in the video screen 1202. Furthermore, as still shown in FIG. 12, the mobile phone can project the video screen 1202 and various controls in the video screen 1202 to the smart TV to form a second screen projection interface 1302.

Then, the user can input a touch operation in the first projection interface 1301 to control the video APP running in the mobile phone (ie, the source device). In addition, the user can also input a touch operation in the second screen projection interface 1302 to control the video APP running in the mobile phone (ie, the source device).

Exemplarily, as shown in FIG. 13, if the smart watch (ie, the first destination device) detects that the user clicks the pause button 1106 in the first screen projection interface 1301, the smart watch may generate a first touch event including the touch point P1 . In addition, the smart watch can convert the touch point P1 in the first screen projection interface 1301 to the touch point P1' in the playback interface 1201 according to the positional relationship of the pause button 1106 recorded in the configuration file 1 before and after the screen projection. Furthermore, the smart watch can send the second touch event including the touch point P1' to the mobile phone, so that the video APP in the mobile phone can execute an instruction to pause the video in response to the second touch event with the touch point P1'.

Exemplarily, as shown in FIG. 14, if the smart TV (ie, the second destination device) detects that the user drags the progress bar 1104 in the second projection interface 1302, the smart TV can generate a first touch that includes the touch point P2. event. In addition, the smart watch can convert the touch point P2 in the second screen projection interface 1302 to the touch point P2' in the aforementioned playback interface 1201 according to the positional relationship of the progress bar 1104 recorded in the configuration file 2 before and after the screen projection. Furthermore, the smart TV can send the second touch event including the touch point P2' to the mobile phone, so that the video APP in the mobile phone can respond to the second touch event with the touch point P2' to switch the video to the point P2' on the progress bar 1104 Play at the corresponding position.

In addition, if the mobile phone not only receives the touch event sent by the first destination device, but also the touch event sent by the second destination device, the mobile phone can respond to each touch event in sequence according to the time sequence of receiving each touch event. Or, when each target device detects a touch operation input by the user, it may also record the touch time of the touch operation. Furthermore, when the destination device sends a corresponding touch event to the source device, it can also send the touch time of the touch event. In this way, the source device can respond to touch events sent by different destination devices in sequence according to the sequence of the touch time.

It can be seen that when the source device simultaneously projects the display content in its display interface to multiple destination devices for display, the user can input a touch operation in any destination device to reversely control the source device to realize the control function corresponding to the touch operation. So as to improve the user's touch experience in the projection scene.

In some usage scenarios, the user can project the display content from multiple source devices to the same destination device for display. Then, according to the above touch method, after the user inputs a corresponding touch operation on a certain control in the target device, the user can control the source device corresponding to the control to implement related application functions.

Exemplarily, as shown in FIG. 15, the user can simultaneously use the mobile phone and the smart watch as the source device of the smart TV (that is, the destination device). Among them, the mobile phone can project the display content in the lock screen interface 1501 being displayed to the smart TV, and at the same time, the smart watch can project the display content in the detection interface 1502 being displayed to the smart TV. Of course, in addition to displaying content projected from mobile phones and smart watches, smart TVs can also display the display screen of the smart TV itself.

Similar to the above screen projection method, the mobile phone can identify the first target control in the lock screen interface 1501 that needs to be projected to the smart TV display according to the configuration file 1 corresponding to the lock screen interface 1501 as: the

icons

1512 and 1512 in the notification message 1511 Message content 1513. In addition, the smart watch can identify the second target control in the detection interface 1502 that needs to be projected to the smart TV display according to the configuration file 2 corresponding to the detection interface 1502 as: heart rate information 1521 and calorie information 1522.

After the smart TV receives the first target control sent by the mobile phone and the second target control sent by the smart watch, it can split and reorganize the first target control, the second target control and the control 1503 in its own display interface. . Furthermore, as still shown in FIG. 15, the smart TV may display the above-mentioned first target control, second target control, and control 1503 in the projection interface 1504. In this way, the destination device can simultaneously display the display content of multiple source devices.

Further, the user can input touch operations on the corresponding controls in the projection interface 1504. If the user inputs a touch operation on the first target control in the projection interface 1504, the mobile phone (ie, the first source device) can be controlled to implement the corresponding function. If the user inputs a touch operation on the second target control in the projection interface 1504, the smart watch (ie, the second source device) can be controlled to implement the corresponding function.

Exemplarily, as shown in FIG. 16, after detecting that the user clicks the message content 1513 in the projection interface 1504, the smart TV (ie, the destination device) may generate a first touch event including the touch point Q1. Since the message content 1513 belongs to the first target control projected by the mobile phone (that is, the first source device), the smart TV can change the position of the message content 1513 recorded in the configuration file 1 before and after the screen projection to change the screen in the projection interface 1504 The touch point Q1 is converted into the touch point Q1' in the lock screen interface 1501. Furthermore, the smart watch can send the second touch event including the touch point Q1' to the mobile phone, so that the mobile phone can expand the message content 1513 in response to the second touch event with the touch point P1'.

Exemplarily, as shown in FIG. 17, after the smart TV (ie, the destination device) detects that the user clicks on the heart rate information 1521 in the projection interface 1504, it may generate a first touch event including the touch point Q2. Since the heart rate information 1521 belongs to the second target control projected by the smart watch (that is, the second source device), the smart TV can set the projection interface 1504 according to the position relationship of the heart rate information 1521 recorded in the profile 2 before and after the projection. The touch point Q2 of is converted into the touch point Q2' in the detection interface 1502. Furthermore, the smart TV can send the second touch event including the touch point Q2' to the smart watch, so that the smart watch can display the detailed content of the heart rate information 1521 in response to the second touch event with the touch point P2'.

It can be seen that when multiple source devices simultaneously project the display content in their display interface to the same target device for display, the user can input touch operations on the controls projected from different source devices in the target device to control the corresponding The source device implements the control function corresponding to the touch operation, thereby improving the user's touch experience in the screen projection scene.

It should be noted that the above embodiments only exemplarily show the application scenario of inputting a touch operation in the destination device to reversely control the source device. It should be understood that the above touch method during projection can also be applied to other In the scenario, the embodiment of the present application does not impose any restriction on this.

Exemplarily, when a video conference is held, an electronic device in a certain conference site can be used as a destination device, and various electronic devices in other conference sites can be used as a source device. Each source device can project the target control to the target device for display according to the above method. Furthermore, the user can input a corresponding control operation to the target control in the target device, thereby controlling the corresponding source device to respond to the control operation to implement reverse control during screen projection.

For another example, students can install the teaching assistant APP on their mobile phones or computers or tablets. When students use the teaching assistant APP to answer questions, their electronic devices can be used as source devices to project the display content of the answer area to the teacher's mobile phone or computer or tablet for display. Then, the teacher can not only preview the answering process of multiple students in their respective answering areas in real time, but also remotely control the student's source device in their own electronic equipment, help students solve problems online, and improve the teaching effect of the teaching assistant APP.

The embodiment of the present application discloses an electronic device including a processor, and a memory, an input device, an output device, and a communication module connected to the processor. Among them, the input device and the output device can be integrated into one device. For example, a touch sensor can be used as an input device, a display screen can be used as an output device, and the touch sensor and display screen can be integrated into a touch screen.

At this time, as shown in FIG. 18, the above electronic device may include: a touch screen 1801, which includes a touch sensor 1806 and a display screen 1807; one or more processors 1802; a memory 1803; a communication module 1808; one or more Application programs (not shown); and one or more computer programs 1804. The above-mentioned devices can be connected through one or more communication buses 1805. The one or more computer programs 1804 are stored in the aforementioned memory 1803 and configured to be executed by the one or more processors 1802, and the one or more computer programs 1804 include instructions, and the aforementioned instructions can be used to execute the aforementioned implementations. The steps in the example. Among them, all relevant content of the steps involved in the above method embodiments can be cited in the functional description of the corresponding physical device, which will not be repeated here.

Exemplarily, the foregoing processor 1802 may specifically be the processor 110 shown in FIG. 3, the foregoing memory 1803 may specifically be the internal memory 121 and/or the external memory 120 shown in FIG. 3, and the foregoing display screen 1807 may specifically be FIG. As shown in the display 194, the touch sensor 1806 may be the touch sensor in the sensor module 180 shown in FIG. 3, and the communication module 1808 may be the mobile communication module 150 and/or the wireless communication module 160 shown in FIG. The embodiment of this application does not impose any restriction on this.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated as needed. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the system, device, and unit described above, reference may be made to the corresponding process in the foregoing method embodiment, which is not repeated here.

The functional units in the various embodiments of the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or the part that contributes to the prior art or all or part of the technical solutions can be embodied in the form of software products, and the computer software products are stored in a storage The medium includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.

The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any changes or substitutions within the technical scope disclosed in the embodiments of the present application shall be covered by this Within the protection scope of the application embodiments. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

A touch method in a projection scene, characterized in that it includes:

The source device displays the first display interface;

In response to the screen projection instruction input by the user, the source device projects the N controls in the first display interface to the screen projection interface displayed by the first destination device, where N is an integer greater than 0;

Receiving, by the source device, a first touch event sent by the first destination device;

The source device executes an operation instruction corresponding to the first touch event.
The method according to claim 1, wherein after the source device receives the first touch event sent by the first destination device, the method further comprises:

Determining, by the source device, a target control corresponding to the first touch event, where the target control is one of the N controls;

Wherein, the operation instruction is a corresponding operation instruction when the target control is triggered on the source device.
The method according to claim 1, wherein the first touch event is: after the first destination device generates a fifth touch event in response to a touch operation input by the user on the projection interface, the The fifth touch event is mapped to a touch event on the first display interface.
The method according to claim 2, wherein the first touch event is: a touch event generated by the first destination device when a user inputs a first touch operation on the screen projection interface.
The method according to claim 4, wherein the source device stores a configuration file corresponding to the first display interface, and the configuration file records that the N controls are displayed on the first display interface. The display position in, and the display position of the N controls in the projection interface;

Wherein, the source device determining the target control corresponding to the first touch event includes:

The source device determines the target control corresponding to the first touch event according to the display positions of the N controls recorded in the configuration file on the projection interface.
The method of claim 5, wherein the first touch event comprises a first coordinate of the first touch operation in the projection interface;

Wherein, the source device determining the target control corresponding to the first touch event according to the display positions of the N controls recorded in the configuration file on the projection interface includes:

When the first coordinate falls within the display position of the first control in the configuration file, the source device determines the first control as the target control.
The method of claim 5, wherein the first touch event comprises a first coordinate of the first touch operation in the projection interface;

Wherein, the source device determining the target control corresponding to the first touch event according to the display positions of the N controls recorded in the configuration file on the projection interface includes:

When the first coordinate falls within the display position of the first control in the configuration file, and at the same time falls within the display position of the second control in the configuration file, the source device will be located at the first The control is determined to be the target control.
The method according to any one of claims 5-7, wherein after the source device determines the target control corresponding to the first touch event, the method further comprises:

The source device maps the first touch event to a second touch event according to the configuration file, and the second touch event is: the user inputs a second touch to the target control in the first display interface The touch event that the source device will generate during operation;

Wherein, the source device executing an operation instruction corresponding to the first touch event includes:

The source device reports the second touch event to the first application; the first display interface is an interface of the first application.
The method according to claim 8, wherein the source device mapping the first touch event to the second touch event according to the configuration file comprises:

According to the corresponding relationship between the first display position of the target control in the first display interface and the second display position in the projection interface, the source device records the target control in the configuration file. The first coordinate in the first touch event is converted to the second coordinate to obtain the second touch event.
The method according to any one of claims 5-6, wherein the source device executing an operation instruction corresponding to the first touch event comprises:

The source device reports the identifier of the target control and the event type of the first touch event to a first application, so that the first application executes a first function, and the first function is the first application When the target control is triggered by the operation indicated by the event type, the corresponding function; the first display interface is the interface of the first application.
The method according to any one of claims 5-6, wherein after the source device determines the target control corresponding to the first touch event, the method further comprises:

The source device generates a third touch event according to the first display position of the target control in the first display interface recorded in the configuration file, and the event type of the third touch event is the same as that of the first touch The event types of the events are the same, and the third coordinate in the third touch event is located in the first display position;

Wherein, the source device executing an operation instruction corresponding to the first touch event includes:

The source device reports the third touch event to the first application; the first display interface is an interface of the first application.
The method according to any one of claims 1-11, wherein after the source device displays the first display interface, the method further comprises:

In response to the second screen projection instruction input by the user, the source device projects the M controls in the first display interface to the second destination device for display, where M is an integer greater than 0;

Receiving, by the source device, a fourth touch event sent by the second destination device;

The source device executes an operation instruction corresponding to the fourth touch event.
A touch method in a projection scene, characterized in that it includes:

The destination device receives a first message sent by the first source device, the first message includes a drawing instruction of a first target control, and the first target control is one of the first display interfaces displayed by the first source device Or multiple controls;

The destination device invokes the drawing instruction of the first target control to draw a screen projection interface, and the screen projection interface includes the first target control;

In response to the first touch operation input by the user to the first target control in the projection interface, the target device generates a first touch event;

The destination device instructs the first source device to execute an operation instruction corresponding to the first touch event.
The method according to claim 13, wherein the destination device instructs the first source device to execute an operation instruction corresponding to the first touch event, comprising:

The destination device sends the first touch event to the first source device, so that the first source device executes an operation instruction corresponding to the first touch event.
The method according to claim 13, wherein after the target device generates the first touch event, the method further comprises:

The target device maps the first touch event to a second touch event, where the second touch event is: when a user inputs a second touch operation to the first target control on the first display interface The touch event to be generated by the first source device;

Wherein, the destination device instructing the first source device to execute an operation instruction corresponding to the first touch event includes:

The destination device sends the second touch event to the first source device, so that the first source device executes an operation instruction corresponding to the second touch event.
The method according to any one of claims 13-15, wherein the method further comprises:

The destination device receives a second message sent by a second source device, the second message includes a drawing instruction of a second target control, and the second target control is in a second display interface displayed by the second source device One or more controls;

The target device invokes the drawing instruction of the second target control to draw the second target control on the projection interface;

In response to a third touch operation input by the user to the second target control in the projection interface, the target device generates a third touch event;

The destination device instructs the second source device to execute an operation instruction corresponding to the third touch event.
An electronic device, characterized in that it comprises:

A touch screen, the touch screen including a touch sensor and a display screen;

Communication module

One or more processors;

One or more memories;

And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, when the instructions are executed by the electronic device When the time, the electronic device is caused to execute the touch method in the projection scene according to any one of claims 1-12 or claims 13-16.
A computer-readable storage medium having instructions stored in the computer-readable storage medium, characterized in that, when the instructions are executed on an electronic device, the electronic device is caused to execute as claimed in claim 1-12 or claim The touch method in the projection scene described in any one of 13-16.
A computer program product containing instructions, characterized in that, when the computer program product runs on an electronic device, the electronic device is caused to execute any one of claims 1-12 or 13-16. The touch method in the screen projection scene.
A touch control system in a projection scene, wherein the system includes at least one source device and at least one destination device; wherein, the source device is used to execute any one of claims 1-12 The touch method in the projection scene, the target device is used to execute the touch method in the projection scene according to any one of claims 13-16.