WO2024037542A1

WO2024037542A1 - Touch input method, system, electronic device, and storage medium

Info

Publication number: WO2024037542A1
Application number: PCT/CN2023/113141
Authority: WO
Inventors: 段鹏; 龚作豪; 董刚刚; 文平; 王威
Original assignee: 华为技术有限公司
Priority date: 2022-08-19
Filing date: 2023-08-15
Publication date: 2024-02-22
Also published as: CN117631939A

Abstract

A touch input method, a system, an electronic device, and a storage medium. The method comprises: a first operating system receives a touch operation of a user on a preset application; in response to the touch operation, the first operating system sends to a second operating system a touch event generated on the basis of the touch operation; the second operating system generates an operation instruction corresponding to the touch event; and, by means of the preset application, the second operating system executes the operation instruction. The method may generate the operation instruction identified by the preset application supporting cross-system operation, and thus improves the consistency of cross-system operations, thereby improving the extension of the application ecology in the operating systems, and meeting the use requirements of users.

Description

A touch input method, system, electronic device and storage medium

This application requires the priority of the Chinese patent application submitted to the State Intellectual Property Office on August 19, 2022, with the application number 202211009728.1 and the application title "A touch input method, system, electronic device and storage medium", which The entire contents are incorporated herein by reference.

Technical field

The present application belongs to the field of device control technology, and in particular relates to a touch input method, system, electronic device and storage medium.

Background technique

With the continuous development of device technology, mobile terminals are becoming more and more powerful. In addition to being able to run applications suitable for mobile terminals, they can also run emulators of other operating systems. Applications suitable for other operating systems can be run through emulators. , to achieve cross-system application operations.

However, existing device control technology can run applications based on other operating systems through an emulator within a certain operating system. For example, a smartphone is equipped with a container running Linux, and the container can run a system based on Linux. However, due to the inconsistent configuration of modules in different operating systems, there are differences in how users operate different operating systems. The touch operations that users can originally perform on smart devices cannot be run through containers. Another operating system identified that when running some applications across systems, the operation methods are not universal, thus reducing the user experience.

Contents of the invention

Embodiments of the present application provide a touch input method, system, electronic device, and computer-readable storage medium, which can solve the problem of non-universal operating methods in existing device control technologies when running some applications across systems. , thereby reducing the problem of user experience.

In a first aspect, embodiments of the present application provide a touch input method. The method is applied to an electronic device. The electronic device is equipped with a first operating system and a second operating system. The first operating system and the second operating system are both equipped with a first operating system and a second operating system. The second operating system is different; the method includes:

The first operating system receives a user's touch operation on a preset application; the preset application is built based on the second operating system;

The first operating system responds to the touch operation and sends a touch event generated based on the touch operation to the second operating system;

The second operating system generates an operation instruction corresponding to the touch event;

The second operating system executes the operation instruction through the preset application.

It can be seen from the above that the touch input method provided by the embodiment of the present application can run a preset application across operating systems in the first operating system, and the user initiates a touch control for the preset application in the first operating system. During operation, the corresponding touch event can be generated through the first touch module of the first operating system and transmitted to the second operating system, and the touch event can be converted through the second touch module in the second operating system to generate a The operation instructions recognized by the second operating system allow the preset application running in the second operating system to execute the operation instructions and generate corresponding response results, thereby realizing interoperability of touch operations across systems. Compared with the existing device control technology, in the embodiments of the present application, corresponding touch modules are configured in different operating systems, and the touch operations are converted across systems through the touch modules, so that it is possible to generate a system that supports cross-system operation. The preset application-identified operation instructions improve the consistency of cross-system operations, thereby improving the expansion of the application ecosystem within the operating system and meeting the user needs.

In a possible implementation of the first aspect, the first operating system and the second operating system are any of the following operating systems: Android operating system, Hongmeng operating system, Linux operating system or windows operating system. .

In a possible implementation of the first aspect, the touch operation is an edge gesture type touch operation, a multi-touch point type touch operation, or a third click on any area in the preset application. Three-touch operation.

In a possible implementation of the first aspect, the touch operation is a first touch operation of an edge gesture type;

In response to the touch operation, the first operating system sends a touch event generated based on the touch operation to the second operating system, including:

The first operating system determines the window type of the target window, and generates a first touch event corresponding to the first touch operation according to the window type; the target window is the first touch operation in the preset time. The window operated in the application; the first touch event includes: the window type of the target window and the operation type corresponding to the first touch operation.

In a possible implementation of the first aspect, the window type is a parent window type or a child window type, and the operation type is closing the window, exiting the application, or performing application management.

In a possible implementation of the first aspect, the touch operation is a second touch operation of a multi-contact type;

The first operating system generates a multi-touch event corresponding to the second touch operation and transmits the multi-touch event to the second operating system; the multi-touch event is included in the The first touch coordinates and touch time of each contact point in the first operating system;

The first touch module sends the touch event to the second touch module in the container running the second operating system, and the second touch module generates the touch event. The operation instructions corresponding to the event include:

The second operating system performs coordinate conversion on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system.

In a possible implementation of the first aspect, the first operating system generates a multi-touch event corresponding to the second touch operation and transmits the multi-touch event to the second Touch module, including:

The first operating system monitors touch operations initiated by the user;

If the first operating system recognizes that the touch operation is a second touch operation, it performs legal verification on the second touch operation;

If the first operating system recognizes that the second touch operation passes the legal verification, it generates the multi-touch event and sends the multi-touch event to the second operating system. .

In a possible implementation of the first aspect, the second operating system performs coordinate transformation on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system. ,include:

The second operating system performs complete verification on the multi-touch event;

If the second operating system recognizes that the multi-touch event passes complete verification, it performs coordinate conversion on each of the touch points in the multi-touch event to determine each of the first touch coordinates. The corresponding second touch coordinates in the second operating system; the second touch coordinates are based on the size of the first window of the preset application in the first operating system and the location of the preset application in the first operating system. The size of the corresponding second window in the second operating system and the first touch coordinates are determined; the first window is the mapping window of the second window under the first operating system;

The second operating system generates the second operation instruction according to the second touch coordinates.

In a possible implementation of the first aspect, the second operating system generates the second operation instruction according to the second touch coordinates, including:

The second operating system determines a target window for a required operation in the preset application, and generates a second operation instruction that acts on the target window based on the second contact point coordinates.

In a possible implementation of the first aspect, the touch operation includes a third touch operation of clicking on any area in the preset application;

The first operating system generates a cursor displacement touch event corresponding to the third touch operation, and sends the cursor displacement touch event to the second operating system; the cursor displacement touch event includes The third touch coordinate corresponding to the above-mentioned touch operation;

The second operating system generates operation instructions corresponding to the touch event, including:

The second operating system identifies the area type corresponding to the third touch coordinate in the preset application, and generates a third operation instruction according to the area type;

After the second operating system generates the operation instruction corresponding to the touch event, it also includes:

The first operating system receives feedback from the second operating system regarding the area type corresponding to the third touch coordinate;

When the first operating system detects that the area type is an editable area type, the soft keyboard is enabled.

In a possible implementation of the first aspect, after the first operating system receives feedback from the second operating system regarding the area type corresponding to the third touch coordinate, the method further includes:

The first operating system does not open the soft keyboard when detecting that the area type is a non-editing area type.

In a possible implementation of the first aspect, the first operating system generates a cursor displacement touch event corresponding to the third touch operation, and sends the cursor displacement touch event to the second Operating system, including:

The first operating system monitors the touch operation initiated by the user;

If the first operating system recognizes that the touch operation is a third touch operation, it performs legal verification on the third touch operation;

If the first operating system recognizes that the third touch operation passes the legal verification, it generates a click recognition event;

In response to the click recognition event, the first operating system identifies the click type corresponding to the third touch operation and the third touch coordinates, and determines the click type and the third touch coordinates based on the click type and the third touch coordinates. The generated cursor displacement event is sent to the second operating system.

In a possible implementation of the first aspect, the second operating system identifies the area type corresponding to the third touch coordinate in the preset application, and generates a third operation instruction according to the area type. ,include:

The second operating system receives the cursor displacement event sent by the first operating system, and updates the virtual cursor in the second operating system to the third touch coordinates;

The second operating system identifies the cursor style of the virtual cursor when updating to the third touch coordinates, and determines the area type corresponding to the third touch coordinates based on the cursor style.

In a possible implementation of the first aspect, receiving a user's touch operation on a preset application in the first operating system further includes:

The first operating system receives a user's startup operation for the preset application;

The first operating system initiates an application startup instruction to the second operating system through a proxy application installed in the first operating system; the application startup instruction carries the application identifier of the preset application;

The second operating system responds to the application startup instruction, runs the preset application, and feeds back the operation interface of the preset application to the proxy application in the first operating system;

The first operating system generates the operation interface through the proxy application.

In a second aspect, embodiments of the present application provide a touch input system. The touch input system includes a first operating system and a second operating system; the first operating system is different from the second operating system. ;

The first operating system includes:

A touch operation receiving module, configured to receive a user's touch operation on a preset application in the first operating system; the preset application is built based on the second operating system;

A first touch module, configured to respond to a touch operation and send a touch event generated based on the touch operation to a second touch module in the second operating system;

The second operating system includes:

The second touch module is used to generate operation instructions corresponding to the touch event;

An operation instruction execution unit is used to execute the operation instruction through the preset application.

In a possible implementation of the second aspect, the touch operation is an edge gesture type touch operation, a multi-touch point type touch operation, or a third-party click on any area in the preset application. Touch operation.

In a possible implementation of the second aspect, the first operating system and the second operating system are any one of the following operating systems: Android operating system, Hongmeng operating system, Linux operating system or windows operating system.

In a possible implementation of the second aspect, the touch operation is a first touch operation of an edge gesture type;

The first touch module is specifically configured to: determine a window type of a target window, and generate a first touch event corresponding to the first touch operation according to the window type; the target window is the first touch event. Control the window operated in the preset application; the first touch event includes: the window type of the target window and the operation type corresponding to the first touch operation;

The second touch module also includes a window management unit;

The window management unit is used to generate a first operation instruction for the target window.

In a possible implementation of the second aspect, the window type is a parent window type or a child window type, and the operation type is closing the window, exiting the application, or performing application management.

In a possible implementation of the second aspect, the first touch module further includes: a touch management unit and a touch window management unit;

The touch management unit is configured to monitor the touch operation initiated by the user, and when the touch operation is recognized as a first touch operation, send the first touch operation to the Touch window management unit;

The touch window management unit is used to establish a cascade relationship between each operation window in the preset application;

The touch window management unit is also configured to determine the window type of the target window according to the first touch operation.

In a possible implementation of the second aspect, the touch operation is a multi-touch type second touch operation;

The first touch module is specifically configured to: generate a multi-touch event corresponding to the second touch operation, and transmit the multi-touch event to the second operating system in the second operating system. Touch module; the multi-touch event includes the first touch coordinates and touch time of each touch point in the first operating system;

The second touch module is specifically configured to perform coordinate conversion on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system.

In a possible implementation of the second aspect, the first touch module includes: a touch management unit and a touch event conversion unit;

The touch management unit is configured to monitor the touch operation initiated by the user, and when detecting that the touch operation is the second touch operation, send the second touch operation to the touch screen. Control event conversion unit;

The touch event conversion unit is used to perform legal verification on the second touch operation;

The touch event conversion unit is also configured to generate the multi-touch event if it is recognized that the second touch operation passes the legal verification, and send the multi-touch event to all The second touch module in the second operating system.

In a possible implementation of the second aspect, the second touch module includes: a multi-touch management unit and a touch window conversion unit;

The multi-touch management unit is used to complete verification of the multi-touch event, and when it is recognized that the multi-touch event passes the complete verification, send the multi-touch event. to the touch window conversion unit;

The touch window conversion unit is used to perform coordinate conversion on each of the touch points in the multi-touch event, and determine the second touch corresponding to each of the first touch coordinates in the second operating system. Coordinates; the second touch coordinates are based on the size of the first window of the preset application in the first operating system and the corresponding second window of the preset application in the second operating system. The size and the first touch coordinates are determined; the first window is a mapping window for the second window under the first operating system;

The touch window conversion unit is also used to generate the second operation instruction according to the second touch coordinates.

In a possible implementation of the second aspect, the control window conversion unit is also used to:

Determine a target window for a required operation in the preset application, and generate a second operation instruction that acts on the target window based on the second contact point coordinates.

In a possible implementation of the second aspect, the touch operation is a third touch operation of clicking on any area in the preset application;

The first touch module is specifically configured to: generate a cursor displacement touch event corresponding to the third touch operation, and send the cursor displacement touch event to the second operating system in the second operating system. Touch module; the cursor displacement touch event includes the third touch coordinate corresponding to the touch operation;

The second touch module is specifically configured to: identify the area type corresponding to the third touch coordinate in the preset application, and generate a third operation instruction according to the area type;

The second touch module is also configured to: send the area type to the first touch module in the first operating system;

The first touch module is also configured to enable the soft keyboard when detecting that the area type is an editable area type.

In a possible implementation of the second aspect, the first touch module is further configured to not open the soft keyboard when it is detected that the area type is a non-editing area type.

In a possible implementation of the second aspect, the first touch module includes: a touch management unit, a touch event conversion unit, and an input touch detection unit;

The touch management unit is configured to monitor the touch operation initiated by the user, and when detecting that the touch operation is a third touch operation, send the third touch operation to the Touch event conversion unit;

The touch event conversion unit is used to perform legal verification on the third touch operation, and when it is detected that the third touch operation passes the legal verification, the input touch detection unit Send a click recognition event generated based on the third touch operation;

The input touch detection unit is configured to, in response to the click recognition event, identify the click type corresponding to the third touch operation and the third touch coordinates, and determine the click type and the third touch coordinate based on the click type and the third touch coordinate. The cursor displacement event generated by the three touch coordinates is sent to the second touch module in the second operating system.

In a possible implementation of the second aspect, the first touch module further includes: an input state recognition unit;

The input touch detection unit is also configured to send click gesture information to the input state recognition unit in the first touch module, so that the input state recognition unit monitors the feedback from the second touch module. Waiting states for zone types;

The input state recognition unit is configured to receive the area type fed back by the second touch module, and when detecting that the area type is an editable area type, enable the soft keyboard.

In a possible implementation of the second aspect, the second touch module includes: a click input subunit and a cursor classification unit;

The click input subunit is configured to receive the cursor displacement event sent by the first touch module of the first operating system, and update the virtual cursor in the second operating system to the third Touch coordinates;

The cursor classification unit is used to identify the cursor style of the virtual cursor when updating to the third touch coordinates, and determine the area type corresponding to the third touch coordinates based on the cursor style;

The cursor classification unit is also used to send the area type to the first touch module in the first operating system.

In a possible implementation of the second aspect, the first operating system includes: a startup operation receiving unit, an application startup unit, and an interface display unit; the second operating system includes: an interface feedback unit;

A startup operation receiving unit configured to receive a user's startup operation for the preset application in the first operating system;

An application startup unit configured to initiate an application startup instruction to the second operating system through a proxy application installed in the first operating system; the application startup instruction carries the application identifier of the preset application;

An interface feedback unit, configured to respond to the application startup instruction, run the preset application in the second operating system, and feed back the operation interface of the preset application to the proxy application;

An interface display unit is configured to generate the operation interface in the first operating system through the proxy application.

In a third aspect, embodiments of the present application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes The computer program implements the touch input method described in any one of the above first aspects.

In a fourth aspect, embodiments of the present application provide a computer-readable storage medium that stores a computer program, which is characterized in that when the computer program is executed by a processor, any of the above-mentioned aspects of the first aspect is implemented. A method of touch input.

In a fifth aspect, embodiments of the present application provide a computer program product, which when the computer program product is run on an electronic device, causes the electronic device to perform the touch input method described in any one of the above first aspects.

In a sixth aspect, embodiments of the present application provide a chip system, including a processor. The processor is coupled to a memory. The processor executes a computer program stored in the memory to implement touch control as described in any one of the first aspects. Input method.

It can be understood that the beneficial effects of the above-mentioned second to sixth aspects can be referred to the relevant descriptions in the above-mentioned first aspect, and will not be described again here.

Description of drawings

Figure 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

Figure 2 is a software structure block diagram of the electronic device according to the embodiment of the present application;

Figure 3 is a schematic diagram of touch operations that can be transmitted between the existing Android system and Linux system;

Figure 4 is a schematic structural diagram of a smart tablet provided by an embodiment of the present application;

Figure 5 is a schematic diagram of a desktop application of a smart tablet provided by an embodiment of the present application;

Figure 6 is a schematic diagram of cross-system data transmission provided by an embodiment of the present application;

Figure 7 is an implementation flow chart of a touch input method provided by an embodiment of the present application;

Figure 8 is a specific implementation flow chart for running a preset application provided by an embodiment of the present application;

Figure 9 is a schematic diagram of an operation interface provided by an embodiment of the present application;

Figure 10 is a schematic diagram of different touch operations provided by an embodiment of the present application;

Figure 11 is a data flow diagram for converting different types of touch operations into operation instructions provided by an embodiment of the present application;

Figure 12 is an implementation flow chart of a method for edge gesture type touch input according to an embodiment of the present application;

Figure 13 is a schematic diagram of multiple edge gestures provided by an embodiment of the present application;

Figure 14 is a comparison chart of the responses of different window types to the same edge gesture provided by this application;

Figure 15 is an implementation flow chart of a method for multi-contact type touch input according to an embodiment of the present application;

Figure 16 is a schematic diagram of a multi-contact type touch operation provided by an embodiment of the present application;

Figure 17 is a schematic diagram of canceling response to a multi-contact operation provided by an embodiment of the present application;

Figure 18 is an implementation flow chart of a touch input method for clicking on any area in a preset application according to an embodiment of the present application;

Figure 19 is a schematic diagram of a cursor style provided by an embodiment of the present application;

Figure 20 is a schematic diagram of opening the soft keyboard provided by an embodiment of the present application;

Figure 21 is a schematic diagram of closing the soft keyboard provided by an embodiment of the present application;

Figure 22 is a structural block diagram of a touch input device provided by an embodiment of the present application;

Figure 23 is a structural block diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and/or components but does not exclude one or more other The presence or addition of features, integers, steps, operations, elements, components and/or collections thereof.

It will also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. ". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, to mean "once determined" or "in response to a determination" or "once the [described condition or event] is detected ]" or "in response to detection of [the described condition or event]".

In addition, in the description of this application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

Reference in this specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Therefore, the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in other embodiments", etc. appearing in different places in this specification are not necessarily References are made to the same embodiment, but rather to "one or more but not all embodiments" unless specifically stated otherwise. The terms “including,” “includes,” “having,” and variations thereof all mean “including but not limited to,” unless otherwise specifically emphasized.

The touch input method provided by the embodiments of this application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR)/virtual reality (VR) devices, notebook computers, and super mobile devices. On electronic devices such as personal computers (ultra-mobile personal computers, UMPCs), netbooks, and personal digital assistants (personal digital assistants, PDAs), in particular, the touch input method can be applied to tablet computers with touch functions. The embodiments of this application do not place any restrictions on the specific types of electronic devices.

For example, the electronic device may be a station (ST) in a WLAN, a cellular phone, a cordless phone, Session Initiation Protocol (SIP) phone, Wireless Local Loop (WLL) station, Personal Digital Assistant (PDA) device, handheld device with wireless communication capabilities, computing device or connected to Wireless modems, other processing devices, computers, laptops, handheld communication devices, handheld computing devices, and/or other devices for communicating over wireless systems and next generation communication systems, such as in 5G networks Mobile terminals or mobile terminals in the future evolved Public Land Mobile Network (Public Land Mobile Network, PLMN) network, etc.

FIG. 1 shows a schematic structural diagram of an electronic device 100 .

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.

It can be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors.

The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.

The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. Interfaces may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiver and transmitter (universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and /or universal serial bus (USB) interface, etc.

The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can separately couple the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the electronic device 100 .

The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 can be coupled with the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to implement the function of answering calls through a Bluetooth headset.

The PCM interface can also be used for audio communications to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to implement the function of answering calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. it will be The transmitted data is converted between serial and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100 . The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100 .

The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, display screen 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.

The USB interface 130 is an interface that complies with the USB standard specification, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through them. This interface can also be used to connect other electronic devices, such as AR devices, etc.

It can be understood that the interface connection relationships between the modules illustrated in the embodiment of the present invention are only schematic illustrations and do not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through the wireless charging coil of the electronic device 100 . While the charging management module 140 charges the battery 142, it can also provide power to the electronic device through the power management module 141.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters. In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.

The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.

The mobile communication module 150 can provide solutions for wireless communication including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.

A modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, adjust The modem and demodulation processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.

The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (bluetooth, BT), and global navigation satellites. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi) -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information. The above-mentioned display screen 194 can specifically display the generated detection report, so that the user can view the detection report through the display screen 194 .

The display screen 194 is used to display images, videos, etc. Display 194 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode). emitting diode (AMOLED), flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. Display 194 may include a touch panel as well as other input devices.

The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.

Camera 193 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other format image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

NPU is a neural-network (NN) computing processor. It draws on the structure of biological neural networks, such as The human brain transmits patterns between neurons, processes input information quickly, and can continuously learn by itself. Intelligent cognitive applications of the electronic device 100 can be implemented through the NPU, such as image recognition, face recognition, speech recognition, text understanding, etc.

The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. Such as saving music, videos, etc. files in external memory card.

Internal memory 121 may be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the stored program area can store an operating system, at least one application program required for a function (such as a sound playback function, an image playback function, etc.). The storage data area may store data created during use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.

The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .

Speaker 170A, also called "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A, or listen to hands-free calls. In particular, the above-mentioned speaker 170A can be used to output prompt information to notify the user of the parts that need to be contacted with the electronic scale.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a call or a voice message, the voice can be heard by bringing the receiver 170B close to the human ear.

Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with the human mouth and input the sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions, etc.

The headphone interface 170D is used to connect wired headphones. The headphone interface 170D may be a USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. For example, the electronic device can obtain the user's weight through the pressure sensor 180A. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor may include at least two parallel plates of conductive material. When a force is applied to pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is performed on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 may also calculate the touched position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch location but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

The gyro sensor 180B may be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of electronic device 100 about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device 100 shakes, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shake of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

Air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

Magnetic sensor 180D includes a Hall sensor. The electronic device 100 may utilize the magnetic sensor 180D to detect opening and closing of the flip holster. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 may detect the opening and closing of the flip according to the magnetic sensor 180D. Then, based on the detected opening and closing status of the leather case or the opening and closing status of the flip cover, features such as automatic unlocking of the flip cover are set.

The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices and be used in horizontal and vertical screen switching, pedometer and other applications.

Distance sensor 180F for measuring distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may utilize the distance sensor 180F to measure distance to achieve fast focusing.

Proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outwardly through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect when the user holds the electronic device 100 close to the ear for talking, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in holster mode, and pocket mode automatically unlocks and locks the screen.

The ambient light sensor 180L is used to sense ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touching.

Fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access to application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

Temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 utilizes the temperature detected by the temperature sensor 180J to execute the temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the low temperature from causing the electronic device 100 to shut down abnormally. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs boosting on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

Touch sensor 180K, also known as "touch device". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is used to detect a touch operation on or near the touch sensor 180K. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 at a location different from that of the display screen 194 .

Bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human body's vocal part. The bone conduction sensor 180M can also contact the human body's pulse and receive blood pressure beating signals. In some embodiments, the bone conduction sensor 180M can also be provided in an earphone and combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibrating bone obtained by the bone conduction sensor 180M to implement the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M to implement the heart rate detection function.

The buttons 190 include a power button, a volume button, etc. Key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100 .

The motor 191 can generate vibration prompts. The motor 191 can be used for vibration prompts for incoming calls and can also be used for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. The motor 191 can also respond to different vibration feedback effects for touch operations in different areas of the display screen 194 . Different application scenarios (such as time reminders, receiving information, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

The indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, or may be used to indicate messages, missed calls, notifications, etc.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 can support 1 or N SIM card interfaces, N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. This embodiment of the present invention takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100 .

Figure 2 is a software structure block diagram of the electronic device according to the embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers. From top to bottom, they are the application layer, the application framework layer, the system layer of the Android runtime (Android runtime), and the kernel layer.

The application layer can include a series of application packages.

As shown in Figure 2, the application package can include applications such as camera, calendar, map, WLAN, Bluetooth, music, video, short message, mailbox, WeChat, WPS, etc.

The application framework layer provides application programming interfaces (application programming) for applications in the application layer.

interface, API) and programming framework. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

A window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make this data accessible to applications. Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

The view system includes visual controls, such as controls that display text, controls that display pictures, etc. A view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

Telephone managers are used to provide communication functions of electronic devices. For example, call status management (including connected, hung up, etc.).

The resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc.

The notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text information is prompted in the status bar, a beep sounds, the electronic device vibrates, the indicator light flashes, etc.

Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.

The core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.

The application layer and application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and application framework layer into binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

The system layer can include multiple functional modules. For example: surface manager (surface manager), media libraries (Media Libraries), 3D graphics processing libraries (for example: OpenGL ES), 2D graphics engines (for example: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition and layer processing.

2D Graphics Engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

The following exemplifies the workflow of the software and hardware of the electronic device 100 in conjunction with capturing the photographing scene.

When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes touch operations into raw input events (including touch coordinates, timestamps of touch operations, and other information). Raw input events are stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation as a camera application icon control as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer. Camera 193 captures still images or video.

With the continuous development of device technology, mobile terminals (such as smart phones, smart tablets and other portable electronic devices) are becoming more and more powerful, and the types of applications that can be run are also increasing. Users can install different types of applications on smart terminals. The application can meet various needs in daily life and work, such as installing photo editing software on smartphones, beautifying and splicing images in the gallery, and editing captured videos on smartphones. , add background music, etc. to meet the needs of editing short videos anytime and anywhere. However, as users' demands for applications continue to increase, and users' demands for applications are also developing in the direction of specialization and complexity, applications developed based on mobile terminals can no longer meet the needs of users, and users need to use specialized For higher-level tasks, it is often necessary to use a computer. For example, when beautifying and editing images packaged in digital negative (RAW) format, you need to use the Lightroom application for processing, and the image editing function of the Lightroom application has only been developed for computers. On the computer side, there is no application suitable for mobile terminals, resulting in users being unable to complete editing of RAW images on mobile terminals, which greatly increases the difficulty of user operations. It can be seen that the computer application ecosystem based on mobile terminals needs to be rapidly improved to meet the application needs of today's users.

Although it is possible to expand the above-mentioned application ecology by developing applications for mobile terminals for different computer applications, due to the large number of applications involved and the complexity of computer-based application functions, the above approach requires It takes a lot of work. If the application developer does not develop applications suitable for mobile terminals, it will cause users to be unable to use a certain type of application on mobile terminals. The existing solution is to run emulators of other operating systems on the mobile terminal, and use the emulator to run applications that have been developed on other operating systems, thereby eliminating the need to re-develop corresponding applications for the mobile terminal.

For example, the operating system of the mobile terminal can be the Android system. In this case, the Linux system can be run on the mobile terminal through an emulator to expand the application ecosystem of the mobile terminal. In view of the fact that the current Linux system already has a relatively complete computer application ecosystem, and because the Linux operating system and the mobile terminal operating system (such as Android system) are both built based on the container structure, they are compatible and compatible with the Android system. Linux applications offer the possibility of implementation. However, because the modules of the Linux system are inconsistent with the modules on the mobile terminal, there is a certain difference between the way users operate the Linux system (mainly keyboard and mouse) and the way they operate the Android system (mainly touch). Difference, specifically, the Linux system is mainly based on keyboard and mouse operations, supplemented by touch operations, while the mobile terminal is mainly dominated by touch operations, supplemented by soft keyboard operations. Therefore, some touch operations on the mobile terminal It is invalid for the Linux system (that is, there is no corresponding touch gesture), which results in a large number of touch operations that cannot be recognized by the Linux system when users run applications based on the Linux system on mobile terminals. It reduces the user's operating efficiency and requires re-adaptation to the operating habits of the Linux system, which greatly increases the learning cost required for applications built based on the Linux system.

Exemplarily, FIG. 3 shows a schematic diagram of touch operations that can be transmitted between the existing Android system and the Linux system. As shown in Figure 3, the mobile terminal runs an Android system, and different types of touch operations can be initiated in the Android system, such as a click selection operation to select a certain display object, and a gesture operation initiated against the edge of the mobile terminal display screen. Or when clicking on the editable area, the text input operation of the soft keyboard is automatically evoked. However, the existing simulation technology of running the Linux system in the Android system cannot recognize the touch operation in the Android system. Of course, when running the Windows system on the Android system, there is also the problem that the above touch operations cannot be recognized.

It can be seen that there are transmission barriers for touch operations across systems. Touch operations cannot be mapped to another operating system. That is, a touch operation that can be recognized in the native operating system cannot be simulated to run another operation. System identification reduces the accuracy and flexibility of cross-system application operations, thereby reducing the user's operating efficiency and providing a poor user experience.

Example 1:

Therefore, in order to solve the problem of existing device control technology, when operating applications running across systems, the touch operations that can be recognized in the original operating system cannot be recognized by the application built based on another operating system. As a result, the user's To solve the problems of low operating efficiency and poor user experience, this application provides a touch input method. The touch input method is executed by an electronic device. The electronic device includes but is not limited to: a smart phone, Tablets, smart watches, computers, laptops and other electronic devices that can receive touch operations initiated by users. For ease of description, in the following embodiments, the electronic device is described using a smart tablet as an example. If the electronic device is a smartphone or other electronic device, the implementation method is exactly the same as that of the smart tablet. You may refer to the specific implementation process of the smart tablet. .

The smart tablet is equipped with a touch screen, can receive touch operations initiated by the user through the touch screen, and realize conversion and transmission of the received touch operations through the built-in first touch module, so that based on another operation The preset application built by the system identifies and performs corresponding response operations.

Exemplarily, FIG. 4 shows a schematic structural diagram of a smart tablet provided by an embodiment of the present application. As shown in FIG. 4 , the smart tablet provided by this embodiment includes at least the following layers: application layer, application framework layer, system layer, hardware abstraction layer, kernel layer and physical layer.

The above-mentioned application layer is specifically used to run various applications, including system-built-in applications (such as system settings) and custom-installed applications. Among them, the above-mentioned application layer includes a proxy application, which is used to realize data interaction between the first operating system (i.e., the native operating system of the smart tablet) and the second operating system (i.e., another operating system running in simulation), For example, a preset application built based on the second operating system can be transmitted through the above-mentioned proxy application. The above-mentioned cross-system application data transmission process specifically includes: through applications that support the displayable X11 protocol and wayland protocol, the window, display After data and other related information are processed by the display module in the second operating system (such as Linux system), the display data is transmitted to the above-mentioned proxy application, and then the proxy application calls the application framework layer in the first operating system (such as Android system). The relevant module completes the display, thereby being able to generate an application window for cross-system applications within the first operating system and the ability to display images in the window.

The agent application is specifically responsible for interface display, window management, application startup and other related functions of applications running across systems, and specifically includes the following modules: main execution module, application execution module, and service module. Among them, the main execution module is responsible for generating desktop icons based on cross-system applications in the desktop, such as Linux desktop icons. Exemplarily, FIG. 5 shows a schematic diagram of a desktop application of a smart tablet provided by an embodiment of the present application. As shown in Figure 5, the desktop of the smart tablet displays all executable applications, such as library application 51, memo application 52, etc., and also displays applications based on other operating systems, such as document editing application 53. The icon generation of the document editing application 53 can be generated through the main execution module in the above-mentioned proxy application. Optionally, in order to distinguish cross-system applications from applications based on the original operating system, a mark, namely mark 54, can be configured in the lower left corner of the application icon of the above-mentioned cross-system application, so that it can be conveniently used to distinguish applications based on different operating systems. . The application execution module is specifically used to generate a window interface of an application program built based on the second operating system; and the service module is specifically used to manage and display images of the above-generated window interface.

The above-mentioned application framework layer includes an application framework for basic functions in the smart tablet, namely Framework 1, and an extension framework for implementing extended functions, namely Framework 2. The above-mentioned native application framework contains different system modules, such as wireless communication module, Bluetooth communication module, multimedia playback module, etc. A multi-window display module can be configured in the extension frame to enable application windows of different applications to be displayed in the same interface of the smart tablet at the same time; the extension frame also includes a first touch module, which is specifically used to realize cross-platform display. Recognition and response of touch operations between systems.

At the same time, the second operating system can be run through the image container, that is, the second operating system is built in the image container. The simulated operating system also includes an application layer, an application framework layer, and a system layer. It should be noted that the above-mentioned image container running the second operating system can be deployed in the cloud server or locally on the smart tablet.

Exemplarily, FIG. 6 shows a schematic diagram of cross-system data transmission provided by an embodiment of the present application. Referring to (a) in Figure 6, the image container running the second operating system is deployed on the smart tablet. In this case, the above-mentioned image container and the first operating system of the smart tablet share the kernel layer in the smart tablet and The physical layer is used to complete the data transmission between the two operating systems through the proxy application. That is, in this case, the first operating system and the second operating system are both installed in the same electronic device. Referring to (b) in Figure 6, the image container running the second operating system is deployed on the cloud server, that is, the first operating system and the second operating system are respectively installed in different electronic devices. In this case, When realizing cross-system data transmission, the required data can be generated through the proxy application in the first operating system and sent to the cloud server through the physical layer in the smart tablet to be received by the mirror container in the cloud service location; Correspondingly, the data generated in the image container can also be sent through the cloud server. to the smart tablet, and is received by the physical layer in the smart tablet and passed to the agent application in the smart tablet, thus realizing cross-system data transmission.

Wherein, the application framework layer of the image container running the second operating system is configured with a second touch module. The second touch module can receive the information sent by the first touch module and convert it into information that can be processed based on the second touch module. Operation instructions identified by applications built by the operating system; the application layer in the image container runs applications built based on the second operating system, that is, subsequent default applications, and the relevant operation windows can be applied to the first operating system through a proxy Internal display enables cross-system application operations and expands the application ecosystem within the smart tablet.

Specifically, based on the structure of the smart tablet mentioned above, the specific implementation process of the touch input method provided by the embodiments of the present application is described in detail below. Figure 7 shows an implementation flow chart of a touch input method provided by an embodiment of the present application. The details are as follows:

In S701, the first operating system receives the user's touch operation on the preset application; the preset application is built based on the second operating system;

In S702, in response to the touch operation, the first operating system sends the touch event generated based on the touch operation to the second operating system through the first touch module in the first operating system.

In this embodiment, the smart tablet is built based on the first operating system, that is, the first operating system is specifically the native operating system of the smart tablet. In a possible implementation, the first operating system is specifically the Android operating system. , Hongmeng operating system, Linux operating system or windows operating system. Of course, the first operating system can also be other operating systems that support touch operations, and the first operating system is not limited here.

In this embodiment, the smart tablet displays application icons corresponding to the default applications built based on other operating systems (ie, the second operating system). The user can click the application icons to start the above-mentioned default applications. Continuing to illustrate with FIG. 5 , the above-mentioned document editing application 53 is an application built based on the Linux system (ie, the above-mentioned default application). The user can click the above-mentioned document editing application 53 to start the default application. After detecting that the user needs to launch a preset application that runs across the system, the smart tablet can generate an operation interface corresponding to the preset application, and the user can initiate a touch operation on the preset application in the operation interface.

Further, as another embodiment of the present application, FIG. 8 shows a specific implementation flow chart of running a preset application provided by an embodiment of the present application. Referring to Figure 8, compared with the embodiment shown in Figure 7, the embodiment of the present application also includes S801~S804 before S701. The specific description is as follows:

In S801, the first operating system receives a user-initiated startup operation for the preset application.

In this embodiment, different types of applications are installed in the smart tablet, including default applications built based on another operating system. As mentioned above, there is a proxy application running in the application layer of the smart tablet. The main execution module of the proxy application can generate the application icon of the preset application in the desktop of the smart tablet, such as the document editing application 53 in Figure 5. When the smart tablet detects that the user clicks on the above-mentioned document editing application 53, it recognizes that the user initiated the above-mentioned startup operation.

In S802, the first operating system sends an application startup instruction to the second operating system through the proxy application installed in the first operating system; the application startup instruction carries the application identifier of the default application.

In this embodiment, a proxy application is installed in the first operating system of the smart tablet. After receiving the startup instruction for the default application, the proxy application can send information about the default application to the container where the second operating system is installed. Start the instruction to run the above-mentioned preset application in the container of the second operating system. It should be noted that if the container is configured in a smart tablet, the agent application can send the above-mentioned startup command to the above-mentioned container running the second operating system through the common kernel layer; if the container is configured in a device other than the smart tablet, Other devices (such as cloud servers) generate communication data packets carrying the communication address of the cloud server and send them to the cloud server through the established communication link with the cloud server to notify the cloud server to run the above-mentioned preset application. .

In this embodiment, the startup command sent by the proxy application carries the application identifier of the default application that needs to be started. The application identifier can be the application name of the default application (such as Ligtroom, Photoshop CS4, etc.), or it can be After receiving the application number of the preset application (such as LR10245, PS23568, etc.), the container running the second operating system can extract the application ID contained in it and compare it with the application ID of each existing local application. Matching is performed, and the existing application corresponding to the application identifier associated with the startup instruction is identified as the above-mentioned default application.

In a possible implementation, the above-mentioned application identification can also be coordinate information in the first operating system. Since the application icon of the default application in the first operating system is generated by the proxy application, the proxy application has Created application icon The association relationship with the default application associated in the second operating system, the association relationship may record the icon position of the default application in the first operating system. In this case, the above-mentioned startup command may carry the coordinate information clicked by the user, and match the coordinate information with the icon position in the above-mentioned association recorded in the container running the second operating system, and match the clicked coordinate position with the icon position recorded in the container running the second operating system. The application associated with the matching icon position is identified as the above-mentioned default application.

In S803, the second operating system responds to the application startup instruction, runs the preset application, and feeds back the operation interface of the preset application to the proxy application.

In this embodiment, after the container running the second operating system generates a startup instruction for the default application, it can obtain the application data package corresponding to the default application, and use the application data package to start the above-mentioned steps in the container. The preset application renders the operation interface of the preset application through the application framework layer in the container, and feeds the operation interface back to the proxy application to realize the display of the cross-system operation interface.

In a possible implementation, the container running the second operating system can convert the operation interface into a corresponding interface image, and send the interface image to the proxy application, that is, the preset generated within the first operating system. The operation interface of the application is specifically displayed in image format.

In S804, the first operating system generates the operation interface in the first operating system through the proxy application.

In this embodiment, after receiving the operation interface fed back by the container running the second operating system, the agent application running in the first operating system can render it in the first operating system of the smart tablet, so that the user can The operation interface for initiating touch operations.

Exemplarily, FIG. 9 shows a schematic diagram of an operation interface provided by an embodiment of the present application. Referring to (a) in Figure 9, the operation interface is specifically a full-screen operation interface. The operation interface is generated based on a preset application running in a container running a second operating system, and is generated through a proxy application. Rendering is performed in the first operating system. Specifically, the operation interface can be an image data, that is, each of the above-mentioned operable controls does not have a corresponding interactive interface in the operation interface, but needs to be fed back to the operation interface through a proxy application. Only the preset application running in the container can respond to the above interactive operations.

As shown in (b) of FIG. 9 , the operation interface may also be a non-full-screen display operation interface, such as displaying the operation interface in a floating window. The specific method used to display the operation interface will be determined according to the specific preset application and is not limited here.

In the embodiment of the present application, cross-system data transmission is realized through the proxy application, and the operation interface of the preset application feedback running through the container is received through the proxy application, thereby realizing the display of the cross-system operation interface and providing convenience for subsequent users in the third application. Initiating a touch operation for cross-system applications within an operating system provides a basis for implementation, and the display process of the above-mentioned operation interface is consistent with the display effect of the operation interface of general applications, so that the above-mentioned cross-system application startup is convenient for users. It is non-perceptual and greatly improves the consistency of user operations, thereby improving the user experience.

In this embodiment, the smart tablet is equipped with a touch module, and the user can initiate touch operations in the touch module. The above touch operations include touch operations of the click selection type, and other touch operations except the click selection type. , such as edge gesture type touch operations, multi-touch touch operations, and character input operations in the editable area through the soft keyboard.

Exemplarily, FIG. 10 shows a schematic diagram of different touch operations provided by embodiments of the present application.

1. Click to select the type of touch operation. Referring to (a) in Figure 10, the user can click to select the display object of the operation window in the application, such as the virtual button in the operation interface and open the sub-interface. For example, the user can click to select the display object in the operation window. Start control 101 to start the menu bar 102 corresponding to control 101.

2. Edge gesture type touch operation. The smart tablet records a variety of smart gestures, which can facilitate the operation of the smart tablet and improve the user's operating efficiency of the smart tablet. For example, as shown in (b) of Figure 10, the user can initiate a sliding operation from the left edge of the smart tablet screen as the starting point, and the corresponding operation command of the sliding operation starting from the edge is "return". Or "Close the current operation window", etc. You can respond to different operation instructions according to the actual situation. By initiating the same gesture as the preset recording rule, you can operate the application without clicking the corresponding control in the operation window, which improves Flexibility of user operations.

3. Multi-touch touch operation. The touch module of the smart tablet supports multi-touch, as shown in (c) in Figure 10. The user can zoom in and out of the display objects in the application through two-finger touch.

4. Input characters into the editable area through the soft keyboard. Since smart tablets generally do not have peripheral input devices, For example, a keyboard or remote control needs to display a soft keyboard when inputting characters. Therefore, the smart tablet can determine whether the soft keyboard needs to be evoked based on the type of area that the user clicks. As shown in (d) in Figure 10, when the user clicks on an editable area, such as the content input area on the conversation page, it will Turn on the soft keyboard, and users can complete character input through the soft keyboard.

It can be seen that the touch operations that can be initiated in a smart tablet are often related to the operating system of the smart tablet and the operating characteristics of the tablet, and some touch operations cannot be directly mapped in other operating systems, such as the above-mentioned edge Gesture-type touch operations are not applicable to devices built on Linux systems and devices built on Windows systems. Since the devices of the above two operating systems are generally controlled by keyboard and mouse, they often do not have corresponding touch screens. , and of course there is no edge touch type operation. Based on this, in order to achieve interoperability and mutual understanding of touch operations between different operating systems, a first touch module is configured in the first operating system, and the smart tablet receives a touch initiated by the user based on a preset application running across the system. During operation, the touch operation can be handed over to the first touch module for processing to generate a corresponding touch event.

In S703, the second operating system generates an operation instruction corresponding to the touch event through the second touch module.

In this embodiment, after generating a touch event, the first touch module in the smart tablet can transmit the touch event to the second touch module configured in the image container running the second operating system. The touch module is specifically used to identify touch events generated in the first operating system and convert them into operation instructions that can be recognized by the second operating system.

In this embodiment, the smart tablet is configured with a proxy application. When the first touch module arranged in the application framework layer generates a touch event, the touch event can be transmitted to the proxy application located in the application layer. The proxy application transmits the above touch event to the second touch module in the container.

In S704, the second operating system executes the operation instruction through the preset application.

In this embodiment, since the operation instructions generated by the second touch module are instructions that can be recognized by the second operating system, the default application built based on the second operating system can also respond to the operation instructions and execute the operation instructions in the second operating system. Generate corresponding response results, such as returning to the previous level interface, closing the default application, adjusting the display content, etc. The specific response results can be determined according to the initiated touch operation, and are not limited here.

Example 2:

Among them, for different types of touch operations, when the first touch module and the second touch module convert the touch operations into corresponding operation instructions, there are certain differences in the specific processing procedures and the calling of units in the modules. difference. Exemplarily, FIG. 11 shows a data flow diagram for converting different types of touch operations into operation instructions provided by an embodiment of the present application. Referring to Figure 11, the above-mentioned first touch module contains a plurality of different sub-units, and correspondingly the second touch module also contains different sub-units, and different sub-units are used to implement different types of touch. Operational transformation. The above data flow diagram specifically includes three data flows, namely data flow 1 for the first touch operation of the edge gesture type, data flow 2 for the second touch operation of the multi-touch type, and data flow 2 for the preset application. Data stream 3 for the third touch operation of clicking on any area. The specific implementation process is described as follows:

Type 1: First touch operation for edge gesture type.

Referring to FIG. 12 , FIG. 12 shows an implementation flow chart of a method for edge gesture type touch input according to an embodiment of the present application. Referring to Figure 12, compared with the embodiment shown in Figure 7, in the touch input method provided by the embodiment of the present application, S702 is specifically S1201, S703 is specifically S1202, and S704 is specifically S1203. The specific description is as follows:

In S1201, the first touch module determines the window type of the target window, and generates a first touch event corresponding to the first touch operation according to the window type; the target window is the first Touch operation operates in the preset application window; the first touch event includes: the window type of the target window and the operation type corresponding to the first touch operation.

In this embodiment, the edge gesture initiated in the smart tablet is specifically used to operate the entire operation window of the application running in the foreground, such as returning to the interface or exiting the application.

In this embodiment, the above-mentioned operation types specifically refer to the operations corresponding to edge gestures. The above-mentioned operation types include but are not limited to: closing windows, exiting applications, or performing application management and other different types of operations. The specific operations can be set according to the actual situation of the user. , which is no longer limited here.

Exemplarily, FIG. 13 shows a schematic diagram of multiple edge gestures provided by an embodiment of the present application. Referring to (a) in Figure 13, the user can start sliding from the right edge of the smart tablet screen to generate a sliding instruction from right to left. Different from the left sliding instruction initiated within the screen, the above sliding instruction is Starting from the right edge of the screen, the smart tablet will recognize the sliding command as an edge gesture touch operation and query the operation command associated with the touch operation. In this example, the left sliding touch on the right edge The corresponding operation is the "exit" operation command. Referring to (b) in Figure 13, the user can start sliding from the lower edge of the smart tablet screen to generate a sliding instruction from bottom to top. Different from the upward sliding instruction initiated within the screen, the above sliding instruction is based on the screen. The lower edge is the starting point, so the smart tablet will recognize that the sliding command is for application window management, as shown in (c) in Figure 13. The user can close the corresponding application through this application window management, or switch to other applications. Perform operations. Of course, the operation instructions corresponding to the edge gestures can be set according to actual conditions, and the operation instructions corresponding to the edge gestures are not limited here.

In this embodiment, since edge gestures are often targeted at a specific operation window, before converting an edge gesture type operation instruction, the operation window operated by the operation instruction needs to be determined first. Since the operation instructions corresponding to the same edge gesture will be different for different window types, in order to improve the accuracy of touch operations, the smart tablet first needs to determine the target window for the required operation, and then determine the window type corresponding to the target window.

In a possible implementation, the above window types include: parent window types and child window types. Each preset application corresponds to an operation window of the parent window type, and the parent window type can contain multiple page jump controls. The user can click on any page jump control in the operation interface of the parent window type to enter The operation interface of the corresponding sub-window type; similarly, the operation interface of the sub-window type can also include a page jump control to enter the secondary operation window, and so on. Among them, the window type of the secondary operation window also belongs to the sub-window type. Window type. Different window types respond differently to the same edge gesture.

By way of example, Figure 14 shows a comparison chart of responses of different window types to the same edge gesture.

Referring to (a) in Figure 14, the window type of the target window that the user needs to operate is specifically a sub-window type. For example, a locally stored document is opened in a text editing application, and the document has not been modified. At this time, the user initiates a left-swipe operation starting from the right edge, and the smart tablet recognizes the left-swipe operation as the first touch operation of the edge gesture type. At this time, it recognizes that it is necessary to exit the target window and return to the previous level. window, therefore the above target window will be closed and the parent operation window corresponding to the default application will be returned (which can also be called the main operation interface).

Referring to (b) in Figure 14, the target window for the user's operation is still a sub-window in the text editing application (that is, the target window type is a sub-window type), but in this example, the user has already edited the document For some content, when initiating a touch operation by edge gesture, you need to return to the previous window. However, since the above document has not been saved, a save prompt box will be generated to confirm whether the user needs to save the edited content. , if you click the save control 141, you will return to the main operation interface; if you click the cancel control 143, you will return to the target window for document editing; if you click the do not save control 142, the edit will not be saved and you will return to the main operation interface.

Referring to (c) in Figure 14, the window type of the target window that the user needs to operate is specifically the parent window type. In this case, when a left sliding operation is initiated from the right edge as the starting point, the window will not exit immediately. The application will prompt the user to exit again. At this time, the user can initiate the above-mentioned left sliding operation starting from the right edge again, and the above-mentioned application will exit and return to the desktop interface of the smart tablet.

It can be seen that for different window types, when receiving the same edge gesture, the corresponding response methods are different. Based on this, the smart tablet needs to preset the window type corresponding to the target window of the current operation of the application, and query the edge touch The response method associated with the operation (such as the above-mentioned exit page, or application window management, etc.) and window type are encapsulated to generate the corresponding first touch event. Among them, the above response mode can be determined based on the corresponding relationship between the locally recorded touch gestures and the response mode. Certainly.

Further, as another embodiment of the present application, the implementation of determining the window type specifically includes the following steps:

In S1201.1, the touch operation initiated by the user is monitored through the touch management unit in the first touch module.

In S1201.2, if the touch management unit identifies that the touch operation is the first touch operation, the touch window management unit in the first touch module is called to determine the target. The window type of the window; the touch window management unit is used to establish a cascade relationship between each operation window in the preset application.

In this embodiment, the first touch module includes a touch management unit and a touch window management unit. Among them, the touch management unit is specifically used to monitor touch operations initiated in the first operating system, and after detecting that the user initiates a touch operation, determine the type of the touch operation and forward it to the corresponding unit for processing. deal with.

In this embodiment, when the touch management unit recognizes that the touch operation initiated by the user is a first touch operation of the edge gesture type, it will determine the response mode corresponding to the first touch operation, and assign the first touch operation to the first touch operation. The touch operation is sent to the touch window management unit to determine the window type of the target window of the current operation. Since the touch window management unit maintains the cascading relationship of each operation window in the application during the process of running the preset application, that is, it is used to determine whether each window belongs to the parent window type or the child window type. Based on this, the touch window The management unit can determine the window type of the target window through the cascade relationship. After determining the response mode and window type, the management unit can generate a first touch event corresponding to the threshold and send it to the second touch unit.

In the embodiment of the present application, the touch management unit identifies the touch operation type and forwards it to the corresponding unit for subsequent response, which can improve the accuracy of touch event generation.

In S1202, a first operation instruction for the target window is generated through the window management unit in the second touch module.

In this embodiment, the second touch window is configured with a window management unit for managing operation windows of each default application running in the second operating system. Therefore, after the second touch module receives the first touch event sent by the first touch module and detects that the first touch event is an edge gesture touch event, it will send the touch event to The window management unit is received, and the first operation instruction corresponding to the first touch event is generated through the window management unit, so that the preset application performs the corresponding operation.

In S1203, the second operating system executes the operation instruction through the preset application.

Since the implementation of S1203 is exactly the same as the implementation of S703, please refer to the relevant description of S703 for specific description, which will not be described again here.

In the embodiment of the present application, since different window types have different response modes, when responding to an edge gesture type touch operation, first determining the window type of the target window for the required operation can improve the accuracy of subsequent window operations. This makes the operation of cross-system preset applications consistent with the operation habits of native applications built based on the first operating system, reducing the learning cost required for user operations.

Type 2: Second touch operation for multi-touch type.

Referring to FIG. 15 , FIG. 15 shows an implementation flow chart of a multi-contact type touch input method according to an embodiment of the present application. Referring to Figure 15, compared with the embodiment shown in Figure 7, in the touch input method provided by the embodiment of the present application, S702 is specifically S1501, S703 is specifically S1502, and S704 is specifically S1503. The specific description is as follows:

In S1501, the first touch module generates a multi-touch event corresponding to the second touch operation, and transmits the multi-touch event to the second touch module. The multi-touch event includes the first touch coordinates and touch time of each touch point in the first operating system.

In this embodiment, under normal circumstances, Android system devices often support multi-touch touch operations, such as image transformation operations such as zooming in, zooming out, and rotating images through two fingers, or long pressing and dragging. Move display objects in the page, etc. Therefore, after the user initiates a multi-touch type touch operation on the smart tablet, the first touch module can generate a corresponding multi-touch event, and the multi-touch event can include the touch operation initiated this time. The first touch coordinates corresponding to each corresponding touch point (that is, the corresponding coordinates in the first operating system), the first touch module will send it to the second touch module for subsequent processing after generating a multi-touch event. deal with.

In this embodiment, each touch point can be connected in sequence according to the first touch coordinates and touch time corresponding to different touch points, thereby forming a touch trajectory corresponding to the second touch operation.

Exemplarily, FIG. 16 shows a schematic diagram of a multi-contact type touch operation provided by an embodiment of the present application. Referring to (a) in Figure 16, an image is displayed in the operation interface of the default application. The user can move the two fingers outward to position the image. big. Referring to (b) in Figure 16, a document is displayed in the operation interface of the default application. Since the document contains a lot of content, a progress bar is configured in the document. The user can drag the progress bar to view the corresponding paragraphs in the document, for example Swipe up to view the content of the next paragraph in the document. It should be noted that multi-contact operations have different response methods according to different applications. The specific settings are based on the actual situation and are not limited.

Further, as another embodiment of the present application, specific ways of generating multi-touch events include:

In S1501.1, monitor the touch operation initiated by the user through the touch management unit in the first touch module;

In S1501.2, if the touch management unit recognizes that the touch operation is a second touch operation, the touch event conversion unit in the first touch module processes the second touch operation. Perform legal verification of control operations.

In S1501.3, if it is recognized that the second touch operation passes the legal verification, a multi-touch event is generated through the touch event conversion unit, and the multi-touch event is sent to The second touch module; the multi-touch event records the first touch coordinates of each contact point in the first operating system.

In this embodiment, the first touch module includes a touch management unit and a touch event conversion unit. Wherein, the above-mentioned touch management unit is used to monitor the touch operation initiated in the first operating system, and after detecting that the touch operation is a second touch operation of multi-contact type, the second touch operation is Forwarded to the touch event conversion unit for subsequent processing.

In this embodiment, after receiving the above-mentioned second touch event, the touch event conversion unit will first verify its legality, that is, whether this multi-touch out operation is a valid operation. If the touch event conversion unit detects that the second touch type passes legal verification, it can convert it into a multi-touch event that can be recognized by the second operating system. The multi-touch event may include first touch coordinates of each touch point in the first operating system.

In the embodiment of the present application, the touch event conversion unit performs legal verification on the touch operation and performs event conversion, thereby generating multi-touch events that can be recognized by other operating systems, thereby improving multi-touch touch control. Operational accuracy reduces responses to invalid operations.

Further, as another embodiment of the present application, after S1501.3, it may also include:

In S1501.4, if it is recognized through the touch event conversion unit that the second touch operation is not an operation for the preset application, send information about the multi-touch event to the second touch module. cancel response instruction to stop the default application from executing the operation instruction related to the second touch operation.

In this embodiment, the above-mentioned touch event conversion unit is also used to cancel an abnormal touch operation. Since the user initiates a second touch operation with multiple touch points, it takes a certain amount of operation time. Therefore, it is possible that the user has not completed the drawing. When the entire sliding trajectory is drawn, the touch event conversion unit has already sent the previously collected multi-touch event to the second touch event. After the entire sliding trajectory is drawn, the sliding trajectory is consistent with the preset shortcut gesture, that is, The operation correspondence is not for the default application. At this time, the default application needs to be stopped from responding to the second touch operation. In this case, the touch event conversion module in the first touch module can generate a cancellation response. command, and sends the cancel response command to the second touch module. After the second touch module receives the cancel response command, it stops the default application from responding to the second touch operation.

Exemplarily, FIG. 17 shows a schematic diagram of canceling response to a multi-touch operation provided by an embodiment of the present application. Referring to (a) in Figure 17, the user initiates a multi-touch touch operation similar to sliding downwards. At this time, the touch event conversion unit performs a complete verification on the above touch trajectory and completes the verification. After passing the verification, the corresponding multi-touch event is generated and sent to the second touch module. At the same time, the user continues to perform sliding operations in the first operating system, and obtains a sliding trajectory as shown in (b) in Figure 17, specifically a circular sliding trajectory, and the circular sliding trajectory corresponds to the improvement of The display brightness of the smart tablet (that is, not for the default application, but for the screen of the smart tablet). At this time, the touch event conversion unit will generate a cancellation response command and send the cancellation response command to the second touch module to stop the default app from responding to the circular sliding trajectory.

In S1502, the second touch module performs coordinate transformation on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system.

In this embodiment, although the first operation interface and the second operation interface can share the same kernel, the operation interface can be adjusted according to the actual display interface when rendering. Therefore, the first touch is determined in the first operating system. There is a certain deviation between the control coordinates and the coordinates in the second operating system, and coordinate conversion needs to be performed through the second touch module, so that the generated second operation instructions can accurately execute the preset application running in the second operating system. Operate on the specified content, and obtain the second operation instruction based on the converted coordinates.

Further, as another embodiment of the present application, the above-mentioned specific process of generating the second operation instruction includes:

In S1502.1, the multi-touch event is completely verified through the multi-touch management unit in the second touch module.

In S1502.2, if it is recognized through the multi-touch management unit that the multi-touch event passes the complete verification, then the touch window conversion unit in the second touch module Each of the touch points in the multi-touch event performs coordinate conversion to determine the second touch coordinates corresponding to each of the first touch coordinates of the preset application running in the container; the second touch The coordinates are based on the size of the first window of the preset application in the first operating system, the size of the second window corresponding to the preset application in the second operating system, and the first touch The coordinates are determined; the first window is a mapping window for the second window under the first operating system.

In S1502.3, the touch window conversion unit determines the target window for the required operation in the preset application, and generates a second operation instruction acting on the target window based on the second touch point coordinates.

In this embodiment, the second touch module includes a multi-touch management unit. The multi-touch management unit is specifically used to manage touch input events of each application running in the second operating system. The above touch input events support multiple Contact processing, and integrity verification of the above touch input events. Therefore, after the second touch module receives the multi-touch event sent by the touch event conversion unit, it can first perform an integrity check through the multi-touch management unit. After detecting the multi-touch event, the corresponding The coordinate transformation is performed only after the sliding trajectory is complete.

In a possible implementation, after passing the complete verification, the above-mentioned multi-touch management unit can also determine the application operated by the multi-touch event, because in the first operating system, multiple windows can be used simultaneously Multiple different applications are run. Therefore, on the second operating system side, a corresponding unit is also needed to determine the preset application corresponding to the multi-touch event. Specifically, the method of determining the preset application can be based on the area where each touch point falls, and which application window the area is, so that the application to which the window where the touch point falls can be identified as Default application.

In a possible implementation, the multi-touch event carries the application identifier of the preset application for the required operation, and the multi-touch management unit can extract the application identifier from the multi-touch event to determine Default applications that run within the second operating system.

In this embodiment, after determining the integrity of the multi-touch event, the multi-touch management unit will send the multi-touch event to the touch window conversion unit. The touch window conversion unit can obtain the multi-touch event from the multi-touch event. Extract the first touch coordinate contained in the touch event, and determine the size of the second window corresponding to the default application when running in the second operating system, and then determine the size of the first window corresponding to the default application in the first operating system. The size of the window and the size of the above-mentioned second window can determine the offset of the coordinates between different operating systems, and adjust each first touch coordinate based on the offset. Finally, the position of each touch point in the second operating system can be calculated. The corresponding coordinates of the second contact point within.

In this embodiment, in addition to implementing coordinate conversion, the touch window conversion unit is also used to determine the target window for the required operation in the preset application, and transmit the second operation instruction based on the coordinates of the second contact point of the multiple contacts. Give the corresponding target window, so that the corresponding target window in the preset application executes the above second operation instruction.

In the embodiment of the present application, coordinate conversion and window confirmation are implemented through the touch window conversion unit, and the application program for the required operation is determined through the multi-touch management unit, thereby improving the accuracy of the touch operation.

In S1503, the second operating system executes the operation instruction through the preset application.

Since the implementation method of S1503 is exactly the same as the implementation method of S703, please refer to the relevant description of S703 for specific description, which will not be described again here.

In the embodiment of the present application, the touch coordinates are converted through the second touch module, so that the touch operation can also accurately operate the preset applications running across the system, thereby realizing multi-touch type operations. It can realize cross-system transmission and identification, improving the usable scope of cross-system operations.

Type 3: The third touch operation of clicking on any area in the preset application.

Referring to FIG. 18 , FIG. 18 shows an implementation flow chart of a touch input method for clicking on any area in a preset application according to an embodiment of the present application. Referring to Figure 18, compared with the embodiment shown in Figure 7, in the touch input method provided by the embodiment of the present application, S702 is specifically S1801, S703 is specifically S1802, and S703 is specifically S1803. After S703, Including S1804, the specific description is as follows:

In S1801, the first touch module generates a cursor displacement touch event corresponding to a third touch operation; the cursor displacement touch event includes a third touch coordinate corresponding to the touch operation.

In this embodiment, the user can click to select the corresponding page control in the operation interface corresponding to the preset application, and can also click to input characters into the editable area. Therefore, in order to distinguish the above two different operation types, the first An operating system can generate the above-mentioned cursor displacement touch event through the first touch module. The cursor displacement touch event includes the coordinates of the user's click in the operation interface corresponding to the default application, that is, the above-mentioned third touch event. coordinate.

In this embodiment, since the operation interface for the default application generated in the first operating system through the proxy application is specifically an image data, that is, there is no corresponding operable control in the first operating system. For area type The identification needs to be determined by the container running the second operating system, that is, the first touch module needs to determine the position of this click operation (ie, the third touch coordinate) and generate the corresponding cursor displacement touch event.

Further, as another embodiment of the present application, generating the above-mentioned cursor displacement touch event specifically includes the following steps:

In S1801.1, the touch operation initiated by the user is monitored through the touch management unit in the first touch module.

In this embodiment, the first touch module includes a touch management unit, wherein the touch management unit is used to monitor the touch operation initiated in the first operating system, and detect that the touch operation is the third After the touch operation, the third touch operation is forwarded to the touch event conversion unit for subsequent processing.

In S1801.2, if the touch management unit identifies that the touch operation is a third touch operation, the touch event conversion unit in the first touch module The operation is checked for legality.

In S1801.3, if the third touch operation passes the legal verification through the touch management unit, a click recognition event is generated through the touch event conversion unit, and the click recognition event is generated. The event is sent to the input touch detection unit.

In this embodiment, the above-mentioned first touch module further includes: a touch event conversion unit and an input touch detection unit, the touch event conversion unit is specifically used to convert into a touch event input that can be recognized by the second operating system; The input touch detection unit is specifically used to identify the specific click type of the click operation initiated by the user.

In this embodiment, after the touch management unit forwards the third touch operation to the above-mentioned touch event conversion unit, the touch event conversion unit may perform legality verification on the third touch operation. If the third touch operation does not pass the legal verification, the first operating system may not respond to the third touch operation; conversely, if the third touch operation is a valid touch operation, the first operating system may respond to the input touch operation. The control detection unit identifies the specific click type of the click operation initiated by the user on any area in the preset application.

In S1801.4, in response to the click recognition event, the click type corresponding to the third touch operation and the third touch coordinate are identified through the input touch detection unit, and based on the click type And the cursor displacement event generated by the third touch coordinate is sent to the second touch module.

In this embodiment, different click types correspond to response modes. The above click types include but are not limited to: click operations such as single click, double click, and long press. For single-click click operations, it can be used to select a display object in a preset application, or when clicking on an editable area, it means that text input is required in this area; for double-click click operations, it may be It is used to cache and select the text content corresponding to the area; for long-press type click operations, it can be used to pop up the corresponding text editing menu to realize operations such as copying, pasting, and deleting the selected text. It can be seen that different click operations can correspond to different response modes (that is, different operation instructions can be generated). Based on this, after receiving the above-mentioned third touch operation, the first operating system needs to determine its operation through the touch detection unit. The corresponding click type is generated, and a cursor movement event is generated according to the click type and the corresponding click position (ie, the third touch coordinate), and the cursor movement event is sent to the second touch module.

In this embodiment, since the operation interface returned by the proxy application about the default application can be specifically an image data, that is, different areas are displayed in the form of pixels in the first operating system, and the area type of each area cannot be determined. , Based on this, the first operating system needs to use the second operating system to determine the area type of the click area, for example, determine whether the area clicked by the user is an editable area or a non-editable area, so it needs to send the cursor movement event to the second operation The system's container performs subsequent identification operations.

In the embodiment of the present application, complete identification is performed through the touch management unit in the first touch module. When it is determined that the third touch operation has passed the verification, the touch detection unit is called to determine the status of the third touch operation. Click type and third touch coordinates, and then generate a corresponding cursor displacement event, and send the cursor displacement event to the second touch module for subsequent recognition processing, so that when providing interface feedback, the second operating system is running The container only needs to send the image of the operation interface to the first operating system. On the basis of ensuring that the user can smoothly select the controls contained in the operation interface, it reduces the amount of data that needs to be transmitted during the cross-system application operation process, thereby improving Improves the response speed of cross-system application operations.

In S1802, the second touch module identifies the area type corresponding to the third touch coordinate in the preset application, and generates a third operation instruction according to the area type.

In this embodiment, after the first touch module sends the cursor displacement event to the second touch module, the second touch module can extract the third touch coordinates from the above-mentioned cursor displacement event and use it in the second operating system The click area corresponding to the third touch coordinate is determined in the default application in the click area, and then the area type corresponding to the click area can be determined, such as whether the area contains clickable controls, input text boxes, image data, etc. Corresponding area types are identified according to different display contents, and the identified area types are fed back to the first touch module.

Further, as another embodiment of the present application, the specific process of identifying the above-mentioned area type includes:

In S1802.1, the cursor displacement event sent by the first touch module is received through the click input sub-unit in the second touch module, and the click input sub-unit in the preset application is The virtual cursor is updated to the third touch coordinate.

In S1802.2, the cursor classification unit in the second touch module identifies the cursor style of the virtual cursor when updating to the third touch coordinates, and determines the third touch based on the cursor style. The area type corresponding to the coordinates.

In this embodiment, the second touch module includes an input subunit and a cursor classification unit. The above-mentioned input subunit is used to control the cursor to move; and the above-mentioned cursor classification unit is used to determine the style corresponding to the cursor in the application program, and identify the area type corresponding to the selected area through the cursor style.

In this embodiment, since the second operating system, such as Linux system and Windows system, supports keyboard and mouse control of the system, that is, during the operation of the second operating system, there will be a corresponding cursor on the interface. , to indicate the current position of the mouse in the interface. For different areas in different interfaces, the cursor styles will also be different. Exemplarily, FIG. 19 shows a schematic diagram of a cursor style provided by an embodiment of the present application. As shown in (a) in Figure 19, when the cursor moves to the non-editing area, such as moving to an interactive control (insert control 191), the mouse cursor will be displayed as an arrow style 192; see (( in Figure 19) As shown in b), when the cursor moves to the editable area, for example, when it moves to the text editing area 193, the mouse cursor will display an "I" style 194 to prompt that text data can be entered in this area. It can be seen that the cursor styles corresponding to different area types will be different, and the area type can be distinguished through the different cursor styles.

In this embodiment, the second touch module can determine the second touch position where the cursor needs to move through the input subunit. Since there may be a coordinate offset between the first operating system and the second operating system, after determining After the user clicks on the third touch coordinate in the first operating system, the second touch module can determine the corresponding second touch coordinate in the second operating system based on the third touch coordinate and the preset coordinate offset. touch position, and generate a movement instruction to control the cursor to move to the second touch position through the input subunit. Then, the cursor is moved to the above-mentioned second touch position through the mouse update unit, and then after the cursor moves to the second touch position, the corresponding cursor style is changed according to the type corresponding to the area, and the cursor is determined through the cursor classification unit The area type associated with the style can recognize the area type and feed the recognition result back to the first touch module.

It should be noted that since the above-mentioned virtual cursor is effective for the second operating system, that is, the above-mentioned virtual cursor only exists inside the second operating system, and the movement and display of the cursor will not be fed back to the first operating system, that is, when the user When the default application is controlled in the first operating system, the virtual cursor is invisible, that is, the change in the cursor style only exists in the second operating system and is used to identify the area type of the user click area.

In the embodiment of the present application, by controlling the cursor to move to a designated area and identifying the area type of the area through the cursor pattern, the logic is simple, thereby improving the efficiency of area type identification.

In S1803, the second operating system executes the operation instruction through the preset application.

In this embodiment, the default application can move the virtual cursor in the application to a designated area according to a click operation initiated by the user. If the user clicks on a control associated with a jump link, it will jump to the specified page; if the user clicks on a virtual button, the menu associated with the virtual button can be opened, or a function corresponding to the virtual button can be executed. Operation, if the area clicked by the user is an editable area, the user can enter the text input state.

In S1804, when it is detected that the area type is an editable area type, the soft keyboard is opened through the first touch module.

In this embodiment, the second touch module can feed back the identified area type to the first touch module. If the above area type is an editable area type, it means that the user may need to input text in this area. In this case The built-in soft keyboard can be turned on through the first touch module, so that the user can complete corresponding input operations through the soft keyboard.

In a possible implementation, if the user has turned on the soft keyboard when the first operating system initiates the third touch operation, then after receiving the third touch operation, the soft keyboard will continue to be turned on. .

Exemplarily, FIG. 20 shows a schematic diagram of opening the soft keyboard provided by an embodiment of the present application. Referring to Figure 20, (a) in Figure 20 shows the operation interface of the preset application. The operation interface contains multiple clickable controls. The user can click on the above controls to open the corresponding menu, such as inserting Control 201; also includes an editable text input area 202. When the user clicks on the editable text area, after the second touch module identifies the area as an editable area, the first touch module can open the soft keyboard 203, As shown in (b) in Figure 20, the user can complete the text input operation through the soft keyboard.

Further, as another embodiment of the present application, the process of opening the soft keyboard may specifically include:

In S1804.1, click gesture information is sent to the input state recognition unit in the first touch module through the input touch detection unit in the first touch module, so that the input state recognition unit enters monitoring The second touch module feeds back the waiting state of the area type.

In this embodiment, an input state recognition unit is also configured in the first touch module. When the touch detection unit sends the above-mentioned cursor displacement event to the second touch module, it will simultaneously send an input state recognition unit to the above-mentioned cursor displacement event. Click gesture information, at this time, the input state recognition unit will enter a waiting state to wait for the second touch unit to feedback the area type of the click area.

In S1804.2, the area type is fed back to the first touch module through the cursor classification unit.

In S1804.3, the input state identification unit receives the area type fed back by the second touch module, and when it is detected that the area type is an editable area type, the input state identification unit Open the soft keyboard.

In this embodiment, after the cursor classification unit in the second touch module determines the corresponding area type, it can feed back to the above-mentioned input state recognition unit. Since the above-mentioned input state recognition unit is always in a listening state, after receiving the above-mentioned After inputting the area type fed back by the status recognition unit, different operations can be performed according to the area type.

In this embodiment, the input state recognition unit determines that the area clicked by the user is an editable area, and therefore opens the soft keyboard so that the user can perform text input operations.

In the embodiment of the present application, by receiving the area type fed back by the second touch module through the input state recognition unit, logical division of labor between units can be realized, and the accuracy of the response operation can be improved.

Further, as another embodiment of the present application, after S1802, it also includes:

In S1805, when it is detected that the area type is a non-editing area type, the soft keyboard is not opened through the first touch module.

In this embodiment, if the above-mentioned area type is a non-editing area type, it means that the user does not need to input text in this area. At this time, the built-in soft keyboard can be closed through the first touch module.

Exemplarily, FIG. 21 shows a schematic diagram of closing the soft keyboard provided by an embodiment of the present application. Referring to Figure 21, as shown in (a) of Figure 21, the area originally selected by the user is the editable area 211. At this time, the first touch module can open the soft keyboard to facilitate the user's input, and when the user clicks the above insert After the control 212 is inserted, since the area corresponding to the inserted control is a non-editing area, at this time, the first touch module will close the above-mentioned soft keyboard and open the control menu 213 corresponding to the inserted control, as shown in (b) of Figure 21 .

In a possible implementation, if the user has turned on the soft keyboard when initiating the third touch operation in the first operating system, after receiving the third touch operation, the soft keyboard continues to be closed. state.

In one possible implementation, when the input state recognition unit determines that the area clicked by the user is a non-editing area, the soft keyboard is closed to prevent the soft keyboard from blocking other displayed content.

In this embodiment of the present application, it is determined whether to enable the soft keyboard by identifying whether the type of user click is an editable area, so as to avoid the inability to automatically operate cross-system applications due to different input methods between different operating systems. The soft keyboard is turned on, which improves the accuracy of cross-system application operations.

Embodiment three:

Corresponding to the touch input method described in the above embodiment, FIG. 22 shows a structural block diagram of the touch input system provided by the embodiment of the present application. For convenience of explanation, only the parts related to the embodiment of the present application are shown. part.

Referring to Figure 22, the touch input system includes a first operating system and a second operating system; the first operating system 221 and the second operating system 222 are different;

The first operating system 221 includes:

The touch operation receiving module 2211 is used to receive the user's touch operation on the preset application in the first operating system; the preset application is built based on the second operating system;

The first touch module 2212 is configured to respond to the touch operation and send the touch event generated based on the touch operation to the second touch module in the second operating system;

The second operating system 222 includes:

The second touch module 2221 is used to generate operation instructions corresponding to the touch event;

The operation instruction execution module 2222 is used to execute the operation instruction through the preset application.

Optionally, the touch operation is an edge gesture type touch operation, a multi-touch point type touch operation, or a third touch operation of clicking on any area in the preset application.

Optionally, the first operating system 221 and the second operating system 222 are any one of the following operating systems: Android operating system, Hongmeng operating system, Linux operating system, and windows operating system.

Optionally, the touch operation is a first touch operation of an edge gesture type;

The first touch module 2212 is specifically configured to: determine the window type of the target window, and generate a first touch event corresponding to the first touch operation according to the window type; the target window is the first Touch operation is performed on the window operated in the preset application; the first touch event includes: the window type of the target window and the operation type corresponding to the first touch operation;

The second touch module 2221 also includes a window management unit;

Optionally, the window type includes a parent window type and a child window type, and the operation types include closing the window, exiting the application, and performing application management.

Optionally, the first touch module 2212 also includes: a touch management unit and a touch window management unit;

Optionally, the touch operation is a second touch operation of a multi-contact type;

The first touch module 2212 is specifically configured to: generate a multi-touch event corresponding to the second touch operation, and transmit the multi-touch event to the third operating system in the second operating system. Two touch modules; the multi-touch event includes the first touch coordinates and touch time of each touch point in the first operating system;

The second touch module 2221 is specifically configured to perform coordinate conversion on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system.

Optionally, the first touch module 2212 includes: a touch management unit and a touch event conversion unit;

Optionally, the second touch module 2221 includes: a multi-touch management unit and a touch window conversion unit;

Optionally, the control window conversion unit is also used for:

Determine the target window for the required operation in the preset application, and generate a target window based on the second contact point coordinates. The second operation command of the mouth.

Optionally, the touch operation is a third touch operation of clicking on any area in the preset application;

The first touch module 2212 is specifically configured to: generate a cursor displacement touch event corresponding to the third touch operation, and send the cursor displacement touch event to the third operating system in the second operating system. Two touch modules; the cursor displacement touch event includes the third touch coordinate corresponding to the touch operation;

The second touch module 2221 is specifically configured to: identify the area type corresponding to the third touch coordinate in the preset application, and generate a third operation instruction according to the area type;

The second touch module 2221 is also used to: send the area type to the first touch module in the first operating system;

The first touch module 2212 is also configured to enable the soft keyboard when detecting that the area type is an editable area type.

Optionally, the first touch module 2212 is also configured to not open the soft keyboard when it is detected that the area type is a non-editing area type.

Optionally, the first touch module 2212 includes: a touch management unit, a touch event conversion unit, and an input touch detection unit;

Optionally, the first touch module 2212 further includes: an input state recognition unit;

Optionally, the second touch module 2221 includes: a click input subunit and a cursor classification unit;

Optionally, the first operating system includes: a startup operation receiving unit, an application startup unit, and an interface display unit; the second operating system includes: an interface feedback unit;

Therefore, the touch input device provided by the embodiment of the present application can also run a preset application across operating systems in the first operating system. When the user initiates a touch operation for the preset application in the first operating system, the device can The corresponding touch event is generated by the first touch module of the first operating system and transmitted to the second touch module deployed in the container running the second operating system, and the touch event is converted by the second touch module , generate an operation instruction that can be recognized by the second operating system, so that the preset application running in the second operating system responds to the operation instruction and generates a corresponding response result, realizing the control of touch operations across systems. Interoperability. Compared with existing device control technology, in the embodiment of this application, by configuring in different operating systems The corresponding touch module converts touch operations across systems through the touch module, thereby generating operation instructions that support preset application recognition for cross-system operation, improving the consistency of cross-system operations, and thus improving the operational efficiency. The expansion of the application ecosystem within the system meets the needs of users.

Figure 23 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As shown in Fig. 23, the electronic device 23 of this embodiment includes: at least one processor 230 (only one processor is shown in Fig. 23), a memory 231, and a device stored in the memory 231 and capable of processing in the at least one processor 230. The computer program 232 runs on the processor 230. When the processor 230 executes the computer program 232, the steps in any of the above touch input method embodiments are implemented.

The electronic device 23 may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The electronic device may include, but is not limited to, a processor 230 and a memory 231. Those skilled in the art can understand that FIG. 23 is only an example of the electronic device 23 and does not constitute a limitation on the electronic device 23. It may include more or fewer components than shown in the figure, or some components may be combined, or different components may be used. , for example, it may also include input and output devices, network access devices, etc.

The so-called processor 230 can be a central processing unit (Central Processing Unit, CPU). The processor 230 can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit). , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

The memory 231 may be an internal storage unit of the electronic device 23 in some embodiments, such as a hard disk or memory of the electronic device 23 . In other embodiments, the memory 231 may also be an external storage device of the electronic device 23, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), or a secure digital card equipped on the electronic device 23. (Secure Digital, SD) card, flash card (Flash Card), etc. Further, the memory 231 may also include both an internal storage unit of the electronic device 23 and an external storage device. The memory 231 is used to store operating systems, application programs, boot loaders (Boot Loaders), data, and other programs, such as program codes of the computer programs. The memory 231 can also be used to temporarily store data that has been output or is to be output.

It should be noted that the information interaction, execution process, etc. between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For details of their specific functions and technical effects, please refer to the method embodiments section. No further details will be given.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional units and modules is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs. Module completion means dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be hardware-based. It can also be implemented in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present application. For the specific working processes of the units and modules in the above system, please refer to the corresponding processes in the foregoing method embodiments, and will not be described again here.

An embodiment of the present application also provides an electronic device. The electronic device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. The processor executes The computer program implements the steps in any of the above method embodiments.

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in each of the above method embodiments can be implemented.

Embodiments of the present application provide a computer program product. When the computer program product is run on a mobile terminal, the steps in each of the above method embodiments can be implemented when the mobile terminal is executed.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, this application can implement all or part of the processes in the methods of the above embodiments by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program When executed by a processor, the steps of each of the above method embodiments may be implemented. Wherein, the computer program includes computer program code, which may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code to a camera/electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. For example, U disk, mobile hard disk, magnetic disk or CD, etc. In some jurisdictions, subject to legislation and patent practice, computer-readable media may not be electrical carrier signals and telecommunications signals.

In the above embodiments, each embodiment is described with its own emphasis. For parts that are not detailed or documented in a certain embodiment, please refer to the relevant descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed devices/network devices and methods can be implemented in other ways. For example, the apparatus/network equipment embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Or components can be combined or can be integrated into another system, or some features can be omitted, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

The above-described embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still implement the above-mentioned implementations. The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of this application, and should be included in within the protection scope of this application.

Claims

A touch input method, characterized in that the method is applied to an electronic device, the electronic device is equipped with a first operating system and a second operating system at the same time, the first operating system and the second operating system are different;

The methods include:

The first operating system receives a user's touch operation on a preset application; the preset application is built based on the second operating system;

The first operating system responds to the touch operation and sends a touch event generated based on the touch operation to the second operating system;

The second operating system generates an operation instruction corresponding to the touch event;

The second operating system executes the operation instruction through the preset application.
The method according to claim 1, characterized in that the first operating system and the second operating system are any one of the following operating systems: Android operating system, Hongmeng operating system, Linux operating system or windows operating system. .
The method of claim 1, wherein the touch operation is an edge gesture type touch operation, a multi-touch point type touch operation, or a third click on any area in the preset application. Three-touch operation.
The method of claim 3, wherein the touch operation is a first touch operation of an edge gesture type;

In response to the touch operation, the first operating system sends a touch event generated based on the touch operation to the second operating system, including:

The first operating system determines the window type of the target window, and generates a first touch event corresponding to the first touch operation according to the window type; the target window is the first touch operation in the preset time. The window operated in the application; the first touch event includes: the window type of the target window and the operation type corresponding to the first touch operation.
The method according to claim 4, characterized in that the window type is a parent window type or a child window type, and the operation type is closing the window, exiting the application or performing application management.
The method according to claim 3, wherein the touch operation is a second touch operation of a multi-contact type;

In response to the touch operation, the first operating system sends a touch event generated based on the touch operation to the second operating system, including:

The first operating system generates a multi-touch event corresponding to the second touch operation and transmits the multi-touch event to the second operating system; the multi-touch event is included in the The first touch coordinates and touch time of each contact point in the first operating system;

The first touch module sends the touch event to the second touch module in the container running the second operating system, and the second touch module generates the touch event. The operation instructions corresponding to the event include:

The second operating system performs coordinate conversion on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system.
The method of claim 6, wherein the first operating system generates a multi-touch event corresponding to the second touch operation and transmits the multi-touch event to the second Touch module, including:

The first operating system monitors touch operations initiated by the user;

If the first operating system recognizes that the touch operation is a second touch operation, it performs legal verification on the second touch operation;

If the first operating system recognizes that the second touch operation passes the legal verification, it generates the multi-touch event and sends the multi-touch event to the second operating system. .
The method according to claim 6, characterized in that the second operating system performs coordinate conversion on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system. ,include:

The second operating system performs complete verification on the multi-touch event;

If the second operating system recognizes that the multi-touch event passes complete verification, it performs coordinate conversion on each of the touch points in the multi-touch event to determine each of the first touch coordinates. The corresponding second touch coordinates in the second operating system; the second touch coordinates are based on the size of the first window of the preset application in the first operating system, the preset application The size of the corresponding second window and the first touch coordinates are determined in the second operating system; the first window is the mapping window of the second window under the first operating system;

The second operating system generates the second operation instruction according to the second touch coordinates.
The method of claim 8, wherein the second operating system generates the second operation instruction according to the second touch coordinates, including:

The second operating system determines a target window for a required operation in the preset application, and generates a second operation instruction that acts on the target window based on the second contact point coordinates.
The method of claim 3, wherein the touch operation is a third touch operation of clicking on any area in the preset application;

In response to the touch operation, the first operating system sends a touch event generated based on the touch operation to the second operating system, including:

The first operating system generates a cursor displacement touch event corresponding to the third touch operation, and sends the cursor displacement touch event to the second operating system; the cursor displacement touch event includes The third touch coordinate corresponding to the above-mentioned touch operation;

The second operating system generates operation instructions corresponding to the touch event, including:

The second operating system identifies the area type corresponding to the third touch coordinate in the preset application, and generates a third operation instruction according to the area type;

After the second operating system generates the operation instruction corresponding to the touch event, it also includes:

The first operating system receives feedback from the second operating system regarding the area type corresponding to the third touch coordinate;

When the first operating system detects that the area type is an editable area type, the soft keyboard is enabled.
The method according to claim 10, characterized in that, after the first operating system receives feedback from the second operating system regarding the area type corresponding to the third touch coordinate, it further includes:

The first operating system does not open the soft keyboard when detecting that the area type is a non-editing area type.
The method of claim 10, wherein the first operating system generates a cursor displacement touch event corresponding to the third touch operation and sends the cursor displacement touch event to the second Operating system, including:

The first operating system monitors the touch operation initiated by the user;

If the first operating system recognizes that the touch operation is a third touch operation, it performs legal verification on the third touch operation;

If the first operating system recognizes that the third touch operation passes the legal verification, it generates a click recognition event;

In response to the click recognition event, the first operating system identifies the click type corresponding to the third touch operation and the third touch coordinates, and determines the click type and the third touch coordinates based on the click type and the third touch coordinates. The generated cursor displacement event is sent to the second operating system.
The method according to claim 10, characterized in that the second operating system identifies the area type corresponding to the third touch coordinate in the preset application, and generates a third operation instruction according to the area type. ,include:

The second operating system receives the cursor displacement event sent by the first operating system, and updates the virtual cursor in the second operating system to the third touch coordinates;

The second operating system identifies the cursor style of the virtual cursor when updating to the third touch coordinates, and determines the area type corresponding to the third touch coordinates based on the cursor style.
The method according to any one of claims 1 to 13, characterized in that receiving a user's touch operation on a preset application in the first operating system further includes:

The first operating system receives a user's startup operation for the preset application;

The first operating system sends an application startup instruction to the second operating system through a proxy application installed in the first operating system; the application startup instruction carries the application identifier of the preset application;

The second operating system responds to the application startup instruction, runs the preset application, and feeds back the operation interface of the preset application to the proxy application in the first operating system;

The first operating system generates the operation interface through the proxy application.
A touch input system, characterized in that the touch input system includes a first operating system and a second operating system; the first operating system and the second operating system are different;

The first operating system includes:

A touch operation receiving module, configured to receive a user's touch operation on a preset application in the first operating system; the preset application is built based on the second operating system;

A first touch module, configured to respond to a touch operation and send a touch event generated based on the touch operation to a second touch module in the second operating system;

The second operating system includes:

The second touch module is used to generate operation instructions corresponding to the touch event;

An operation instruction execution unit is used to execute the operation instruction through the preset application.
The system according to claim 15, wherein the first operating system and the second operating system are any one of the following operating systems: Android operating system, Hongmeng operating system, Linux operating system or windows operating system. .
The system of claim 15, wherein the touch operation is an edge gesture type touch operation, a multi-touch point type touch operation, or a third click on any area in the preset application. Three-touch operation.
The system according to claim 17, wherein the touch operation is a first touch operation of an edge gesture type;

The first touch module is specifically configured to: determine a window type of a target window, and generate a first touch event corresponding to the first touch operation according to the window type; the target window is the first touch event. Control the window operated in the preset application; the first touch event includes: the window type of the target window and the operation type corresponding to the first touch operation;

The second touch module also includes a window management unit;

The window management unit is used to generate a first operation instruction for the target window.
The system according to claim 18, characterized in that the window type is a parent window type or a child window type, and the operation type is closing the window, exiting the application or performing application management.
The system according to claim 19, wherein the first touch module further includes: a touch management unit and a touch window management unit;

The touch management unit is configured to monitor the touch operation initiated by the user, and when the touch operation is recognized as a first touch operation, send the first touch operation to the Touch window management unit;

The touch window management unit is used to establish a cascade relationship between each operation window in the preset application;

The touch window management unit is also configured to determine the window type of the target window according to the first touch operation.
The system according to claim 17, wherein the touch operation is a second touch operation of a multi-contact type;

The first touch module is specifically configured to: generate a multi-touch event corresponding to the second touch operation, and transmit the multi-touch event to the second operating system in the second operating system. Touch module; the multi-touch event includes the first touch coordinates and touch time of each touch point in the first operating system;

The second touch module is specifically configured to perform coordinate conversion on each touch point in the multi-touch event to obtain a second operation instruction suitable for the second operating system.
The system according to claim 21, wherein the first touch module includes: a touch management unit and a touch event conversion unit;

The touch management unit is configured to monitor the touch operation initiated by the user, and when detecting that the touch operation is the second touch operation, send the second touch operation to the touch screen. Control event conversion unit;

The touch event conversion unit is used to perform legal verification on the second touch operation;

The touch event conversion unit is also configured to generate the multi-touch event if it is recognized that the second touch operation passes the legal verification, and send the multi-touch event to the The second touch module in the second operating system.
The system according to claim 21, wherein the second touch module includes: a multi-touch management unit and a touch window conversion unit;

The multi-touch management unit is used to complete verification of the multi-touch event, and when it is recognized that the multi-touch event passes the complete verification, send the multi-touch event. to the touch window conversion unit;

The touch window conversion unit is used to perform coordinate conversion on each of the touch points in the multi-touch event, and determine the second touch corresponding to each of the first touch coordinates in the second operating system. coordinates; the second touch coordinates are based on the size of the first window of the preset application in the first operating system and the second corresponding second window of the preset application in the second operating system. The size of the window and the first touch coordinates are determined; the first window is a mapping window for the second window under the first operating system;

The touch window conversion unit is also used to generate the second operation instruction according to the second touch coordinates.
The system according to claim 23, characterized in that the control window conversion unit is also used for:

Determine a target window for a required operation in the preset application, and generate a second operation instruction that acts on the target window based on the second contact point coordinates.
The system according to claim 17, wherein the touch operation includes a third touch operation of clicking on any area in the preset application;

The first touch module is specifically configured to: generate a cursor displacement touch event corresponding to the third touch operation, and send the cursor displacement touch event to the second operating system in the second operating system. Touch module; the cursor displacement touch event includes the third touch coordinate corresponding to the touch operation;

The second touch module is specifically configured to: identify the area type corresponding to the third touch coordinate in the preset application, and generate a third operation instruction according to the area type;

The second touch module is also configured to: send the area type to the first touch module in the first operating system;

The first touch module is also configured to enable the soft keyboard when detecting that the area type is an editable area type.
The system according to claim 25, wherein the first touch module is further configured to not open the soft keyboard when it is detected that the area type is a non-editing area type.
The system according to claim 25, wherein the first touch module includes: a touch management unit, a touch event conversion unit and an input touch detection unit;

The touch management unit is configured to monitor the touch operation initiated by the user, and when detecting that the touch operation is a third touch operation, send the third touch operation to the Touch event conversion unit;

The touch event conversion unit is used to perform legal verification on the third touch operation, and when it is detected that the third touch operation passes the legal verification, the input touch detection unit Send a click recognition event generated based on the third touch operation;

The input touch detection unit is configured to, in response to the click recognition event, identify the click type corresponding to the third touch operation and the third touch coordinates, and determine the click type and the third touch coordinate based on the click type and the third touch coordinate. The cursor displacement event generated by the three touch coordinates is sent to the second touch module in the second operating system.
The system according to claim 27, wherein the first touch module further includes: an input state recognition unit;

The input touch detection unit is also configured to send click gesture information to the input state recognition unit in the first touch module, so that the input state recognition unit monitors the feedback from the second touch module. Area type wait states;

The input state recognition unit is configured to receive the area type fed back by the second touch module, and when detecting that the area type is an editable area type, enable the soft keyboard.
The system according to claim 25, wherein the second touch module includes: a click input subunit and a cursor classification unit;

The click input subunit is configured to receive the cursor displacement event sent by the first touch module of the first operating system, and update the virtual cursor in the second operating system to the third Touch coordinates;

The cursor classification unit is used to identify the cursor style of the virtual cursor when updating to the third touch coordinates, and determine the area type corresponding to the third touch coordinates based on the cursor style;

The cursor classification unit is also used to send the area type to the first touch module in the first operating system.
The system according to any one of claims 15 to 29, characterized in that the first operating system includes: a startup operation receiving unit, an application startup unit and an interface display unit; the second operating system includes: an interface feedback unit ;

A startup operation receiving unit configured to receive a user's startup operation for the preset application in the first operating system;

An application startup unit configured to initiate an application startup instruction to the second operating system through a proxy application installed in the first operating system; the application startup instruction carries the application identifier of the preset application;

An interface feedback unit, configured to respond to the application startup instruction, run the preset application in the second operating system, and feed back the operation interface of the preset application to the proxy application;

An interface display unit is configured to generate the operation interface in the first operating system through the proxy application.
An electronic device, characterized in that the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the computer program is as described in the claim. The steps of the method according to any one of claims 1 to 14.
A computer-readable storage medium stores a computer program, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.