WO2023137996A1

WO2023137996A1 - Application control method, gesture processing apparatus, intelligent terminal, and vehicle

Info

Publication number: WO2023137996A1
Application number: PCT/CN2022/103039
Authority: WO
Inventors: 黄嘉桐; 丁冠源; 回姝; 郑红丽; 郭富琦; 郑彤
Original assignee: 中国第一汽车股份有限公司
Priority date: 2022-01-24
Filing date: 2022-06-30
Publication date: 2023-07-27
Also published as: CN114546549B; CN114546549A

Abstract

The technical solution of the present invention provides an application control method, a gesture processing apparatus, an intelligent terminal, and a vehicle. The intelligent terminal is provided with two gesture recognition channels, the channel 1 is a gesture recognition channel based on an Android system, the channel 2 is a gesture recognition channel based on a linux kernel, and the channel 1 does not need to modify a native architecture of the Android system. A gesture event interface is newly added and implemented in an input management server of the intelligent terminal, i.e., a gesture event can be sent to an application in a native framework of the Android system and responded. The gesture recognition channel based on the linux kernel serves as a gesture recognition channel parallel to the channel 1, the channel 2 operates in parallel and does not influence normal operation of the channel 1, and because the two gesture recognition channels are loaded in the intelligent terminal, when one of the channels fails to operate, the other channel can send the gesture event to the application, so as to ensure that the application executes a response operation on the gesture event.

Description

App control method, gesture processing device, smart terminal and vehicle

technical field

The invention belongs to the technical field of data processing of data processing equipment, and in particular relates to an application program control method, an intelligent terminal and a vehicle with the intelligent terminal.

Background technique

At present, touch screens have been widely used in various intelligent terminal devices. Operators use the touch screens to input operation instructions into the intelligent terminal devices equipped with the touch screens to realize human-computer interaction.

With the improvement of the processing capability of smart terminals and the rapid development of information technology, the number and types of applications supported by smart terminals are increasing day by day. With the popularization of touch screens, applications that use input gesture events to trigger and execute various operating instructions have emerged and applied to multiple technical fields. Especially in the field of vehicles, with the intelligent upgrade of cars, the traditional interaction method is replaced by touch interaction, thus establishing an interaction strategy based on multi-finger touch. In actual driving scenarios, the user's multi-finger gestures trigger the execution of applications to enhance the human-computer interaction experience.

However, the current application program has a single response method to gesture events. When the gesture recognition channel fails, the application program will not be able to perform the response operation. The single response method of the application program to gesture commands greatly reduces the fault tolerance rate of the system, and it is difficult to guarantee the speed and accuracy of gesture recognition.

Contents of the invention

In order to solve the above-mentioned technical problems, the technical solution of the present invention provides a control method of an application program. The smart terminal installed with the application program is loaded with an Android system based on the linux kernel, and the smart terminal is configured with two gesture recognition paths, which are:

Path 1: a gesture recognition path based on the Android system. The implementation steps of path 1 include: 1) Pre-create an input monitoring object related to gesture events on the smart terminal, and the input monitoring object is used to implement an interface in the input management server of the smart terminal; 2) The input management server reads and analyzes the gesture events reported by the linux kernel in real time, and sends them to the input monitoring object and the foreground window.

Path 2: Gesture recognition path based on the linux kernel. The input device registered by the linux kernel detects, reads, parses, and distributes gesture events to the application in real time. The application receives the gesture event in real time and executes the response operation to the gesture event on the application page;

Both the channel 1 and the channel 2 are set in parallel in the smart terminal to control the response of the application program to the gesture event.

Further, gesture events include sliding gestures, multi-finger touch gestures, multiple-click gestures, and long-press gestures.

Further, the input management server of channel 1 reads and analyzes input gesture events in real time, and distributes the analyzed gesture events to the application programs through the window management server.

Further, at the same time, the window management server only calls back the parsed input gesture to at most one application program.

Further, at the same time, only one application in path 2 receives the gesture event distributed by the input device.

The technical solution of the present invention also provides a gesture processing device, which includes:

The gesture acquisition module is used to acquire gesture events in real time;

The gesture sending module sends gesture events to specified applications based on the Android system and the Linux kernel, and guides the applications to perform response operations on the gesture events.

Further, the gesture processing device further includes a gesture analysis module, configured to perform analysis operations on the acquired gesture events.

Further, the parsing operation includes judging the number of contacts, clustering processing of contacts, and analyzing the track of contacts.

The technical solution of the present invention also provides an intelligent terminal. The intelligent terminal is loaded with an Android system based on the linux kernel. The intelligent terminal is also equipped with a memory and a processor. The memory is used to store program instructions, and the processor is used to call and execute the program instructions in the memory. The control method of the application program is executed in parallel. Path 1 and path 2. Path 1: a gesture recognition path based on the Android system. The implementation steps of path 1 include: 1) Pre-create an input monitoring object related to gesture events on the smart terminal, and the input monitoring object is used to realize an interface in the input management server of the smart terminal; Read and analyze the gesture events reported by the linux kernel, and send them to the input monitoring object and the foreground window. The foreground window receives and calls back to the application program layer by layer. The application program receives the gesture event and executes the response operation to the gesture event on the application program page; Path 2: Based on the gesture recognition channel of the linux kernel, the input device registered by the linux kernel detects, reads and distributes the gesture event to the application program in real time, and the application program receives the gesture event and executes the response operation to the gesture event on the application program page.

The technical solution of the present invention also provides a vehicle, the vehicle is equipped with an intelligent terminal, and the intelligent terminal is loaded with an Android system and a linux kernel at the same time, and the intelligent terminal is provided with a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions in the memory, execute the control method of the application program, and run the channel 1 and the channel 2 in parallel.

The beneficial technical effects of the control method of the application program, the intelligent terminal and the vehicle provided by the present invention are as follows:

① There are two gesture recognition paths in the smart terminal. Path 1 is a gesture recognition path based on the Android system, and path 2 is a gesture recognition path based on the Linux kernel. Path 1 does not need to modify the native architecture of the Android system. A gesture event interface is added and implemented in the input management server of the smart terminal, and gesture events can be sent to the application program in the native framework of the Android system and respond. The gesture recognition path based on the Linux kernel is parallel to the gesture recognition path of path 1. Parallel operation of path 2 does not affect the normal operation of path 1. When one of the channels fails, the other channel can also send gesture events to the application to ensure that the application performs operations in response to gesture events;

②Gesture events of the present invention include sliding gestures, multi-finger touch gestures, multiple click gestures, long press gestures and various personalized self-setting gestures. Since the channel 2 can add or modify gestures in the application program, the gesture recognition channel 2 can realize the recognition of emerging personalized gestures by the application program in a short period of time by customizing the application program layer. , so as to obtain a gesture recognition path with more stable functions, and the two paths are set in parallel, which is conducive to ensuring that the smart terminal realizes orderly and stable support for the personalized gestures added or modified by the application;

③The input management server of channel 1 reads and analyzes input gesture events in real time, and distributes the analyzed gesture events to applications through the window management server, which is conducive to efficient and unified management of application windows and improved gesture recognition efficiency;

④In order to avoid display conflicts caused by multiple applications responding at the same time, the window service manager in path 1 only allows one application to execute the callback operation at the same time; in path 2, only one application obtains the gesture events distributed by the input device registered by the linux kernel at the same time;

⑤ In order to improve the gesture recognition accuracy and gesture recognition efficiency, the gesture processing device is also provided with a gesture analysis module, which is used to perform analysis operations on the acquired gesture events.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that are required in the description of the embodiments or prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings can also be obtained based on these drawings without creative work.

Fig. 1 is a schematic diagram of the event processing flow of the native Android system;

Fig. 2 is the workflow schematic diagram of input management server in embodiment 1;

FIG. 3 is a schematic diagram of the gesture event processing flow of the parallel path 1 and path 2.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations as sequential processing, many of the operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of operations can be rearranged. A process may be terminated when its operations are complete, but may also have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subroutine, or the like.

Specific structural and functional details disclosed herein are representative only and for purposes of describing example embodiments of the present invention. This invention may, however, be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a", "an" and "an" are intended to include the plural unless the context clearly dictates otherwise. It should also be understood that the terms "comprising" and/or "comprising" as used herein specify the presence of stated features, integers, steps, operations, units and/or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, units, components and/or combinations thereof.

The content of the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Example 1:

However, the current application program has a single way of responding to gesture events. When the gesture recognition channel fails, the application program will not be able to perform the response operation, and the single response method of the application program to gesture commands greatly reduces the fault tolerance rate of the system, making it difficult to guarantee the speed and accuracy of gesture recognition.

In order to solve the above-mentioned technical defects, the technical solution of the present invention provides an application control method. Two gesture recognition paths are configured on the smart terminal. Path 1 is a gesture recognition path based on the Android system, and path 2 is a gesture recognition path based on the Linux kernel. Path 1 does not need to modify the native architecture of the Android system. As a gesture recognition path parallel to path 1, the parallel operation of path 2 does not affect the normal operation of path 1, and since there are two gesture recognition paths loaded in the smart terminal, when one path fails, the other path can also send gesture events to the application program to ensure that the application program executes the response operation to the gesture event.

Specifically, the gesture recognition path based on the Android system, that is, the steps for path 1 to realize gesture recognition and application response are as follows:

First, an input monitoring object related to gesture events is pre-created in the smart terminal. As shown in Figure 2, the input monitoring object is connected to the input event distributor, and the input monitoring object is used to implement an interface in the input management server of the smart terminal.

To further illustrate, the system service module of the smart terminal is one of the cores of the Android system, and is used to support most of the Android system services. In order to realize the response of the application program to the gesture event, the native input monitoring object of the Android system can only be created in the system service module. The present invention is different from the traditional Android system. The input monitoring object related to the gesture event is selected to be set in the input management server. The gesture event will be triggered when the user puts the finger on the screen, slides on the screen or moves away from the screen. The newly added input monitoring object is connected to the input management server of the input smart terminal, and can realize an interface in the input management server, so that the input management server holds the input monitoring event and monitors the gesture events input by the touch screen in real time.

Then, the input management server reads and analyzes gesture events in real time, sends them to the input monitoring object and the foreground window, and calls back to the application program layer by layer, and executes the response operation to the preset gesture on the application program page. Specifically, in the gesture recognition path, the input management server (IMS, InputManagerServer) will open two threads including event processing and event distribution in the native layer. The event processing thread first reads input events from the dev/input directory through eventhub, and throws the data into a queue after processing, wakes up the sleeping event distribution intermediary, and the event distribution intermediary sends the data to the foreground window. The foreground window receives and calls back to the upper-level window or application. Actions in response to events. Path 1 is different from the existing Android system. It innovatively sets the input monitoring object in the input management server to realize one of the interfaces without changing the original design of the Android system. The input management server can receive gesture events reported by the Linux kernel, realize the monitoring of all gesture times on the touch screen, and send gesture events to all monitoring objects and designated applications, and the gesture events received by the monitoring objects will not affect the input manager to distribute gesture events to applications.

As shown in FIG. 3 , path 2 is a gesture recognition path based on the Linux kernel, and this path is set in parallel with path 1 in the smart terminal to control the response of the application program to the gesture event.

To further explain, the linux kernel includes a system call interface, a process management module, a memory management module, a virtual file system, a network module, and a device driver module. Its function is an intermediate layer between hardware and software, and its purpose is to pass the request of the application layer sequence to the hardware, and act as the underlying driver to address various devices and components in the system. The kernel is responsible for resource management, which allocates available shared resources (CPU time, disk space, network connection, etc.) to each system process, and provides a set of system-oriented commands that can call applications like calling ordinary functions, which is sufficient to support a gesture recognition path independent of the native Android system. The linux kernel can detect, read and distribute gesture events in real time through the registered input devices, and analyze and process the gesture events. It is equivalent to adding a background service to the linux kernel and allocating system resources. The application program can monitor the gestures sent from the bottom layer. Since the channel adopts the independent data acquired by the input device of the linux kernel, which is parallel to the channel 1, it is beneficial to maintain the independence of the two channels, and the channel 2 does not need to be recognized by the system before being sent back to the APP, which shortens the processing time of the touch event. Since the response time and response accuracy of the application program are the key factors for evaluating the response effect of the application program, and the quality of the response is crucial to the user experience, the technical solution of the present invention has a more stable and efficient response effect in the control of the application program compared to the smart terminal with only one gesture recognition path.

The R&D personnel tested the response effect of multiple applications installed on three different smart terminals to the same gesture event, and calculated the average response speed, response accuracy rate, and non-response rate. The three different smart terminals refer to smart terminal 1 with access 1 and access 2, smart terminal 2 with only access 1, and smart terminal 3 with only access 2.

After testing, compared with smart terminal 2, the average response time of the applications loaded on smart terminal 1 is increased by 47%, the average response accuracy rate is increased by 85%, and the average non-response rate is reduced by 70%.

In order to further improve gesture recognition efficiency, the input management server of channel 1 reads and analyzes input gesture events in real time, and distributes the analyzed gesture events to applications through the window management server. The window management server, also known as "WindowManagerServer", uses the input monitoring object as a middleware to connect the input management server and the window management server. The window service manager receives gesture events and outputs the display requests of the application to the display device in an orderly manner. Specifically, the window management server manages the addition and deletion of windows, window size, window hierarchy, window startup, window animation, event dispatch, surface management, etc. Through the settings of the window management server, it is possible to efficiently and uniformly manage application windows and improve gesture recognition efficiency.

In path 1, in order to avoid display conflicts caused by simultaneous responses of multiple applications, the window service manager only allows one application to execute the callback operation at the same time.

In order to avoid multiple application programs responding to display conflicts at the same time, in path 2, only one application program obtains the gesture events distributed by the input device registered by the linux kernel at the same time.

The java description of the application programming interface involved in path 2 is as follows:

With the development of computer technology and artificial intelligence, human-computer interaction gradually develops from traditional mechanical buttons to touch operation and voice operation, and the ways of human-computer interaction become more and more diversified. For touch operation, it has also gradually upgraded from the initial single-touch to multi-touch and multi-finger touch. Especially when some applications need to cooperate with special touch methods based on their own tonality to highlight their product features, the required touch gestures gradually tend to be personalized and customized. In the embodiment of the present invention, in the field of vehicle technology, drivers tend to use multi-finger touch gestures, and this gesture is also one of the trends in the development of multi-mode interaction in the future. In other embodiments, gesture events can also be sliding gestures, multiple-click gestures, and long-press gestures. Since path 2 can add or modify gestures in the application program, it is convenient for gesture recognition path 2 to realize the recognition of emerging personalized gestures by the application program in a short period of time by customizing the application program layer, which is in line with the multi-mode development trend of human-computer interaction in the future. Gesture recognition path 1 set in parallel with path 2 can realize the modification of the system core by placing gesture event-related input monitoring objects in the input management server when there is sufficient time, so as to obtain a gesture recognition path with more stable functions. Optimize gestures to achieve orderly and stable support.

Example 2:

On the basis of Embodiment 1, the technical solution of the present invention also provides a gesture processing device, which is set on a smart terminal, and the gesture processing device includes:

The gesture acquisition module is used to acquire gesture events in real time;

The gesture sending module sends gesture events to the application based on the Android system and the Linux kernel, and guides the application to perform response operations on the gesture events.

In order to further refine the solution, the gesture sending module uses path 1 and path 2 to send gesture events to the application program, and guide the application program to perform response operations on the gesture events. Among them, channel 1 is a gesture recognition channel based on the Android system, and channel 2 is a gesture recognition channel based on the Linux kernel. Channel 1 does not need to modify the native architecture of the Android system. A gesture event interface is added and implemented in the input management server of the smart terminal, and gesture events can be sent to the application program within the native framework of the Android system and respond. The gesture recognition channel based on the Linux kernel is parallel to the gesture recognition channel of channel 1. The parallel operation of channel 2 does not affect the normal operation of channel 1. Another channel can also send gesture events to the application to ensure that the application performs operations in response to gesture events.

In order to improve gesture recognition accuracy and gesture recognition efficiency, the gesture processing device is also provided with a gesture analysis module for performing analysis operations on the acquired gesture events. Specifically, gesture events fire when the user places their finger on the screen, swipes across the screen, or lifts it off the screen. This is because when the user's finger touches the touch screen (capacitive screen, resistive screen) and generates a contact signal, at the same time, the contact signal obtains the coordinates on the touch screen when it occurs. The gesture analysis module can analyze the contact signal based on the coordinate information. The specific analysis operation includes the judgment of the number of contacts, the clustering processing of the contacts, and the analysis of the contact track.

Example 3:

On the basis of Embodiment 1, the technical solution of the present invention also provides an intelligent terminal. The intelligent terminal is loaded with an Android system and a linux kernel at the same time. The intelligent terminal is provided with a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions in the memory, execute the control method of the application program, and run path 1 and path 2 in parallel.

Path 1: Gesture recognition path based on the Android system. The implementation steps of path 1 include: 1) Pre-create an input monitoring object related to gesture events in the system service module of the smart terminal. The input monitoring object is used to implement an interface in the input management server of the smart terminal; 2) The input management server reads and analyzes gesture events in real time and sends them to the input monitoring object and the foreground window. Execute operations in response to preset gestures;

Path 2: Gesture recognition path based on the linux kernel. The input device registered by the linux kernel detects, reads, and distributes gesture events to the application in real time. When the application detects the preset gesture of the application, it executes the response operation to the preset gesture on the application page according to the mapping relationship between the preset gesture and the control command of the application;

Both the channel 1 and the channel 2 are set in parallel in the smart terminal to control the response of the application program to gesture events including sliding gestures, multi-finger touch gestures, multiple-click gestures, and long-press gestures.

Pass 1 does not need to modify the native architecture of the Android system. In the input management server of the smart terminal, it can be added and realized a gesture event interface. You can send gesture events to the application and respond in the Android system native framework. And because there are two hand -to -way recognition channels in the intelligent terminal, when one of the channels running fails, the other pathway can also send gesture events to the application to ensure the application of the application of the opponent's gesture event.

In addition, to rationalize the solution, the input management server of channel 1 reads and analyzes input gestures in real time, and distributes the analyzed input gestures to applications through the window management server, and the window management server only calls back the analyzed input gestures to at most one application program. In path 2, only one application obtains the gesture events distributed by the input device registered by the linux kernel at the same time.

Example 4:

The technical solution of the present invention also provides a vehicle, the vehicle is equipped with an intelligent terminal, and the intelligent terminal is loaded with an Android system and a linux kernel at the same time, and the intelligent terminal is provided with a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions in the memory, execute the control method of the application program, and run the path 1 and path 2 in parallel.

Path 1: Gesture recognition path based on the Android system. The implementation steps of path 1 include: 1) Pre-create an input monitoring object related to gesture events in the system service module of the smart terminal. The input monitoring object is used to implement an interface in the input management server of the smart terminal; 2) The input management server reads and analyzes gesture events in real time, and sends them to the input monitoring object and the foreground window. Response actions for preset gestures;

The control method of an application provided by the technical solution of the present invention, the gesture processing device, the smart terminal and the vehicle equipped with the smart terminal have been introduced in detail above. In this paper, specific examples are used to illustrate the principle and implementation of the present invention. The description of the above embodiment is only used to help understand the core idea of the present invention. At the same time, for those of ordinary skill in the art, according to the ideas and methods of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

The control method of application program is loaded with the Android system built based on linux kernel in the intelligent terminal that described application program is installed, it is characterized in that, two gesture recognition pathways are arranged in described intelligent terminal, are respectively:

Path 1: a gesture recognition path based on the Android system. The implementation steps of path 1 include: 1) Pre-create an input monitoring object related to gesture events on the smart terminal, and the input monitoring object is used to implement an interface in the input management server of the smart terminal; 2) The input management server reads and analyzes the gesture events reported by the linux kernel in real time, and sends them to the input monitoring object and the foreground window.

Path 2: Gesture recognition path based on the linux kernel. The input device registered by the linux kernel detects, reads, parses, and distributes gesture events to the application in real time. The application receives the gesture event in real time and executes the response operation to the gesture event on the application page;

Both the channel 1 and the channel 2 are set in parallel in the smart terminal to control the response of the application program to the gesture event.
The application control method according to claim 1, wherein the gesture event includes a sliding gesture, a multi-finger touch gesture, a multiple-click gesture, and a long-press gesture.
The application program control method according to claim 1, wherein the input management server of the channel 1 reads and analyzes input gesture events in real time, and distributes the analyzed gesture events to the application programs through the window management server.
The application program control method according to claim 3, wherein at the same time, the window management server only calls back the analyzed input gesture to at most one application program.
The application program control method according to claim 1, wherein only one application program in the path 2 receives the gesture event distributed by the input device at the same time.
A gesture processing device, characterized in that it comprises:

The gesture acquisition module is used to acquire gesture events in real time;

The gesture sending module sends gesture events to specified applications based on the Android system and the Linux kernel, and guides the applications to perform response operations on the gesture events.
The gesture processing device according to claim 6, further comprising a gesture analysis module configured to perform analysis operations on the acquired gesture events.
The gesture processing device according to claim 7, wherein the analyzing operation includes determining the number of touch points, clustering the touch points, and analyzing the track of the touch points.
An intelligent terminal, characterized in that the intelligent terminal is loaded with an Android system built based on the linux kernel, and the intelligent terminal is also provided with a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call and execute the program instructions in the memory, execute the control method of the application program as described in any one of claims 1-5, and run the passage 1 and the passage 2 in parallel.
A vehicle, characterized in that the vehicle is equipped with the intelligent terminal as claimed in claim 9 .