WO2022179239A1

WO2022179239A1 - Touch behavior recognition method and apparatus, and device

Info

Publication number: WO2022179239A1
Application number: PCT/CN2021/136066
Authority: WO
Inventors: 陈维
Original assignee: 华为技术有限公司
Priority date: 2021-02-26
Filing date: 2021-12-07
Publication date: 2022-09-01
Also published as: CN115047993A

Abstract

Disclosed in the embodiments of the present application are a touch behavior recognition method, a related apparatus, and a robot. The method is applied to the robot, and the robot comprises a capacitive touch sensor, an inertial measurement unit (IMU), and a processor, and is used for improving the accuracy of touch behavior recognition. In the method of the embodiments of the present application, the capacitive touch sensor detects the touch state information of a touch action, and the touch state information of the touch action indicates a touch state or a non-touch state; the IMU detects the acceleration information of the touch action; and the processor determines a touch duration of the touch action according to the touch state information of the touch action, and determines a touch action type according to the touch duration of the touch action and the vibration amplitude of the touch action, wherein the vibration amplitude of the touch action is determined according to the acceleration information of the touch action, and the touch action type is used for determining a touch behavior.

Description

A touch behavior recognition method, device and device

This application claims the priority of the Chinese patent application filed on February 26, 2021 with the application number 202110217491.5 and the invention titled "A touch behavior recognition method, device and device", the entire contents of which are incorporated by reference in in this application.

technical field

The embodiments of the present application relate to the field of artificial intelligence, and in particular, to a touch behavior recognition method, device, and device.

Background technique

Touch Sensing in robots helps robots understand how objects interact in the real world, depending on their weight and stiffness, how a surface feels when touched, how it deforms when touched, and how it moves when pushed . Only by equipping robots with advanced touch sensors, known as "Tactile Sensing" systems, can they become aware of their surroundings, stay away from potentially damaging influences, and provide information for subsequent tasks such as hand manipulation. However, due to the lack of effective application of tactile sensing technology, most of the current robotic interactive technology systems have inaccurate and unstable movements, and the interaction process is "clumsy", which greatly limits their interaction and cognitive abilities.

At present, human-computer interaction based on capacitive touch sensors has developed rapidly. Multi-channel touch sensors can recognize more complex touch behaviors such as sliding and zooming. Multi-channel capacitive touch sensors are generally presented in the form of touch pads and are mostly used in touch screens. In existing robots, touch sensors are generally installed locally to identify the touch behavior of the touched part. For example, a touch sensor is installed on the head of the robot. When a single contact with the head is sensed, the robot may give corresponding feedback (such as playing a certain voice or displaying a certain expression on the screen).

However, each touch action is isolated. When there are uninterrupted touch actions of different types, such as double-clicking and then long-pressing after continuous click, the correct touch action cannot be continuously reported, thus reducing touch action recognition. , thereby reducing the accuracy of touch behavior recognition.

SUMMARY OF THE INVENTION

The present application provides a touch behavior recognition method, related devices and equipment, which are used to jointly determine the touch action type according to the touch duration of the touch action and the vibration amplitude of the touch action, thereby improving the accuracy of touch action type recognition. The improved touch action type determines the touch action, which can improve the accuracy of touch action recognition.

A first aspect of the present application provides a touch behavior recognition method, which is applied to a robot, and the robot includes a capacitive touch sensor, an inertial measurement unit (IMU) and a processor. Based on this, the capacitive touch sensor detects the touch state information of the touch action, the touch state information of the touch action indicates the touch state, or the non-touch state, and then the IMU detects the acceleration information of the touch action, and secondly, the processor according to the touch state of the touch action The information determines the touch duration of the touch action, and determines the touch action type according to the touch duration of the touch action and the vibration amplitude of the touch action. The vibration amplitude of the touch action is determined according to the acceleration information of the touch action, and the touch action type is used to determine touch behavior.

In this embodiment, the touch duration of the touch action is determined by detecting the touch state information of the touch action, and the vibration amplitude of the touch action is determined by the acceleration information of the touch action, and then the touch duration of the touch action and the vibration amplitude of the touch action are jointly determined. The touch action type is used to improve the accuracy of touch action type recognition. Based on this, the touch action type is determined according to the touch action type with improved accuracy, which can improve the accuracy of touch action recognition.

In an optional implementation manner of the present application, the processor first determines, according to the touch duration of each touch action, a time interval threshold corresponding to each touch action, where the time interval threshold is proportional to the touch duration of the touch action, and then processes The controller splits touch actions according to the time interval threshold.

In this embodiment, the time interval threshold between different touch actions is positively correlated with the touch duration of the latest touch action or touch sub-action, thereby enabling adaptive update of the time interval threshold. Based on this, the dynamic time interval threshold setting can better determine the continuity of the touch action.

In an optional implementation manner of the present application, in the touch behavior recognition method provided by the present application, the processor can further determine the touch start time point and the touch end time point of the touch action according to the touch state information of the touch action, And according to the touch start time point and the touch end time point, the touch duration of the touch action and the time interval between every two adjacent touch actions are determined. Based on this, when the time interval is greater than or equal to the first time interval threshold, the processor determines that the touch action is a new touch action, and the first time interval threshold value is corresponding to the previous touch action among the two adjacent touch actions If the time interval is less than the first time interval threshold, the processor determines that the touch action is a touch sub-action.

In this embodiment, the processor can handle the case of continuous changes between touch actions, that is, identify new touch actions and new touch sub-actions that appear in the touch actions, so as to accurately capture transitions between consecutive touch actions, Therefore, different touch actions are no longer isolated from each other, so that the determined result is more in line with the real touch action, thereby improving the accuracy of determining the touch action.

In an optional embodiment of the present application, the robot includes a plurality of touch parts, and the capacitive touch sensor includes a plurality of capacitive touch sensors located at the plurality of touch parts. Based on this, since some touch behaviors involve multiple touch parts, the touch behavior needs to be determined according to the touch actions performed at multiple touch parts. In the touch behavior recognition method provided in this application, the processor can The touch action types of the multiple touch actions of the multiple touch parts determine the touch behavior. For example, a handshake is a touch action of continuing to stroke or long-pressing on the palm of the hand, and a touch action of continuing to stroke or long-pressing on the back of the hand. Based on this, touch actions at different touch parts can be combined into touch actions.

In this embodiment, on the basis that the touch actions sent by the multiple touch parts included in the robot are determined, the type of touch action corresponding to each part can be further determined, so as to realize the recognition of complex touch behaviors, and further improve the touch The accuracy of behavior recognition.

In an optional embodiment of the present application, on the basis that the robot includes multiple touch parts and the capacitive touch sensor includes multiple capacitive touch sensors located at the multiple touch parts, the processor can Touch action types of multiple touch actions, and touch actions are determined according to preset rules.

In this embodiment, a method is provided for determining a touch behavior through preset rules according to touch action types of multiple touch actions, thereby improving the feasibility of touch action recognition.

In an optional implementation manner of the present application, on the basis that the robot includes multiple touch parts, and the capacitive touch sensor includes multiple capacitive touch sensors located at the multiple touch parts, the processor can The touch action type of each touch action, and the touch action is determined according to the classification model, and the classification model is a model obtained by training the historical touch data as training data.

In this embodiment, the historical touch data is used as the training data, and the classification model is obtained through training. Since the historical touch data is used as the model training data, the result of the obtained classification model output is more in line with the real touch behavior. Based on this, According to the touch action types of the multiple touch actions, the touch action is determined by the classification model, which can further improve the feasibility of touch action recognition.

In an optional implementation manner of the present application, the touch duration of the touch action is used to distinguish the type of the touch action according to a preset duration threshold.

In this implementation manner, by distinguishing touch action types according to a preset duration threshold, the feasibility of distinguishing touch action types is improved, thereby improving the feasibility of this solution.

In an optional implementation manner of the present application, the preset duration threshold includes a first duration threshold and a second duration threshold, and the first duration threshold is smaller than the second duration threshold. Based on this, in one case, a touch action in which the touch duration of the touch action is less than the first duration threshold is a tap. In another case, the touch duration of the touch action is greater than or equal to the first duration threshold, and the touch action whose duration is less than the second duration threshold is stroking. In yet another case, a touch action whose touch duration is greater than or equal to the second duration threshold is a long press. In practical applications, the first duration threshold and the second duration threshold between different touch action types are based on the average interval duration of each touch sub-action and the average contact duration of each touch sub-action in each touch action, Determined by conducting experiments and/or statistics based on large amounts of data.

In this embodiment, the preset duration thresholds include different duration thresholds, and different duration thresholds can be divided into different touch duration ranges, so that touch actions in different duration ranges can be refined and classified. The touch duration of the action can determine the touch action type of the touch action, so as to realize the division of different touch action types. On the basis of ensuring the feasibility of the solution, the flexibility of the solution is improved.

In an optional implementation manner of the present application, the vibration amplitude of the touch action is used to distinguish the type of the touch action according to a preset vibration amplitude threshold. In one case, when the vibration amplitude of the touch action is less than the touch action of the vibration amplitude threshold, the touch action type is any one of tapping, stroking, or light pressing. In another case, when the vibration amplitude of the touch action is greater than or equal to the touch action of the vibration amplitude threshold, the touch action type is any one of re-tapping, re-stroking or re-pressing.

In this embodiment, without a pressure sensor, the purpose of re-tapping and tapping, or stroking and re-stroking, or tapping and re-pressing can be achieved, thereby improving the accuracy of detecting the type of touch action.

A second aspect of the present application provides a touch behavior recognition device, comprising:

a detection module, configured to detect touch state information of a touch action, wherein the touch state information of the touch action indicates a touch state, or a non-touch state;

The detection module is also used to detect the acceleration information of the touch action;

a processing module, configured to determine the touch duration of the touch action according to the touch state information of the touch action;

The processing module is further configured to determine the touch action type according to the touch duration of the touch action and the vibration amplitude of the touch action, wherein the vibration amplitude of the touch action is determined according to the acceleration information of the touch action, and the touch action type is used to determine the touch behavior.

In an optional implementation manner of the present application, the processing module is further configured to determine a time interval threshold corresponding to each touch action according to the touch duration of each touch action, wherein the time interval threshold is proportional to the touch of the touch action duration;

The processing module is further configured to split the touch action according to the time interval threshold.

In an optional implementation manner of the present application, the processing module is further configured to determine a touch start time point and a touch end time point of the touch action according to the touch state information of the touch action;

The processing module is further configured to determine the touch duration of the touch action and the time interval between every two adjacent touch actions according to the touch start time point and the touch end time point;

Processing module, specifically for:

When the time interval is greater than or equal to the first time interval threshold, the processor determines that the touch action is a new touch action, where the first time interval threshold is the time corresponding to the previous touch action in the two adjacent touch actions interval threshold;

When the time interval is less than the first time interval threshold, the processor determines that the touch action is a touch sub-action.

In an optional implementation manner of the present application, the processing module is further configured to determine the touch behavior according to the touch action types of the multiple touch actions that occur at the multiple touch locations.

In an optional implementation manner of the present application, the processing module is specifically configured to determine touch behaviors according to preset rules according to touch action types of multiple touch actions that occur at multiple touch locations.

In an optional implementation manner of the present application, the processing module is specifically configured to determine touch behaviors according to a classification model according to touch action types of multiple touch actions occurring at multiple touch locations, wherein the classification model is based on historical touches. The data is the training data, the model obtained through training.

In an optional implementation manner of the present application, the duration threshold includes a first duration threshold and a second duration threshold, and the first duration threshold is smaller than the second duration threshold;

A touch action whose touch duration is less than the first duration threshold is a tap;

The touch duration of the touch action is greater than or equal to the first duration threshold, and the touch action whose duration is less than the second duration threshold is stroking;

A touch action whose touch duration is greater than or equal to the second duration threshold is a long press.

In an optional embodiment of the present application, the vibration amplitude of the touch action is used to distinguish the type of the touch action according to a preset vibration amplitude threshold;

The vibration amplitude of the touch action is less than the vibration amplitude threshold, which is any one of tapping, stroking or tapping;

A touch action whose vibration amplitude is greater than or equal to the vibration amplitude threshold is any one of re-tapping, re-stroking, or re-pressing.

In a third aspect, a robot is provided, including a capacitive touch sensor, an IMU and a processor. The capacitive touch sensor, the IMU and the processor are coupled to the memory, and can be used to execute instructions in the memory, so as to implement the method in any possible implementation manner of the first aspect above. Optionally, the touch behavior recognition device further includes a memory. Optionally, the touch behavior recognition device further includes a communication interface, the processor is coupled to the communication interface, the communication interface is used for inputting and/or outputting information, and the information includes at least one of instructions and data.

In another implementation manner, the touch behavior recognition device is a chip or a chip system configured in the robot. When the touch behavior recognition device is a chip or a chip system configured in the robot, the communication interface may be an input/output interface, an interface circuit, an output circuit, an input circuit, a pin or a related circuit, and the like. The processor may also be embodied as a processing circuit or a logic circuit.

In a fourth aspect, a processor is provided, including: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any one of the possible implementation manners of the first aspect.

In a specific implementation process, the above-mentioned processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a flip-flop, and various logic circuits. The input signal received by the input circuit may be received and input by, for example, but not limited to, a receiver, the signal output by the output circuit may be, for example, but not limited to, output to and transmitted by a transmitter, and the input circuit and output The circuit can be the same circuit that acts as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation manners of the processor and various circuits.

In a fifth aspect, a touch behavior recognition device is provided, including a communication interface and a processor. The communication interface is coupled with the processor. The communication interface is used to input and/or output information. The information includes at least one of instructions and data. The processor is configured to execute a computer program, so that the touch behavior recognition apparatus executes the method in any possible implementation manner of the first aspect.

Optionally, there are one or more processors and one or more memories.

In a sixth aspect, a touch behavior recognition device is provided, including a processor and a memory. The processor is configured to read instructions stored in the memory, and can receive signals through a receiver and transmit signals through a transmitter, so that the apparatus performs the method in any possible implementation manner of the first aspect.

Optionally, there are one or more processors and one or more memories.

Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

In the specific implementation process, the memory can be a non-transitory memory, such as a read only memory (ROM), which can be integrated with the processor on the same chip, or can be separately set in different On the chip, the embodiment of the present application does not limit the type of the memory and the setting manner of the memory and the processor.

It should be understood that the relevant information exchange process, for example, sending a message may be a process of outputting a message from the processor, and receiving a message may be a process of inputting a received message to the processor. Specifically, the information output by the processing can be output to the transmitter, and the input information received by the processor can be from the receiver. Among them, the transmitter and the receiver may be collectively referred to as a transceiver.

The touch behavior recognition device in the fifth aspect and the sixth aspect may be a chip, and the processor may be implemented by hardware or software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. ; When implemented by software, the processor can be a general-purpose processor, and is implemented by reading software codes stored in a memory, which can be integrated in the processor or located outside the processor and exist independently.

In a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code, or instructions), which, when the computer program is executed, causes the computer to execute any one of the above-mentioned first aspects. A method in a possible implementation.

In an eighth aspect, a computer-readable storage medium is provided, the computer-readable storage medium stores a computer program (which may also be referred to as code, or an instruction), when it is run on a computer, causing the computer to execute the above-mentioned first aspect method in any of the possible implementations.

In a ninth aspect, a non-volatile computer-readable storage medium is provided, the non-volatile computer-readable storage medium stores a computer program (also referred to as code, or instructions) when it runs on a computer , so that the computer executes the method in any possible implementation manner of the first aspect.

In a tenth aspect, the present application provides a chip system, the chip system includes a processor and an interface, the interface is used to obtain a program or an instruction, and the processor is used to call the program or instruction to realize or support a robot to realize the first On the one hand the functions involved.

In a possible design, the chip system further includes a memory for storing necessary program instructions and data of the robot. The chip system may be composed of chips, or may include chips and other discrete devices.

It should be noted that, the beneficial effects brought by the embodiments of the second aspect to the tenth aspect of the present application can be understood with reference to the embodiments of the first aspect, and thus are not repeated.

Description of drawings

FIG. 1 is a schematic diagram of an embodiment of a system architecture in an embodiment of the present application;

2 is a schematic diagram of an embodiment of a robot in an embodiment of the present application;

FIG. 3 is a schematic diagram of an embodiment of a touch action and a touch sub-action in an embodiment of the present application;

FIG. 4 is a flowchart of an embodiment of a method for detecting a long-term touch behavior in an embodiment of the present application;

FIG. 5 is a flowchart of an embodiment of a method for detecting the end of a touch action in an embodiment of the present application;

FIG. 6 is a flowchart of an embodiment of a touch behavior recognition method in an embodiment of the present application;

7 is a flowchart of a method for determining a touch action type in combination with an IMU and a capacitive touch sensor in an embodiment of the present application;

FIG. 8 is a schematic diagram of an embodiment of touch behavior recognition based on a classification model in an embodiment of the present application;

FIG. 9 is a schematic structural diagram of a touch behavior recognition device according to an embodiment of the present application.

Detailed ways

In order to make the above objects, technical solutions and advantages of the present application easier to understand, a detailed description is provided below. The detailed description presents various embodiments of devices and/or processes through the use of block diagrams, flowcharts, and/or examples. Since these block diagrams, flowcharts and/or examples contain one or more functions and/or operations, those skilled in the art will understand that these block diagrams, Each function and/or operation within a flowchart or example. The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of this application and the drawings are used to distinguish similar objects and are not necessarily used to Describe a particular order or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

Touch is one of the main ways that humans coordinate interactions. Sense of Touch can help humans evaluate the properties of objects, such as size, shape, texture, temperature, etc. In addition, tactile sense can be used to detect slippage of objects, thereby developing human awareness of the body. Touch transmits various sensory information such as pressure, vibration, pain, and temperature to the central nervous system, helping humans perceive the surrounding environment and avoid potential harm. Research has shown that human touch is better at processing physical features and detailed shapes of objects than sight and hearing. Like humans, touch sensing in robots helps robots understand how objects interact in the real world, which depends on their weight and stiffness, how a surface feels when touched, how it deforms when touched, and how it moves when pushed. . Only by equipping robots with advanced touch sensors, known as "tactile sensing" systems, can they become aware of their surroundings, stay away from potentially damaging influences, and provide information for subsequent tasks such as hand manipulation. However, due to the lack of effective application of tactile sensing technology, most of the current robotic interactive technology systems have inaccurate and unstable movements, and the interaction process is "clumsy", which greatly limits their interaction and cognitive abilities.

Therefore, human-computer interaction based on capacitive touch sensors has developed rapidly. Single-channel touch sensors can generally realize the recognition of single touch/click, double touch/click and long-term touch (long press). Multi-channel touch Sensors can recognize more complex touch behaviors such as swiping and zooming. Multi-channel capacitive touch sensors are generally presented in the form of touch pads and are mostly used in touch screens. In existing robots, touch sensors are generally installed locally to identify the touch behavior of the touched part. For example, a touch sensor is installed on the head of the robot. When a single contact with the head is sensed, the robot may give corresponding feedback (such as playing a certain voice or displaying a certain expression on the screen). However, although the touch sensor can recognize single-click, double-click and long-press detection, each touch action is isolated. When there are uninterrupted different types of touch actions, such as continuous double-click followed by long-press , at this time, the correct touch action cannot be continuously reported, thereby reducing the accuracy of the touch action recognition, thereby reducing the accuracy of the touch behavior recognition.

In order to solve the above problem, an embodiment of the present application provides a touch behavior recognition method, and the touch behavior recognition method is applied to a robot. For ease of understanding, please refer to FIG. 1 , which is a schematic diagram of an embodiment of the system architecture in the embodiment of the present application. In combination with touch detection and vibration detection of each touch part, touch action recognition is performed, thereby improving the accuracy of touch behavior recognition. Spend. In this embodiment, the touched parts include but are not limited to: the top of the head, the left side of the head, the right side of the head, the chest, the left armpit, the right armpit, the stomach, the back of the left hand, the center of the left hand, the back of the right hand, and the center of the right hand. As shown in FIG. 1 , in the system architecture provided by the embodiment of the present application, the user's touch behavior on the robot can be identified according to the touch detection and vibration detection of each touch part.

Since the touch behavior recognition method provided by the embodiment of the present application is applied to the robot, the robot used in the embodiment of the present application will be introduced below. Please refer to FIG. 2 , which is a schematic diagram of an embodiment of the robot in the embodiment of the present application, as shown in FIG. 2, the robot 200 includes one or more capacitive touch sensors; optionally, the robot 200 may further include one or more inertial measurement units (IMUs).

Specifically, the robot 200 can also include a processor. The processor is arranged in the robot 200 . The processor is used to control the operation of the entire robot 200, output control information after receiving, responding or executing relevant control instructions to control the operation of various parts of the robot, and the robot outputs corresponding responses according to the relevant control information. The processor may be referred to as a central processing unit (CPU). The processor may be an integrated circuit chip with logic and/or signal processing capability and computing capability. The processor may also be a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), system on chip (SoC) or other programmable logic device, discrete gates or transistors Logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Further, the robot 200 can also include the shape, frame, and structure of the robot itself, as well as multiple internal and external components. For example, the robot 200 may further include a housing (not shown in FIG. 2 ), a drive assembly (not shown in FIG. 2 ) for driving the robot 200 to move, and a motion assembly (not shown in FIG. 2 ) for moving the robot 200 , such as wheels or legs, etc., an audio component for receiving external sound/voice information and sound (not shown in Figure 2), a display component for displaying image information, text information or facial expression information (not shown in Figure 2) ), an actuator (not shown in FIG. 2 ) for making the robot perform other movements, such as a robotic arm, etc., as well as various electronic components, circuit structures, parts structures, and the like.

The robot 200 provided in this embodiment has a total of 11 touching parts in the whole body. The 11 touching parts of the robot 200 include the top of the head 201, the right side of the head 202, the left side of the head 203, the chest 204, the stomach 205, the right armpit 206, the left Armpit 207, right heart 208, right back 209, left palm 210 and left back 211. In addition, copper skins are attached to the inner shell of the aforementioned 11 touch parts of the robot 200, and a corresponding capacitive touch sensor is installed on the copper skin of each touch part, and each capacitive touch sensor is used to detect the touch action of the corresponding touch part. Secondly, the position of the head touching part of the robot 200 (including the top of the head 201 , the right side of the head 202 and the left side of the head 203 ) is also installed with an IMU, which is used to detect the vibration generated when touching, so as to assist the processor in distinguishing between taps and retakes . Therefore, when an IMU is installed at the position of the head touching part of the robot 200, the touch actions that can be recognized include but are not limited to tapping, continuous tapping, stroking, continuous stroking, long pressing, head re-tapping, head continuous re-tapping, etc. .

It can be understood that this embodiment is introduced with 11 touch parts, and in practical applications, the touch parts of the robot can be further divided according to requirements. Secondly, the IMU introduced in this embodiment is installed on the head of the robot. In practical applications, the IMU can be deployed on any touching part of the robot according to actual requirements. Therefore, the robot shown in FIG. 2 is only used to understand this solution, and should not be understood as a limitation of this solution.

Specifically, the touch behavior recognition method provided by the embodiment of the present application can be applied to various scenarios, and please refer to Table 1 for a specific example.

Table 1

As can be seen from the above introduction, since the touch action detection of each touch part relies on the corresponding capacitive touch sensor, the touch action detection of each touch part is independent of each other, but the algorithm principle used is the same, how to detect The touch action on each touch site is particularly critical.

Therefore, in order to facilitate understanding, the touch action splitting algorithm provided in this embodiment is introduced first. The touch action splitting algorithm is used to detect the touch action (Touch Event) and the touch sub-action (Touch Event) performed by the user on a certain touch part of the robot. Sub-touch Event). The touch action and touch sub-action are described below. Please refer to FIG. 3. FIG. 3 is a schematic diagram of an embodiment of a touch action and a touch sub-action in an embodiment of the present application. As shown in FIG. 3, the horizontal axis shown in FIG. 3 is the time axis, and the vertical line above the time axis is used for For the touch start time point representing the touch action or touch sub-action, for example, the touch start time point 301 , the touch start time point 303 and the touch start time point 305 . The vertical lines below the time axis are used to represent touch termination time points of the touch action or touch sub-action, for example, touch termination time point 302 , touch termination time point 304 , and touch termination time point 306 .

Based on this, in the time period between the adjacent previous touch start time point and the subsequent touch termination time point, the user is in contact with a certain touch part of the robot, for example, the user and a certain touch part of the robot are in contact The contact state is in the time period between the touch start time point 301 and the touch end time point 302, and the user and a certain touch part of the robot are in the time period between the touch start time point 303 and the touch end time point 304. is in a contact state, and the user is in a contact state with a certain touch part of the robot during the time period between the touch start time point 305 and the touch end time point 306 . Secondly, in the time period between the adjacent previous touch termination time point and the subsequent touch start time point, the user is in a non-contact state with a touch part of the robot. For example, the user and a touch part of the robot are in a non-contact state. The non-contact state is in the time period between the touch end time point 302 and the touch start time point 303, and the time period between the touch end time point 304 and the touch start time point 305 between the user and a certain touch part of the robot in a non-contact state.

For example, if the user performs touch action A and touch action B on the robot. The touch action B is a touch action between the user and a touch part of the robot during the time period between the start time point 305 and the touch end time point 306 , and the touch action B may include a touch state. The touch action A includes a touch sub-action 1 and a touch sub-action 2. The touch sub-action 1 is the touch between the user and a certain touch part of the robot during the time period between the touch start time point 301 and the touch end time point 302. The touch sub-action 2 is a touch action between the user and a certain touch part of the robot in the time period between the touch start time point 303 and the touch end time point 304 . Although there is a non-touch state time interval between touch sub-action 1 and touch sub-action 2 (ie, the time period between the touch end time point 302 and the touch start time point 303 ), the time interval is smaller than that of touch action A and The time interval between the touch actions B (ie, the time period between the touch end time point 304 and the touch start time point 305 ). Therefore, the touch action A is the touch action between the user and a touch part of the robot in the time period between the touch start time point 301 and the touch end time point 304, and the touch action A may include a touch state and a non-touch state .

Optionally, the touch action A shown in FIG. 3 may correspond to the touch action of continuous tapping, and the touch action B may correspond to the touch action of stroking or the touch action of long pressing. It can be understood that the example in FIG. 3 is only used to understand touch actions and touch sub-actions. In practical applications, each touch action may include multiple touch sub-actions, and the number of touch sub-actions should not be construed as a limitation to this solution. limited.

Because the capacitive touch sensor can detect the touch state between the user and a touch part of the robot based on the change of capacitance, and report to the processor that the touch state between the user and a touch part of the robot, for example, is in the touch state When the touch state flag "1" is reported, in the non-touch state, the touch state flag "0" is reported. When the capacitive touch sensor detects that there is a touch state between the user and a touch part of the robot, it reports the touch part and the touch state to the processor. The status flag is "1", therefore, after the processor receives the touch state flag "1", it can determine that a touch behavior has occurred at a certain touch part of the robot. On the contrary, when the capacitive touch sensor detects that the user and a certain touch part of the robot are in a non-touch state, it reports the touch part and the touch state flag "0" to the processor. Therefore, after the processor receives the touch state flag "0" , it can be determined to stop the touch behavior at a certain touch part of the robot. Based on this, the method for detecting a long-time touch action (which may be referred to as "detectLongTouch") provided by this embodiment will be introduced below to determine whether the touch between the user and a certain touch part of the robot is the start of a new touch action. Please refer to FIG. 4 , which is a flowchart of an embodiment of a method for detecting a long-term touch behavior in an embodiment of the present application, as shown in FIG. 4 , and the specific steps are as follows.

Step S400, the processor starts long-term touch behavior detection.

Step S401, the processor determines whether the last touch action ends.

In this embodiment, the processor needs to determine whether the last touch action ends. If yes, step S402 is executed. If not, step S403 is executed.

Specifically, after the capacitive touch sensor senses a touch action on a certain touch portion, the processor may determine whether the previous touch action ends by using a time interval threshold. Optionally, the time interval threshold may be adaptively adjusted according to the duration and/or time interval of the last touch action and/or the previous one or more touch actions, or dynamically changed, and each time interval threshold may be It is adaptively adjusted according to artificially formulated rules, or it can be automatically calculated based on historical data and learning models using artificial intelligence technology.

If the time interval between the current touch action and the previous touch action is greater than or equal to the time interval threshold, it can be determined that the previous touch action has ended, and the current touch action is a new touch action. Specifically, the time interval between the current touch action and the previous touch action is the touch start time point of the current touch action and the touch end time point of the previous touch action. For example, referring to FIG. 3 again, when the time interval between the touch end time point 304 of the touch action A and the touch start time point 305 of the touch action B is greater than or equal to the time interval threshold, it can be determined through step S401 that the previous The touch action has ended, and the current touch action is a new touch action, that is, the previous touch action is touch action A, and the current touch action is touch action B.

Similarly, if the time interval between the current touch action and the previous touch action is less than the time interval threshold, it can be determined that the previous touch action has not ended, and the current touch action is a new touch sub-action of the previous touch action. For example, referring to FIG. 3 again, when the time interval between the touch termination time point 302 of the touch sub-action 1 and the touch start time point 303 of the touch sub-action 2 is less than the time interval threshold, it can be determined through step S401 that the A touch action has not ended, and the current touch action is not a new touch action, then the previous touch action may be touch action A including touch sub-action 1, and the current touch action is touch sub-action 2. Before touch sub-action 2 occurs (ie, before touch start time point 303 ), touch action A includes touch sub-action 1 but not touch sub-action 2 , and after touch sub-action 2 occurs (eg, touch end time point 304 ) After or after the touch start time point 303 ), touch action A includes both touch sub-action 1 and touch sub-action 2 , and touch sub-action 2 is a new touch sub-action of touch action A.

In a possible implementation manner, the time interval threshold may be positively correlated with the touch duration of a touch action or touch sub-action that is the closest to the current touch action. Specifically, the time interval threshold between different touch actions can be determined by formula (1):

I=min(k·T _touch +I _min ,I _max ); (1)

Wherein, k indicates an adjustable parameter, T _touch indicates the touch duration of the most recent touch action or touch sub-action of the current touch action, I _min indicates the preset shortest time interval, and I _max indicates the preset maximum time interval.

For example, referring to FIG. 3 again, if the previous touch action is touch action A and the current touch action is touch action B, then T _touch indicates the time period from touch start time point 303 to touch end time point 304 . Assume that the touch duration of the touch sub-action 2 (T _touch in this example) is 100 milliseconds (ms), the duration between the touch termination time point 304 and the touch start time point 305 , that is, the time interval, is 350 ms, and the preset shortest time The interval I _min is 50 ms, the preset maximum time interval I _max is 2 seconds (s), and the adjustable parameter k is configured to be 0.7, then the time interval threshold calculated according to formula (1) is 120 ms. The time interval of 350ms is greater than the time interval threshold of 120ms, so it can be determined that the touch action A has ended and the touch action B is a new touch action.

Step S402, the processor determines the touch start time point of the current touch action.

In this embodiment, before step S402 is executed, the processor first determines through step S401 that a new touch action is started, that is, the previous touch action has ended. Therefore, the current touch action is a new touch action after the previous touch action is completed, and at this time, the processor can determine the start time of the current touch action. Exemplarily, referring to FIG. 3 again, if the previous touch action is touch action A and the current touch action is touch action B, the processor will determine the touch start time point 305 .

Step S403, the processor determines the current touch action as a touch sub-action of the previous touch action.

In this embodiment, before step S403 is executed, the processor first determines through step S401 that no new touch action occurs, that is, the previous touch action has not ended. Therefore, the current touch action will be determined as the touch sub-action of the previous touch action. Optionally, the processor can also determine the touch start time point of the touch sub-action and/or the identification information of the touch sub-action, and the like. Exemplarily, please refer to FIG. 3 again, if the previous touch action is touch action A including touch sub-action 1, and the current touch action is touch sub-action 2, then touch sub-action 2 will be determined to be included in touch sub-action A. A new touch sub-action, then the processor can determine the touch start time point 303 of the touch sub-action 2 .

In this case, the number of touch sub-actions included in the "last touch action" is increased by one. For example, referring to FIG. 3 again, before and after touch sub-action 2 occurs, the number of touch sub-actions included in touch action A is increased from one (touch sub-action 1) to two (touch sub-action 1 and touch sub-action 2). ). Based on this, after determining the touch sub-action, the processor can also update the number of intervals of the touch action (which can be recorded as "numIntervals") and the average interval duration of each touch sub-action in the touch action (which can be recorded as "meanTouchIntervals"). For example, it is assumed that the previous touch action includes the first touch sub-action and the second touch sub-action, and the time interval between the first touch sub-action and the second touch sub-action is 80 ms. In step S403, the current touch action determines the third touch sub-action of the previous touch action, and the time interval between the second touch sub-action and the third touch sub-action is 100ms, then the processor determines the interval number of the previous touch action Update from 1 to 2, and update the average interval duration of the last touch action from 80sm to (80+100)/2=90ms.

Based on this, when the touch action changes, for example, continuous tapping is changed to continuous stroking, at this time, the average interval duration of each touch sub-action will gradually become longer. Therefore, when the processor determines the type of the touch action, it can also refer to the average interval duration obtained after updating to make determination, so as to satisfy the possibility of different touch actions and improve the accuracy of determining the type of the touch action.

Step S404, the processor starts the next long-term touch behavior detection.

In this embodiment, after the processor completes step S403 or step S402, it will start the next long-term touch behavior detection, that is, repeatedly execute the process shown in FIG. 4 . When the next touch behavior occurs, the processor can determine whether a new touch action occurs in a manner similar to step S401, and then complete the reporting of the touch start time of the new touch action in a manner similar to step S402, or through the same method as step S402. In a similar manner to S403, reporting of the result of determining the current touch action as a new touch sub-action included in the previous touch action is completed.

The long-term touch action detection method has been introduced above. The following will introduce the touch action end detection method (which may be referred to as "detectTouchEvent") provided by the embodiment of the present application, which is used to determine the contact between the user and a certain touch part of the robot. Whether the touch action has ended. Please refer to FIG. 5 , which is a flowchart of an embodiment of a method for detecting the end of a touch action in an embodiment of the present application, as shown in FIG. 5 , and the specific steps are as follows.

Step S500, the processor starts the touch action end detection.

Step S501, the processor determines whether a new touch sub-action occurs.

In this embodiment, when a touch action occurs, after the capacitive touch sensor senses a touch action on a touch part, the processor can determine whether a new touch action occurs in a manner similar to step S401, that is, according to the time The interval threshold is used for judgment, and the description is not repeated here.

If the determination in step S501 is yes, it means that no new touch action occurs, so the current touch action can be determined as a touch sub-action in a manner similar to step S403, that is, it is determined that a new touch sub-action occurs, and step S502 is executed. 3 , when the time interval between the touch termination time point 302 of the touch sub-action 1 and the touch start time point 303 of the touch sub-action 2 is less than the time interval threshold, it can be determined by step S501. The last touch action has not ended, and the current touch action is a new touch sub-action. Then, the previous touch action may be touch action A including touch sub-action 1, and the current touch action is touch sub-action 2. Before touch sub-action 2 occurs (ie, before touch start time point 303 ), touch action A includes touch sub-action 1 but not touch sub-action 2 , and after touch sub-action 2 occurs (eg, touch end time point 304 ) After or after the touch start time point 303 ), touch action A includes both touch sub-action 1 and touch sub-action 2 , and touch sub-action 2 is a new touch sub-action of touch action A.

If the last touch action (last touch sub-action) is touch sub-action 1, then the current touch action (new touch sub-action) is touch sub-action 2.

Next, if the determination in step S501 is NO, it means that a new touch action has occurred, so it is determined that no new touch sub-action has occurred, and step S503 is executed.

Step S502, the processor determines the result.

In this embodiment, before step S502 is executed, the processor first determines through step S501 to start a new touch sub-action. Therefore, the current touch action is a new touch sub-action in the previous touch action, and the processor determines this result. Optionally, the processor may further determine the touch start time point of the touch sub-action and/or determine the identification information of the touch sub-action, and the like. For example, referring to FIG. 3 again, if the processor determines that the new touch sub-action is touch sub-action 2 , the processor can determine the touch start time point 303 of touch sub-action 2 .

In this case, similar to the foregoing implementation, the number of touch sub-actions included in the "last touch action" is increased by one. For example, referring to FIG. 3 again, before and after touch sub-action 2 occurs, the number of touch sub-actions included in touch action A is increased from one (touch sub-action 1) to two (touch sub-action 1 and touch sub-action 2). ). Based on this, after the processor determines the result, it can also update the number of touch sub-actions in the touch action (can be recorded as "numTouches"), and the average touch duration of each touch sub-action in the touch action (can be recorded as "meanTouchDuration" ). For example, assuming that the previous touch action includes a first touch sub-action and a second touch sub-action, the touch duration of the first touch sub-action is 200 milliseconds, and the touch duration of the second touch sub-action is 240 milliseconds. In step S502, the current touch action determines the third touch sub-action of the previous touch action, and the touch duration of the third touch sub-action is 280 milliseconds, then the processor updates the number of touch sub-actions of the previous touch action from 2 to 3, and the average touch duration of each touch sub-action in the previous touch action is updated from (200+240)/2=220 milliseconds to (200+240+280)/3=240 milliseconds.

Step S503, the processor determines the touch termination time point of the touch action.

In this embodiment, before step S503 is executed, the processor first determines through step S501 that no new touch sub-action occurs, that is, the previous touch action has ended, and starts a new touch action. Therefore, the current touch action is a new touch action after the previous touch action is completed. If the previous touch action includes multiple touch sub-actions, the processor may determine the touch termination time of the last touch sub-action in the previous touch action as the touch termination time of the touch action. Illustratively, referring to FIG. 3 again, the previous touch action (touch action A) has ended, and a new touch action (touch action B) occurs. Since the last touch sub-action in touch action A is touch sub-action 2, and the touch termination time point of touch sub-action 2 is 304, it can be determined that the touch termination time point of touch action A is 304. Optionally, the processor may also determine the touch start time point 303 of the touch sub-action 2, or the touch duration from the touch start time point 303 to the touch end time point 304 in the touch sub-action 2, and the like.

If the touch action does not include a touch sub-action, after the capacitive touch sensor senses a touch action on a touch part, the processor can receive the touch status flag "1" reported by the capacitive touch sensor, and after the capacitive touch sensor senses the touch status flag "1" When the touch action is stopped for a certain touch part, the processor can receive the touch state flag "0" reported by the capacitive touch sensor, and the time point when the touch state flag "0" is received is the touch termination time of the touch action. Exemplarily, please refer to FIG. 3 again, for a new touch action (touch action B), since the touch action B does not include a touch sub-action, the processor receives the touch state identifier "reported by the capacitive touch sensor" at time point 305. 1", at this time the processor can determine that the time point 305 is the touch start time point 305 of the touch action B, and the processor receives the touch status flag "0" reported by the capacitive touch sensor at the time point 306, and the processor can determine The time point 306 is the touch termination time point of the touch action B. Optionally, the processor may further determine the touch start time point 305 of the touch action B, or the touch duration from the touch start time point 305 to the touch end time point 306 in the touch action B, and the like.

Step S504, the processor starts the next touch action end detection.

In this embodiment, after the processor completes step S503 or step S502, it will start the next touch action end detection, that is, repeatedly execute the process shown in FIG. 5 . When the next touch action occurs, the processor can determine whether a new touch sub-action occurs in a manner similar to step S501, and then complete the reporting of the result of determining the new touch sub-action in a manner similar to step S502, or through The reporting of the touch termination time for the new touch sub-action is completed in a manner similar to step S503.

In some possible implementations of the embodiments shown in FIG. 3 and FIG. 5 , the time interval threshold between different touch actions is positively correlated with the touch duration of the latest touch action or touch sub-action, so that the time interval threshold can be adjusted. Adaptive update. Based on this, the dynamic time interval threshold setting can better determine the continuity of touch actions, and the touch action splitting algorithm proposed in the embodiments of the present application can handle the situation of continuous changes between touch actions. For example, traditional recognition algorithms can only recognize single clicks, double clicks, and long presses, and each action is isolated. The touch action splitting algorithm proposed in the present invention can continuously identify the previous touch action and the current touch action, and can accurately capture the transition between consecutive touch actions, so different touch actions are no longer isolated from each other, so that The determined result is more in line with the real touch action.

Based on this, the following will introduce the touch behavior recognition method in the embodiment of the present application based on the touch action detection method introduced in the foregoing embodiments. Please refer to FIG. 6. FIG. 6 is a flowchart of an embodiment of the touch behavior recognition method in the embodiment of the present application. As shown in the figure, the touch behavior recognition method in the embodiment of the present application can be applied to the robot shown in FIG. 2, based on the Therefore, the specific steps of the touch behavior recognition method are as follows.

Step S601, the processor determines the touch duration of the touch action.

In this embodiment, the processor may implement the detection of a long-time touch action by executing steps S401 to S404 shown in FIG. 4 when the touch starts. Secondly, the detection of the end of the touch action can be realized by executing steps S501 to S504 shown in FIG. 5 when the touch ends.

Based on this, since the processor can determine the touch start time and touch end time of the touch action in the long-time touch action detection method and the touch action end detection method, the processor can determine the touch start time and touch end time of the touch action based on the touch start time and touch end time of the touch action. The time is calculated to obtain the touch duration of the touch action, so as to achieve the purpose of determining the touch duration of the touch action.

Specifically, when the user starts to touch a certain part of the robot, the processor will execute the touch action start algorithm (which can be denoted as "actionOnTouch"), and call the long-time touch action detection shown in FIG. 4 through the touch action start algorithm. (Can be recorded as "detectLongTouch") method to achieve the purpose of long-term touch action detection. Secondly, when the user ends the touch action on a certain part of the robot, the processor will execute the touch action end algorithm (which can be denoted as "actionOnQuit"), and call the touch action end detection shown in FIG. Denoted as "detectTouchEvent") method, to achieve the purpose of detecting the end of the touch action.

Step S602, the processor determines the type of the touch action according to the touch duration of the touch action.

In this embodiment, different touch action types may be divided according to the touch duration of the touch action. For example, the touch duration corresponding to the touch action type A is the touch duration range A, the touch duration corresponding to the touch action type B is the touch duration range B, the touch duration corresponding to the touch action type C is the touch duration range C, and the touch action type D corresponds to the touch duration range C. The touch duration is the touch duration range D. When the touch duration of the touch action acquired in step S601 is in the touch duration range C, it can be determined that the touch action type is touch action type C.

In a possible implementation manner, the touch action type of a touch action with a touch duration less than 200ms may be defined as "pat", and the touch action type of a touch action with a touch duration greater than or equal to 200ms and less than 2s may be defined as "stroking" , and define the type of touch action whose touch duration is greater than or equal to 2s as "long press". It should be understood that in this embodiment, two duration thresholds of 200ms and 2s are used as the division of the three types of touch actions: stroking, stroking, and long pressing, which is only an example of a possible implementation, not a limitation; Whether the divided time interval is an open interval or a closed interval does not constitute a limitation. In practical applications, the time interval division between different touch action types may be based on the average interval duration of each touch sub-action in the touch action obtained in the foregoing embodiment and the time interval of each touch sub-action in each touch action. The average duration of exposure, determined by conducting experiments and/or statistics based on a large number of data.

Optionally, the touch duration ranges corresponding to different touch action types may be the same or overlap, and the touch action type cannot be accurately determined only based on the touch duration range. The above-mentioned other information can be, for example, the motion state of the robot (such as acceleration), the sound/voice information obtained by the robot, the pictures or videos collected by the image acquisition device of the robot, or the information obtained by the robot from other electronic devices (such as sound/voice information, sensor information, picture or video information) and other information.

In some scenarios, some touch actions cannot be accurately distinguished only by the touch state identifier reported by the capacitive touch sensor. For example, it is difficult to distinguish re-tap and tap only by the touch state identifier reported by the capacitive touch sensor. At this time, the acceleration information detected by the IMU can be used, and then the processor determines the vibration amplitude of the touch action according to the acceleration information, and determines the touch action according to the vibration amplitude of the touch action and the touch duration of the touch action. For example, if the vibration amplitude of tapping on the touched part is smaller than the vibration amplitude threshold, it can be further determined that the touch action type is tap. If the vibration amplitude of the tapping at the touched part is greater than or equal to the vibration amplitude threshold, it can be further determined that the touch action type is re-tapping. It should be understood that in practical applications, the IMU can further distinguish more touch actions, such as stroking and heavy stroking, or light pressing and heavy pressing, and the vibration amplitude thresholds corresponding to different touch actions are also different. The touch action determined based on the information detected by the IMU and the capacitive touch sensor can be more accurate.

FIG. 7 exemplarily shows a flowchart of a method for determining a touch action type in combination with an IMU and a capacitive touch sensor provided by an embodiment of the present application. As shown in FIG. 7 , the method for determining a touch action type specifically includes the following steps.

Step S700, the processor starts the identification of the touch action type.

Step S701, the processor determines the touch action type.

In this embodiment, the processor determines the type of the touch action according to the touch duration of the touch action in a similar manner to the foregoing embodiment.

Step S702, the processor determines the magnitude relationship between the vibration amplitude and the vibration amplitude threshold.

In this embodiment, the IMU detects the acceleration information of the touch action, and reports the acceleration information of the touch action to the processor included in the robot. The processor determines the vibration amplitude of the touch action according to the acceleration information of the touch action, and determines the vibration amplitude and the vibration amplitude. The size of the threshold relationship.

Step S703, whether the vibration amplitude is greater than or equal to the vibration amplitude threshold.

In this embodiment, the processor determines whether the vibration amplitude of the touch action is greater than or equal to the vibration amplitude threshold, and if so, executes step S704. If not, step S705 is executed.

S704, the processor modifies the touch action type to heavy.

In this embodiment, it can be known from step S703 that the vibration amplitude of the touch action is greater than or equal to the vibration amplitude threshold. Based on this, if it is determined in step S701 that the type of touch action is tap, then combined with the result that the vibration amplitude of the touch action is greater than the vibration amplitude threshold, it can be further determined that the type of touch action is re-tapping, and the touch action determined in step S701 Action type corrected from "beat" to "retake". Similarly, if the touch action type is determined to be stroking in step S701, then combined with the result that the vibration amplitude of the touch action is greater than the vibration amplitude threshold, it can be further determined that the touch action type is re-stroking, and the determined in step S701 is Corrected touch action type from "stroking" to "stroking". Other touch action types can also be corrected based on a similar manner in step S704, and the correction of all touch action types is not exhaustive here.

S705, the processor modifies the touch action type to light.

In this embodiment, it can be known from step S703 that the vibration amplitude of the touch action is less than the vibration amplitude threshold. Based on this, if it is determined in step S701 that the type of touch action is tap, then combined with the result that the vibration amplitude of the touch action is less than the vibration amplitude threshold, it can be further determined that the type of touch action is tap, and the touch action determined in step S701 Action type corrected from "Tap" to "Tap". Similarly, if it is determined in step S701 that the touch action type is a long press, then combined with the result that the vibration amplitude of the touch action is less than the vibration amplitude threshold, it can be further determined that the touch action type is a tap, and the determined in step S701 Fixed touch action type from "long press" to "tap". Other touch action types can also be corrected based on a similar manner in step S705, and the correction of all touch action types is not exhaustive here.

S706, the processor outputs the corrected touch action type.

In this embodiment, after step S704 or step S705 is completed, the modified touch action type is output, so that the processor can determine the touch action according to the modified touch action type, so as to further improve the accuracy of the determination of the touch action. For example, if the touch action type determined in step S701 is modified from "tap" to "re-tap" in step S704, the modified touch action type "re-tap" will be output in step S706. Similarly, if the touch action type determined in step S701 is modified from "long press" to "tap" in step S705, the modified touch action type "tap" will be output in step S706.

Therefore, in some scenarios introduced in step S602, that is, in the absence of a pressure sensor, the acceleration information detected by the IMU and the touch state identifier reported by the capacitive touch sensor can be used to achieve re-tapping and tapping, or light-tapping. The purpose of stroking and re-stroking, or tap and re-tap, and improve the accuracy of touch action type detection.

Step S603, the processor determines the touch behavior according to the touch action type.

In this embodiment, the touch action type of each touch part can be determined through step S602. Since some touch behaviors only involve one touch part, the touch behavior can be determined according to the touch action performed on one touch part. For example, stroking the head is the touch action of stroking the head, and patting the belly is patting the belly. touch action.

However, some touch behaviors involve multiple touch parts, so the touch behavior needs to be determined according to the touch actions performed at multiple touch parts. Continue the touch action of stroking or long pressing. Based on this, touch actions at different touch parts can be combined into touch actions.

Since the robot shown in FIG. 2 introduced in this embodiment includes 11 touch parts, and each touch part has a high degree of discrimination, the rules for fusion perception can be encoded and formulated. However, when the number of parts touched by the robot is very large (such as electronic skin), coding the rules for fusion perception will be labor-intensive and the generalization ability may be poor. At this time, artificial intelligence technology can be used to train a classification model that can identify different complex touch behaviors driven by a large amount of data. Optionally, the above classification model may be a learning model such as a machine learning model, a deep learning model or a reinforcement learning model. In the following, the touch behavior recognition method based on the preset rule and the touch behavior recognition method based on the classification model will be introduced respectively.

1. Touch behavior recognition based on preset rules

In this embodiment, the preset rule is a rule formulated by humans through coding. For example, if (touch action 1 in touch part 3) and (touch action 4 in touch part 6) then touch behavior 1, it means that if touch action 1 is performed on touch part 3 and touch action 4 is performed on touch part 6, it is a touch Behavior 1. Or, if (touch action 1 in touch part 4) and (touch action 1 in touch part 3) then touch behavior 3, it means that if touch action 1 is performed on touch part 4 and touch action 1 is performed on touch part 3, it is a touch Behavior 1.

It can be understood that the foregoing example is only a rule made by humans. In practical applications, the preset rule is not limited to two single-part touch actions, and may be a touch action formed by touching one or more parts. For example, (stroking in the left palm) and (stroking in the back of the left hand) and (stroking in the right heart) and (stroking in the back of the right hand) then hold both hands. It means stroking on the center of the left hand, stroking on the back of the left hand, stroking on the center of the right hand, and stroking on the back of the right hand to hold both hands. Therefore, the touch action and the number of touch parts should not be construed as a limitation of this application.

2. Touch Behavior Recognition Based on Classification Model

In this embodiment, the touch behavior of a plurality of touch parts is used as the input of the to-be-trained touch behavior recognition model, and the preset touch behavior is used as the output, and the training is performed by a common model training method. Exemplarily, if the robot is equipped with capacitive touch sensors at N touch locations, please refer to FIG. 8 , which is a schematic diagram of an embodiment of touch behavior recognition based on a classification model in an embodiment of the present application. As shown in FIG. 8 , the touch actions detected at the N touch parts are used as the input of the touch behavior recognition model (a classification model for classifying touch behaviors) obtained after the training is completed. The touch behavior recognition model can be based on the above input , which outputs the touch behavior category. Specifically, as shown in FIG. 8 , the touch part 1 and the touch part N are tapped, and the touch part 2 and the touch part N-1 are not touched (in FIG. The information such as stroking at the touch part 3 is used as the input of the touch behavior recognition model, and the touch behavior recognition model can determine the category of the touch behavior according to the above input, and finally output the classification result touch behavior 4 .

It can be understood that the foregoing examples are all used to understand this solution, and the touch behavior corresponding to the touch actions detected by the multiple touch parts needs to be flexibly determined according to the actual situation.

The touch behavior recognition method in the embodiment of the present application can realize the recognition of complex touch behaviors of multiple touch parts, for example, handshake and Hugs, etc., further improve the accuracy and reliability of touch behavior recognition.

In some possible implementations, the touch sensors in the above embodiments of the present application may also be other types of sensors other than capacitive sensors, for example, touch sensors based on surface acoustic waves and infrared.

In some possible implementations, the IMU in the above-mentioned embodiments of the present application may also be other types of sensors that can detect the motion information of the robot. For example, a contact-type microelectromechanical system (microelectromechanical system, MEMS) microphone can also realize vibration. detection.

In some possible implementation manners, the robot provided by the above embodiments of the present application can react according to the user's touch action type or touch behavior determined by the robot processor to realize anthropomorphic interaction. For example, if the robot processor determines that the user has made a "hold hands" touch behavior on the robot, the display screen on the robot head can display a smiley expression, and the robot speaker can play the voice "Do you want to play a game with me?". For another example, if the robot processor determines that the user has performed a "continuous heavy blow" touch action on the robot, the display screen of the robot head may display a crying expression, and the robot speaker will play a voice of "Ah, it hurts". Based on the touch behavior recognition method provided by the embodiments of the present application, the robot can accurately recognize and distinguish the continuous actions of the user, improve the accuracy of touch behavior recognition, and then make more accurate responses, conduct more realistic interactions with users, and improve the user experience. experience.

In some possible implementations, the methods provided by the above embodiments of the present application can be applied to any interactive electronic device, such as mobile phones, foldable electronic devices, tablet computers, desktop computers, laptop computers, in addition to robots. Top computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, Virtual reality (VR) equipment, artificial intelligence (AI) equipment, wearable equipment, brain-computer interface equipment, in-vehicle equipment, smart home equipment, smart medical equipment, smart fitness equipment or smart city equipment, etc.

Embodiments of the present application provide a robot, where the robot includes one or more touch sensors and a processor.

Among them, one or more touch sensors are used to detect changes in capacitance when a touch behavior is performed.

The processor is configured to execute the methods shown in FIG. 4 , FIG. 5 , FIG. 6 and FIG. 7 .

Optionally, the above-mentioned robot further includes an IMU, and the above-mentioned IMU is used to detect acceleration information;

The above-mentioned processor is further configured to determine the vibration amplitude of the touch action according to the acceleration information.

The solutions provided by the embodiments of the present application have been introduced above mainly from the perspective of methods. It can be understood that, in order to realize the above-mentioned functions, the touch behavior recognition apparatus includes corresponding hardware structures and/or software modules for executing each function. Those skilled in the art should easily realize that the present application can be implemented in hardware or in the form of a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

The embodiments of the present application may divide the touch behavior recognition device into functional modules based on the foregoing method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. It should be noted that, the division of modules in the embodiments of the present application is schematic, and is only a logical function division, and there may be other division manners in actual implementation.

Therefore, the touch behavior recognition device in the present application will be described in detail below, and the touch behavior recognition device is set in the robot. Please refer to FIG. 9 , which is a schematic structural diagram of the touch behavior recognition device in the embodiment of the application, as shown in the figure. As shown, the touch behavior recognition device 900 includes:

A detection module 901, configured to detect touch state information of a touch action, wherein the touch state information of the touch action indicates a touch state, or a non-touch state;

The detection module 901 is further configured to detect the acceleration information of the touch action;

a processing module 902, configured to determine the touch duration of the touch action according to the touch state information of the touch action;

The processing module 902 is further configured to determine the touch action type according to the touch duration of the touch action and the vibration amplitude of the touch action, wherein the vibration amplitude of the touch action is determined according to the acceleration information of the touch action, and the touch action type is used to determine the touch behavior .

In an optional implementation manner, on the basis of the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition device 900 provided in this embodiment of the present application, the processing module 902 is further configured to For the touch duration of the touch action, a time interval threshold corresponding to each touch action is determined, wherein the time interval threshold is proportional to the touch duration of the touch action;

The processing module 902 is further configured to split the touch action according to the time interval threshold.

In an optional implementation manner, on the basis of the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition device 900 provided in this embodiment of the present application, the processing module 902 is further configured to perform a touch action according to the touch action touch state information, determine the touch start time point and touch end time point of the touch action;

The processing module 902 is further configured to determine the touch duration of the touch action and the time interval between every two adjacent touch actions according to the touch start time point and the touch end time point;

The processing module 902 is specifically used for:

In an optional implementation manner, on the basis of the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition apparatus 900 provided in this embodiment of the present application, the processing module 902 is further configured to The touch action types of the multiple touch actions of the multiple touch parts determine the touch behavior.

In an optional implementation manner, on the basis of the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition device 900 provided in this embodiment of the present application, the processing module 902 is specifically configured to For the touch action types of the multiple touch actions of the multiple touch parts, the touch behavior is determined according to a preset rule.

In an optional implementation manner, on the basis of the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition device 900 provided in this embodiment of the present application, the processing module 902 is specifically configured to For the touch action types of the multiple touch actions of the multiple touch parts, the touch behavior is determined according to a classification model, wherein the classification model is a model obtained by training the historical touch data as training data.

In an optional implementation manner, based on the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition device 900 provided by this embodiment of the present application, the touch duration of the touch action is used according to a preset The duration threshold for distinguishing touch action types.

In an optional implementation manner, on the basis of the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition apparatus 900 provided in this embodiment of the present application, the duration threshold includes a first duration threshold and a second duration threshold. duration threshold, the first duration threshold is less than the second duration threshold;

In an optional implementation manner, based on the embodiment corresponding to FIG. 9 above, in another embodiment of the touch behavior recognition device 900 provided in this embodiment of the present application, the vibration amplitude of the touch action is used according to a preset The vibration amplitude threshold of distinguishing the type of touch action;

An embodiment of the present application provides a robot, including at least one capacitive touch sensor, an IMU, at least one processor, a memory, and an input-output (I/O) interface;

the at least one capacitive touch sensor, the IMU and the at least one processor are coupled to the memory and the input-output interface;

The at least one capacitive touch sensor, the IMU and the at least one processor are used to execute a computer program stored in the memory to execute the method executed by the at least one capacitive touch sensor, the IMU and the at least one processor in any of the above method embodiments.

The present application further provides a touch behavior recognition device, including at least one processor, where the at least one processor is configured to execute a computer program stored in a memory, so that the touch behavior recognition device executes the capacitance in any of the foregoing method embodiments. A touch sensor, an IMU, and a method performed by a processor.

It should be understood that the above-mentioned touch behavior recognition device may be one or more chips. For example, the touch behavior recognition device may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a system on chip (SoC), It can also be a central processing unit (CPU), a network processor (NP), a digital signal processing circuit (DSP), or a microcontroller (microcontroller). controller unit, MCU), it can also be a programmable logic device (PLD) or other integrated chips.

The embodiment of the present application also provides a touch behavior recognition device, which includes a processor and a communication interface. The communication interface is coupled with the processor. The communication interface is used to input and/or output information. The information includes at least one of instructions and data. The processor is configured to execute a computer program, so that the touch behavior recognition apparatus executes the method executed by the capacitive touch sensor, the IMU and the processor in any of the above method embodiments.

The embodiments of the present application also provide a touch behavior recognition device, which includes a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the touch behavior recognition apparatus executes the capacitive touch sensor, the IMU and the processor in any of the foregoing method embodiments method performed.

In the implementation process, each step of the above-mentioned method can be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The steps of the methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, detailed description is omitted here.

It should be noted that the processor in this embodiment of the present application may be an integrated circuit chip, which has a signal processing capability. In the implementation process, each step of the above method embodiments may be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software. The aforementioned processors may be general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components . The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

It can be understood that the memory in this embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically programmable Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory may be random access memory (RAM), which acts as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) ) and direct memory bus random access memory (direct rambus RAM, DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

According to the method provided by the embodiment of the present application, the present application also provides a computer program product, the computer program product includes: computer program code, when the computer program code is run on a computer, the computer is made to execute FIG. 2 , FIG. 5 and A method performed by each unit in the embodiment shown in FIG. 9 .

According to the method provided by the embodiment of the present application, the present application also provides a computer-readable storage medium, where the computer-readable storage medium stores program codes, and when the program codes are run on a computer, the computer is made to execute FIG. 2 , and FIG. 5 and the method performed by each unit in the embodiment shown in FIG. 9 .

The modules in the above-mentioned device embodiments correspond to the units in the method embodiments completely, and the corresponding modules or units perform corresponding steps. Other steps may be performed by a processing unit (processor). For functions of specific units, reference may be made to corresponding method embodiments. The number of processors may be one or more.

The terms "component", "module", "system" and the like are used in this specification to refer to a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computing device and the computing device may be components. One or more components may reside within a process and/or thread of execution, and a component may be localized on one computer and/or distributed between 2 or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. A component may, for example, be based on a signal having one or more data packets (eg, data from two components interacting with another component between a local system, a distributed system, and/or a network, such as the Internet interacting with other systems via signals) Communicate through local and/or remote processes.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units may refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

A touch behavior recognition method, the method is applied to a robot, the robot includes a capacitive touch sensor, an inertial measurement unit IMU and a processor, wherein the method includes:

The capacitive touch sensor detects touch state information of a touch action, wherein the touch state information of the touch action indicates a touch state, or a non-touch state;

The IMU detects acceleration information of the touch action;

The processor determines the touch duration of the touch action according to the touch state information of the touch action;

The processor determines the type of the touch action according to the touch duration of the touch action and the vibration amplitude of the touch action, wherein the vibration amplitude of the touch action is determined according to the acceleration information of the touch action, and the touch action is determined according to the acceleration information of the touch action. Action type is used to determine touch behavior.
The method according to claim 1, wherein the method further comprises:

The processor determines, according to the touch duration of each of the touch actions, a time interval threshold corresponding to each of the touch actions, wherein the time interval threshold is proportional to the touch duration of the touch action;

The processor splits the touch action according to the time interval threshold.
The method according to claim 2, wherein the method further comprises:

The processor determines, according to the touch state information of the touch action, a touch start time point and a touch end time point of the touch action;

The processor determines, according to the touch start time point and the touch end time point, the touch duration of the touch action, and the time interval between every two adjacent touch actions;

The processor splits the touch action according to the time interval threshold, which specifically includes:

If the time interval is greater than or equal to a first time interval threshold, the processor determines that the touch action is a new touch action, wherein the first time interval threshold is two adjacent touches the time interval threshold corresponding to the previous touch action in the action;

If the time interval is less than the first time interval threshold, the processor determines that the touch action is a touch sub-action.
The method according to any one of claims 1 to 3, wherein the robot includes a plurality of touch parts, the capacitive touch sensor includes a plurality of capacitive touch sensors located at the plurality of touch parts, and the Methods also include:

The processor determines the touch behavior according to the touch action types of the plurality of touch actions occurring at the plurality of touch locations.
The method according to claim 4, wherein the processor determines the touch behavior according to the touch action types of a plurality of touch actions that occur at the plurality of touch locations, specifically comprising:

The processor determines the touch behavior according to a preset rule according to the touch action types of the touch actions occurring at the touch parts.
The method according to claim 4, wherein the processor determines the touch behavior according to the touch action types of a plurality of touch actions that occur at the plurality of touch locations, specifically comprising:

The processor determines the touch behavior according to a classification model according to the touch action types of the touch actions occurring at the touch parts, wherein the classification model takes historical touch data as training data, and uses the historical touch data as training data. The trained model.
The method according to any one of claims 1 to 6, wherein the touch duration of the touch action is used to distinguish the type of the touch action according to a preset duration threshold.
The method according to claim 7, wherein the duration threshold comprises a first duration threshold and a second duration threshold, and the first duration threshold is smaller than the second duration threshold;

A touch action whose touch duration of the touch action is less than the first duration threshold is a tap;

The touch duration of the touch action is greater than or equal to the first duration threshold, and the touch action whose duration is less than the second duration threshold is stroking;

The touch action in which the touch duration of the touch action is greater than or equal to the second duration threshold is a long press.
The method according to claim 7 or 8, wherein the vibration amplitude of the touch action is used to distinguish the touch action type according to a preset vibration amplitude threshold;

The touch action whose vibration amplitude of the touch action is less than the vibration amplitude threshold value is any one of tapping, stroking or light pressing;

The touch action in which the vibration amplitude of the touch action is greater than or equal to the vibration amplitude threshold value is any one of re-patching, re-stroking or re-pressing.
A touch behavior recognition device, characterized in that the touch behavior recognition device comprises:

a detection module, configured to detect touch state information of a touch action, wherein the touch state information of the touch action indicates a touch state, or a non-touch state;

The detection module is further configured to detect acceleration information of the touch action;

a processing module, configured to determine the touch duration of the touch action according to the touch state information of the touch action;

The processing module is further configured to determine the touch action type according to the touch duration of the touch action and the vibration amplitude of the touch action, wherein the vibration amplitude of the touch action is determined according to the acceleration information of the touch action , the touch action type is used to determine the touch behavior.
The device according to claim 10, wherein the processing module is further configured to determine a time interval threshold corresponding to each of the touch actions according to the touch duration of each of the touch actions, wherein the The time interval threshold is proportional to the touch duration of the touch action;

The processing module is further configured to split the touch action according to the time interval threshold.
The device according to claim 11, wherein the processing module is further configured to determine a touch start time point and a touch end time point of the touch action according to the touch state information of the touch action;

The processing module is further configured to determine the touch duration of the touch action and the time interval between every two adjacent touch actions according to the touch start time point and the touch end time point ;

The processing module is specifically used for:

If the time interval is greater than or equal to a first time interval threshold, the processor determines that the touch action is a new touch action, wherein the first time interval threshold is two adjacent touches the time interval threshold corresponding to the previous touch action in the action;

If the time interval is less than the first time interval threshold, the processor determines that the touch action is a touch sub-action.
The device according to any one of claims 10 to 12, wherein the processing module is further configured to determine the type of the touch action according to the touch action types of the plurality of touch actions that occur at the plurality of touch parts. described touch behavior.
The device according to claim 13, wherein the processing module is specifically configured to determine the touch according to a preset rule according to the touch action types of the touch actions that occur at the touch parts Behavior.
The apparatus according to claim 13, wherein the processing module is specifically configured to determine the touch behavior according to a classification model according to the touch action types of a plurality of touch actions that occur at the plurality of touch locations , wherein the classification model is a model obtained through training with historical touch data as training data.
The apparatus according to any one of claims 10 to 15, wherein the touch duration of the touch action is used to distinguish the type of the touch action according to a preset duration threshold.
The apparatus according to claim 16, wherein the duration threshold comprises a first duration threshold and a second duration threshold, and the first duration threshold is smaller than the second duration threshold;

A touch action whose touch duration of the touch action is less than the first duration threshold is a tap;

The touch duration of the touch action is greater than or equal to the first duration threshold, and the touch action whose duration is less than the second duration threshold is stroking;

The touch action in which the touch duration of the touch action is greater than or equal to the second duration threshold is a long press.
The device according to claim 16 or 17, wherein the vibration amplitude of the touch action is used to distinguish the touch action type according to a preset vibration amplitude threshold;

The touch action whose vibration amplitude of the touch action is less than the vibration amplitude threshold value is any one of tapping, stroking or light pressing;

The touch action in which the vibration amplitude of the touch action is greater than or equal to the vibration amplitude threshold value is any one of re-patching, re-stroking or re-pressing.
A robot, characterized in that it includes:

Including capacitive touch sensor, inertial measurement unit IMU, processor, memory and input output (I/O) interface;

the capacitive touch sensor, the IMU and the processor are coupled to the memory and the input and output interface;

The capacitive touch sensor, IMU and processor execute the method of any one of claims 1 to 9 by running computer instructions in the memory.
A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 9.