WO2019113868A1

WO2019113868A1 - Ear-based human-computer interaction technology for interaction with touch surface device

Info

Publication number: WO2019113868A1
Application number: PCT/CN2017/116042
Authority: WO
Inventors: 喻纯; 王若琳; 史元春; 何纬捷
Original assignee: 清华大学
Priority date: 2017-12-14
Filing date: 2017-12-14
Publication date: 2019-06-20

Abstract

Disclosed is an ear-based human-computer interaction method for interaction with a touch surface device, wherein an ear, serving as the human organ that interacts with the device, provides an input for the device. The method comprises: identifying an input orientation and/or position of an ear; and in response to an identification result, a touch surface device executing a corresponding control operation. The ear, serving as the human organ that interacts with the device, provides an input for the device, such that access to the entire touch surface of the touch surface device can be realized under the conditions of a stable single-hand grip, and both the use thereof by sighted people in eyes-free conditions and the use thereof by blind people can be supported. Therefore, finger operations can be completely replaced or are just supplementary.

Description

Human-computer interaction technology based on interaction between ear and touch surface device

Technical field

The invention relates to a human-computer interaction method based on interaction between an ear and a touch surface device, and a human-computer interaction system implementing the method.

Background technique

When the user is occupied with one hand and interacts with a touch surface device, especially a mobile phone with a touch screen, with one hand, the thumb can only be used to access the screen, and the range of motion is limited to the lower right corner of the screen (holding in the right hand) In the case of equipment). If the user wants to access any position on the screen, he has to move his hand up and down, or deflect the orientation of the screen of the mobile phone, so that it cannot be easily achieved in a stable grip state, especially for a large-screen mobile phone of 5 inches or more.

For the blind, it is more difficult to interact with touch-surface devices with one hand because they hold the phone tighter. The reliance on speech feedback makes them often need to lift the device to the ear to hear the speech feedback, which results in more limited range of thumb activity. Although special, blind people can also use the thumb to slide left and right in the lower right corner to access the full screen through sequential browsing, but this operation is more difficult and less efficient.

These limitations have led to a reduction in the ability of a person to access a full screen with a finger in the event that one hand is occupied.

Summary of the invention

In view of the above problems, the present invention provides a human-computer interaction method and a corresponding device based on interaction between an ear and a touch surface device, and the ear provides input as a human body interacting with the device, enabling a stable one-handed posture. Access to the entire touch surface of the touch surface device can be supported to support the use of a blind person in an eye-free situation and a blind person. Thus, the finger operation can be completely replaced or only supplemented.

According to a first aspect of the present invention, a human-computer interaction method based on interaction of an ear with a touch surface device is proposed, wherein the ear provides input to a human body that interacts with the device, the method comprising:

- identifying the input finger and/or posture of the ear, and

- in response to the result of the recognition, the touch surface device performs a corresponding control operation.

In the method, the inputting the pointing finger and/or the gesture includes: obtaining a proximity image associated with the ear to identify one of an input location, a position, a force, and/or an action of the ear on the device or Multiple or a combination thereof.

According to one embodiment, the input site comprises one or more of the upper auricle, the lower auricle, the earlobe, or a combination thereof, or the entire ear.

According to an embodiment, the location comprises an edge location and a non-edge location of the touch surface.

According to one embodiment, the actions include: touching, pressing, rotating, sliding, and detaching.

Preferably, the method further comprises: detecting a density value and a pattern of the region of interest as an ear candidate, wherein the region of interest refers to a region of pixels of a point at which the density value exceeds a certain threshold.

Advantageously, the movement of the ear relative to the ear when it is in contact with the touch surface is identified and tracked, and a touch navigation operation command is implemented based on the recognition result.

According to one embodiment, the touch surface device senses the input of the ear using a capacitive sensor, and the contact state of the ear with the screen is measured by the sum of the touch surface capacitance value intensity.

Advantageously, the complete contact of the ear with the touch surface is defined as an ear contact event, the summed value falling back to the ear contact frame for the first time during the contact of the ear with the touch surface; and wherein the ear is The touch surface lift-off is defined as an ear disengagement event that rapidly drops during the process of the ear moving away from the touch surface, taking the last frame above the set threshold to disengage the ear.

Advantageously, the movement of the ear on the touch surface is defined as an ear movement event, and the proximity image is tracked to obtain a relative amount of displacement between successive two frames of images.

Advantageously, the relative displacement between successive two frames of images is calculated using a kernel correlation filter KCF algorithm.

Advantageously, the method uses two KCF trackers to work alternately: the first tracker is in an active working state, and when the tracking result is output, the second tracker is in a background running state; when the first tracker works for a certain period of time or When a tracker fails to track, the state of the two trackers is exchanged, and the replaced first tracker is reinitialized, and so on.

Advantageously, said time is 500 ms.

Advantageously, the contact coordinates of the ear are calculated.

Preferably, the intent to define an ear image at a position between the edge of the upper auricle and the ear hole is defined Click on the point.

Preferably, the minimum rectangle of the surrounding image of the ear is used as a reference near the vertex of the posterior superior auricle, and the distance boundary is a certain value at the intent click point.

The inventors have noted that when interacting, the user tends to move the touch surface device toward the ear rather than moving the ear toward the device. Thus, advantageously, the contact coordinates of the ear are mapped to the touch surface to minimize movement of the user's arms and thumb.

Advantageously, the operation comprises the touch surface device providing feedback to the user.

Preferably, the feedback comprises audible feedback, in particular in the form of sound effects, speech or a combination of both.

Preferably, the feedback comprises haptic feedback, in particular providing the haptic feedback in a vibrating manner.

Additionally or alternatively, the hand inputs an operational command to the touch surface device as an auxiliary human organ that interacts with the touch surface device.

Advantageously, the touch surface device also recognizes input from the hand.

Alternatively, the touch surface defines an ear interaction area and a hand interaction area.

Preferably, when an ear or hand inputs an operation instruction in a non-self-interaction area, the touch surface device prompts an error by feedback.

Advantageously, the method further comprises: identifying the orientation of the ear on the screen to distinguish the left and right ears, and performing different controls based on the result of the differentiation.

According to another aspect of the present invention, there is provided a touch surface device for facilitating human interaction with an ear, the ear being a human body interacting with the touch surface device, the touch surface device comprising:

a touch surface element comprising a sensor component for sensing the operation of the ear, receiving an operation command input by the ear,

- processor; and

a memory storing computer executable instructions that, when executed by the processor, perform the method as previously described.

Preferably, the touch surface device is a mobile phone and the touch surface is a touch screen.

According to still another aspect of the present invention, there is provided a computer storage medium having stored thereon computer executable instructions that, when executed by a computer, perform the method as defined above.

DRAWINGS

1 shows a schematic flow chart of a human-computer interaction method 1 based on an ear interacting with a touch surface device according to an embodiment of the present invention;

2 is a schematic diagram showing a structural configuration of a touch surface device according to an embodiment of the present invention;

Figure 3 is a schematic illustration of the life cycle of an ear contact event when the ear interacts with the screen;

Figure 4 illustrates a method of alternately operating two core correlation filters KCF employed in tracking an image of a moving ear;

Figure 5 shows a schematic diagram of calculating the contact coordinates of the ear; and

Figure 6 shows a schematic diagram of the mapping of the ear contacts to the touch surface.

Detailed ways

The present invention will be further described in detail below in conjunction with the drawings and specific embodiments.

In this article, the input finger of the ear, here draws on the concept of "pointing" in the input field. Pointing generally refers to the coordinates of the click point of the mouse, indicator pointer, etc. in the screen coordinate system. In this case, when the ear is used as the input organ, An input position point determined based on the detection result and a predetermined algorithm.

By "proximity image" herein is meant that the touch surface device is capable of detecting objects as they approach and/or contact the touch surface. Each sensing element (ie, "pixel") in a two-dimensional array of sensing elements (ie, a touch surface) produces an output signal to indicate an electric field disturbance (for a capacitive sensor) at the sensor element, force (for pressure) Sensor) or photo-affinity (for photosensors). The overall pixel value represents a "proximity image". As described herein, various embodiments of the present invention provide the ability to detect and discern touch surface signals (represented as proximity images) produced by identified input locations, locations, force paths, and/or motion types.

The touch surface elements herein may be touch panels or touch screens, and the touch surface elements include sensors capable of sensing input, such as capacitive sensors, pressure sensors, photosensors, image sensors, and the like.

Compared to the fingers, the ear has its own characteristics as a human organ that provides input for interaction with the touch surface device:

1, complex shape

The shape of the finger on the touch surface is relatively simple and can be easily reduced to a single spot. However, The ear itself has a more complex shape, a larger area, and a more uneven surface. The shape of the contact area varies from person to person when in contact with the touch surface, and even for the same user, there is a huge difference every time it is input. . Therefore, the recognition and tracking of the ear input is more difficult.

2, can provide rich input

Different actions that can be performed when the ear interacts with the touch surface device, making the input richer. For example, actions that the ear can make include, but are not limited to:

- touch: part or all of the ear contacts the touch surface;

- Pressing: Part or all of the ear touches the touch surface and slowly increases the strength of the contact;

- Rotation: a part or all of the ear maintains a state of contact with the touch surface;

- sliding: part or all of the ear contacts the touch surface and produces displacement;

- Pull away: All of the ears are in close contact with the touch surface, slowly reducing the strength of the contact.

Also, the area of the ear is large, resulting in more options for the parts that can provide input. For example, one or more of the upper auricle, the lower auricle, and the earlobe, or a combination thereof, may be used as an input site, or the entire ear may also serve as an input site. Therefore, performing some or all of the same operation of the ear is two kinds of interactions, which can trigger different operation instructions. As an example of an operation in which different parts of the ear are combined to define an interaction mode, the upper part of the ear is first contacted on the screen, and as a contact rotation point, the lower end of the mobile phone can be attached or lifted to the head. The lower half of the image of the ear will appear and disappear, and the operation can be identified accordingly, and the operation instructions are defined based on this.

In addition, since the ear is very soft and has a complicated shape, it deforms during contact with the touch surface. Therefore, different forces—light or heavy—will also bring new information to the input, and these differences can be used to distinguish between different interactions.

In addition, the location in which the ear interacts with the touch surface device can also provide more options for input. For example, the ear may be in contact at the edge of the touch surface or in a non-edge position; of course, the ear may not be in contact with the touch surface, but at a distance from it, and/or at an angle.

Moreover, the orientation of the left and right ears presented on the screen is different: the operation of the left and right ears can be given different meanings.

The input position, force, position and/or motion of the ear combined with the action time can form an extremely rich input, making different operational instructions, including but not limited to:

- Click: part or all of the ear, short-time contact with the touch surface and leave;

- Double click / multiple consecutive clicks: part or all of the ear is consecutively and twice in a certain period of time Contact and leave the touch surface;

- Long press: Part or all of the ear is left in a single time and continuously touches the touch surface for a certain period of time;

- fitting: a part of the ear contacts the touch surface, and all the ears are in contact with the touch surface by the action of the fitting;

- Partially away: all of the ear touches the touch surface, keeping part of the ear in contact with the touch surface, and other parts away from the touch surface;

- Pressing: All of the ear gently touches the touch surface, slowly increasing the strength of the contact;

As an application example of the ear motion described above, one of the most basic operations is to move the ear in a state of being attached to the touch surface, and the touch surface device reads the information content of the corresponding position. As another example, the user can touch the ear twice on the touch surface to make a selected operational command. For another example, when the user rotates the ear on the touch surface to make an adjustment operation instruction for a certain control amount, for example, the control amount may be the brightness of the light control, the opening degree of the curtain control, the volume of the video control, and the playing. Tracks, set temperature of air conditioning control, air volume, and so on.

Next, a touch surface device that facilitates human interaction with an ear is described in accordance with an embodiment of the present invention, wherein the ear acts as a human organ that interacts with the touch surface device. As shown in FIG. 2, the touch surface device 200 includes a touch surface element 230 that receives an operational command input by an ear, the touch surface element 230 includes a sensing device 231 that senses an input of the ear; it further includes a processor 210 and stores A memory 220 of computer executable instructions that can execute the human computer interaction method in accordance with the present invention. The touch surface device can be, for example, a mobile phone, a tablet computer, a smart watch, or the like.

It should be noted that, hereinafter, the touch of the ear to the touch surface is often taken as an input, but this is a preferred example, and in the case of sensor support, other actions of the ear on the touch surface, such as proximity, may also form an input, such as on a touch surface. A photoelectric sensor is provided thereon, which can be detected by the photoelectric sensor when the ear is close enough.

As shown in FIG. 1, in a human-computer interaction method 100 based on interaction between an ear and a touch surface device according to an embodiment of the present invention, the following steps are performed: step S110, identifying an input pointing and/or a gesture of an ear; and step S120, The touch surface device performs phase in response to the result of the recognition The control operation should be.

In one example, the point at which the center of the ear is upward is defined as the ear position. In one example, the positional movement and orientation change of the ear are discerned by comparing the matches between the ear shapes.

A specific explanation is now made by an embodiment. In this embodiment, the touch surface device is a mobile phone, which may be, for example, an Android (Android) system based mobile phone, and thus the touch screen of the mobile phone is a touch surface. The handset, for example, has a capacitive sensor that senses the input received from the ear. Of course, the touch surface device of the present invention can also utilize other sensors such as pressure sensors, photo sensors, image sensors, sonic sensors, and the like.

When the ear interacts with the phone, it recognizes the input finger and/or posture of the ear. In particular, a proximity image associated with the ear is obtained to identify one or more of the input locations, locations, forces, and/or motions of the ear on the device, or a combination thereof.

The input site can include one or more of the upper auricle, the lower auricle, the earlobe, or a combination thereof, or the entire ear.

The position includes: the edge position and the non-edge position of the touch surface. Among them, the edge position of the touch surface is a position that is easy to be perceived by the ear, so it is preferable to set some events triggered by the ear motion at the edge position. For example, the auricle portion of the ear can be slid up and down along the edge of the screen and slide left and right to achieve an up and down slide and a left and right slide operation similar to the finger on the screen.

In one example, the capacitance matrix of the screen can be obtained from the Android HAL layer (hardware abstraction layer), and processed, interpolated, denoised, etc. to obtain a capacitive screen image suitable for analysis.

The capacitive image produced by the ear and screen contact is then identified and tracked. Specifically, the density value and the pattern of the region of interest as the ear candidate are detected, wherein the region of interest may refer to a region composed of pixels of a point whose density value exceeds a certain threshold.

As mentioned before, the unique attribute based on the interaction of the ear with the touch surface corresponds to the definition of the existing finger touch event (down, up, move). In the embodiment, a unique ear-touch event can be proposed for ear interaction, that is, a concept definition of Ear-on, Ear-off, and Ear-move. .

For example, in the present embodiment, the "full-screen capacitance value intensity summation value" can be used to identify the region of interest, and the contact state of the ear with the screen can be measured to extract two stable frames as the ear contact Ear-on and the ear disengaged Ear. -off. Figure 3 shows the ear contact event life cycle when the ear interacts with the screen once, with the Y-axis being the "addition value."

Ear contact with Ear-on: The ear is "fully in contact" with the screen. The "addition value" fluctuates in the process of contact between the ear and the screen, taking its first fall, that is, the first trough appearing in the curve is an Ear-on frame.

Ear off the ear: When the ear is lifted off the screen. The "addition value" drops rapidly as the ear moves away from the screen, taking the last frame above the set threshold as an Ear-off frame, as shown in Figure 3.

Ear-move: The KCF kernel correlation filter (Kernelized Correlation Filter) algorithm is used to track the image of the ear moving on the screen to obtain the relative displacement between two consecutive frames of images.

In addition, for the characteristics of easy deformation of the ear, in order to obtain a more stable tracking effect, two KCF trackers can be used to alternately work, as shown in FIG. 4:

- the first tracker is in an active working state, and when the tracking result is output, the second tracker is in a background running state;

- When the first tracker reaches a predetermined time threshold, for example 500 ms, or the first tracker fails to track, the status of the two trackers is swapped and the replaced first tracker is reinitialized.

Here, the time 500ms is a preferred example, and other time thresholds can be set as needed.

Then, based on the results of the recognition and tracking, the contact coordinates of the ear are calculated and mapped to the full screen.

In this embodiment according to the present invention, as shown in Fig. 5, for example, a certain point located between the edge of the upper auricle and the ear hole is defined as an intent click point of the ear image. The point at which the distance boundary is a certain value is taken as the reference by using the smallest rectangle surrounding the image of the ear near the apex of the posterior superior auricle. The first point (ear contact ear-on) uses this definition of absolute position coordinates, and the ear movement of ear-move is calculated by the absolute coordinates and the relative displacement of adjacent frames. Other locations may be employed in the present invention as intent click points.

In this embodiment, the ears interact only within a certain area of the screen, thereby defining an interactive area of the ear, as shown in Figure 5, in which the interactive area as shown is defined as a comfortable boundary area for ear interaction. The movement of the ear in this area is then linearly mapped to full screen, as shown in Figure 6. Thereby, the range of movement of the user's arm and thumb can be reduced as much as possible.

Thereafter, after smoothing the mapped coordinate trajectory, for example, simulating the structure of the touch event, the touchevent is injected back into the Android system by the adb using the sendevent command.

The touch surface device may provide to the user during operation of the method according to an embodiment of the invention Feedback. The feedback includes audible feedback, for example, during sliding of the ear on the touch surface, the information content on the ear sliding path can be output to prompt the user. The audible feedback can be provided, for example, in the form of sound effects, speech, or a combination of both. Alternatively, the feedback includes haptic feedback, such as providing tactile feedback to alert the user when the ear is mishandling. For example, the haptic feedback can be provided in a vibrating manner. Alternatively, the feedback can include a combination of audible and tactile feedback.

In addition, the hand can also input operational commands to the touch surface device as an auxiliary human organ that interacts with the touch surface device. The touch surface device can recognize input from the hand. Alternatively, the touch surface defines an ear interaction area and a hand interaction area such that when an ear or hand inputs an operation instruction in a non-self interaction area, the touch surface device prompts an error by feedback. As an example of ear-to-finger cooperation, when the finger clicks on the screen and does not press and hold the screen (just like the shift key), the ear operates as a different interaction.

In the previous example, a capacitor is used as an example of a sensor that senses ear input, but this is by way of example and not limitation, and other types of sensors may be used instead or in combination, such as: a pressure sensor, a light sensor, a camera, as an example of a camera, You can get a traditional flat image or you can use a depth map.

In the previous example, the contact of the ear with the surface of the screen is sensed, but this is by way of example and not limitation, and alternatively, or in combination, the distance of the ear from the screen may be recognized when the surface of the screen that the ear does not touch. The orientation of the ear relative to the screen, and different input signals are obtained based on different distances and different orientations, so that different input signals can be corresponding to different operation commands.

In the foregoing, a mobile phone (handset) is taken as an example of a touch surface device, but this is by way of example and not limitation, and the touch surface device may also be, for example, a tablet computer system, a handheld computer system, a portable music player system, a portable video player system, etc. Wait.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Numerous modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

A human-computer interaction method based on interaction between an ear and a touch surface device, wherein the ear provides input as a human body interacting with the device, the method comprising:

- identifying the input finger and/or posture of the ear, and

- in response to the result of the recognition, the touch surface device performs a corresponding control operation.
The human-computer interaction method according to claim 1, wherein the recognizing an input pointing and/or a gesture of the ear comprises: obtaining a proximity image associated with the ear to identify an input portion, a position of the ear on the device, One or more of the forces and/or actions or a combination thereof.
The human-computer interaction method according to claim 2, wherein the input portion comprises one or more of an upper auricle, a lower auricle, and an earlobe or a combination thereof, or an entire ear.
The human-computer interaction method according to claim 2, wherein the position comprises: an edge position and a non-edge position of the touch surface.
The human-computer interaction method according to claim 2, wherein the action comprises: touching, pressing, rotating, sliding, and detaching.
The human-computer interaction method according to claim 2, further comprising: detecting a density value and a pattern of the region of interest as an ear candidate, wherein the region of interest refers to a region composed of pixels of a point whose density value exceeds a certain threshold .
The human-computer interaction method according to any one of claims 1 to 6, wherein the movement of the ear with respect to the touch surface when it is in contact with the touch surface is recognized and tracked, and a touch browsing operation instruction is implemented according to the recognition result.
The human-computer interaction method according to claim 7, wherein the touch surface device senses an input of the ear using a capacitance sensor, and measures a contact state of the ear with the screen by means of a touch surface capacitance value intensity addition value.
The human-computer interaction method according to claim 8, wherein the full contact of the ear with the touch surface is defined as an ear contact event, and the first fall of the gradual increase in the contact of the ear with the touch surface is An ear contact frame; and wherein the lifting of the ear from the touch surface is defined as an ear disengagement event, the summing value rapidly decreasing as the ear moves away from the touch surface, taking the last time above the set threshold One frame, the frame is released for the ear.
The human-computer interaction method according to claim 9, wherein the movement of the ear on the touch surface is defined as an ear movement event, and the proximity image is tracked to obtain two consecutive frames of images. The relative amount of displacement between.
The human-computer interaction method according to claim 10, wherein the relative displacement amount between successive two frames of images is calculated using a kernel correlation filter KCF algorithm.
The human-computer interaction method according to claim 11, wherein two KCF trackers are alternately operated: the first tracker is in an active working state, and when the tracking result is output, the second tracker is in a background running state; When the first tracker works for a certain period of time or when the first tracker fails to track, the state of the two trackers is exchanged, and the replaced first tracker is re-initialized, and so on.
The human-computer interaction method according to claim 12, wherein said time is 500 ms.
The human-computer interaction method according to claim 7, wherein the contact coordinates of the ear are calculated.
The human-computer interaction method according to claim 14, wherein the position between the upper auricle edge and the ear hole is defined as an intention click point of the ear image.
The human-computer interaction method according to claim 15, wherein the imaginary click point is determined by using a minimum rectangle surrounding the ear image near a vertex of the upper rear auricle.
The human-computer interaction method according to claim 16, wherein the contact coordinates of the ear are mapped to the touch surface.
The human-computer interaction method according to any one of claims 1 to 6, wherein the operation comprises the touch surface device providing feedback to the user.
The human-computer interaction method of claim 18, wherein the feedback comprises audible feedback.
The human-computer interaction method according to claim 19, wherein the auditory feedback is provided in a sound effect, a voice, or a combination of both.
The human-computer interaction method of claim 18, wherein the feedback comprises haptic feedback.
The human-computer interaction method according to claim 21, wherein said tactile feedback is provided in a vibration manner.
The human-computer interaction method according to any one of claims 1 to 6, wherein the hand inputs an operation instruction to the touch surface device as an auxiliary human body that interacts with the touch surface device.
The human-computer interaction method according to claim 23, wherein the touch surface device further recognizes Input from the hand.
The human-computer interaction method according to claim 23, wherein the touch surface defines an ear interaction area and a hand interaction area.
The human-computer interaction method according to claim 24, wherein the touch surface device prompts an error by feedback when an ear or a hand inputs an operation instruction in the non-self-interaction area.
The human-computer interaction method according to any one of claims 1 to 6, further comprising: recognizing an orientation of the ear on the screen to distinguish the left ear and the right ear, and performing different control based on the result of the discrimination.
A touch surface device that facilitates human interaction with an ear, the ear being a human body that interacts with the touch surface device, the touch surface device comprising:

a touch surface element comprising a sensor component for sensing the operation of the ear, receiving an operation command input by the ear,

- processor; and

a memory storing computer executable instructions which, when executed by the processor, perform the method of any one of claims 1 to 27.
The touch surface device of claim 28, the touch surface device is a mobile phone, and the touch surface is a touch screen.
A computer storage medium having stored thereon computer executable instructions for performing the method of any one of claims 1 to 27 when the instructions are executed by a computer.