WO2014108150A2

WO2014108150A2 - User interface for handwritten character input in a device

Info

Publication number: WO2014108150A2
Application number: PCT/EP2013/003528
Authority: WO
Inventors: Michael SCHLITTENBAUER; Lorenz Bohrer; Martin Roehder
Original assignee: Audi Ag
Priority date: 2013-01-08
Filing date: 2013-11-22
Publication date: 2014-07-17
Also published as: DE102013000072A1; WO2014108150A3

Abstract

The invention relates to a method for operating a user interface for inputting characters into a device (12) in a contactless manner. A sensor device (10) determines a 2D character shape (48) which represents at least one handwritten character (50) of a user using the input element (24), and a character recognition device (16) generates, in accordance with the 2D character shape (48), at least one recognized character (50) as the character input and is sent to the device (12) The aim of the invention is to provide a user interface for a device which allows characters to be input into a narrow space in the device. The sensor device (10) with the help of a TOF camera (TOF - time of flight) (26) detects a 3D movement path (44) of the input element (24) in a pre-determined spatial area (38) and the detected 3D movement path (48) is projected onto a 2D projection plane. The projection of the 3D movement path (44) is then transmitted as the 2D character track (48) to the character recognition device (16). Image detection can also take place from above.

Description

Operator interface for a handwritten character input into a device

DESCRIPTION The invention relates to a method for operating a user interface of a device, which has a handwriting recognition for a contactless input of handwritten characters. The invention also includes a motor vehicle with such an operator interface, a computer program product for providing such operating interface parts in a mobile terminal, in particular a smartphone or a tablet PC, and such a mobile terminal.

In a motor vehicle is usually provided to call individual functions of the motor vehicle, such as entering the navigation destination in a navigation device, by pressing it provided, defined function keys. To sense such key presses, today each key an electrical or capacitive switch is necessary. The keys are localized and fixed. Searching the buttons or even searching for control panels on a touch-sensitive control panel, for example a so-called touchpad or touch screen, can distract the driver during a journey in an undesirable manner from the traffic situation. In addition, each key requires a relatively expensive evaluation unit. The more different equipment variants of a motor vehicle must be, the more different buttons must be installed and connected accordingly. This makes variant diversity costly.

In order to simplify an input of text, for example for setting a navigation destination in a navigation device, character recognition may be provided instead of hardware buttons or emulated buttons on a touchscreen. In this case, a user can draw individual characters, that is to say letters, numbers or even special characters, in succession on one and the same touch-sensitive control panel with a finger or also with a pen. An automatic handwriting Identification, also referred to as text recognition or optical character recognition, can then recognize, for example on the basis of pattern recognition from the respectively drawn character trace, the drawn letters and pass on corresponding digital information to the navigation device. Automatic handwriting recognition allows the recognition of single characters or even several characters drawn one after the other with a single character track or intermittently.

By means of an automatic handwriting recognition it is possible to omit the input of letters for the input of texts on individual keys and instead to provide a single touch-sensitive area on which the user can input different characters. In the context of handwriting text input to a workstation computer, US 201 1/0254765 A1 discloses an automatic handwriting recognition system in which a user with one hand independently of a hand-held item is free in space, i. in the air, can write and this movement of the hand is detected as a 3D motion trajectory of an optical sensor. The 3D motion trajectory is projected onto a plane. The 2D image of the 3D motion trajectory thus formed is then fed to automatic handwriting recognition. The system assumes that the user is standing in front of the optical sensor when performing his hand movements, performing writing movements like on a blackboard. The system can detect when the user finishes an entry with a special gesture. As an alternative to the special gesture, in the case of fully written words with several letters, a statistical analysis can be used to check which part of the 3D motion trajectory belongs to the actual character input.

The invention has for its object to provide a user interface for a device by means of which characters can be entered into the device in a small space.

The object is achieved by a method according to claim 1, a motor vehicle according to claim 6, a computer program product according to claim 9 and a mobile terminal according to claim 10. solves. Advantageous developments of the invention are given by the dependent claims.

In the method according to the invention, an automatic handwriting recognition is also used in a manner known per se for recognizing a character which has been handwritten by a user with an input element, for example his finger or a drawing instrument. The character track represents one or more characters in the usual way as a two-dimensional, ie flat image as the user has drawn. By a character recognition device is then generated in a known manner depending on the 2D character track at least one recognized character, i. ie e.g. output at an output, and the recognized characters are provided as a character input to the device to be operated. The text input is possible by a gesture without touching an input element. For this purpose, a 3D movement trajectory (trajectory) of the input element in a predetermined spatial area is detected by a sensor device according to the method. In the case of the 3D motion trajectory, the coordinates of the points which make up the motion trajectory can differ in all three spatial directions. It is therefore a three-dimensional structure. The detected 3D movement trajectory is then projected onto a two-dimensional projection plane by the sensor device and the projection of the 3D motion trajectory is transmitted as the 2D character trace to the character recognition device.

In the method according to the invention, the 3D motion trajectory is now detected with a TOF camera (TOF) and from a viewing direction whose direction vector lies in the projection plane or parallel to it or at an acute angle smaller than 45 ° to the projection plane, preferably less than 30 °, is arranged. In other words, for example, the TOF camera observes the user's drawing finger from above or below, for example, while the user draws the characters upright in space. With this arrangement, there is the advantage that the TOF camera can be arranged very inconspicuously in an environment of the operator. It does not have to be in front of the user. In particular, in a motor vehicle, it has proven to be favorable to arrange the 3D camera, for example, in the headliner of the motor vehicle and to image the driver from above by means of the 3D camera. By capturing the thus drawn 3D motion trajectory through a TOF camera, the Person does not pay attention to what angle to the 3D camera she performs the movement. As experiments show, the detection of the 3D motion trajectory by means of the TOF camera in all three spatial directions takes place with sufficient reliability.

It can also be provided in the invention that a TOF camera is part of a stereo camera arrangement. This advantageously avoids the shading effect known for TOF cameras, which results when the TOF camera detects an object located freely in space only from one side and thus does not detect the depth dimensions of the object along the optical axis of the TOF camera can be. Another way to compensate for shadowing is by a hand model, as e.g. of the product "Kinect" ® of the company Microsoft ® is known.

Another advantage arises when an inclination of the projection plane in space is adapted to the respectively detected 3D motion trajectory. For this purpose, an envelope of the 3D motion trajectory is determined. For example, for this purpose, dimensions of the edges of a virtual cuboid can be determined, which completely encompasses (envelopes) the 3D motion trajectory and from which, when the dimension of one of the edges is reduced, the 3D motion trajectory would stand out of the cuboid. The expansion direction along which the shell has a smallest extension and the projection plane perpendicular to this expansion direction is then determined. The direction of expansion can also be determined, for example, by means of a principal component analysis (PCA) of the coordinates of the points describing the 3D movement trajectory. By aligning the projection plane there is the advantage that the character recognition device can detect a 2D character trace with particular reliability.

A further advantage results if the detection of the 3D movement trajectory is started for each detection operation, if it is detected by the sensor device that the input element penetrates into said spatial area, and the detection is terminated again, if it is detected by the sensor device, that the input element leaves the room area again. For this purpose, for example, the space area may be defined as a cuboid volume in space. Based on the coordinates the input element, so for example a fingertip, can then be checked whether the fingertip is inside this cuboid or not. The user can then start detecting a 3D motion trajectory by dipping his fingertip into the volume and then, by pulling his hand out of the volume, stop the recognition process.

Another possibility, in order to be able to reliably differentiate between an input procedure on the one hand and a hand movement which is not intended to cause an input, is achieved according to an embodiment of the method by detecting the 3D motion trajectory in the spatial area only if, as a result Sensor device is detected that a hand of the user is at least partially in the space area and at least one finger of the hand has a respective predetermined finger position. In other words, detection of the 3D motion trajectory is only triggered when the user makes a certain finger gesture when drawing by hand.

As already stated, the invention also includes a motor vehicle. The motor vehicle according to the invention has an operator interface with a character recognition device, that is to say an automatic handwriting recognition, for handwritten character input into a component of the motor vehicle. The component may be, for example, a navigation device or an infotainment system. The character recognition device can be a per se known embodiment for a 2D character trace recognition. The motor vehicle according to the invention is characterized in that a TOF camera for handwritten character input is provided at the operator interface and the operator interface is designed to carry out an embodiment of the method according to the invention.

The camera is preferably arranged in a headliner of the motor vehicle. As already described, this results in no disadvantages in the detection of a 2D character trace for the character recognition device when the inventive method is used. In addition, there are advantages due to the unrestricted visibility of the hand, as there are usually no objects above the hand. The non-contact text input can be provided on the basis of the invention, but not only in a motor vehicle, but also in a mobile device, so for example in a smartphone or a tablet PC. In order to be able to retrofit a conventional mobile terminal so that it can carry out the method according to the invention, a computer program product with a program stored on at least one storage medium, which is designed to execute the program by a processor device of the mobile terminal on the Based on camera data of a camera of the mobile terminal to perform an embodiment of the method according to the invention.

Finally, the invention also includes a mobile terminal which has an embodiment of the computer program product according to the invention. In other words, the mobile terminal can be operated in such a way that hands-free character input is possible without contact.

In the following, the invention is explained again with reference to a concrete exemplary embodiment. This shows:

Fig. 1 is a block diagram of an optical sensor device which may be installed in an embodiment of the motor vehicle according to the invention, and

2 shows a sketch of an operating procedure, as is possible for an operator on the basis of an embodiment of the method according to the invention. The examples shown represent preferred embodiments of the invention.

FIG. 1 shows an optical sensor device 10 and a re-input device 12 of a motor vehicle, for example a passenger car. The reproduction device 12 can be, for example, an infotainment system, an audio system, a navigation system, a television system, a telephone, a combination instrument or a head-up display. The sensor device 10 comprises a measuring device 14 and a calculation unit 16. The measuring device 14 comprises an sensor 18, which may for example be a TOF camera or PMD camera (PMD - photon mixing detector). The optical sensor 18 may also be a stereo arrangement, for example. In the example shown in FIG. 1, it has been assumed that the optical sensor 18 is a PMD camera. The optical sensor 18 may be arranged, for example, in a headliner of the motor vehicle.

The optical sensor 18 may be configured in a manner known per se, i. a light source 20, e.g. an infrared light illuminates a detection area 22, for example a space above a center console of the motor vehicle. If there is an object in it, for example a hand 24 of the driver of the motor vehicle, then the electromagnetic radiation emitted by the light source 20 is reflected back by the hand 24 to a sensor array 26. By means of the sensor array 26 then 3D image data can be generated, which indicate 3D coordinates to individual surface elements of the hand 24. The 3D image data are transmitted from the measuring device 14 to the calculation unit 16. The calculation unit 16 may be, for example, a control unit of the motor vehicle. In the calculation unit 16, the signals are evaluated and then the evaluated data are made available to the vehicle, for example, by being transmitted to the reproduction device 12. By the calculation unit 16 limbs, such as a hand, can be segmented from the 3D image data, whereby, for example, the position of a fingertip in the detection area 22 can be determined. Here, known per se segmentation algorithms can be used. The 3D image data of the sensor array 26 of the optical sensor 8 may also represent a sequence of consecutive 3D images, i. With the optical sensor 18 and movements of the hand 24 can be detected. By tracing the trajectory, for example the fingertip in this 3D image sequence, in particular by tracking the position and the speed of the fingertip, a motion gesture indicated by the fingertip can be extrapolated from the trajectory.

In Fig. 2, an interior 28 of a motor vehicle, such as a passenger car, is shown. Shown are a center console 30 with a gear selector lever 32 and a dashboard 34. A driver can draw with his hand 24 in a space above the center console 30 freely in space and so set, for example, a place name for a selection of a navigation target in the display device 12, in this case, a navigation device or a navigation module in an infotainment system. He can concentrate fully on the traffic, ie he is not distracted from his driving task. The character input also requires no electrical evaluation units for buttons or a touchpad.

The non-contact automatic character recognition is provided by means of the optical sensor device 10, which represents an operating interface for the playback unit 12. For this purpose, the optical sensor 26 of the sensor device 14 can be arranged, for example, in a headliner above the center console 30. An optical axis of a camera of the optical sensor can point vertically or obliquely downward. A direction vector V of the optical axis is shown in FIG.

The optical sensor 26 films the interior space 28 above the center console 30. From the 3D image data thus generated, the calculation unit 16 extracts the image information belonging to the hand 24 and thus determines a position P of the fingertip of the hand 24. The position P of the fingertip is is a feature that is tracked by the computing unit 16 over time. In the example, with his finger outstretched, the driver drew the letter A in the air with his fingertip. The position P of the fingertip has thereby described a trajectory 36 which has been detected by the calculation unit 16 on the basis of the 3D image data of the sensor unit 14. In this case, the calculation unit monitors whether the position P is within a predetermined spatial area 38. The spatial region 38 can be described, for example, by coordinates of its corners and edges in a coordinate system 40. In the example shown, it is provided that that part of the trajectory 36 which is located within the spatial area 38 is used for the character recognition. In this case, the calculation unit 16 detects an entry point 40 of the trajectory 36 and an exit point 42 from the spatial area 38, for example by means of a coordinate comparison. The portion of the trajectory 36 located within the spatial area 38 forms a 3D movement trajectory 44, on the basis of which the character recognition takes place. The 3D motion trajectory 44 is projected onto a projection plane 46 by the computing unit 16. This takes place computationally within the calculation unit 16 and is shown in FIG. 2 only for the sake of clarity for the spatial region 28 as well. The directional vector V of the optical axis may be aligned in the projection plane 46 or parallel to the projection plane 46. An angle included by the directional vector V and the projection plane 46 may also have a value between 0 ° and 45 °. In the example shown in FIG. 2, the projection plane 46 is the yz plane of a coordinate system K, as defined by the vehicle vertical axis and the vehicle lateral axis. An orientation of the projection plane 46 in space can also be adjusted depending on the 3D motion trajectory 44. The projection of the 3D movement trajectory 44 onto the projection plane 46 results in a 2D character trace 48, which is fed to a 2D handwriting recognition known from the prior art realized in the calculation unit 16. Through the handwriting recognition, the 2D character trace 48 is assigned in a manner known per se to a recognized character 50, in this case the correctly recognized letter A. The recognized character 50 is then output to the display device 12 as a character input.

The example shows how certain features can be traced (tracked) by the TOF camera or body parts of the occupant due to the physically perfect location of the occupant and the trajectory obtained thereby can be evaluated. In order to enter texts in a system of the motor vehicle, for example, the tip of the outstretched finger can be followed. The y-z plane of the trajectory then reflects the painted letter.

Claims

CLAIMS:

Method for operating a user interface for a non-contact character input into a device (12), wherein a 2D character track (48) is detected by a sensor device (10) which represents at least one character handwritten by a user with an input element (24), and at least one recognized character (50) is generated as the character input and provided to the device (12) by a character recognition device (16) as a function of the 2D character trace (48), wherein a 3D movement trajectory (44) is generated by the sensor device (10) ) of the input element (24) is detected in a predetermined spatial area (38) and the detected 3D motion trajectory (48) is projected onto a two-dimensional projection plane (46) and the projection of the 3D motion trajectory (44) as the 2D character trace (48 ) is transmitted to the character recognition device (16),

characterized in that

the 3D movement trajectory (44) is detected with a TOF camera (26) from a direction whose direction vector (V) lies in the projection plane (46) or offset in parallel or at an acute angle smaller than 45 ° to the projection plane (46) is arranged.

The method of claim 1, wherein an expansion direction is determined along which a shell of the 3D movement trajectory has a smallest extension, and the projection plane is arranged perpendicular to this extension direction.

The method of claim 1 or 2, wherein for each one detection operation, the detection of the 3D movement trajectory (44) is started if it is detected by the sensor device (10) that the input element (24) penetrates into the space region (38), and the Detection is terminated again if it is detected by the sensor device (10) that the input element (24) leaves the space area (38) again.

Method according to one of the preceding claims, wherein the 3D movement trajectory (44) in the spatial area (38) is detected only if it is detected by the sensor device (10) that a hand (24) of the user at least partially in the space area (38) and at least one finger of the hand (24) has a respective predetermined finger position.

A method according to any one of the preceding claims, wherein as the input element (24), a fingertip of an outstretched finger of the user is tracked by detecting the 3D movement trajectory (44) of the fingertip in the spatial region.

Motor vehicle comprising an operator interface with a character recognition device (16) for a handwritten character input in a component (12) of the motor vehicle, wherein the character recognition device (16) is adapted from a 2D character track (48), which at least one handwritten input character (16) 50) describes to recognize at least one character (50), characterized in that

at the operator interface, a TOF camera (26) for handwritten character input is provided and the operator interface is adapted to perform a method according to any one of the preceding claims.

Motor vehicle according to claim 6, wherein the TOF camera (26) is part of a stereo camera arrangement.

Motor vehicle according to claim 6 or 7, wherein the TOF camera (26) is arranged in a headliner of the motor vehicle.

A computer program product comprising a program stored on at least one storage medium and configured to execute a method according to one of claims 1 to 5 when the program is executed by a processor device of a mobile terminal on the basis of camera data of a camera of the mobile terminal.

Mobile terminal comprising a computer program product according to claim 9.