DE10313019B4 - One-hand operation of terminals using gesture recognition - Google Patents

Abstract

Mobile terminal with
A user interface (4) with a selection means (5) which can be positioned on the user interface,
Means for measuring a position of a user (3) relative to the mobile terminal (1),
Means for deriving a relative movement of the mobile terminal (1) relative to the user (3) from the position of the user (3),
- Means for positioning the selection means (5) on the user interface (4) in dependence of the relative movement.

Description

With the miniaturization of mobile communication terminals Keyboards and other haptic input options in favor of device size on a Minimum reduced. In future terminals will even be dispensed with keypads or keypads. Nevertheless, a practicable and comfortable interaction possibility to ensure, new operating concepts become necessary. Restrictive limit due to Cost, however, is usually the existing hardware equipment, the this must be used intelligently.

Devices without Keypad or with virtual keypads have been either through Speech recognition or by pen input, as for example usual for PDAs is. The speech input alone is especially in heavily noisy Environments difficult. On the other hand, the pen input allows no one - handed operation for mobile devices often desired becomes.

since some time are mobile devices available, in which a camera is integrated.

From that Based on the object of the invention, an input option for mobile terminals to create a one-handed Operation allowed and no keyboard needed.

These The object is achieved by those specified in the independent claims Inventions solved. Advantageous embodiments emerge from the dependent claims.

Accordingly has a mobile terminal a displayable on a display, in particular optical and further preferably graphical user interface. The user interface let yourself in the form of, for example, a display, in particular an LCD or LED display, illustrated user interface or user interface realize or as a very simple visual display with e.g. different illuminated digits, in the non-selected Digits. green shine and selected red. On the graphical user interface is a virtual and / or real selection means for selecting of individual functional areas of the user interface, which each functionalities are assigned, in particular positionable by moving. Further features the mobile terminal via means for measuring a relative movement of the mobile terminal relative to a User of the mobile terminal. Furthermore has the mobile device also means for positioning the selection agent on the User interface in dependence the measured relative movement.

So For example, the user can simply put the mobile terminal in hold his hand and move towards the selection agent a desired one Area of the user interface to move. Is the selection agent in this way, for example, to a desired (virtual) operation button (Button) of the user interface has been moved, so the user can by speaking a command or by pressing an input button on the mobile terminal the one with the virtual operation button trigger connected action of the mobile terminal.

The Means for measuring the relative movement preferably have means for detecting the position of the lips, an eye and / or the face of the person User on. In that case, the user only needs the mobile terminal to align and move on his face while doing the Trace the selection mark movement on the user interface.

At the easiest Realize the measurement of relative motion when the mobile terminal to an optical sensor, in particular a camera, for recording a respective image of the user. From the recorded episode of pictures then determine the relative movement.

On The graphical user interface can be used to control the relative movement as feedback, the currently recorded image of the user from which the user then clicks on his current Position opposite the terminal shut down can.

For a use in the dark, the mobile terminal may have lighting means, to illuminate the user and also here use the optical To enable sensors.

Preferably the mobile terminal has one Evaluation device, through which the image of the user can be analyzed is by setting segments in the captured image that are brightness-specific Have characteristics, a positional relationship of the specified Segments to each other is checked and a presence of a face is derived in the image, if a selection of fixed segments a particular positional relationship having.

The selection means can be displayed as a display means on the graphical user interface the. But it is also possible to signal the user the position of the selection means by not the selection means itself is displayed, but only each of the currently selected on the user interface element is displayed as selected.

Around a flowing one To ensure movement of the selection agent, the mobile exhibits terminal advantageous stabilizing agent for the movement of the selection agent on.

In a method of operating a mobile terminal measured a relative movement of the mobile terminal towards a user and a selection agent on a mobile user interface terminal in dependency of positioned measured relative movement. Advantageous embodiments of the method are analogous to the advantageous embodiments of mobile terminal and vice versa.

So For example, it is preferably used to measure the relative movement from the mobile device a picture of the user taken by a camera. The picture of the user can then be parsed by setting segments in the image that have brightness-specific features, a positional relationship the defined segments to each other is checked and the presence of the user, in particular a face of the user, in the detected Image is derived when a selection of fixed segments one certain positional relationship having.

The A method for analyzing the image of the user comprises in particular setting segments in the captured image that are brightness-specific Have features. As will be explained in more detail below, the brightness-specific features such as light-dark transitions and / or Include dark-light transitions. Subsequently, will a positional relationship of fixed segments checked each other, with a presence a user, in particular a (human) face, in derived from the captured image when a selection of specified Segments has a particular positional relationship. The means by the process just described can only by Analysis of certain areas of the captured image, namely the Segments with brightness-specific features, more precisely by checking the positional relationship the specified segments on the presence of the user, in particular his Face, and its position closed to the mobile terminal become. Thus, to carry out the method requires only a small amount of computing power, thereby a resource reduction or resource optimization of a data processing system to perform of the method allows becomes.

According to one advantageous embodiment, it is in the captured image an image captured by an electronic camera that is electronic is processable. In particular, the captured image is single Assembled pixels associated with each gray level values are.

According to one Another advantageous embodiment, segments are detected in the Image set in which the brightness-specific features sharp or abrupt brightness transitions, for example from dark to bright or from light to dark. such find (sharp) brightness transitions for example, in a human's face, in particular at the transition from the forehead to the eyebrows or (in people with light hair color) at the transition from the forehead in the shadow of the eye sockets. Find such (sharp) brightness transitions but also during the transition from the upper lip area or lip area to the mouth opening or from the mouth opening to the lip area of the lower lip or to the lower lip area. One further brightness transition stands between the lower lip and the chin area, more precisely said as a shadow area (depending on the light ratio or incidence of light) based on a slight protrusion the lower lip, a. As explained in more detail below it will be possible based on these brightness transitions or based on the positional relationship a selection of the brightness transitions the presence of a human face or in particular to close the position of an oral area.

According to an advantageous embodiment, the step of defining segments in the acquired image comprises processing the image by means of a gradient filter, in particular a horizontal gradient filter, in particular horizontal (sharp) brightness transitions, such as those on the eyebrows, on the eyes, or on the mouth to make visible. The processing by means of the gradient filter produces a first processed image in which corresponding pixels in the first processed image with a grayscale value are provided at the location of a brightness transition in the acquired image, which depend on the sharpness of the brightness transition in the acquired image. This means that in the case of a brightness transition within a few pixels from a low brightness value (low gray scale value) to a high brightness value (high gray scale value), corresponding information is entered in the first processed image. It is assumed that the first edited image is displayed in the form of a grayscale image For example, when using a positive horizontal gradient filter applied from top to bottom to the captured image, areas of the image in which no brightness transitions are present can be represented in medium gray level values, while light to dark transitions (from above depending on the sharpness of the transitions, a correspondingly higher gray value than the average gray value is obtained, and dark-light transitions (from top to bottom) receive a correspondingly lower gray value as a function of the sharpness of the respective transition.

According to one Further advantageous embodiment, the step of setting of segments in the captured image binarization processing of the first include processed image. This can be a second edited Image are generated at the corresponding pixels, their gray scale value a first predetermined threshold (which depends on the brightness of the captured image is selected) is assigned a binary "1" value while the remaining pixels a binary "0" value is assigned. In other words, can the pixels whose gray scale value exceeds the predetermined first threshold, the value "white" is assigned, while the rest Pixels the value "black" is assigned. Thus, in the second processed image only brightness transitions with a definite one sharpness to recognize.

Related pixels, which is a white value is assigned are seen as a unit of pixels leading to the following evaluation in a geometric shape, such as a rectangle inscribed become. This geometric shape can then be considered a fixed segment To be defined. Since it is necessary from the recorded in the Image captured scene biometric features, especially a face concerning a human to filter or extract to be able to infer the presence of a human face is it is advantageous to include certain areas or structures in the Image, for example, due to their size or shape, no biometric Characteristics regarding of a face, discard.

According to one advantageous embodiment, certain areas of the second edited image with contiguous pixels based on of predetermined parameters regarding size and / or shape of the area or discarded this inscribing geometric shape and In particular, the pixels associated with the areas become set a black value. A criterion concerning the shape of areas related Pixels with a white value may be, for example, that the height of this area or this Structure too big a value in contrast to the width of the structure. That means that relationship from height too wide is too big. Such a determination becomes clear when - as mentioned above - segments based on brightness-specific features of horizontally extending Eyebrows or horizontally extending lips or mouth parts be determined. This in turn means that the previously captured Picture with a big one Number of pixels to only a small number of specified Segments is returned, from their positional relationship to each other now the presence and position of a face can be derived.

In accordance with an advantageous embodiment, in the step of checking the positional relationship in a first examination step corresponding to a first detection mechanism, each of the fixed segments is examined as to whether a second segment fixed to a segment to be examined exists on a horizontal line Substantially horizontal line to the currently examined fixed segment lies. Starting from a captured image consisting of a plurality of pixels, the second segment does not necessarily have to lie on one of the horizontal lines of pixels covered by the segment to be examined, it may also be a predetermined small amount of pixels higher or lower with respect to the horizontal or horizontal line lie. If a second fixed horizontal segment is found, a search is made for a third fixed segment located below the examined and second fixed segment and for which there is a distance from the examined to the second fixed segment and a distance of a link between the examined and the second fixed segment to the third fixed segment has a first predetermined ratio. In particular, a normal to the link between the examined and the second fixed segment can be defined, wherein the distance from the third segment (along the normal) to the link between the examined and the second fixed segment in the first predetermined ratio enters. The distance between the examined and the second segment may be defined such that the distance from a respective center of the fixed segments is used. In particular, when using segments with the geometric shape of a rectangle, a center point can be easily determined. The presence of a face can thus be concluded by the first examination step just described, by determining the positional relationship between three fixed segments. Herein, it is assumed that the examined and the second predetermined segment represent a respective eyebrow portion in the face of a human, which normally has a sharp chiaroscuro brightness transition from top to bottom and thus is easily recognizable in the second processed image. The third predetermined segment represents a segment of an oral region or the border area forming between the upper lip and lower lip. It has been found that, for the determination of a human face, the first predetermined relationship between the distance of the examined and second defined segments to the distance of the connecting path of the both segments to the third fixed segment with a value of 1: 1 to 1: 1.7, in particular from 1: 1.2 to 1: 1.5 is advantageous. In addition to the ability to use eyebrows as distinctive segments with brightness-specific features, it is also possible to use shadow-forming areas of the eye sockets instead of the eyebrows or the eyes or the iris itself.

According to one Another advantageous embodiment is in the step of checking the positional relationship in a second examination step (corresponding to a second examination step) Detection mechanism) of each of the designated segments examines whether there is a fourth fixed segment to this segment exists below, especially along a vertical, the examined fixed segment and for which that the horizontal length of the examined fixed segment and the distance of the fourth specified Segments to examine fixed segment a second predetermined Ratio. This second investigation step assumes that in particular in individuals who wear glasses or sunglasses, the individual eyebrows are not recognizable, but that due to the strong contours the spectacles only one segment or structure is recognizable, the geometrical of the length the two eyebrows together corresponds. Thus stands also the Length of this the both eyebrows comprehensive segment again in a certain relationship to remove the mouth on the user's face. Become thus detected two specified segments, in which the length of the first segment to the distance of the second segment a second predetermined relationship may also according to the second examination step be closed to the presence of a human face. As valid Area for the second predetermined ratio can again a value of 1: 1 to 1: 1.7, but especially from 1: 1.2 to 1: 1.5 are accepted.

According to one advantageous embodiment is in the step of checking the positional relationship first the first examination step carried out in particular for faces to look for people who are not wearing glasses. Shows the first examination step no result at which to the respective examined segments no second and third segment has been found, so in Following the second examination step performed. It but it is also possible first to carry out the second and then the first examination step.

In the method and the advantageous embodiments thereof described above, in particular in the first examination step, the positional relationship of three fixed segments to each other or a first predetermined ratio of the distances of the segments to each other and is in the second examination step, the positional relationship of (two) segments in Depending on the size and / or shape, in particular the (horizontal) length, the first segment can be used as a criterion for deriving a presence of a face in the captured image of a scene. In order to improve the method, ie to increase the probability that it is actually a face, according to a further advantageous embodiment, in a further method step, extracting biometric features with respect to the mouth or lip from an analysis area around a found third or fourth fixed segment be performed. This means that in this additional step further investigations are to be carried out as to whether the third or fourth fixed segment representing a mouth region actually represents a mouth region or an oral region. In this case, light-dark and dark-bright-brightness transitions can be determined in the detected image in an area located at the position of the third or the fourth fixed segment and the respective segment, in order to detect the presence of an upper lip and lower lip of a Derive mouth. Advantageously, the analysis region of the acquired image, the region with the presumed mouth part, is subjected to processing with a horizontally oriented gradient filter in order to obtain a third processed image. The third processed image, viewed from top to bottom, can then represent a sharp light-dark transition from the upper lip region to the "intermediate lip region" or the mouth opening region according to the first processed image, it being possible to deduce the shape of the upper lip from this transition. This determined shape can be subjected to a first test step, which is to determine whether it is actually an upper lip or any other structure based on the shape of the upper lip line. Does the upper lip have specific geometric properties for it, such as a certain (horizontal) length or one or more specific curvatures, it can be concluded first on the actual presence of an oral area and thus also on the presence of a human's face. Thus, the method described above, including its advantageous embodiments, which conclude the presence of a face from the positional relationship of various defined segments, can be improved by feature extraction, in this case the extraction of geometric properties of the upper lip in the first test step become. This means that after this first section of the feature extraction in the first checking step, the probability can already be increased that the third or fourth fixed segment or the analysis area is an image area in which a mouth part of a face is located ,

Fulfill the certain geometric properties (for example, in terms of length or curvature) the for one Upper lip specific criteria, so can the feature extraction proceed to a second extraction section or a second checking step, in which now a lower lip or lower lip line are extracted should. A lower lip, for example, by means of a vertical Search algorithm be recognized or extracted, in the analysis area first the Grayscale values of the upper lip are determined and then corresponding gray scale values to be searched in a section that is vertical located below the area of the upper lip. That way is it is possible very bright areas, such as the teeth, as well as very dark areas, such as the throat area, to skip and one regarding the To identify the upper lip as a lower lip. A another possibility extracting the lower lip, based on the fact that when appropriate greater Choice of analysis area where part of the chin area is with is included, especially in the third edited image of viewed down to the top, a dark area (shadow area) below the lower lip, which is based on the fact that the lower lip normal to the face by one small amount curved forward is. Thus, according to this method based on the lowest and most striking brightness transition viewed from dark to bright from bottom to top, on the lower lip or lower lip line are closed.

are now identified upper lip and lower lip, so it is now possible viseme or extract oral characteristics by the geometric properties the upper lip and lower lip are examined separately.

It It can thus be said in summary that a method according to the present invention it first allows, based on an image in a single pass, brightness-specific Segments to determine when checking their positional relationship on the presence and position of a (human) face close.

One Program product for a data processing system containing code sections, with one of the described methods on the data processing system accomplished can be, can be by suitable implementation of the method in a programming language and translation in execute executable code from the data processing system. The Code sections are stored for this purpose. It is under a program product understood the program as a tradable product. It can be in any Form, for example, on paper, a computer-readable medium or via a Network distributed.

Further Advantages and features of the invention will become apparent from the description an embodiment based on the drawing. Showing:

1 a user with a mobile terminal;

2 a graphical user interface with control of a selection means by a movement of the user's head;

3 a representation of a selection agent as a mouse pointer;

4 a schematic arrangement of the essential components for analyzing an image of a face according to a preferred embodiment;

5 a schematic arrangement of the method steps according to a primary detection mechanism for finding a face, in particular an oral area, in an image;

6 an image of a user in the raw state and after two specific processing steps;

7 an image of a user with a marked segment of a mouth area;

8th an enlarged view of the in 5 illustrated mouth area with marked upper lip and lower lip lines;

9 a picture of a user in raw and after a plurality of processing steps.

Of the Collection of camera functionalities into mobile devices The third generation allows new ways of interacting with the Devices. Using face detection and tracking can be done by the camera recorded movement of the user translated directly into motion data become. This movement data can be evaluated in such a way that as a positioning for the selection of buttons displayed on a graphical user interface be used.

As in the 1 and 2 can be shown by slight movement of a mobile device 1 and / or the head 2 a user 3 a selection means (selection mark) are moved to a desired button. In the illustrated embodiment, this is the button 9 on the graphical user interface 4 , Then either by acoustic input, in particular speech input, "yes", whistle, tongue clicks, by optical input, in particular winking, opening the mouth, by pressing a button or by pausing the selection means on the selected object in the form of the button 9 the function of the mobile terminal belonging to this object is triggered.

The selection agent 5 and thus its position on the graphical user interface 4 can also be like in 3 shown optically, in particular in the form of a mouse pointer, are output. The mouse pointer is thereby, analogous to optical mice, triggered by the movement of the device and is moved according to the detected motion estimation. In this case, the mouse pointer movement is no longer measured and calculated as with an optical mouse via the relative movement to the underlying surface, but from the relative movement of the mobile terminal to the user. Intuitive operation is provided by the already familiar with the user mouse concept.

Thereby can be used on mobile devices, especially with mobile devices with less than 24 or less than 10 buttons, with mobile phones and / or for PDAs, with camera functionality a very comfortable one-hand operation realized without additional hardware become. When implemented as a moving mouse pointer, the user can the familiar operating concept of the mouse on the movement of his Transfer terminal.

It can Continue stabilization method for smooth movement of the mouse pointer or the button selection.

For visual feedback, either the image of the head 2 the user as in 2 can be played, recorded, or the method can be as in 3 shown without image recording.

For a service in dark surroundings, the mobile terminal can use lighting devices such as infrared light emitting diodes are equipped.

The mobile terminal and the proposed method can be used in a variety of ways used or applied. This includes entering phone numbers, with appropriate size of the ad or a soft-scrolling also a selection of letters or special characters, for example SMS, a menu control of the device and / or an application control.

To the Measuring the relative movement, ie the localization of the user or his face the terminal, For example, the method described below can be used.

First, be on it 4 referenced, in which a schematic representation of the components contained in the mobile terminal or mechanisms for detecting and analyzing an image are shown. As in 4 can be seen, a camera K is provided as an optical sensor, which is able to capture a scene (a real environment with certain things and / or people). The camera can be designed, for example, as a CCD camera. The camera K is connected to an optical evaluation section OAA, in which various mechanisms or methods for analyzing the scene detected by the camera can take place. The optical evaluation section OAA is capable of performing a first or primary detection mechanism PLD for finding a face or mouth part in the captured image of the scene. Accordingly, the optical evaluation section can perform an alternative or additional second or secondary detection mechanism SLD for finding a face or an oral area. After finding the position of the face according to the primary or secondary detection mechanism, an improved plausibility check can be carried out by means of a feature extraction ME, whether it is actually the face found, in particular its mouthparts. The results obtained from the mechanisms PLD, SLD and ME, namely that a face of a person or an oral area is present in the image, can be forwarded to a control device SE.

In the case of the preferred embodiment explained below, the in 4 Darge presented components in a data processing system in the form of a mobile phone (of course, other implementations of a data processing system are possible in which the in 4 components can be provided), wherein the camera K can be permanently integrated in the mobile phone or can be provided as a replaceable peripheral component. Especially in small mobile devices such as mobile phones, it is important that applications, such as the analysis of a scene described below, resource-saving algorithms, which are minimized in terms of memory requirements, such as the computational power, the resources of the devices to make the most possible use, but without necessitating additional processor or memory expansion.

in the The following will now be in the optical evaluation section OAA feasible Mechanisms or individual procedures are presented in detail.

Primary detection mechanism

It is now up 5 referenced, in which a schematic representation of the components or individual method steps for realizing the primary detection mechanism are shown. An image captured by the camera K is first supplied to an optional image contrast enhancement BKV by a histogram-based Histogram Equalization per KV of the individual pixels of the captured image. The image thus improved with the contrast is then fed to a processing by a horizontally oriented positive gradient filter GF, by means of which brightness-specific horizontal contours, ie sharp chiaroscuro transitions or dark-light transitions can be extracted. The first processed image thus obtained is fed to a segmentation algorithm SA, in which all pixels with a gray value above a certain threshold are assigned the gray scale value "white", while the remaining pixels are assigned the gray scale value "black". In this way, for example, only certain sharp horizontal chiaroscuro transitions can be made visible. In such a second processed image obtained, the remaining "white" regions can be examined for their area, position or geometry. The segments defined in the segmentation algorithm SA can then be subjected to a first examination step US1 by determining the presence of a face and the basis of the positional relationship of corresponding defined segments or the positional relationship and the shape or extent of the segments, including biogeometric criteria for a face on the position of the face or the corresponding mouth can be closed.

Such a described primary detection mechanism will now be described with reference to 6A to 6C be presented in detail. It is first in 4A the image captured by the camera of the user in a vehicle, which is present as a gray scale image, that is, the image consists of individual pixels, which is assigned a respective specific gray level value. The optional image contrast enhancement BKV (cf. 5 ) is then subjected to a processing by a horizontally oriented positive gradient filter to determine horizontal structures, ie significant brightness transitions. Such a (first) processed image is in 6B seen in which light-dark transitions viewed from top to bottom, starting from an average gray scale value, according to the sharpness of the transition to an increased gray scale value (toward "white") transition, while dark-bright-brightness transitions from top to bottom starting from a middle grayscale value into a lower grayscale value (towards "black"). By applying thresholding (where the threshold value is selected as a function of the brightness of the captured image, for example based on a histogram representation of the gray scale values of the individual pixels in the acquired image), pixels whose gray scale value is above the brightness-dependent threshold are assigned the value "white", while the other pixels are assigned the value "black". In this way, a large number of unimportant details can be removed in a (second) edited image. Furthermore, in the segmentation algorithm SA now obtained white areas can be removed whose size or shape does not correspond to specific criteria, in particular biogeometric criteria with respect to a face. After discarding these too small or too large white areas or areas not relevant in the form and by assigning the value "black" to the corresponding pixels, the actual segments relevant for the later examination are determined. For better analysis, these segments are inscribed in rectangles, as in 6C can be seen. In the first examination step US1, the defined remaining segments are examined in sequence for their geometric or positional relationship to one another. The investigation is based on the fact that the two eyebrows (see the eyebrow areas ABB in 4A . 4B ) or eyeglass contours in a human face, a third segment located below these first two segments is sought, which corresponds to a possible oral segment (cf. area MB in 6A . 6B ) based on the brightness transition from the upper lip region to the intermediate lip region. If a first and a second horizontally lying segment are found, such as the segments S1 and S2, their distance from one another, in this case the distance b, is determined starting from the respective center points of the segments. Subsequently, below the two segments S1 and S2, a third segment is searched whose distance, the distance a, to the first two segments or a connecting line thereof is in a first biogeometric relationship with respect to a human face. Such a ratio of a to b may have a value between 1.0: 1 to 1.7: 1, but in particular a value of 1.2: 1 to 1.5: 1. Depending on the scope of use for the first ratio of a to b, in 6C the segment S3 and S4 may be considered as a possible third segment (potential oral segment). According to an advantageous embodiment, therefore, in this case it is still possible to search for a fourth relevant segment, which is located immediately below the third segment. Based on the fact that under normal circumstances, the lower lip of a mouth segment is curved slightly forward with respect to the face and thus creates a shadow area below the lower lip, this shadow area may, in the case of a kind of plausibility check, be a fourth segment which must be located immediately below the third segment. serve. Based on this review, segment S4 can then be identified as a potential oral segment, with segment S5 representing the shadow below the lower lip. Now, an area in the captured image of 6A stored, which corresponds to the area identified as a potential oral segment (segment S4) in size and position. This area, which will be referred to as the analysis area, will be used for the following feature extraction.

It It should be noted that the entire process of the primary detection mechanism, as just described, in a single pass carried out which at the end of the passage is the one originally recorded Image can be discarded and only the stored pixel data of the analysis area needed become. As will be further appreciated, within this single pass the image data of the captured image with very simple algorithms significantly reduced the requirement for minimizing memory footprint, Calculation time and lead time.

feature extraction

The following feature extraction is used, for example, to check the plausibility of whether there is really a face with mouth or lips in the analysis area found. First up 7 referenced, in the again of the camera K in 5 captured scene accordingly 6A is shown, wherein the now to be analyzed area AB of the mouth area, which was stored during the implementation of the first detection mechanism is marked. This area is enlarged in 8th shown. According to the in 5 processing with a horizontally oriented gradient filter is now also the in 8th magnified section AB subjected to processing with a horizontally oriented positive gradient filter. It should be noted that prior to processing with the gradient filter again an image contrast enhancement corresponding to the image contrast enhancement BKV in 5 can be carried out. Processing with the horizontally oriented positive gradient filter now produces a third processed image in which (horizontal) light-dark transitions, in particular from the upper lip region to the intermediate lip region, or in other words extracted from the region of the upper lip to the region below the upper lip become. In this way it is possible to obtain individual segments of the upper lip, which can be connected together and smoothed. The now found or smoothed upper lip line can now be evaluated with regard to its geometric properties (length, curvature, etc.), ie whether it corresponds to biogeometric features of an actual upper lip or upper lip line. It is now possible that there are several light-dark transitions in the analysis area AB, which can represent a top lip line. If a first upper lip line or a combination of several segments identified as the upper lip line, as just described, is evaluated negatively, a further light-dark transition or a further section of a plurality of light-dark transitions, which can be combined to form a top lip line, is formed investigated the presence of certain geometric properties corresponding to an actual upper lip. In experiments it has been shown that a repetition of the investigation with a third most likely bright-dark transition region is possible, but usually unsuccessful, which is why according to this embodiment of the feature extraction to the explained below in more detail secondary detection mechanism for finding the position of a face or an oral part is passed.

If a found upper lip line is evaluated as positive (corresponding to an actual upper lip or upper lip line, respectively), a second test step or extraction step is carried out in which the lower lip or lower lip line is to be identified. Because the exact extraction of the lower lip due to generally weaker contouring of Lower lip is difficult by means of a filter, such as the gradient filter described above, in this case, a vertical search algorithm is applied, starting from the relatively well recognizable upper lip or upper lip line, which by the upper white squares OL in 8th is looking down the first color-like or grayscale-like pixel value along the vertical. This makes it possible to skip bright areas, such as the teeth, as well as dark areas, such as the pharynx area, and to have as close a lower lip area as possible to the area 8th formed by the line of white squares UL is identified. The corresponding found upper and lower lip lines OL and UL can then be examined with respect to their geometric properties (length and curvature).

Secondary detection mechanism

As above the feature extraction mentioned has been possible, it is possible that after performing of the primary Detection mechanism 'and after a plausibility check in a first test step the feature extraction of those found by the primary detection mechanism Analysis area no actual Mouthpiece may have. In this case, the secondary detection mechanism used to find a mouthpiece analysis area. It should be mentioned, that the secondary Detection mechanism already algorithmic in the passage of the primary detection mechanism considered can be, d. H. after capturing the first image of the scene. It but it is also possible that the second detection mechanism but only in a second Pass with respect to one second captured image of the scene can be performed. Will the secondary detection mechanism already performed while going through the primary detection mechanism, so can The identified or specified segments then only when Call the secondary Detection mechanism are evaluated.

The reason for the failure of the primary detection mechanism may be that with heavily contoured glasses, sunglasses, or continuous eyebrows, there are no two separate eyebrow segments available for facial geometric assessment, so in this case a different detection mechanism should be used. In the case of the second detection mechanism, the same processing steps as those in FIG 5 described steps BKV, GF and SA performed. Instead of the first examination step US1, however, here a second examination step US2 is used, in which the face geometry is used again to find the lips within the image captured by the camera K, a segment being sought here instead of the first and the second segment, which may correspond geometrically to the length of the two eyebrows together.

Starting from a in 9A The image shown having the face of a user (a person in a vehicle) having highly contoured glasses is intended to be the face or mouth part of that face. After performing the in 5 explained processing steps BKV, GF, SA with respect to the image of 9A , will be in 9B shown (second) edited image (corresponding to 6C ), in which only significant light-dark transitions of certain relevant size and shape are present. First, the horizontal length b 'of each of these existing segments is determined, as shown in segment S6. Now, for each segment, shown here by way of example on the segment S6, a subjacent segment is sought, which has a certain distance a 'therefrom, wherein the ratio a' to b 'must have a second biogeometric ratio with respect to a face. This second ratio of a to b is in the range of 1: 1 to 1.7: 1, but in particular between 1.2: 1 to 1.5: 1. As it is in 9B can be seen, a further segment S7 is found to the segment S6, through which the conditions of the second ratio can be satisfied. If there are several segments through which the second ratio can be satisfied, it is possible to find further test criteria for these plurality of segments. For example, the length of the further below the eyebrow representing segment can be checked, which has to lie within predetermined parameters. If, for example, the length of the further segment is too long (eg greater than the length b), this can not be identified as a potential mouth segment.

is now by means of the secondary Detection mechanism to a first corresponding to the eyebrows Segment a second of the mouthpiece corresponding segment found has been found, so this area found in position and size in the captured image as an analysis area and the above feature extraction shown are supplied.

In summary, it can thus be stated that the method carried out in the optical evaluation section OAA is advantageous for analyzing a scene or for finding, tracking and analyzing an oral area, in particular for data processing installations with limited resources. Through the use of facial geometry segmentation methods, the visual or optical data volume for further processing can very quickly be reduced to small segment quantities be made. Built-in plausibility and robustness test (in the feature extraction) also allow a high degree of reliability and prevent misdetection.

All Procedures can advantageous for determining the position of the user relative to the mobile terminal be used by moving from the position of the face, in particular the mouth part, in the picture the position of the user is computed. there it is not even necessary to determine the absolute position, but only the relative in successive pictures that make up the relative movement of the user the mobile terminal can be calculated. According to this relative movement then the selection agent moved to the graphical user interface.

Claims (24)

Mobile terminal with - a user interface ( 4 ) with a selection means positionable on the user interface ( 5 ), Means for measuring a position of a user ( 3 ) relative to the mobile terminal ( 1 ), - means for deriving a relative movement of the mobile terminal ( 1 ) to the user ( 3 ) from the position of the user ( 3 ), - means for positioning the selection agent ( 5 ) on the user interface ( 4 ) in dependence of the relative movement. Mobile terminal according to claim 1, characterized in that the means for measuring the position of the user ( 3 ) Means for detecting the position of the face ( 2 ) of the user ( 3 ) exhibit. Mobile terminal according to claim 1 or 2, characterized in that the means for measuring the position of the user ( 3 ) Means for detecting the position of the lips and / or an eye of the user ( 3 ) exhibit. Mobile terminal according to one of the preceding claims, characterized in that the mobile terminal for measuring the position of the user ( 3 ) an optical sensor (K) for taking a picture of the user ( 3 ) having. Mobile terminal according to claim 4, characterized in that on the user interface ( 4 ) to control the relative movement as a feedback respectively the currently recorded image of the user ( 3 ) can be displayed. Mobile terminal according to one of claims 4 or 5, characterized in that the mobile terminal ( 1 ) Lighting means to the user ( 3 ) to illuminate. Mobile terminal according to one of claims 4 to 6, characterized in that the mobile terminal ( 1 ) has an evaluation device (OAA, SE) through which the image of the user ( 3 ) can be analyzed with the following steps: - defining segments (S1-S7) in the image which have brightness-specific features; - Checking a positional relationship (a, b) of the predetermined segments to each other; Deriving a presence and / or a position of the user, in particular a face of the user, in the image when a selection of fixed segments (S1, S2, S4, S6, S7) has a certain positional relationship. Mobile terminal according to one of the preceding claims, characterized in that the selection means ( 5 ) on the user interface ( 4 ) can be displayed. Mobile terminal according to one of the preceding claims, characterized in that the mobile terminal ( 1 ) Has stabilizing agent in order to ensure a fluid movement of the selection agent ( 5 ) to ensure. Method for operating a mobile terminal, in which - a position of a user ( 3 ) relative to the mobile terminal ( 1 ), - from the position of the user ( 3 ) a relative movement of the mobile terminal ( 1 ) to the user ( 3 ), - a selection agent ( 5 ) on a user interface ( 4 ) of the mobile terminal ( 1 ) is positioned in dependence of the relative movement. Method according to Claim 10, - in which an image of the user ( 3 ) - where the user's position ( 3 ) relative to the mobile terminal ( 1 ) is determined by analyzing the image. Method according to Claim 11, in which the image of the user ( 3 ) is analyzed with the following steps: - defining segments (S1-S7) in the image which have brightness-specific features; - Checking a positional relationship (a, b) of the predetermined segments to each other; Deriving a presence and / or a position of the user, in particular a face of the user, in the image when a selection of fixed segments (S1, S2, S4, S6, S7) has a certain positional relationship. The method of claim 12, wherein the captured image is composed of individual pixels having respective gray level values and / or color values are assigned. A method according to claim 12 or 13, wherein the brightness-specific features sharp brightness transitions and / or have sharp color transitions. The method of claim 14, wherein the step setting segments in the captured image is a modification (GF) of the image by means of a gradient filter, in particular one positive horizontal gradient filter, includes a first to produce an edited image, at the location of a brightness transition and / or a color transition in the captured image corresponding pixels in the first edited Image with a grayscale value provided by the sharpness of the Brightness transfer in the captured image. The method of claim 15, wherein the step setting segments in the captured image binarization processing (SA) of the first processed image to make it a second to produce a processed image, at the corresponding pixels, whose greyscale value exceeds a predetermined threshold, assigned a "white" value will, while the remaining pixels assigned a "black" value becomes. The method of claim 16, wherein the step specifying segments in the captured image (SA) of a region of the second processed image with contiguous ones Pixels assigned a "white" value is, in a geometric shape, in particular a rectangle, as a fixed Segment includes. The method of claim 16 or 17, wherein certain Regions of the second processed image with contiguous "white" pixels Reason of certain parameters regarding size and / or shape of the areas or discarding the geometric shape that inscribes it. Method according to one of claims 12 to 18, wherein in the Step of checking the positional relationship in a first examination step each of the defined segments is examined as to whether a second (S2) fixed to this Segment exists, which is essentially horizontal to the examined fixed segment (S1), wherein in the presence of the second fixed segment after a third fixed segment (S4) is sought, which is below the examined and the second Segments is located, and for that is, a distance (b) from the examined to the second fixed segment, and a distance (a) of a link between the examined and the second fixed segment to the third fixed segment has a first predetermined ratio. Method according to one of claims 12 to 19, wherein in the Step of checking the positional relationship in a second examination step each of the defined segments is examined to see whether a fourth fixed Segment (S7) exists, which is set below the examined Segment (S6) is located, and for that is, the horizontal length (b ') of the examined fixed segment and the distance (a ') of the fourth fixed segment to the examined fixed segment a second predetermined one relationship exhibit. A method according to claim 19 or 20, wherein said first or the second predetermined ratio has a value of 1: 1 to 1: 1.7, in particular from 1: 1.2 to 1: 1.5. A method according to any one of claims 19 to 21, further comprising a Step of extracting biometric features with respect to a Mouth part of an analysis area around the found third (S4) or fourth (S7) segment. The method of claim 22, wherein in the detected Image in an analysis area (AB) located at the position of the third or fourth fixed segment, and the includes respective segment, light-dark and / or dark-bright-brightness transitions determined be to the presence of an upper lip and lower lip of a Derive mouth. Program product that when it comes to a data processing system loaded and executed is a method according to any one of claims 10 to 23 or a device according to one of the claims 1 to 9 goes into effect.