WO2023156475A1

WO2023156475A1 - Method for protecting information displayed on a display device and display device

Info

Publication number: WO2023156475A1
Application number: PCT/EP2023/053792
Authority: WO
Inventors: Sharat RAVI SHANKAR; Christian Lennartz; Christian Hess; Friedrich SCHICK; Jonas GROSSE HOLTHAUS; Vera Charlotte KOCKLER
Original assignee: Trinamix Gmbh
Priority date: 2022-02-15
Filing date: 2023-02-15
Publication date: 2023-08-24

Abstract

A method for protecting information displayed on a display device (1) from unauthorized views, the method comprising the steps: receiving (S1) an image showing surroundings of the display device (1), detecting (S2) one or multiple relevant faces on the image, determining (S3), for each relevant face detected on the image, whether it is authorized to look at the display device (1), generating (S4) a verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device (1) or not based on a result of the determining step (S3), and outputting (S5) the verification signal.

Description

Method for protecting information displayed on a display device and display device

The present disclosure relates to a method for protecting information displayed on a display device from unauthorized views. The present disclosure further relates to a display device, which is, in particular, adapted to protect information shown thereon from unauthorized views.

Display devices such as laptops or smartphones are sometimes used for displaying sensitive information, for example online banking information. It is desirable to protect such information against unauthorized access and from unauthorized viewers. In particular, it is desirable to prevent an unauthorized viewer spying over the shoulder of an authorized viewer from seeing sensitive information, and to prevent an unauthorized viewer seeing the sensitive information when the authorized viewer has left the display device unattended.

It is therefore an object of the present disclosure to improve the protection of information displayed on a display device.

According to a first aspect, a method for protecting information displayed on a display device from unauthorized views is provided. The method comprises the steps of: a) receiving an image showing surroundings of the display device, b) detecting one or multiple relevant faces on the image, c) determining, for each relevant face detected on the image, whether it is authorized to look at the display device, d) generating a verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device or not based on a result of the determining of step c), and e) outputting the verification signal.

According to a second aspect, a display device is provided. The display device comprises: a display unit, an imaging unit for recording an image showing surroundings of the display device, a processor unit adapted to: detect one or multiple relevant faces on the image, determining, for each relevant face detected on the image, whether it is authorized to look at the display device, and generating a verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device or not based on a result of the determining performed by the processor unit, and an output unit for outputting the verification signal.

The features and embodiments described in the following apply to both the method of the first aspect and the display device according to the second aspect.

The step of detecting and/or determining may comprise using a machine learning model, in particular an artificial neural network

The display device may automatically recognize that an unauthorized person is viewing the display device using face recognition. Accordingly, the display device can for example recognize that an unauthorized person is spying on the information on the display device by looking over the shoulder of an authorized user or that an unauthorized person is looking at the information on the display device when the authorized user has briefly and/or rapidly left the display device. Accordingly, a verification signal indicating whether all relevant faces capable of viewing the information on the display device are authorized can be generated. This verification signal can be used to emit a warning and/or to hide sensitive information when at least one of the detected faces are unauthorized. Thereby, the privacy of the sensitive information can be ensured.

Determining, for each relevant face detected on the image, whether it is authorized to look at the display device corresponds to checking an access right for the face.

The display device can be a smartphone, a tablet, a laptop, a personal computer (PC), an automated teller machine (ATM), a media player, a mobile device or any similar device. In particular, the display device includes a display (display unit) which may be a screen or a touch screen. The display device can be used to display visual information, in particular including sensitive information to be protected against the view of unauthorized persons. Such information can include text, images, videos or the like. In the following, the term "person" can be used as a synonym to the term "user". The term "face" in particular always refers to a human face.

The image represents the surroundings of the display device, in particular a portion of the surroundings that faces the display unit of the display device. The image can be captured by an imaging unit such as a camera. The surroundings can correspond to an area (for example a three-dimensional area) covered by a field of view of the imaging unit. For example, the surroundings include an area that is within a predefined distance of the imaging unit and that in particular allows collecting an image of a face of a person with a current setting and resolution of the imaging unit. For example, the surroundings correspond to an area in front of the display unit. The surroundings can have a cuboid shape, a shape of a half-sphere or a shape of a sphere portion. The half sphere or sphere portion can be centered around the center of the display unit and in particular have a radius of 4m, 5m or 10m.

The imaging unit can be orientated such that it is capable of sensing the presence of a person's while the person is interfacing, either passively or actively, with the display unit of the display device. In particular, the imaging unit is a front camera of the display device. The imaging unit can be orientated on a same side of the display device as the display unit. The imaging unit can face the faces of persons interfacing, either passively or actively, with the display unit. The imaging unit can be provided within a housing of the display device (embedded within the display device, in particular within the display unit) or as an add-on unit, such as a webcam. The imaging unit can be adapted to capture one image, a sequence of images and/or a video of the surroundings.

The imaging unit can be an infrared (IR) camera. Using an IR camera can be advantageous because a red blue green (RBG) camera of the display device is often already used or blocked by an application having exclusive access to the RBG camera. In smartphones for example, IR cameras are often used for less purposes, and are typically only used by the operating system of the smartphone for authentication. In embodiments, at least two imaging devices, e.g. cameras are provide, wherein a first device is configured to capture a visual image of the use and/or the surroundings and a second imaging device is configured to capture an infrared image of the surroundings,

The imaging unit may record a flood light image (which can be an image illuminated by a flood light source), so the image is taken from the scene (surrounding the display device) which is either lighted by ambient light or a flood light source. The imaging unit may also record an image while the scene is illuminated with patterned light, for example a point cloud. Such an image can contain information like distance or materials, for example skin and/or skin patterns. Using flood light, patterned light or a combination of both allows analyzing the scene in great detail and false analyses can be avoided.

The display device may include a processor (such as a processor unit), such as a central processing unit (CPU). The processor can perform image processing on the image to detect relevant faces thereon. Face detection can include the process of detecting and/or locating a face within the image. In particular, all faces on the image are detected and form the relevant faces. Relevant faces can also be faces that agree with a predetermined relevance criterion, that will be described in detail below. In particular, at least one face can be detected in the image. If a face is detected, the method steps c) to e) can be subsequently performed. If no face is detect- ed, the method steps c) to e) may not be performed, which avoids unnecessary energy consumption, and the steps a) and b) may instead be repeated.

In embodiments, detecting one or multiple relevant faces on the image, and/or determining whether a face is authorized to look at the display device, i.e. authenticating the face, includes the use of a data-driven model, e.g. a classification model, a machine learned model, an artificial neural network, in particular, a convolutional neural network or a vision transformer. One may contemplate of encoder devices for generating the low-level representation.

In embodiments, the step of determining/authenticating comprises: classifying the image data associated with the detected face using a trained machine learning model. The machine learning model may include an artificial neural network, in particular a convolutional neural network.

The neural network (NN) may be trained using training data sets mapping ground truth feature vectors associated with captured images of surroundings and/or faces and their assigned access rights. The trained NN then receives feature vectors corresponding to the image and outputs an access right information for the user.

In order to detect faces on the image, the image can be analyzed for the presence of a face using a variety of face detection techniques available on the market. For example, a trained neural network can be used. Such a neural network can be trained using labelled images including images with and without faces and corresponding labels.

The processor unit may be capable of determining, for each detected relevant face, whether it is authorized to look at the display device. The process of determining whether each face is authorized in particular involves recognizing that a detected relevant face is associated with a particular person or user. Once the face recognition is performed, it can be determined whether the recognized relevant face is an allowed (in particular authorized) face, in particular in accordance with data prestored in a database that indicates which faces are authorized or not. The process of determining whether a detected relevant face is that of an authorized person is also referred to as "authentication". In particular, the determination whether a relevant face is authorized to look at the display device involves authenticating the detected relevant face and determining whether it is an authorized or unauthorized relevant face.

The detected relevant face can be compared to a database containing prestored representations of faces of authorized persons. The database may be part of the display device and/or located in a cloud that can be accessed by the display device. For example, for each detected relevant face, one or multiple features (such as a skin pattern, color of eyes, skin or hair or the shape of nose, mouth, eyes or face) of the face are determined and compared with corresponding features stored in the database. If a detected relevant face has features identical with features of a prestored representation of a face of an authorized person (or if more than a predetermined number of features of the detected relevant face, such as 90%, are identical with the prestored features), it is determined that the detected relevant face is that of an authorized person. Otherwise, if a detected relevant face has not all features identical with features of prestored representations of a face of authorized persons (or if less than a predetermined number of features of the detected relevant face, such as 90%, are identical with the prestored features), it is determined that the detected relevant face is not that of an authorized person and that it is rather the face of an unauthorized person. The features can represent biometric information.

An authorized person can be a person that is an authorized user of the display device and that is allowed to access any information displayed on the display device in accordance with a prestored indication regarding the allowability of this person. For example, a user that is currently logged on to the display device is an authorized person. In another example, all persons of a same family or of a same company can be considered as authorized persons. An authorized person is a person that may look at the display device in accordance with a prestored indication regarding the allowability of this person.

An unauthorized person can be a person that is an unauthorized user of the display device and that may not access to all the information displayed on the display device. For example, users that are not currently logged on to the display device can be unauthorized persons. An unauthorized person is a person that is not authorized to look at the display device.

The verification signal that can be generated by the processor can be a signal summarizing the results of the determinations whether relevant faces are authorized or not for all relevant faces. In particular, the verification signal may be a binary signal indicating either that all detected relevant faces are authorized or that at least one of the detected relevant faces is unauthorized.

In embodiments, the image of the surroundings is an infrared image, a reflection image, an image obtained by illuminating the surroundings with coherent light.

Outputting the verification signal can be performed by the output unit. Outputting the verification signal can include storing the verification signal in a storage unit, transmitting the verification signal to another device or to another unit of the display device, transmitting the verification sig- nal to an application executed on the display device, and/or outputting the verification signal to a user, for example by displaying it visually on the display unit.

According to an embodiment, the method further comprises the step of: f) if the verification signal indicates that not all relevant faces detected on the image are authorized to look at the display device: hiding at least part of the information displayed on the display device and/or emitting a warning to warn a user.

A protective measure involving hiding at least part of the information displayed on the display device and/or warning a user (preferably, an authorized user) that at least one unauthorized person is viewing the display device can be performed to protect the information displayed on the display device. Preferably, the parts of the information hidden from the display device can be sensitive information such as a bank account number, a telephone number, a password, private photos, or the like. Hiding sensitive information can be performed by applying a filter, for example a blurring filter or a distorting filter, onto all or parts of the information displayed on the display unit. Hiding sensitive information can also be achieved by covering some or parts of the information displayed display unit and/or by preventing the display unit from displaying some or parts of the information.

The warning can be a visual warning, an audible alarm, a warning on a mobile phone or the like. The protective measure can be performed or initiated by the unit, device or application receiving the output verification signal.

According to a further embodiment, the method further comprises the step of: g) displaying at least part of the information on the display device if the verification signal indicates that all relevant faces detected on the image are authorized to look at the display device.

In particular, if the verification signal indicates that all recognized faces viewing the display device are those of authorized persons, the display device starts or continues displaying the information without any restrictions and in particular including sensitive information. Method step g) may be performed by the display unit of the display device.

According to a further embodiment, the method further comprises the steps of: h) detecting all faces on the image, i) determining, for each face detected on the image, whether it is a relevant face or not in view of a predetermined relevance criterion. A face can be considered as relevant if it complies with properties defined in the predetermined relevance criterion. In particular, to determine the relevance of a face, features of the face are extracted and compared with the predetermined relevance criterion. If the extracted features comply with the predetermined relevance criterion, the detected face is classified as being relevant. Otherwise, it is classified as being irrelevant.

There may be cases where the presence of an unauthorized viewer is irrelevant, for example because he is too far away (further away than a maximum distance), looks in a different direction or from an angle which makes it unlikely that the viewer can see sensitive information, for example because a protective film is used which darkens the screen when viewed from an angle. These examples will be described below.

A relevant face may be one that is capable of seeing information displayed on the display unit of the display device. The relevant faces may be all faces that are detected in the received image, a part of all faces that are detected in the received image or none of the faces that are detected in the received image. Considering only relevant faces allows reducing the required computational power for determining whether each face is authorized to look at the display device. In particular, the determination whether a detected face is authorized to look at the display device is performed only for the relevant faces and not for the irrelevant faces.

According to a further embodiment, the predetermined relevance criterion indicates: a distance between the face and the display device, a direction towards which the face is looking, an angle between the face and the display device, the presence of a living subject, and/or a liveliness of the face.

The relevance of the detected face can be determined by considering features of the detected faces using the processor unit. The relevance of a detected face can be determined from the image obtained by the imaging unit, for example by a gaze analysis or a distance determination.

For example, the processor unit may perform image processing on the image to determine a distance to the detected face. The distance can be a distance between a specific point of the imaging unit or the display unit (such as a center of the imaging unit or the display unit) and a specific point of the detected face (such as a nose or a closest eye of the face). This determined distance can then be compared to the predetermined relevance criterion indicating a maximum distance between the face and the display device. For example, if the determined distance is larger than a maximum distance included in the predetermined relevance criterion, the detected face is considered as being irrelevant.

For example, the processor unit may perform image processing on the image to determine a direction towards which the face is looking (gaze direction). The gaze direction can be defined as the vector that is normal to a specific portion of an eye of the detected face (such as a vector that is normal to an iris of the detected face). This determined direction towards which the face is looking can then be compared to the predetermined relevance criterion indicating allowed directions towards which the face may be looking. For example, if the determined direction towards which the detected face is looking is not within the allowed directions defined in the predetermined relevance criterion, the detected face is considered as being irrelevant. For example, the allowed directions can be any gaze directions that do not cross the display unit.

For example, the processor unit may perform image processing on the image to determine an angle between the face and the display device. In particular, an angle between the face and the display device may be defined as an angle between a vector normal to a specific point of the detected face (in particular of one of the eyes or of the nose) and a vector normal to a specific point of the display unit or imaging unit (such as a center of the display unit or imaging unit). An angle of 0° would correspond to a face being located centrally in front of the display device, and the angle would increase as the face is located more and more on sides of the display device. The determined angle can be compared to the predetermined relevance criterion indicating a maximal angle between the face and the display device. For example, if the determined angle is larger than the maximum angle included in the predetermined relevance criterion, the detected face is considered as being irrelevant.

Liveliness detection can correspond to determining whether the detected face is that of a real and living human or not. In other words, liveliness detection allows distinguishing real and living human faces from photos, sculptures, drawings or other representations of a face. Liveliness detection may be performed such that faces on posters, photos on a desk or the like do not accidentally falsify the verification signal. Liveliness detection can be performed by detecting the material skin in a face, for example from a pattern light image, or by detecting blood flow or cardiac activity detected by recording several images at a short interval and comparing these.

In embodiments, determining the liveliness includes determining a vital sign of the detected faces or the image of the surroundings. In embodiments, a vital sign measure is determined for detecting a liveliness. E.g. a high vital sign measure indicates a high liveliness with respect to a lower vital sign measure derived from a face having less liveliness.

A speckle contrast may represent a measure for a mean contrast of an intensity distribution within an area of a speckle pattern. In particular, a speckle contrast K over an area of the speckle pattern may be expressed as a ratio of standard deviation o to the mean speckle intensity <l>, i.e.,

Speckle contrast values are generally distributed between 0 and 1.

A vital sign measure, preferably, represents a measure indicative of whether the object from which the coherent electromagnetic radiation was reflected shows a vital sign. The vital sign measure is determined based on the speckle contrast. Thus, the vital sign measure may depend on the determined speckle contrast. If the speckle contrast changes, the vital sign measure derived from the speckle contrast may change accordingly. A vital sign measure may be a single number or value that may represent a likelihood that the object is a living subject.

Preferably, for determining the speckle contrast, the complete speckle pattern of the image of the surroundings is used. Alternatively, for determining the speckle contrast, a section of the complete speckle pattern may be used. The section of the complete speckle pattern, preferably, represents a smaller area of the speckle pattern than an area of the complete speckle pattern. The section of the speckle pattern may be obtained by cropping the reflection image. One may contemplate cropping identified faces in the image of the surroundings.

In embodiments, the method further comprises the step of predicting the presence of a living subject, wherein the step includes at least one sub-step of: determining a confidence score based on the determined vital sign measure, comparing the confidence score to a predefined confidence threshold, and predicting the presence of a living subject based on the comparison. 20 11.

The method may comprise the step of using a trained artificial neural network configured to receive the determined vital sign measure as input and to provide the confidence score as output. The confidence score may be generated from a vital sign measure, e.g., represented by a single number or a value, or from a vital sign map, e.g., represented by a matrix of vital sign measures. The confidence score may represent a degree of confidence indicative of a presence of a living subject. The confidence score may be expressed by a single number or a value. Preferably, the confidence score is determined by comparing the determined vital sign measure to a reference, e.g., to one or more reference vital sign measures each being preferably associated with a particular confidence score.

Alternatively, the confidence score may be determined using a neural network that is trained for receiving the determined vital sign measure as input and for providing the confidence score as output. The neural network may be trained with historic data representing historic vital sign measures and associated confidence scores.

The confidence threshold is predetermined to ensure a certain level of confidence that the object indeed is a living subject. The confidence threshold may be predetermined in dependence on a security level required for a specific application, e.g., for providing access to a device. For example, the confidence threshold may be set such that the confidence score may represent a comparatively high level of confidence, e.g., 90 % or more, e.g., 99 % or more, that the object presented to a camera is a living subject. Only when the comparison with the confidence threshold yields that the confidence score is high enough, i.e. , exceeding the confidence threshold, the presence of a living subject is approved. If the confidence score is below the confidence threshold, access to the device will be denied. A denial of access may trigger a new measurement, e.g. a repetition of the method of detecting a vital sign and of making use of the vital sign measure for predicting the presence of a living subject as described before.

In particular, the liveliness of a face can be determined based on the vital sign measure as referred to above.

Other information that can be included in the predetermined relevance criterion includes a current time, lighting condition or the like.

According to a further embodiment, the steps a) to e) are executed by an operating system of the display device.

In particular, the display device is constructed such as to be capable to perform steps a) to e) without any additional programming or the like. For example, programs for performing steps a) to e) can be part of the firmware of the display device. According to a further embodiment, the step e) of outputting the verification signal includes out- putting the verification signal to an application displaying the information when executed on the display device.

The application (app) can be a program which, when executed on the display device, performs specific tasks including displaying the information on the display device. The application may be linked to the display device such as to obtain the verification signal from the display device. The application may then perform an adequate measure, such as one of the protective measures described above, upon reception of the verification signal.

According to a further embodiment, the predetermined relevance criterion is provided by the application.

For example, depending on the sensitivity of the displayed information, the application specifies the predetermined relevance criterion and accordingly indicates how the display device should set the boundary between relevant and irrelevant faces.

The application (requesting application) may provide a threshold value indicating the sensitivity required to shield information from unauthorized users. For very sensitive information, the threshold may be high, so even remote viewers or those not directly looking on the screen result in a signal of unauthorized viewers. Other apps may display less sensitive information, so the threshold is lower to avoid unnecessary shielding action.

According to a further embodiment, the method further comprises the steps of: j) initiating (in particular triggering) the execution of steps a) to e), wherein the initiating is performed: at predetermined times, repeatedly every time that a predefined time interval elapses (in particular periodically), upon receiving a new login by the display device; upon detecting a motion and/or acceleration of the display device; and/or upon request by an application displaying the information when executed on the display device.

Initiating the execution of the method steps at predetermined times means that predetermined time stamps are stored and that when a timer reaches the predetermined time stamps, the initiation is started. Initiating the execution of the method steps repeatedly every time that a predefined time interval elapses in particular means that a timer triggers the process repeatedly after the predefined time interval, for example every second or every five seconds.

It is also possible to trigger upon change of a user, for example a new login into the display device. Also, handing over the display device or putting it on a table, which may be detected by a motion detector, may serve as trigger (motion and/or acceleration detection).

Further, the initiation can be triggered by the request of the application which displays sensitive information and hence requires avoidance of unauthorized access. Such "on demand" triggering consumes less power which is particularly relevant for mobile devices running on a battery.

According to a further embodiment, the method further comprises the steps of: k) repeating the steps a) to c) a predetermined verification time interval after determining, in step c), that one relevant face is not authorized to look at the display device, and l) generating the verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device or not based on a result of the determining of the repeated step c).

A person may accidentally look at the display unit of the display device for a very short amount of time. A false verification signal to hide information may not be desired in such cases. To avoid such unnecessary hidings of information, a second image may be triggered a short time after an unauthorized view is detected, for example after 0.5 seconds or 1 second. Instead of a single second image, a sequence of second images can be captured at predetermined time intervals. If the second image still has an unauthorized view, a corresponding verification signal is generated. Request and analysis for a second image by the processor and/or the imaging unit may be made depending on the security level, which may be indicated by the application. If the application indicates the need for a very high security level, every unauthorized view may lead to a corresponding verification signal while a second image is only analyzed for lower levels of security.

As indicated above, the authentication of the detected relevant faces can include determining features of the detected relevant faces. An example of such features includes skin patterns. The following explains how such skin patterns features can be determined from the image. A "skin pattern feature" refers to a pattern feature which has been reflected by skin. Skin pattern features can be determined by making use of the fact that skin has a characteristic way of reflect- ing light: It is both reflected by the surface of the skin and also partially penetrates the skin into the different skin layers and is scattered back therefrom overlying the reflection from the surface. This leads to a characteristic broadening or blurring of the pattern features reflected by skin which is different from most other materials. This characteristic broadening can be detected in various ways. For example, it is possible to apply image filters to the pattern features, for example a luminance filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or median filter; a grey-level-occurrence-based contrast filter; a grey-level-occurrence-based energy filter; a grey-level-occurrence-based homogeneity filter; a grey-level-occurrence-based dissimilarity filter; a Law’s energy filter; a threshold area filter. In order to achieve best results, at least two of these filters are used. Further details are described in WO 2020/187719.

The result when applying the filter can be compared to references. The comparison may yield a similarity score, wherein a high similarity score indicates a high degree of similarity to the references and a low similarity score indicates a low degree of similarity to the references. If such similarity score exceeds a certain threshold, the pattern feature may be qualified as skin pattern feature. The threshold can be selected depending on the required certainty that only skin pattern features shall be taken into account, so minimizing the false positive rate. This comes at the cost of identifying too few pattern features are recognized as skin pattern features, i.e. yield a high false negative rate. The threshold is hence usually a compromise between minimizing the false positives rate and keeping the false negative rate at a moderate level. The threshold may be selected to obtain an equal or close to equal false negative rate and false negative rate.

It is possible to analyze each pattern feature separately. This can be achieved by cropping the image showing the body part while it is illuminated with patterned light into several partial images, wherein each partial image contains a pattern feature. It possible that a partial image contains one pattern feature or more than one pattern features. If a partial image contains more than one pattern feature, the determination if a particular pattern feature is a skin pattern feature is based on more than one partial images. This can have the advantage to make use of the correlation between neighboring pattern features.

The determination of skin pattern features can be achieved by using a machine learning algorithm. The machine learning algorithm is usually based on a data-driven model which is parametrized to receive images containing a pattern feature and to output the likelihood if the pattern feature is skin or not. The machine learning algorithm needs to be trained with historic data comprising pattern features and an indicator indicating if the pattern feature has been reflected by skin or not. Particularly useful machine learning algorithms are neural networks, in particular convolutional neural networks (CNN). The kernels of the CNN can contain filters as described above capable of extracting the skin information out the broadening or blurring of the pattern feature.

In embodiments method comprises capturing the image of the surroundings by an imaging device associated with the display device, wherein, preferably, the display device comprises at least two imaging devices, a first imaging device configured to capture a visual image of the user and a second imaging device configured to capture the image of the surroundings.

Using the second imaging device, e.g. an infrared camera, to capture the surroundings does not interfere with the camera for visual imaging, which may be used by the user for other applications. E.g. a videoconference call can be performed using the first camera and contemporaneously verifying as to whether by-standing persons may attend the call using the infrared camera device (second imaging device).

According to an embodiment, the display device of the second aspect is adapted to perform the method steps according to the first aspect or according to an embodiment thereof.

According to a further embodiment: the display unit, the imaging unit, the processor unit and the output unit are part of the hardware of the display device, the display device is configured to execute an application displaying the information when executed on the display device, and the output unit is adapted to output the verification signal to the application.

According to a third aspect, a computer-readable data medium, in particular a non-transitory computer-readable data medium, storing a computer program including instructions for executing steps of the method according to the first aspect or an embodiment of the first aspect is provided.

In embodiments, a computer-program or computer-program product comprises a program code for executing the above-described methods and functions by a computerized control device when run on at least one control computer, in particular when run on the display device. A computer program product, such as a computer program means, may be embodied as a memory card, USB stick, CD-ROM, DVD or as a file which may be downloaded from a server in a network. For example, such a file may be provided by transferring the file comprising the computer program product from a wireless communication network. According to a fourth aspect, use of the verification signal obtained by the method according to the first aspect or an embodiment of the first aspect for information protection on a display device is provided.

In particular, the display device and/or the application uses the verification signal to accordingly decide whether a corresponding protective measure is required or not and/or to initiate such a protective measure.

In a further aspect, the display device, in particular the display device according to the second aspect or an embodiment thereof, is a smartphone or a tablet having a translucent screen as the display unit. In this aspect, the imaging unit is for example a front camera. The imaging unit can be located on an interior of the display device, behind the translucent screen. The imaging unit can include an illumination source for emitting light through the translucent screen to illuminate the surroundings. The imaging unit can further include an optical sensor for receiving light from the surroundings and passing through the translucent screen. The optical sensor may general a sensor signal in a manner dependent on an illumination of a sensor region or light sensitive area of the optical sensor. The sensor signal may be passed onto the processing unit to reconstruct an image of the surroundings and/or to process the image, in particular along the lines defined above.

Further possible implementations or alternative solutions of the invention also encompass combinations - that are not explicitly mentioned herein - of features described above or below in regard to the embodiments. The person skilled in the art may also add individual or isolated aspects and features to the most basic form of the invention.

Further embodiments, features and advantages of the present invention will become apparent from the subsequent description and dependent claims, taken in conjunction with the accompanying drawings, in which:

Fig. 1 shows a display device according to a first embodiment;

Fig. 2 shows components of the display device of Fig. 1 ;

Fig. 3 shows a method for protecting information displayed on a display device according to a first embodiment; Fig. 4 shows a method for protecting information displayed on a display device according to a second embodiment;

Fig. 5 shows a method for protecting information displayed on a display device according to a third embodiment;

Fig. 6 shows an example of a face authentication; and

Fig. 7 shows a display device according to a second embodiment.

In the Figures, like reference numerals designate like or functionally equivalent elements, unless otherwise indicated.

Fig. 1 shows a display device 1 according to a first embodiment. The display device 1 is a smartphone and includes a translucent touchscreen 3 as a display unit. The display unit 3 is configured for displaying information. Such information can include a text, image, diagram, video, or the like. Besides the display unit 3, the display device 1 includes an imaging unit 4, a processor unit 5 and an output unit 6. In Fig. 1 , the imaging unit 4, the processor unit 5 and the output unit 6 are represented by dashed squares because they are located within a housing 2 of the display device 1, and behind the display unit 3 when viewed from an exterior of the display device 1.

Fig. 2 shows the components of the display device 1 located on the interior of the housing 2 in more detail. Fig. 2 corresponds to a view onto the display unit 3 from an interior of the display device 1, with the imaging unit 4, the processor unit 5 and the output unit 6 being located in front of the display unit 3.

The imaging unit 4 is a front camera. The imaging unit 4 is configured to capture an image of surroundings of the display device 1. In detail, an image of a scene in front of the display unit 3 of the display device 1 can be captured using the imaging unit 4. The surroundings are here defined as a half-sphere located in front of the imaging unit 4 and centered around a center of the display. The radius of the half-sphere is 5m.

The imaging unit 4 includes an illumination source 9 and an optical sensor 7 having a light sensitive area 8. The illumination source 9 is an infrared (IR) laser point projector realized by a vertical-cavity surface-emitting laser (VCSEL). The IR light emitted by the illumination source 9 shines through the translucent display unit 3 and generates multiple laser points on the scene surrounding the display device 1. When an object, such as a person, is located in front of the display device 1 (in the surroundings of the display device 1, facing the display unit 3 and the imaging unit 4), an image of the object is reflected towards the imaging unit 4. This reflected image also includes reflections of the laser points.

Instead of the illumination source 9 being an IR laser pointer, it may be realized as any illumination source capable of generating at least one illumination light beam for fully or partially illuminating the object in the surroundings. For example, other spectral ranges are feasible. The illumination source may be configured for emitting modulated or non-modulated light. In case a plurality of illumination sources is used, the different illumination sources may have different modulation frequencies. The illumination source may be adapted to generate and/or to project a cloud of points, for example the illumination source may comprise one or more of at least one digital light processing (DLP) projector, at least one Liquid crystal on silicon (LCoS) projector, at least one spatial light modulator, at least one diffractive optical element, at least one array of light emitting diodes, at least one array of laser light sources.

The optical sensor 7 Is here realized as a complementary metal-oxide-semiconductor (CMOS) camera. The optical sensor 7 looks through the display unit 3. In other words, it receives the reflection of the object through the display unit 3. The image reflected by the object, such as the person, is captured by the light sensitive area 8. When light from the reflected image reaches the light sensitive area 8, a sensor signal indicating an illumination of the light sensitive area 8 is generated. Preferably, the light sensitive area 8 is divided into a matrix of multiple sensors, which are each sensitive to light and each generate a signal in response to illumination of the sensor.

Instead of a CMOS camera, the optical sensor 7 can be any type of optical sensor designed to generate at least one sensor signal in a manner dependent on an illumination of the sensor region or light sensitive area 8. The optical sensor 7 may be realized as a charge-coupled device (CCD) sensor.

The signals from the light sensitive area 8 are transmitted to the processor unit 5. The processor unit 5 is configured to process the signals received from the optical sensor 7 (which form an image). By analyzing a shape of the laser spots reflected by the object and captured by the optical sensor 7, the processor unit 5 can determine a distance to the object and a material information of the object. In the example of Fig. 1 and 2, the imaging unit 4, the processor unit 5 and the output unit 6 can exchange data via connection cables 10. The output unit 6 is a portion of the display unit 3 on which the verification signal can be output. In other embodiments, the out- put unit 6 can be a loudspeaker, a storage unit and/or a light emission unit for outputting the verification signal.

The display device 1 shown in Fig. 1 and 2 is capable of protecting information displayed on the display unit 3 thereof. This can be achieved by executing a method for protecting information displayed on the display device 1 using the display device 1 of Fig. 1 and 2. Example for such methods will be described in the following in conjunction with Fig. 3 to 6.

Fig. 3 shows a method for protecting information displayed on a display device 1 according to a first embodiment. In a step S1 , an image showing surroundings of the display device 1 is received by the processor unit 5. In detail, the image showing the surroundings (the front) of the display device 1 is captured using the imaging unit 4 described above.

In a step S2, a relevant face is detected in the image received by the processor unit 5. To this end, the processor unit 5 performs face detection on the image using a neural network trained specifically for human face detection. Such a neural network can be trained using labelled images including images with and without faces and corresponding labels. In the example of Fig.

3, the processor unit 5 detects all faces in the received image. The result of the face detection is an image highlighting the presence and position of one or multiple faces in the image. In particular, the position of the detected faces is indicated by marks on the image that is being processed by the processor unit 5. A copy of the image with the marks indicating the position of the detected faces can be stored in a database of the display device 1.

In a step S3, the processor unit 5 determines, for each detected face, whether it is authorized to look at the display device 1 . To this end, the processor unit 5 uses another trained neural network for extracting face features from the detected faces. This further neural network can be trained using labelled images including images with and without specific features (such as the size, shape and color of the face, nose, mouth, eyes, ears, glasses, teeth, hair, and the like) and corresponding labels.

For each relevant face detected in step S2, the processor unit 5 provides a list of features (which is a list of characteristics of the face). The processor unit 5 then compares, for each relevant face, the list of features associated with the faces with lists of features prestored in a database (not shown in the figures, but the database is located in the display device 1 or in a cloud that the processor unit 5 can access) representing faces of authorized users. For example, only the user currently logged in to the display device 1 and his assistant are authorized persons, while every other person is unauthorized. In particular, for each session on the display device 1 , the display device 1 stores information regarding the authorized persons, for example by storing features of the faces of the authorized persons. Accordingly, in step S3, the processor unit 5 determines the features of all the relevant faces from the captured image and determines whether they correspond to the stored features of the faces of the current user and his assistant. If the processor unit 5 determines that the features of a relevant face correspond to the stored features of the faces of the current user or of his assistant, the face is authenticated as being that of an authorized person. Otherwise, if the processor unit 5 determines that the features of a relevant face do not correspond to the features of the stored faces of the current user or of his assistant, the face is authenticated as being that of an unauthorized person.

In a step S4, the processor unit 5 generates a verification signal indicating whether all relevant faces detected in step S2 are faces of authorized persons based on the determination of step S3. If all detected relevant faces are faces of authorized persons, a first type of verification signal is generated. If one or more detected relevant faces are faces of unauthorized persons, a second type of verification signal is generated.

In a step S5, the verification signal is output. This is performed by the output unit 6. The verification signal is here stored in a storage unit. If the verification signal indicates that one or more detected relevant faces are faces of unauthorized persons, the output unit further displays a warning message on the display unit 3, thereby also hiding any sensitive information that is being displayed on the display device 1. Here, the hiding is achieved by displaying a warning message in front of the sensitive information. Thereby, it can be avoided that an unauthorized person views the sensitive information.

Fig. 4 shows a method for protecting information displayed on a display device 1 according to a second embodiment. The method of Fig. 4 can also be executed by the display device 1 of Fig. 1 and 2. The method of Fig. 4 includes the same method steps S1 to S5 as Fig. 3, the description of which will hence be omitted to avoid repetitions. In addition to the steps S1 to S5, the method of Fig. 4 further includes steps SO and S6 to S9, which are detailed in the following.

The step SO is an initiation step in which the execution of the subsequent method steps S1 to S9 by the display device 1 is triggered. In detail, the initiation is triggered by a mobile application (app) that is executable by the display device 1 and displays information on the display unit 3 upon its execution. The app is a banking app which displays sensitive information such as a bank account number and the like.

At the end of step S2, if no relevant faces are detected in the image, no further action is taken in the method of Fig. 4 (this corresponds to step S9 of Fig. 4). According to the method of Fig. 4, step S3 is performed only if relevant faces have been detected in step S2.

Further, as explicitly shown in the method of Fig. 4, two different verification signals can be output, depending on the content of the verification signal. Namely, if a verification signal is generated in step S4 that indicates that all relevant faces detected on the image are from authorized persons, a verification signal indicating that all viewers are authorized is emitted in step S5. Otherwise, if a verification signal is generated in step S4 that indicates that at least one of the relevant faces detected on the image is from an unauthorized person, a verification signal indicating that a viewer is unauthorized is emitted in step S6.

In steps S5 and S6 of Fig. 4, the output unit 6 outputs the verification signal to the app that initiated the entire process. Accordingly, in steps S7 and S8, the app decides upon the action to be performed based on the content of the received verification signal. In the present example, in step S7, the app decides not to perform any preventive measure because there is no unauthorized viewer detected. In step S8, the app decides to hide all sensitive information displayed by the app on the display unit 3 because an unauthorized viewer was detected. In step S8, the sensitive information is hidden by blurring all parts of the display unit 3 that show information relating to the app.

Fig. 5 shows a method for protecting information displayed on a display device 1 according to a third embodiment. The method of Fig. 5 can also be executed by the display device 1 of Fig. 1 and 2. The method of Fig. 5 includes the same method steps SO to S9 as Fig. 4, the description of which will hence be omitted to avoid repetitions. In addition to the steps SO to S9, the method of Fig. 5 further includes steps S10 and S11 , which are detailed in the following.

In detail, in step S10, the initiating (requesting) app provides a predetermined relevance criterion to the processor unit 5. The predetermined relevance criterion allows the processor unit 5 to determine, in a step S11 , whether a face detected in step S2 is a relevant face or not. Indeed, there may be cases where the presence of an unauthorized viewer is irrelevant, for example because he is too far away, looks in a different direction or from an angle which makes it unlikely that the viewer can see sensitive information, for example because a protective film is used which darkens the screen when viewed from an angle. The predetermined relevance criterion provides thresholds for the maximal distance, angle, or orientation of the viewer. Using these thresholds and information extracted from the detected faces, the processor unit 5 can determine whether each face is relevant or not. Only the relevant faces are authenticated in step S3. If only irrelevant faces are detected, no further action is taken (similarly to step S9).

Fig. 6 shows a detailed example of how the face authentication of step S3 described above can be performed. Namely, based on the 2D image received from the IR camera 8 (step S31), features of the detected 2D face are compared with the stored features of the authorized persons (step S32). Only if the features match in step S32, the method continues with step S33. In step S33, the liveliness of the face is determined using the dots on the image corresponding to the reflected laser spots (step S33). Determining the liveliness allows distinguishing from faces on photos or paintings, for example. Only if the liveliness is confirmed in step S33, then the face is determined as being an authorized face in step S34 of Fig. 6.

Step S33 may involve determining speckle contrast values to obtain a vital sign measure to predict the presence of a living subject in the captured image. Determining a vital sign can be carried out as described above.

Fig. 7 shows a display device 1 according to a second embodiment. The display device 1 according to the second embodiment is equally configured to perform the method of any one of Fig. 3 to 6. However, in addition to the IR laser point projector 9 (patterned light projector) of Fig. 1 and 2, the display device 1 of Fig. 7 further includes a flood light projector 11 for emitting flood light through the display unit 3 toward the surroundings of the display device 1.

In Fig. 7, the face detection neural network used by the processor unit 5 in step S2 is represented by the reference numeral 12. In Fig. 7, the output unit 6 forms a more general interface to apps for communication between the processor unit 5 and the apps. Since the information relating to which viewer is authorized is security relevant, it is provided on a secure enclave 13 including the neural network 14 used for face authentication in step S3.

Although the present invention has been described in accordance with preferred embodiments, it is obvious for the person skilled in the art that modifications are possible in all embodiments. For example, instead of storing feature information on authorized persons and comparing them with features extracted from the detected relevant faces, feature information on unauthorized persons may instead or additionally be stored for comparison purposes. Further, the order of the described method steps can be modified. Reference signs:

1 display device

2 housing

3 display unit

4 imaging unit

5 processor unit

6 output unit

7 optical sensor

8 light sensitive area

9 illumination source

10 connection cable

11 flood light projector

12 neural network

13 secure enclave

14 neural network

50 initiating the execution of the method

51 receiving an image showing surroundings of the display device

52 detecting one or multiple relevant faces on the image

53 determining, for each face, whether it is authorized to look at the display device

54 generating a verification signal

55 outputting the verification signal indicating authorized viewers only

56 outputting the verification signal indicating at least one unauthorized viewer

57 displaying at least part of the information on the display device

58 hiding at least part of the information displayed on the display device

59 terminating the execution of the method

510 receiving a predetermined relevance criterion from an app

511 determining whether each face it a relevant face or not

531 receiving 2D image from IR camera

532 comparing features of detected 2D face with stored features of authorized persons

533 determining liveliness of detected face

534 determining that face is authorized

Claims

1. A method for protecting information displayed on a display device (1) from unauthorized views, the method comprising the steps of: receiving (S1) an image showing surroundings of the display device (1), detecting (S2) one or multiple relevant faces on the image, determining (S3), for each relevant face detected on the image, whether it is authorized to look at the display device (1), generating (S4) a verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device (1) or not based on a result of the determining step (S3), and outputting (S5) the verification signal.

2. The method according to claim 1, further comprising the step of: if the verification signal indicates that not all relevant faces detected on the image are authorized to look at the display device (1): hiding (S8) at least part of the information displayed on the display device (1) and/or emitting a warning to warn a user.

3. The method according to any one of the preceding claims, further comprising the step of: displaying (S7) at least part of the information on the display device (1) if the verification signal indicates that all relevant faces detected on the image are authorized to look at the display device (1).

4. The method according to any one of the preceding claims, further comprising the steps of: detecting all faces on the image, determining (S11), for each face detected on the image, whether it is a relevant face or not in view of a predetermined relevance criterion.

5. The method according to claim 4, wherein the predetermined relevance criterion indicates: a distance between the face and the display device (1), a direction towards which the face is looking, an angle between the face and the display device (1), the presence of a living subject, and/or a liveliness of the face.

6. The method according to any one of the preceding claims, further comprising the steps of: detecting all faces on the image; and determining (S11), for each face detected on the image a liveliness of the face in terms of a vital sign measure.

7. The method of claim 6, further comprising determining the vital sign measure as a function of a speckle contrast within an area of the image of the surroundings where at least one face is detected.

8. The method of claim 6, further comprising determining the vital sign measure as a function of a speckle contrast of the entire image of the surroundings.

9. The method according to any one of the preceding claims, further obtaining a speckle pattern of the image of the surroundings; and determining a speckle contrast of the image of the surroundings.

10. The method according to any one of the preceding claims, further comprising the step of predicting the presence of a living subject including at least one sub-step of: determining a confidence score based on the determined vital sign measure, comparing the confidence score to a predefined confidence threshold, and predicting the presence of a living subject based on the comparison.

11. The method according to any one of the preceding claims, wherein the step of detecting (S1) and/or determining (S2) comprises using a machine learning model, in particular an artificial neural network.

12. The method of claim 10 and 11, comprising using a trained artificial neural network configured to receive the determined vital sign measure as input and to provide the confidence score as output.

13. The method according to any one of the preceding claims, wherein the steps of receiving (S1) an image, detecting (S2) one or multiple faces, determining (S3) whether each face is authorized, generating (S4) a verification signal and outputting (S5) the verification signal are executed by an operating system of the display device (1).

14. The method according to any one of the preceding claims, wherein the step of outputting (S5) the verification signal includes outputting the verification signal to an application displaying the information when executed on the display device (1).

15. The method according to claims 4 and 7, wherein the predetermined relevance criterion is provided by the application.

16. The method according to any one of the preceding claims, further comprising the steps of: initiating (SO) the execution of the steps of receiving (S1) an image, detecting (S2) one or multiple faces, determining (S3) whether each face is authorized, generating (S4) a verification signal and outputting (S5) the verification signal, wherein the initiating is performed: at predetermined times, repeatedly every time that a predefined time interval elapses, upon receiving a new login by the display device (1); upon detecting a motion and/or acceleration of the display device (1); and/or upon request by an application displaying the information when executed on the display device (1).

17. The method according to any one of the preceding claims, further comprising the steps of: repeating the steps of receiving (S1) an image, detecting (S2) one or multiple faces and determining (S3) whether each face is authorized at a predetermined verification time interval after determining, in the step of determining (S3) whether each face is authorized that one relevant face is not authorized to look at the display device (1), and generating the verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device (1) or not based on a result of the determining of the repeated step of determining (S3) whether each face is authorized.

18. The method according to any one of the preceding claims, further comprising capturing the image of the surroundings by an imaging device (7) associated with the display device (1).

19. The method according to claim 18, wherein the display device (1) comprises at least two imaging devices (7), a first imaging device (7) configured to capture a visual image of the user and a second imaging device configured to capture the image of the surroundings.

20. The method according to any one of the preceding claims, wherein the image of the surroundings is an infrared image.

21. A computer-readable data medium storing a computer program including instructions for executing steps of the method according to any one of the preceding claims.

22. Use of the verification signal obtained by the method according to any one of the preceding claims for information protection on a display device (1).

23. A display device (1) comprising a display unit (3), an imaging unit (4) for recording an image showing surroundings of the display device (1), a processor unit (5) adapted to: detect one or multiple relevant faces on the image, determining, for each relevant face detected on the image, whether it is authorized to look at the display device (1), and generating a verification signal indicating whether all relevant faces detected on the image are authorized to look at the display device (1) or not based on a result of the determining performed by the processor unit, and an output unit (6) for outputting the verification signal.

24. The display device according to claim 23, which is adapted to perform the method steps according to any one of claims 2 to 20.

25. The display device according to claim 23 or 24, wherein the display unit (3), the imaging unit (4), the processor unit (5) and the output unit (6) are part of the hardware of the display device (1), the display device (1) is configured to execute an application displaying the information when executed on the display device (1), and the output unit (6) is adapted to output the verification signal to the application.