SE1450769A1 - Improved tracking of an object for controlling a non-touch user interface - Google Patents

Description

SUMMARY It is an aspect of the teachings of the present application to solve the problems listed above by providing a computer device comprising a screen and a control unit, the control unit being arranged to detect and track an object via a camera provided by a camera. video stream and adjusting an illumination of said screen to illuminate the object so that the computer device can successfully track the object, the computer device being characterized in that the control unit is further arranged to determine that the object cannot be tracked at a prevailing light condition and in response adjusting the illumination of the screen , the illumination of the object being provided by illumination of the screen.

Such a computer device is capable of properly illuminating an object to be tracked without the need for additional photo transmitters.

In one embodiment, the control unit is further arranged to detect a distance to the object to be tracked and to adjust the illumination of the screen based on the distance.

In one embodiment, the control unit is further arranged to detect an ambient light condition and to adjust the illumination of the screen based on the ambient light condition.

In one embodiment, the control unit is further arranged to determine that it is not possible to track the object at a prevailing light condition and in response to this to adjust the illumination of the screen.

In one embodiment, the computer device is a mobile communication terminal. In one embodiment, the computer device is a tablet or a (laptop) computer. In one embodiment, the computer device is a game console. In one embodiment, the computer device is a media device such as a television set or a media system.

It is also an aspect of the teachings of the present application to solve the problems listed above by providing a method for use in a computer device comprising a screen, the method comprising detecting and tracking an object via a video stream provided by a camera and adapting a illuminating the screen to illuminate the object so that the computer device can successfully track the object, the method being characterized in that it further comprises determining that the object cannot be tracked at a prevailing light condition and in response adjusting the illumination of the screen, the lighting of the object being provided by screen illumination. It is further an aspect of the teachings of the present application to solve the problems listed above by providing a computer readable medium comprising instructions which, when loaded and executed by a control unit, such as a processor, in a computer device entail execution. of a method in accordance with this.

The inventors of the present invention have, after creative and insightful reasoning, realized that by using a camera designed to operate in the visible light spectrum, the ambient light can be advantageously used to illuminate the object.

By coming to the realization that the lighting provided by an (active) screen is part of the ambient light and as such can be used to illuminate the object, the need for special extra lamps is also reduced, which is of paramount importance. By coming to this ingenious insight, the inventors also overcame the prevailing view in the field that in order to reduce energy consumption, the illumination of the screen should be reduced in dark environments as the illumination needed to display the content distinctly, compared to a bright environment, is reduced. In addition, there is a well-rooted delusion in the area towards using strong lighting in a dark environment as a brightly lit screen reduces a user's night vision.

The method taught herein thus provides a simple solution to a long-standing problem and runs counter to the prevailing preconceived notion of screen lighting.

The teachings herein find use in control systems for devices with user interfaces such as mobile phones, smartphones, touchpads, computers (laptops and desktops), game consoles and media and other information entertainment devices.

Other features and advantages of the described embodiments will become apparent from the following detailed description, from the appended dependent claims, and from the drawings. In general, all terms used in the claims are to be construed in accordance with their usual meaning in the technical field, unless otherwise expressly stated herein.

All references to "an element, device, component, organ, step, etc.]" shall be construed as referring to at least one example of the element, device, component, organ or step, etc., if nothing else is explicitly stated.

The steps in the various methods described herein need not be expressly performed in exactly the order described, unless expressly stated otherwise. BRIEF DESCRIPTION OF THE DRAWINGS The invention will be described in more detail with reference to the accompanying drawings, in which: Figures 1A, 1B and 1C each show a schematic view of a computer device in accordance with the teachings herein; Figure 2 shows a schematic view of the components of a computer device in accordance with the teachings herein; Figure 3 shows a schematic view of a computer readable memory in accordance with the teachings herein; Figures 4A, 4B and 4C show an example of an embodiment of a computer device in accordance with the teachings herein; and Figure 5 shows a fate diagram illustrating a general method in accordance with an embodiment of the teachings herein.

DETAILED DESCRIPTION The described embodiments will now be described in more detail with reference to the accompanying drawings, in which certain embodiments of the invention are shown. However, this invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example for the purpose of this description to be accurate and complete and to fully explain the scope of the invention to those skilled in the art. The same number always refers to the same element.

Figure 1 generally shows a computer device 100 in accordance with an embodiment herein. In one embodiment, the computer device 100 is provided for network communication, either wireless or wired. Examples of a computer device 100 are: a personal computer, desktop or laptop computer, a tablet, a mobile communication terminal such as a mobile phone, a smartphone, a personal digital assistant and a game console.

Three embodiments will be exemplified and described as being a smartphone in Figure 1A, a laptop in Figure 1B and a media unit 100 in Figure 1C. A media device is considered to be a computer device within the scope of this application in that it is arranged to receive digital content, process or calculate the content and present resulting or calculated media, such as image (s) and / or sound.

Referring to Figure 1A, a communication terminal in the form of a smartphone 100 includes a housing 110 in which a screen 120 is provided. In one embodiment, the screen 120 is a touch screen. In other embodiments, the screen 120 is a non-touch screen. The smartphone 100 further includes two buttons 130a, 130b. In this embodiment, there are two buttons 130, but any number of buttons is possible and depends on the design of the smartphone 100. In one embodiment, the smartphone 100 is arranged to display and operate a virtual button 135 on the touch core 120. It should be noted that the number of virtual buttons 135 depends on the design of the Smartphone 100 and an application executed on the Smartphone 100. The Smartphone 100 is also equipped with a camera 160. The camera 160 is a digital camera which is arranged to take video or still images by recording images on an electronic image sensor (not shown). ). In one embodiment, the camera 160 is an external camera. In one embodiment, the camera is, as a possibility, replaced by a source that provides an image stream.

Referring to Figure 1B, a laptop 100 includes a monitor 120 and a housing 110. The housing includes a controller or CPU (not shown) and one or more computer readable storage media (not shown), such as storage devices and internal memory.

Examples of storage devices are disk drives or hard drives. The portable computer 100 further includes at least one data port. Data ports can be wired and / or wireless. Examples of data ports are USB (Universal Serial Bus) ports, Ethemet ports or WiFi (according to IEEE standard 802.11) ports. Data ports are provided to enable a laptop 100 to connect to other computer devices or a server.

The portable computer 100 further includes at least one input device such as a keyboard 130. Other examples of input devices are computer heads, touch pads, touch screens, or joysticks to name a few.

The laptop 100 is further equipped with a camera 160. The camera 160 is a digital camera which is arranged to take video or still images by recording images on an electronic image sensor (not shown). In one embodiment, the camera 160 is an extreme camera. In one embodiment, the camera is alternatively replaced by a source that provides an image stream.

Referring to Figure 1C, a media unit, such as a television set, TV 100, includes a screen 120 and a housing 110. The housing includes a controller or CPU (not shown) and one or more computer readable storage media (not shown), such as storage units. and internal memory, for storing user settings and control software. The computer device 100 may further include at least one data port (not shown). Data ports can be wired and / or wireless. Examples of data ports are USB (Universal Serial Bus) ports, Ethemet ports or WiFi (according to IEEE standard 802.11) ports. Such data ports are provided to enable the TV 100 to be connected to an external storage medium, such as a USB stick, or to be connected to other computer devices or a server.

The TV 100 may further include an input unit such as at least one button 130 or a remote control 130b for operating the TV 100.

The TV 100 is further equipped with a camera 160. The camera 160 is a digital camera which is arranged to take video or still images by recording images on an electronic image sensor (not shown). In one embodiment, the camera 160 is an extreme camera. In one embodiment, the camera is alternatively replaced by a source that provides an image stream.

Figure 2 shows a schematic view of the general structure of a device according to Figure 1. The device 100 comprises a control unit 210 which is responsible for the overall operation of the computer device 200 and which is preferably implemented by any commercially available CPU ("Central Processing Unit"). ”Or central processor), DSP (“ Digital Signal Processor ”or signal processor) or any other electronically programmable logic device. The controller 210 is arranged to read instructions from the memory 240 and execute these instructions to control the operation of the computer device 100. The memory 240 can be implemented using any generally known technology for computer readable memories such as ROM, RAM, SRAM, DRAM, CMOS, FLASH , DDR, SDRAM or any other memory technology. The memory 240 is used for various purposes by the controller 210, one of which is for storing application data and program instructions 250 for various software modules in the computer device 200. The software modules comprise a real-time operating system, drivers for a user interface 220, an application manager as well as various applications 250 .

The computer device 200 further comprises a user interface 220, which in the computer device in Figures 1A, 1B and 1C consists of the screen 120 and the buttons 130, 1 3 5.

The computer device 200 may further include a radio frequency interface 230, which is adapted to allow the computer device to communicate with other devices via a radio frequency band using various radio frequency technologies.

Examples of such technologies are IEEE 802.11, IEEE 802.15, ZigBee, WirelessHART, WIFI, Bluetooth®, W-CDMA / HSPA, GSM, UTRAN and LTE to name a few.

The computer device 200 is further equipped with a camera 260. The camera 260 is a digital camera which is arranged to take video and still images by recording images on an electronic image sensor (not shown). The camera 260 is operatively connected to the controller 210 to provide the controller with a video stream 265, i.e. series of images taken, for further processing and, if possible, for use in and / or in accordance with one or more of your applications 250. In one embodiment, the camera 260 is an extreme camera or source of an image stream.

References to "computer-readable storage medium", "computer software product", "real / tangible / concretely designed computer program" etc. or a "control unit", "computer", "processor" etc. should of course include not only computers with different architectures such as simple / multi processor architectures and sequential (Von Neumann) / parallel architectures but also specialized circuits such as field programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other devices. References to computer programs, instructions, code, etc. shall be understood to include software for a programmable processor or hardware such as, for example, the programmable contents of a hardware device, whether instructions for a processor, or configuration settings for a fixed function unit, gate array or programmable logic unit etc. .

Figure 3 shows a schematic view of a computer readable medium as described above. The computer readable medium 30 is in this embodiment a data disk 30. In one embodiment, the data disk 30 is a magnetic data storage disk. The data disk 30 is arranged to carry instructions 31 which, when loaded in a control unit, such as a processor, execute a method or a method in accordance with the embodiments described above. The data disk 30 is arranged to be connected to or inside and read by a reading device 32, for inputting the instructions into the control unit. An example of a reading device 32 in combination with a (or fl era) data disk (s) 30 is a hard disk. It should be noted that the computer readable medium may also be other media such as compact discs, digital video discs, memory memory or other frequently used memory technologies. the instructions 31 may also be downloaded to a device which reads computer data 34, such as a laptop or other device capable of reading computer coded data on a computer readable medium by including the instructions 31 in a computer readable signal 33 transmitted via a wireless (or wired) interface (for example via the Internet) to the device which reads computer data 34 for inputting the instructions 31 in a control unit. In such an embodiment, the computer readable signal 33 is a type of a computer readable medium 30.

The instructions may be stored in a memory (not explicitly shown in Figure 3, but designated 240 in Figure 2) of the laptop 34. References to computer programs, instructions, code, etc. are to be understood as including software for a programmable processor or hardware such as, for example, the programmable contents of a hardware device, whether instructions for a processor, or configuration settings for a fixed function device, gate matrix or programmable logic unit, etc.

An improved way of tracking an object will be described below with reference to the accompanying figures. The example will be illustrated with a focus on the resulting illumination provided by a screen, but it should be understood that the processing is performed in part or in whole in a computer device comprising a control unit as described above with reference to Figures 1 and 2 or obtained performed by executing the instructions stored on a computer readable medium as described with reference to Figure 3.

Figure 4A shows an example of a computer device, in this example a laptop 100 as in Figure 1B, which is arranged to detect and track an object, such as a hand H, via a video stream provided by a camera 160. The laptop has a screen 120, on which object 135 is displayed. The screen is set to radiate or illuminate at an initial (or normal) level. Figure 4A indicates the initial illumination with dashed lines referred to as IL1. The initial lighting level depends on a number of factors that are obvious to a person skilled in the art and which can also be user-configurable.

In Figure 4A, the hand is at a distance D1 from the screen. In the example in Figure 4A it is assumed that the ambient light condition is bright enough to properly illuminate the hand H so that the camera and control unit using the associated computer instructions can track the hand H. How such an object H is detected and tracked is described in Swedish patent application SE 1250910 -5 and will not be discussed in detail in the present application. For further details regarding this, reference is made to the said Swedish patent application. It should be noted, however, that the teachings in the present application can be implemented by using other tracking methods than those described in Swedish patent application SE 1250910-5.

In the example of Figures 4A and 4B, the ambient light condition is not sufficient to successfully track a detected object, such as the hand H, when the hand H is placed at a greater distance, such as the distance D2, from the screen 120.

In one embodiment, the portable computer 100 is arranged to detect that an object is present in front of the screen 120 / camera 160 by analyzing the provided image stream. A method of detecting an object is based on the fact that an object to be tracked is most likely not statically placed in front of the camera 160 and motion can thus be detected by changes between the images in the image stream constituting the video stream.

Since it is only necessary for the control unit to detect changes to determine that there is movement of an object and thereby detect an object (such as the area where the changes are detected), the light required may be less than what is needed to actually track an object. . When tracking an object, details of the object are needed to determine how the object is moving and that it is the object being tracked that is actually moving.

Some factors affect how well an object can be detected. Examples of such factors are the color, reflection and structure (sharp or smooth edges) of the object. In a poorly lit room, for example, it is easier to detect a white object than a black object.

When it becomes impossible to track the object H using the illumination provided by the ambient light conditions, the laptop 100 is arranged to adjust the illumination of the screen 120 to increase the illumination and thereby the ambient light to better illuminate the hand H and enable successful detection of the object.

Referring to Figure 4B, the notebook 100 is arranged to detect that the hand H is at a distance D2 from the screen and in response adjust the illumination of the screen 120. In Figure 4B this is indicated by longer dashed lines emanating from the screen 120 and the increased the lighting is designated IL2.

By increasing the illumination of the screen, the ambient light conditions are improved and the notebook 100 is able to successfully track the hand H to receive control input as part of the user interface of the notebook 100. The 1 examples according to Figures 4A and 4B detect the control unit that the hand H is moved away from the screen 120 and in response increases the illumination of the screen 120.

Referring to Figure 4C, the portable computer 100 is arranged to detect that the hand H is detectable but not traceable and, in response, to adjust the illumination of the screen 120. This determination can be made by measuring the ambient light ratio, for example by analyzing the video stream provided by the camera 160. In Figure 4B, this is indicated by longer dashed lines starting from the screen 120 and the increased illumination is denoted IL2. In the examples of Figures 4A and 4C, the screen 120 is initially at a first (initial) illumination IL1 (Figure 4A), when it is either determined (as explained above) that the illumination is insufficient or that the surrounding lighting conditions change and become insufficient. Figure 4C illustrates the insufficient light condition by being shaded. The laptop 100 is arranged to detect that the light condition is not sufficient and in response increase the illumination of the screen 120. In Figure 4C this is indicated by longer dashed lines starting from the screen 120 and the increased illumination is denoted IL3.

In one embodiment, the portable computer 100 is arranged to determine that the object cannot be tracked by attempting to perform a tracking procedure without success and in response increase the illumination of the screen 120. Such tracking methods are described in, but are not limited to, Swedish patent application SE 1250910 -5 and will not be discussed in detail in the present application.

Thus, in one embodiment, the portable computer 100 is arranged to detect an object and determine that the object cannot be tracked under prevailing light conditions (possibly using initial illumination IL1), based on at least one of the alternatives of measuring the ambient light condition, detecting a distance and / or determine that a tracking procedure is unsuccessful and in response increase the illumination of the screen 120.

In one embodiment, the portable computer 100 is arranged to stepwise or linearly adjust the illumination of the screen 120 until and / or while the object to be tracked is able to be tracked successfully, for example by adjusting the illumination of the screen 120 so that the object is clearly distinguishable. , which can be determined by analyzing the image (s) in the video stream.

It should be noted that although the adaptation based on light ratio and the adaptation based on distance are described separately in the above, the two can be combined into an adaptation based on both the distance and the light ratio. The adaptation based on the determination of whether the object to be tracked can be traced can also be combined with the adaptation based on distance, the adaptation based on light conditions or a combination of these.

As mentioned above, the distinguishability of an object depends on a number of factors. In one embodiment, the portable computer 100 is arranged to store an appearance profile of a user's preferred control objects or objects to be tracked. Such as the user's hand or finger. The stored factors may relate to color, reflective features and / or structure. By having access to information about the object to be traced 10 15 20 25 30 ll and how easily it can be distinguished, the illumination level, possibly the initial illumination level, can be adjusted to enable a successful detection and detection of an object without having to determine a suitable lighting level by trying out. This can be done, for example, when a new user logs in or is detected by the computer device.

The stored appearance profile may differ depending on the ambient light condition and the laptop 100 may be arranged to take into account the ambient light condition when determining the initial illumination level (IL1).

In one embodiment, the laptop 100 is arranged to illuminate the screen 120 at the increased illumination level IL2, IL3 during a first time period and to illuminate the screens 120 at the initial illumination level IL1 when the first time period has elapsed.

Examples of the first time period are in the order of 1 to 10 seconds, 1 to 5 seconds, 1 second, 2 seconds or 3 seconds.

Figure 5 shows a fate diagram of a general method in accordance with the teachings herein.

A computer device detects and tracks 510 an object, such as a hand. The computer device determines that an object is insufficiently illuminated 520 and in response adjusts the illumination of the screen 530. The illumination of the object can be determined on the basis of the distance 523, the ambient light ratio 526 or an image analysis of the detected object 529.

The invention thus teaches that the computer device can use the lighting as well as the lighting condition of the screen to ensure that the lighting of the object to be tracked is sufficient to track the object and to adjust the lighting accordingly.

The teachings herein provide the advantage that an object can be tracked even in poorly lit conditions and without requiring expensive equipment.

Another advantage is that the teachings herein can even be implemented in existing devices through a software upgrade.

The invention has mainly been described above with reference to a few embodiments. However, it will be readily apparent to one skilled in the art that embodiments other than those described above are also possible within the scope of the invention as defined by the appended claims.

Claims (9)

10 15 20 25 30 35 12 PATENT REQUIREMENTS A computer device (100, 200) comprising a screen (120) and a control unit (210), the control unit (210) being arranged to: detect and track an object (H) via one provided by a camera (160, 260). holding video stream (265) and adapting an illumination (IL1, IL2, IL3) of said screen (120) to illuminate the object (H) so that the computer device can successfully track the object (H), the computer device (100, 200) being characterized by that the control unit (210) is further arranged to determine that the object (H) cannot be tracked at a prevailing light condition and in response to this adjust the illumination (IL1, IL2, ILS) of the screen (120), the illumination of the object (H) being provided of screen illumination (120). The computer device (100, 200) according to claim 1, wherein the control unit (210) is further arranged to detect a distance (D1, D2) to the object to be tracked (H) and to adjust the illumination (IL1, IL2, IL3) of the screen. (120) on the basis of the distance (D1, D2), so that if the control unit detects an increased distance, the illumination is increased. The computer device (100, 200) according to claim 1 or 2, wherein the control unit (210) is further arranged to detect an ambient light condition and to adjust the illumination (IL1, IL2, IL3) of the screen (120) on the basis of the ambient light condition. The computer device (100, 200) according to claim 1, wherein the control unit (210) is further arranged to dynamically adjust the illumination (IL1, IL2, ILS) of the screen (120) until the object (H) is clearly distinguishable. The computer device (100, 200) according to any one of the preceding claims further comprising a memory (240), and wherein the control unit (210) is further arranged to store in the memory an appearance profile for a known object to be tracked and adapt the lighting (IL1, IL2, IL3) of the screen (120) based on the stored appearance profile. The computer device (100, 200) of claim 5, wherein the stored appearance profile is associated with an ambient light condition. 10 15 20 13 The computer device (100, 200) according to any one of the preceding claims, wherein the control unit (210) is further arranged to illuminate the screen (120) at an adapted illumination level (IL2, IL3) during a first time period and when the first time period has for illuminating the screen. (120) at an initial illumination level (IL1). A method of use in a computer device (100, 200) comprising a screen (120), the method comprising: detecting and tracking an object via a video stream (265) provided by a camera (160, 260) and adapting a illumination (IL1, IL2, IL3) of the screen (120) to illuminate the object (H) so that the computer device can successfully track the object (H), the method being characterized in that it further comprises determining that the object cannot be traced at a prevailing light condition and in response, adjusting the illumination (IL1, IL2, ILS) of the screen (120), the illumination of the object (H) being provided by illumination of the screen (120). Computer readable storage medium (40) encoded with instructions (41) which, when loaded and executed on a processor, cause the method of claim 8 to be performed.