PT107038A - PROCESS THAT POSSIBLE THE USE OF ANY DIGITAL MONITOR AS A MULTI-TOUCH AND NEXT TOUCH SCREEN - Google Patents

Abstract

The present invention is based on a program executed by a computer, which uses as an image capture device, a 3D sensor (3) mounted on the projection of the top of the screen (2), perpendicular to it. The computer program has the capability of extracting the information FROM THE 3D SENSOR, THE PROCESSING OF WHICH THE USER FINGERS THAT ARE IN THE INTERACTION AREA AND AVAILABLE THIS INFORMATION TO THE GRAPHIC APPLICATIONS. THE PROGRAM ALSO ENABLES TO PERFORM A CALIBRATION OF THE SCREEN TO DEFINE THE AREA OF USE (2-A). AFTER CALIBRATION, THE PROGRAM BEGINS TO TRY INFORMATION FROM THE SENSOR AND TWO INTERACTIVE AREAS ARE DEFINED IN FRONT OF THE SCREEN: THE DISTANCE CONTROL AREA (B), BETWEEN THE DISTANCE CONTROL BARRIER (6) AND THE ALMOST- TOUCH (5), AND THE TOUCH AREA (A), BETWEEN THE NEAR-TOUCH BARRIER (5) AND THE SCREEN (2). THE TOUCH AREA ASSEMBLY AND DISTANCE CONTROL AREA ARE THE AREA OF INTERACTION (7).

Description

DESCRIPTION "PROCESS THAT POSSIBLE THE USE OF ANY DIGITAL MONITOR AS MULTI-TOUCH SCREEN AND QUICK-TOUCH" "

Technical domain

It's a process that turns any screen of any size into a multi-touch screen and almost touches a fraction of the cost of a touchscreen on the market. The near-touch name is due to the fact that it is not really necessary to touch the screen to interact, contrary to the current touch screens. The multi-touch name defines that it is possible for multiple people with different fingers to interact simultaneously, controlling the same content on the transformed screen.

In addition to the above, the touch experience is enhanced by a new modality: remote content control, which allows users to have two states of interaction.

By analogy, using the computer mouse and their respective buttons, you can move the mouse without pressing, allowing you to position the mouse freely without interacting with the contents, and only interact with them by clicking, such as " or " drag " or simply by pressing the left mouse button.

It should be noted that in the described process the definition of which interactions are desired with the remote control and quasi-touch states present in the displayed program are performed on the side of the graphic content construction.

This technology also allows for the massification of multi-touch technology, as it allows reusing any existing screen. The said process is based on a program executed by a computer, which uses as an image capture device a depth camera (3D sensor), taking advantage of its three-dimensional images to interpret the world around it. Said depth chamber should be mounted to the projection at the top of the screen, perpendicular thereto. The physical assembly of the components is decisive in their operation. Said assembly then enables the program to recognize the user and his fingers, as well as their distance from the screen, which is the main factor for the near-touch detection and remote control mentioned above.

The near-touch contents and desired remote control make the invented process suitable for use in places where hygiene and safety have to be guaranteed, such as operating blocks or the operation of heavy machinery or work involving dirt.

Public displays placed in shopping malls or airports, for example, are also exposed to hygiene problems, making the ability of the invention to detect touching actions without the user actually touching the screen highly valued.

What's more, the overall cost of this solution is lower than traditional touch-screen solutions.

State of the art The touch and multi-touch displays with the presented program have already been mentioned and implicitly previously compared, these being the main competitors to the process of the invention.

In addition to the touch screens there are also interactive projections made by a video projector, where it is possible to control content by touch and near-touch.

For touch screens, they have a single interaction state: the touch. It is not possible, for example, to determine the position of the finger relative to the screen and once the interaction occurs, using the analogy of the mouse again, it is as if we were always with the finger to press the left button, which sometimes leads to involuntary and undesirable interactions with the contents.

Only predefined graphical applications for such controls, which are very present in today's mobile devices, are able to circumvent this situation, using only and exclusively touch as a means of interaction.

This new paradigm of graphic application design has led to the disruption between what has already been done and what is being done for a specific device type, increasing the cost for companies to be present on different platforms.

As for interactive projections, most of them also use computer vision - the vision of a camera - to interpret interactions on a given surface where, in some cases, quasi-touch interaction already exists.

Again, these projections lack the remote control present in the disclosed invention. They also have other significant disadvantages, such as the shadow produced by the hand and even by the user's body, which is inefficient for good interaction.

On the other hand, the cost / life of a video projector is expensive and time-consuming, and there is not enough mass use compared to digital displays.

There are still some cases where universities and institutes are using a technique close to the invention, using the two sensors simultaneously, but whose intention is to reconstruct the user's hands three-dimensional, for interpretation of the interaction. In this case we are facing a remote control scenario but without notion of near-touch or even touch. Said solution significantly increases the logistics of the facility and has already been evaluated by several in-house experts as extremely complex to be a commercial product.

Description of the figures. Fig. 1 shows a simplified schematic of the four main components of the invention, the 3D sensor 3 being shown in the upper projection of the screen 2 the computer 1 and the computer program 4; 2 schematically shows the interaction area 7, comprising a near-touch area (A) and a remote control area (B), delimited by a near-touch barrier (5) and a remote control barrier (6), the display (2), the area of use (2-a) after calibration and a 3D sensor (3) in the upper projection of the screen; Fig. 3 shows, in flowchart, the steps of the screen calibration process (2): selection of the lower left corner (I), selection of the lower left corner (II) for the definition of the rectangular zone, selection of the lower right corner (III) to give continuity to the definition of the partial area and selection of the upper right corner (IV) to complete the definition of the interactive area; 4 shows, in simplified flowchart, the operation of the invention; 5 shows, in more detailed flowchart, the application of the invented process.

The process comprising the invention comprises four main components (Fig. 1): a 3D sensor (3), a computer program (4) that tracks information from the 3D sensor, a computer (1) ) or embedded device for running the computer program and a screen (2), area where the calibration and the display medium of the contents will be carried out. The 3D sensor allows, like a conventional camera, to acquire digital images, which are a set of data in computational form, but with one more field of information: depth.

The 3D sensors have the ability to provide the distance with an error, in an ideal scenario, of a millimeter, to which a certain point is of the sensor.

That extra dice, depth, vastly expanded the amount of applicabilities inherent in the area of computer vision.

Through the information acquired by the 3D sensor, the program analyzes the data obtained, in order to interpret the events that are in its field of vision, transforming them into data easily understood by the human being. This data is then made available in two different ways: - By controlling the mouse of the operating system where the program is running; or - Through a communication channel and a port, present in any operating system.

In the second case, a communication protocol is defined where the graphic program will have to interpret and process the information received in the desired way. The computer program requires a computer to run and perform all the calculation necessary to track the users' fingers.

Within the range of computers we have different operating systems, some that can be installed on any computer, but others that only for the platforms to which they are intended. On the fringe of this current complexity and diversity of supply, the presented software has the ability to run on Linux, MacOSX, Windows and Android.

Finally, a digital screen is required, that is, any type of video output that uses LCD, TFT, LED or even obsolete plasmas. It is these screens which are scattered and massified to a large scale and which can readily be used in the invention.

Presented that were the main components that participate in the invented process, it is now important to detail a key component of it: the computer program. The software developed has as its main capacity the extraction of the information coming from the 3D sensor, processing which fingers are in the area of interaction and make this information available in different ways to graphic applications that are developed or reused for interaction with the user.

This process of interaction is related to the area of the digital screen to be used and, therefore, variable. The developed software allows to perform a calibration in order to define this area with respect to the present installation. The same digital display will be tighter in terms of image for processing if the sensor is farther away and vice versa.

Generally digital displays are rectangular surfaces, but the software in question works well with any standard guadrilateral surface. The calibration process (Fig. 3) is based on the location definition of each of these corners. The program will now know which corner is being calibrated: upper left corner (I), lower left corner (II), lower right corner (III) or upper right corner (IV). The said calibration will define the area of the screen that will be used in the interaction with the user or users. The screen can thus be used by several users after each defining their area of use (2-a). As a merely exemplary - and never limiting - of the field of application of the invention, a number of tourists may define a more restricted area on a digital screen which, by way of example, informs the layout of a metropolitan network.

In order to do this calibration you can use your own finger, simply place it in the corner indicated by the calibration process and click on the tip of the finger present in the depth image.

After calibration is performed - which can be repeated as often as desired - the computer program starts to trace the information from the sensor. The program has two interactive regions in front of the screen (Fig. 2): the remote control area (B) and the near-touch area (A).

For better understanding, these areas should be viewed as a volumetric extrapolation of the screen to a defined distance: A distance to the barrier from the remote control and one to the near-touch barrier.

These barriers are invisible surfaces in 3D at a constant distance from the screen, predefined by the user through the computer program: They only exist in the computer program to define which zone of interaction in which the finger of a user is and its processing.

Any finger in the remote control area (B) will be tracked by the program, allowing you to interact with the surface.

A touch action is identified when a finger passes the near-touch barrier (5) and enters the near-touch area (A).

Any finger of a user within said near-touch area (A) will be tracked as a touch, allowing touch sensing to be detected at a desired distance, ie near-touch. This particularity is one of the greatest potentialities of the invention if one thinks that for cultural and hygienic reasons many users are not willing to touch the screens directly with their fingers.

That is, whenever the remote control barrier (6) is passed by a finger of the user, interaction becomes possible; if the finger exceeds the near-touch barrier (5), it will be perceived by the invention as a real touch. The remote control barrier (6) and the near-touch barrier (5) can be defined with precision of 1 millimeter, depending on the configuration. The maximum distance also depends on the installation configuration, more specifically the field of view of the 3D sensor.

With the 3D sensor placed farther, greater distances can be achieved. The position of the finger is a three-dimensional position - x, y, z - in relation to the screen. The z-depth coordinate represents the distance between the finger and the screen, measured in millimeters and perpendicular to the plane of the screen. Using this direction, the finger position is vectored on the screen and, using this projection, the - x, y - values are calculated in relation to the shape of the screen. The - x, y - values are relative irrational values between 0 and 1 which measure the relative distance between the left side of the screen and the upper part respectively. The upper left corner (I) corresponds to (0, 0) and the lower right corner (III) to (1, 1). Thus, the relative position is measured within the shape of the screen, making it adaptable to any resolution of the diqital screen.

When the finger is on the remote control barrier (6), the percentage is 1 and decreases linearly towards the touch barrier, reaching 0 in that position. The touch percentage is 1 when the finger is on the near-touch barrier (5) and 0 on the screen.

Using these two values one can measure the distances of the important interaction planes, ie the barrier of the remote control (6), the near-touch barrier (5) and the screen (2). The area between the screen and the barrier of the remote control is the interaction area (7).

All this information is sent by the data channel to the final application, whether it is an information panel of a metro network, a tourist information panel or a control panel during a surgery.

In summary, Figs. 4 and 5 represent the set of phases and procedures that support the invented process: The 3D sensor captures the information, communicating the computer program, which will process the acquired data (namely, the number of users and the position of the fingers - fundamental aspect -). By using the protocol, this data is sent to a graphical application, being possible the interaction with the user (s) (Fig.4).

In a more explicit and detailed way, Fig. 5 presents the set of phases, operations and procedures that make the invented process work: from the beginning, with the connection of the 3D sensor to the computer and from this to the screen, finger tracking program of the user (s) and consequent projection of the depth image of the finger (s) received by the 3D sensor and projected on the screen (each finger and its distance relative to the screen being identified) , sending the information of each finger to the graphical program (through network through pre-defined protocol) and the consequent creation of a new image. The invention functions as an intercom between the program and the operating system, so that consumers can create the applications that best suit their needs, reusing them if they are already developed, or in a working environment or internet.

This means that the end user does not have to throw away any application to use the invention by simply using its protocol; - Through a channel, which means that any final application can receive and use it, following only the defined protocol used.

This is the demonstration of the universality of the application of the invention and, as has been said, it allows to use any digital screen as a near-touch surface, that is, to transform the interaction in the touch area as if it were a real touch.

Porto, July 3, 2013.

Claims (1)

A method which enables the use of any digital monitor as a multi-touch and quasi-touch screen characterized by, by means of a screen 2, a 3D sensor 3 placed in the top projection thereof and a computer program 4, , establish a near-touch area (A) and a remote control area (B) on said screen and allow any screen after calibration to be used as a multi-touch and quasi-touch screen by one or more users on comprising: a) The calibration of the screen (2), which allows to define the respective area of use (2-a), greater or less, based on said process in the definition of location of each of the corners of the surface to be used on the screen; b) Recognition by the program of which corner is being calibrated: upper left corner (I), lower left corner (II), lower right corner (III) and upper right corner (IV); c) Extraction of information from the 3D sensor (3); d) processing which fingers are in the interaction area (7); e) The availability of the collected information to the graphical applications that are developed or reused for interaction with the users; f) The possibility of operation on any quadrilateral surface; g) The ability of the user to use his / her own finger to calibrate the screen; h) For the said calibration process, place the finger in the corner indicated by the calibration process and click on the tip of the finger present in the depth image on the screen; i) the tracking of information from the 3D sensor after calibration is performed, which can be repeated as often as desired; j) The possibility of said tracking of the information coming from the 3D sensor can be repeated as often as desired; k) The definition of two interactive areas in front of the digital screen: the area of the remote control (A) and the near-touch area (B); l) Distance to the distal control barrier (6) and distance to the near-touch barrier (5); m) Constant distances from said remote control barrier (6) and quasi-touch barrier (5), predefined by the user through the computer program; η) The possibility of any finger in the remote control area being tracked by the program, allowing interaction with the screen surface; o) The identification of a touch action whenever a finger of a user passes the near-touch barrier (5) and enters the touch area (A); (p) The definition of the barriers of the remote control area (6) and quasi-touch area (5) with an accuracy of 1 millimeter, depending on the configuration; q) tracking the user's finger position as a three-dimensional position relative to the screen; r) The vector projected position of the finger on the screen; s) the calculation of the shape of the screen from the use of the width and length values; t) A range of values ranging from 1 (when the finger is on the remote control barrier) and 0 (the touch barrier is reached), the touch percentage being 1 when the finger is on the touch barrier and 0 on the screen; u) The use of values between 1 and 0 to measure the distances of important interaction planes, ie the barrier of the remote control, the near-touch barrier and interactive screen; v) The possibility for the user to be able to use the invention, the use of the respective protocol suffices. Porto, August 27, 2013.