JP2025541589A - Application and system for dual control of a game, application and system for displaying virtual buttons, system for determining the value of at least one parameter of a user's finger, and system for determining and presenting the position of a user's finger on a display - Google Patents

Abstract

The present invention relates to an application and system for dual control of a game, an application and system for displaying virtual buttons, a system for determining the value of at least one parameter of a user's finger, and a system for determining and presenting the position of a user's finger on a display.
[Selected Figure] Figure 1

Description

The present invention relates to an application and system for dual control of a game, an application and system for displaying virtual buttons, a system for determining the value of at least one parameter of a user's finger, and a system for determining and presenting the position of the user's finger on a display. The system, including software installed on a smartphone, allows a user to hover over the screen and then accurately display the finger on the screen beneath the fingertip, allowing the user to control the game by means of the finger's specified parameters. This is achieved by providing feedback information regarding the exact position of the finger relative to the phone screen and the type of movement made by the finger.

In the prior art, there are many known wearable devices intended to monitor the position of a user's hands or fingers. Many of them are based on the recognition of gestures to control computer programs or VR environments. This allows commands to be more intuitive, without the need to learn the rules that control complex input systems.

An example of such a device is disclosed in document WO2019165139. The invention discloses a wearable device that can determine the position and gestures of a user's fingers based on the flexion of their joints using only magnetic fields. The device can display the outline of the hand on the screen, but no trace of the device's position relative to the screen is disclosed. The method for locating elements with magnetic sensors can also prove useful in tablet technology.
Another document, WO2019005586, discloses a device that can be worn on one of the fingers. It has optical, acceleration, and pressure sensors and can be wirelessly connected to other devices. The device can track the movement of the finger in a VR environment. This publication does not disclose any use of the invention in relation to a touchscreen or in relation to determining the position of the finger before touching the screen.

Another example of a device for determining the position of a user's hands is disclosed in US Patent US10642356. The invention discloses a device in the form of a flexible material containing an acceleration sensor and a gyroscope. It is intended to determine the position and movement of the hand in space in order to transfer the hand into a VR environment. A method for monitoring the user's movements involves tracking the relative movements of at least two fingers by means of this device. Tracing the movements relative to an external device such as a touchscreen is not disclosed.

Patent document US10579099B2 discloses a system including a ring device. The ring device may have a housing adapted to be worn on a user's finger. Sensors that can be used to collect input data from the user include force sensors, ultrasonic sensors, inertial measurement units, optical sensors, touch sensors, and other elements. A control circuit can wirelessly transmit information collected from the sensors and other input devices to an associated electronic device. This information can be used to control the operation of the electronic device. The housing of the ring device may have an annular body and an extendable portion connected to the body. The extendable portion may include a flap with a hinge, a rotatable housing element, a housing with an internal adjustable frame and lid, an extendable housing made of an extendable tube connected between the first and second ring devices, and other extendable structures. The presented solution does not define the type of input achieved after touching and/or pressing the screen.

The inventions disclosed in the above documents focus on applications related to VR environments, detecting gestures based on hand movements. Furthermore, users do not receive feedback information regarding the position of their fingers. In the case of devices with touchscreens, users are forced to input commands by touching the touchscreen and then modify the input position according to their intentions. The prior art does not provide a solution that enables the determination of the exact position of a user's finger on a touchscreen and displays the determined position on the screen.
Therefore, one objective of the present invention is to enable a user to stream a phone screen to an external display device and perform precise actions on the display without looking at it, but instead simply looking at the external display device. High accountability and precision of gestures input into applications displayed on the external display can then be maintained without the need to check the position of the fingers on the control device. For example, in embodiments including buttons located under the fingertips, it is even possible to transfer input gestures without the need to use a touchscreen, which allows the control device to be replaced with any flat surface, allowing the values of finger parameters to be checked, and based on these, actions in a video game corresponding to the gesture to be performed.

The object of the present invention is an application for controlling a video game, configured to receive information from a device including a touch layer and data from an additional device for inputting data, and to control the progress of the game in response to this information and data.

Preferably, the application is adapted to transfer data, in particular video data, to a display or a device having a display, where the display is in the form of a monitor, a smart TV, a projector, a VR or AR device, a smartphone screen or a tablet screen, in particular to a device for inputting data, in particular when the device for inputting data is a game controller, gamepad, joystick or device that can be worn on a user's finger, to said display and/or to an external device.

The invention also relates to a system for controlling a video game, comprising: a first control device comprising a processor, a memory and an application for controlling a video game according to the invention; a device comprising a touch layer configured to determine the position of a user's finger on said touch layer and to transmit information about this position to said first control device; an additional device for inputting said data, configured to transmit data to the first control device; and a display, wherein the first control device is configured to receive said information from the device comprising the touch layer and data from the additional device for inputting data, and is configured to control the progress of the game depending on this information and these data, and is also configured to transmit said data, in particular video data, to the display,
The display is characterized in that it is adapted to receive data, in particular video data, from the first control device and to display a video image based thereon.

Preferably, the first control device, the device with the touch layer, and the display are a single integrated device, in particular a smartphone or tablet.

Preferably, the device including the touch layer is a smartphone or tablet, and the touch layer is the screen of the smartphone or tablet.

Preferably, the first control device is a smartphone, tablet, smart TV, or video game console.

Preferably, the additional device for inputting data is a game controller, gamepad, joystick, or a device that can be worn on the user's finger.

Preferably, the device with the touch layer is configured to transmit information regarding the user's finger position and/or input and/or data and/or triggered actions input by means of the finger on the touch layer directly to the first control device.

Preferably, the device with the touch layer is configured to transmit information regarding the position of the user's finger and/or input and/or data, and/or trigger actions input by means of the finger on the touch layer, to the first control device via an additional device for inputting data.

Preferably, the additional device for inputting data is configured to transmit the data directly to the first control device.

Preferably, the additional device for inputting data is configured to transmit data to the first control device via the device having the touch layer.

Preferably, the display is configured to receive additional data, in particular video data, from additional external sources, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images based thereon.

Preferably, the display has the form of a monitor, smart TV, projector, VR or AR device, smartphone screen, or tablet screen.

Preferably, the touch layer and/or additional device for inputting data and/or the first control device and/or the device comprising the display are configured to communicate wirelessly via Bluetooth and/or Wi-Fi and/or near field communication NFC and/or Li-Fi.

The present invention also provides an application for generating virtual buttons, configured to display the virtual buttons on an application or game.

Preferably, the virtual button has the form of a context menu and is configured to receive information from the device including the touch layer and possibly data from additional devices, and to generate data in the form of a context menu, in particular video data, that depends on this information and data.

Preferably, the application is configured to allow virtual buttons to be assigned to selected actions in the game, or physical buttons on a game controller, gamepad, mouse buttons, or keyboard keys.

Preferably, the application is configured to send information about the activated button to the display or an additional device for inputting data.

It is further preferred that the application is configured to transmit data, in particular video data, to the display.

An object of the present invention is also a system for displaying virtual buttons, comprising: a second control device comprising a processor, a memory and an application for generating virtual buttons according to the invention; optionally a system for controlling a video game according to the invention; information from a device comprising a touch layer and adapted to determine the position of a user's finger on said touch layer and to transmit information about this position to the second control device, and possibly from an additional device adapted to transmit data to the second control device; and a display, wherein said second control device is adapted to receive data from the device comprising the touch layer and possibly from said additional device, to generate data, in particular video data, in the form of a context menu depending on this information and these data, and to transmit these data, in particular video data, to the display,
The display is configured to receive data, particularly video data, from the second controller and to display a video image based thereon.

Preferably, the context menu includes any number of buttons, distributed in a circle around a location corresponding to the position of the user's finger as determined by the device including the touch layer.

Preferably, the button is configured to transmit data, after activation, to an external device, in particular an additional device for inputting data, and these data are transmitted to a second control device, in particular a tablet, phone or television, as information about pressing a physical button corresponding to the selected button on a game controller or keyboard and/or mouse.

Preferably, the device including the touch layer is a smartphone or tablet, and the touch layer is the screen of the smartphone or tablet.

Preferably, the second control device is a smartphone, tablet, smart TV, or video game console.

Preferably, the additional device for inputting the data is a game controller, gamepad, joystick, or a device that can be worn on the user's finger.

Preferably, the display is configured to receive additional data, in particular video data, from an additional external source, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images based thereon.

Preferably, the display has the form of a monitor, smart TV, projector, VR or AR device, smartphone screen or tablet screen.

Preferably, the touch layer and/or additional device for inputting data and/or the second control device and/or the device with the display are configured to communicate wirelessly via Bluetooth and/or Wi-Fi and/or near field communication (NFC) and/or Li-Fi.

The invention also relates to a system for determining the value of at least one parameter of a user's finger, comprising:
a device wearable on a user's finger; at least one sensor wearable on the user's finger; a processor; a memory; a third control application; optionally a system for controlling a video game according to the present invention; and optionally a system for displaying virtual buttons according to the present invention, wherein the at least one sensor is configured to send a signal to the third control application;
further comprising a configuration for determining a value of at least one parameter of the user's finger, the configuration including at least one sensor and configured to determine a value of the at least one parameter of the user's finger and to transmit information regarding the determined value of the at least one parameter of the user's finger to the third control device;
The third control device is configured to receive information regarding the determined value of at least one parameter of the user's finger.

Preferably, the sensor is in the form of a motion sensor, a pressure sensor, an optical sensor, and/or a proximity sensor.

Preferably, the system for determining the value of at least one parameter of a user's finger further comprises a navigation surface, in particular a touchscreen of a smartphone or tablet, or a navigation surface in the form of a touchpad or trackpad.

Preferably, the navigation surface is configured to determine a value of at least one parameter of the user's finger, and the navigation surface is configured to transmit the value of the at least one parameter of the user's finger to a third control device.

Preferably, the system for determining the value of at least one parameter of the user's finger is configured to determine the value of the parameter of the user's finger (meaning the angle between the distal phalanges and the proximal phalanges).

Preferably, the system for determining the value of at least one parameter of the user's finger is configured to determine the value of the parameter of the user's finger (meaning the pressure applied to the sensor by the finger).

Preferably, the system for determining the value of at least one parameter of the user's finger is configured to determine the value of the parameter of the user's finger (meaning the pressure applied by the finger on the navigation surface).

Preferably, the system for determining the value of at least one parameter of the user's finger further comprises a display.

Preferably, the display is a projector, monitor, smart TV, VR or AR device, smartphone screen, or tablet screen.

Preferably, the display is configured to receive data from a third control device and display an image based thereon.

Preferably, the display is configured to receive additional data, in particular video data, from additional external sources, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images, including images, on the basis thereof.

Preferably, the third control device is configured to perform an action based on the value of a parameter of the user's finger.

Preferably, the third control device is a tablet or smartphone.

Preferably, the third control device is integrated with a display, and in particular the third control device is a smart TV.

Preferably, the third control device is a video game console.

Preferably, the device wearable on a user's finger has the form of a clip intended to be placed on the user's finger from above or from the side, or in the form of an open or closed ring with a sensor.

Preferably, the device wearable on the user's finger comprises a sensor adapted to measure the angle between the distal and proximal phalanges of the user's finger.

Preferably, the device wearable on the user's finger comprises a motion sensor adapted to measure the pressure applied by the finger to the navigation surface and to measure the positional and angular differences between the finger and the navigation surface.

Preferably, the device wearable on the user's finger comprises a pressure sensor and/or an optical sensor adapted to measure the pressure applied by the finger on the navigation surface.

Preferably, the device wearable on the user's finger has an additional portion extending over the distal phalanx of the finger, with a motion sensor located on the additional portion.

Preferably, the device wearable on the user's finger comprises a button under the fingertip, in particular a pressure sensor configured to generate and transmit a signal to a third control device when pressed.

Preferably, the third control device and/or the device wearable on the user's finger and/or the configuration for determining the value of at least one parameter of the user's finger and/or the navigation surface and/or the display are configured to communicate wirelessly via Bluetooth and/or Wi-Fi and/or near field communication NFC and/or Li-Fi.

Preferably, the system for determining the value of at least one parameter of the user's finger is configured so that the user defines a portion of the surface of the touch layer in the form of a touchscreen of a smartphone or tablet constituting the navigation surface.

The present invention also provides a system for determining and presenting on a display the position of a user's finger, comprising:
a fourth control device comprising a processor, a memory, a fourth control application, the fourth control device being configured to determine a position of a finger on a navigation surface, the navigation surface being configured to transmit information about the position to the fourth control device, a display, and optionally a system for controlling a video game according to the present invention, a system for displaying virtual buttons according to the present invention, and a system for determining at least one parameter of a finger according to the present invention;
further comprising an arrangement for determining a user's finger position on the navigation surface, the arrangement having at least one sensor and configured to determine the user's finger position and to transmit the determined information to the fourth control device;
the fourth control device receives the information from the arrangement for determining a user's finger position and transmits data, in particular video data, to the display, the data, in particular video data, including a finger position indicator corresponding to the user's finger position on the navigation surface;
The display is configured to receive data, in particular video data, from the fourth control device and to display a video image based thereon.

Preferably, the fourth control device is a tablet, smartphone, or video game console.

Preferably, the fourth control device is integrated with a display, and in particular the fourth control device is a smart TV.

Preferably, the display has the form of a monitor, smart TV, projector, or VR or AR device, smartphone screen, or tablet screen.

Preferably, the display is configured to receive additional data, in particular video data, from additional external sources, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and is configured to display video images including images on that basis.

Preferably, the navigation surface is a touch layer, in particular the surface of a touchscreen on a tablet or smartphone.

Preferably, the system for determining and presenting the position of a user's finger on a display is in the form of a touchscreen of a smartphone or tablet, where the user forms the navigation surface, and is configured to form the surface of the touch layer.

Preferably, the arrangement for determining the position of a user's finger on the navigation surface is adapted to be placed on at least one of the user's fingers, the arrangement comprising an accelerometer and/or a gyroscope, and configured to use data from the accelerometer and/or gyroscope to determine the position of the user's finger on the navigation surface in real time.

Preferably, the arrangement for determining the position of the user's finger on the navigation surface further comprises an optical sensor configured to assist in determining the position of the user's finger on the navigation surface by determining the angle between the distal and proximal phalanges and thereby determining the finger flexion level.

Preferably, the fourth control device and/or the navigation surface and/or the arrangement for determining the position of the user's finger on the navigation surface and/or display are configured to communicate wirelessly via Bluetooth and/or Wi-Fi and/or near field communication (NFC) and/or Li-Fi.

Preferably, the system for determining and presenting the position of the user's fingers on the display is configured to recognize gestures performed by the user's fingers through and/or on the navigation surface and/or in the form of a particular configuration of one or more of the user's fingers.

Preferably, the first control device and/or the second control device and/or the third control device and/or the fourth control device are the same device.

The system according to the present invention combines touch navigation, known from mobile devices such as phones or tablets, with additional external input and feedback originating from one or more external devices worn on the fingers to form a controller for games, in particular, but not exclusively. That is, it combines the positive properties of touch navigation known from mobile games, such as the precision of the touch interface and control, with an additional input that allows the execution of multiple actions simultaneously, thanks to feedback functionality, without the need to use several fingers, making it possible to control the system, application, and game without looking at the fingers. Combined with virtual buttons displayed on a transparent layer above the game, which, after activation, are configured to transmit information corresponding to pressed physical buttons on the game controller or keyboard and/or mouse, allowing players to use touch navigation to play any game streamed from the cloud or otherwise on any device, even if these games do not officially support touch navigation.

The present invention will now be presented in more detail in preferred embodiments with reference to the accompanying drawings, in which:
[Figures 1 to 3] User's finger parameter values (meaning the angle between the distal and proximal phalanges)
1 shows a system for determining
4-6 show a system for determining the value of a parameter of a user's finger (pressure applied by the finger to a sensor).
7-11 show a system for determining and presenting the position of a user's finger on a display, where the user is configured to define a portion of the surface of a touch layer.
[Figures 12-13] Show a device that can be worn on a user's finger in the form of a clip, which means that it can be placed on the user's finger from above the ring, or from the side, or from an open or closed ring, with a sensor.
14-15 show a device wearable on a user's finger that includes a sensor adapted to measure the angle between the distal and proximal phalanges of the user's finger.
17-19 show a device wearable on a user's finger having an additional portion that extends over the distal phalanges of the finger, with a motion sensor located on the additional portion.

In one preferred embodiment, the present invention takes the form of a system for controlling a video game. The system includes a first control device, the first control device including a processor, a memory, and an application for controlling the video game. The application for controlling the video game is configured to receive information from a device with a touch layer and data from an additional device for inputting data, and to control the progress of the video game in response to this information and data. Preferably, the application for controlling the video game is configured to transmit data, in particular video data, to a display or device with a display and/or to an external device, where the display is in the form of a monitor, a smart TV, a projector, a VR or AR device, a smartphone screen, or a tablet screen. In particular, the device for inputting data, in particular when the device for inputting data is a game controller, gamepad, joystick, or device wearable on a user's finger. The system includes a device with a touch layer configured to determine the position of a user's finger thereon and to transmit information about this position to a first control device. The system also includes an additional device for inputting data and configured to transmit data to the first control device, and a display. The first control device is configured to receive the information from the device with the touch layer and data from the additional device for inputting data, and is configured to control the progress of the game in response to this information and data, and to transmit data, particularly video data, to the display. The display is then configured to receive the data, particularly video data, from the first control device and display a video image based thereon.

Preferably, the device including the touch layer is a smartphone or tablet, and the touch layer is the screen of the smartphone or tablet.

The first control device may be a smartphone, tablet, smart TV, or video game console. The additional device for inputting data may be a game controller, gamepad, joystick, or a device wearable on the user's finger.

The device including the touch layer is configured to transmit information about the position of the user's finger and/or input and/or data, and/or triggered actions, that are input directly to the first control device by means of a finger on the touch layer.

Preferably, the device including the touch layer is configured to transmit information about the position of the user's finger on the touch layer to the first control device via an additional device for inputting data. The additional device for inputting data is configured to transmit data directly to the first control device.

According to another preferred embodiment, the additional device for inputting data is configured to transmit data to the first control device via the device with the touch layer.

Preferably, the display is configured to receive additional data, in particular video data, from additional external sources, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images based thereon.

Preferably, the display may have the form of a monitor, smart TV, projector, VR or AR device, smartphone screen, or tablet screen.

The device with the touch layer and/or additional devices for inputting data and/or the first control device and/or display is preferably configured to communicate wirelessly via Bluetooth and/or Wi-Fi and/or near field communication (NFC) and/or Li-Fi.

According to a further embodiment, the invention comprises a second control device in the form of a system for displaying virtual buttons in the form of a context menu, the second control device comprising a processor, a memory and an application for generating the virtual buttons, and the application for generating data, in particular video data, in the form of a context menu, is configured to display them on an application or game. The virtual buttons are in the form of a context menu and are configured to receive information from a device comprising a touch layer and possibly data from an additional device, and to generate data, in particular video data, in the form of a context menu that depends on this information and on these data.
The application is additionally configured so that virtual buttons can be assigned to selected actions in the game or physical buttons on a game controller, gamepad, mouse button, or keyboard key. The application can also be configured to send information about the activated button to a display or an additional device for inputting data and to send data, particularly video data, to the display. The system further includes a device with a touch layer configured to determine the position of a user's finger on the touch layer and send information about this position to a second control device. The system can also include an additional device for inputting data configured to send data to the second control device. The system also includes a display on which virtual buttons are displayed, for example in the form of a context menu. The second control device is configured to receive the information from the device with the touch layer and possibly data from the additional device, generate data, particularly video data, in the form of a context menu depending on this information and data, and send this data, particularly video data, to the display. The display is configured to receive data, particularly video data, from the second control device and display a video image based on it.

Furthermore, the system may be a system including a first control device for controlling a video game, the system including a processor, a memory, and an application for controlling the video game. The application for controlling the video game is configured to receive information from the device with a touch layer and data from an additional device for inputting data, and to control the progress of the game in accordance with this information and data. Preferably, the application for controlling the video game is further configured to transmit data, in particular video data, to a display. Thus, the system further includes a device with a touch layer configured to determine the position of a user's finger on the touch layer and transmit information about this position to the first control device. The system also includes an additional device for inputting data, configured to transmit data to the first control device, and a display. The first control device is configured to receive the information from the device with a touch layer and data from the additional device for inputting data, control the progress of the game in accordance with this information and data, and transmit data, in particular video data, to the display. The display is then configured to receive data, in particular video data, from the first control device and display a video image based thereon.

Preferably, the context menu includes any number of buttons, preferably four buttons, distributed in a circle around a location corresponding to the user's finger position as determined by the device including the touch layer. This allows the user to select an appropriate command from the context menu depending on their needs or the game situation. Preferably, the buttons correspond in appearance and function to physical buttons on a standard game controller, which is usually a pad or joystick. After activation, the buttons are configured to transmit data to an external device, in particular an additional device for inputting data, and these data are transmitted to a second control device, in particular a tablet, phone or television, as information regarding the pressing of a physical button corresponding to a selected button on the game controller or keyboard and/or mouse.

The device including the touch layer is a smartphone or tablet, and the touch layer is the screen of the smartphone or tablet. The second control device may be a smartphone, tablet, smart TV, or video game console.

Preferably, the additional device for inputting data is a game controller, gamepad, joystick, or a device that can be worn on the user's finger.

Preferably, the display is configured to receive additional data, in particular video data, from additional external sources, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images based thereon.

Preferably, the display has the form of a monitor, smart TV, projector, VR or AR device, smartphone screen, or tablet screen.

Preferably, the device comprising the touch layer and/or additional device for inputting data and/or second control device and/or display is configured for wireless communication, preferably via Bluetooth and/or Wi-Fi and/or near field communication NFC and/or Li-Fi.

By using a system for displaying virtual buttons, the virtual buttons can appear anywhere in the interface, or in the form of a menu that appears within the touch range of one of the user's fingers on the screen, or on a transparent layer above the game or application, particularly a game streaming application. At launch, the virtual buttons are not in the form of performing an action; rather, the game or application performs an action assigned to a physical button on the game controller, but rather the game or application performs an action that corresponds to any button on a conventional controller, or to conventional navigation, e.g., a keyboard.

The purpose of a context menu is to add more additional inputs to perform side actions. This context menu can take the form of a menu that appears on the interface at the location of the finger, whether it is positioned where the finger contacts the navigation surface or by a feedback function.

Launching the menu:
The menu may be activated using one of the buttons on a device wearable on the user's finger b. The menu may also be activated by pressing a button on the game or application interface.
b. The menu may be activated by pressing a button on the game or application interface.
c. The menu may be activated by an external mechanical button that depresses the game or application interface. Preferably, these may be triggers that, when pressed by a finger after a call is made, mechanically contact the screen where the button is on the interface.
d. The menu may be activated using a gesture.

The context menu may be displayed only on the screen of the navigation space and/or on an external screen. The menu may, for example, be displayed within one of the fingers surrounding a given finger (where it touches the navigation space or where in the space the finger is located). This may be the same finger that activates the menu, or a different finger. For example, one of the buttons on the tip of the left thumb activates the menu around the right finger.

The context menu may have one or more buttons that correspond to actions assigned to buttons on a conventional controller. In this case, a button on the conventional controller is assigned to the menu, and the game/application performs the action previously assigned in the conventional controller configuration. The context menu may also have one or more buttons that correspond to actions assigned in the game/application.

The application displaying the virtual buttons in the form of the context menu communicates back with the controller, which, after receiving information about the selected position from the context menu, sends information about the input, i.e., the pressed button, to the device. For example, main buttons and side actions from the context menu may be assigned to specific buttons on a conventional controller supported by the game or application. In such a case, the controller, upon receiving information about the selection of an action in the input menu, returns information about the input corresponding to, for example, pressing button A on the controller. An alternative solution envisions using a motion sensor that recognizes the direction of movement toward the button/action displayed in the menu being approached by the finger. In such a situation, data about the type of input sent, e.g., button A or B, is sent directly by the controller without having to receive data from the application beforehand. A context menu displayed in an application or on an external screen is merely one form of visual feedback information when making a selection.

Activation of the context menu is accomplished by touching/pressing the selected button, or by swiping your finger and releasing it over the button.

The context menu can take any form that places action icons within a short distance of the user's finger from which they can select an action. The simplest example is in the form of a circle divided into any number of sections. Where context menu forms with multiple levels are possible, selecting one of the buttons in the context menu displays the next level of the context menu with additional actions.

An interface with virtual buttons on a transparent layer over a game or application, where the virtual buttons are configured to correspond to actions in the game or to trigger inputs corresponding to any button on a popular game controller, gamepad, mouse button, or keyboard key.

This solution is specifically intended to use game-supported navigation using a game controller, keyboard, and mouse, and aims to generate virtual navigation based on them, providing the ability to play games streamed from the cloud that do not have native touch navigation on devices with touch screens.

The use of the context menu system makes it possible to play games streamed from the cloud on devices with a touch layer that do not have native touch navigation, but only traditional navigation that supports game controllers, keyboards, and mice.

The system for displaying data in the form of said context menu, in particular video data, is within the range of one of the user's fingers touching the touch layer, making it possible to select any/each action in this menu simply by moving the finger, without any significant hand movement.

Preferably, a system for displaying data in the form of a context menu, in particular video data, is arranged around one of the fingers that contacts the touch layer.

According to another preferred embodiment of the present invention, selection of an action from a context menu occurs by swiping and releasing a finger on the selected button, without withdrawing the finger from the moment the menu appears on the touch layer.

The context menu may be launched in another application, in particular an external application launched via a game or game streaming application. The context menu may also be launched in a game streaming application on a layer with streamed images of the game. It may also be displayed on the graphics layer of the game or application/game or application.

Preferably, activation of the context menu virtual button is assigned in the application code to indirectly trigger an action via a game controller or keyboard and/or mouse button assigned thereto.

Preferably, information about input from the context menu received by the external device is further transmitted to another device, e.g., a smart TV, or returned to the device from which it was received, e.g., a phone.

Preferably, information about the button selected from the context menu is sent to an external device, in particular an additional device for inputting data, and returned to the control device, in particular a tablet, phone or television, as information about pressing a physical button/physical input corresponding to a button on a game controller or keyboard and/or mouse.

The purpose of adding external inputs is to allow players to perform multiple actions simultaneously while gaming without having to use additional fingers. This solution not only improves the ergonomics, efficiency, comfort, and experience of playing more complex mobile titles, but also enables the playing of console and PC titles using touch navigation via cloud gaming, including but not limited to:

Yet another aspect of the present invention provides a system for determining a value of at least one parameter of a user's finger, comprising: a device wearable on the user's finger; at least one sensor disposed within or on the device wearable on the user's finger; and a third control device comprising a processor, memory, and a third control application. The at least one sensor is adapted to transmit a signal to the third control device. The system further comprises a configuration for determining a value of at least one parameter of the user's finger, the configuration comprising the at least one sensor and configured to determine the value of the at least one parameter of the user's finger and transmit information regarding the determined value of the at least one parameter of the user's finger to the third control device. The third control device is configured to receive information regarding the determined value of the at least one parameter of the user's finger.

Additionally, the system for determining the value of at least one parameter of a user's finger includes a first control device for controlling a video game, the device including a processor, a memory, and an application for controlling the video game. The application for controlling the video game is configured to receive information from a device with a touch layer and data from a device for inputting data, and to control the progress of the game in accordance with the information and data. Preferably, the application for controlling the video game is further configured to transmit data, in particular video data, to a display. Accordingly, the system further includes a device with a touch layer, configured to determine the position of the user's finger on the touch layer and transmit information about this position to the first control device. The system also includes an additional device for inputting data, configured to transmit data to the first control device, and a display. The first control device is configured to receive the information from the device with a touch layer and data from the additional device for inputting data, control the progress of the game in accordance with the information and data, and transmit data, in particular video data, to the display. The display is then configured to receive data, in particular video data, from the first control device and display a video image based thereon.

Additionally, a system for determining the value of at least one parameter of a user's finger may include a second control device consisting of a processor, memory, and an application for a context menu, and an application for generating virtual buttons, the second control device including a processor, memory, and an application for the virtual buttons. The application for generating data, particularly video data, in the form of a context menu is configured to display them in an application or game. The virtual buttons have the form of a context menu and are configured to receive information from a device including a touch layer and possibly data from an additional device, and to generate data, particularly video data, in the form of a context menu depending on this information and data. The application is additionally configured so that the virtual buttons can be assigned selected actions of physical buttons in a game or on a game controller, gamepad, mouse button, or keyboard key. The application may also be configured to send information about the activated buttons to a display or additional device for inputting data and to send data, particularly video data, to the display. The system further includes a device including a touch layer configured to determine the position of the user's finger on the touch layer and send information about this position to the second control device. The system may include an additional device for inputting data, configured to transmit the data to a second control device. The system also includes a display for displaying, for example, a context menu. The second control device is configured to receive the information from the device with the touch layer and possibly data from the additional device, generate data in the form of a context menu, in particular video data, in response to the information and data, and transmit the data, in particular video data, to a display. The display is configured to receive the data, in particular video data, from the second control device and display a video image based thereon.

At least one sensor may be in contact with the surface of the finger.

Preferably, the sensor is in the form of a camera and/or a proximity sensor and/or a motion sensor and/or a pressure sensor and/or an optical sensor.

The system for determining the value of at least one parameter of a user's finger may further comprise a navigation surface, in particular in the form of a touch layer, in particular in the form of a touchscreen or touchpad or trackpad of a smartphone or tablet, configured to determine the value of at least one parameter of the user's finger, the navigation surface being configured to transmit the value of the at least one parameter of the user's finger to a third control device.

Preferably, the system for determining the value of at least one parameter of the user's finger is configured to determine the value of the parameter of the user's finger, meaning the angle between the distal phalanges and the proximal phalanges, which is shown in Figures 1-3.

Preferably, the system for determining the value of at least one parameter of the user's finger is configured to determine the value of the parameter of the user's finger, meaning the pressure applied by the finger to the sensor, as presented in Figures 4-6.

Preferably, the system for determining the value of at least one parameter of a user's finger is configured to determine the value of a parameter of the user's finger, meaning the pressure applied by the finger on the navigation surface and/or the angle between the distal and proximal phalanges. In this case, information about the parameter is collected when the finger touches the navigation surface, preferably through the value of the pressure applied by the finger to the navigation surface.

The system for determining the value of at least one parameter of a user's finger also includes a display. The display may be a projector, a monitor, a smart TV, a VR or AR device, a smartphone screen, or a tablet screen. The display is configured to receive data from the third control device and display an image based thereon.

Preferably, the display is configured to receive additional data, in particular video data, from an additional external source, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images based thereon from a third control device, including images obtained based on the received data.

Preferably, the third control device is configured to perform an action or operation based on the value of the parameter of the user's finger.

The third control device may be a tablet or a smartphone. Furthermore, the third control device may be integrated with a display, and in particular, the third control device is a smart TV. Preferably, the third control device is a video game console.

Preferably, the device wearable on a user's finger has the form of a clip, and is intended to be placed on top of or on the side of the user's finger, or on an open or closed ring with sensors as shown in Figures 12 and 13. The ring only has a sensor (e.g., but not limited to, EMG) and detects changes in tension in the muscles or tissues of the ring finger. It can detect finger pressure, and information about the tension change can be used to read the input, and information about the finger tension can be used to identify the finger flexion angle, which signifies the type of input. The ring can be supplemented with additional sensors, for example, a camera/proximity sensor/infrared sensor, which help evaluate finger flexion.

Preferably, the device wearable on the user's finger comprises a sensor adapted to measure the angle between the distal and proximal phalanges of the user's finger, as presented in Figures 14 and 15. Preferably, an additional element extends from the top and is made of a flexible material, for example rubber, so as to always press against the top of the distal phalange of the finger. There are many more fastening methods possible; it could also extend from the side or bottom. The important feature is that the motion sensor should abut against the distal phalange and move with it during flexion.

Preferably, the device wearable on a user's finger has an additional part extending over the distal phalanges of the finger, with the motion sensor located in the additional part shown in Figures 16-19. Although the structure has a completely different visual appearance, this solution can be derived from the solutions shown in Figures 14 and 15. This solution has the form of a clip or open ring, which in this case is placed from the side. The motion sensor is still located on the distal phalanges of the finger and performs the same function. A "fingertip button" element below the fingertip is an additional part that generates an input when pressed.

Preferably, the device wearable on the user's finger may comprise a button under the fingertip, in particular a pressure sensor configured to generate and transmit a signal to a third control device when pressed.

In another embodiment, the device wearable on a user's finger comprises a pressure sensor and/or an optical sensor adapted to measure the pressure applied by the finger on the navigation surface.

Preferably, the device wearable on a user's finger may have an additional portion extending beyond the distal phalanges of the finger, with the motion sensor located on the additional portion.

Preferably, the device wearable on the user's finger comprises a button under the fingertip, in particular a pressure sensor configured to generate and transmit a signal to a third control device when pressed.

Preferably, the third control device and/or the device wearable on the user's finger and/or the configuration for determining the value of at least one parameter of the user's finger and/or the navigation surface and/or the display are configured to communicate via wireless communication, preferably via Bluetooth and/or Wi-Fi and/or near field communication NFC and/or Li-Fi.

Preferably, the system for determining the value of at least one parameter of a user's finger is configured so that the user defines a portion of the surface of the touch layer in the form of a touchscreen of a smartphone or tablet, which in particular constitutes a navigation surface for the smartphone or tablet, the touchscreen of the smartphone or tablet being configured to determine the value of at least one parameter of the user's finger, and the navigation surface being configured to transmit the value of the at least one parameter of the user's finger to a third control device.

Another aspect of the present invention is a system for determining and presenting a user's finger position on a display, the system comprising: a fourth control device configured to determine the position of the finger across the navigation surface; a navigation surface comprising a processor, memory, and a fourth control application configured to then transmit information regarding the position to the fourth control device; and a display. The fourth control device configured to determine the finger position on the navigation surface also comprises an arrangement for determining the position of the user's finger on the navigation surface, the arrangement comprising at least one sensor, configured to determine the position of the user's finger, and to transmit information regarding the determined position to the fourth control device. The fourth control device is configured to receive the information from the arrangement for determining the user's finger position and transmit data, particularly video data, to a display, the video data comprising a finger position indicator corresponding to the position of the user's finger across the navigation surface. The display is configured to receive the data, particularly video data, from the fourth control device and display a video image based thereon, including the finger position indicator.

The system for determining and presenting the position of a user's finger on a display may also comprise a system for controlling a video game, the system comprising a first control device, a processor, a memory, and an application for controlling the video game. The application for controlling the video game is configured to receive information from the device with a touch layer and data from an additional device for inputting data, and to control the progress of the game in accordance with the information and data. Preferably, the application for controlling the video game is further configured to transmit data, in particular video data, to a display. Thus, the system further comprises a device with a touch layer configured to determine the position of the user's finger on the touch layer and transmit information about the position to the first control device. The system also comprises the additional device for inputting data, configured to transmit data to the first control device, and a display. The first control device is configured to receive the information from the device with a touch layer and data from the additional device for inputting data, control the progress of the game in accordance with the information and data, and transmit data, in particular video data, to a display. The display is then configured to receive data, in particular video data, from the first control device and display a video image based thereon.

Additionally, the system for determining and presenting the position of a user's finger on a display may include a system for displaying virtual buttons, the system including a second control device having a processor, a memory, and an application for generating the virtual buttons. The application for generating data, particularly video data, in the form of a context menu is configured to display them across applications or games. The virtual buttons have the form of a context menu and are configured to receive information from a device including a touch layer and possibly data from an additional device, and to generate data, particularly video data, in the form of a context menu depending on the information and data. The application is additionally configured so that the virtual buttons can be assigned selected actions of physical buttons of a game or of a game controller, gamepad, mouse button, or keyboard key. The application may also be configured to send information about activated buttons to a display or additional device for inputting data and to transmit data, particularly video data, to the display. The system may further include a device including a touch layer configured to determine the position of a user's finger on the touch layer and transmit information about the position to a second control device. The system may include an additional device for inputting data configured to transmit data to the second control device. The system also includes a display for displaying, for example, a context menu. The second control device is configured to receive the information from the device with the touch layer and, optionally, data from additional devices, generate data in the form of a context menu, in particular video data, in response to the information and data, and transmit the data, in particular video data, to the display. The display is configured to receive data, in particular video data, from the second control device and display a video image based thereon.

Additionally, the system for determining and presenting the position of a user's finger on a display may also include a system for determining a value of at least one parameter of a user's finger, comprising: a device wearable on the user's finger; at least one sensor disposed in or on the device wearable on the user's finger; and a third control device comprising a processor, a memory, and a third control application. The at least one sensor is adapted to transmit a signal to the third control device. The system further comprises a configuration for determining a value of at least one parameter of the user's finger, the configuration comprising the at least one sensor and configured to determine the value of the at least one parameter of the user's finger and transmit information regarding the determined value of the at least one parameter of the user's finger to the third control device. The third control device is configured to receive information regarding the determined value of the at least one parameter of the user's finger.

The fourth control device may be a tablet, a smartphone, or a video game console. The fourth control device may be integrated with a display, and in particular the fourth control device may be a smart TV.

The display may have the form of a monitor, a smart TV, a projector, a VR or AR device, a smartphone screen, or a tablet screen. The display is configured to receive additional data, in particular video data, from additional external sources, for example a video game console, a smartphone, a tablet, or to receive video streaming from the Internet, and to display video images, including images, on those bases.

Preferably, the navigation surface of the system for determining and presenting the position of the user's finger on the display is the surface of a touch layer, in particular the touchscreen of a tablet or smartphone.

Preferably, the system for determining and presenting the position of a user's finger on the display is configured so that the user defines a portion of the surface of the touch layer in the form of a touchscreen on a smartphone or tablet, which constitutes the navigation surface presented in Figures 7-11.

Preferably, the arrangement for determining the position of a user's finger across a navigation surface is adapted to be placed on at least one of the user's fingers, the arrangement comprising an accelerometer and/or a gyroscope and configured to use data from the accelerometer and/or gyroscope to determine the position of the user's finger on the navigation surface in real time.

In yet another embodiment of the present invention, an arrangement for determining the position of a user's finger on a navigation surface is adapted to be placed on at least one of the user's fingers, the arrangement comprising a magnetometer and configured to use data from the magnetometer to determine the position of the user's finger on the navigation surface in real time.

In another embodiment of the present invention, the arrangement for determining the position of a user's fingers on a navigation surface comprises at least one camera and is configured to use data from the at least one camera to determine the position of the user's fingers on the navigation surface in real time, preferably the arrangement being a device wearable on at least one of the user's fingers and having an indicator.

In accordance with yet another aspect of the present invention, an arrangement for determining the position of a user's finger on a navigation surface comprises a device configured to detect light intensity and configured to use data from said device to determine in real time the position of the user's finger across the navigation surface, said arrangement forming a light barrier or reflector of incident light by said navigation.

Preferably, the arrangement for determining the position of the user's finger across the navigation surface further comprises an optical sensor configured to assist in determining the position of the user's finger on the navigation surface via determining the angle between the distal and proximal phalanges to determine the finger flexion level.

Preferably, the fourth control device and/or the arrangement for determining the position of the user's finger across the navigation surface, and/or the navigation surface and/or the display are configured to communicate wirelessly, preferably via Bluetooth and/or Wi-Fi and/or near field communication (NFC) and/or Li-Fi.

Preferably, the system for determining and presenting the position of the user's fingers on the display is configured to recognize gestures performed by the user's fingers above and/or on the navigation surface and/or in the form of a particular configuration of one or more of the user's fingers.

According to another embodiment of the present invention, the first control device and/or the second control device and/or the third control device and/or the fourth control device are the same device.

Each of the systems according to the present invention, i.e., the system for dual control of a game, the system for displaying virtual buttons, the system for determining the value of at least one parameter of a user's finger, and the system for determining and presenting the position of a user's finger on a display, may be implemented individually as a sole system without the implementation of the remaining systems, or any two of them may be implemented, i.e., 1) a combination of a system for dual control of a game and a system for displaying data, particularly video data, in the form of a context menu; or 2) a combination of a system for dual control of a game and a system for determining the value of at least one parameter of a user's finger; or 3) a combination of a system for dual control of a game and a system for determining and presenting the position of a user's finger on a display; 4) a combination of a system for displaying data, particularly video data, in the form of a context menu and a system for determining the value of at least one parameter of a user's finger; or 5) a combination of a system for displaying data, particularly video data, in the form of a context menu and a system for determining the value of at least one parameter of a user's finger and a system for determining and presenting the position of a user's finger on a display; or 6) a combination of a system for determining the value of at least one parameter of a user's finger and a system for determining and presenting the position of a user's finger on a display.

Three combinations of the above-mentioned systems are possible: 1) a combination of a system for dual control of a game, a system for displaying data, particularly video data, in the form of a context menu, and a system for determining the value of at least one parameter of a user's finger; 2) a combination of a system for displaying data, particularly video data, in the form of a context menu, a system for determining the value of at least one parameter of a user's finger, and a system for determining the value of at least one parameter of a user's finger and presenting the position of the user's finger; 3) a combination of a system for dual control of a game, a system for displaying data, particularly video data, in the form of a context menu, a system for determining and presenting the position of a user's finger on a display, and a system for dual control of a game; and 4) a combination of a system for dual control of a game, a system for determining the value of at least one parameter of a user's finger, and a system for determining the value of at least one parameter of a user's finger and presenting the position of the user's finger.

It is also possible to implement all four of the above-mentioned systems simultaneously, i.e. a system for dual control of a game, a system for displaying data in the form of a context menu, in particular video data, a system for determining the value of at least one parameter of a user's finger, and a system for determining and presenting the position of the user's finger on the display.

As used herein, an application is a computer program that, when launched on appropriate hardware, performs the actions and operations referred to herein. They refer to computer programs that contain instructions that, when executed by a computer, cause the computer to perform the appropriate operations. In the context of this invention, the above description of what an application is includes all applications referred to herein, i.e., those relating to dual control of a game, display of a context menu, and display of third and fourth applications.

In a sample use of the system according to the invention, touch navigation is performed by contact with a navigation surface, particularly a touch-sensitive one, in particular a phone or tablet, but this could also be a different surface covered with any touch sensor, or a surface that is not touch-sensitive. The purpose of the navigation surface is to return information about the user's finger position and/or input and/or data and/or trigger action by finger means. When the surface is touch-sensitive, this is done similarly to the operation of a touchscreen on a mobile device or trackpad. For surfaces that are not touch-sensitive, this is done based on measurements from system sensors. The purpose of the navigation surface is to perform a function similar to touch control in mobile games, or a mouse on a computer, or an analog knob on a traditional pad. The player does not need to retract their thumbs and touch virtual buttons to trigger targeting or shooting actions. They can also use their left thumb to move their character and their right thumb to rotate the camera. At the same time, a slight pressure on the left thumb triggers a target action, and a slight pressure on the right thumb triggers a shooting action.

An example of usage could be a user wearing two devices wearable on the user's fingers, one on each thumb, and playing a "first-person shooter" type mobile game on the phone. Basic control is provided by the phone's inherent capabilities and the game's functions, with the left thumb being used to move the character by means of a virtual knob and the right thumb being used to rotate the camera by swiping the finger against the right part of the screen. They are also free to use actions assigned to virtual buttons placed on the interface in the form of context menus. At the same time, the user can use additional inputs allowed by the devices wearable on the user's fingers.

Another use case might be a user playing a "first-person shooter" type game that is streamed from the cloud to a television set. They use a phone and two devices that can be worn on the user's fingers, one on the thumb of each hand. These three devices, along with an application installed on the phone, make up the entire controller. The player sends information about the tilt of the virtual knob and the movement of the fingers relative to the screen directly to the television set or to the device worn on the user's fingers. These control the character's movement and camera rotation in the application on the phone, which then sends it to the television set via the device worn on the user's fingers, along with additional input provided by the player. Controlling a game streamed to a television set requires more actions than simply moving the character and rotating the camera. Due to the external main action, the player can also perform other necessary actions, such as targeting and shooting.

The primary external input involves pressing a finger against the navigation surface. Several distinct inputs may, but need not, be distinguished depending on the angle at which the finger, particularly the distal phalanges, are pressed against the navigation surface, or depending on the specific part of the fingertip that is pressed against the navigation surface. Additional external inputs include selecting an action from a context menu. The menu is displayed on the navigation surface and/or on the external screen, preferably in a location corresponding to the position of one of the fingers. Selecting a location from a context menu involves swiping the finger or fingers displayed on the external screen toward the selected element by means of the finger and releasing it when the element is selected, or typically involves pressing the selected element. External spatial input involves performing a specific gesture in space with one or more fingers wearing a device wearable on the user's finger. This gesture may be performed without reference to another object in space and/or may be performed with respect to the navigation surface and/or a second or multiple devices wearable on the user's finger. The purpose of external input is to allow a single finger to perform more actions simultaneously without interrupting other actions, for example, to use screen touch or a control plane to control a pointer similar to a mouse pointer. Touching the screen or control plane with a finger and swiping across it controls the pointer displayed on the external screen. At the same time, the finger can also be pressed down firmly to generate a click.

The purpose of adding external inputs is to perform multiple actions simultaneously during a game without having to use additional fingers. This solution also aims to improve ergonomics, efficiency, comfort, and the playing experience, allowing, for example, a player to touch the screen with their right thumb to rotate the camera and start shooting without stopping the camera rotation even for a split second.

Feedback includes feedback of information about the position of the fingers relative to the navigation surface, particularly via a phone or tablet, and also via another external navigation surface, such as a trackpad, or another navigation surface, particularly but not necessarily having a touch layer, including gathering data from motion sensors. The objective is to be able to precisely touch any point on the navigation surface without looking at it and/or one's hand. Instead, the only information used includes the position of the fingers, as presented by a visible pointer on an external screen, including being able to display additional information about the game, application, or system (e.g., application icon) over which one or more fingers are positioned, similar to hovering a particular interface element with a mouse pointer, even when the pointer is not visible.

The primary purpose of the feedback is to display a pointer that indicates the position of a finger relative to a navigation surface, particularly a phone or tablet, allowing a player to easily tap game or application interface elements with their finger on the navigation surface, particularly a phone or tablet, without having to look at the navigation surface, particularly a phone or tablet. The pointer and interface elements may be displayed directly on the navigation surface, particularly a mobile device, or may be displayed only on an external screen. Hovering over an interface element with a pointer (visible or invisible) can cause additional information to be displayed.

In another use of the present invention, a user wears two devices wearable on the user's fingers, one for each thumb on each hand, and is playing a "first-person shooter" type mobile game on the phone, transmitting the images displayed on it to a television set. A pointer displayed on the phone screen, or directly on the television set screen, indicates the position of the thumb on the phone screen. This allows the player to look only at the television set and still use their thumb on the phone to hit virtual buttons displayed on the interface without having to look at the phone or their hands.

Another usage example is where the user plays a "first-person shooter" type game streamed from the cloud to a television set. They use a phone and two devices that can be worn on the user's fingers, one on each thumb. The set of these three devices and an application installed on the phone constitutes the entire controller. Apart from transmitting information about the tilt of a knob or the rotation of a camera, the phone also serves as a reference point for displaying a pointer on an external navigation surface and on the television set. The application transforms the entire phone screen, or part of it, into a space that corresponds to the ratio of the television set's screen.

One additional use of the system for determining and presenting the position of the user's finger on the display is in the form of so-called feedback, i.e. in the form of a pointer that is visible on the external screen. The position of the pointer corresponds to the position of the thumb on the phone screen. This allows the player to touch selected buttons on the interface without looking at his or her hands, focusing only on the external screen. Another application is the use of additional input in the form of one or more buttons placed on the fingertips, allowing the player to perform several main actions simultaneously by using only the thumb. The input is further supported by context menus, which make it possible to perform even more side actions.

Ways to achieve additional input:
a) Mechanical Buttons - Place one or more mechanical buttons of any kind on the fingertip. The controller element that abuts the fingertip may be flexible and/or replaceable to allow for proper sizing to fit a person's finger or to replace it when worn, or it may be fixedly attached to the rest of the device.

b) Pressure Sensors: One or more pressure sensors of any type are attached to the fingertip. Each pressure sensor can send one or more inputs depending on the pressure applied. The controller element that abuts the fingertip may be flexible and/or replaceable to allow for proper sizing to fit a person's finger or to allow replacement when worn. It may also be fixedly attached to the rest of the device.

c) Touch Sensors One or more touch sensors of any type are attached to the fingertip. The controller element that abuts the fingertip may be flexible and/or replaceable to allow for proper sizing to fit a person's finger or to replace it when worn, or it may be fixedly attached to the rest of the device.

d) Motion Sensor: This may be a simple gyroscope, a gyroscope with an accelerometer, or a whole set of sensors located anywhere on the finger that constitute a motion sensor. When performing a tap/click, the finger performs a specific micro-movement. Collecting data from the motion sensor allows this movement/gesture to be distinguished from a regular touch and swipe across the screen. The tilt angle of the distal phalanges relative to the navigation surface, particularly but not exclusively, allows the motion sensor, i.e., the movement sensor located on the distal phalanges, to distinguish different types of inputs. For example, Input A is a flat finger placed parallel to the navigation surface and pressing against it, with the distal phalanges straightened. Input B is a finger bent at a 45-degree angle. Input C is a finger bent at a 90-degree angle relative to the navigation surface and pressing with the tip of the finger (the part of the fingertip just below the nail). Preferably, the motion sensor is further supported by machine learning to achieve better accuracy.

e) Motion sensor + heart rate monitor/EMG
A heart rate monitor is added to the solution from point d to receive additional data.

f) Optical Sensor This refers to an optical sensor that measures finger tension via infrared light to obtain information about pressure.

Claims (64)

An application for controlling a video game, configured to receive information from a device including a touch layer and data from an additional device for inputting data, and to control the progress of the game in accordance with this information and data. The application according to claim 1, further characterized in that it is configured to transmit data, in particular video data, to a display or a device comprising a display, wherein the display is in the form of a monitor, a smart TV, a projector, a VR or AR device, a smartphone display or a tablet display, in particular when the data input device is a game controller, a gamepad, a joystick or a device wearable on the user's finger, and/or to an external device. 1. A system for controlling a video game, comprising:
a first control device for determining the position of a user's finger on the touch layer, the first control device comprising a processor, a memory, and an application according to claim 1 or 2 for controlling the video game;
a device having a touch layer, the device determining the position of a user's finger on the touch layer and transmitting this information to the first controller;
a display;
said first control device is configured to receive said information from said device with the touch layer and to input data from said further device, to control the progress of the game depending on said information and these data, and to transmit said data, in particular video data, to said display;
The display device is configured to receive data, particularly video data, from the first control device and to display a video image based thereon. The system described in claim 3, characterized in that the first control device, the device with the touch layer, and the display are a single integrated device, in particular a smartphone or tablet. The system described in claim 3 or 4, characterized in that the device including the touch layer is a smartphone or a tablet, and the touch layer is the screen of the smartphone or the tablet. The system described in claim 3, 4, or 5, wherein the first control device is a smartphone, tablet, smart TV, or video game console. The system described in any one of claims 3 to 6, wherein the additional device for inputting data is a game controller, a gamepad, a joystick, or a device that can be worn on a user's finger. The system described in any one of claims 3 to 7, characterized in that the device equipped with a touch layer is configured to directly transmit information regarding the position of the user's finger and/or the input and/or the data and/or the trigger action input by means of the finger on the touch layer to the first control device. The system of any one of claims 3 to 7, characterized in that the device with a touch layer is configured to transmit information regarding the position of the user's finger and/or the input and/or the data, and/or the triggered action input by means of the finger on the touch layer, to the first control device via the additional device for inputting data. The system described in any one of claims 3 to 9, characterized in that the additional device for inputting data is configured to transmit the data directly to the first control device. The system described in any one of claims 3 to 9, wherein the additional device for inputting data is configured to transmit the data to the first control device via the device having a touch layer. The system described in claims 3 to 11, characterized in that the display is configured to receive additional data, in particular video data, from an additional external source, such as a video game console, smartphone, tablet, or to receive video streaming from the Internet, and to display video images based thereon. The system described in any one of claims 3 to 12, characterized in that the display has the form of a monitor, smart TV, projector, VR or AR device, smartphone screen or tablet screen. A system according to any one of claims 3 to 13, characterized in that the touch layer and/or additional device for inputting data, and/or the first control device and/or the device comprising the display are configured for wireless communication, preferably via Bluetooth and/or Wi-Fi and/or near field communication NFC and/or Li-Fi. An application for generating virtual buttons, configured to display the virtual buttons within an application or game. The application described in claim 15, characterized in that the virtual button has the form of a context menu and is configured to receive information from the device having a touch layer, including information from the device and possibly data from the additional device, and to generate data in the form of a context menu, in particular video data, depending on this information and these data. An application wherein the virtual buttons are additionally configured to be able to be assigned selected actions of games or physical buttons on a game controller, gamepad, mouse buttons, or keyboard keys. An application as described in any one of claims 15 to 17, further configured to transmit information about the activation button to a display or an additional device for inputting data. The application described in claim 14, further configured to transmit data, in particular video data, to the display. 1. A system for displaying virtual buttons, comprising:
a second control device comprising a processor, a memory and an application for generating virtual buttons according to claims 15 to 19;
Optionally, a system for controlling a video game according to claims 3 to 14,
a device comprising a touch layer, the touch layer configured to determine a position of a user's finger on the touch layer and to transmit information regarding the position to a second controller;
Optionally, an additional device configured to transmit data to a second controller;
The display and
The second control device receives the information of the device with a touch layer, possibly data from an additional device, and generates data, in particular video data, in the form of a context menu based on this information and data, and transmits this data, in particular video data, to a display, which is configured to receive the data, in particular video data, from the second control device and display a video image based thereon, for displaying virtual buttons. The system of claim 20, wherein the context menu includes any number of buttons distributed in a circle around a location corresponding to the position of the user's finger as determined by the device including the touch layer. A system as claimed in claim 20 or 21, wherein the button is configured, after activation, to transmit data to an external device, in particular an additional device for inputting data, and these data are transmitted to the second control device, in particular a tablet, phone or television, as information about pressing a physical button corresponding to a selected button on a game controller or keyboard and/or mouse. The system described in any one of claims 20 to 22, wherein the device including the touch layer is a smartphone or tablet, and the touch layer is the screen of the smartphone or tablet. The system described in any one of claims 20 to 23, wherein the second control device is a smartphone, a tablet, a smart TV, or a video game console. The system described in any one of claims 20 to 24, wherein the additional device for inputting data is a game controller, a gamepad, a joystick, or a device wearable on a user's finger. A system as described in any one of claims 20 to 25, characterized in that the display is configured to receive additional data, in particular video data, from an additional external source, such as a video game console, smartphone, or tablet, or to receive video streaming from the Internet, and to display video images based thereon. The system described in any one of claims 20 to 26, characterized in that the display has the form of a monitor, smart TV, projector, VR or AR device, smartphone screen or tablet screen. A system according to any one of claims 20 to 27, characterized in that the touch layer and/or additional device for inputting data and/or the second control device and/or the device with the display are configured to communicate wirelessly, preferably via Bluetooth and/or Wi-Fi and/or near field communication NFC and/or Li-Fi. 1. A system for determining a value of at least one parameter of a user's finger, comprising:
a device wearable on a user's finger;
at least one sensor disposed in or on the device wearable on the user's finger;
a third control device comprising a processor, a memory, and a third control application;
Optionally, a system for controlling a video game according to claims 3 to 14;
Optionally, a system for displaying virtual buttons according to claims 20 to 28, wherein said at least one sensor is configured to send a signal to said third control device;
a configuration for determining a value of at least one parameter of the user's finger, configured to determine the value of the at least one parameter of the user's finger and to transmit information about the determined value of the at least one parameter of the user's finger to the third control device;
The system, wherein the third controller is configured to receive information about a determined value of at least one parameter of the user's finger. The system described in claim 29, characterized in that the sensor has the form of a motion sensor, a pressure sensor, an optical sensor, and/or a proximity sensor. The system according to claim 29 or 30, further comprising a navigation surface, in particular in the form of a touch layer, in particular in the form of a touchscreen of a smartphone or tablet, or a touchpad or trackpad. The system of claim 31, wherein the navigation surface is configured to determine a value of at least one parameter of the user's finger, and the navigation surface is configured to transmit the value of the at least one parameter of the user's finger to the third control device. The system described in any one of claims 29 to 31 is configured to determine the value of a parameter of the user's finger, i.e., the angle between the distal phalange and the proximal phalange. The system described in claim 29, 30, 31 or 33 is configured to determine the value of a parameter of the user's finger, i.e., the pressure applied by the finger on the sensor. The system described in claim 29, 30, 31, 33 or 34, characterized in that it is configured to determine values of parameters of the user's finger, i.e., the pressure exerted by the finger on the navigation surface and/or the angle between the distal and proximal phalanges. The system of claim 31, configured to determine a value of a parameter of the user's finger, i.e., the pressure applied by the finger on the navigation surface. The system described in any one of claims 29 to 36, further comprising a display. The system of claim 37, wherein the display is a projector, monitor, smart TV, VR or AR device, smartphone screen, or tablet screen. The system described in claim 37 or 38, wherein the display is configured to receive data from the third control device and display an image based thereon. A system as described in any one of claims 37 to 39, characterized in that the display is configured to receive additional data, in particular video data from an additional external source, such as a video game console, smartphone, or tablet, or to receive video streaming from the Internet, and to display video images based thereon, including the image described in claim 39. The system described in any one of claims 29 to 40, wherein the third control device is configured to perform an action based on the value of a parameter of the user's finger. The system described in any one of claims 29 to 41, wherein the third control device is a tablet or a smartphone. The system described in any one of claims 29 to 42, characterized in that the third control device is integrated with the display, in particular the third control device is a smart TV. The system described in any one of claims 29 to 43, wherein the third control device is a video game console. The system described in any one of claims 29 to 44, characterized in that the device wearable on the user's finger has the form of a clip intended to be placed on the user's finger and has an open or closed ring on top, or on the side, or on which the sensor is mounted. The system described in any one of claims 29 to 45, wherein the device wearable on the user's finger includes a sensor adapted to measure the angle between the distal phalanges and the proximal phalanges of the user's finger. The system of any one of claims 29 to 46, wherein the device wearable on the user's finger comprises a motion sensor adapted to measure positional and angular differences between the finger and the navigation surface so as to measure the pressure exerted by the finger on the navigation surface. The system of any one of claims 29 to 47, wherein the device wearable on the user's finger comprises a pressure sensor and/or an optical sensor adapted to measure the pressure applied by the finger on the navigation surface. The system described in any one of claims 29 to 48, wherein the device wearable on the user's finger has an additional portion extending over the distal phalanx of the finger, and a motion sensor is located on the additional portion. The system described in any one of claims 29 to 49, characterized in that the device wearable on the user's finger includes a button under the fingertip, in particular a pressure sensor configured to generate and transmit a signal to the third control device when pressed. The system of any one of claims 29 to 50, characterized in that the third control device and/or the device for determining the value of at least one parameter of the user's finger and/or the navigation surface and/or the display are configured to communicate wirelessly, preferably via Bluetooth and/or Wi-Fi and/or near field communication (NFC) and/or Li-Fi. A system as claimed in any one of claims 29 to 51, wherein a user is configured to define a portion of the surface of the touch layer in the form of a touchscreen of a smartphone or tablet, thereby forming a navigation surface as claimed in claim 31. 1. A system for determining and presenting on a display the position of a user's finger, comprising:
a fourth control device configured to determine a position of a finger on the navigation surface, the fourth control device comprising a processor, a memory, and a fourth control application;
The display and
a navigation surface, the processor configured to transmit information regarding its position to the fourth control device;
The display and
Optionally, a system for controlling a video game according to claims 3 to 14;
Optionally, a system for displaying virtual buttons according to claims 20 to 28;
Optionally, a system for determining a value of at least one parameter of a finger according to claims 29 to 52, and an arrangement for determining a position of a user's finger on a navigation surface,
the arrangement comprises at least one sensor configured to determine the position of a user's finger and to transmit information about the determined position to the fourth control device;
the fourth control device is configured to receive the information from the arrangement for determining the position of a user's finger and to transmit data, in particular video data, to the display, the data, in particular video data, comprising a finger position indicator corresponding to the position of the user's finger on the navigation surface;
The system wherein the display is configured to receive data, particularly video data, from a fourth control device and display a video image based thereon including a finger position indicator. The system described in claim 53, wherein the fourth control device is a tablet, a smartphone, or a video game console. The system described in claim 53 or 54, characterized in that the fourth control device is integrated with the display, in particular the fourth control device is a smart TV. The system described in any one of claims 53 to 45, characterized in that the display is in the form of a monitor, smart TV, projector, or VR or AR device, or has the screen of a smartphone or tablet. A system as described in any one of claims 53 to 56, characterized in that the display is configured to receive additional data, in particular video data from an additional external source, such as a video game console, smartphone, or tablet, or to receive video streaming from the Internet, and is configured to display video images based thereon, including the image described in claim 53. A system as described in any one of claims 43 to 57, characterized in that the navigation surface is a touch layer, in particular the surface of a touchscreen of a tablet or smartphone. The system of claim 58, in the form of a touchscreen for a smartphone or tablet, comprising the navigation surface of claim 58, wherein the user is configured to define a portion of the surface of the touch layer. The system of any one of claims 53 to 59, wherein the arrangement for determining the position of the user's fingers on the navigation surface is adapted to be placed on at least one of the user's fingers, the arrangement comprises an accelerometer and/or a gyroscope, and is configured to use data from the accelerometer and/or gyroscope to determine the position of the user's finger on the navigation surface in real time. The system of claim 60, wherein the configuration for determining the position of the user's finger on the navigation surface further comprises an optical sensor configured to assist in determining the position of the user's finger on the navigation surface by determining the angle between the distal phalange and the proximal phalange to determine the finger flexion level. A system as described in any one of claims 53 to 60, characterized in that the fourth control device and/or the navigation surface and/or the configuration for determining the position of the user's finger on the navigation surface and/or the display are configured to communicate wirelessly, preferably via Bluetooth and/or Wi-Fi and/or near field communication (NFC) and/or Li-Fi. The system of any one of claims 60 to 62, characterized in that it is configured to recognize the gestures performed by the user's fingers across and/or on the navigation surface and/or in the form of a particular configuration of one or more of the user's fingers. A system according to any one of claims 3 to 63, wherein the first control device and/or the second control device and/or the third control device and/or the fourth control device are the same device.