RU2577467C1 - Mobile device for sensor plane - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: device relates to interactive displays, user interfaces and systems for recognition of gestures. Device comprises a source of illumination and optically interfaced with it first and second television cameras, directed towards the user's fingers and connected to the first and second inputs of special-purpose computer. Besides, proposed device comprises a bracelet and retainer engaged with the bracelet, the bracelet is placed on the wrist, and the first and second television cameras and illumination source are arranged in retainer. Television cameras create viewing plane passing through specified in advance image line on the photodetector matrix and optical lens centre of each television camera, creating sensor plane. Whereupon computer detects intersection of said plane with finger and determines by spatial position of said intersection.

EFFECT: technical result consists in providing the possibility of remote interaction with personal computers without using built-in sensor surface.

2 cl, 5 dwg

Description

The proposed device relates to techniques for interactive displays, user interfaces and gesture recognition systems, the use of which is constantly growing.

A number of devices for this purpose are known. So, in [1], a system is described that implements a projection of a virtual keyboard image on the user's palm. It includes an optical projector and a television camera located on the arms of the glasses, and an associated computing device. The finger touch of the virtual keyboard image at certain points is captured by the television camera and transmitted to a computing device that determines which element of the virtual keyboard touched the user's finger.

In [2], a mobile wearable system is considered, fixed on the forearm, perceiving the biopotentials of the corresponding muscles involved in the development of gestures made by fingers. The disadvantage of this system is the complex technical implementation and low noise immunity, the need for its individual calibration.

In [3], a device for digitizing the movements of limbs based on a structured illumination source similar to that used in the Kinect system [4] is proposed. However, devices of this class are not designed to work with small-sized objects such as fingers.

In [5], an interactive measuring system was proposed, which determines the position of the user's fingers in the area limited by the viewing areas from two to four or more television cameras, characterized by the possibility of 3D measurements, but it is very cumbersome for mobile applications.

A device of Canesta, Inc. is known. [6], the functional diagram of which is closest to the proposed invention and which is its prototype. This device contains an infrared (IR) emitter that creates a flat radiation field above the controlled surface, and an array photodetector with a lens located cantilever above the surface of the "virtual keyboard", the field of view of which covers the controlled surface. The receiving optical system is connected to a computing device into which a digitized image of the controlled surface is introduced. Touching with a finger or some kind of pointer (stylus), the surface of the “virtual keyboard” causes reflection from the finger (stylus) of part of the infrared radiation generated by the emitter, and its registration with the receiving optical system. The determination of the contact coordinate is based on solving a system of equations that describes the spatial position of the image of the illuminated part of the finger on the surface of the photodetector, which is part of the receiving optical system, and the surface formed by the radiation generated by the emitter (triangulation equation). As a result, the flat coordinates of the finger (stylus) on the surface of the “virtual keyboard” are determined, and a decision is made about which key was “pressed”.

The mobile device for organizing the touch plane, proposed in this application, differs from the known advanced functionality, namely, that it allows you to organize interaction with a personal or tablet computer, smartphone, etc. remotely without using the built-in touch surfaces. To do this, it is made in the form of a bracelet located on the wrist, and allows the user to start typing, drawing, editing, making phone calls, etc. at any time. In addition, finding in the environment a piece of a flat surface (table, window, window sill, etc.). item), you can "turn" this surface into a touch. It contains a backlight and optically coupled to it the first and second television cameras directed towards the user's fingers, connected to the first and second inputs of a specialized computer and placed in the latch. In this case, television cameras form viewing planes passing through a predetermined line of the image on the photodetector and the optical center of the lens of each television camera. The touch plane is created by the intersection of the viewing planes of the first and second television cameras and intersects the possible positions of the fingers of the user's hand. Next, a specialized computer calculates the intersection of the user's finger with the touch plane and determines the spatial position of this intersection. In addition, the side of the latch opposite to that which it is attached to the bracelet may have a flat surface parallel to the touch plane. This allows, having fixed the flat surface of the latch with any plane, to obtain tactile feedback that occurs when a finger touches a hard surface, as with a conventional touch screen or touchpad.

In FIG. 1 shows a functional diagram of the proposed device, where:

1, 2 - the first and second television cameras,

3 - infrared emitter,

4 - brush user

5 - finger

6 - bracelet,

7 - latch,

8 - specialized calculator,

Ψ ₁ , Ψ ₂ - solid viewing angles of the first (1) and second (2) television cameras,

Ω is the flow generated by the infrared emitter 3,

Φ ₁ is the viewing plane of the first television camera 1,

Φ ₂ is the viewing plane of the second television camera 2,

Θ - sensory plane.

In FIG. 2 illustrates the formation of the touch plane Θ, where:

M ₁ - photodetector matrix of the first television camera 1,

M ₂ - photodetector matrix of the second television camera 2,

O ₁ - the optical center of the lens of the first television camera 1,

O ₂ - the optical center of the lens of the second television camera 2,

C ₁ - the center of the photodetector matrix M ₁ ,

C ₂ - the center of the photodetector matrix M ₂ ,

P - point belonging to the finger of the operator,

O ₁ C ₁ - the Central optical axis of the first television camera 1,

O ₂ C ₂ - the Central optical axis of the second television camera 2,

f is the focal length of the lenses of the first and second television cameras 1 and 2,

l is the distance between the optical centers of the lenses O ₁ and O ₂ ,

S _M1 (1) is the 1st row of the photodetector matrix M ₁ ,

S _M2 (1) - the first row of the photodetector matrix M ₂ .

S _M1 (n) is the nth row of the photodetector matrix M ₁ ,

S _M2 (n) is the n-th row of the photodetector matrix M ₂ ,

S _M1 (i) is the i-th row of the photodetector matrix M ₁ ,

S _M2 (i) is the i-th row of the photodetector matrix M ₂ ,

- координата изображения точки П на матрице M₁, $$x_{\mathrm{one}}^P$$

- the coordinate of the image of the point P on the matrix M ₁ ,

- координата изображения точки П на матрице М₂. $$x_2^P$$

- the coordinate of the image of point P on the matrix M ₂ .

In FIG. 3 shows an example of imaging of a finger 5, where:

5 (t ₁ ) - the position of the finger 5 at time t ₁ ,

5 (t ₂ ) - the position of the finger 5 at time t ₂ ,

I _M1 (t ₁ ) - image of finger 5 on the photodetector matrix M ₁ at time t ₁ ,

I _M1 (t ₂ ) - image of finger 5 on the photodetector matrix M ₁ at time t ₂ ,

I _M2 (t ₁ ) - the image of the finger 5 on the photodetector matrix M ₂ at time t ₁ ,

I _M2 (t ₂ ) - image of finger 5 on the photodetector matrix M ₂ at time t ₂ .

In FIG. 4 shows a functional diagram of a device made according to the second claim, where:

9 - flat surface of the retainer 7,

10 - working surface.

In FIG. 5 shows an example of the design of the device, where:

RSV - printed circuit board.

ξ is the distance between the touch plane and the working surface.

The operation of this device is illustrated in FIG. 1 and proceeds as follows. The device covers the wrist near the wrist of user 4 using a bracelet 6. An infrared emitter 3 and two television cameras 1 and 2 optically paired with it are placed in the latch 7, which is rigidly connected to the bracelet 6. The infrared emitter 3 creates a backlight, the flow of which Ω crosses the fingers 4 brushes, including finger 5.

In FIG. 2 shows how on the line S _M1 (n) an image of the surrounding scene is formed, located in the plane Φ ₁ , hereinafter referred to as the viewing plane of the first television camera 1.

Similarly, on line S _M2 (n), an image of the surrounding scene is formed, located in the plane Φ ₂ , hereinafter referred to as the viewing plane of the second television camera 2.

Define the touch plane

as the intersection of the planes Φ ₁ and Φ ₂ . This plane is actually a stereo viewing area [7] of television cameras 1 and 2. When finger 5 crosses the touch plane Θ at time t _{2, the} images I _M1 (t ₂ ) and I _M2 (t ₂ ) will intersect lines S _M1 (n) and S _M2 (n), respectively. If the optical axes O ₁ C ₁ and O ₂ C _{2 are} parallel, changing the number n in the direction indicated by arrow α causes the plane Θ to move in the direction indicated by arrow β. This allows the user, if necessary, to adjust the position of the touch plane Θ relative to the brush 4.

The expression for calculating the coordinates of the point P corresponding to the position of the finger 9 located at the intersection with the touch plane Θ is obtained from FIG. 2 and has the form:

и $$x_2^{П}$$

пересечения I_M1(t₂) и I_M2(t₂) с S_M1(n) и S_M2(n) соответственно, специализированный вычислитель 8 рассчитывает координаты точки пересечения пальцем 5 сенсорной плоскости Θ по формуле 1. Координаты X_П и Y_П поступают далее в персональный компьютер, смартфон и т.п. для организации интерактивного взаимодействия с его программным обеспечением.The infrared flux power Ω generated by the emitter 3 should be sufficient to exceed the level of spurious components in the images of the surrounding scene in the video signals of the first and second television cameras 1 and 2. Thus, a specialized computer 8, having detected the i-th line of both television cameras in the video signals Levels 1 and 2 exceeding threshold values (for example, U _ref1 and U _ref2 ), makes a decision on the presence of I _M1 (t ₂ ) and I _M2 (t ₂ ). Next, fixing the coordinates $$x_{\mathrm{one}}^P$$

and $$x_2^P$$

the intersections of I _M1 (t ₂ ) and I _M2 (t ₂ ) with S _M1 (n) and S _M2 (n), respectively, specialized calculator 8 calculates the coordinates of the point of intersection of finger 5 of the touch plane Θ by formula 1. Coordinates X _P and Y _P proceed further to a personal computer, smartphone, etc. for organizing interactive interaction with its software.

In FIG. 4 shows a modification of the proposed device, namely, the latch 7 on the side opposite to that which it is attached to the bracelet 6, supplemented by a plane 9, optically paired with the first and second television cameras parallel to the touch plane Θ. If the user has a portion of a flat surface, such as a table, work with the device can be organized as follows. The user presses the latch 7 with the plane 9 to the work surface 10. Then he can touch the work surface 10 and move finger 5 on it, making interactive interaction with the software. In order for this touch to occur almost simultaneously with the intersection of the touch plane Θ, the distance ξ between the touch plane 0 and the working surface 10 should be no more than a few millimeters, so that the fingertip passes this distance almost instantly for the user.

In FIG. Figure 5 illustrates the use of television cameras of the Omni Vision type OVM7675 [8], having dimensions of 2.9 × 2.9 × 2.3 mm, which makes it possible to obtain a touch plane Θ located at a distance ξ from surface 10 of no more than 1.5 mm. Structurally, television cameras and the emitter can be located on a single multilayer printed circuit board 11, on the reverse side of which is placed a specialized computer 8.

To transfer the calculated coordinate values to a personal computer, smartphone, etc. the device may be equipped with a wireless data transmission device, such as Wi-Fi or Bluetooth.

Information sources

1. US patent No. 8228315 B1 from 01.24.2012

2. US patent No. 7333090 B2 of 02/19/2008.

3. US Patent No. 7259747 of 08.21.2007

4. US patent No. 7170492 B2 dated January 30, 2007.

5. Publication No. US 2010/0199228 A1 of 08/05/2010.

6. US patent No. 7006236 B2 dated February 28, 2006.

7. V.V. Korotaev, A.V. Blushing. Television measuring systems. St. Petersburg: St. Petersburg State University ITMO, 2008, p. 43.

8. Material from Omni Vision. OV7695. VGA product brief. Verison 1.0, August 2012.

http://www.ovt.com/download_document.php?type=sensor&sensorid=124.

Claims (2)

1. A mobile device for organizing a touch plane containing a source of illumination and a television camera optically paired with it, directed towards the fingers of a user, the television camera being connected to an input of a computer, characterized in that it further comprises a second television camera, optically paired with the first and connected to the second input of the calculator, a bracelet and a latch mechanically connected to the bracelet, with the bracelet placed on the wrist, and the first and second television cameras and the source the backlights are placed in the latch, and the touch plane is created by the intersection of the viewing planes of the first and second television cameras, in turn, these planes pass through a given image line of each television camera and the optical center of its lens, as a result, the touch plane intersects the possible position of the user's fingers, and the calculator fixes the intersection with the user's fingers of the touch plane and determines the spatial position of this intersection. 2. A mobile device for organizing a touch plane according to claim 1, characterized in that the side of the latch opposite to that which it is attached to the bracelet has a flat surface parallel to the touch plane.

Images