RU2486608C2 - Device for organisation of interface with object of virtual reality - Google Patents

Abstract

FIELD: measurement equipment.

SUBSTANCE: device comprises two optically coupled TV chambers of IR-range, a source of IR-light, a radiation-scattering surface optically connected with them and a space indicator, besides, outputs of TV chambers are connected to an input of a calculating device, and the scattering surface is a screen of a 3D-monitor with appropriate glasses of a user-operator. Besides, there is an interface module that generates video signals for right and left eyes, which represent a 3D-image of a virtual reality object, and changing it depending on coordinates of a space indicator, besides, the input of the interface module is connected to the output of the calculating device and the input of the 3D-monitor.

EFFECT: expansion of functional capabilities, namely provision of potential interaction between a human operator and virtual space objects.

2 cl, 10 dwg

Description

The invention relates to techniques for television measurement systems and can be used to build three-dimensional user interfaces in three-dimensional virtual reality systems.

Known is the gesticulation data input device developed by Microsoft (the commercial name of the product is Kinect XBX-360), which allows you to interact with virtual reality objects by tracking the position of the hands and body of the operator. The device includes a sensor consisting of an IR (infrared) -band television camera and an infrared emitter optically coupled to it, forming a set of beams, a visible-range television camera and a computing device associated with them. The range sensor determines the spatial position of the points highlighted on the operator, and the camera of the visible range is designed to detect gestures and facial expressions.

The disadvantages of this device include its low noise immunity.

The closest in technical solution to the proposed invention is the RF patent "Device for three-dimensional manipulation." This device uses two backlight sources and an infrared television camera optically paired with them, and its field of view covers part of the perimeter of the surface being monitored. In this case, images of the controlled surface are recorded alternately illuminated by the first and second sources of illumination, and the pointer located on the path of illumination — the operator’s finger or stylus — creates darkened areas on the illuminated surface. By entering the position of the shaded areas in the computing device, you can determine the spatial coordinates and orientation of the pointer.

The disadvantages of this device are limited functionality that does not allow the human operator to interact with virtual spatial objects formed by three-dimensional image monitors (3D monitors).

The objective of the invention is to expand the functionality, namely the possibility of interaction of the human operator with virtual spatial objects.

To this end, a device for organizing a user interface with a virtual reality object containing an IR illumination source, an infrared television camera, and the television camera output being connected to a 3D position calculator of a real spatial pointer, an additional optically coupled second infrared television camera is introduced from the first, the output of the second television camera is connected to the second input of the 3D position calculator of the real spatial index, and the IR illumination source is optical and is coupled to the screen of the newly introduced 3D monitor in such a way that the radiation reflected from it is directed towards the first and second infrared television cameras, the first and second television cameras being optically paired with the eyes of the human operator of the system observing the images generated on the 3D monitor images using glasses appropriate for the 3D monitor, the 3D monitor being connected to a newly introduced shaper of virtual reality objects, for example, a personal computer, which, in turn, is connected to a 3D computer -position of the real spatial pointer. In this case, the working area of the device is formed by the intersection of the solid viewing angles of television cameras with the solid viewing angles of the eyes of the human operator of the system and is limited by the plane of the 3D monitor screen.

Moreover, the screen of the entered 3D monitor with the corresponding glasses of the user-operator is used as a scattering surface, in addition, an interface module has been introduced that generates video signals for the left and right eyes of the operator, which are a 3D image of a virtual reality object, and changes it depending on coordinates of the real spatial pointer located in the working area of the device, which are fed to the input of the interface module from the output of the 3D position calculator of the real space a pointer, which, in turn, determines these coordinates by the flat coordinates of the images of the real spatial pointer coming from the output of the first and second television cameras of the infrared range, and the operator observes the image of the virtual reality object and the real image of the spatial pointer, and by changing the position of the pointer in space, it can combine a 3D image of a virtual reality object formed by a 3D monitor with the actual position of the pointer, and if they coincide, the interface module It produces a change 3D-image of the object of virtual reality and its provisions, with the elaboration of an appropriate control signal to this event.

The interface module is necessary for changing the position of the spatial pointer on the screen of the 3D monitor, as well as for interacting with the computer.

A second infrared television camera is needed to form a stereo zone of the device.

A 3D monitor is used to form an image and to reflect radiation from the backlight to the IR cameras.

Figure 1 shows the functional diagram of the proposed device, where:

1, 2 - the first and second television IR cameras, with solid viewing angles ψ ₁ and ψ _2, respectively,

3 - 3D monitor screen,

4 - IR emitter with a stream ω,

5 is a real spatial pointer,

6 - points

7 - calculator 3D-position of the real spatial pointer,

8 - interface module.

Figure 2 shows the optical-geometric diagram of the device, where:

9 - radiation flux generated by the infrared emitter 4 and incident on the screen of a 3D monitor,

10 - reflected from the screen, the radiation flux directed towards the television cameras 1 and 2.

Figure 3 shows the optical-geometric diagram of the image formation of the illuminated screen of the 3D monitor, where:

11, 12 - photodetector arrays,

13, 14 - optical centers of the lenses,

15, 16 - images of the illuminated surface of the screen of a 3D monitor on photodetector arrays,

17 - the working area of the stereo system, for television cameras 1 and 2.

Figure 4 shows the optical-geometric diagram of the formation of images of a real spatial index, where:

18, 19 are images of a real spatial index on the photodetector arrays of television cameras 1 and 2.

Figure 5 shows the geometric diagram of the formation of the working space of the device, where:

20 - human operator of the system,

21 - zone stereo vision of a human operator,

22 - stereo viewing area of television cameras 1 and 2,

23 - the working space of the system.

Figure 6 shows an illustration of a shift of flat images on the screen of a 3D monitor, forming a virtual reality object (hereinafter OVR), where:

τ is the interpupillary distance of the human operator,

ρ - parallax (shift) of the images along the horizontal axis,

s is the distance from the pupil to the screen of the 3D monitor,

q is the distance from the pupil to the point belonging to the virtual reality object,

0XYZ is the coordinate system associated with the left pupil of a human operator.

Figure 7 shows an illustration of the formation on the screen of a 3D monitor of a virtual reality object, where:

24, 25 - images on the screen of the 3D monitor, forming the ID;

26 is a virtual reality object observed by an operator.

On Fig shows examples of images on photodetector arrays, where:

L, P - pointer images,

a _L and b _L , a _L , b _P - informative points of the image of the pointer on the photodetector arrays of television cameras 1 and 2.

Figure 9 shows an example of a virtual reality object - a switching lever, where:

27 - the initial position of the lever,

28 - the position of the lever after touching the spatial pointer.

Figure 10 shows the functional diagram according to the second dependent claim.

The device operates as follows:

The interface module generates video signals for the left and right eyes of the operator, which is a 3D image of a virtual reality object, and changes it depending on the coordinates of the real spatial pointer located in the working area of the device, which are input to the interface module from the output of the computing device, which, in turn, determines these coordinates by the flat coordinates of the images of the real spatial index coming from the output of the first and second television cameras p, and the operator observes the image of the virtual reality object and the real image of the spatial pointer, and by changing the position of the pointer in space, it can combine the 3D image of the virtual reality object formed by the 3D monitor with the actual position of the pointer, and when they coincide, the interface module changes 3D image of a virtual reality object and its position, with the development of a control signal corresponding to this event.

The emitter 4 (figure 1) creates a stream of infrared radiation, the solid angle of propagation of which covers the screen of the 3D monitor 3 and is placed so that its radiation, reflected from it, is directed towards the real spatial index 5 and television cameras 1 and 2, as illustrated in figure 2.

The interface module 8 generates a video signal that creates a stereo image on the screen of the 3D monitor, consisting of 2 flat images shifted along the horizontal axis of the monitor and optically separated using glasses 6.

The 3D position calculator of the real spatial index from the stereo image obtained using the infrared television cameras determines the spatial coordinates of the pointer and passes them to the interface module, which determines the moments of coincidence of the spatial coordinates of the pointer with the spatial coordinates of the virtual reality object, and accordingly changes its image on the screen of a 3D monitor and generates the corresponding interface signals.

Considering the different spectral range of the image (visible) formed on the screen of the 3D monitor and the IR cameras 1 and 2, the signal received by them does not depend on the image itself formed on the monitor screen. In this case, the high contrast associated with the shading of the monitor screen, reflecting the flow of infrared radiation, provides high noise immunity. OVR, the construction of the interface with which the proposed device is intended may be, for example, an imaginary image of a control panel containing buttons, indicators, switches and other objects whose states in space can be changed by commands sent to interface module 8, “touching” their spatial images real pointer 5. In this case, the shift of the elements (parallax), flat components (for the left and right eyes of the system operator) of the images of the virtual reality object, is determined by the ratio one of Fig.6, and has the form:

$$\rho -\left(\frac{s-q}{q}-\mathrm{one}\right)\tau (\mathrm{one})$$

where: ρ is the parallax (shift) of the images along the horizontal axis, q is the distance from the pupil to the point belonging to the virtual reality object, s is the distance from the pupil to the 3D monitor screen, τ is the interpupillary distance of the human operator.

To ensure the imposition of the spatial index 5, "acting" in real space with a virtual reality object, it is necessary to combine the stereo vision zone of the human operator of this system with the stereo viewing area of television cameras 1 and 2.

The working area of the system is formed by the intersection of the stereo viewing zone 21 of the person operator 20 of the system with the stereo viewing area 22 of the television cameras 1 and 2. Moreover, the stereo viewing area of television cameras is in turn formed by the intersection of the solid angles ψ ₁ and ψ ₂ .

The calculation of the spatial position of the pointer 5 is carried out according to the coordinates of its images on the photodetector arrays. Images are entered into the 3D position calculator of the real spatial index 7 in the form of a digital video signal stream, and Fig. 6 shows an example of images on photodetector arrays. In this case, the spatial position of the pointer 5 is calculated according to the following algorithm:

1. Search on images of characteristic points of the pointer (points a _L , b _L , a _P , b _P ),

2. The calculation of the spatial position of these points according to the perspective transformation formulas defined below.

The formulas for calculating the spatial position of the points are as follows:

$$\{\begin{array}{c}\tilde{A}=P^{-\mathrm{one}}{\tilde{a}}_L,\\ \tilde{A}=T^{-\mathrm{one}}{P}^{-\mathrm{one}}{\tilde{a}}_P,\\ \tilde{B}=P^{-\mathrm{one}}{\tilde{b}}_L,\\ \tilde{B}=T^{-\mathrm{one}}{P}^{-\mathrm{one}}{\tilde{b}}_P;\end{array}(2)$$

Where:

$$P^{-\mathrm{one}}-\left[\begin{array}{cccc}\mathrm{one}& 0& 0& 0\\ 0& \mathrm{one}& 0& 0\\ 0& 0& \mathrm{one}& 0\\ 0& \frac{\mathrm{one}}{\mathrm{<figure-callout\; id="0"\; label="f"\; filenames="00000015.png,00000017.png"\; state="\{\{state\}\}">f</figure-callout>}}& 0& \mathrm{one}\end{array}\right]$$

is the matrix of the inverse perspective transformation, and f is the focal length of the lens, $$T^{-\mathrm{one}}=\left[\begin{array}{cccc}\mathrm{one}& 0& 0& \Delta x\\ 0& \mathrm{one}& 0& 0\\ 0& 0& \mathrm{one}& 0\\ 0& 0& 0& \mathrm{one}\end{array}\right]$$

is the matrix inverse to the shift matrix along the 0X axis by Δx is the shift of the right television camera relative to the left.

, $$\tilde{B}$$

- однородные координаты точек A и B, $$\tilde{A}$$

, $$\tilde{B}$$

- homogeneous coordinates of points A and B,

, $${\tilde{a}}_P$$

, $${\tilde{b}}_L$$

, $${\tilde{b}}_P$$

- однородные координаты точек a_L, a_p, b_L, b_P, и могут быть получены, исходя из теории перспективных преобразований, изложенной, например, в [3]. $${\tilde{a}}_L$$

, $${\tilde{a}}_P$$

, $${\tilde{b}}_L$$

, $${\tilde{b}}_P$$

 are the homogeneous coordinates of the points a_L, a_p, b_Lb_P, and can be obtained on the basis of the theory of promising transformations described, for example, in [3].

The disclosure of system 2 leads to 12 linear equations with respect to the 6 component coordinates of points A and B.

- фазовое состояние точки А. Это позволит в дальнейшем определить направление подхода пространственного указателя к заданной точке объекта виртуальной реальности.Further, the orientation of the pointer (Euler angles) relative to the coordinates 0XYZ, and changes in the coordinates of point A in time, i.e., a set $$(X_A,Y_A,Z_A,{\dot{X}}_A,{\dot{Y}}_A,{\dot{Z}}_A)$$

- the phase state of point A. This will further determine the direction of the spatial pointer approach to a given point of the virtual reality object.

Thus, the 3D position calculator of the real spatial pointer implements the algorithm described above (points 1 and 2), calculates the orientation of the spatial pointer, and the phase state of point A belonging to the pointer. These calculations can be performed on the basis of a productive processor oriented to processing video information, for example, TMS320DM368 DaVinci [4].

To determine the moment of coincidence of the spatial coordinates calculated by formulas (2) with the OVA generated on the screen of the 3D monitor 3, the coordinates of the spatial pointer 5 are entered into the interface module 8. When they coincide with the coordinates of the individual elements of the virtual reality object, the interface module 8 changes the position of the individual elements of the object of virtual reality and "translates" it into a new spatial state. For example, the imaginary image of the spatial control lever is moved to a new position when the pointer 5 is “touched” (the coordinates of the pointer 5 coincide with the coordinate of a certain point of the 3D image) with the imaginary image of this lever. The foregoing is illustrated in Fig. 9, where the initial 3D image of the lever 27 changes to the image 28 after the coordinates of the spatial pointer point coincide with the coordinates of the 3D image point of this lever. Naturally, a certain number of intermediate 3D images of a virtual reality object is possible, making the transition of a virtual reality object from one spatial position to another more smooth.

The above is illustrated by the following algorithm that describes the operation of the interface module:

, фазовые координаты точки А, принадлежащие данному пространственному указателю в момент времени t1. Check if the domain is in the given region for a certain i-th virtual reality object in state I $$R_I^{J}i$$

, phase coordinates of point A belonging to a given spatial index at time t

. $${(X_A,Y_A,Z_A,{\dot{X}}_A,{\dot{Y}}_A,{\dot{Z}}_A)}_t\in R_I^J$$

.

2. If this condition is met, the i-th virtual reality object goes into a new state i + 1, the image of which is rebuilt on the screen of a 3D monitor.

The detection of the real spatial index 5 can also be carried out by directly illuminating it with the emitter 4, while the emitter should be located in the area of television cameras 1 and 2, as shown in Fig.10. The brightness of the illumination source should ensure the detection of a useful signal against the background spurious emissions of the surrounding scene, and provide illumination of the pointer in the working area of the device (defined above as the intersection of the stereo vision zone of a human operator with the stereo viewing area of television cameras 1 and 2). In addition, in the working area there should not be glare objects that reflect radiation into the lenses of television cameras 1 and 2. Thus, the emitter in this case should be optically paired with television cameras 1 and 2.

This algorithm of work can be implemented on a specialized computing device or on a typical personal computer.

Information sources

1. US patent US 20100199228. Gesture Keyboarding.

2. RF patent No. 2362216. Device for three-dimensional manipulation (prototype).

3. R. Duda, P. Hart, Pattern Recognition and Scene Analysis. M., World, 1976

Claims (2)

1. A device for organizing a user interface with a virtual reality object, comprising an IR illumination source, an infrared television camera, a surface that is optically coupled to them and a real spatial pointer, and the output of the television camera is connected to a 3D position calculator of a real spatial pointer, characterized in that it further comprises a second television camera, optically paired with the first, the output of which is connected to the second input of the 3D-polo computer a real spatial index, which determines the spatial coordinates of the real spatial index from the stereo image obtained with the help of infrared television cameras, and the screen of the entered 3D monitor with the corresponding operator’s glasses is used as a scattering surface, in addition, an interface module is introduced, generating video signals for the left and right eyes of the operator, which is a 3D image of a virtual reality object that defines The moments of coincidence of the coordinates of the real spatial pointer, calculated by the 3D position calculator of the real spatial pointer, with the coordinates of the virtual reality object, and changing it depending on the coordinates of the real spatial pointer. 2. A device for organizing a user interface with a virtual reality object according to claim 1, characterized in that the backlight is optically coupled to the first and second infrared television cameras and is located in close proximity to the first and second infrared television cameras.

Images