RU2693197C2 - Universal operator intelligent 3-d interface - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to non-contact interaction of user with controlled devices. Technical result is achieved by the fact that control signal is sent by user, capturing a three-dimensional image, recognizing commands and transmitting the corresponding instruction to the controlled device. Signal is transmitted in minimum redundant form with volume of 1–10 Mb, at that, on the controlled device using the intelligent processing inverse transforms, the original command is restored, wherein the operator universal intelligent 3D interface in real time forms, complements, modifies distributed database of spatial images of operator space operator and working object, develops knowledge base.

EFFECT: technical result consists in providing universal intelligent 3D-interface, providing a universal, most close to natural, human interaction of the operator with different technical systems.

1 cl, 5 dwg

Description

The invention relates to the field of contactless user interaction with controlled devices in a distributed multi-agent information network, namely, the management of a local operating system, a distributed network computing system, robots, production automated control systems.

Known patent application US 20100020078, which discloses a system and method for creating a three-dimensional map of the object [1]. The system contains (FIG. 1) an image device (1) that generates and projects a plurality of rays on an object and captures the image reflected by the object.

The processor (2) processes the image data generated by the image device to recreate a 3D object map. The image processor calculates the 3D coordinates of points on the object's surface, determining the difference in illumination intensity. The image processor may be a separate device or a chip in the body of the image device. The 3D map (3) obtained by the image processor can be used for various purposes. For example, the card may be directed to an output device, such as a display. If the object (4) is a body or part of it, for example, a hand, the system can be used to provide, based on user interface gestures, to interactively control the application, instead of tactile interface elements, such as a mouse or joystick.

Known patent application US 2010199228 "Typing gestures" [2]. According to the description, the system (Fig. 2) consists of a computational means (5) and a capture device (6). The computing means may be a computer, a gaming system or a console, and may include hardware and / or software components.

According to the description of this method, the capture device is connected to the computing device through a communication channel. This can be a wired connection (for example, a USB connection, a Firewire connection, an Ethernet cable connection) or a wireless connection. The capture device generates a skeleton model and transmits via communication channel to the computing facility that receives depth data for recognizing the user's gesture (7) to control the application (8). Computing means may include a gesture recognizer. Captured data can be compared with gestures in the recognizer for controlling applications.

Known patent RU 2455676 "A method of controlling a device using gestures and a 3D sensor for its implementation." The device control method using gestures (Fig. 3) given by the user (9) consists in capturing a three-dimensional image using 3D-сoncopa (10), recognizing the gesture (11) and issuing a command to the controlled device, when this in the surrounding user space is allocated at least one sensory area (12), at least one expected gesture is associated with each sensory area and one command is associated with each combination of the sensory area and gesture, the position of his head and the position of his hand, determine the conventional line of sight (13), coming from the user's organ of sight and directed to the point of the surrounding space visible by the user, check whether the conventional line of sight is directed to the sensory area, analyze the gesture given by the user's hand and, if the gesture is served simultaneously with the guidance of a conditional line of sight at the sensory region, then the command associated with the given combination of the sensory region and the gesture is issued to the controlled device [3].

Known patent US 8933876 "Three-dimensional user interface for managing the session." The method (Fig. 4) describes the process of contactless interaction of the user (14) with the computer (15) by means of hand gestures (16) located in the field of view of the sensor (17) connected to the computer. After detecting a gesture, the contactless three-dimensional user interface moves from one state to another [4].

Patent US 8933876 closest to the proposed invention and adopted for the prototype.

The disadvantage of the prototype and other known gesture recognition devices is the use of external interfaces for image transmission, such as an S-video cable, a coaxial cable, a USB cable, etc. A sufficiently large amount of transmitted data and the bandwidth of the external interface limit the speed of the system's response to user gestures, and, accordingly, the speed of the control system as a whole.

Another disadvantage of existing systems for determining gestures is the problem of recognition by one technical device of large gestures and small gestures simultaneously. This problem is mainly related to the low resolution of the perceived 3D image, the limited accuracy of determining the coordinates and the insufficient speed of the system as a whole.

The proposed control method using gestures (Fig. 5), supplied by the operator (s) (18), is that on the transmitting side (19) connected by a communication channel (20) to the receiving side (21) through which the signal is transmitted in the minimum redundant form with a volume of 1-10 MB, the position of the operator (s) in the specified operator, work and global (for example, geo-location) spaces is made. Based on the distributed databases of spatial images (22) and corresponding knowledge bases (23), the code sends the coordinates of the operator’s skeleton nodes (s), the name of the operator (s) (spatial image), the control signals of the virtual device (mouse (26), keyboard ( 27), joystick (28), touch screen (29), microphone (30), kinect (31)), data of the exchange of distributed knowledge bases and databases, coordinate data of work objects present in the operator space, their name and description of the state, as well as interaction data of objects and subjects of multi-agent information environment. On the receiving side, on the basis of the data received from the transmitting side using the inverse transformations of intellectual processing, the original information flow is restored.

The main element of the system is an intelligent module (24). 3D video and audio information obtained from a 3D scanner (25) is being input to IM. Functions of the intellectual module:

1) Select from the general information flow the part corresponding to the individual objects (subjects) of the 3D operator space.

2) Recognition of selected objects based on a distributed database.

3) For the operator (s) and the working object (s), the coordinates of the skeletal nodes are determined, which make it possible to unambiguously determine their position.

4) On the basis of the received data, in accordance with the established specification, commands are recognized and transmitted in the form of code parcels to the receiving side.

5) Supplement the database and transfer via the communication channel the necessary information.

6) IM monitors dynamic changes in the coordinates of objects (subjects) of the operator space, saves information about these changes to the database. Based on the information accumulated in the database and on the basis of machine learning algorithms, it gains new knowledge, adding to the corresponding knowledge base.

The task which the claimed method is intended to solve is the recognition, coding, transmission and reproduction of operator control actions (s) for the local computing system (32), the distributed network computing system (33), robots (34), production automated control systems (35).

The technical result consists in the possibility of using this operator universal intelligent 3D-interface:

- to provide universal, closest to the natural human interaction of the operator with various technical systems (local computing system, distributed network computing system, robots, production automated control systems) based on the same type of movements, gestures, voice commands, and also manage these systems in any data generation mode conventional hardware operator control (keyboard, mouse, joystick, Kinect, touch screen, etc.).

- significantly reduce the amount of information transmitted via a communication channel by intelligently extracting and transmitting coordinates of skeletal nodes in real time, which are used to form and recognize a given list of control commands, and transfer changing information about 3D spatial images of the operator space (s) and operating space object (s) that are required to interpret operator control scenes.

INFORMATION SOURCES

1. The patent application US 20100020078 A1, cl. G09G 5/00, G06T 17/00. "Depth mapping using multi-beam illumination." Alexander Shpunt.

2. The patent application US 20100199228 A1, cl. G06F 17/28, G06F 3/033. "Gesture Keyboarding". Stephen G. Latta.

3. Patent RU 2455676, cl. G06F 3/00. "The method of controlling the device using gestures and a 3D sensor for its implementation." A.V. Roller, P.A. Zaitsev, D.A. Morozov.

4. Patent US 8933876 B2, cl. G06F 3/00, G06F 3/01, G09G 5/00, G06F 3/03. "Three dimensional user interface session control". Micha Galor, Jonathan Pokrass, Amir Hoffhung.

Claims (1)

Operator universal intelligent 3D-interface, which includes a control signal (gesture, voice command, movement) by the user, capturing a three-dimensional image, recognizing the command and transmitting the corresponding command to the controlled device, characterized in that the signal is transmitted in a minimally redundant form with a volume of 1-10 MB , while on the managed device using the inverse transformations of intellectual processing, the original command is restored, while the operator universal intelligent 3D Interface allows you to generate real-time, add, edit a distributed database of spatial images of the operator operator data space and the work object, to develop the knowledge base for problems of intellectual computer control online interaction and for process manufacturing and robotics.

Images