RU2701090C1 - System and method for automatic execution of user-defined commands - Google Patents

Abstract

FIELD: electronic equipment.

SUBSTANCE: invention relates to a method, an electronic device and an automatic command execution system. System comprises an agent which collects information on the environment in which a command is to be executed, and compiles an environment map based on the collected information, predictor providing prediction of information which is collected by an agent, wherein the agent and the predictor are configured to be trained based on information obtained by the agent after prediction, a database for obtaining, storing and updating the environment map, a command processing unit which enables receiving and processing a command in accordance with a medium map, and generating a script for executing a command, wherein the agent enables executing a script of executing a command received from the instruction processing unit.

EFFECT: technical result consists in automatic command execution.

11 cl, 7 dwg

Description

Technical field

The present invention relates to the automatic execution by an electronic device of instructions given by a user in a natural language.

Description of the Related Art

At the moment there are many applications in which user commands are set in a natural language, namely words, gestures, facial expressions. Such applications can be used in various smart devices, for example, smartphones, devices of home assistant robots, etc.

The scenario of user interaction with the electronic device application usually consists in the fact that the user sets up informal commands with voice, gestures, or facial expressions. The application recognizes user commands and executes them. The application is pre-trained on the servers, and during the operation of such an application almost no online algorithms are used to learn directly by direct feedback, that is, when an application learns online when it tries to execute a user command, it receives feedback from the user. To date, there are no effective online learning algorithms for electronic device applications.

Known solutions are characterized in that they can only be pre-trained, and only after such a separate training, well-known applications can be used in electronic devices.

The prior art document US 20130254139 A1, published September 26, 2013, which discloses systems and methods for constructing a universal intelligent assistant with the possibility of training. The systems and methods disclosed in the document are associated with the creation of an intelligent assistant that can perceive human requests / commands in simple text form, also in a natural language format, and perform tasks assigned by the user. Moreover, before use, the user himself must train the assistant through the provided user interface and / or use only some knowledge that the assistant already knows. A trained assistant can generate more general knowledge based on what he is learning, and can apply more general knowledge to execute commands that he never saw or never studied directly, and can revise and improve knowledge according to the feedback result . The document does not disclose the methods of training an assistant immediately during his work, that is, online.

The prior art document US 20140095931 A1, published 03.04.2014, which discloses a method and system for automating the testing process of electronic devices. A well-known solution provides the basis for automation testing of devices for automation testing of embedded systems. The Device Test Automation Engine (DTAF) allows the user to test the firmware of the embedded device using test scripts that can capture various interfaces of the device under test. A graphical user interface (GUI tool) is created based on the device in the test configuration and user input. This GUI tool shows the various interfaces of the device under test. Hardware for device testing automation provides communication between the test instrument and the device under test. DTAF enables the testing process to significantly increase the performance, efficiency, and test coverage of embedded software. A disadvantage of the known solution is that machine learning is not used, that is, changes in the user interface are not allowed.

From the prior art at the moment there are two approaches to testing smartphones.

The first simplest approach is based on the fact that the user reads the instructions on his own and, following the instructions, conducts testing of his smartphone. Although the human user has the ability to learn and can make decisions in non-standard situations, the user spends a lot of time reading and reproducing the instructions, it is also clear that there is a human error factor, which also takes time to understand and correct. Moreover, a person cannot perceive and reproduce several instructions at the same time.

The second approach involves the use of automatic tools that perceive instructions in the form of a formal language, that is, commands come in the form of instructions from a predetermined set of instructions (for example, a set of mobile phone commands, a set of robot movement commands), such instructions are unambiguous, then there are those that do not allow ambiguous interpretations, and do not require user participation directly (a person is needed to use the result). Automatic tools can at the same time perceive and reproduce several instructions simultaneously. However, when using automatic means there is no error factor. However, well-known automatic means are unable to solve non-standard situations that were not previously trained.

SUMMARY OF THE INVENTION

An intelligent system for executing commands by an electronic device is proposed, which differs from well-known applications in two key characteristics:

- automatically collects information online about the structure of the environment in which it is necessary to execute commands, for example, the robot assistant collects information about the premises before doing work, and, for example, the application for testing a smartphone before work collects information about the smartphone itself.

Such devices, using the collected information, are able to carry out the commands that a person sets.

The proposed system can create, for example, a mobile phone application map and use such a map to execute human commands immediately after receiving a command.

A system for automatically executing at least one command by an electronic device is provided, comprising: an agent configured to collect information regarding an environment in which at least one command should be executed, with the ability to map the said environment based on the information collected; a predictor configured to predict information that the agent collects, the agent and the predictor configured to learn based on information received by the agent after the prediction; a database configured to receive a map of said environment from an agent, store a map of said environment, update a map of said environment upon receipt of a new map of said environment from an agent; a command processing unit, configured to receive and process at least one command in accordance with the map of the aforementioned environment, and to generate a script for executing at least one command, the agent being also configured to execute a script for executing at least one command, received from the command processing unit. Moreover, the agent is configured to calculate a predictor prediction error. Moreover, based on the calculated predictor prediction error, the agent learns to find the state of the target environment with the maximum predictor error. Moreover, based on the calculated predictor prediction error, the predictor is trained to minimize the prediction error. Moreover, at least one command is a user command. Moreover, the user command is visual information or audio information, or text information. Moreover, the database is a neural network.

Also proposed is a method of operating a system for automatically executing commands by an electronic device, comprising the steps of:

a) agent:

surveys the current state of the environment in which at least one command should be executed,

selects an action from a variety of actions, and

before performing the action, the agent passes the mentioned current state of the environment and the selected action to the predictor;

b) predictor:

from the said current state of the target environment and the action selected by the agent predicts the subsequent state of the said environment,

transmits the prediction to the agent;

c) agent:

performs the selected action, while receiving information about the real state of the mentioned environment:

compares the real state of said medium with the prediction received from the predictor,

calculates the predictor error,

based on the calculated error, it is trained to find the states of the mentioned medium with the maximum predictor error;

d) the predictor, on the basis of the calculated error, is trained to minimize the error;

d) after each action, the agent updates the map of the said environment, entering the corrective information into the map of the mentioned environment and sends the map of the mentioned environment to the database;

f) the database stores a map of the environment mentioned, received from the agent;

g) command processing unit:

processes at least one command in accordance with a map of said environment obtained from a database,

generates a script for the execution of at least one command based on the processing of at least one command and a map of the above environment,

passes the script for the execution of at least one command to the agent for execution,

moreover

in the case when the map of the mentioned environment stored in the database matches the current state of the environment in which at least one command should be executed, the agent executes the script for the execution of at least one command,

in the case when the map of the mentioned environment stored in the database does not coincide with the current state of the environment in which at least one command should be executed, steps (a) - (g) are repeated

Moreover, at least one command is a user command. Moreover, the user command is visual information, or audio information, or text information.

An electronic device is also provided, comprising a system for automatically executing at least one instruction by an electronic device, comprising: an agent configured to collect information regarding an environment in which at least one instruction should be executed, with the possibility of compiling a map of said environment based on the collected information; a predictor configured to predict information that the agent collects, the agent and the predictor configured to learn based on information received by the agent after the prediction; a database configured to receive a card, said environment from an agent, store a map of said environment, update a map of said environment when a new card of said environment is received from an agent; a command processing unit, configured to receive and process at least one command in accordance with the map of the aforementioned environment, and to generate a script for executing at least one command, the agent being also configured to execute a script for executing at least one command, received from the command processing unit.

Brief Description of the Drawings

The above and other features and advantages of the present invention are explained in the following description, illustrated by the drawings, in which the following is presented:

FIG. Figure 1 shows schematically two modes of operation of a system for automatically executing instructions in a natural language by an electronic device.

FIG. 2 schematically presents the first mode of operation of the system for automatic execution by an electronic device of commands given in a natural language.

FIG. 3 schematically presents the second mode of operation of the system for automatic execution by an electronic device of commands specified in a natural language.

FIG. 4 illustrates how the map of the target environment changes using the example of the proposed system when testing a mobile phone.

FIG. 5 schematically shows a second mode of operation of the system when testing a mobile device.

FIG. 6 schematically shows the second mode of operation of the system when the robot cleaner is operating.

7 schematically shows the second mode of operation of the system when using the system as an electronic assistant.

DETAILED DESCRIPTION OF THE INVENTION

The present invention allows the use of natural human language, as well as sign language and facial expressions, for commanding. A user using an electronic device containing the proposed system can simply set commands in the usual language and get the result in the form of command execution. Using the present invention, it is possible to train an electronic device that executes commands directly at the time of receipt of the command, and at the same time it can solve non-standard situations that were not laid down during the pre-training period, which occurs due to updating the map of the target environment and due to methods based on reinforcement learning (trial and error methods).

A method and system for automatic execution by an electronic device of commands given by a user in a natural language.

Natural language here refers to human speech, human gestures and human facial expressions.

The proposed system receives data from the source of the command, which can be visual, sound, text, and the space (target environment) in which the system must execute commands.

The target environment here will be called the environment in which user commands are executed. This can be, for example, a smartphone application that needs to be tested or performed with other actions, or a room in which the robot should clean, etc.

The proposed system contained in the electronic device instructs the electronic device to investigate the target environment for which the commands will be given, and compiles a map (structure) of the features of the specified target environment. This is achieved by preliminary machine learning of the blocks of the system located in the electronic device, or by using a user who can enter data through the user interface to create a map or structure of the features of the target environment.

In other words, the blocks of the system are pre-trained using machine learning to automatically explore the target environment and create a map or structure of the target environment. Moreover, an electronic device containing the proposed system, with the help of this system is able to investigate the target environment, detect its features and create a map of the target environment. In addition, the proposed system can solve the problem of studying the target environment and mapping the target environment through interaction with the user interface, that is, at any time, the map of the target environment can be changed as needed by the user. The proposed system requires the electronic device to execute commands given by the user in a natural language using a compiled map of the target environment. Moreover, the electronic device will interact with the target environment according to the map of the target environment, which the system can change at any time in accordance with the interests of the user, for example, through the user interface, and also automatically, periodically re-examining the target environment, for example, at certain intervals, which can be set by user.

The system can detect semantic (semantic) relationships between map elements and is able to follow inaccurate human commands in the form of a natural language. That is, teams that require a large number of identical actions can be automated.

With the proposed approach, there is no need to create special software for each individual task that an electronic device should perform.

The figure 1 schematically shows two modes of operation of a system for automatically executing by an electronic device commands specified by a user in a natural language.

The proposed system can operate in two key modes. In the first mode, the system generates and updates the target environment map, in the second mode, the system recognizes and executes user commands in natural language in accordance with the target environment map received in the first mode or updating the target environment map immediately after receiving the user command.

So:

- in the first mode, the system examines the target environment in the most efficient way and builds a map of the target environment;

- in the second mode, the system recognizes and executes a user command based on a map of the target environment.

In the second mode, the system uses user commands and a map of the target environment as input. In this mode, the system can also update the map of the target environment by feedback from the user.

The proposed system can also execute user commands without a pre-compiled map of the target environment. In this case, the system can map the target environment online, that is, at the same time as receiving commands from the user.

The figure 2 schematically shows the first mode of operation of the system for automatic execution by an electronic device of commands specified in a natural language. In the first mode of operation, machine learning is performed to automatically build a map of the target environment without user control.

In the first operation mode, the first block containing the agent, the second block containing the predictor, and the third block containing the database participate.

The database may be a neural network or an object capable of storing data and updating it. The database may store data regarding the target environment.

A system for automatically executing user commands by an electronic device, operating in the first mode, using the first block (agent), receives information from the target medium, which can be visual information, text information or sound information, depending on the type of medium in which the commands should be executed user.

An agent is a block that can be represented by a machine learning algorithm, for example, a neural network, which serves to decide what action needs to be performed in the current state of the target environment.

The state of the target environment is the visible (monitored) predetermined part or the entire target environment at a given time. In the case of a mobile phone, the state of the target environment is, for example, the screen of a mobile phone and all information about its settings. For example, in the case of a vacuum cleaner as a cleaner, the state of the target environment is respectively a photograph of the house at the current moment and so on.

The predictor is the second block, which can be represented by a machine learning algorithm, for example, a neural network, which serves to predict the next state of the target environment, provided that the current state is known and some planned action is taken. The prediction error by the predictor of the next state is used by the agent to decide how efficiently the selected action is. A large error means that this state is less investigated than the state in which we receive a small error.

In the training mode and in the process, at each moment of time, the agent sees only one state of the target environment. The agent obtains the state of the target environment from its sensors and, on the basis of this, generates the next action to go to the next state of the target environment. The agent's task is to examine the map of the target environment. Accordingly, the agent learns to do actions that lead to the least known states (parts) of the target environment. The degree of knowledge is determined precisely by the predictor. Each time before an agent performs an action, the predictor predicts the result of this action, that is, what state the target environment will go into, in other words, what one state will look like that will be visible (visible) at the next moment in time. After the action is taken, the agent sees the real state of the target environment and, accordingly, can calculate the error between reality and prediction, due to this both the agent and the predictor are trained.

The predictor predicts the next state of the target environment always, upon receipt of information from the agent about the intention to perform an action. At the moments when the agent observes a state very different from that predicted by the predictor, the agent updates the map of the target environment. Thus, the agent and the predictor are always trained and paired. The predictor is trained to predict the next state of the target environment. An agent is trained to head into the least predictable states of the target environment. This happens all the time both at the stage of map building and at the stage of interaction with the user. If the map is already built, then the map updates will go with less intensity, since the predictions will more or less correspond to reality.

Let us describe in more detail the work of the agent and predictor in the first mode of the system, at the stage of constructing a map of the target environment.

Mode 1. Building a map of the target environment

A) The agent reviews the state of the target environment, selects the action (for example, where to direct the robot vacuum cleaner for further study of the target environment). Before performing an action, the agent passes the state of the target environment and the selected action to the predictor.

B) The predictor of the current state of the target environment and the action selected by the agent predicts the next state of the target environment (that is, what the agent sees after the action is completed), and then transmits the prediction to the agent.

C) The agent performs the action selected in (A) (for example, forces the robot cleaner to rotate, etc.), obtains the real state of the target environment.

D) The agent compares the received real state with the state received from the predictor, then the agent calculates the predictor error.

E) An agent, based on the calculated error, is trained to find the state of the target environment with the maximum predictor error. A predictor, based on the calculated error, is trained to minimize the prediction error.

E) All the states of the target environment detected by the agent are written to the target environment map, in turn, simply in the order in which the agent selects them and this is an important point, because the agent tries to select the most relevant, poorly studied states of the target environment by searching for states with the maximum a mistake.

An agent, as a rule, receives information about the target environment from sensors that are determined directly by the target environment.

The predictor receives the current state of the target environment and the action that the agent is about to perform as an input. Each environment has its own set of actions. For example, for the task of testing the phone, the set of actions may be as follows: pressing a button, swipe the screen, and so on. For a robot, this is moving left, right, right, back) in the current state. Based on this data received from the agent, the predictor tries to predict the next state of the target environment. After the action is completed, the agent receives the real state, passes it to the predictor, and it compares the real state with the already predicted one. Thus, the agent receives a predictor prediction error.

The agent, perceiving information from the target environment, is in a fixed state. As soon as the agent notices that changes have occurred in the target environment, the agent enters the state of investigation of the part of the target environment where the changes occurred, that is, the state of the target environment in order to record the changes. The change that occurred in the target environment is determined by the agent processing the input signal corresponding to this environment.

The predictor predicts what the agent will detect, that is, what information will come from the agent at the next point in time. Then the agent receives real information. Predicted information and real information from the agent enters the database. The predictor does not interact directly with the database. The agent writes the observable states to the database in some form, this may depend on the implementation of the database or other structure that is used to store the map.

Incorrect predictor prediction and correct information from the agent are used to train the agent and predictor.

Based on the observed states of the target environment, the agent forms an updated map of the target environment. The target environment map can be updated both in the environment information collection mode and in the user command execution mode.

The figure 3 schematically shows the second mode of operation of the system for automatic execution by an electronic device of commands specified in a natural language.

In the second mode of operation, the system recognizes and executes the user command (s) given by the user in a natural language.

In the second mode of the system, the input data in the form of user commands and maps of the target environment from the database are received in the command processing unit.

The command processing block creates a language model suitable for command recognition. Then, the command processing unit generates a script script, which is a set of commands that will be executed directly on the electronic device in which the system is used for the corresponding target environment. Script script is formed based on user commands and maps of the target environment in the command processing block.

The script script from the command processing block is transferred to the agent that runs the script script. The script is executed based on the most promising and effective ways to execute scripts. For each state of the target environment, the most suitable commands are determined that will correspond to the instructions received. First of all, the system tries to follow exactly these instructions, and if the system cannot execute such instructions, it searches semantically for the next ones based on the map compiled.

The target environment map is constantly updated in the database, and the agent is responsible for updating the target environment map, this process can be performed in parallel with the script execution, that is, online directly when the electronic device is working with the user.

Let us describe in more detail the mode of operation of the system in which user commands are executed.

Mode 2. Execution of user commands

A) The command processing unit (s) of the user and the map of the target environment from the database are received in the command processing block.

B) Based on the user command and the target environment map, the command processing block creates a script for the user command execution script.

C) The script script (the next action) is transferred to the agent, the agent passes the script script to the predictor.

D) Agent and predictor see the state of the target environment.

If the target environment map received by the command processing unit from the database and used to compose the script script coincides with the actual state of the target environment at the moment the agent executes the script, the agent runs the script script.

If, at the moment the agent executes the script, the real state of the target environment does not match the map received by the command processing unit from the database and used to compose the script,

the agent determines that the script cannot be executed,

the first operating mode of the system is turned on, as a result, the agent receives the real state and predicted by the predictor, compares them, considers the error, and rebuilds the map of the target environment,

transmits a map of the target environment to the command processing unit and to the database,

the command processing unit, based on a new map of the target environment and the same user command, creates a new script script,

the new script is passed to the agent, and the agent executes it.

The figure 4 illustrates how changing the map of the target environment for example, the application of the proposed system when testing a mobile phone.

Suppose the user instructs the system to automatically execute electronic commands to find the “Font size” option in the mobile phone menu.

Figure 4 on the left shows a map of the target environment, which was stored in the system at the time of the command from the user. The system initially begins to use a map of the target environment stored in the database, after a command is sent from the user, the second module is responsible for the direct execution of the command by the electronic device. Since the “Font size” option was not found on the mobile phone when using the target environment map, the exploration of the target environment and updating the map of the target environment are automatically activated. That is, the first mode of operation of the system described above is turned on. The target environment map is updated in the database. The updated map is shown in figure 4 on the right, after which the system switches to the second mode of operation to execute the command execution script.

The system may include a reconnaissance mode, when it is turned on, a map of the target environment can be updated periodically.

The figure 5 shows an example of a second mode of operation of the system when testing a mobile device. A test script is submitted to the input of the command processing unit, that is, some description of the error reproducing process, or of any script on the phone that is set in the usual human language, and a map of the target environment obtained from the database. In this particular case, the target environment is the mobile phone application that needs to be tested. A script script is generated in the command processing block based on the application map and test script, in this case a list of commands that need to be run on the mobile phone to execute each of the test script commands. Received from the command processing device the list of commands goes to the agent. The agent gives a command to the smartphone application to play commands one by one or all at the same time, depending on the type of list of commands received. Thus, testing a smartphone is done without user intervention.

The figure 6 schematically shows the second mode of operation of the system when the robot cleaner. The database periodically updates the map of the target environment. Visual and / or voice user commands are sent to the input of the data processing unit. Upon receipt of voice commands from the user, a voice command processing device is used to process them, and a visual command processing device is used to process visual commands, both of which are in the command processing block. An updated map of the target environment is also fed to the command processing block. In this case, the target environment is the room in which the assistant robot must execute user commands. After processing the user command and the map of the target environment, the command processing block generates a script script, the data is transferred to the agent, which issues the corresponding set of commands to the assistant robot. The assistant robot goes to the right place and performs the necessary action, for example, cleaning.

If the next command does not produce the expected result, or if the predictor cannot predict the expected result, this means that the map of the target environment has changed. It detects an agent that interacts with the predictor. If during the execution of commands by the robot it is detected that the map of the target environment has changed, the map is updated immediately before being transmitted to the command processing unit.

Assistant robots known from the prior art do not use machine learning, so such robots can only execute a programmed set of commands and cannot respond to an unusual situation.

For example, a user wants a robot to bring an item. If the robot cannot find the item in the place that is stored in its database, it will not be able to complete the task, since such a robot cannot learn and adapt to unknown situations.

Assistant robot using the proposed system for automatic execution by an electronic device of commands given by a user in a natural language, it will be able to adapt to changing the positions of objects in online mode, that is, directly upon receipt of a command from the user.

In a final product that uses the proposed system to automatically execute electronic instructions by a user in natural language, the assistant robot will be able to:

Stage 1. Explore the map of the house:

The robot examines the environment (target) environment (for example, a house), updates the map in the database, trains the neural network: builds associations (kitchen - for food, bedrooms - for sleep, etc.), studies the similarities and proportions of objects, and so on . It should be noted that both the agent and the predictor have software modules that are machine learning algorithms, respectively, any of them can be a neural network or another machine learning algorithm capable of self-learning.

Stage 2. Execution of commands:

The robot is trained and ready to execute commands.

An instruction processing unit, for example, containing a neural network, translates user commands into a sequence of actions — a script and transfers the script to the agent. The robot follows the created scenario. If something goes wrong at any stage, the robot will be able to adapt, thanks to the associations in the neural network obtained at the training stage, and also because the room map can be updated directly during the robot’s task.

Periodically, such a robot must update its knowledge thanks to the presence of an agent and a predictor.

The figure 7 schematically shows another example of a second mode of operation of the system when using the system as an electronic assistant. The target environment in this case will be a set of preferences and interests of the user. Such information is collected by the agent when observing the user's preferences, for example, during his Internet surfing, that is, statistics are collected on the user's visits to the sites, watching videos, listening to music, etc. Then the agent, on the basis of sorting the collected information using statistics that displays the user's preferences, forms a map of the user's interests.

The database periodically updates the map of the user's interests.

Visual and / or speech user commands and a map of user interests are fed to the input of the command processing block. In this case, user commands may concern, for example, the selection of news on one topic or music tracks, films, or the selection in the online store of products that the user prefers, or the choice of a hotel, etc.

When receiving commands from the user and the interest card, the command processing device generates a script script and passes it to the agent. The agent issues to the electronic device the appropriate script commands for selecting, for example, news, music tracks or products, etc. or any recommendations matching the user's request.

Here are just a few examples of possible uses of the invention. The proposed system for automatic execution by an electronic device of commands given by a user in a natural language can find applications in almost all spheres of human life where electronic devices are used.

Although the invention has been described in connection with some illustrative embodiments, it should be understood that the invention is not limited to these specific embodiments. On the contrary, it is assumed that the invention includes all alternatives, corrections and equivalents that may be included in the essence and scope of the claims.

In addition, the invention retains all equivalents of the claimed invention, even if the claims change during the review process.

Claims (38)

1. A system for automatically executing at least one command by an electronic device, comprising: an agent configured to collect information regarding the environment in which the at least one team should be executed, with the ability to map the said environment based on the collected information; a predictor configured to predict information that the agent collects, the agent and the predictor configured to learn based on information received by the agent after the prediction; a database configured to receive a map of said environment from an agent, store a map of said environment, update a map of said environment upon receipt of a new map of said environment from an agent; a command processing unit, configured to receive and process at least one command in accordance with the map of the aforementioned environment, and with the possibility of generating a script for the execution of at least one command, moreover, the agent is also configured to implement a script script to execute at least one command received from the command processing unit. 2. The system of claim 1, wherein the agent is configured to calculate a predictor prediction error. 3. The system according to claim 2, in which, based on the calculated predictor prediction error, the agent is trained to find the states of the medium with the maximum predictor error. 4. The system of claim 2, wherein, based on the calculated predictor prediction error, the predictor is trained to minimize the prediction error. 5. The system according to any one of claims 1 to 4, in which at least one command is a user command. 6. The system of claim 5, wherein the user command is visual information, or audio information, or text information. 7. The system according to any one of claims 1 to 6, in which the database is a neural network. 8. A method of operating a system for automatically executing commands by an electronic device, comprising the steps of: a) agent: surveys the current state of the environment in which at least one command should be executed, selects an action from a variety of actions, and before performing the action, the agent passes the mentioned current state of the environment and the selected action to the predictor; b) predictor: by the said current state of the medium and the action selected by the agent, the subsequent state of the said medium is predicted, transmits the prediction to the agent; c) agent: performs the selected action, while receiving information about the real state of the mentioned environment: compares the real state of said medium with the prediction received from the predictor, calculates the predictor error, based on the calculated error, it is trained to find the states of the mentioned medium with the maximum predictor error; d) the predictor, on the basis of the calculated error, is trained to minimize the error; d) after each action, the agent updates the map of the mentioned environment, entering the corrective information into the map of the mentioned environment, and sends the map of the mentioned environment to the database; f) the database stores a map of the environment mentioned, received from the agent; g) command processing unit: processes at least one command in accordance with a map of said environment obtained from a database, generates a script for the execution of at least one command based on the processing of at least one command and a map of the above environment, passes the script for the execution of at least one command to the agent for execution, moreover in the case when the map of the mentioned environment stored in the database matches the current state of the environment in which at least one command should be executed, the agent executes the script for the execution of at least one command, in the case when the map of said medium stored in the database does not coincide with the current state of the medium in which at least one command should be executed, steps (a) - (g) are repeated. 9. The method of claim 8, wherein the at least one command is a user command. 10. The method according to claim 9, in which the user command is visual information, or audio information, or text information. 11. An electronic device containing a system according to any one of claims 1 to 7.

Images