RU2696221C1 - Method of transmitting multimodal information on mission-critical objects - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: invention relates to computer engineering. Method comprises steps of: receiving, from a first user interface through protocol data units of a first modality; converting protocol data units of the first modality into protocol data units of the second modality; before receiving the protocol data units of the first modality, generating a request for the service of the multimodal info-communication system from the second user; transmitting generated request to communication controller; converting, in a communication controller, protocol data units of a first modality into protocol data units of a second modality by selecting those messages of a first modality which are necessary for realizing the service of the multimodal info-communication system; transmitting, through VPN tunnels, a service of a multimodal info-communication system with protocol data units of second modality to a second user interface.

EFFECT: high efficiency of using transmission capacity of data transmission networks.

1 cl, 1 tbl, 4 dwg

Description

The invention relates to communication technology, namely, processes for transmitting multimodal information and can be used to increase the efficiency of using the bandwidth of the channels of data transmission networks (SPD) of critical objects (CVO).

Currently, the reliability and safety of the operation of strategic infrastructure facilities, such as info-communication, auto or railway networks, water protection nodes, gas transmission, oil pipeline and energy systems, etc., determine the potentially achievable level of economic viability of the country, affecting its defense and national security. Such objects are critical objects of the state, since the failure (damage) of their elements leads to significant negative military-political and economic consequences, and sometimes to numerous casualties. For such objects, VPN tunnels are used as LDS channels, the redundancy of information in which can be used to transmit multimodal information.

To understand the essence of the invention, we introduce a number of definitions.

The first modality is physically recorded information.
about the individual (his condition; attitude to the message, to the interlocutor, to communication, etc.).

The second modality is the result of the multimodal infocommunication system service, which determines the state of the user of the workstation of a critical facility.

A multimodal infocommunication system (PIKS) is an interconnected set of information processing and storage systems, telecommunication systems that unite them, operating under a single control for the purpose of collecting, processing, storage, protection, transmission and distribution, display and use of multimodal information in the interests of the required quality.

Modal presentation of information - individual and autonomous processing of two or more modalities.

The multimodal presentation of information is an interconnected combined processing of modalities that allows you to manipulate the redundancy of the transmitted information.

A critical object is an object whose violation or termination of operation will result in loss of control of the economy of the Russian Federation, a constituent entity of the Russian Federation, or an administrative-territorial unit of a constituent entity of the Russian Federation, its irreversible negative change (destruction) or a significant decrease in the safety of the population [Federal Law of 23.06 .2016 No. 68-ФЗ “On the protection of the population and territories from emergency situations of natural and man-made nature”].

User interface - an interface that enables the transfer of information between a human user and the hardware and software components of a computer system [ISO / IEC / IEEE 24765-2010].

Multimodal interface - an interface that provides the choice of modalities for the transmission of various types of information and their joint use [Ronzhin A.L., Karpov A.A. Multimodal interfaces: basic principles and cognitive aspects // Transactions of SPIIRAS. 2006. Issue. 3. S. 300-319].

Automated workplace (AWS) - a software and hardware complex of an automated system designed to automate the activities of a certain type [p. 2.22 GOST 34.003-90].

A known method for implementing distributed multimodal applications (RF patent RU 2494444, G10L 17/22, 2013). The method comprises the steps of visualizing a visual display that includes at least one multimodal display element for which input data is received by the client device through visual modality and voice modality, the focus of the visual presentation being set to the first multimodal display element from this at least one multimodal display element; send the first request for a voice event to the application server, while the first request for a voice event is an asynchronous request using the Hypertext Transfer Protocol (HTTP); receiving an audio signal, which may represent a fragment of a user's speech, through a voice modality; sending uplink audio data representing this audio signal to a speech recognition module and receiving a response to a voice event request from an application server in response to said voice event request; and send a second voice event request to the application server in response to receiving a response to the voice event request.

A known method for distributing tasks between service robots and means of cyberphysical intellectual space with multimodal user service is a patent (RF RU 2638003, H04W 92/08, 2017). The method consists in updating the visual display on the client device in accordance with the result of speech recognition and sending a message along the control path of the SP / robot to the service robot, which includes the result of speech recognition, after which they are received from the application / robot server control path from application server message, which includes the result of speech recognition and which instructs the service robot to execute the appropriate action program, execute the action program, with responsible acceptance of the results of speech recognition, are sent to the application server on the path management application server / client an indication of the fact that the service robot performed the prescribed program; update the visual display on the client device in accordance with the result of the gesture recognition and send a message along the control path of the joint venture / robot to the service robot, which includes the result of the recognition of the gesture, and then receive a message along the path from the application server / robot that includes the result of gesture recognition and which requires the service robot to execute the appropriate action program, execute the action program corresponding to the accepted p As a result of gesture recognition, an indication is sent to the application server along the control path of the application server / client that the service robot has completed the prescribed program.

The closest in technical essence to the claimed method and selected as a prototype is a multimodal communication method through modality-specific interfaces (US patent 0180807, H04L 12/58, 2015), comprising the steps of receiving from the first user interface through a computer network, the protocol data blocks of the first modalities, converting protocol data blocks into a second modality and transmitting, after converting protocol data blocks of the first modality to a second modality, protocol blocks of a second modalities through a computer network into a second user interface adapted for the second modality.

The disadvantage of the subject area is that each information signal carries data about a communication service in a standard infocommunication system (speech, video, etc.). With this approach, in order to increase the quality and nomenclature of communication services, it is necessary to include additional technical means and extensively increase the capacity of communication channels. At the same time, there is redundant information in the IPO Datagrams protected by the CVO that are completely encapsulated in another packet for transmission through VPN tunnels.

The technical problem to which the invention is directed is to increase the efficiency of using the bandwidth of the KVO SPD channels.

Assessment of the efficiency of using the bandwidth of the SPD KVO channels will be determined by the indicator of the degree of using the bandwidth of the SPD KVO channels:

 (one)

,

– выделяемый канальный ресурс для обслуживания нагрузки до и после применения способа передачи модальной информации на КВО.Where

,

- the allocated channel resource for servicing the load before and after applying the method of transmitting modal information to the ATS.

The technical problem is solved by the fact that in the method of transmitting multimodal information on critical objects, which consists in receiving protocol data units of the first modality from the first user interface through a computer network, they convert protocol data units of the first modality into protocol data units of the second modality and additionally before by the way, from the first user interface via the computer network, the protocol data units of the first modality are received, a request for the multimodal infocommunication system service from the second user, the generated request is transmitted to the communication controller, while the protocol data units of the first modality form the service type, flags, fragment offset, lifetime and header checksum of IP datagrams of standard communication services for transmission through tunnels VPNs, convert the protocol data units of the first modality into protocol data units of the second modality in the communication controller by selecting those messages of the first modality that are necessary we have to implement multimodal services info-communication system is transmitted through VPN tunneling service infocomm multimodal system with protocol data unit of the second modality of the second user interface.

A new set of essential features allows you to achieve the specified technical result through the use of redundant information in secure SPD KVO for the transmission of multimodal information. When using VPN tunnels as SPD channels, the entire source IP-datagram (service and information parts) is encapsulated in a new format packet. At the same time, such fields of the original datagram as the type of service, flags, fragment offset, lifetime and header checksum are not used during routing and do not undergo any changes until the original form is restored on the receiving side after authentication and deletion procedures created for transmission via VPN tunnels routing and authentication headers. Consequently, in secure SPDs, it is possible to organize a modal information message transmission path (48 bits in each packet) without additional bandwidth costs.

Thus, the claimed method improves the efficiency of using the bandwidth of the SPD channels.

The claimed invention is illustrated by a detailed description, given with reference to the relevant figures:

in FIG. 1 is a flowchart illustrating the interaction between users of the CPO SPD;

in FIG. 2 - format of the original encapsulated datagram
TCP segment

in FIG. 3 - format of a datagram with an encapsulated TCP segment and messages of the first modality;

in FIG. 4 - format of the converted packet with messages of the first modality transmitted in the tunnel mode over the protected SPD KVO.

Implementation of the claimed method is as follows (Fig. 1). When the connection between users is initially established, first of all, a request for a PICS service from a second user is generated. In order to determine from the totality of messages of the first modality necessary for the implementation of this PICS service, the request is transmitted through the computer network to the communication controller. Next, the request is transmitted through the computer network to the first user for the formation and transmission to the controller of communication packets with messages of the first modality.

During the formation of packets for transmission to the controller of communication with messages of the first modality in the processing unit of the first modality, such fields as the type of service, flags, fragment offset, lifetime, and checksum of the header of the original datagram (Fig. 2) are replaced by messages of the first modality from user interfaces ( Fig. 3). In addition, a new packet routing headers are generated in the VPN client for transmission through the secure SAC KVO and user authentication on the receiving side of the VPN tunnel (Fig. 4).

An authentication check occurs on the receiving side of the VPN tunnel, after which packet routing and user authentication headers are deleted. From the messages of the first modality received from the processing unit of the first modality of the communication controller, only those messages of the first modality that are most suitable for the implementation of the PICS service generated by the second user are determined in the second modality generation unit.

After conversion in the communication controller of the first modality received from the first user through the secure SPD KVO to the second modality, the generated result of the PICS service is transmitted to the second user.

The validity of the theoretical assumptions was checked on the basis of calculating redundant information in the implementation of standard communication services through VPN tunnels. Based on the maximum size of the useful data block of one packet that can be transmitted by the protocol without fragmentation (MTU = 1500 bytes) and the sizes of fields such as service, flag, fragment offset, lifetime and header checksum (48 bits) in this packet, we can calculate the excess information in each standard communication service (table 1).

Table 1

Redundant information in standard communication services
when using secure SPD

application Typical data rates Excess
information Telephony 4-64 kbps 16-256 bps Voice messaging 4-32 kbps 16-128 bps High quality streaming audio 16-128 kbps 64-512 bps Videotelephony 16-384 kbps 64-1536 bps Video transfer 16-384 kbps 64-1536 bps Web navigation 100 kbps 400 bps Data Array Transfer 10-10000 Mbps 40-40000 kbps Transaction <100 kbps <400 bps Teams (management) 10 kbps 40 bps Still image <1000 kbps 4000 bps Email
(access to server) <100 kbps <400 bps Email (server-server) <100 kbps <400 bps

Accordingly, in the method of transmitting multimodal information to the CVT when using redundant information in standard communication services, the amount of occupied bandwidth of the SPD KVO channel when transmitting a standard communication service and the same message volume of the first modality will be less (Fig. 5).

To determine the gain in comparison with the method of the prototype, the calculation of the degree of utilization of the channel capacity for standard communication services after applying the method of transmitting multimodal information to the CVO was performed:

1) for video telephony with a data rate of 384 kbit / s:

2) for the transfer of data arrays with a transfer rate of 10 Mbit / s:

Due to the fact that the amount of redundant information in each packet is the same, the degree of utilization of the throughput capacity of the KVO SPD channels after applying the method of transmitting multimodal information to the KVO will be 0.4% and there will be no envy of the standard communication service provided.

The given examples clearly illustrate the effect of the application of the method of transmitting multimodal information to the CWO, which consists in increasing the efficiency of using the bandwidth of the channels of the LDS CWO.

The claimed method is not limited to the presented examples and, as follows from the foregoing, according to it, information can be exchanged between users using various types of modalities, both currently known and those that will become known in the future.

The analysis of the prior art made it possible to establish that there are no analogues that are characterized by a set of features identical to all the features of the claimed method of transmitting multimodal information at critical facilities. Therefore, the claimed invention meets the condition of patentability "novelty."

Search results for known solutions in this and related fields of technology in order to identify features that match the distinctive features of the claimed object from the prototype showed that they do not follow explicitly from the prior art. The prior art also did not reveal the popularity of the impact provided by the essential features of the claimed invention, the transformations on the achievement of the specified technical result. Therefore, the claimed invention meets the condition of patentability "inventive step".

Claims (1)

A method of transmitting multimodal information at critical facilities, which consists in receiving protocol data units of the first modality from the first user interface via a computer network, converting the protocol data units of the first modality into protocol data units of the second modality, characterized in that before receiving from the first user interface through the computer network, the protocol data units of the first modality form a request for a multimodal infocommunication service of the system from the second user, the generated request is transmitted to the communication controller, while the protocol data blocks of the first modality form the service type, flags, fragment offset, lifetime and header checksum of IP datagrams of standard communication services for transmission through VPN tunnels, in the communication controller, convert the protocol data blocks of the first modality into the protocol data blocks of the second modality by selecting those messages of the first modality that are necessary for the implementation of the multimodal service hydrochloric info-communication system, is transmitted through the VPN tunnel multimodal service info-communication system with the protocol data units of the second modality in the second user interface.

Images