CN116055468A

CN116055468A - Service efficiency evaluation system and method for unmanned cluster system

Info

Publication number: CN116055468A
Application number: CN202211664429.1A
Authority: CN
Inventors: 金松昌; 史殿习; 杨烟台; 李�杰; 孟杨瑞
Original assignee: Tianjin Binhai Artificial Intelligence Innovation Center
Current assignee: Tianjin Binhai Artificial Intelligence Innovation Center
Priority date: 2022-12-23
Filing date: 2022-12-23
Publication date: 2023-05-02

Abstract

The invention provides an unmanned cluster system-oriented service efficiency evaluation system and method, comprising the following steps: the system comprises a Web front end, a server end and a plurality of unmanned nodes; the Web front end is used for acquiring user request parameters, sending the user request parameters to the server end and receiving user response results sent by the server end; the server side is used for analyzing the user request parameters, determining user request information and sending the user request information to the unmanned node; the system is also used for storing the evaluation response information sent by the unmanned node and outputting a user response result to the Web front end based on the evaluation response information; and the unmanned node is used for performing performance evaluation according to the user request information sent by the server, outputting evaluation response information and sending the evaluation response information to the server. The invention can improve the efficiency level and the service capability of different service capabilities for providing services in the group unmanned system executing cooperative tasks, and improve the data service reliability, the data communication adaptability and the data updating instantaneity of the unmanned system.

Description

Service efficiency evaluation system and method for unmanned cluster system

Technical Field

The invention belongs to the technical field of unmanned system performance evaluation, and particularly relates to an unmanned cluster system-oriented service performance evaluation system and method.

Background

At present, the execution capacity and efficiency of a single unmanned system are low, so that in order to improve the requirement of collaborative task completion, a group unmanned system is adopted to finish tasks in a collaborative manner according to the service capacity of the group unmanned system, the factors of poor execution capacity and small coverage range of the single unmanned system are broken through, but services in different unmanned systems, such as data service, network service, algorithm service and the like, are required to be acquired in the collaborative task process of the group unmanned system, different service capacities can influence the success or failure of the collaborative task of the group unmanned system, and the capability of verifying the performance and grade of the service provided by the different service capacities in the collaborative task execution of the group unmanned system cannot be realized in the current prior art.

Disclosure of Invention

In order to overcome the shortcomings of the prior art, the present invention provides an unmanned cluster system service performance evaluation system, which includes:

the system comprises a Web front end, a server end and a plurality of unmanned nodes;

the server side is respectively in communication connection with the Web front end and the unmanned node;

The unmanned nodes are in communication connection through a server;

the Web front end is used for acquiring user request parameters and sending the user request parameters to the server end, and receiving user response results sent by the server end;

the server side is used for analyzing the user request parameters, determining user request information and sending the user request information to the unmanned node; the system is also used for storing the evaluation response information sent by the unmanned node and outputting a user response result to the Web front end based on the evaluation response information;

the unmanned node is used for performing efficiency evaluation according to the user request information sent by the server, outputting evaluation response information and sending the evaluation response information to the server.

Preferably, the server side includes:

the system comprises a Web server, an application server, a database server and a file server;

the Web server is respectively in communication connection with the application server, the file server and the database server;

the Web server is in communication connection with the Web front end;

the application server and the file server are respectively in communication connection with the unmanned node;

The Web server is used for analyzing the user request parameters sent by the Web front end to obtain index parameter information, generating an index query request and a plug-in query request according to the index parameter information, and generating user request information as an evaluation task to be uploaded to the application server based on the evaluation index and the plug-in stored in the file server and according to the index query request and the plug-in query request; the system is also used for receiving user response information sent by the application server and storing the user response information to a database server; the server is also used for retrieving a user response result from the database server according to the user request parameters and uploading the user response result to the Web front end;

the file server is used for storing preset evaluation indexes and plug-ins;

the application server is used for uploading the user request information to the unmanned node; the system is also used for receiving the evaluation response information sent by the unmanned node, analyzing and aggregating the evaluation response information to obtain user response information, and sending the user response information to the Web server through ROSBridge;

The database server is used for storing the user response information;

wherein the index parameter information includes: and the corresponding evaluation index in the user request parameters, the corresponding plug-in name of the evaluation index and the plug-in parameters.

Preferably, the application server includes:

and the node management module: the method comprises the steps of storing node address information and node state information of unmanned nodes;

and an information management module: the method comprises the steps of receiving user request information sent by a Web server, and uploading the user request information to a corresponding unmanned node based on node address information and node state information of the unmanned node; the system is also used for receiving the evaluation response information sent by the unmanned node, analyzing and aggregating the evaluation response information to obtain user response information, and sending the user response information to the Web server through ROSBridge;

wherein the node address information includes: node IP address and node port number; the node status information includes: a node available state and a node unavailable state.

Preferably, the Web server includes:

and the evaluation index system management module: the method comprises the steps of analyzing user request parameters sent by the Web front end, obtaining corresponding evaluation indexes, obtaining plug-in names and plug-in parameters corresponding to the evaluation indexes according to the evaluation indexes, and taking the evaluation indexes, the plug-in names corresponding to the evaluation indexes and the plug-in parameters as index parameter information;

And (3) evaluating an engineering management module: the evaluation engineering is established according to the index parameter information;

an evaluation task management module: the system comprises an evaluation project, an application server, an index query request and a plug-in query request, wherein the evaluation project is used for generating an index query request and a plug-in query request according to the evaluation project, and based on evaluation indexes and plug-ins stored in the file server, user request information is generated according to the index query request and the plug-in query request and is used as an evaluation task to be uploaded to the application server; the system is also used for receiving user response information sent by the application server and storing the user response information to the database server; the system is also used for receiving user response information sent by the application server and storing the user response information to a database server; and the server is also used for retrieving a user response result from the database server according to the user request parameters and uploading the user response result to the Web front end.

Preferably, the evaluation task management module generates user request information as an evaluation task and uploads the evaluation task to an application server according to the index query request and the plug-in query request based on the evaluation index and the plug-in stored in the file server, and the evaluation task management module includes:

Generating an index query request and a plug-in query request according to the index parameter information in the evaluation engineering;

accessing an index library in the file server according to the index query request, judging whether an evaluation index in the index parameter information exists in the index library, and acquiring address parameter information corresponding to the index when the evaluation index exists in the index library;

accessing a plugin library in the file server according to the plugin query request, judging whether plugins in the index parameter information exist in the plugin library, and acquiring address parameter information corresponding to the plugins when the plugins exist in the plugin library;

packaging the index and the address parameter information corresponding to the index and the plug-in and the address parameter information corresponding to the plug-in as user request information, and uploading the user request information as an evaluation task to an application server; wherein the address parameter information includes: IP address, port number, and parameter authentication information.

Preferably, the unmanned node includes:

the system comprises an unmanned system client, an index verification module and an unmanned system database;

the index verification module is respectively in communication connection with the unmanned system client and the unmanned system database;

The unmanned system client is in communication connection with the server;

the unmanned system client is used for judging whether the plug-in exists in the index verification module according to the received user request information, and downloading the plug-in to the index verification module according to the acquired address parameter information of the plug-in at the server side when the plug-in does not exist in the index verification module; the index verification module is also used for issuing an index verification instruction to the index verification module according to the user request information; the system is also used for receiving the evaluation response information uploaded by the index verification module and synchronizing the evaluation response information to the application server;

the plug-in verification module is used for loading and starting a corresponding plug-in the user request information based on the unmanned system database and the index verification instruction to obtain evaluation response information, and uploading the evaluation response information to an unmanned system client;

the unmanned system database is used for storing node address information and node state information of unmanned nodes and synchronizing the node address information and the node state information to the server side;

Wherein the address parameter information includes: IP address, port number and parameter verification information; the node address information includes: node IP address and node port number; the node status information includes: a node available state and a node unavailable state.

Preferably, the Web front end includes:

and the evaluation operation management module: the method comprises the steps of judging whether a current user has access rights according to acquired user request parameters, and sending the user request parameters to a server side when the user has the access rights;

and the data connection management module is used for: the method comprises the steps of receiving a user response result sent by a server side and scoring the user response result;

visual result display module: the system is used for carrying out visual analysis according to the user response result and carrying out multidimensional display aiming at the visual analysis result; wherein the visual analysis comprises: contrast analysis and sensitivity analysis; the multi-dimensional presentation includes: radar chart display and pie chart display.

Based on the same inventive concept, the invention also provides an unmanned cluster system-oriented service efficiency evaluation method, which comprises the following steps:

resolving the acquired user request parameters to determine user request information;

Based on the stored node address information and node state information of the unmanned node, sending the user request information to the corresponding unmanned node;

and outputting a user response result based on the received evaluation response information sent by the unmanned node.

Preferably, the analyzing the obtained user request parameter to determine the user request information includes:

acquiring user request parameters, and analyzing the user request information to obtain index parameter information;

generating an index query request and a plug-in query request according to the index parameter information;

generating user request information based on a pre-stored index library and a plug-in library and according to the index query request and the plug-in query request;

Preferably, the generating the user request information based on the pre-stored index library and the plug-in library and according to the index query request and the plug-in query request includes:

judging whether an evaluation index in the index parameter information exists in an index library or not based on the index library and the index query request, and acquiring address parameter information corresponding to the evaluation index when the evaluation index exists in the index library;

Judging whether the plug-in the index parameter information exists in a plug-in library or not based on the plug-in library and the plug-in query request, and acquiring address parameter information corresponding to the plug-in when the plug-in exists in the plug-in library;

packaging the evaluation index and the address parameter information corresponding to the evaluation index and the plug-in and the address parameter information corresponding to the plug-in as user request information;

wherein the address parameter information includes: IP address, port number, and parameter authentication information.

Preferably, the outputting the user response result based on the received evaluation response information sent by the unmanned node includes:

receiving evaluation response information sent by an unmanned node, and analyzing and aggregating the evaluation response information to obtain user response information;

and storing the user response information, and outputting a user response result based on the user request parameters.

receiving user request information;

performing efficiency evaluation according to the user request information, and outputting evaluation response information;

wherein the user request information is the user request information as described above.

Preferably, the performance evaluation is performed according to the user request information, and evaluation response information is output, including:

judging whether a preset database contains an insert in the user request information, acquiring address parameter information of the insert according to the user request information when the database does not contain the corresponding insert, and downloading the insert to the database according to the address parameter information;

loading and starting a plug-in the user request information based on the database to obtain evaluation response information;

Compared with the closest prior art, the invention has the following beneficial effects:

the invention provides an unmanned cluster system-oriented service efficiency evaluation system and method, comprising the following steps: the system comprises a Web front end, a server end and a plurality of unmanned nodes; the server side is respectively in communication connection with the Web front end and the unmanned node; the unmanned nodes are in communication connection through a server; the Web front end is used for acquiring user request parameters and sending the user request parameters to the server end, and receiving user response results sent by the server end; the server side is used for analyzing the user request parameters, determining user request information and sending the user request information to the unmanned node; the system is also used for storing the evaluation response information sent by the unmanned node and outputting a user response result to the Web front end based on the evaluation response information; the unmanned node is used for performing efficiency evaluation according to the user request information sent by the server, outputting evaluation response information and sending the evaluation response information to the server. The service efficiency evaluation system adopts a B/S+C/S hybrid architecture, a user submits user request parameters through a Web page, a server side processes the user request parameters, and forwards related requests to unmanned nodes, the unmanned nodes run unmanned system cluster tasks and service capacity to perform related evaluation, the result is fed back to the server side, the server side collects the evaluation result and processes the evaluation result according to corresponding logic, and the result is fed back to a front end for display; the invention can improve the efficiency level of providing services in the group unmanned system executing cooperative tasks by different service capacities, and improve the data service reliability, the data communication adaptability and the data updating instantaneity of the unmanned system; and the evaluation and management problems of multiple service capabilities in the unmanned system can be effectively solved under the unmanned cluster condition, and various service capabilities of the unmanned system are improved.

Drawings

Fig. 1 is a schematic diagram of a service performance evaluation system for an unmanned cluster system according to the present invention;

fig. 2 is a schematic diagram of functional modules in the service performance evaluation system for the unmanned cluster system according to the present invention;

FIG. 3 is a schematic diagram of a service performance evaluation system deployment for an unmanned cluster system according to the present invention;

FIG. 4 is a schematic diagram of an evaluation flow of a service performance evaluation system according to an embodiment of the present invention;

FIG. 5 is a typical task state machine of a unmanned cluster performing a search task in an embodiment of the present invention;

FIG. 6 is a network topology of a semi-physical verification environment in accordance with an embodiment of the present invention;

fig. 7 is a schematic flow chart of Web front-end and back-end execution in the service performance evaluation method for the unmanned cluster system provided by the invention;

fig. 8 is a schematic flow chart of unmanned node execution in the service performance evaluation method for an unmanned cluster system provided by the invention.

Detailed Description

The following describes the embodiments of the present invention in further detail with reference to the drawings.

Example 1:

the invention provides an unmanned cluster system service effectiveness evaluation system, a structural schematic diagram of which is shown in figure 1, comprising:

the unmanned nodes are in communication connection through a server;

the unmanned node is used for performing efficiency evaluation according to the user request information sent by the server side, outputting evaluation response information and sending the evaluation response information to the server side;

the Web front end can be deployed on any computer PC communicated with the server; the server side can be deployed on the same server or on different servers; the unmanned node is also an application client and is deployed on an unmanned system node in the test environment;

The server side comprises:

the Web server is in communication connection with the Web front end;

The file server is used for storing preset evaluation indexes and plug-ins;

the database server is used for storing the user response information;

The application server comprises:

The Web server includes:

The evaluation task management module generates user request information as an evaluation task to be uploaded to an application server according to the index query request and the plug-in query request based on the evaluation index and the plug-in stored in the file server, and comprises the following steps:

The unmanned node comprises:

the unmanned system client is in communication connection with the server;

The Web front end includes:

From the perspective of user interaction, the system provided by the invention adopts a B/S architecture; the user views the Web page through the Web front end (i.e. browser) and uses the unmanned system service to evaluate the efficiency system through the Web page, so as to perform experimental design, experimental implementation, monitor experimental process, view experimental results and the like. The operation of the user is transmitted to the Web background program through socket and the like for processing and responding, the corresponding Web background is used as a server end, the request of the user is analyzed, the user is processed according to the established logic, and finally the responding result is transmitted to the front-end page through socket for displaying. Because the system is oriented to the unmanned cluster system, the Web background program is not a background program in the traditional sense, namely the Web background program is processed according to established logic, the front-end request can be responded by local calculation or database query unlike the traditional Web background program, probes, plug-ins and the like are arranged in corresponding test environments to perform data detection, data acquisition and the like, and then the data acquired from the test environments are aggregated, processed correspondingly and stored in a service database, and then a result responding to the user request can be generated.

From the perspective of acquiring evaluation data, the system background provided by the invention adopts a C/S architecture, and the server side comprises a database server, a file server, a Web server and an application server; the application server runs an application service program, the file server runs a file service program, the Web server runs a Web background service program, the application client serves as an unmanned node in the evaluation environment, the application client runs an application client program, and the server and the unmanned node are interconnected through the DDS. The application service program and the Web background service program are communicated through the ROSBridge, so that the requirements of data acquisition, related test commands and the like which need to be unfolded by the Web background program are sent to the application service program developed by the C++, and the application service program transmits related commands and messages to each unmanned node in the assessment environment in a cross-node mode through the communication module. The client application program on the unmanned node is in charge of analyzing the message and the command transmitted by the server side besides communicating with the application service program on the server, and then communicating with the file service program to acquire the probe program and the plug-in program of the server side; and then loading and running the pulled probe and plug-in program to evaluate the corresponding database efficiency index. The client application program is also responsible for packaging the evaluation result into a message with a specified format, and then feeding back the result to the application service program by means of the communication module;

The invention provides an unmanned cluster system service efficiency evaluation system, which mainly comprises 9 large modules such as an index library, a plug-in library, evaluation engineering management, evaluation index system management, evaluation task management, data connection and data management, evaluation operation management, node management, visual result display and the like, wherein the functional modules form a schematic diagram as shown in figure 2; the specific functions of each module are as follows:

the index library is a warehouse for storing and managing evaluation indexes, 37 evaluation indexes are provided aiming at the functions, performances, stability and compatibility of the service capability of the unmanned system, and a verification method and corresponding plug-ins are designed according to each index.

The plug-in library is a repository in which plug-ins developed by the system run corresponding evaluation metrics are stored. Each plug-in corresponds to an evaluation index, which is a specific evaluation method of the index. The plug-ins are obtained by adopting a cross compiling mode, the same index can correspond to a plurality of plug-ins, and each plug-in can be realized in a plurality of concrete modes according to different architectures and the like.

The evaluation index system management module supports functions of managing index types and indexes, designating index evaluation rules, index evaluation commands and the like. Index type management is responsible for maintaining index type processing under an evaluation system and depends on index types and index libraries (also called evaluation dictionaries); the index management is responsible for maintaining index processing under an evaluation system, and is responsible for creating, deleting, modifying, inquiring and setting parameters of the index. The index evaluation rule is responsible for maintaining the index evaluation rule data, and supports index evaluation methods such as a weighted average method, an ADC efficiency analysis method, a hierarchical analysis method and the like. The weighted average method calculates the weight into the weight to average to represent the average level of the service capacity of the unmanned system, and the service capacities of different unmanned systems can be analyzed and compared from two aspects of transverse and longitudinal; the ADC efficiency model predicts the measurement of the degree that the system meets a set of specific task requirements under the specified condition, it is a function of the products of availability (Avail-availability), credibility (Dependability) and Capability (Capability), A represents the effectiveness vector, can be used for representing the probability that the unmanned cluster system is in different states at the moment of starting to execute the task; d represents a credibility matrix, wherein the numerical value in the matrix represents the probability that the system is in one state at the beginning moment and is transferred to the other state in the running process; c represents a capacity matrix in which the values represent some performance indicator achieved in the last possible state. The analytic hierarchy process decomposes and combines the system evaluation problems to form a plurality of evaluation elements, and different layers are divided according to different attributes to form a layer relationship which is governed layer by layer from top to bottom. The index evaluation command is responsible for issuing and maintaining index names under the index type, and depends on the index type and the perfection of an index library. The index system is a directed graph which describes the dependency relationship among indexes and is obtained by extracting various characteristics of the same type of evaluation objects layer by layer. The module supports visual creation and editing of four types of indexes of functionality, performance, stability and compatibility, supports new construction of an index system, supports editing of the index system, supports deletion of the index system, supports setting of an aggregation flow of the index system and supports deletion of the aggregation flow of the index system.

The evaluation engineering management module provides a uniform resource management platform for efficiency evaluation, can rapidly search and display key attributes of each resource, and can start a selected resource editing tool for editing. The method supports the operations of creating, editing, deleting and exporting the evaluation engineering, is used for maintaining the evaluation engineering information, and can load corresponding engineering configuration information by self when the engineering is created according to the data perfection degree of the evaluation system module and the related configuration is needed in the evaluation system module.

The evaluation task management module is responsible for creating, editing and deleting the evaluation task, creating an evaluation task template, and creating, deleting and comparing an evaluation instance. An evaluation task refers to a process of evaluating one or more related evaluation objects once by using a uniform evaluation scheme. The evaluation task is established by firstly setting an evaluation object and an evaluation grade, and then inputting data obtained by data preprocessing into an evaluation flow by configuring operator parameters in the evaluation flow. Generating various index data under the evaluation engineering depends on the perfection degree of the evaluation engineering module and the perfection degree of the evaluation index configuration, and the work can be completed after the evaluation engineering module and the evaluation system configuration module are perfected. And automatically loading the generated evaluation data into a visual pattern interface, and automatically loading the module after the evaluation data are generated.

The data connection and data management module is used for displaying various data index details under the evaluation engineering to the user, and comprises an evaluation object, an evaluation time, an evaluation person, an evaluation environment and various index values and weights of the index values under the evaluation object; finally, scoring, evaluation description and the like are carried out on the test objects according to various indexes, and an operator can export the evaluation result into a PDF document to be archived.

The evaluation operation management module is mainly used for maintaining the management of the system function authority and controlling the role authority management and the access authority of the system user, is a basic stone for the system operation, and the access authority and the data security of the subsequent modules basically depend on the module.

The node management is mainly to manage unmanned nodes in the system, including checking available nodes in the system, checking node states, deleting nodes and the like.

The visual result display module supports the comparison analysis, the sensitivity analysis function and the pattern display and report automatic generation function of the evaluation result. The pattern display is divided into a radar chart and a pie chart, and the radar chart can display the analysis and comparison conditions of multiple indexes in the same coordinate system. It is a graph consisting of a set of coordinates and a plurality of concentric circles. The radar graph analysis method is a method commonly used in comprehensive evaluation, is particularly suitable for global and overall evaluation of objects described by a multi-attribute architecture, and has good effect in the aspect of showing scores or performances of multiple dimensions. The difference of each evaluation index can be intuitively seen by means of the radar chart, and further, the evaluation index is better improved through analysis; the content displayed by the pie chart is consistent with the content displayed by the radar chart, the same data is displayed by adopting another pattern, and the same data is displayed from different angles, so that the weights and scores occupied by different index types under the same evaluation object can be better displayed, whether the performance emphasis of the measured object reaches an expected target or not can be known, whether optimization adjustment is needed or not, and the like;

The service efficiency evaluation system for the unmanned cluster system provided by the invention is specifically configured as shown in a schematic diagram in FIG. 3; the deployment mode realizes the idea that: the platform service end and the unmanned node in the tested environment together form a ROS system or a system compatible with ROS, the platform service end serves as a Master (server, master node and service end) of ROS, and the unmanned node serves as a client (slave node) of ROS. Under the condition, each unmanned node clearly knows the IP and the port of the central node of the cluster, so that the ROS message sent by the platform server can accurately reach the unmanned node cluster, and the ROS-based message subscription and release mechanism running on the unmanned node can timely collect the command related to the evaluation task sent by the platform server, and can also return the local evaluation result to the platform server (the platform server only needs to subscribe topics related to the evaluation result). In the deployment mode, communication between the unmanned node and the platform server in the tested environment is realized by a message subscription and release mechanism compatible with ROS, so that cross-machine communication is realized.

In this deployment mode, the data flow in the overall system is divided into the following steps:

1. The user performs corresponding operation on the Java Web front end, such as starting an evaluation test, and the operation of the user and corresponding parameters, such as index names, corresponding plug-in names, parameters and the like, are sent to a Java Web background program running on the platform server through an http protocol;

2. after receiving the user request, the Java Web background program running on the platform server analyzes the user request parameters such as the evaluation index, the corresponding plug-in name, the parameters and the like, then goes to a local plug-in library, places the plug-in library in a Web container, conveniently uses an http protocol to remotely pull the plug-in file, searches whether the corresponding plug-in exists in the plug-in library, and is consistent with the evaluated plug-in name. If all are normal, the web server address, port number and parameters required by obtaining the plug-in file are packaged into a user request message structure together with the user request parameters; if the user request parameter analysis fails or the plug-in the local plug-in library is inconsistent with the requested plug-in, returning corresponding error information to the Java Web front end;

the Java Web background program sends the packaged user request message to a local ROS server through the ROSB; the local ROS server has established websocket connection to the Jave Web background and keeps communication;

The ROS server publishes the user request information to the ROS cluster managed by the ROS server through a message subscription publishing mechanism of the ROS;

5. after receiving a user request message sent by an ROS server end, an ROS client on an unmanned node participating in the evaluation at the time judges whether an insert to be pulled exists in the index verification program set at the time or not according to an index name, an insert name, a Web server address, a port number, framework information of a local machine and the like in a user message structure body, and if the insert to be pulled exists, the next step is directly carried out; otherwise, splicing to generate a plug-in URL applicable to the local index verification, and then pulling a corresponding plug-in from a platform server through an http protocol, and storing the plug-in the index verification program set;

loading and starting a corresponding index verification program by the ROS client according to the user request message;

7. the index verification program is a specific implementation of an index verification method, after the index verification method is operated, interaction is carried out with service information of the unmanned system according to specific logic, and when the interaction is carried out, the service information is not only local service information but also possibly service information on other unmanned nodes;

8. the index verification program receives the result of the unmanned system service data and carries out corresponding logic processing, such as time cost calculation, throughput rate calculation and the like;

9. The index verification program collects the results of the index verification program and encapsulates the results into a user response message structure;

10. the unmanned node publishes the user response message to the ROS cluster through a message subscription and publishing mechanism of the ROS;

11. the ROS server side of the platform server side receives the user response messages sent by the unmanned nodes, then analyzes and aggregates the user response messages, and sends the results to the Java web background program through the ROSB;

after further analyzing the result, the Java Web background program stores the result into a local MySQL service database;

and (3) refreshing the page by the Java Web front end, and acquiring and displaying an updated result from the platform server database.

An embodiment is taken as an example to describe how an unmanned cluster system service performance evaluation system provided by the invention organizes and manages evaluation nodes, evaluation environments and evaluation processes in terms of data service capability evaluation and displays and evaluates data service capability aiming at evaluation data, in this embodiment, the data service capability required by 3 unmanned aerial vehicles to execute search tasks in a semi-physical environment is taken as an example, a specific evaluation flow diagram is shown in fig. 4, and core indexes and verification methods thereof required by evaluating the tasks are determined according to current key nodes, key steps and task types which are concerned by executing the tasks. Based on the system, an evaluation project is created, an index set is determined according to demand analysis, an index library is created in an evaluation index system management module, corresponding indexes are selected, and a proper plug-in is selected according to a method of the required verification indexes. And checking a node list maintained by the current platform, selecting nodes required to participate in the current evaluation, and configuring an evaluation environment. After the deployment of the evaluation environment is completed, a connection relation between the system and each evaluation node is established through a data connection and data management module, an index evaluation command is issued to automatically distribute indexes and plug-ins to each node, an evaluation rule suitable for the requirement of executing the task at the present time is selected in an evaluation index system management module, then an evaluation project is selected in an evaluation task management module, an evaluation task is executed, the system generates data interaction according to the selected plug-ins and unmanned aerial vehicles, and returns result information to the rear end of the system through webSocket, and a system background application analyzes the read returned data and generates an evaluation report list and an index ratio diagram, and an evaluation result is obtained in a visual result display module and is analyzed and evaluated according to the result.

1. Unmanned aerial vehicle group collaborative search task planning

The unmanned aerial vehicle group collaborative search task is supposed to be completed by 3 unmanned aerial vehicles in a collaborative way, a 3-task version is adopted to deploy unmanned aerial vehicle systems and node service evaluation systems to represent three unmanned aerial vehicles, and the unmanned aerial vehicles execute the following tasks: the unmanned aerial vehicle is simulated to take off and then is assembled in a certain area, a search task is carried out in a certain area after the unmanned aerial vehicle is assembled, a suspicious target is found and then reported, and finally landing is returned. Because the semi-physical task board cannot take off, the data are acquired from the simulation environment, and the unmanned system in the task board receives the transmitted data of the ROSBag data packet after being started, so that the data problem is solved. The flow of the unmanned aerial vehicle cluster executing task is controlled by a task state machine which is planned in advance, the state switching needs to meet certain conditions, and the unmanned aerial vehicle cluster autonomously judges. One exemplary task state machine for a drone cluster to perform search tasks is shown in fig. 5, and includes task states such as ready, take off, maneuver, search, return, and landing. The mobile and search clusters are in a leader/follower mode, namely one leader and the rest are followers, and the leader is used as the brain of the cluster to control the cluster.

The preparation state can be switched to the take-off state, the take-off state can be switched to the maneuvering state and the return state, the maneuvering state can be switched to the searching state and the return state, the searching state can be switched to the tracking state and the return state, the tracking state can be switched to the searching state and the return state, and the return state can be switched to the landing state. The task state machine is characterized in that any state after the unmanned aerial vehicle takes off can be switched to a return state according to the requirement. The purpose of setting like this is to guarantee as far as possible that unmanned aerial vehicle cluster also can safe return to the voyage once there is unexpected condition to take place.

2. Unmanned aerial vehicle group collaborative search task demand analysis and design

In the collaborative task idea, the data service capability bears the data management function of the application, and mainly comprises local data storage, remote data sharing, local data query, remote data query and the like. In the whole cooperative task application execution process, the data service capability requirement of the unmanned system mainly comprises the following aspects.

(1) The storage aspect is as follows: support create file (table file, data file) operations, write data, update data, create index, write index, record update, whether transactions are supported;

(2) Query aspect: supporting data retrieval of cross nodes, including conditional retrieval under time constraint and space constraint, and concurrent query;

(3) System expansibility aspect: the system has automatic scalability, supports the access of unmanned nodes along with the meeting and abnormal or normal exit, and does not influence the normal work of data service;

(4) Resource consumption aspects: excessive CPU, memory and network resources are not occupied;

(5) Real-time aspect: the method can respond to the user request quickly, and has small delay and response time.

For the above aspects, the present concept selects several indicators representative thereof to verify the ability of the data service capability to meet the above requirements, and mainly includes:

(1) Perceptual data index QPS: transmitting a data query request for a single file (table) to unmanned node equipment, and measuring the number of data service processing queries per second (1 s);

(2) DDL throughput index: performing the average response time required for create, delete, alter operations on the unmanned node device database sheet table;

(3) The unmanned node monitors the state response index online in real time: acquiring the time spent by the current online node equipment data;

(4) Unmanned system data service response time index: sending a processing request for inquiring, updating and deleting the list table operation to node equipment, and processing the average processing time required by one request;

(5) Unmanned system data service concurrency rate index: concurrency = QPS response time;

(6) Unmanned system data service hotspot row update index: adopting 5 threads, adding about 2M file resources to node equipment at the same time, and updating the time spent by corresponding rows of a database at the same time;

(7) Unmanned system data service throughput rate index: within 1 second, sending a query list table request and file reading to node equipment, and successfully delivering the data size;

(8) Unmanned system data service resource utilization rate and occupancy rate index: executing 5-table joint query sentences with fields not less than 20 to node equipment, and simultaneously executing the writing or reading of file resources with the size of more than 2M, wherein the utilization rate and occupancy rate of the resources of the current node equipment are the same as those of the current node equipment;

(9) Unmanned system data service stability index: within 2 seconds, sending inquiry, updating and deleting list table operation and reading files above 2M to node equipment, and successfully executing the completion rate;

(10) Transaction capability index in unmanned system data service: inquiring whether the node equipment has the capability of processing the transaction;

(11) DDS delay evaluation performance index: the DDS delay refers to the time when one node sends a request to another node or a plurality of nodes and receives a reply, wherein the time comprises average response time, minimum response time and maximum response time;

(12) System stability assessment index: DDS communication stability and database stability, DDS communication stability increases and decreases the node through the cluster, DDS communication delay offset scope.

In the aspect of comprehensive evaluation, an analytic hierarchy process weighting model is considered, wherein the analytic hierarchy process weighting model is a method for dividing an evaluation target into a plurality of layers and a plurality of indexes and carrying out comprehensive evaluation according to different weights.

3. System architecture deployment

Semi-physical verification adopts an unmanned aerial vehicle development board to replace a real unmanned aerial vehicle to carry out verification experiments, and has the capability of highly restoring real scene conditions.

The ROS system, the DDS module, the index verification program set and the SQLite database are installed on the development board, and data collected in the verification process in the simulation environment are loaded. Two servers are deployed in the same network environment of the development board and respectively serve as a transit server and a web application server. The transfer server is used for deploying ROSBridge, ros master and Fast DDS modules and is used for transferring Java Web and messages among development boards. The Web application server deploys Java Web, mysql, redis, and Java file services.

By accessing Java Web application project address, logging in an unmanned system service effectiveness evaluation index verification prototype system, configuring, creating, executing, evaluating, analyzing and managing data service evaluation on an development board.

The semi-physical verification environment mainly comprises 3 development boards, 2 servers and 1 router, and specific parameters are shown in table 1; the devices in the verification environment are connected in a wired connection mode, and the network topology is shown in fig. 6.

Table 1 collaborative search application semi-physical verification environment device inventory

4. Test procedure and run

1) Entering a system: and inputting a prototype system login address, entering a login interface, inputting an authorized account number and a password into the system in the client PC browser.

2) Configuration indexes: and selecting an evaluation dictionary from a menu list of the system to enter a relevant function of configuration indexes, displaying the index information configured by the current system in a list form, and displaying the sequence number, the index type name, the sequence number, remarks, operations and the like of the indexes.

3) Increasing evaluation indexes: a new button is arranged at the upper right of the index type main list, a data entry popup window is opened, and information such as index type names, sequence numbers, remarks and the like is sequentially entered in the popup window according to form prompt information; clicking the "ok" button in the pop-up window closes the data entry pop-up window. The method comprises the steps of configuring 3 items of functional indexes, performance indexes and safety indexes.

4) The new index plug-in is added: opening an evaluation dictionary menu in a horizontal menu of the system, sequentially selecting index configuration-index evaluation command menu buttons in a vertical menu at the left side of the page, and entering an index evaluation command list page. Clicking a new button on the upper right of the main list in the page, opening a data entry popup window, entering information in the popup window according to the header on the left side of the list, and submitting an evaluation program (plug-in) for verifying the new index, wherein the program can have different versions (the verification modes of applying the same index by different ZZ can be different, and the programs of development boards of different architectures can be incompatible and can cause the same evaluation program to need different codes); clicking a 'confirm' button in the popup window, closing the data entry popup window, displaying newly added index list information on the main page, and configuring 10 performance indexes and 2 functional indexes in the present example.

5) Configuration evaluation environment: opening an evaluation dictionary menu in the transverse menu, sequentially selecting an evaluation environment configuration menu button, an evaluation environment type menu button and an evaluation environment type menu button in a longitudinal menu on the left side of the page, entering an evaluation environment configuration interface, and displaying configuration information in a list form. Clicking a new button at the upper right of the main list, opening a data entry popup window, and entering information such as environment type names, sorting numbers, remarks and the like in the popup window according to the header at the left side of the form in a corresponding input box; clicking a 'confirm' button in the popup window, closing the data input popup window, and prompting successful interface feedback operation. Sequentially selecting a menu button of 'evaluation environment configuration', 'evaluation environment' from a longitudinal menu on the left side of the page; entering an evaluation environment main interface, displaying the current evaluation environment in a list form, clicking a newly added button at the upper right of the main list, opening a data input popup window, and selecting information such as environment types, input environment names, environment values, sequence numbers, remarks and the like in the popup window according to the gauge heads at the left side of the form in the corresponding input frame; clicking a 'confirm' button in the popup window, closing the data input popup window, and prompting successful interface feedback operation.

6) Configuring the corresponding score of the index result: opening an evaluation dictionary menu in the transverse menu, and selecting an evaluation environment index configuration menu button in a longitudinal menu on the left side of the page; displaying the configuration information of the current evaluation environmental index in a list form, including information such as configuration name, environment, index name, configuration value, sequence number and the like, clicking a new button at the upper right of the main list, opening a data entry popup window, and entering information in a corresponding input frame according to the header at the left side of the form in the popup window. After the index of the list row in the popup window is selected to be pulled down, the system automatically generates a data entry form according to the lower part of the popup window, and displays the data range information of the index, including the minimum value, the maximum value, the units, the range preview information, the score, the grade, the remarks, the operation and the like. The operator can input the corresponding parameter information in the table according to the header prompt, and if one row needs to be added, click "+" under the operation column "

Number, adding one row, if deleting, clicking the "-" number, removing the selected row information; clicking on "confirm" in a popup window "

And the button is used for closing the data input popup window, and the interface feedback operation is successfully prompted.

7) Configuration evaluation system: and opening an evaluation system menu in the transverse menu, wherein the system automatically loads corresponding system main list data in a main interface, and the list displays information such as system names, configured index numbers, system accessories, sequence numbers, remarks and the like. Clicking a new button at the upper right corner of the list, entering/selecting corresponding data information in an entry frame according to an operation prompt of a first column of the list in a pop-up window, when selecting an option of 'DDS delay' after 'index configuration' line, loading detailed information of the selected 'DDS delay' in a 'index configuration' list below the list, clicking a removal button under the list operation column, removing the data detail in the list, and selecting a drop-down option after 'index configuration' line in the list to cancel the option of 'DDS delay'. Clicking on "ok" in pop-up window

And a 'button' is confirmed, the recorded 'evaluation system' data is saved, and the recorded data information is automatically loaded in the main list.

8) Creating an evaluation project: and opening a transverse menu 'evaluation project', loading all evaluation project list data in a list form on the main interface, and displaying information such as project names, clusters, project description information, sequence numbers, current states and the like. Clicking a new button at the upper left of the interface list, opening a data entry popup window, and entering data information such as project names, whether clusters are selected, system data information is selected, project description and the like according to form prompts; after the 'system' row is selected, loading corresponding index information from an index configuration list below the form; clicking the 'confirm' button to save the engineering information.

9) Starting an evaluation task: opening a transverse menu task evaluation button, and loading engineering information added in the 8 th operation step by a main interface; clicking a start button after the engineering information line created in the 8 th operation step, and opening a bullet window of the selection evaluating equipment; the device to be evaluated is checked, in this embodiment, 3 devices are selected, and clicked

The "confirm" button, the system begins to evaluate the project for an index. The index can monitor the index evaluation result in real time in the evaluation process, and click the data with the evaluation completion and the state of 'in progress'; the operation interface opens and evaluates a new message popup window; the bullet window is loaded with each index evaluation result information under the selected equipment, and the index evaluation result information comprises serial number, equipment information, index name and state information (the evaluation index has data return, the returned value is prompted by green, and the evaluation index number is calculated)

And if not returned, the gray prompt is not finished) when all the evaluation indexes are green, the index test is completed. 10 Viewing the evaluation result: opening a transverse menu data processing button, loading the tested engineering information click on the left side of the main interface, and displaying the engineering information click in a tree line structure; selecting the evaluation project at the left side, and loading index configuration configured under the selected project in the middle of a main interface; the right side of the main interface displays the evaluation values of the selected equipment to the configured indexes under the current engineering respectively, wherein the evaluation values comprise evaluation engineering information, test object information, test time information, tester information, information (CPU, kernel, bandwidth and the like) of each evaluation environment and actual evaluation conditions of each index in each equipment, and displays the actual result data, score information, occupied weight information and grade information of the index in the equipment, so that the evaluation personnel can check conveniently. Meanwhile, the lower half part of the report displays the evaluation index information by a radar chart and a pie chart respectively, and finally displays the overall evaluation information of the evaluation project.

11 Save and view the evaluation result: and opening a 'derive PDF' button on a page in the step 10, and exporting all indexes and equipment evaluation values under the selected engineering into a PDF file, so that the evaluation personnel can conveniently and durably store the evaluation result information. The user clicks the exported file, opens the PDF file, and checks whether the file content is complete.

12 Analysis of test results: from the results, the performance of the data service capability of the unmanned system can reach 290 minutes (300 minutes in full) under the condition that the ground does not fly practically, which shows that the unmanned system is well behaved in a static environment. The main reasons for the occurrence of such results may be that under static environment, the consumption of system resources is relatively small, and the CPU, network, memory and the like of the system are free, so that the data requirements of the performance evaluation prototype system can be processed timely and efficiently.

If each onboard development board is mounted on an unmanned aerial vehicle and the unmanned aerial vehicle group performs a test during the movement, the effect may not change little, that is, the test result is also affected by the environment.

In the setting process of the evaluation system, the weights of the indexes are assigned according to historical experience, and meanwhile, the returned values of the indexes are assigned in a segmented mode. The rationality of the weights, the rationality of the segments, etc. all have an impact on the results. Therefore, the result of the evaluation cannot represent the final conclusion, but a large number of test and verification results are required, and the application developer, the data service research and development team and the service efficiency evaluation prototype system together analyze the objective data obtained by the evaluation, so as to obtain the final conclusion, and the defects and shortcomings of each system, so as to promote the iterative development of the data service function and the efficiency evaluation system of the application and unmanned system.

According to the invention, by introducing the B/S+C/S hybrid architecture, the rapid implementation, deployment and system display of the efficiency evaluation system are facilitated, the remote end of a user can conveniently collect and display evaluation data in real time, the operation and remote control of the user are facilitated, the interconnection and intercommunication of unmanned system data and service between clusters can be realized by a system deployment mode suitable for a cluster environment, the transmission efficiency is improved, the service capacity is improved, 37 evaluation indexes aiming at the unmanned system data service evaluation are provided, the aspects of functions, performances, stability, compatibility and the like are covered, the characteristics of unmanned system data service reliability, data communication adaptability, data updating instantaneity, unmanned node topological structure change dynamics and the like are embodied, the user can select, call and evaluate the evaluation data in the form of index plug-in, the evaluation capacity and efficiency level of the evaluation system are improved, the evaluation and management problems of various service capacities in the unmanned system under the condition of the cluster can be effectively solved, and the service capacity of the unmanned system is improved.

Example 2:

based on the same inventive concept, the invention also provides a service efficiency evaluation method for the unmanned cluster system, wherein a flow diagram is shown in fig. 7, and the method comprises the following steps:

Step 1: resolving the acquired user request parameters to determine user request information;

step 2: based on the stored node address information and node state information of the unmanned node, sending the user request information to the corresponding unmanned node;

step 3: and outputting a user response result based on the received evaluation response information sent by the unmanned node.

Specifically, step 1 includes:

The generating user request information based on the pre-stored index library and the plug-in library according to the index query request and the plug-in query request comprises the following steps:

Step 3, including:

Example 3:

based on the same inventive concept, the invention also provides a service efficiency evaluation method for the unmanned cluster system, wherein a flow chart is shown in fig. 8, and the method comprises the following steps:

step S1: receiving user request information;

step S2: performing efficiency evaluation according to the user request information, and outputting evaluation response information;

Specifically, step S2 includes:

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the scope of protection thereof, and although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that various changes, modifications or equivalents may be made to the specific embodiments of the application after reading the present invention, and these changes, modifications or equivalents are within the scope of protection of the claims appended hereto.

Claims

1. An unmanned cluster system-oriented service performance evaluation system, comprising:

the unmanned nodes are in communication connection through a server;

2. The system of claim 1, wherein the server side comprises:

the Web server is in communication connection with the Web front end;

the file server is used for storing preset evaluation indexes and plug-ins;

The database server is used for storing the user response information;

3. The system of claim 2, wherein the application server comprises:

4. The system of claim 2, wherein the Web server comprises:

5. The system of claim 4, wherein the evaluation task management module generates user request information as an evaluation task to be uploaded to an application server according to the index query request and the plug-in query request based on the evaluation index and the plug-in stored in the file server, and includes:

6. The system of claim 5, wherein the unmanned node comprises:

the unmanned system client is in communication connection with the server;

7. The system of claim 1, wherein the Web front-end comprises:

8. The service effectiveness evaluation method for the unmanned cluster system is characterized by comprising the following steps of:

9. The method of claim 8, wherein the parsing the obtained user request parameters to determine user request information comprises:

10. The method of claim 9, wherein generating user request information based on the pre-stored index library and plug-in library and according to the index query request and plug-in query request comprises:

11. The method of claim 8, wherein outputting a user response result based on the received evaluation response information transmitted by the unmanned node comprises:

12. The service effectiveness evaluation method for the unmanned cluster system is characterized by comprising the following steps of:

receiving user request information;

wherein the user request information is the user request information of any one of claims 8 to 11.

13. The method of claim 12, wherein performing performance assessment based on the user request information, outputting assessment response information, comprises: