CN114154611B

CN114154611B - Man-machine confrontation system supporting Turing test mode and intelligent agent test method

Info

Publication number: CN114154611B
Application number: CN202111328333.3A
Authority: CN
Inventors: 倪晚成; 徐佳乐; 王士贤; 黄凯奇; 杨旭阳
Original assignee: Institute of Automation of Chinese Academy of Science
Current assignee: Institute of Automation of Chinese Academy of Science
Priority date: 2021-11-10
Filing date: 2021-11-10
Publication date: 2022-12-06
Anticipated expiration: 2041-11-10
Also published as: CN114154611A

Abstract

The invention provides a man-machine confrontation system supporting a Turing test mode and an intelligent agent test method, wherein the system comprises: the man-machine confrontation module is used for finishing man-machine confrontation; wherein, human-computer confrontation is completed by human confrontants and intelligent agents in a double-blind environment; the confrontation data acquisition module is used for acquiring a man-machine confrontation result and countercheck confrontation data of man-machine confrontation; the confrontation data analysis module is used for acquiring the efficiency data of the intelligent agent according to the counteroffer data; the intelligent agent comprises a smart questionnaire module, a database module and a database module, wherein the smart questionnaire module is used for acquiring external evaluation information of the intelligent agent in a smart questionnaire form; and the confrontation result management module is used for receiving and storing the man-machine confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent, and acquiring the capability test result of the intelligent agent according to the man-machine confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent. The invention realizes the comprehensive and effective test of the decision-making capability of the intelligent agent.

Description

Man-machine confrontation system supporting Turing test mode and intelligent agent test method

Technical Field

The invention relates to the technical field of man-machine confrontation, in particular to a man-machine confrontation system supporting a Turing test mode and an intelligent agent test method.

Background

Artificial intelligence technology is undergoing the evolution from perceptual intelligence to cognitive intelligence, and is working on machines developing from being able to understand data to being able to understand the real world, and from being able to hear, say, see, develop as being able to think like humans. The perception intelligence usually takes machine learning as a technical route, the feature representation which can be obviously fitted with an expected target is learned from a large-scale data set, the final output of the feature representation is usually instant and clear, and the feature representation can be matched and compared with known standard answers, so that the quantitative evaluation of indexes such as accuracy and error values is convenient to use. In the field of cognitive intelligence, an agent is often in various flexible application scenes, and a certain action purpose is achieved through a continuous cognitive decision process of perception-judgment-decision-action. The cognitive decision process is continuous and dynamically changed, uncertainty is full, and sometimes it is difficult to obtain a Reward (Reward) of a short-term decision behavior, a traditional test method usually defines a final objective index (such as an arbitration score) for evaluation, but the method ignores the evaluation of the decision of the middle process of the intelligent body, cannot comprehensively reflect the capability level of the intelligent body in all aspects, and is easy to bring a profit "drilling blank" behavior which is dominated by improving the score, so that the accuracy of the decision capability test result of the intelligent body cannot be ensured. Therefore, how to comprehensively and effectively test the decision-making capability of the cognitive and decision-making type intelligent agents is an urgent problem in the cognitive intelligence era.

Allen-turing has proposed a "turing test" in 1950, which is a test to determine whether a machine has human intelligence, to determine whether a machine has intelligence by countermeasures and analogies. "Turing test" defines: in the case where the tester is separated from the testee (one person and one machine), the testee is asked questions at will by some means. After multiple tests, a machine is considered to have human intelligence if it has more than 30% false positives on average per participant.

However, there is no man-machine countermeasure system supporting the turing test mode in the prior art, and there is a need to provide a man-machine countermeasure system supporting the turing test mode to obtain performance test data of the intelligent agent and to comprehensively and effectively test the cognitive and decision-making capabilities of the intelligent agent according to the test data.

Disclosure of Invention

The invention provides a man-machine confrontation system supporting a Turing test mode and an intelligent agent test method, which are used for solving the defect that the capability level of an intelligent agent in each aspect cannot be reflected in all aspects in the prior art and realizing comprehensive and effective test on the cognitive and decision-making capability of the intelligent agent.

The invention provides a man-machine confrontation system supporting a Turing test mode, which comprises:

the man-machine confrontation module is used for finishing man-machine confrontation; wherein the human-machine confrontation is accomplished by a human confronter and an agent in a double-blind environment;

the confrontation data acquisition module is used for acquiring a man-machine confrontation result and countermark confrontation data of man-machine confrontation;

the confrontation data analysis module is used for acquiring the efficiency data of the intelligent agent according to the counteroffer data of the multi-disk;

the intelligent agent comprises a smart questionnaire module, a database module and a database module, wherein the smart questionnaire module is used for acquiring external evaluation information of the intelligent agent in a smart questionnaire form;

and the confrontation result management module is used for receiving and storing the man-machine confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent, and acquiring the capability test result of the intelligent agent according to the man-machine confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent.

According to the man-machine confrontation system supporting the Turing test mode, the man-machine confrontation module comprises:

the confrontation course generating module is used for generating a confrontation circular course and coding the system codes of the human confronters and the intelligent agents; wherein encoding system codes of the human competitor and the agent is used to create the double-blind environment;

the confrontation scheduling management module is used for automatically scheduling the man-machine confrontation process according to the circular race course;

and the man-machine confrontation function module is used for completing the man-machine confrontation in a simulation environment through the human confronters and the intelligent agents under the process scheduling of the man-machine confrontation.

According to the man-machine confrontation system supporting the Turing test mode, the man-machine confrontation function module comprises:

a confrontational environment simulation module for generating the simulated environment;

the engine kernel module is used for receiving action instructions sent by the human competitors and the intelligent agent, updating the environment state and the battlefield situation of the simulation environment according to the action instructions and generating real-time deduction data;

the UI module is used for analyzing and displaying the real-time deduction data to the human competitor, sending an action instruction sent by the human competitor to the engine kernel module and filtering identity information of the human competitor;

and the AI module is used for analyzing the real-time deduction data to the intelligent agent, sending the action instruction sent by the intelligent agent to the engine kernel module, and filtering the identity information of the intelligent agent.

According to the man-machine confrontation system supporting the Turing test mode, the confrontation data analysis module obtains the efficiency data of the intelligent agent according to the countercheck data of the multi-disk, and the efficiency data comprises the following steps:

according to the countercheck data, acquiring operation data of the intelligent agent in the man-machine countercheck process, and respectively acquiring performance data of the intelligent agent from a plurality of different aspects according to the operation data; wherein the operational data of the agent comprises: own resources use class data, own strategy class data, enemy perception class data and human-computer distinguishing characteristic class data.

The man-machine confrontation system supporting the Turing test mode further comprises a confrontation and sightseeing module;

the confrontation data acquisition module is also used for acquiring real-time confrontation data in the man-machine confrontation process;

the confrontation and sightseeing module is used for analyzing the real-time confrontation data and/or the counteroffer data, and carrying out anonymous 2D or 3D confrontation graphic display and switching display of different battlefield situations according to the analysis result of the real-time confrontation data and/or the counteroffer data.

The invention also provides an intelligent agent testing method, which comprises the following steps:

acquiring a man-machine confrontation result and countercheck data of man-machine confrontation; wherein the human-machine confrontation is completed by a human confronter and an agent in a double-blind environment;

acquiring efficiency data of the intelligent agent according to the countermeasures data, and acquiring external evaluation information of the intelligent agent in the form of a Tuoling questionnaire;

and acquiring a capability test result of the intelligent agent according to the man-machine confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent.

According to the intelligent agent testing method provided by the invention, the human-computer confrontation completed by the human confronter and the intelligent agent in a double-blind environment comprises the following steps:

generating a cyclic course of confrontation and encoding the system codes of the human confronters and the agents; wherein encoding system codes of the human competitor and the agent is used to create the double-blind environment;

and automatically scheduling the process of the man-machine confrontation according to the circular competition process, and completing the man-machine confrontation in a simulation environment through the human confronters and the intelligent agents under the scheduling of the process of the man-machine confrontation.

According to the invention, the intelligent agent testing method is provided, wherein the human-computer confrontation completed by the human confronter and the intelligent agent in a simulated environment comprises the following steps:

an environment generation step: generating the simulation environment;

a countermeasure step: receiving action instructions sent by the human opponents and the intelligent agents, updating the environment state and the battlefield situation of the simulation environment according to the action instructions, and generating real-time deduction data;

circulating the confrontation steps until the man-machine confrontation is completed;

wherein, before the countermeasure step, the method further comprises: filtering identity information of the human competitor and the agent.

According to the intelligent agent testing method provided by the invention, the step of acquiring the efficiency data of the intelligent agent according to the countercheck data comprises the following steps:

according to the countercheck data, acquiring operation data of the intelligent agent in the man-machine countercheck process, and respectively acquiring performance data of the intelligent agent from a plurality of different aspects according to the operation data; wherein the operational data of the agent comprises: the own resource application class data, the own strategy class data, the enemy perception class data and the man-machine distinguishing characteristic class data.

According to the intelligent agent testing method provided by the invention, in the process of the man-machine confrontation and/or after the man-machine confrontation is completed, the intelligent agent testing method further comprises the following steps:

acquiring real-time confrontation data in the man-machine confrontation process;

and analyzing the real-time countermeasure data and/or the countersignature data, and performing anonymous 2D or 3D countermeasure graphic display and switching display of different battlefield situations according to the analysis result of the real-time countermeasure data and/or the countersignature data.

According to the man-machine confrontation system supporting the Turing test mode and the intelligent body test method, the Turing questionnaire module is creatively introduced into the man-machine confrontation system, the outside evaluation information of the intelligent body is acquired through the Turing questionnaire module, the effectiveness data of the intelligent body is obtained through the analysis of confrontation data in the man-machine confrontation process, the decision-making capability of the intelligent body is tested by combining the man-machine confrontation result, the outside evaluation information of the intelligent body and the effectiveness data of the intelligent body, and the comprehensive and effective test of the cognition and decision-making capability of the intelligent body is realized; meanwhile, the man-machine confrontation process is finished in a double-blind environment, so that the fairness of data in the cognitive and decision-making ability test process of the intelligent agent is further ensured, and the accuracy of the decision-making ability test result of the intelligent agent is improved.

Drawings

In order to more clearly illustrate the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a schematic structural diagram of a human-machine confrontation system supporting Turing test mode according to the present invention;

FIG. 2 is a schematic flow diagram of the method for testing an agent according to the present invention;

fig. 3 is a schematic structural diagram of an electronic device provided in the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The man-machine confrontation system supporting the turing test mode of the invention is described below with reference to fig. 1, and the man-machine confrontation system supporting the turing test mode is shown in fig. 1 and comprises:

a man-machine confrontation module 710 for completing man-machine confrontation; wherein, human-computer confrontation is completed by human confronters and intelligent agents in a double-blind environment;

the confrontation data acquisition module 720 is used for acquiring a man-machine confrontation result and counteroffer data of man-machine confrontation;

the confrontation data analysis module 730 is used for acquiring the efficiency data of the intelligent agent according to the counteroffer data of the multiple disks;

the Turing questionnaire module 740 is used for acquiring external evaluation information of the intelligent agent in the form of Turing questionnaire; that is, the turing questionnaire module 740 collects the guessing and evaluation information of the external world to the agent in the man-machine interaction mode of the electronic questionnaire;

and the countermeasure result management module 750 is used for receiving and storing the human-computer countermeasure result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent, and acquiring the capability test result of the intelligent agent according to the human-computer countermeasure result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent.

As an alternative, the man-machine confrontation system supporting the Turing test mode further comprises a confrontation and sightseeing module; the confrontation data acquisition module 720 is further used for acquiring real-time confrontation data in the man-machine confrontation process; the confrontation and sightseeing module is used for analyzing the real-time confrontation data and/or the counteroffer data, and carrying out anonymous 2D or 3D confrontation graphic display and switching display of different battlefield situations according to the analysis result of the real-time confrontation data and/or the counteroffer data. The module is mainly characterized in that the identity information of both parties of the battle is screened, the real man-machine condition and the user information of the confrontation players cannot be known during the sightseeing, and only the anonymous system code of the confrontation players can be known. The confrontation and sightseeing module comprises a real-time sightseeing unit and a re-disk sightseeing unit; the real-time fighting observation unit is used for analyzing real-time fighting data in the man-machine fighting process and providing a real-time graphic display interface according to the fighting process; the countersignature watching unit is used for analyzing countersignature data of the man-machine countersignature and providing a graphic display interface after the countersignature is finished; the double-disk fighting unit also has flexible progress bar dragging and speed multiplying playing functions.

In the present invention, the human-machine confrontation module 710 includes:

the confrontation course generating module 711 is used for generating a confrontation cyclic course and realizing the human-computer mixed double-blind cyclic confrontation competition system through the generated cyclic course; in this embodiment, in the generated cyclic course, the contestants compete in turn with each other to ensure the fairness and the rationality of the competition scheme. During the generation process of the cycle course, cycle confrontation competitions in different modes are formulated according to the number of players, for example, when the number of players is small, a single cycle competition system or a double cycle competition system is adopted; when the number of players is excessive, a grouping and circulating competition system is adopted. After the cycle course is formulated, the confrontation course generating module 711 is further configured to encode the system codes of the human confronters and the intelligent agents, for example, to uniformly name the human confronters and the intelligent agents; through the coding, only the system knows the identity of the competitor, different competitors cannot mutually know the identity information of the other party, and the spectator cannot know the identity information of the competitor. There are various methods for unifying naming, for example, m1 human competitors and m2 agents participating in human-computer competition are mixed by the competition course generation module 711, and anonymous system codes "1 to (m 1+ m 2) players" are respectively assigned, so that the front-end interface can only display the system codes of the players.

The confrontation scheduling management module 712 is configured to automatically schedule a human-computer confrontation process according to a cyclic course, that is, the confrontation scheduling management module 712 automatically generates a plurality of confrontation rooms according to the course arrangement, and schedules players to reach the operation interfaces of corresponding seats of the corresponding rooms, so as to enter the confrontation process, and implement conventional operations of selecting scenes, opponents, seats and the like without intervention of confrontation personnel, thereby effectively preventing the players from obtaining some operation information of the opponents in the course operation, assisting in judging the identity of the opponents, ensuring conditions of double-blind confrontation, and further improving the accuracy and effectiveness of the external evaluation information of the intelligent objects collected by the turing questionnaire module 740. For example, in a chess deduction system, the form of the confrontation is a game confrontation which must decide two seats of red and blue, so as to plan for a scene of summarizing the confrontation. Specific conditions for confrontation are pointed out, including map selection, combat types, positions of control points, force configuration and initial positions of two parties and the like, and some common chess deductions are given as follows: mountain land channel seizing and controlling war, water net paddy field war and medium-sized lodging land war, etc. For example, according to the course arrangement, if player No. 1 and player No. 2 are to develop a confrontation under the assumption of mountain channel racing, player No. 1 executes the red square and player No. 2 executes the blue square, the confrontation scheduling management module 712 generates a confrontation room under the assumption, and directly schedules player No. 1 to enter the operation interface of the red square seat in the room and player No. 2 to enter the operation interface of the blue square seat in the room.

The man-machine confrontation function module 713 is used for completing man-machine confrontation in a simulation environment through a human confronter and an intelligent agent under the process scheduling of the man-machine confrontation; the human-machine confrontation function module 713 is a basic module of the human-machine confrontation system supporting the turing test mode and is used for the human confronters and intelligent agents to develop cognitive and decision-making game confrontation in a simulation environment. The human-machine confrontation function module 713 includes:

a confrontational environment simulation module 714 for generating a simulated environment; for example, in a chess push system, the simulation environment is embodied as a simulation map area with different terrain and topography, and a plurality of confrontation environments can be operated in parallel in the chess push system.

The engine kernel module 715 is used for receiving action instructions sent by human opponents and intelligent agents, updating the environment state and battlefield situation of the simulation environment according to the action instructions, and generating real-time deduction data; the engine kernel module 715 is connected to a rule knowledge base and a decision knowledge base, and updates the environment state and battlefield situation of the simulation environment according to the rule knowledge base and the decision knowledge base, and calculates and generates real-time deduction data.

The UI module 716 is configured to perform interaction and information exchange between the confrontation environment and the human confronter, parse and display real-time deduction data to the human confronter, and send an action instruction sent by the human confronter to the engine kernel module 715; the UI module 716 displays the analyzed real-time deduction data through an image interface; meanwhile, the UI module 716 is further configured to filter identity information of the human confronter, so that the human confrontation function module 713 cannot acquire real identity information of the player, and anonymity of human confrontation is ensured. The UI module 716 includes a first countermeasure unit and a first environment unit, where the first countermeasure unit is configured to perform pre-battle deployment, action generation, and resource clearing, so as to implement a full countermeasure flow in which a human competitor deploys an action list before countermeasure, receives a battlefield situation and outputs an action instruction during countermeasure, and clears data resources after countermeasure; the first environment unit is used for operating the confrontation environment so as to realize the resetting, pushing, saving and restoring of the environment.

The AI module 717, configured to perform interaction and information exchange between the confrontation environment and the agent, analyze real-time deduction data for the agent, and send an action instruction sent by the agent to the engine kernel module 715; meanwhile, the AI module 717 is also used to filter the identity information of the agent, so that the man-machine confrontation function module 713 cannot obtain the real identity information of the player, and the anonymity of the man-machine confrontation is ensured. The AI module 717 comprises a second countermeasure unit and a second environment unit, wherein the second countermeasure unit is used for initializing, deploying before battle, generating actions and clearing resources, so as to realize the full countermeasure flow of initializing countermeasure basic information of the agent, deploying an action list before countermeasure, receiving battlefield situation and outputting action instructions during countermeasure and clearing resources such as data and models after countermeasure; the second environment unit is used for operating the countermeasure environment to realize the resetting, pushing, saving and restoring of the environment.

The UI module 716 and the AI module 717 adopt a mask technology to filter out the identity information of human competitors and agents.

The confrontation data analysis module 730 obtains the performance data of the agent according to the countermeasures data of the compound disk, including:

acquiring operation data of the intelligent agent in a man-machine countermeasure process according to the counteroffer data, and respectively acquiring efficiency data of the intelligent agent from a plurality of different aspects according to the operation data; wherein the several different aspects include: available confrontation resource application capacity, confrontation deployment and planning capacity, awareness and coping capacity for enemy actions and situation changes and intelligent agent remarkable characteristics; the operation data of the agents corresponding to each aspect are respectively as follows: the method comprises the following steps of self resource application class data (for example, in a military chess deduction system, the number of actions of an intelligent body for operating different operators and the number of different weapons in all attack actions of the intelligent body), self strategy class data (for example, in the military chess deduction system, the motion track and the heat degree of track positions of the operators operated by the intelligent body, the action number of the intelligent body for cooperatively attacking firepower among the operators), enemy perception class data (for example, in the military chess deduction system, the average time interval between the actions of the intelligent body and the actions of an opponent and the action number of the intelligent body in unit time), and human-machine resolution characteristic class data (for example, in the military chess deduction system, the number of illegal operations of the intelligent body).

The turing questionnaire module 740 includes a confrontation person questionnaire unit and a spectator person questionnaire unit; wherein, the confrontation personnel questionnaire unit collects guessing and evaluating information of the human confronters to the opponents through the UI module 716 in a man-machine interaction mode of an electronic questionnaire after each opponent is ended; the interviewer questionnaire unit collects guessing and evaluation information of the interviewer to the interviewer in a man-machine interaction mode of an electronic questionnaire through the interview module (the real identity information of the interviewer is shielded) after the interview of each opponent is finished. As an alternative, the design scheme of the turing questionnaire is as follows: the Turing questionnaire comprises guesses of the appraiser on the identity of the competitor and scores of various aspects of the competitor performance; guessing the opponent's identity is: guessing whether the competitor is a human or an agent; the various scores of the performance of the competitors are: within the standard score range of 0-P, the available confrontation resource application capacity, the confrontation deployment and planning capacity, the sensing and coping capacity for the action and situation change of the enemy and the obvious characteristics of the intelligent agent of the confrontation person are graded and evaluated.

The confrontation achievement management module 750 comprises a system adjudication score management unit, an efficiency data management unit, a turing questionnaire result management unit and a statistical unit; the system adjudication score management unit is used for storing a man-machine confrontation result, namely a system adjudication score, wherein the man-machine confrontation result comprises the following steps: attack score, seize score, remaining operator score, total score and net win score; the efficiency data management unit is used for storing the efficiency data of the intelligent agent; the picture questionnaire result management unit is used for storing external evaluation information of human confronters and intelligent agents; the statistical unit is used for performing statistical analysis on the human-computer confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent, and acquiring a capability test result of the intelligent agent according to the statistical analysis result.

As an alternative, the man-machine confrontation system supporting the turing test mode further comprises a confrontation database module, wherein the confrontation database module is used for storing and managing countervailing data of the compound disk; the countermeasure database module generates a file after each countermeasure node is ended, the file records the whole process of the whole countermeasure, the file comprises a plurality of frame data, each frame data comprises a plurality of fields and is divided into a basic field and an action field, the basic field records the basic information at the time step, and the action field records the action executing condition at the time step. For example, in a war game deduction system, the countercheck data is in the form of json files, and each game of countercheck is finished and then a json file is generated to record the whole process of the whole game of countercheck. The multi-disk confrontation data stores the deduction situation information according to the time sequence, one frame of data is recorded every 1 second, and each frame of data comprises a plurality of fields which are divided into a basic field and an action field. Basic fields record basic information under the time step, such as operator information, time information, control point seizing information, system arbitration scores and other fields; the action field records the condition of executing the action at the time step, and is related to specific actions, such as aiming point information, shooting arbitration and other fields. If no action occurs at this time step, the action field in the frame is empty, and only the base field is reserved.

The following describes the testing method of the smart agent provided by the present invention with reference to fig. 2, and the testing method of the smart agent described below and the above-described man-machine confrontation system supporting the turing test mode can be referred to correspondingly. The intelligent agent testing method is shown in fig. 2 and comprises the following steps:

s100, acquiring a man-machine confrontation result and multi-disk confrontation data of man-machine confrontation; wherein, human-computer confrontation is completed by human confronters and intelligent agents in a double-blind environment; in the step, the human-computer confrontation is completed by the human confronters and the intelligent agent in a double-blind environment, and the step comprises the following steps:

s110, generating a cyclic competition process of the confrontation, and coding system codes of human confronters and intelligent agents; wherein, the system code of the human confronter and the intelligent agent is coded for creating a double-blind environment; the method comprises the steps of determining human competitors and intelligent agents participating in man-machine competition before generating a cyclic competition process of the competition; in this embodiment, players whose ability levels are known through various events are selected and have different ability levels so as to be analogized as the ability of the agent; specifically, the selected human competitors have higher confrontation level and rich confrontation experience in the confrontation application scene, so that effective comparison evaluation with the intelligent agent can be carried out, and the confrontation application scene can be obtained by means of invitations of elite players. In this embodiment, the human opponent system sends invitations to m human elite players ahead according to the total score of the user's historical oppositions, thereby determining the human opponents participating in the human opponent. Meanwhile, the selected intelligent agents need to have higher deduction level so that the testing process of the intelligent agents is more challenging, and more comprehensive and valuable evaluation on the intelligent agents can be obtained. In this embodiment, the selection of the agents is performed by machine-to-machine confrontation, and it is assumed that the number of agents registered to participate in the evaluation is large, so that machine-to-machine confrontation in the two stages of the election race and the promotion race is adopted to obtain the agents who finally enter into the machine-to-machine confrontation. The method comprises the following specific steps:

s111, developing a grouped circulating competition of the mechanical countermeasures in a countermeasure environment with the initial configuration determined to be unchanged, and entering the promotion competition by N intelligent agents before selecting according to the integral;

in this embodiment, in the election competition stage of step S111, the mechanical countermeasure is performed under the initial assumption that is not changed by using the grouping cycle point competition system. In order to reduce the influence of the machine on the chance and the game asymmetry, a plurality of battles are specified between the same opponent, each battle has two hands, and after one hand is finished, the two hands change and exchange seats, for example, when the No. 1 player in the first game carries on the basketball side, the No. 2 player carries on the red side, and after the second game carries on the basketball side, the No. 1 player carries on the red side, and the No. 2 player carries on the basketball side. Therefore, each player and the opponent develop multiple countermeasures, and finally N agents before being drawn out are selected to enter the promotion competition of the second stage.

S112, a double-circulation match of the mechanical-mechanical countermeasures is developed in the initially configured and dynamically adjusted countermeasures environment, the former N intelligent body promotion Turing test man-machine countermeasures playground matches are selected according to the points, and N is smaller than N.

In this embodiment, in the promotion competition stage of step S112, a two-cycle point competition system is adopted to perform the mechanical competition under the dynamically adjusted online assumption. And the same opponent is used for fighting a plurality of fields, each field is used for fighting two rounds, and after one round is finished, the opponents exchange seats to carry out the confrontation. Finally, the top n strong promotion levels are selected to carry out the man-machine confrontation final match with the man-machine confronters. The opportunistic thought in this stage is to increase the emergency situation at random on the basis of the initial thought, such as randomly setting roadblocks, changing positions of robbing control points, changing initial positions of military forces, reinforcing the military forces in the confrontation process and the like. The arrangement of the contingency plan can provide flexible scene selection for the countermeasure implementation, increase the countermeasure difficulty without losing rationality, test the autonomous strain capacity of the intelligent agent more, and provide higher requirements for the research and development of the intelligent agent.

The process of generating the cyclic competition process of the confrontation comprises the following steps: the method comprises the following steps of (1) compiling a mixture of human competitors and agents without distinguishing identities into a large group, setting a competition scene through a competition course generation module 711, and generating a competition cycle course; for example, taking a war act deduction as an example: the specific scenes of the confrontation are summarized by imagination and divided into initial imagination and opportunistic imagination. The initial plan means that various configurations in each confrontation condition are fixed, and the opportunistic plan means that certain random configurations occur in each confrontation condition, such as adding a roadblock with random positions, changing positions of control-robbing points, and the like. In this embodiment, the form of the game is determined as a circular confrontation, and multiple rounds of confrontations are played, so that all players can play one confrontation with each other. Multiple games are simultaneously carried out in each round, two games are played between different players, each game adopts one game and two games, and after the game is finished, the two players change seats. Specific contents of the course include a confrontational scene, a confrontational round and an opponent arrangement for each round. In this embodiment, the system administrator sets the confrontation scene through the confrontation course generating module 711, for example, the confrontation scene can be set as mountain passage fighting envisaged by the opponent. The competition course generating module 711 then automatically determines the competition rounds and the arrangement of the opponents in each round according to the two-cycle competition system, and generates a round course of the competition.

There are various methods of encoding the system code numbers of human competitors and agents, for example, uniformly naming human competitors and agents; through coding, only the system can know the identity of the competitor, different competitors cannot mutually know the identity information of the other party, and the spectator cannot know the identity information of the competitor. There are various methods for unifying naming, for example, m1 human competitors and m2 agents participating in human-computer competition are mixed by the competition course generation module 711, and anonymous system codes "1 to (m 1+ m 2) players" are respectively assigned, so that the front-end interface can only display the system codes of the players.

And S120, automatically scheduling a man-machine confrontation process according to the circular competition process, and completing the man-machine confrontation in a simulated environment through the human confronters and the intelligent bodies under the scheduling of the man-machine confrontation process. In this step, under the condition that the previous work ensures the man-machine double-blind circulation, the confrontation scheduling management module 712 centrally schedules the whole confrontation process according to the arrangement of the race course. The confrontation scheduling management module 712 starts the engine kernel module 715 according to the specific conditions of the confrontation to generate a confrontation room meeting the conditions, matches the corresponding seats of all the players entering the corresponding room according to the course arrangement, and schedules the players to directly enter the confrontation process until all rounds of confrontation are finished.

Accomplishing human-machine confrontation in a simulated environment by human confrontants and agents includes:

s121, generating a simulation environment through the confrontation environment simulation module 714;

s122, receiving action instructions sent by human confronters and intelligent agents, updating the environment state and battlefield situation of the simulation environment according to the action instructions, generating real-time deduction data, and completing human-computer confrontation; the method specifically comprises the following steps: after receiving a signal that the preparation of the players of the two parties is finished, starting timing by the system and formally entering a confrontation state; both players sense and judge the current environment state and battlefield situation, and select in the selectable action set according to the sensing result to make action decision; the engine core module 715 receives the action command of the player, updates the environment status and battlefield situation of the simulation environment according to the action command, and feeds back the result to the user terminal through the UI module 716 and the AI module 717. Repeating the step S122 until meeting the confrontation ending condition, and ending the local confrontation; after the opposing competition of the local bureau ends, the opposing competition scheduling management module 712 schedules the exchanging seats of the players of both parties, enters the appointed competition process to carry out the hand-over competition, and repeats the steps S121 to S122 until the competition of one (two bureaus) is completed.

Wherein, before the man-machine confrontation begins, still include: filtering the identity information of the human competitor and the agent through the UI module 716 and the AI module 717, respectively, so that the competitor's identity information cannot be known to the contestants participating in the competition; the UI module 716 and AI module 717 employ masking techniques to filter out identity information of human competitors and agents.

S130, after the confrontation is finished, the man-machine confrontation result is transmitted to the confrontation result management module 750 through the engine kernel module 715, and meanwhile, the countermark data is transmitted to the confrontation database module for persistent storage. In this embodiment, after the confrontation is completed, the engine kernel module 715 sends the man-machine confrontation result, that is, the system adjudication score obtained by the player, to the confrontation result management module 750 for unified storage management, where the system adjudication score includes: attack score, seize score, remaining operator score, total score and net win score; the statistic unit in the confrontation achievement management module 750 ranks the achievements of human confronters and intelligent agents according to the system adjudication score, which specifically comprises the following steps: the statistic unit is used for counting the large scores and the small scores of the players in all the participating games according to the system adjudication scores obtained by the players in each game, and then ranking all the players according to the scores. Wherein, the large score is the number of the game players winning all the participated games, and the small score is the sum of the systematically adjudicated net wins of the game players in all the participated games.

S200, acquiring efficiency data of the intelligent agent according to the confrontation data of the real copy disc, and acquiring external evaluation information of the intelligent agent in the form of a Tuoling questionnaire;

in this step, obtaining performance data of the agent according to the countermeasures data includes:

the countercheck data stored in the countercheck database module is read by the countercheck data analysis module 730, the operation data of the intelligent body in the man-machine countercheck process is obtained according to the countercheck data, and the efficiency data of the intelligent body is respectively obtained from a plurality of different aspects according to the operation data; alternatively, the acquired operation data is mapped to a score range of 0 to P in degree as an evaluation score S of the player in the corresponding dimension _j 0 denotes the lowest level of player expression in the dimension, P denotes the highest level of player expression in the dimension, and P denotes>0. Several different aspects include: the system can be used for resisting resource application capacity, resisting deployment and planning capacity, sensing and coping capacity for enemy action and situation change and remarkable characteristics of an intelligent agent. The operation data of the agents corresponding to each aspect are respectively as follows: the own resource application class data, the own strategy class data, the enemy perception class data and the man-machine distinguishing characteristic class data.

The acquisition of performance data in different aspects comprises:

obtaining the evaluation score of the available countermeasure resource application capability of the intelligent agent according to the resource application class data of the intelligent agent; for example, in a war game deduction system, the intelligent agent can embody the comprehensive application capability of operators and the comprehensive application capability of weapons by using the antagonistic resource application capability; the comprehensive operator application capacity is obtained according to the number of actions of the intelligent agent for operating different operators, and the more balanced the action proportion among the different operators is, the stronger the comprehensive operator application capacity is; for example, the combat operators operated by players include various types such as infantry, chariot, tank, patrol missile, artillery and the like, the action quantity of different types of operators in the overall countermeasure is counted, the proportion of the action quantity is used as quantitative data, and the more balanced the proportion is, the stronger the comprehensive application capacity of the operators is; the comprehensive utilization capacity of the weapons is obtained according to the use number of different weapons in all the striking actions of the intelligent agent, and the more balanced the use proportion among the weapons is, the stronger the comprehensive utilization capacity of the weapons is; for example, the shooting weapons are divided into various types such as direct-fired guns, rapid-fired guns, missiles, infantry light weapons and vehicle-mounted light weapons, the number of different weapons used by all shooting actions of the player in the whole confrontation is counted, the ratio of the using number of the different weapons is used as quantitative data, and the more balanced the ratio is, the stronger the comprehensive utilization capacity of the weapons is.

Obtaining the evaluation score of the confrontation deployment and planning capacity of the intelligent agent according to the policy class data of the intelligent agent; for example, in a war game deduction system, the confrontation deployment and planning capacity of an intelligent body is embodied in two aspects of tactical novelty and action cooperative control capacity; tactical novelty is obtained according to the heat of the motion track and the track position of an operator operated by the intelligent agent, the heat of the motion track and the track position of the operator operated by the intelligent agent is compared with the heat of a historical track, and the lower the matching degree is, the more novel the tactical is; for example, movement is the basis for operators to execute high-level actions, and a movement path is the most intuitive embodiment of a player tactical strategy, so that the movement locus of all operators belonging to the player on the whole map and the heat (measured by the total time length) of each position on the locus are counted, and the longer the operator is at a certain position, the higher the heat of the position is; after statistics is finished, comparing the track heat degree table of the local countermeasure with the track heat degrees of all historical countermeasures under the same countermeasure condition, wherein the lower the matching degree is, the more novel the battle tactics of the field countermeasure is; the action cooperative control capability is obtained according to the action quantity of cooperative fire striking of the intelligent agent between operators, and the more cooperative actions, the stronger the action cooperative control capability is; for example, the cooperative fire striking action is defined as: the attack behaviors sent by two (and more) operators have respective fire shooting action occurrence time intervals not exceeding a threshold value T, and the distance between targets shot by two adjacent fire shots not exceeding a threshold value D, the number of cooperative fire striking actions executed by players in the whole confrontation is counted, and the more cooperative actions, the stronger the cooperative control capability of the actions.

Obtaining the evaluation score of the perception and the coping ability of the intelligent agent on the action and situation change of the enemy according to the perception class data of the intelligent agent on the enemy; for example, in a war game deduction system, the perception and coping ability of an intelligent body on the action and situation change of an enemy is embodied in two aspects of the perception judgment ability of an opponent and the telepresence strain ability; the opponent perception judging capability is obtained according to the average time interval between the intelligent body action and the opponent action, and the smaller the time interval is, the more the player can perceive the change of the opponent action and timely respond; specifically, an enemy operator action list in the visual field range of the player and a my operator action list following the corresponding action of the enemy are counted, the time interval of the action at the corresponding index of the two action lists is calculated, and then the average value of all the intervals is obtained; the smaller the average time interval is, the more the enemy can discover the action of the enemy and respond, namely the stronger the adversary perception judgment ability is; the opportunistic capacity is obtained according to the action number of the agent in unit time, and the higher the action density is, the stronger the capacity is; for example, the opportunistic responses are expressed in various action types such as shooting, robbing control and moving, and whether a player can make a reasonable and timely action decision under the current situation is an important factor for measuring the strain capacity.

Obtaining evaluation scores of the remarkable characteristics of the intelligent agent according to the man-machine distinguishing characteristic class data of the intelligent agent; for example, in a military chess deduction system, an evaluation score of the remarkable characteristics of an intelligent agent is obtained according to the number of illegal operations of the intelligent agent, if an action violating the normal operation of human beings occurs in the countermeasure process, punishment is made on the evaluation score, and the more unreasonable conditions are, the greater the punishment degree is; for example, in a war game pursuit, if a player has an ideal situation in which the manual operation speed of a human being is difficult to achieve, such as a plurality of operators shooting at the same time or an infantry and a cruise missile operator getting off at the same time, the player can be significantly judged as an agent. And counting the number of the types of unreasonable conditions in the whole fight, wherein the larger the number is, the larger the fraction punishment degree is.

Thus, the performance data for the agent obtained from 4 different aspects is expressed as S ₁ ,S ₂ ,S ₃ ,S ₄ ]。

The method for acquiring the external evaluation information of the intelligent agent in the form of the Tuoling questionnaire comprises the following steps:

after each confrontation is completed, guessing and evaluating information of the confrontation personnel for the opponent and the spectator for the confrontation personnel is collected through the Tuoling questionnaire module 740. The Turing questionnaire comprises guesses of the appraisers about the identities of the competitors and scores of various aspects of the competitor's performance; guessing the opponent's identity is: guessing whether the competitor is a human or an agent; the various scores of the performance of the competitors are: within the standard score range of 0-P, the available countermeasure resource application capacity, the countermeasure deployment and planning capacity, the perception and coping capacity of the enemy action and situation change and the obvious characteristics of the intelligent agent of the competitor are scored and evaluated. Wherein, the confrontation achievement management module 750 reads and stores the guess and evaluation information from the turing questionnaire module 740, and then the statistics unit counts the times that each player is guessed as human and guessed as machine in all the turing questionnaires, and calculates the man-machine guess misjudgment rate of the intelligent agent; the man-machine guess misjudgment rate is calculated according to the times of guessing as human and the times of guessing as machine of the player in all the Tuoling questionnaires, and specifically comprises the following steps:

wherein, P _Misjudgment Guessing the misjudgment rate t for human _{Human being} Number of times guessed as human, t _{Machine with a rotatable shaft} The number of times it was guessed as a machine.

S300, acquiring a capability test result of the intelligent agent according to the man-machine confrontation result, the external evaluation information of the intelligent agent and the efficiency data of the intelligent agent. In the step, the capability test result of the intelligent agent comprises a comprehensive evaluation score, a man-machine guess misjudgment rate and a system score; the comprehensive evaluation score is obtained according to the efficiency data of the agent and the external evaluation information of the agent, for example, the sum of the scores of the efficiency data of the agent in the four aspects of the available countermeasure resource application capability, the countermeasure deployment and planning capability, the perception and coping capability for the enemy action and situation change and the significant characteristics of the agent is obtained as the efficiency score, the sum of the scores of the external evaluation information of the agent in the four aspects of the available countermeasure resource application capability, the countermeasure deployment and planning capability, the perception and coping capability for the enemy action and situation change and the significant characteristics of the agent is obtained as the external evaluation score, and the sum or the average of the efficiency score and the external evaluation score is obtained as the comprehensive evaluation score. The system score is obtained according to a human-computer countermeasure result, namely a system adjudication score, and comprises the following steps: the system adjudges the total and average score, total points, victory arenas, battle-defeat arenas, victory, heat and ELO (ELO) scores of the scores; wherein, the total integration is the winning times in each confrontation, and the heat is the confrontation times which are participated in by the user in total.

During and/or after the man-machine competition, the method further comprises the following steps:

acquiring real-time confrontation data in a man-machine confrontation process;

and analyzing the real-time countermeasure data and/or the multi-disc countermeasure data, and performing anonymous 2D or 3D countermeasure graphic display and switching display of different battlefield situations according to the analysis result of the real-time countermeasure data and/or the multi-disc countermeasure data.

The exhibition of the confrontation image and the exhibition of the switching of different battlefield situations are convenient for observers to observe and massage, wherein the external evaluation information of the intelligent body comprises evaluation information made by human confronters and evaluation information made by observers.

According to the man-machine confrontation system supporting the Turing test mode and the intelligent body test method, the Turing questionnaire module is creatively introduced into the man-machine confrontation system, the outside evaluation information of the intelligent body is acquired through the Turing questionnaire module, the effectiveness data of the intelligent body is obtained through the analysis of confrontation data in the man-machine confrontation process, the decision-making capability of the intelligent body is tested by combining the man-machine confrontation result, the outside evaluation information of the intelligent body and the effectiveness data of the intelligent body, and the comprehensive and effective test of the cognition and decision-making capability of the intelligent body is realized; meanwhile, the system center scheduling is carried out through the countermeasure scheduling management module, the front-end anonymous processing is completed through unified coding of the player system codes, the rear-end interface information filtering is carried out through the UI module and the AI module, the man-machine double-blind countermeasure environment is realized, the man-machine countermeasure process is ensured to be completed in the double-blind environment, and therefore the fairness of data in the intelligent agent cognition and decision-making capability test process is ensured; in addition, the fairness of data in the cognitive and decision-making ability test process of the intelligent agent is further improved through anonymous observation (the observation interface displays the system code and does not display the real identity of an opponent), and therefore the accuracy of the decision-making ability test result of the intelligent agent is improved.

Fig. 3 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 3: a processor (processor) 810, a communication Interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication Interface 820 and the memory 830 communicate with each other via the communication bus 840. Processor 810 may invoke logic instructions in memory 830 to perform an agent testing method comprising: acquiring a man-machine confrontation result and countercheck data of man-machine confrontation; wherein, human-computer confrontation is completed by human confronters and intelligent agents in a double-blind environment;

acquiring efficiency data of the intelligent agent according to the counterwork data of the multiple disks, and acquiring external evaluation information of the intelligent agent in the form of a map questionnaire;

In addition, the logic instructions in the memory 830 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the method for testing an agent provided by the above methods, the method comprising: acquiring a man-machine confrontation result and multi-disk confrontation data of man-machine confrontation; wherein, human-computer confrontation is completed by human confronters and intelligent agents in a double-blind environment;

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor, implements an agent testing method provided by the above methods, the method comprising: acquiring a man-machine confrontation result and countercheck data of man-machine confrontation; wherein, human-computer confrontation is completed by human confronters and intelligent agents in a double-blind environment;

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A human-machine confrontation system supporting a turing test mode, comprising:

the man-machine confrontation module is used for finishing man-machine confrontation; wherein the human-machine confrontation is completed by a human confronter and an agent in a double-blind environment;

the countercheck data analysis module is used for acquiring operation data of the intelligent agent in the man-machine countercheck process according to the countercheck data and acquiring efficiency data of the intelligent agent from a plurality of different aspects according to the operation data; wherein the operational data of the agent comprises: own resource application class data, own strategy class data, enemy perception class data and man-machine distinguishing characteristic class data;

2. The system of claim 1, wherein the warrior countermeasure module comprises:

and the man-machine confrontation function module is used for completing the man-machine confrontation in a simulated environment through the human confronter and the intelligent body under the process scheduling of the man-machine confrontation.

3. The system of claim 2, wherein the human-machine confrontation function module comprises:

4. The system of claim 1, further comprising a confrontation and massage module;

the countercheck sightseeing module is used for analyzing the real-time countercheck data and/or the multi-disc countercheck data, and performing anonymous 2D or 3D countercheck graphic display and switching display of different battlefield situations according to the analysis result of the real-time countercheck data and/or the multi-disc countercheck data.

5. An agent testing method, comprising the steps of:

according to the countercheck data, acquiring operation data of the intelligent agent in the man-machine countercheck process, and respectively acquiring performance data of the intelligent agent from a plurality of different aspects according to the operation data; wherein the operational data of the agent includes: own resource application class data, own strategy class data, enemy perception class data and man-machine distinguishing characteristic class data;

acquiring external evaluation information of the intelligent agent in the form of a Tuoling questionnaire;

6. The agent testing method of claim 5, wherein completing the human-machine confrontation in a double-blind environment by a human confronter and an agent comprises:

generating a cyclic course of confrontations and encoding the system codes of the human confronters and the agents; wherein encoding system codes of the human competitor and the agent is used to create the double-blind environment;

and automatically scheduling the process of the man-machine confrontation according to the circular competition process, and finishing the man-machine confrontation in a simulated environment through the human confronter and the intelligent agent under the scheduling of the process of the man-machine confrontation.

7. The agent testing method of claim 6, wherein completing the human-machine confrontation in a simulated environment by the human confrontator and the agent comprises:

an environment generation step: generating the simulation environment;

cycling the step of confrontation until the human-machine confrontation is completed;

wherein, before the countermeasure step, further comprising: filtering identity information of the human competitor and the agent.

8. The agent testing method of claim 5, further comprising, during and/or after the human-machine confrontation,:

and analyzing the real-time countermeasure data and/or the multi-disc countermeasure data, and carrying out anonymous 2D or 3D countermeasure graphic display and switching display of different battlefield situations according to the analysis result of the real-time countermeasure data and/or the multi-disc countermeasure data.