CN101303589A - Dynamic Multi-target Collaborative Tracking Method Based on Finite State Automata - Google Patents

Abstract

The invention discloses a dynamic multi-target cooperative tracking method based on a finite state automaton, which is characterized in that: a composite AI entity detects environment information I, a server or other AI entities or AI entities The task information M that needs to be completed conveyed by the group manager, and/or the artificially specified information H conveyed by the server, among multiple finite automatic state machines, select a finite automatic state machine as the Finite Automatic State Machines for Behavioral State Models. The AI entity is driven by factors such as behavior status and emotional information, and performs centralized control or individual information interaction control of the AI entity through information interaction or coordination with the server team. The present invention is applicable to different architectures such as centralized, distributed and hybrid.

Description

Dynamic Multi-target Collaborative Tracking Method Based on Finite State Automata

technical field

The invention belongs to the field of robot navigation and application, and relates to a dynamic multi-target cooperative tracking method based on finite state automata.

Background technique

The complexity and diversification of modern tasks have put a higher demand on the combination of multi-robots and the completion of team tasks. How to cooperate with multi-robots to conduct reconnaissance on the environment through existing video acquisition equipment, information interaction and fusion, The ability to guide behavior through vision is particularly important. However, there is currently no general tracking method in this regard that can be widely used.

In an unknown environment, to use a large-scale robot team with visual equipment to observe and track dynamic multi-targets collaboratively, it is necessary to observe the environment in real time through the AI real-time model on the multi-robots asynchronously and solve local problems. Synchronization and communication between the entire multi-ai entity system ensures the real-time and accuracy of information, and makes decisions based on global information. In terms of synchronization, it can be synchronized through the time stamp in the protocol, the information interaction of multiple AI entity groups, or combined with the global monitoring and management on the server side, while asynchronously uses the AI entity model based on finite state automata. Through the positioning of the state, Independent of the motion control platform, it can mobilize heterogeneous multi-ai entities without video equipment, distribute computing information, and complete tasks collaboratively. The combination of the two solves the synchronous and asynchronous problems of information and decision-making in a good compromise, increases the adaptability of the robot team to the environment, is conducive to the efficient cooperation of the robot team, and ensures the comprehensiveness, accuracy and accuracy of the reconnaissance information. Rapidity.

Contents of the invention

The technical problem to be solved by the present invention is to provide a dynamic multi-target cooperative tracking method based on finite state automata.

The technical scheme that the present invention adopts for solving the problems of the technologies described above is:

A dynamic multi-objective cooperative tracking method based on finite state automata, characterized in that: the compound AI entity is based on the environment information I, server or other AI entities or AI entity group managers detected by itself The conveyed task information M that needs to be completed, and/or the artificially specified information H conveyed by the server, among multiple finite automatic state machines, select a finite automatic state machine as the one used to maintain the compound AI entity behavior state model Finite automatic state machine.

Described finite automatic state machine is full-automatic state machine and semi-automatic state machine; Full-automatic state machine is:

It consists of five states: waiting, observing, tracking, lost and busy. The specific transition relationship of each state is as follows:

(a) For the "waiting" state, if the input is "prohibit connection", the original state will be maintained; if the input is "connection", it will switch to the "observation" state;

(b) For the "observation" state, if the input is "lost target", it will maintain the original state; if the input is "forbidden connection", it will switch to the "waiting" state; command" will switch to the "lost" state; if the input is "found target" then it will switch to the "tracking" state;

(c) For the "tracking" state, if the input is "found target", it will keep the original state; if the input is "lost target", it will switch to the "lost" state; if the input is "task", it will switch to the "busy" state; if the input is " command" will switch to the "observation" state; if the input is "prohibit connection" then it will switch to the "waiting" state;

(d) For the "lost" state, if the input is "lost target", it will maintain the original state; if the input is "found target", it will switch to the "tracking" state; if the input is "task", it will switch to the "busy" state; if the input is " command" to switch to the "observation" state;

(e) For the "busy" state, if the input is "prohibit connection", it will switch to the "waiting" state; if the input is "task completed", it will return to the state before accepting the task according to the record, which can be "observation" state, "lost " state or "tracking" state;

The semi-automatic state machine is:

It consists of five states: waiting, observing, tracking, lost and busy. The specific transition relationship of each state is as follows:

(a) For the "waiting" state, if the input is "prohibit connection", the original state will be maintained; if the input is "connection", it will switch to the "observation" state;

(b) For the "observation" state, if the input is "lost target", it will maintain the original state; if the input is "forbidden connection", it will switch to the "waiting" state; Target found" will switch to "Tracking" state;

(c) For the "tracking" state, if the input is "found target", it will maintain the original state; if the input is "lost target", it will switch to the "lost" state; if the input is "task", it will switch to the "busy" state;

If the input is "command", it will switch to the "observation" state; if the input is "prohibit connection", it will switch to the "waiting" state;

(d) For the "lost" state, if the input is "lost target", it will maintain the original state; if the input is "found target", it will switch to the "tracking" state; if the input is "task", it will switch to the "busy" state;

If the input is "command", it will switch to the "observation" state;

(e) For the "busy" state, if the input is "prohibit connection", it will switch to the "waiting" state; if the input is "task completed", it will return to the state before accepting the task according to the record, which can be "observation" state, "lost " state or "tracking" state.

The behavior of the compound type AI entity can be expressed as M=(Q, ∑, δ, q ₀ , F), which is a mathematical model consisting of the following parts:

A finite set of states Q={waiting, observing, tracking, lost, busy}, that is

Q = {Wait, Detect, Track, Lost, Busy};

Acceptable input set ∑, which specifies all symbols that allow input, and the finite automatic state machine performs state changes according to the input of symbols in this set, which is expressed as follows:

∑={connect, target found, target lost, connection prohibited, task, task completed, order},

That is, ∑={connect, findobj, lostobj, unconnect, work, finishwork, order}

The initial state q ₀ = {Wait}, that is, q ₀ = {Wait}, the first state after the AI entity is turned on, and will remain in this state when the video device cannot be connected;

The end state F={wait}, that is, F={Wait}, when the AI entity cannot continue to actively participate in the cooperative tracking, it will enter this state after disconnecting the video device. Before the AI entity physical equipment finishes all tasks, Also disconnect the video equipment first, and declare to leave the Aizhen body group;

The transfer function δ is a mapping of Q×Σ→Q, which is recognized by the finite automatic state machine.

Through the behavior state model maintained by the compound AI agent finite automatic state machine and the constraints of real-time tasks and environments, multiple composite AI agents are separated into several groups, and the composite AI agents within the group can directly exchange information ; Each group has an AI entity group manager, and each group can communicate directly or indirectly through the AI entity group manager; the direct communication between groups is to stimulate information interaction and/or timing information interaction through information update The indirect communication mentioned above is in the multi-ai entity system with a server, and the groups communicate through the server.

The AI-entity group manager obtains three types of information from the composite AI-entity individuals:

The first category is target information, which is used to ensure that the information of the target library on the AI entity group manager or the server is the latest information;

The second category is the detailed information of the video equipment and the status information of the compound AI entity; the AI entity group manager regularly receives new information from the AI entity individual, and in the case of a server, the AI entity group manager according to The content of the blackboard in the update server of the updated information;

The third type of information is the request information; when the composite AI entity encounters an emergency situation and the accident handling is invalid, an assistance request signal will be generated, the request sequence will be written, and the request processing module will be activated; the request processing module will let the The groups of groups carry out module scheduling to meet the request sent by a compound AI entity; when the task of a single group cannot be completed, the single group dispatches other groups to assist.

Described compound formula body comprises following several parts:

1) A behavioral state model maintained by a finite automatic state machine. The behavioral state model includes a planning scheme. The behavioral state model decides to select an appropriate module from the module library for the next calculation according to the current state and the planning scheme;

2) A module library, including video processing modules and modeling modules, the modules can be expanded according to requirements; video processing modules include target detection based on three-frame difference, morphological noise removal, target segmentation, target screening, target information calculation, target Merging and extraction and mean shift algorithm tracking; modeling modules include memory process for unexpected handling, predictive process and self-protection handling;

3) A communication module, used for communicating with other compound AI entities, AI entity group managers and/or servers.

The negotiation scheme adopted includes three parts: negotiation agreement, negotiation method and negotiation approach:

1) Negotiate agreement

The form of the negotiation protocol includes multi-bit start and end flags, instruction length negotiation primitives, message numbers and message content; the negotiation primitives include requests, commands and information instructions, and the message content includes message recipient types, Effective information content and message sending time;

2) Negotiation method

One kind of negotiation method is to activate the task allocation algorithm, competition negotiation algorithm, information calculation part and The request processing module obtains the optimal negotiation plan, and writes the result back to the task list; then the manager sends the instruction to the relevant compound AI entity with the highest priority according to the task list.

Another negotiation method is that in the compound AI entity group, the individual composite AI entities exchange information, directly fuse information with each other, and negotiate to complete the task;

3) Negotiation channels

Communicate through a reconfigurable point-to-point communication platform for multi-mobile robots. When the state of the compound AI entity changes, it will send the current information to the AI entity group manager, and the AI entity group manager will update the information interactively or update the server. Information: In the case of no state change, update the information of the AI entity group manager every time T, and at the same time update the status and information of other compound AI entity individuals in the group maintained by the AI entity group manager .

The described dynamic multi-objective cooperative tracking method based on finite state automata comprises the following steps:

The first step is the preparation of the composite AI entity: after the composite AI entity is turned on, it will carry out activities through the behavior state model maintained by the finite automatic state machine, or communicate with other composite AI entities; when the composite AI entity When it is turned on, it is in the waiting state, that is, the wait state. When the composite AI body can connect to the video device, it will leave this state; the composite AI body that remains in this state may be due to the failure of the video device, no video device or It is designated as not allowed to conduct investigative activities, unable to self-observe changes in the external environment, and only waits to receive external instructions to carry out activities;

The second step is to complete the specified task: when the compound AI entity is turned on, if it is in the group, it will start to regularly update the information on the group manager of the compound AI entity, and if there is a server, update the server information at the same time; when accepted When it comes to a designated task, if the compound AI body is in a fully automatic state, it will switch to a semi-automatic state; if the accepted task is arranged in a formation, all compound AI bodies in a non-busy state will enter the task group , distribute the tasks according to the number of compound AI entities currently completing the task, and guide the composite AI entities to complete; if the accepted task is to find or track a designated target, all currently under observation or a certain area The robot enters the task group to accept the task; it is marked by the finding of the target by a compound AI entity that accepts the task, indicating that the task is completed, and notifying other composite AI entities in the task group to give up the task; when the composite AI entity After the body accepts the task, its state is in the busy state; the state transition is converted according to the assigned instructions. If the state to be transformed is not specified, it will automatically return to the state before entering the busy state, restore the saved task point, and continue complete the interrupted work;

The third step is environmental investigation: after the composite AI entity is connected to the video equipment, it will automatically enter the observation state to carry out investigation activities; the composite AI entity in the observation state will observe the moving individuals within the field of vision, calculate and record their information ; If it is connected to the server or other compound AI entities, the information will be shared and notified; if it is in a fully automatic state, then the target will be selected and tracked according to the given rules, and it will enter the tracking state, that is, the Track state; if it is semi-automatic In the fully automatic state, when the target is extracted, if the target is tracked, it will meet the criteria of first-come-first-track, the largest tracking viewing area, and the closest distance between the target and the compound AI entity , select the optimal target to switch to the tracking state for tracking, and other targets can perform basic visual tracking; if the target is not tracked, return to the observation state and continue to observe;

The fourth step, target tracking: the compound AI entity obtains a large amount of data information from the external environment through the sensor, analyzes the data through the video processing module in the module library, obtains the tracked target information, and uses the modeling module to analyze the data. The information is stored and calculated; the tracking is mainly realized by the color-based mean shift algorithm; when the tracked target is lost, it enters the target loss state, that is, the Lost state;

The fifth step, accident handling: including memory process and prediction process;

1) Memory process, memory is a process in which a compound AI entity keeps known information in storage while tracking the state, which is divided into short-term memory and long-term memory; short-term memory will remember the most recent actions , and the long-term memory will remember the moving path of the composite AI entity and the target during the whole process of tracking a target; according to the obtained information, the composite AI entity and the target movement can be calculated by curve fitting The path information is summarized; the fitting on the client side only considers the most recent part of the data, and performs segmented curve fitting on it; the compound AI entity manager or server side will calculate the overall data;

2) In the prediction process, when the area of the calculated target is less than a certain threshold, the target is considered lost; then it enters the lost state, that is, the Lost state; the short-term memory generally does not exceed three actions, and the most recent action plays the main role, and the others play a role. Auxiliary inspection function; if the target is not found again through the guidance of short-term memory, then the target trajectory fitted by the polynomial curve of long-term memory will guide the composite AI entity to turn to the predicted angle for observation; when the composite AI If the real body has not found the target within a certain time limit, and thinks that the target is really lost, it sends out a target loss message.

The beneficial effects of the present invention have:

The finite-state automata-based dynamic multi-target cooperative tracking method for multiple AI entities proposed by the present invention is independent of the action control platform, and supports centralized control and interactive control of AI entity individual information. The proposal of the composite multi-ai body will be applicable to different multi-ai body system architectures, and is suitable for multi-target collaborative tracking of multi-robot teams. Each AI entity can work independently, and can also cooperate to complete the task. Since it is divided into different AI entity groups according to the task, the work efficiency is higher. When there is a server participating, the server is the general command center, commanding the coordination and cooperation of each entity or group from the overall situation.

Description of drawings

Fig. 1 is the abstract application of the dynamic multi-objective cooperative tracking method based on finite state automaton in the present invention;

Fig. 2 is the design framework of compound type AI phantom model among the present invention;

Fig. 3 is the detailed invocation relation schematic diagram between the state and the module of the behavioral model of the dynamic multi-objective cooperative tracking method based on finite state automata in the present invention;

Fig. 4 is the finite automatic state machine of the behavior model in the fully automatic state in the dynamic multi-objective cooperative tracking method of multiple entities based on the finite state automata of the present invention;

Fig. 5 is the finite automatic state machine of the behavior model in the semi-automatic state in the dynamic multi-objective cooperative tracking method of multiple entities based on the finite state automata of the present invention;

Fig. 6 is the format of the negotiation agreement of the AI entity team in the present invention;

Fig. 7 is the module flow chart between the Track state and the Lost state of the behavior model in the dynamic multi-target cooperative tracking method of multi-ai entities in the present invention;

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Example 1:

The present invention proposes a multi-agent dynamic multi-target cooperative tracking method based on finite state automata, and its application basis is a multi-agent system (Multi-Agent System, referred to as MAS). This method is applied to an AI entity that can continue to function autonomously. It selects a finite automatic state machine (Deterministic Finite Automation, DFA for short) according to the environment and task requirements to maintain the behavioral state model of the compound AI entity, and then combines the video The environmental information sensed by the device sensor and the shared resources in the AI entity group, communicates, collaborates and negotiates with other AI entities, performs modeling, prediction, planning, and decision-making, and guides the individual AI entities to control the actuators to perform certain actions. . The present invention abstracts the individual AI entity into a composite model. In this model, for the AI entity itself, the finite automatic state machine can freely choose according to the environment and task requirements to maintain the behavior state. For the AI entity social group, it can Manage heterogeneous teams of multiple entities from an abstraction level. The AI entity is driven by factors such as behavioral state and emotional information, such as the AI entity's tracking fatigue, and the individual preference of the AI entity reflected by the evaluation function. Through information interaction or in combination with server team coordination, Centralized control or AI entity individual information interactive control. The multi-target cooperative tracking method can be applied to different architectures such as centralized, distributed, and hybrid.

In order to solve the technical problems existing in the existing multi-target cooperative tracking of heterogeneous multiple entities, the present invention proposes a dynamic multi-target cooperative tracking based on finite state automata according to the characteristics of finite state automata method, which is independent of the motion control platform and supports centralized control and interactive control of individual information of AI entities. The proposal of this composite multi-ai body will be applicable to different multi-ai body system architectures, and a multi-target cooperative tracking scheme suitable for multi-robot teams is given.

Each AI entity group obtains valid data from each AI entity, integrates it, and shares it with all AI entities, and calls different decisions according to the updated information and individual task requests of AI entities to realize the task Allocation, collaborative group cooperation, to achieve the purpose of collaborative detection and collaborative tracking of dynamic multi-targets in an unknown environment, dynamic background; in order to achieve this function, on the AI entity group leader AI entity, the decision generation module includes the task allocation part , knowledge resources such as modeling target prediction, and guide the overall robot group to perform actions.

The ai entity has a composite ai entity model that can continue to function autonomously in a collaborative computing environment. It perceives its environment through video equipment sensors and acts on the environment through motion control. It chooses to use the finite automatic state machine to maintain the behavioral state model by itself through the environment information obtained by itself, and through the shared resources in the AI entity group and the communication, collaboration and negotiation with other AI entities, to perform modeling, prediction, planning, Decision-making, guiding actions, controlling actuators to perform actions, acting on the environment, and synchronously updating the corresponding resources on the server if there is a server.

The present invention will be further described below in conjunction with the accompanying drawings.

1. Ai entity group abstraction

The method implements two levels of abstraction through state localization based on the Ai entity model of finite state automata. First, the heterogeneous multiple AI entities are abstracted into a consistent AI entity model through the AI entity model independent of the motion control platform; second, the behavior state is maintained by the finite automatic state machine in the AI entity model The model separates multiple AI entities into several groups through real-time tasks and environmental constraints, and each group has a manager with strong performance, and then each group can be connected through Managers communicate directly or indirectly. And within the group, each entity can directly exchange information. The entire abstract form is shown in Figure 1.

The AI entity group manager obtains three types of information from AI entity individuals.

The first type is target information, which is used to ensure that the information of the group manager or the target library on the server is the latest information. On the one hand, it triggers the information calculation part, which performs polynomial curve segment fitting on the known information that is valid for the updated target. , and correct the error. On the other hand, according to the choice of decision, the task assignment algorithm is triggered to generate a new tracking decision, and instructions are sent to the relevant AI entities according to the robot task list.

The second category is the detailed information of the video equipment and the state information of the AI entity. The group manager regularly receives new information from AI entities. If there is a server, the group manager updates the content of the blackboard according to the updated information.

The third type of information is request information. When the Ai Entity encounters an emergency and the emergency handling is invalid, an assistance request signal will be generated, and the request sequence will be written to activate the request processing module. The request processing module will allow the AI entity group where it is located to perform module scheduling to meet the request sent by an AI entity. When a single group task cannot be completed, other groups are dispatched to assist.

2. Composite Ai entity individual model

According to the proposed method, the compound AI body is designed as shown in Figure 2, including the following parts:

1) A behavioral state model maintained by DFA, the model includes a planning scheme, and the model decides to select an appropriate module from the module library for the next calculation according to the current state and planning scheme;

2) A module library, mainly including video processing modules and modeling modules, the modules can be expanded according to requirements; video processing modules include target detection based on three-frame difference, morphological noise removal, target segmentation, target screening, target information calculation, Target merging and extraction, Meanshift tracking (mean shift algorithm), etc.; the modeling module includes the memory process of unexpected processing, prediction process and self-protection processing.

3) Communication module

3. Behavior state model of AI entities

The behavior state model maintained by DFA and the module library of video processing module and modeling module constitute the realization entity of behavior decision-making layer, which can be expressed as a mathematical model of M=(Q, ∑, δ, q ₀ , F), which is introduced in detail as follows.

1) A finite set of states Q={Wait, Detect, Track, Lost, Busy} There are five states in the state set: waiting state: Wait, observation state: Detect, tracking state: Track, lost tracking state, referred to as tracking Status: Lost, Busy status: Busy. For a detailed introduction, see the detailed introduction of the behavioral state model. The planning solutions in the Detect state, Track state, and Lost state in the state set completely cover the autonomous behavior planning of the AI entity. The planning scheme of these three states, combined with the video processing module and modeling module provided in the module library, processes the obtained information, and gives its own decision on the behavior of the AI agent according to the result. The detailed relationship between these three states and the modules to be called by their corresponding planning schemes is shown in Figure 3.

2) Acceptable input set Σ, which specifies all symbols that allow input, and the finite automatic state machine performs state changes according to the input of symbols in this set, expressed as follows:

∑={connection, target found, target lost, connection prohibited, task, task completed, command}, that is

∑={connect, findobj, lostobj, unconnect, work, finishwork, order}

This set includes 7 symbols that can be entered, which represent the occurrence of corresponding physical events in practice. The details are as follows:

Connect connect: Indicates that the AI entity has successfully connected to available video devices;

Find the target findobj: Indicates that the AI entity has found an unknown or known target within the range of the field of vision through the currently acquired video image in the current state;

Lost target lostobj: Indicates that in the current state, the AI entity cannot find a target that matches the historical information within the range of the field of vision through the currently acquired video image;

Prohibition to connect unconnect: It means that when the physical conditions limit or the overall resource allocation is restricted by the AI entity, it is stipulated that it cannot use video equipment to obtain environmental information and actively participate in collaborative tracking activities;

work: Indicates that the AI entity accepts specified commands in the current state and assigns new tasks. After accepting the work command, the AI entity enters the Busy state and does not allow itself to perform new task deployment.

Task completion finishwork: Indicates that the AI entity has completed all the tasks in the command queue. After completing the task, Ai Zhenbody returns to the state before accepting the order, and continues to complete the task interrupted by the specified order;

Order order: Indicates that the AI entity accepts the instruction, switches to a new state, and completes the assigned task.

3) The initial state q ₀ ={Wait}, when the AI body is turned on, it will directly enter this state, and if it cannot connect to the video device, it will remain in this state.

4) End state F={Wait}, when Ai-Print cannot continue to take the initiative to participate in the collaborative tracking, then disconnect the video device and enter this state. Equipment, announced to leave the group of Ai entities.

5) The transfer function δ is a mapping of Q×∑→Q, which is recognized by the finite automatic state machine. The finite automatic state machine is divided into two types according to the individual behavior state of the AI entity, one is the finite automatic state machine in the fully automatic state, and the other is the finite automatic state machine in the semi-automatic state.

Fully automatic state: In the course of the behavior of the AI entity, without the support of other AI entities and server information, it can also complete the independent discovery target, perform tracking tasks, and independently search for other AI entities. Known as the fully automatic state.

Semi-automatic state: During the behavior of the AI entity, under the specified cooperation situation, the AI entity accepts the instruction, tracks and searches for the designated target, communicates with the known AI entity, and cooperates to complete the task, and cannot give up the existing task at will. Carrying out other non-authorized voluntary actions is called a semi-autonomous state.

The state sets of the finite automatic state machine in the fully automatic state and the semi-automatic state are consistent, and the two states can be freely switched according to the actual environment and task requirements. Freely convert the environmental information I detected by itself, the task information M conveyed by the server or other entities, and the artificially specified information H conveyed by the server. In a variety of finite automatic state machines, N=f(I , M, H) is the selection index to select the optimal automatic state machine. If the environment information detected by the current ai entity itself includes target information, and the ai entity group manager or server does not transmit task instructions, the ai entity operates in a fully automatic state; and if the ai entity group manager or After the server conveys the task instructions, it will choose the semi-automatic state, and for the purpose of cooperation, it will give priority to the execution of the task instructions. After the task is completed, it will return to the fully automatic state, and then make self-decisions; , until the task is completed and other instructions are received.

Refer to Figure 4 and Figure 5 for the specific transfer function of the finite automatic state machine.

4. Negotiation plan

The negotiation scheme adopted by this method includes three parts: negotiation agreement, negotiation method and negotiation approach.

1) Negotiate agreement

Communication is a key link in realizing collaboration in the method of this embodiment, and it plays a role of a bridge in the process of realizing collaboration. The form of the negotiation protocol in this method is shown in FIG. 6 . The multi-bit start and end flags and the length of the instruction are all to ensure that an instruction can be completely parsed during network transmission, because when the information is packaged and transmitted, it may not be possible to give a complete piece of information at a time, so in continuous reception In the information, it is necessary to be able to restore the split instructions. The message number and message sending time are to ensure that no repeated or outdated information is used to update the data in the AI entity group manager or server and the AI entity individual when there is network congestion and retransmission.

2) Negotiation method

The present invention can adopt the following two negotiation methods, one is to activate the task allocation algorithm through the real-time changes of the robot status list and global target library, decision selection, task list, request sequence and other information provided by the agent group manager or server , Competitive negotiation algorithm, information calculation part and request processing module and other knowledge resources, get the optimal negotiation plan, and write the result back to the task list. The manager then sends instructions to the relevant AI entities with the highest priority according to the task list. Another negotiation method is to exchange information between AI entities and AI entities, directly fuse information with each other, and negotiate to complete tasks. The selection among AI entities is the result of the information exchange between AI entity group managers or the server's group division according to tasks. AI entities that meet similar conditions will be divided into an AI entity group (that is, an AI entity group), and the AI entity individuals and other members of the AI entity group can exchange required information.

3) Negotiation channels

The heterogeneous multi-agents communicate through a reconfigurable multi-mobile robot point-to-point communication platform. When the status changes, the individual ai entities will send the current information to the ai entity group managers, and the ai entity group managers will interact and update information, or update server information (if a server exists). In the absence of state changes, update the information of the group manager of the ai entity every time T, and at the same time update the status and information of other ai entity individuals in the group it maintains.

The present invention is applied to a large-scale robot team with visual equipment in an unknown environment for collaborative observation and tracking of dynamic multi-targets. The main steps are as follows:

The first step is AI entity preparation: when an AI entity is turned on, it will carry out activities or communicate with other AI entities through the behavior state model maintained by DFA. When the AI body is turned on, it is in the Wait state, and when the AI body can connect to the video device, it will leave this state. Ai entities maintained in this state may be unable to self-detect changes in the external environment due to video equipment failure, lack of video equipment, or being designated as not allowed to conduct investigative activities. Only wait to receive external instructions to carry out activities;

The second step is to complete the specified task: when the AI entity is turned on, if it is in the group, it will start to regularly update the information on the group manager of the AI entity, and if there is a server, it will update the server information at the same time. The server is designed to accept human intervention part of the information. When receiving a designated task, if the Ai Entity is in a fully automatic state, it will switch to a semi-automatic state. If the accepted task is arranged in a formation, all AI entities in the non-Busy state will enter the task group, and the task will be assigned according to the number of AI entities currently completing the task, and they will be guided to complete. If the accepted task is to find or track a specified target, all robots currently in the Detect state or in a certain area will enter the task group and accept the task. It is marked by the finding of the target by a certain agent accepting the task, indicating that the task is completed, and notifying other robots in the task group to give up the task. If the Ai Entity accepts the task, it will be in the state with the highest priority among the individual states of the Ai Entity: the Busy state. It shows that the Ai entity is currently completing an assigned work, which cannot be interrupted by its own instructions, and other orders should be carried out after it completes the work. The state conversion is converted according to the assigned instructions. If the state to be converted is not specified, it will automatically return to the state before entering the busy state, restore the saved task point, and continue to complete the interrupted work.

The third step is environmental investigation: after Ai Zhentu is connected to the video equipment, it will automatically enter the observation state and carry out investigation activities. Ai entities in the observation state observe moving individuals within the field of vision, calculate and record their information. If the server or other entity is contacted, the information will be shared and notified. If it is in the fully automatic state, the target will be selected and tracked according to the given rules, and then transferred to the Track state. If it is in the semi-automatic state, it will act according to the authorization; for the detection of moving individuals within the field of vision, the technology used is to use three frames first The difference background clipping method is used to extract the target, and then through morphological denoising and Gaussian filtering, the separated part in the motion space is obtained to make a mask, the target is segmented, the target is merged and extracted, and finally the original target information is searched iteratively. And use the watershed algorithm to obtain individual target information. They are screened by the defined target screening principles, which are as follows:

1. If the position of the center of gravity of the target is similar, the tone value of the target is similar, and the size of the area is similar, it is considered that it is caused by the wrong segmentation of the same target, and the one with a larger area is selected to keep, and the mean value of the center of gravity and the mean value of the tone are taken.

2. In the actual target observation, there will be basically no cases where the RGB value is 0 or the H value is 0.

3. When there is no target, the measured noise due to the influence of light generally has a large area, but when there is a target, the measured noise information is small due to the influence of light. The physical actions of individual AI entities may also produce some small-area information, and such small-area targets need to be screened.

After target screening, it is necessary to number the target, and the number of the target needs to be assigned by the AI entity group manager or server. After defining the screening target, first assign a hue number to the target according to the hue value, and then the server sends the information number to each AI entity, check whether there is the hue number target in the target library, if not, you can assign the target number as 1. If there is already a target for that hue number, check to see if it is the same target. Through the known information of the target, the position of the center of gravity of the target in the image, the position and direction of the individual AI entity, and the data information of the sonar, the position or direction of the target can be obtained. If the target is similar to the known target in the global map, it will be considered as a known target, assigned a known number, and its information will be notified to other AI entities that track this target. Then track several Ai entities of the same target, and in the subsequent process, several Ai entities will exchange information with each other and correct the information. If it is not the same target, continue to assign the next number according to the known number.

In the fully automatic state, when the target is extracted, if the target is tracked, then the optimal target is selected for action conversion to meet the criteria of first-come-first-tracked, the largest tracking visible area, and the closest distance between the target and the AI entity. Track state for tracking, and other targets for basic visual tracking. If the target is not tracked, return to the Detect state to continue observing.

The fourth step, target tracking: the state that interacts with the environment most during the behavior of the AI entity is Track, the tracking state. In this state, the AI entity obtains a large amount of data information from the external environment through the sensor, analyzes the data through the video processing module in the module library, obtains the tracking target information, and stores the analyzed information through the modeling module and calculate. Due to the description of the target, two invariants are used: color information and contour information, and a variable variable area information is used to represent, so the algorithm used for tracking is mainly realized through the color-based Meanshift algorithm. If the tracked target is lost, it enters the Lost, target lost state. In theoretical analysis, the transition between the Track state and the Lost state is very deterministic. When the target is within the field of vision, there will be no loss and state transition. But in a moving AI entity and an unknown environment, because the physical movement of the AI entity will produce a certain inertia, the light and shadow of the environment will change with the movement of the AI entity. In the environment of multiple AI entities, the occlusion It is also a problem that cannot be ignored. When the state transition is too deterministic, the loss rate of the object tracking by the agent will be too high. Therefore, the process of unexpected handling is designed in the method. When the target is lost, first use the memory model in the accident processing to guide certain actions, and perform multiple searches and confirmations. If the target is lost after multiple confirmations and the target cannot be tracked, it is considered that the target is finally lost. , proceed to the next step;

The fifth step, accident handling: Under the influence of various unpredictable conditions such as light and physical inertia, the accidental loss of the target is very likely. Therefore, in order to deal with such accidents, memory and prediction functions are designed for Ai entity individuals.

(1) Memory process. Memory is a process in which an individual AI entity stores known information in a container while in the Track state. It is divided into short-term memory and long-term memory. Short-term memory remembers actions recently made, while long-term memory remembers the resulting entity and the path the target traveled throughout the course of tracking a target. After a certain amount of information is obtained, the curve fitting can be used to summarize the information of the moving path of the entity and the target. On the client side, due to the individual performance limitations of AI entities and a large number of tasks such as image data processing and communication, as well as the trajectory uncertainty of the target movement, only the nearest part of the data is considered for fitting, and segmented curve fitting is performed on it. Ai entity managers or servers will perform more detailed calculations through overall data

(2) During the prediction process, when the calculated area of the target is smaller than a certain threshold, the target can be considered lost. Then enter the Lost state. If there is no temporary action through short-term memory after losing the target, complete a temporary action first, and then observe. The short-term memory generally does not exceed three actions, the most recent action plays a major role, and the others serve as auxiliary inspection work. If the target is not found again through the short-term memory guidance, the target trajectory fitted by the polynomial curve of the long-term memory is used to guide the entity to turn to the predicted angle for observation. When the AI entity has not found the target within a certain time limit, it considers that the target is indeed lost, and then sends a message to the server, and the server will guide the AI entity group to make corresponding decisions and actions.

Claims (8)

1, a kind of dynamic multi-target cooperative tracking method based on finite state automata, is characterized in that: composite AI entity according to the environmental information I, server or other AI entity or AI entity group by self detection The task information M to be completed conveyed by the manager, and/or the artificially specified information H conveyed by the server, among multiple finite automatic state machines, select a finite automatic state machine as the one used to maintain the behavioral state of the composite agent Model a finite automatic state machine. 2. A kind of dynamic multi-target cooperative tracking method based on finite state automata as claimed in claim 1, characterized in that: said finite automatic state machine is a fully automatic state machine and a semiautomatic state machine; The fully automatic state machine is: It consists of five states: waiting, observing, tracking, lost and busy. The specific transition relationship of each state is as follows: (a) For the "waiting" state, if the input is "prohibit connection", the original state will be maintained; if the input is "connection", it will switch to the "observation" state; (b) For the "observation" state, if the input is "lost target", it will maintain the original state; if the input is "forbidden connection", it will switch to the "waiting" state; command" will switch to the "lost" state; if the input is "found target" then it will switch to the "tracking" state; (c) For the "tracking" state, if the input is "found target", it will keep the original state; if the input is "lost target", it will switch to the "lost" state; if the input is "task", it will switch to the "busy" state; if the input is " command" will switch to the "observation" state; if the input is "prohibit connection" then it will switch to the "waiting" state; (d) For the "lost" state, if the input is "lost target", it will maintain the original state; if the input is "found target", it will switch to the "tracking" state; if the input is "task", it will switch to the "busy" state; if the input is " command" to switch to the "observation" state; (e) For the "busy" state, if the input is "prohibit connection", it will switch to the "waiting" state; if the input is "task completed", it will return to the state before accepting the task according to the record, which can be "observation" state, "lost " state or "tracking" state; The semi-automatic state machine is: It consists of five states: waiting, observing, tracking, lost and busy. The specific transition relationship of each state is as follows: (a) For the "waiting" state, if the input is "prohibit connection", the original state will be maintained; if the input is "connection", it will switch to the "observation" state; (b) For the "observation" state, if the input is "lost target", it will maintain the original state; if the input is "forbidden connection", it will switch to the "waiting" state; Target found" will switch to "Tracking" state; (c) For the "tracking" state, if the input is "found target", it will keep the original state; if the input is "lost target", it will switch to the "lost" state; if the input is "task", it will switch to the "busy" state; if the input is " command" will switch to the "observation" state; if the input is "prohibit connection" then it will switch to the "waiting" state; (d) For the "lost" state, if the input is "lost target", it will maintain the original state; if the input is "found target", it will switch to the "tracking" state; if the input is "task", it will switch to the "busy" state; if the input is " command" to switch to the "observation" state; (e) For the "busy" state, if the input is "prohibit connection", it will switch to the "waiting" state; if the input is "task complete", it will switch to the "observation" state; if the input is "task complete", it will return to accept according to the record The previous state of the task can be "observed", "lost" or "tracked". 3. A kind of dynamic multi-objective cooperative tracking method based on finite state automata as claimed in claim 2, characterized in that: the behavior of said composite AI entity can be expressed as M=(Q, ∑, δ, q ₀ , F), is a mathematical model composed of the following parts: A finite set of states Q={waiting, observing, tracking, lost, busy}, that is Q = {Wait, Detect, Track, Lost, Busy}; Acceptable input set ∑, which specifies all symbols that allow input, and the finite automatic state machine performs state changes according to the input of symbols in this set, which is expressed as follows: ∑={connect, target found, target lost, connection prohibited, task, task completed, order}, That is, ∑={connect, findobj, lostobj, unconnect, work, finishwork, order} The initial state q ₀ = {Wait}, that is, q ₀ = {Wait}, the first state after the AI entity is turned on, and will remain in this state when the video device cannot be connected; The end state F={wait}, that is, F={Wait}, when the AI entity cannot continue to actively participate in the cooperative tracking, it will enter this state after disconnecting the video device. Before the AI entity physical equipment finishes all tasks, Also disconnect the video equipment first, and declare to leave the Aizhen body group; The transfer function δ is a mapping of Q×Σ→Q, which is recognized by the finite automatic state machine. 4. A kind of dynamic multi-objective cooperative tracking method based on finite state automata as claimed in claim 1, characterized in that: the behavioral state model maintained by the composite AI entity finite automatic state machine and the real-time Due to the constraints of tasks and the environment, multiple compound AI entities are separated into several groups, and the composite AI entities within the group can directly exchange information; each group has an AI entity group manager, and each group can communicate with each other. Communicate directly or indirectly through AI entity group managers; direct communication between groups is carried out through information update to stimulate information interaction and/or timed information interaction, and the indirect communication is performed in a multi-AI entity system with a server In , groups communicate through servers. 5. A method for multi-Ai entity dynamic multi-target cooperative tracking based on finite state automata as claimed in claim 4, characterized in that: the AI entity group manager obtains three types of information from the compound AI entity individual : The first category is target information, which is used to ensure that the information of the target library on the AI entity group manager or server is the latest information; The second category is the detailed information of the video equipment and the status information of the compound AI entity; the AI entity group manager regularly receives new information from the AI entity individual, and in the case of a server, the AI entity group manager according to The content of the blackboard in the update server of the updated information; The third type of information is the request information; when the composite AI entity encounters an emergency situation and the accident handling is invalid, an assistance request signal will be generated, the request sequence will be written, and the request processing module will be activated; the request processing module will let the The groups of groups carry out module scheduling to meet the request sent by a compound AI entity; when the task of a single group cannot be completed, the single group dispatches other groups to assist. 6. a kind of dynamic multi-objective cooperative tracking method based on finite state automata as claimed in claim 4, is characterized in that: described compound type entity comprises following several parts: 1) A behavioral state model maintained by a finite automatic state machine. The behavioral state model includes a planning scheme. The behavioral state model decides to select an appropriate module from the module library for the next calculation according to the current state and the planning scheme; 2) A module library, including video processing modules and modeling modules, the modules can be expanded according to requirements; video processing modules include target observation based on three-frame difference, morphological noise removal, target segmentation, target screening, target information calculation, target Merging and extraction and mean shift algorithm tracking; modeling modules include memory process for unexpected handling, predictive process and self-protection handling; 3) A communication module, used for communicating with other composite AI entities, AI entity group managers and/or servers. 7. a kind of dynamic multi-objective cooperative tracking method based on finite state automata as claimed in claim 4 is characterized in that: the negotiation scheme adopted comprises three parts of negotiation protocol, negotiation method and negotiation approach: 1) Negotiate agreement The form of the negotiation protocol includes multi-bit start and end flags, instruction length negotiation primitives, message numbers and message content; the negotiation primitives include requests, commands and information instructions, and the message content includes message recipient types, Effective information content and message sending time; 2) Negotiation method One kind of negotiation method is to activate the task allocation algorithm, competition negotiation algorithm, information calculation part and The request processing module obtains the optimal negotiation plan, and writes the result back to the task list; then the manager sends the instruction to the relevant compound AI entity with the highest priority according to the task list. Another negotiation method is that in the compound AI entity group, the individual composite AI entities exchange information, directly fuse information with each other, and negotiate to complete the task; 3) Negotiation channels Communicate through a reconfigurable point-to-point communication platform for multi-mobile robots. When the state of the compound AI entity changes, it will send the current information to the AI entity group manager, and the AI entity group manager will update the information interactively or update the server. Information: In the case of no state change, update the information of the AI entity group manager every time T, and update the status and information of other compound AI entity individuals in the group maintained by the AI entity group manager . 8. a kind of dynamic multi-target cooperative tracking method based on finite state automata as described in any one of claim 1～7, is characterized in that, comprises the following steps: The first step is the preparation of the composite AI entity: after the composite AI entity is turned on, it will carry out activities through the behavior state model maintained by the finite automatic state machine, or communicate with other composite AI entities; when the composite AI entity When it is turned on, it is in the waiting state, that is, the Wait state. When the composite AI body can connect to the video device, it will leave this state; the composite AI body that remains in this state may be due to the failure of the video device, no video device or It is designated as not allowed to conduct investigative activities, unable to self-observe changes in the external environment, and only waits to receive external instructions to carry out activities; The second step is to complete the specified task: when the compound AI entity is turned on, if it is in the group, it will start to regularly update the information on the group manager of the compound AI entity, and if there is a server, update the server information at the same time; when accepted When it comes to a designated task, if the compound AI body is in a fully automatic state, it will switch to a semi-automatic state; if the accepted task is arranged in a formation, all compound AI bodies in a non-busy state will enter the task group , distribute the tasks according to the number of compound AI entities currently completing the task, and guide the composite AI entities to complete; if the accepted task is to find or track a designated target, all currently under observation or a certain area The robot enters the task group to accept the task; it is marked by the finding of the target by a compound AI entity that accepts the task, indicating that the task is completed, and notifying other composite AI entities in the task group to give up the task; when the composite AI entity After the body accepts the task, its state is in the busy state; the state transition is converted according to the assigned instructions. If the state to be transformed is not specified, it will automatically return to the state before entering the busy state, restore the saved task point, and continue complete the interrupted work; The third step is environmental investigation: after the composite AI entity is connected to the video equipment, it will automatically enter the observation state to carry out investigation activities; the composite AI entity in the observation state will observe the moving individuals within the field of vision, calculate and record their information ; If it is connected to the server or other compound AI entities, the information will be shared and notified; if it is in a fully automatic state, then the target will be selected and tracked according to the given rules, and it will enter the tracking state, that is, the Track state; if it is semi-automatic In the fully automatic state, when the target is extracted, if the target is tracked, it will meet the criteria of first-come-first-track, the largest tracking viewing area, and the closest distance between the target and the compound AI entity , select the optimal target to switch to the tracking state for tracking, and other targets can perform basic visual tracking; if the target is not tracked, return to the observation state and continue to observe; The fourth step, target tracking: the compound AI entity obtains a large amount of data information from the external environment through the sensor, analyzes the data through the video processing module in the module library, obtains the tracked target information, and uses the modeling module to analyze the data. The information is stored and calculated; the tracking is mainly realized by the color-based mean shift algorithm; when the tracked target is lost, it enters the target loss state, that is, the Lost state; The fifth step, accident handling: including memory process and prediction process; 1) Memory process, memory is a process in which a compound AI entity keeps known information in storage while tracking the state, which is divided into short-term memory and long-term memory; short-term memory will remember the most recent actions , and the long-term memory will remember the moving path of the composite AI entity and the target during the whole process of tracking a target; according to the obtained information, the composite AI entity and the target movement can be calculated by curve fitting The path information is summarized; the fitting on the client side only considers the most recent part of the data, and performs segmented curve fitting on it; the compound AI entity manager or server side will calculate the overall data; 2) In the prediction process, when the area of the calculated target is less than a certain threshold, the target is considered lost; then it enters the lost state, that is, the Lost state; the short-term memory generally does not exceed three actions, and the most recent action plays the main role, and the others play a role. Auxiliary inspection function; if the target is not found again through the guidance of short-term memory, then the target trajectory fitted by the polynomial curve of long-term memory will guide the composite AI entity to turn to the predicted angle for observation; when the composite AI The real body has not found the target within a certain time limit, and if it thinks that the target is really lost, it sends out a target loss message.

Images