CN101303589A - Dynamic Multi-target Collaborative Tracking Method Based on Finite State Automata - Google Patents
Dynamic Multi-target Collaborative Tracking Method Based on Finite State Automata Download PDFInfo
- Publication number
- CN101303589A CN101303589A CNA2008100315434A CN200810031543A CN101303589A CN 101303589 A CN101303589 A CN 101303589A CN A2008100315434 A CNA2008100315434 A CN A2008100315434A CN 200810031543 A CN200810031543 A CN 200810031543A CN 101303589 A CN101303589 A CN 101303589A
- Authority
- CN
- China
- Prior art keywords
- state
- entity
- target
- information
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种基于有限状态自动机的多艾真体动态多目标协作跟踪方法,其特征在于:复合式艾真体根据通过自身探测的环境信息I、服务器或其他艾真体或艾真体群体管理者传达的需要完成的任务信息M、和/或服务器传达的人为的指定信息H,在多个有限自动状态机中,选择一个有限自动状态机作为用以维持该复合式艾真体行为状态模型的有限自动状态机。该艾真体以行为状态及情感信息等因素为驱动,通过信息交互或者结合服务器团队协调,进行集中式控制或艾真体个体信息交互控制。本发明可适用于集中式、分布式、混合式等不同体系结构。
The invention discloses a dynamic multi-target cooperative tracking method based on a finite state automaton, which is characterized in that: a composite AI entity detects environment information I, a server or other AI entities or AI entities The task information M that needs to be completed conveyed by the group manager, and/or the artificially specified information H conveyed by the server, among multiple finite automatic state machines, select a finite automatic state machine as the Finite Automatic State Machines for Behavioral State Models. The AI entity is driven by factors such as behavior status and emotional information, and performs centralized control or individual information interaction control of the AI entity through information interaction or coordination with the server team. The present invention is applicable to different architectures such as centralized, distributed and hybrid.
Description
技术领域 technical field
本发明属于机器人导航和应用领域,涉及一种基于有限状态自动机的多艾真体动态多目标协作跟踪方法。The invention belongs to the field of robot navigation and application, and relates to a dynamic multi-target cooperative tracking method based on finite state automata.
背景技术 Background technique
现代任务的复杂化和多样化使得对多机器人结合和团队任务的完成有了较高的需求,协同多元艾真体如何通过已具备的视频获取设备,通过信息交互与融合,对环境进行侦察,通过视觉来指导行为的能力就特别重要。但是,目前在这方面并未存在一个通用性强的跟踪方法,可供广泛应用。The complexity and diversification of modern tasks have put a higher demand on the combination of multi-robots and the completion of team tasks. How to cooperate with multi-robots to conduct reconnaissance on the environment through existing video acquisition equipment, information interaction and fusion, The ability to guide behavior through vision is particularly important. However, there is currently no general tracking method in this regard that can be widely used.
在一个未知环境中,要使用一个具有视觉设备的大型机器人团队协作观测与跟踪动态多目标,需要异步的通过多机器人上的艾真体模型对环境实时地进行观察,并解决局部问题,同时需要整个多艾真体系统之间的同步与通信,保证信息的实时性和准确性,并根据全局信息进行决策。同步方面可以通过协议中的时间戳,多艾真体群体信息交互,或者结合服务器端的全局监控和管理来进行同步,而异步则使用基于有限状态自动机的艾真体模型,通过状态的定位,独立于动作控制平台,可调动不带视频设备的异质多艾真体,分布式计算信息,协同完成任务。这两者的结合,较好的折中解决了信息和决策的同步和异步问题,增加机器人团队对环境的自适应性,有利于机器人团队高效协同合作,保证侦察信息的全面性、准确性和快速性。In an unknown environment, to use a large-scale robot team with visual equipment to observe and track dynamic multi-targets collaboratively, it is necessary to observe the environment in real time through the AI real-time model on the multi-robots asynchronously and solve local problems. Synchronization and communication between the entire multi-ai entity system ensures the real-time and accuracy of information, and makes decisions based on global information. In terms of synchronization, it can be synchronized through the time stamp in the protocol, the information interaction of multiple AI entity groups, or combined with the global monitoring and management on the server side, while asynchronously uses the AI entity model based on finite state automata. Through the positioning of the state, Independent of the motion control platform, it can mobilize heterogeneous multi-ai entities without video equipment, distribute computing information, and complete tasks collaboratively. The combination of the two solves the synchronous and asynchronous problems of information and decision-making in a good compromise, increases the adaptability of the robot team to the environment, is conducive to the efficient cooperation of the robot team, and ensures the comprehensiveness, accuracy and accuracy of the reconnaissance information. Rapidity.
发明内容 Contents of the invention
本发明所要解决的技术问题是,提供一种基于有限状态自动机的多艾真体动态多目标协作跟踪方法。The technical problem to be solved by the present invention is to provide a dynamic multi-target cooperative tracking method based on finite state automata.
本发明为解决上述技术问题所采用的技术方案是:The technical scheme that the present invention adopts for solving the problems of the technologies described above is:
一种基于有限状态自动机的多艾真体动态多目标协作跟踪方法,其特征在于:复合式艾真体根据通过自身探测的环境信息I、服务器或其他艾真体或艾真体群体管理者传达的需要完成的任务信息M、和/或服务器传达的人为的指定信息H,在多个有限自动状态机中,选择一个有限自动状态机作为用以维持该复合式艾真体行为状态模型的有限自动状态机。A dynamic multi-objective cooperative tracking method based on finite state automata, characterized in that: the compound AI entity is based on the environment information I, server or other AI entities or AI entity group managers detected by itself The conveyed task information M that needs to be completed, and/or the artificially specified information H conveyed by the server, among multiple finite automatic state machines, select a finite automatic state machine as the one used to maintain the compound AI entity behavior state model Finite automatic state machine.
所述的有限自动状态机为全自动状态机和半自动状态机;全自动状态机为:Described finite automatic state machine is full-automatic state machine and semi-automatic state machine; Full-automatic state machine is:
由等待、观测、跟踪、丢失和忙碌5个状态组成,各状态的具体转换关系如下:It consists of five states: waiting, observing, tracking, lost and busy. The specific transition relationship of each state is as follows:
(a)对于“等待”状态,输入为“禁止连接”则保持原状态;输入为“连接”则切换到“观测”状态;(a) For the "waiting" state, if the input is "prohibit connection", the original state will be maintained; if the input is "connection", it will switch to the "observation" state;
(b)对于“观测”状态,输入为“丢失目标”则保持原状态;输入为“禁止连接”则切换到“等待”状态;输入为“任务”则切换到“忙碌”状态;输入为“命令”则切换到“丢失”状态;输入为“发现目标”则切换到“跟踪”状态;(b) For the "observation" state, if the input is "lost target", it will maintain the original state; if the input is "forbidden connection", it will switch to the "waiting" state; command" will switch to the "lost" state; if the input is "found target" then it will switch to the "tracking" state;
(c)对于“跟踪”状态,输入为“发现目标”则保持原状态;输入为“丢失目标”则切换到“丢失”状态;输入为“任务”则切换到“忙碌”状态;输入为“命令”则切换到“观测”状态;输入为“禁止连接”则切换到“等待”状态;(c) For the "tracking" state, if the input is "found target", it will keep the original state; if the input is "lost target", it will switch to the "lost" state; if the input is "task", it will switch to the "busy" state; if the input is " command" will switch to the "observation" state; if the input is "prohibit connection" then it will switch to the "waiting" state;
(d)对于“丢失”状态,输入为“丢失目标”则保持原状态;输入为“发现目标”则切换到“跟踪”状态;输入为“任务”则切换到“忙碌”状态;输入为“命令”则切换到“观测”状态;(d) For the "lost" state, if the input is "lost target", it will maintain the original state; if the input is "found target", it will switch to the "tracking" state; if the input is "task", it will switch to the "busy" state; if the input is " command" to switch to the "observation" state;
(e)对于“忙碌”状态,输入为“禁止连接”则切换到“等待”状态;输入为“任务完成”则根据记录,回复到接受任务之前的状态,可以为“观测”状态,“丢失”状态或者是“跟踪”状态;(e) For the "busy" state, if the input is "prohibit connection", it will switch to the "waiting" state; if the input is "task completed", it will return to the state before accepting the task according to the record, which can be "observation" state, "lost " state or "tracking" state;
半自动状态机为:The semi-automatic state machine is:
由等待、观测、跟踪、丢失和忙碌5个状态组成,各状态的具体转换关系如下:It consists of five states: waiting, observing, tracking, lost and busy. The specific transition relationship of each state is as follows:
(a)对于“等待”状态,输入为“禁止连接”则保持原状态;输入为“连接”则切换到“观测”状态;(a) For the "waiting" state, if the input is "prohibit connection", the original state will be maintained; if the input is "connection", it will switch to the "observation" state;
(b)对于“观测”状态,输入为“丢失目标”则保持原状态;输入为“禁止连接”则切换到“等待”状态;输入为“任务”则切换到“忙碌”状态;输入为“发现目标”则切换到“跟踪”状态;(b) For the "observation" state, if the input is "lost target", it will maintain the original state; if the input is "forbidden connection", it will switch to the "waiting" state; Target found" will switch to "Tracking" state;
(c)对于“跟踪”状态,输入为“发现目标”则保持原状态;输入为“丢失目标”则切换到“丢失”状态;输入为“任务”则切换到“忙碌”状态;(c) For the "tracking" state, if the input is "found target", it will maintain the original state; if the input is "lost target", it will switch to the "lost" state; if the input is "task", it will switch to the "busy" state;
输入为“命令”则切换到“观测”状态;输入为“禁止连接”则切换到“等待”状态;If the input is "command", it will switch to the "observation" state; if the input is "prohibit connection", it will switch to the "waiting" state;
(d)对于“丢失”状态,输入为“丢失目标”则保持原状态;输入为“发现目标”则切换到“跟踪”状态;输入为“任务”则切换到“忙碌”状态;(d) For the "lost" state, if the input is "lost target", it will maintain the original state; if the input is "found target", it will switch to the "tracking" state; if the input is "task", it will switch to the "busy" state;
输入为“命令”则切换到“观测”状态;If the input is "command", it will switch to the "observation" state;
(e)对于“忙碌”状态,输入为“禁止连接”则切换到“等待”状态;输入为“任务完成”则根据记录,回复到接受任务之前的状态,可以为“观测”状态,“丢失”状态或者是“跟踪”状态。(e) For the "busy" state, if the input is "prohibit connection", it will switch to the "waiting" state; if the input is "task completed", it will return to the state before accepting the task according to the record, which can be "observation" state, "lost " state or "tracking" state.
所述的复合式艾真体的行为可表示为M=(Q,∑,δ,q0,F),是一个由以下几部分组成的数学模型:The behavior of the compound type AI entity can be expressed as M=(Q, ∑, δ, q 0 , F), which is a mathematical model consisting of the following parts:
一个状态的有穷集合Q={等待,观测,跟踪,丢失,忙碌},即A finite set of states Q={waiting, observing, tracking, lost, busy}, that is
Q={Wait,Detect,Track,Lost,Busy};Q = {Wait, Detect, Track, Lost, Busy};
可接受的输入集合∑,它指明了所有允许输入的符号,有限自动状态机根据这个集合中的符号的输入,进行状态的变化,其表示如下:Acceptable input set ∑, which specifies all symbols that allow input, and the finite automatic state machine performs state changes according to the input of symbols in this set, which is expressed as follows:
∑={连接,发现目标,丢失目标,禁止连接,任务,任务完成,命令},∑={connect, target found, target lost, connection prohibited, task, task completed, order},
即∑={connect,findobj,lostobj,unconnect,work,finishwork,order}That is, ∑={connect, findobj, lostobj, unconnect, work, finishwork, order}
起始状态q0={等待},即q0={Wait},艾真体开启之后的第一个状态,在无法连接视频设备的情况下,将维持在这个状态;The initial state q 0 = {Wait}, that is, q 0 = {Wait}, the first state after the AI entity is turned on, and will remain in this state when the video device cannot be connected;
结束状态F={等待},即F={Wait},当艾真体无法继续主动参与完成协作跟踪的情况下,断开视频设备后进入该状态,在艾真体物理设备结束一切任务之前,也首先断开视频设备,宣告离开艾真体群体;The end state F={wait}, that is, F={Wait}, when the AI entity cannot continue to actively participate in the cooperative tracking, it will enter this state after disconnecting the video device. Before the AI entity physical equipment finishes all tasks, Also disconnect the video equipment first, and declare to leave the Aizhen body group;
转移函数δ是Q×∑→Q的一个映射,由所述的有限自动状态机所识别。The transfer function δ is a mapping of Q×Σ→Q, which is recognized by the finite automatic state machine.
通过复合式艾真体有限自动状态机所维持行为状态模型以及实时的任务和环境的约束,将多个复合式艾真体分离成若干群体,群体内部复合式艾真体可以直接进行信息的交流;每个群体有一个艾真体群体管理者,各个群体之间可通过艾真体群体管理者直接或间接进行交流;群体之间的直接交流是通过信息更新激发信息交互和/或定时信息交互来进行,所述的间接交流是在有服务器的多艾真体系统中,群体间通过服务器进行交流。Through the behavior state model maintained by the compound AI agent finite automatic state machine and the constraints of real-time tasks and environments, multiple composite AI agents are separated into several groups, and the composite AI agents within the group can directly exchange information ; Each group has an AI entity group manager, and each group can communicate directly or indirectly through the AI entity group manager; the direct communication between groups is to stimulate information interaction and/or timing information interaction through information update The indirect communication mentioned above is in the multi-ai entity system with a server, and the groups communicate through the server.
艾真体群体管理者从复合式艾真体个体中获取三类信息:The AI-entity group manager obtains three types of information from the composite AI-entity individuals:
第一类为目标信息,用于保证艾真体群体管理者或服务器上的目标库的信息为最新信息;The first category is target information, which is used to ensure that the information of the target library on the AI entity group manager or the server is the latest information;
第二类为视频设备的详细信息和复合式艾真体个体的状态信息;艾真体群体管理者定时从艾真体个体接收新的信息,在有服务器情况下,艾真体群体管理者根据所更新信息的更新服务器中黑板的内容;The second category is the detailed information of the video equipment and the status information of the compound AI entity; the AI entity group manager regularly receives new information from the AI entity individual, and in the case of a server, the AI entity group manager according to The content of the blackboard in the update server of the updated information;
第三类信息为请求信息;当复合式艾真体个体遇到紧急情况,在意外处理无效的情况下,会发生援助请求信号,写入请求序列,激发请求处理模块;请求处理模块会让所在的群体进行模块调度,满足某个复合式艾真体发送的请求;在单个群体任务无法完成时,该单个群体再调度其他群体进行协助。The third type of information is the request information; when the composite AI entity encounters an emergency situation and the accident handling is invalid, an assistance request signal will be generated, the request sequence will be written, and the request processing module will be activated; the request processing module will let the The groups of groups carry out module scheduling to meet the request sent by a compound AI entity; when the task of a single group cannot be completed, the single group dispatches other groups to assist.
所述的复合式艾真体包括如下几个部分:Described compound formula body comprises following several parts:
1)一个用有限自动状态机所维持的行为状态模型,行为状态模型中包括规划方案,行为状态模型根据当前的状态和规划方案决定从模块库中选择合适的模块进行下一步的计算;1) A behavioral state model maintained by a finite automatic state machine. The behavioral state model includes a planning scheme. The behavioral state model decides to select an appropriate module from the module library for the next calculation according to the current state and the planning scheme;
2)一个模块库,包括视频处理模块和建模模块,模块可根据需求进行扩展;视频处理模块包括基于三帧差的目标检测、形态学去噪声、目标分割、目标筛选、目标信息计算、目标合并与提取以及均值偏移算法跟踪;建模模块包括意外处理的记忆过程、预测过程和自保护处理;2) A module library, including video processing modules and modeling modules, the modules can be expanded according to requirements; video processing modules include target detection based on three-frame difference, morphological noise removal, target segmentation, target screening, target information calculation, target Merging and extraction and mean shift algorithm tracking; modeling modules include memory process for unexpected handling, predictive process and self-protection handling;
3)通讯模块,用于与其他复合式艾真体、艾真体群体管理者和/或服务器通信。3) A communication module, used for communicating with other compound AI entities, AI entity group managers and/or servers.
所采用的协商方案包括协商协议、协商方法和协商途径三个部分:The negotiation scheme adopted includes three parts: negotiation agreement, negotiation method and negotiation approach:
1)协商协议1) Negotiate agreement
协商协议的形式包括多位起始和结束标志、指令长度协商元语、消息编号和消息内容;所述的协商元语包括请求、命令和信息指令,所述的消息内容包括消息接收者类型、有效信息内容和消息发送时间;The form of the negotiation protocol includes multi-bit start and end flags, instruction length negotiation primitives, message numbers and message content; the negotiation primitives include requests, commands and information instructions, and the message content includes message recipient types, Effective information content and message sending time;
2)协商方法2) Negotiation method
一种协商方法是通过艾真体群体管理者或服务器提供的机器人状态列表和全局目标库、决策选择、任务列表和请求序列信息的实时变化,激活任务分配算法、竞争协商算法、信息计算部分和请求处理模块,获得最优协商方案,将结果回写任务列表;然后管理者根据任务列表以最高优先权将指令发送给相关的复合式艾真体个体。One kind of negotiation method is to activate the task allocation algorithm, competition negotiation algorithm, information calculation part and The request processing module obtains the optimal negotiation plan, and writes the result back to the task list; then the manager sends the instruction to the relevant compound AI entity with the highest priority according to the task list.
另外一种协商方法是在复合式艾真体群体中,复合式艾真体个体之间交互信息,相互直接信息融合,协商完成任务;Another negotiation method is that in the compound AI entity group, the individual composite AI entities exchange information, directly fuse information with each other, and negotiate to complete the task;
3)协商途径3) Negotiation channels
通过可重构的多移动机器人点对点通信平台进行通讯,复合式艾真体个体在状态变化时会将当前信息发送给艾真体群体管理者,艾真体群体管理者交互更新信息,或更新服务器信息;在没有状态变化的情况下,每隔时间T更新一次艾真体群体管理者的信息,同时更新一次艾真体群体管理者所维护的群体内其他复合式艾真体个体的状态和信息。Communicate through a reconfigurable point-to-point communication platform for multi-mobile robots. When the state of the compound AI entity changes, it will send the current information to the AI entity group manager, and the AI entity group manager will update the information interactively or update the server. Information: In the case of no state change, update the information of the AI entity group manager every time T, and at the same time update the status and information of other compound AI entity individuals in the group maintained by the AI entity group manager .
所述的基于有限状态自动机的多艾真体动态多目标协作跟踪方法,包括以下步骤:The described dynamic multi-objective cooperative tracking method based on finite state automata comprises the following steps:
第一步,复合式艾真体准备:复合式艾真体开启之后,通过有限自动状态机维持的行为状态模型进行活动,或和其他复合式艾真体进行交流;当该复合式艾真体开启时,处于等待状态,即wait状态,当复合式艾真体能够连接视频设备后,就离开这个状态;维持在这个状态的复合式艾真体个体,可能由于视频设备故障,没有视频设备或者是被指定为不允许进行侦查活动,无法自我观测外界环境变化,只等待接收外界指令来进行活动;The first step is the preparation of the composite AI entity: after the composite AI entity is turned on, it will carry out activities through the behavior state model maintained by the finite automatic state machine, or communicate with other composite AI entities; when the composite AI entity When it is turned on, it is in the waiting state, that is, the wait state. When the composite AI body can connect to the video device, it will leave this state; the composite AI body that remains in this state may be due to the failure of the video device, no video device or It is designated as not allowed to conduct investigative activities, unable to self-observe changes in the external environment, and only waits to receive external instructions to carry out activities;
第二步,指定任务完成:当复合式艾真体开启之后,如果在群体内,则开始定时更新复合式艾真体群体管理者上的信息,如果有服务器,则同时更新服务器信息;当接受到一个指定任务时,复合式艾真体如果是全自动状态,则会转换成半自动状态;接受的任务如果是队形排列,则所有处于非忙碌状态的复合式艾真体都进入任务群组,将任务根据当前完成任务的复合式艾真体个数来进行分配,并指导复合式艾真体完成;如果接受的任务是指定目标查找或跟踪,则所有当前处于观测状态的或某个区域的机器人进入任务群组接受任务;以接受任务的某一个复合式艾真体找到目标为标志,表示任务完成,并通知任务群组中其他复合式艾真体放弃该任务;当复合式艾真体接受任务后,其状态为忙碌状态;状态的转换根据分配的指令来转换,在没有指定要转换的状态的情况下,则自动返回到进入忙碌状态之前的状态,恢复保存的任务点,继续完成被中断的工作;The second step is to complete the specified task: when the compound AI entity is turned on, if it is in the group, it will start to regularly update the information on the group manager of the compound AI entity, and if there is a server, update the server information at the same time; when accepted When it comes to a designated task, if the compound AI body is in a fully automatic state, it will switch to a semi-automatic state; if the accepted task is arranged in a formation, all compound AI bodies in a non-busy state will enter the task group , distribute the tasks according to the number of compound AI entities currently completing the task, and guide the composite AI entities to complete; if the accepted task is to find or track a designated target, all currently under observation or a certain area The robot enters the task group to accept the task; it is marked by the finding of the target by a compound AI entity that accepts the task, indicating that the task is completed, and notifying other composite AI entities in the task group to give up the task; when the composite AI entity After the body accepts the task, its state is in the busy state; the state transition is converted according to the assigned instructions. If the state to be transformed is not specified, it will automatically return to the state before entering the busy state, restore the saved task point, and continue complete the interrupted work;
第三步,环境侦查:复合式艾真体个体连接视频设备后,自动进入观测状态进行侦查活动;处于观测状态的复合式艾真体个体,观察视野范围内移动个体,计算和记录它们的信息;如果联系到服务器或其他复合式艾真体,则将信息共享和通知;若是全自动状态,则根据给定的规则对目标进行选取和跟踪,并转入跟踪状态,即Track状态;若是半自动状态,根据授权进行动作;在全自动状态,当目标被提取出来之后,如果跟踪到了目标,则以满足先到先跟踪、跟踪可视面积最大、目标与复合式艾真体个体距离最近的标准,选取最优目标进行动作转换到跟踪状态进行跟踪,其他目标可进行基本的视觉跟踪;如果没有跟踪到目标,则回到观测状态继续观测;The third step is environmental investigation: after the composite AI entity is connected to the video equipment, it will automatically enter the observation state to carry out investigation activities; the composite AI entity in the observation state will observe the moving individuals within the field of vision, calculate and record their information ; If it is connected to the server or other compound AI entities, the information will be shared and notified; if it is in a fully automatic state, then the target will be selected and tracked according to the given rules, and it will enter the tracking state, that is, the Track state; if it is semi-automatic In the fully automatic state, when the target is extracted, if the target is tracked, it will meet the criteria of first-come-first-track, the largest tracking viewing area, and the closest distance between the target and the compound AI entity , select the optimal target to switch to the tracking state for tracking, and other targets can perform basic visual tracking; if the target is not tracked, return to the observation state and continue to observe;
第四步,目标跟踪:复合式艾真体个体通过感知器从外界环境获取大量数据信息,通过模块库中视频处理模块对数据进行分析,得到跟踪的目标信息,并通过建模模块对分析后的信息进行存储和计算;跟踪主要通过基于颜色的均值偏移算法来实现;当所跟踪的目标丢失之后,则进入目标丢失状态,即Lost状态;The fourth step, target tracking: the compound AI entity obtains a large amount of data information from the external environment through the sensor, analyzes the data through the video processing module in the module library, obtains the tracked target information, and uses the modeling module to analyze the data. The information is stored and calculated; the tracking is mainly realized by the color-based mean shift algorithm; when the tracked target is lost, it enters the target loss state, that is, the Lost state;
第五步,意外处理:包括记忆过程和预测过程;The fifth step, accident handling: including memory process and prediction process;
1)记忆过程,记忆是复合式艾真体个体在跟踪状态的同时,将已知信息在存储中进行保存的一个过程,分为短期记忆和长期记忆;短期记忆会记住最近所做的动作,而长期记忆会记住在跟踪一个目标的整个过程中,所得到的复合式艾真体个体和目标移动的路径;根据获取的信息,通过曲线拟合对复合式艾真体个体和目标移动的路径的信息进行归纳;在客户端进行拟合只考虑最近部分数据,对其进行分段的曲线拟合;复合式艾真体管理者或服务器端会对总体的数据进行计算;1) Memory process, memory is a process in which a compound AI entity keeps known information in storage while tracking the state, which is divided into short-term memory and long-term memory; short-term memory will remember the most recent actions , and the long-term memory will remember the moving path of the composite AI entity and the target during the whole process of tracking a target; according to the obtained information, the composite AI entity and the target movement can be calculated by curve fitting The path information is summarized; the fitting on the client side only considers the most recent part of the data, and performs segmented curve fitting on it; the compound AI entity manager or server side will calculate the overall data;
2)预测过程,当计算目标的面积小于某个阈值之后,认为目标丢失;则进入丢失状态,即Lost状态;短期记忆一般不超过三个动作,起主要作用的是最近的动作,其他的起辅助检验作用;如果通过短期记忆引导,没有再次找到目标,则通过长期记忆的多项式曲线拟合出的目标轨迹,指导复合式艾真体转到所预测到的角度来进行观测;当复合式艾真体在一定时限内还没有找到目标,认为目标确实丢失了,则发出目标丢失信息。2) In the prediction process, when the area of the calculated target is less than a certain threshold, the target is considered lost; then it enters the lost state, that is, the Lost state; the short-term memory generally does not exceed three actions, and the most recent action plays the main role, and the others play a role. Auxiliary inspection function; if the target is not found again through the guidance of short-term memory, then the target trajectory fitted by the polynomial curve of long-term memory will guide the composite AI entity to turn to the predicted angle for observation; when the composite AI If the real body has not found the target within a certain time limit, and thinks that the target is really lost, it sends out a target loss message.
本发明的有益效果有:The beneficial effects of the present invention have:
本发明所提出的基于有限状态自动机的多艾真体动态多目标协作跟踪方法,独立于动作控制平台,支持集中式控制和艾真体个体信息交互控制。复合式多艾真体的提出,将适用于不同的多艾真体系统体系结构,适合多机器人团队的多目标协作跟踪。各个艾真体能独立工作,还能协同完成任务,由于按照任务分成不同的艾真体群体,因此工作效率更高。有服务器参与时,服务器即为总的指挥中枢,从全局上指挥各个艾真体或群体协调配合。The finite-state automata-based dynamic multi-target cooperative tracking method for multiple AI entities proposed by the present invention is independent of the action control platform, and supports centralized control and interactive control of AI entity individual information. The proposal of the composite multi-ai body will be applicable to different multi-ai body system architectures, and is suitable for multi-target collaborative tracking of multi-robot teams. Each AI entity can work independently, and can also cooperate to complete the task. Since it is divided into different AI entity groups according to the task, the work efficiency is higher. When there is a server participating, the server is the general command center, commanding the coordination and cooperation of each entity or group from the overall situation.
附图说明 Description of drawings
图1为本发明中基于有限状态自动机的多艾真体动态多目标协作跟踪方法的抽象应用;Fig. 1 is the abstract application of the dynamic multi-objective cooperative tracking method based on finite state automaton in the present invention;
图2为本发明中复合式艾真体模型的设计构架;Fig. 2 is the design framework of compound type AI phantom model among the present invention;
图3为本发明中基于有限状态自动机的多艾真体动态多目标协作跟踪方法行为模型的状态与模块之间的详细调用关系示意图;Fig. 3 is the detailed invocation relation schematic diagram between the state and the module of the behavioral model of the dynamic multi-objective cooperative tracking method based on finite state automata in the present invention;
图4为本发明基于有限状态自动机的多艾真体动态多目标协作跟踪方法中行为模型在全自动状态下的有限自动状态机;Fig. 4 is the finite automatic state machine of the behavior model in the fully automatic state in the dynamic multi-objective cooperative tracking method of multiple entities based on the finite state automata of the present invention;
图5为本发明基于有限状态自动机的多艾真体动态多目标协作跟踪方法中行为模型在半自动状态下的有限自动状态机;Fig. 5 is the finite automatic state machine of the behavior model in the semi-automatic state in the dynamic multi-objective cooperative tracking method of multiple entities based on the finite state automata of the present invention;
图6为本发明中艾真体团队协商协议的格式;Fig. 6 is the format of the negotiation agreement of the AI entity team in the present invention;
图7为本发明中多艾真体动态多目标协作跟踪方法中行为模型的Track状态和Lost状态之间的模块流程图;Fig. 7 is the module flow chart between the Track state and the Lost state of the behavior model in the dynamic multi-target cooperative tracking method of multi-ai entities in the present invention;
具体实施方式 Detailed ways
下面结合附图和具体实施方式对本发明作进一步说明。The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
实施例1:Example 1:
本发明提出一种基于有限状态自动机的多艾真体动态多目标协作跟踪方法,其应用基础是一个多艾真体系统(Multi-Agent System,简称MAS)。该方法应用在一个可持续自主发挥作用的艾真体上,它根据环境和任务需求选择一个有限自动状态机(Deterministic Finite Automation,简称DFA)维持复合式艾真体的行为状态模型,然后结合视频设备传感器所感知的环境信息与艾真体群体中的共享资源,与其他艾真体通信协作与协商,进行建模、预测、规划、决策,指导艾真体个体自身动作控制执行器进行一定动作。本发明将艾真体个体抽象为一个复合式模型,在该模型中,对艾真体自身,有限自动状态机可根据环境和任务需求自由选择来维持行为状态,对艾真体社会群体,可从一个抽象层次来管理异质的多艾真体团队。该艾真体以行为状态及情感信息,如艾真体的跟踪疲劳度,以估值函数为衡量所反映的艾真体的个性偏好等因素为驱动,通过信息交互或者结合服务器团队协调,进行集中式控制或艾真体个体信息交互控制。该多目标协作跟踪方法可适用于集中式、分布式、混合式等不同体系结构。The present invention proposes a multi-agent dynamic multi-target cooperative tracking method based on finite state automata, and its application basis is a multi-agent system (Multi-Agent System, referred to as MAS). This method is applied to an AI entity that can continue to function autonomously. It selects a finite automatic state machine (Deterministic Finite Automation, DFA for short) according to the environment and task requirements to maintain the behavioral state model of the compound AI entity, and then combines the video The environmental information sensed by the device sensor and the shared resources in the AI entity group, communicates, collaborates and negotiates with other AI entities, performs modeling, prediction, planning, and decision-making, and guides the individual AI entities to control the actuators to perform certain actions. . The present invention abstracts the individual AI entity into a composite model. In this model, for the AI entity itself, the finite automatic state machine can freely choose according to the environment and task requirements to maintain the behavior state. For the AI entity social group, it can Manage heterogeneous teams of multiple entities from an abstraction level. The AI entity is driven by factors such as behavioral state and emotional information, such as the AI entity's tracking fatigue, and the individual preference of the AI entity reflected by the evaluation function. Through information interaction or in combination with server team coordination, Centralized control or AI entity individual information interactive control. The multi-target cooperative tracking method can be applied to different architectures such as centralized, distributed, and hybrid.
为了解决现有异质多艾真体多目标协作跟踪存在的技术问题,本发明根据有限状态自动机的特性针对该问题提出了一种基于有限状态自动机的多艾真体动态多目标协作跟踪方法,该方法独立于动作控制平台,支持集中式控制和艾真体个体信息交互控制。这种复合式多艾真体的提出,将适用于不同的多艾真体系统体系结构,给出了一种适合多机器人团队的多目标协作跟踪的方案。In order to solve the technical problems existing in the existing multi-target cooperative tracking of heterogeneous multiple entities, the present invention proposes a dynamic multi-target cooperative tracking based on finite state automata according to the characteristics of finite state automata method, which is independent of the motion control platform and supports centralized control and interactive control of individual information of AI entities. The proposal of this composite multi-ai body will be applicable to different multi-ai body system architectures, and a multi-target cooperative tracking scheme suitable for multi-robot teams is given.
各个艾真体群体从各艾真体中获取到有效的数据,将其融合,共享给所有的艾真体,并根据更新的信息和艾真体个体的任务请求,调用不同的决策,实现任务分配,协同好群体的合作,达到在未知环境动态背景动态多目标的协作检测与协作跟踪的目的;为了实现这个功能,在艾真体群体领导艾真体上,决策生成模块中包含任务分配部分,建模目标预测等知识资源,指导整体机器人群体进行动作。Each AI entity group obtains valid data from each AI entity, integrates it, and shares it with all AI entities, and calls different decisions according to the updated information and individual task requests of AI entities to realize the task Allocation, collaborative group cooperation, to achieve the purpose of collaborative detection and collaborative tracking of dynamic multi-targets in an unknown environment, dynamic background; in order to achieve this function, on the AI entity group leader AI entity, the decision generation module includes the task allocation part , knowledge resources such as modeling target prediction, and guide the overall robot group to perform actions.
艾真体个体上具有一个可在协同计算的环境中持续自主发挥作用的一个复合式艾真体模型,通过视频设备传感器感知其环境,并通过动作控制作用于该环境。它通过自己获取的环境信息,并通过艾真体群体中的共享资源和与其他艾真体通信协作与协商,选择自身使用有限自动状态机来维持行为状态模型,进行建模,预测,规划,决策,指导动作控制执行器进行动作,作用于环境,如果有服务器,则同步更新服务器上的相应资源。The ai entity has a composite ai entity model that can continue to function autonomously in a collaborative computing environment. It perceives its environment through video equipment sensors and acts on the environment through motion control. It chooses to use the finite automatic state machine to maintain the behavioral state model by itself through the environment information obtained by itself, and through the shared resources in the AI entity group and the communication, collaboration and negotiation with other AI entities, to perform modeling, prediction, planning, Decision-making, guiding actions, controlling actuators to perform actions, acting on the environment, and synchronously updating the corresponding resources on the server if there is a server.
下面结合附图对本发明作进一步的说明。The present invention will be further described below in conjunction with the accompanying drawings.
1.艾真体群体抽象1. Ai entity group abstraction
该方法通过基于有限状态自动机的艾真体模型的状态定位进行了两个层次的抽象。第一,通过独立于运动控制平台的艾真体模型将异质的多艾真体抽象成一致的艾真体个体模型;第二,通过艾真体模型中的有限自动状态机所维持行为状态模型将多个艾真体个体,通过实时的任务和环境的约束,将该多个个体分离成若干群组,每个群组有一个性能较强的管理者,然后各个群组之间可通过管理者直接或间接进行交流。而群组内部直接各艾真体可以直接进行信息的交流。整个抽象形式如图1所示。The method implements two levels of abstraction through state localization based on the Ai entity model of finite state automata. First, the heterogeneous multiple AI entities are abstracted into a consistent AI entity model through the AI entity model independent of the motion control platform; second, the behavior state is maintained by the finite automatic state machine in the AI entity model The model separates multiple AI entities into several groups through real-time tasks and environmental constraints, and each group has a manager with strong performance, and then each group can be connected through Managers communicate directly or indirectly. And within the group, each entity can directly exchange information. The entire abstract form is shown in Figure 1.
艾真体群体管理者从艾真体个体中获取三类信息。The AI entity group manager obtains three types of information from AI entity individuals.
第一类为目标信息,用于保证群体管理者或服务器上的目标库的信息为最新信息,一方面触发信息计算部分,它对被更新的目标有效的已知信息进行多项式曲线分段拟合,并修正误差,另一方面,根据决策的选择,触发任务分配算法产生新的跟踪决策,根据机器人任务列表发送指令给相关的艾真体个体。The first type is target information, which is used to ensure that the information of the group manager or the target library on the server is the latest information. On the one hand, it triggers the information calculation part, which performs polynomial curve segment fitting on the known information that is valid for the updated target. , and correct the error. On the other hand, according to the choice of decision, the task assignment algorithm is triggered to generate a new tracking decision, and instructions are sent to the relevant AI entities according to the robot task list.
第二类为视频设备的详细信息和艾真体个体的状态信息。群体管理者定时从艾真体个体接收新的信息,如果有服务器,则群体管理者根据所更新信息的更新黑板的内容。The second category is the detailed information of the video equipment and the state information of the AI entity. The group manager regularly receives new information from AI entities. If there is a server, the group manager updates the content of the blackboard according to the updated information.
第三类信息为请求信息。当艾真体个体遇到紧急情况,在意外处理无效的情况下,会发生援助请求信号,写入请求序列,激发请求处理模块。请求处理模会让所在的艾真体群体进行模块调度,满足某个艾真体发送的请求。在单个群体任务无法完成时,再调度其他群体进行协助。The third type of information is request information. When the Ai Entity encounters an emergency and the emergency handling is invalid, an assistance request signal will be generated, and the request sequence will be written to activate the request processing module. The request processing module will allow the AI entity group where it is located to perform module scheduling to meet the request sent by an AI entity. When a single group task cannot be completed, other groups are dispatched to assist.
2.复合式艾真体个体模型2. Composite Ai entity individual model
复合式艾真体按照提出的方法进行了如图2设计,包括如下几个部分:According to the proposed method, the compound AI body is designed as shown in Figure 2, including the following parts:
1)一个用DFA所维持的行为状态模型,模型中包括规划方案,模型根据当前的状态和规划方案决定从模块库中选择合适的模块进行下一步的计算;1) A behavioral state model maintained by DFA, the model includes a planning scheme, and the model decides to select an appropriate module from the module library for the next calculation according to the current state and planning scheme;
2)一个模块库,主要有视频处理模块和建模模块,模块可根据需求进行扩展;视频处理模块包括基于三帧差的目标检测,形态学去噪声,目标分割,目标筛选,目标信息计算,目标合并与提取,Meanshift跟踪(均值偏移算法)等;建模模块包括意外处理的记忆过程,预测过程和自保护处理。2) A module library, mainly including video processing modules and modeling modules, the modules can be expanded according to requirements; video processing modules include target detection based on three-frame difference, morphological noise removal, target segmentation, target screening, target information calculation, Target merging and extraction, Meanshift tracking (mean shift algorithm), etc.; the modeling module includes the memory process of unexpected processing, prediction process and self-protection processing.
3)通讯模块3) Communication module
3.艾真体的行为状态模型3. Behavior state model of AI entities
DFA维持的行为状态模型和视频处理模块和建模模块的模块库组成了行为决策层的实现实体,可表示成一个M=(Q,∑,δ,q0,F)的数学模型,详细介绍如下。The behavior state model maintained by DFA and the module library of video processing module and modeling module constitute the realization entity of behavior decision-making layer, which can be expressed as a mathematical model of M=(Q, ∑, δ, q 0 , F), which is introduced in detail as follows.
1)一个状态的有穷集合Q={Wait,Detect,Track,Lost,Busy}状态集中有五个状态:等待状态:Wait,观测状态:Detect,跟踪状态:Track,丢失跟踪状态,简称为跟踪状态:Lost,忙碌状态:Busy。详细的介绍参见行为状态模型的详细介绍。状态集中的Detect状态,Track状态和Lost状态中的规划方案对艾真体的自主行为规划进行了完整的覆盖。这三种状态的规划方案,结合模块库中提供的视频处理模块和建模模块,对获得的信息进行处理,根据结果对艾真体的行为给出自身的决策。这三种状态与它们所对应的规划方案要调用的模块详细关系如图3所示。1) A finite set of states Q={Wait, Detect, Track, Lost, Busy} There are five states in the state set: waiting state: Wait, observation state: Detect, tracking state: Track, lost tracking state, referred to as tracking Status: Lost, Busy status: Busy. For a detailed introduction, see the detailed introduction of the behavioral state model. The planning solutions in the Detect state, Track state, and Lost state in the state set completely cover the autonomous behavior planning of the AI entity. The planning scheme of these three states, combined with the video processing module and modeling module provided in the module library, processes the obtained information, and gives its own decision on the behavior of the AI agent according to the result. The detailed relationship between these three states and the modules to be called by their corresponding planning schemes is shown in Figure 3.
2)可接受的输入集合∑,它指明了所有允许输入的符号,有限自动状态机根据这个集合中的符号的输入,进行状态的变化,表示如下:2) Acceptable input set Σ, which specifies all symbols that allow input, and the finite automatic state machine performs state changes according to the input of symbols in this set, expressed as follows:
∑={连接,发现目标,丢失目标,禁止连接,任务,任务完成,命令},即∑={connection, target found, target lost, connection prohibited, task, task completed, command}, that is
∑={connect,findobj,lostobj,unconnect,work,finishwork,order}∑={connect, findobj, lostobj, unconnect, work, finishwork, order}
这个集合中包括7个可输入的符号,它们代表在实际中相应的物理事件的发生,详细介绍如下:This set includes 7 symbols that can be entered, which represent the occurrence of corresponding physical events in practice. The details are as follows:
连接connect:表示艾真体个体成功的进行可用视频设备连接;Connect connect: Indicates that the AI entity has successfully connected to available video devices;
发现目标findobj:表示艾真体个体在现有的状态下,通过当前获取的视频图像在视野可及的范围内,搜寻到一个未知或已知的目标;Find the target findobj: Indicates that the AI entity has found an unknown or known target within the range of the field of vision through the currently acquired video image in the current state;
丢失目标lostobj:表示艾真体个体在现有状态下,通过当前获取的视频图像在视野可及范围内,无法找到一个和历史信息吻合的目标;Lost target lostobj: Indicates that in the current state, the AI entity cannot find a target that matches the historical information within the range of the field of vision through the currently acquired video image;
禁止连接unconnect:表示艾真体个体在物理条件限制或整体进行资源调配时,规定其不能使用视频设备获取环境信息和主动参与协同跟踪活动;Prohibition to connect unconnect: It means that when the physical conditions limit or the overall resource allocation is restricted by the AI entity, it is stipulated that it cannot use video equipment to obtain environmental information and actively participate in collaborative tracking activities;
work:表示艾真体个体在当前状态接受指定命令,分配新的任务,在接受work指令后,艾真体进入Busy状态,不允许自身进行新的任务调配。work: Indicates that the AI entity accepts specified commands in the current state and assigns new tasks. After accepting the work command, the AI entity enters the Busy state and does not allow itself to perform new task deployment.
任务完成finishwork:表示艾真体个体完成指令队列中所有的任务。在完成任务后,艾真体恢复到接受命令之前的状态,继续完成被指定命令中断的任务;Task completion finishwork: Indicates that the AI entity has completed all the tasks in the command queue. After completing the task, Ai Zhenbody returns to the state before accepting the order, and continues to complete the task interrupted by the specified order;
命令order:表示艾真体个体接受指令,转换到新的状态,完成指定的任务。Order order: Indicates that the AI entity accepts the instruction, switches to a new state, and completes the assigned task.
3)起始状态q0={Wait},当艾真体开启之后,直接进入这个状态,如果无法连接视频设备,则维持在这个状态。3) The initial state q 0 ={Wait}, when the AI body is turned on, it will directly enter this state, and if it cannot connect to the video device, it will remain in this state.
4)结束状态F={Wait},当艾真体无法继续主动参与完成协作跟踪时,则断开视频设备,进入该状态,在艾真体物理设备结束一切任务之前之前,也首先断开视频设备,宣告离开艾真体群体。4) End state F={Wait}, when Ai-Print cannot continue to take the initiative to participate in the collaborative tracking, then disconnect the video device and enter this state. Equipment, announced to leave the group of Ai entities.
5)转移函数δ是Q×∑→Q的一个映射,被有限自动状态机所识别。有限自动状态机根据艾真体个体的行为状态分为两种,一种是全自动状态下的有限自动状态机,一种是半自动状态下的有限自动状态机。5) The transfer function δ is a mapping of Q×∑→Q, which is recognized by the finite automatic state machine. The finite automatic state machine is divided into two types according to the individual behavior state of the AI entity, one is the finite automatic state machine in the fully automatic state, and the other is the finite automatic state machine in the semi-automatic state.
全自动状态:在艾真体行为过程中,在没有其他艾真体和服务器信息支持的情况下,也能够完成自主的发现目标,进行跟踪任务,自主搜寻其他艾真体个体等工作的状态,被称为全自动状态。Fully automatic state: In the course of the behavior of the AI entity, without the support of other AI entities and server information, it can also complete the independent discovery target, perform tracking tasks, and independently search for other AI entities. Known as the fully automatic state.
半自动状态:在艾真体行为过程中,在指定协作的情形下,艾真体接受指令,跟踪搜寻指定目标,和已知的艾真体进行通讯,协作完成任务,不能随意放弃现有任务,进行其他非授权的自主行为,被称为半自动状态。Semi-automatic state: During the behavior of the AI entity, under the specified cooperation situation, the AI entity accepts the instruction, tracks and searches for the designated target, communicates with the known AI entity, and cooperates to complete the task, and cannot give up the existing task at will. Carrying out other non-authorized voluntary actions is called a semi-autonomous state.
全自动状态和半自动状态下的有限自动状态机的状态集是一致的,两个状态可以根据现实环境和任务需求自由转换。自由转换通过自身探测的环境信息I,服务器或其他艾真体传达的需要完成的任务信息M,和服务器传达的人为的指定信息H,在多种有限自动状态机中,以N=f(I,M,H)为选择指标,选择最优的自动状态机。如果当前艾真体自身探测的环境信息中包括目标信息,且艾真体群体管理者或服务器没有传达任务指令,则该艾真体以全自动状态进行运行;而如果艾真体群体管理者或服务器传达了任务指令,则选择半自动状态,以合作为目的,优先执行任务指令,任务完成后,回到全自动状态,然后进行自我决策;如果接收到人工指令信息,则处于半自动,无条件服从指令,直到任务完成,接收到其他指令为止。The state sets of the finite automatic state machine in the fully automatic state and the semi-automatic state are consistent, and the two states can be freely switched according to the actual environment and task requirements. Freely convert the environmental information I detected by itself, the task information M conveyed by the server or other entities, and the artificially specified information H conveyed by the server. In a variety of finite automatic state machines, N=f(I , M, H) is the selection index to select the optimal automatic state machine. If the environment information detected by the current ai entity itself includes target information, and the ai entity group manager or server does not transmit task instructions, the ai entity operates in a fully automatic state; and if the ai entity group manager or After the server conveys the task instructions, it will choose the semi-automatic state, and for the purpose of cooperation, it will give priority to the execution of the task instructions. After the task is completed, it will return to the fully automatic state, and then make self-decisions; , until the task is completed and other instructions are received.
有限自动状态机具体的转移函数可参见图4和图5。Refer to Figure 4 and Figure 5 for the specific transfer function of the finite automatic state machine.
4.协商方案4. Negotiation plan
该方法采用的协商方案包括协商协议、协商方法和协商途径三个部分。The negotiation scheme adopted by this method includes three parts: negotiation agreement, negotiation method and negotiation approach.
1)协商协议1) Negotiate agreement
通信是本实施例方法实现协作的关键环节,它在实现协作的过程中起到了桥梁作用。该方法中协商协议的形式如图6所示。多位起始和结束标志,指令长度,均是为了保证在网络的传输中,能完整的解析一条指令,因为当信息打包传输时,不一定一次能给出一条完整的信息,因此在连续接收的信息中,需要能将被拆分的指令还原。消息编号和消息发送时间是为了在出现网络堵塞和重发时,保证不用重复或过时的信息来更新艾真体群体管理者或服务器以及艾真体个体中的数据。Communication is a key link in realizing collaboration in the method of this embodiment, and it plays a role of a bridge in the process of realizing collaboration. The form of the negotiation protocol in this method is shown in FIG. 6 . The multi-bit start and end flags and the length of the instruction are all to ensure that an instruction can be completely parsed during network transmission, because when the information is packaged and transmitted, it may not be possible to give a complete piece of information at a time, so in continuous reception In the information, it is necessary to be able to restore the split instructions. The message number and message sending time are to ensure that no repeated or outdated information is used to update the data in the AI entity group manager or server and the AI entity individual when there is network congestion and retransmission.
2)协商方法2) Negotiation method
本发明可以采用以下两种协商方法,一种是通过艾真体群体管理者或服务器提供的机器人状态列表和全局目标库、决策选择、任务列表、请求序列等信息的实时变化,激活任务分配算法、竞争协商算法、信息计算部分和请求处理模块等知识资源,得到最优协商方案,将结果回写任务列表。然后管理者根据任务列表以最高优先权将指令发送给相关的艾真体个体。另外一种协商方法是艾真体个体和艾真体个体之间交互信息,相互直接信息融合,协商完成任务。艾真体个体之间的选择是通过艾真体群体管理者之间信息交互后或服务器根据任务进行群组划分的结果。满足相似条件的艾真体个体将被划分为一个艾真体群组(即一个艾真体群体),艾真体个体和艾真体群组中其他成员交互所需信息。The present invention can adopt the following two negotiation methods, one is to activate the task allocation algorithm through the real-time changes of the robot status list and global target library, decision selection, task list, request sequence and other information provided by the agent group manager or server , Competitive negotiation algorithm, information calculation part and request processing module and other knowledge resources, get the optimal negotiation plan, and write the result back to the task list. The manager then sends instructions to the relevant AI entities with the highest priority according to the task list. Another negotiation method is to exchange information between AI entities and AI entities, directly fuse information with each other, and negotiate to complete tasks. The selection among AI entities is the result of the information exchange between AI entity group managers or the server's group division according to tasks. AI entities that meet similar conditions will be divided into an AI entity group (that is, an AI entity group), and the AI entity individuals and other members of the AI entity group can exchange required information.
3)协商途径3) Negotiation channels
该异质多艾真体通过可重构的多移动机器人点对点通信平台进行通讯,艾真体个体在状态变化时会将当前信息发送给艾真体群体管理者,艾真体群体管理者交互更新信息,或更新服务器信息(在存在服务器的情形下)。在没有状态变化的情况下,每隔时间T更新一次艾真体群体管理者的信息,同时更新一次它所维护的群组内其他艾真体个体的状态和信息。The heterogeneous multi-agents communicate through a reconfigurable multi-mobile robot point-to-point communication platform. When the status changes, the individual ai entities will send the current information to the ai entity group managers, and the ai entity group managers will interact and update information, or update server information (if a server exists). In the absence of state changes, update the information of the group manager of the ai entity every time T, and at the same time update the status and information of other ai entity individuals in the group it maintains.
本发明应用于在一个未知环境中,使用一个具有视觉设备的大型机器人团队上,用于进行协作观测与跟踪动态多目标,主要步骤如下:The present invention is applied to a large-scale robot team with visual equipment in an unknown environment for collaborative observation and tracking of dynamic multi-targets. The main steps are as follows:
第一步,艾真体准备:当一个艾真体开启之后,它就通过DFA维持的行为状态模型,进行活动,或和其他艾真体进行交流。当该艾真体开启时,处于Wait状态,当艾真体能够连接视频设备后,就离开这个状态。维持在这个状态的艾真体个体,可能由于视频设备故障,没有视频设备或者是被指定为不允许进行侦查活动,无法自我检测外界环境变化。只等待接收外界指令来进行活动;The first step is AI entity preparation: when an AI entity is turned on, it will carry out activities or communicate with other AI entities through the behavior state model maintained by DFA. When the AI body is turned on, it is in the Wait state, and when the AI body can connect to the video device, it will leave this state. Ai entities maintained in this state may be unable to self-detect changes in the external environment due to video equipment failure, lack of video equipment, or being designated as not allowed to conduct investigative activities. Only wait to receive external instructions to carry out activities;
第二步,指定任务完成:当艾真体开启之后,如果在群体内,则开始定时更新艾真体群体管理者上的信息,如果有服务器,则同时更新服务器信息。服务器被设计为可以接受人工干预部分的信息。当接受到一个指定任务时,艾真体如果是全自动状态,则会转换成半自动状态。接受的任务如果是队形排列,所有处于非Busy状态的艾真体都进入任务群组,将任务根据当前完成任务的艾真体个数来进行分配,并指导它们完成。如果接受的任务是指定目标查找或跟踪,则所有当前处于Detect状态的或某个区域的机器人进入任务群组,接受任务。以接受任务的某一个艾真体找到目标为标志,表示任务完成,并通知任务群组中其他机器人,放弃该任务。如果当艾真体接受任务后,则将处于在艾真体个体的状态中优先级最高的一个状态:Busy状态。说明艾真体个体当前正在完成一个被指派的工作,不可被自身的指令中断,其他的命令应当在它完成该工作后进行。状态的转换根据分配的指令来转换,在没有指定要转换的状态的情况下,则自动返回到进入忙碌状态之前的状态,恢复保存的任务点,继续完成被中断的工作。The second step is to complete the specified task: when the AI entity is turned on, if it is in the group, it will start to regularly update the information on the group manager of the AI entity, and if there is a server, it will update the server information at the same time. The server is designed to accept human intervention part of the information. When receiving a designated task, if the Ai Entity is in a fully automatic state, it will switch to a semi-automatic state. If the accepted task is arranged in a formation, all AI entities in the non-Busy state will enter the task group, and the task will be assigned according to the number of AI entities currently completing the task, and they will be guided to complete. If the accepted task is to find or track a specified target, all robots currently in the Detect state or in a certain area will enter the task group and accept the task. It is marked by the finding of the target by a certain agent accepting the task, indicating that the task is completed, and notifying other robots in the task group to give up the task. If the Ai Entity accepts the task, it will be in the state with the highest priority among the individual states of the Ai Entity: the Busy state. It shows that the Ai entity is currently completing an assigned work, which cannot be interrupted by its own instructions, and other orders should be carried out after it completes the work. The state conversion is converted according to the assigned instructions. If the state to be converted is not specified, it will automatically return to the state before entering the busy state, restore the saved task point, and continue to complete the interrupted work.
第三步,环境侦查:艾真体个体连接视频设备后,自动进入观测状态,进行侦查活动。处于观测状态的艾真体个体,观察视野范围内移动个体,计算和记录它们的信息。如果联系到服务器或其他艾真体,则将信息共享和通知。若是全自动状态,则根据给定的规则对目标进行选取和跟踪,并转入Track状态,若是半自动状态,根据授权进行动作;对于视野范围内移动个体的侦查,采用的技术是先使用三帧差背景剪除方法来对目标的提取,然后通过形态学去噪和高斯滤波,获取运动空间中的分离部分制作掩模,进行目标分割,目标合并与提取,最后进行迭代式的原始目标信息查找,并使用分水岭算法获取单个目标信息。通过定义的目标筛选的原则对它们进行筛选,其目标筛选的原则如下:The third step is environmental investigation: after Ai Zhentu is connected to the video equipment, it will automatically enter the observation state and carry out investigation activities. Ai entities in the observation state observe moving individuals within the field of vision, calculate and record their information. If the server or other entity is contacted, the information will be shared and notified. If it is in the fully automatic state, the target will be selected and tracked according to the given rules, and then transferred to the Track state. If it is in the semi-automatic state, it will act according to the authorization; for the detection of moving individuals within the field of vision, the technology used is to use three frames first The difference background clipping method is used to extract the target, and then through morphological denoising and Gaussian filtering, the separated part in the motion space is obtained to make a mask, the target is segmented, the target is merged and extracted, and finally the original target information is searched iteratively. And use the watershed algorithm to obtain individual target information. They are screened by the defined target screening principles, which are as follows:
1.目标重心点位置相近,目标色调值相似,面积大小相似,则认为它是由同一目标误分割所致,选取面积较大的一个保留,取重心点均值,色调均值。1. If the position of the center of gravity of the target is similar, the tone value of the target is similar, and the size of the area is similar, it is considered that it is caused by the wrong segmentation of the same target, and the one with a larger area is selected to keep, and the mean value of the center of gravity and the mean value of the tone are taken.
2.实际目标观测,基本不会出现RGB值为0或H值为0的情况。2. In the actual target observation, there will be basically no cases where the RGB value is 0 or the H value is 0.
3.无目标时光线影响测出的噪声一般面积很大,但是存在目标时,由于光线影响,测出的噪声信息较小。艾真体个体的物理动作也可能产生部分小面积信息,需要对这样的小面积目标进行筛选。3. When there is no target, the measured noise due to the influence of light generally has a large area, but when there is a target, the measured noise information is small due to the influence of light. The physical actions of individual AI entities may also produce some small-area information, and such small-area targets need to be screened.
进行目标筛选之后需要给目标编号,其中给目标编号需要通过艾真体群体管理者或服务器的分配。定义筛选目标之后,先根据色调值,给目标分配色调编号,然后服务器将信息编号发送给各艾真体个体,在目标库中查看是否有该色调编号目标,如果没有,则可以给目标编号为1。如果已有该色调编号的目标,则查看是否是同一个目标。通过目标已知信息,目标重心在图像中的位置,艾真体个体的位置和方向,声纳的数据信息等,可以得出目标的位置或者方向。如果在全局地图中,该目标和已知目标信息相似,则认为是一个已知目标,分配已知编号,并将其信息通知给其他跟踪此目标的艾真体个体。然后跟踪同一个目标的几个艾真体个体,在之后的过程中,几个艾真体个体将相互交流信息,对信息进行修正。若不是同一个目标,则根据已知编号,继续分配下一个编号。After target screening, it is necessary to number the target, and the number of the target needs to be assigned by the AI entity group manager or server. After defining the screening target, first assign a hue number to the target according to the hue value, and then the server sends the information number to each AI entity, check whether there is the hue number target in the target library, if not, you can assign the target number as 1. If there is already a target for that hue number, check to see if it is the same target. Through the known information of the target, the position of the center of gravity of the target in the image, the position and direction of the individual AI entity, and the data information of the sonar, the position or direction of the target can be obtained. If the target is similar to the known target in the global map, it will be considered as a known target, assigned a known number, and its information will be notified to other AI entities that track this target. Then track several Ai entities of the same target, and in the subsequent process, several Ai entities will exchange information with each other and correct the information. If it is not the same target, continue to assign the next number according to the known number.
在全自动状态,当目标被提取出来之后,如果跟踪到了目标,则以满足先到先跟踪,跟踪可视面积最大,目标与艾真体个体距离最近的标准,选取最优目标进行动作转换到Track状态进行跟踪,其他目标可进行基本的视觉跟踪。如果没有跟踪到目标,则回到Detect状态继续观测。In the fully automatic state, when the target is extracted, if the target is tracked, then the optimal target is selected for action conversion to meet the criteria of first-come-first-tracked, the largest tracking visible area, and the closest distance between the target and the AI entity. Track state for tracking, and other targets for basic visual tracking. If the target is not tracked, return to the Detect state to continue observing.
第四步,目标跟踪:艾真体行为过程中和环境交互最多的状态就是Track,跟踪状态。在此状态中,艾真体个体通过感知器从外界环境获取大量数据信息,通过模块库中视频处理模块对数据进行分析,得到跟踪的目标信息,并通过建模模块对分析后的信息进行存储和计算。由于目标的描述,采用了两个不变量:颜色信息与轮廓信息,和一个可变量区域信息来表示,因此跟踪采用的算法是主要通过基于颜色的Meanshift算法来进行实现。如果当所跟踪的目标丢失之后,则进入Lost,目标丢失状态。理论分析上,Track状态和Lost状态之间的转换确定性非常强,当目标在视野范围内,就不会丢失和状态转换。但是在一个移动的艾真体和未知的环境中,由于艾真体物理移动会产生一定的惯性,环境的光影会随着艾真体移动而发生变化,在多艾真体的环境中,遮挡也是不可忽视的问题。当状态转换确定性过强的时候,艾真体对目标跟踪的丢失率就会过高。因此,在方法中设计了意外处理的过程。当目标发生丢失时,首先通过意外处理中的记忆模型,指导进行一定的动作,进行多次查找和确认,如果对目标丢失的这个过程进行多次确认还是无法跟踪目标,则认为该目标最终丢失,进行下一步动作;The fourth step, target tracking: the state that interacts with the environment most during the behavior of the AI entity is Track, the tracking state. In this state, the AI entity obtains a large amount of data information from the external environment through the sensor, analyzes the data through the video processing module in the module library, obtains the tracking target information, and stores the analyzed information through the modeling module and calculate. Due to the description of the target, two invariants are used: color information and contour information, and a variable variable area information is used to represent, so the algorithm used for tracking is mainly realized through the color-based Meanshift algorithm. If the tracked target is lost, it enters the Lost, target lost state. In theoretical analysis, the transition between the Track state and the Lost state is very deterministic. When the target is within the field of vision, there will be no loss and state transition. But in a moving AI entity and an unknown environment, because the physical movement of the AI entity will produce a certain inertia, the light and shadow of the environment will change with the movement of the AI entity. In the environment of multiple AI entities, the occlusion It is also a problem that cannot be ignored. When the state transition is too deterministic, the loss rate of the object tracking by the agent will be too high. Therefore, the process of unexpected handling is designed in the method. When the target is lost, first use the memory model in the accident processing to guide certain actions, and perform multiple searches and confirmations. If the target is lost after multiple confirmations and the target cannot be tracked, it is considered that the target is finally lost. , proceed to the next step;
第五步,意外处理:在光线,物理惯性等各种不可预知的条件影响下,目标的意外丢失是很有可能出现的事情。因此为了处理这种意外,为艾真体个体设计了记忆和预测功能。The fifth step, accident handling: Under the influence of various unpredictable conditions such as light and physical inertia, the accidental loss of the target is very likely. Therefore, in order to deal with such accidents, memory and prediction functions are designed for Ai entity individuals.
(1)记忆过程,记忆是艾真体个体在Track状态的同时,将已知信息在容器中进行保存的一个过程,分为短期记忆和长期记忆。短期记忆会记住最近所做的动作,而长期记忆会记住在跟踪一个目标的整个过程中,所得到的艾真体个体和目标移动的路径。在获取到了一定量信息后,可以使用曲线拟合对艾真体个体和目标移动的路径的信息进行归纳。在客户端,由于艾真体个体性能限制和图像数据处理,通信等大量任务,以及目标运动的轨迹不确定性,进行拟合只考虑最近部分数据,对其进行分段的曲线拟合。艾真体管理者或服务器端会通过总体的数据进行更详细的计算(1) Memory process. Memory is a process in which an individual AI entity stores known information in a container while in the Track state. It is divided into short-term memory and long-term memory. Short-term memory remembers actions recently made, while long-term memory remembers the resulting entity and the path the target traveled throughout the course of tracking a target. After a certain amount of information is obtained, the curve fitting can be used to summarize the information of the moving path of the entity and the target. On the client side, due to the individual performance limitations of AI entities and a large number of tasks such as image data processing and communication, as well as the trajectory uncertainty of the target movement, only the nearest part of the data is considered for fitting, and segmented curve fitting is performed on it. Ai entity managers or servers will perform more detailed calculations through overall data
(2)预测过程,当计算目标的面积小于某个阈值之后,可认为目标丢失。则进入Lost状态。如果丢失目标后,没有通过短期记忆来进行临时动作的话,则先完成一个临时动作,再进行观测。短期记忆一般不超过三个动作,起主要作用的是最近的动作,其他的起辅助检验工作。如果通过短期记忆引导,没有再次找到目标,则通过长期记忆的多项式曲线拟合出的目标轨迹,指导艾真体转到所预测到的角度来进行观测。当艾真体在一定时限内还没有找到目标,则认为目标确实丢失了,则发送信息到服务器,服务器将指导艾真体群体做出相应决策与动作。(2) During the prediction process, when the calculated area of the target is smaller than a certain threshold, the target can be considered lost. Then enter the Lost state. If there is no temporary action through short-term memory after losing the target, complete a temporary action first, and then observe. The short-term memory generally does not exceed three actions, the most recent action plays a major role, and the others serve as auxiliary inspection work. If the target is not found again through the short-term memory guidance, the target trajectory fitted by the polynomial curve of the long-term memory is used to guide the entity to turn to the predicted angle for observation. When the AI entity has not found the target within a certain time limit, it considers that the target is indeed lost, and then sends a message to the server, and the server will guide the AI entity group to make corresponding decisions and actions.
Claims (8)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2008100315434A CN101303589B (en) | 2008-06-20 | 2008-06-20 | Multi-agent dynamic multi-target collaboration tracking method based on finite-state automata |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN2008100315434A CN101303589B (en) | 2008-06-20 | 2008-06-20 | Multi-agent dynamic multi-target collaboration tracking method based on finite-state automata |
Publications (2)
Publication Number | Publication Date |
---|---|
CN101303589A true CN101303589A (en) | 2008-11-12 |
CN101303589B CN101303589B (en) | 2011-04-13 |
Family
ID=40113518
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN2008100315434A Expired - Fee Related CN101303589B (en) | 2008-06-20 | 2008-06-20 | Multi-agent dynamic multi-target collaboration tracking method based on finite-state automata |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN101303589B (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101937341A (en) * | 2010-09-09 | 2011-01-05 | 西北工业大学 | An Autonomous Multimodal Management Approach |
CN102176122A (en) * | 2011-03-07 | 2011-09-07 | 华中科技大学 | Automatic control system based on finite automaton and control method thereof |
WO2014101558A1 (en) * | 2012-12-27 | 2014-07-03 | 深圳先进技术研究院 | Processing method and system for controlling service amounts of multiple robots |
CN104076800A (en) * | 2014-07-10 | 2014-10-01 | 哈尔滨工程大学 | UUV group task coordination system and method |
CN104246749A (en) * | 2012-04-13 | 2014-12-24 | Infnis网络公司 | Method, server, terminal device, and computer-readable recording medium for selectively removing nondeterminism of nondeterministic finite automata |
CN105637432A (en) * | 2013-08-30 | 2016-06-01 | 慧与发展有限责任合伙企业 | Identifying anomalous behavior of a monitored entity |
CN106681322A (en) * | 2016-12-21 | 2017-05-17 | 华东师范大学 | Ground autonomous mobile robot safety navigation method based on formal description |
CN109416867A (en) * | 2016-07-04 | 2019-03-01 | 索尼公司 | Robot, robot system and recording medium |
CN111596691A (en) * | 2020-07-07 | 2020-08-28 | 福州大学 | A decision modeling and collaborative control method and system for a human-in-the-loop multi-robot system |
-
2008
- 2008-06-20 CN CN2008100315434A patent/CN101303589B/en not_active Expired - Fee Related
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101937341B (en) * | 2010-09-09 | 2014-07-30 | 西北工业大学 | Autonomic multi-mode management method |
CN101937341A (en) * | 2010-09-09 | 2011-01-05 | 西北工业大学 | An Autonomous Multimodal Management Approach |
CN102176122A (en) * | 2011-03-07 | 2011-09-07 | 华中科技大学 | Automatic control system based on finite automaton and control method thereof |
CN104246749B (en) * | 2012-04-13 | 2016-01-13 | Infnis网络公司 | Probabilistic method of non deterministic finite automaton, server and end device is eliminated for selectivity |
CN104246749A (en) * | 2012-04-13 | 2014-12-24 | Infnis网络公司 | Method, server, terminal device, and computer-readable recording medium for selectively removing nondeterminism of nondeterministic finite automata |
WO2014101558A1 (en) * | 2012-12-27 | 2014-07-03 | 深圳先进技术研究院 | Processing method and system for controlling service amounts of multiple robots |
CN105637432A (en) * | 2013-08-30 | 2016-06-01 | 慧与发展有限责任合伙企业 | Identifying anomalous behavior of a monitored entity |
CN104076800A (en) * | 2014-07-10 | 2014-10-01 | 哈尔滨工程大学 | UUV group task coordination system and method |
CN109416867A (en) * | 2016-07-04 | 2019-03-01 | 索尼公司 | Robot, robot system and recording medium |
US11200787B2 (en) | 2016-07-04 | 2021-12-14 | Sony Corporation | Robot, robot system, and storage medium |
US11837064B2 (en) | 2016-07-04 | 2023-12-05 | Sony Group Corporation | Robot, robot system, and storage medium |
CN106681322A (en) * | 2016-12-21 | 2017-05-17 | 华东师范大学 | Ground autonomous mobile robot safety navigation method based on formal description |
CN111596691A (en) * | 2020-07-07 | 2020-08-28 | 福州大学 | A decision modeling and collaborative control method and system for a human-in-the-loop multi-robot system |
CN111596691B (en) * | 2020-07-07 | 2021-08-31 | 福州大学 | A decision modeling and collaborative control method and system for a human-in-the-loop multi-robot system |
Also Published As
Publication number | Publication date |
---|---|
CN101303589B (en) | 2011-04-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN101303589B (en) | Multi-agent dynamic multi-target collaboration tracking method based on finite-state automata | |
Wan et al. | Artificial intelligence for cloud-assisted smart factory | |
US11179843B2 (en) | Method for operating a robot in a multi-agent system, robot, and multi-agent system | |
Weyns | Architecture-based design of multi-agent systems | |
CN106378780A (en) | Robot system and method and server for controlling robot | |
Ukita et al. | Real-time cooperative multi-target tracking by communicating active vision agents | |
CN108449383A (en) | Distributed thin cloud computing system mobile in real time | |
CN114721806A (en) | Task scheduling and executing method and system based on digital twin | |
Hu et al. | An overview: Attention mechanisms in multi-agent reinforcement learning | |
CN117851570A (en) | Star crowd intelligent cooperative control method and system based on large language model | |
Lv et al. | Multi-robot distributed communication in heterogeneous robotic systems on 5G networking | |
CN115716278A (en) | Robot target searching method based on active sensing and interactive operation cooperation and robot simulation platform | |
CN118644055B (en) | A robot task management system based on deep reinforcement learning | |
Abbas et al. | Statistically correlated multi-task learning for autonomous driving | |
Yang et al. | Integrated communication and computing maritime networks design for green metaverse | |
CN118450529A (en) | A distributed multi-channel access method for collaborative transmission of industrial heterogeneous data | |
Dai et al. | Cooperative path planning of multi-agent based on graph neural network | |
CN113269329B (en) | Multi-agent distributed reinforcement learning method | |
Liu et al. | CoDriveVLM: VLM-Enhanced Urban Cooperative Dispatching and Motion Planning for Future Autonomous Mobility on Demand Systems | |
Pack et al. | Intelligent machine architecture for object-based system integration | |
Ahmed et al. | Efficient frontier management for collaborative active SLAM | |
Liu et al. | Manipulating complex robot behavior for autonomous and continuous operations | |
Rambow et al. | A framework for information distribution, task execution and decision making in multi-robot systems | |
CN119987380B (en) | Multi-robot elastic formation control method and system based on state machine | |
Chen et al. | Semantic Interaction Strategy of Multiagent System in Large‐Scale Intelligent Sensor Network Environment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
C17 | Cessation of patent right | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20110413 Termination date: 20110620 |