JP2009517225A - System and method for image mapping and visual attention - Google Patents

Abstract

A technology for an intelligent machine, particularly an adaptive autonomous robot is provided.
High density sensory data is mapped to a sensory self-centered sphere (SES). Find and rank relevant areas in images that form a complete visual scene on SES. Image data attention processing is best performed on each full-size image, mapping each attention position to the closest node, and summing all attention positions at each node. The attention process is repeated for each image in the sequence, and much information is available. Attention points that persist in several adjacent images will have a high activity value and are considered more prominent than attention points found in only one image. Thus, the reliability that is characterized by the position considered prominent is higher than with an alternative processing method in which attention processing is performed only once on the image restored from the foveal window located above. .
[Selection] Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is incorporated herein by reference, claims the benefit of US Provisional Patent Application Serial No. 60 / 726,033, filed Oct. 11, 2005 is there.

  A related patent is US Pat. No. 6,697,707, which relates to an architecture for robot intelligence, incorporated herein by reference.

STATEMENT The present invention relates to the United States federal government support research was carried out with the assistance from the through some NASA approval NNJ04HI19G the United States government. The US government has certain rights in this invention.

  The present invention relates to the field of intelligent machines. More specifically, the present invention relates to the field of autonomous adaptive robots.

  An autonomous robot is a robot that works entirely independently by considering its own situation in its own environment and can determine what action should be taken without human intervention to achieve its goal. A robot is adaptive if it can improve its capabilities to achieve its goals.

  Autonomous adaptive robots must be able to sense the environment and interact with it. Therefore, the robot needs to include sensors and actuators. A sensor is any device that can generate a signal that can be mapped to a characteristic of the environment. The sensor may be a self-accepting sensor that measures aspects of the robot, such as, for example, the angle formed by two members at a joint or the angular velocity of a motor shaft. The sensor can be an external receptive sensor that measures aspects external to the robot, such as, for example, light intensity from a certain direction or presence of force applied to the robot. An actuator is any device that allows a robot to perform actions in whole or in part. The physical state of the robot can be described by the (S + A) dimensional state vector R (t), where S is the dimension of the robot sensor data and A is the dimension of the robot's actuator controller. The state vector R (t) is the only information accessible to the robot. In addition to sensors, actuators, and mechanical support structures, the robot can receive one signal from the sensor, transmit commands to the actuator, and execute one or more programs. You should have more computers than that.

  The task of building an autonomous adaptive robot is so complex that the research group concentrates on dividing the problem into several easier-to-handle tasks and solving each task independently of the others . In robotics, the three tasks or behaviors of learning, planning, and world representation are considered the most difficult.

  Initial efforts to implement these behaviors in the robot process environmental information from sensors, resulting in behavior similar to human learning, planning, and abstraction (to represent the robot world or surroundings). It was aimed at building complex programs that generate commands to actuators.

  While efforts to build a single complex control program continue, many new and surprising advances in robotics are based on the denial of the notion that complex behavior requires complex control programs. Instead, control is spread to many interacting autonomous agents. An agent is a small program that functions independently of other agents while interacting with other agents. The complex behavior that is learned or abstracted arises from the interaction of many independent agents rather than being controlled by any one agent.

  “Cambrian Intelligence: The Early History of the New AI”, published by MIT Publishing, 1999, “Learning a Learning of Diffusion Mapping Based on Navigation Behavior (Learning a "Distributed Map Representation Based on Navigation Behaviors") reveals that complex behaviors such as guided navigation to a target can emerge from a simpler interaction of behaviors called "reflections". A reflection is an agent that combines an actuator signal with a sensor signal. For example, avoidance reflection can generate a signal to the wheel motor based on the signal from the proximity sensor. If the proximity sensor detects an object in the robot's danger zone, the reflection generates a signal to stop the wheel motor. Matric and Brooks show that starting with only four reflections, target guided navigation can emerge from the interaction of these reflections. However, the reflection did not occur by the robot, but required manual coding by the programmer.

  Pfeifer, R.A. And C.I. Scheier's “Sensory-motor coordination: the metaphor and beyond”, robotics and autonomous systems, “Practice and Future of Autonomous Agents” No. 20, Vol. 2-4, pp. 157-178, 1997, the signals from sensors and actuators tend to cluster for repetitive tasks, and such clustering is called sensorimotor coordination ( It is shown to be called category formation through SMC). Cohen shows that the robot can divide the continuous data stream it receives from the sensor into episodes that can be compared to other episodes and clustered so that typical episodes are formed. Yes. A typical episode is representative of a cluster of episodes and can be determined by averaging over episodes that include each cluster. Typical episodes are self-generated (by robots) and replace external programmers. As the robot is trained, it will identify a set of typical episodes that can be used to complete the assigned task. The robot's ability to identify episodes from a continuous sensor data stream and create “categories” (typical episodes) from clustered episodes can be considered a basic form of robot learning.

  The robot should be trained to collect a sufficient number of episodes for category identification. Training is usually accomplished by reinforcement learning (RL) techniques known to those skilled in the art. In one example of an RL, the robot is allowed to generate actions randomly, while the trainer rewards the action that moves the robot toward the desired goal. Since rewards reinforce the robot's most recent actions and are similarly rewarded for similar actions, episodes corresponding to rewarded actions over time will begin to cluster. However, training requires many iterations for each action that includes the desired task.

  An autonomous robot should be able to select an action that will lead to or achieve its desired goal. One known method for robot planning is an activated diffusion network (SAN), which, when linked together, initiates a command sequence that the robot can execute to achieve a desired goal. With the ability module (CM). The capability module contains information characterizing the state of the robot both before (pre-condition state) and after (post-condition state) commands to the actuator. Capability modules are linked by adapting the pre-condition state of one CM to the post-condition state of another CM.

  The programming method begins by identifying all terminal CMs that are defined as CMs that have a post-conditional state that corresponds to the state of the robot after achieving the assigned goal. Next, each pre-condition state of the terminal CM is used to find another CM having a post-condition state that matches the pre-condition state of the terminal CM. This process is repeated until the pre-condition state of the CM corresponds to the condition of the current state of the robot.

  In one method of searching for the shortest path to a target, each CM is assigned an activity value determined by the CM that touches that CM (with matching endpoints). The order of execution is determined by the activity value of each CM, and the CM having the maximum activity value is executed next.

  As the number of CMs increases, the time required to complete a search increases very rapidly, and the robot's reaction time increases until the robot is unable to respond to dynamic changes in its environment. Such a search is considered acceptable in the plan before starting the task, but the search time increases exponentially when more CMs are added (ie when the robot learns) By doing so, such a search becomes inappropriate for the real-time response of the robot to the changing environment.

  CM link backpropagation is avoided in robot responsiveness because the robot cannot begin to execute the linked CM until a complete chain of CMs is found that moves the robot from its current state to the target state. Causes impossible delays. This unavoidable delay limits the operating environment of the robot to a normally predictable situation.

  Thus, there remains a need for an efficient method for robot planning that can react to abrupt or dynamic situations in the robot's environment while considering the addition of CM as the robot learns. .

  As with humans, the amount of sensory information received by a robot greatly exceeds the processing function of the robot. To function in any environment, a robot should be able to densify a large amount of sensor data stream to a data rate that can be processed by its processor while retaining information critical to its operation. It is. In one method of densifying the sensor data stream, the robot builds a representation of the robot's environment (world model) and compares the received sensory information against this representation stored by itself. The world model allows the robot to orient itself within its environment and allows rapid characterization of sensory data for objects in the world model.

  The world model can be others-centric or self-centric. The others-centric world model places objects in a coordinate grid that does not change with the position of the robot. Self-centered models are always centered on the current position of the robot. An example of an autocentric model is Albus, J. et al. S. "Outline for a theory of intelligence", IEEE Trans. Syst. Man, and Cybern. , Vol. 21, No. 3, 1991. Albus describes an “an Ego-sphere” in which the environment of a robot is projected onto a sphere surface centered on the current position of the robot. The “self-centered sphere” is a high-density representation of the world in the sense that all sensory information is projected onto the “self-centered sphere”. Also, Albus's “self-centered sphere” is continuous because the projection is affine. The advantage of a “self-centered sphere” is a complete representation of the world and its ability to deal with the direction of the object. However, “self-centered spheres” still require processing of sensory data streams into objects and filtering mechanisms that distinguish important objects from unimportant objects. Furthermore, Albus does not disclose or suggest any method for creating an action plan for a robot using a “self-centered sphere” or suggesting linking the “self-centered sphere” to the robot's learning mechanism. Absent.

  Another example of a self-centric model is the “sensory self-centered sphere (SES)” described in US Pat. No. 6,697,707, which is incorporated herein by reference. Again, the environment of the robot is projected onto a sphere surface centered on the current position of the robot. More specifically, in one embodiment, the SES is structured as a geodesic dome, which is a quasi-uniform triangular mosaic into a spherical polyhedron. The geodesic dome is composed of 12 pentagons and a variable number of hexagons depending on the frequency (or mosaic) of the dome. The frequency is determined by the number of vertices connecting the center of one pentagon with the center of another pentagon, and all pentagons are uniformly distributed on the dome. Illustratively, the SES has 14 mosaics and thus 1963 nodes.

  SES facilitates the detection of events in the environment that stimulate multiple sensors simultaneously. Each sensor in the robot sends information to one or more sensory processing modules (SPMs) designed to extract specific information from the data stream associated with these sensors. The SPMs are independent of each other and preferably run sequentially and simultaneously on different processors. Each SPM sends an information message to a SES manager agent that stores data including direction sense information in SES, if available. Specifically, the sensor data is stored in a node closest to the data source on the sphere (in space). For example, an object visually localized in the environment is projected onto a sphere with an azimuth and elevation corresponding to the pan and tilt angles of the camera head when the object is seen. Labels identifying objects and other related information are stored in the database. The vertex on the sphere closest to the object projection is the location where the registration node or information is stored in the database. Each message received by the SES manager is given a time stamp indicating the time when the message was received.

  SES eliminates the need to process the entire spherical projection field to find related items. Processing the entire projection field is very time consuming and reduces the ability of the robot to respond quickly to dynamic changes in the environment. Significant events are quickly identified by SES by identifying the most active areas of SES. Processing resources are only used to identify objects in the most active areas and are not wasted in areas outside the projection field or unrelated. Furthermore, because each SPM writes to the SES independently of each other, the SES can fuse or associate independent sense information written at the same vertex with little additional cost (in terms of computer resources). .

  In one embodiment, the SES vertices are uniformly distributed across the sphere surface such that the nearest neighbor distance at each vertex is approximately the same. Discretization of a continuous sphere surface into a set of vertices allows the SES agent to quickly associate independent SPM information based on the direction of each sensor source. The choice of SES size (number of vertices) can be determined by one skilled in the art by balancing the time delay increase caused by more vertices against the maximum angular resolution of the robot sensor. In a preferred embodiment, the vertices are arranged to match the vertices in the geodesic dome structure.

  FIG. 1 is an example of SES reproduced from FIG. 3 of the aforementioned 6,697,707 patent. In FIG. 1, the SES is represented as a polyhedron 300. Polyhedron 300 includes a flat triangular surface 305 with vertex 310 defining one corner of the surface. In the polyhedron of FIG. 1, each vertex has either 5 or 6 nearest neighbor vertices, and the nearest neighbor distance is substantially the same, but mosaics that generate various nearest neighbor distances are also present. It is the scope of the invention. The SES is centered on the current position of the robot, which is at the center 301 of the polyhedron. Axis 302 defines the current path of the robot, axis 304 defines a direction perpendicular to the robot, and axis 303 together with axis 302 defines the horizontal plane of the robot.

The object 350 is projected onto the SES by a ray 355 connecting the center 301 with the object 350. Ray 355 intersects surface 360 at point 357 defined by azimuth angle φ _s and elevation angle (or polar angle) θ _s . Information about the object 350 such as φ _s and θ _s is stored at the vertex 370 closest to the point 357.

  In one embodiment, the SES is implemented as a multiple linked list of pointers to data structures each representing a vertex on the mosaicked sphere. Each vertex record includes a pointer to the nearest vertex and an additional pointer to a tagged format data structure (TFDS). TFDS is a terminal list of objects, and each object consists of an alphanumeric tag, a time stamp, and a pointer to a data object. The tag identifies the sensory data type and the time stamp indicates when the data was written to the SES. A data object includes any functional specification such as sensor data and links to other agents associated with the data object. The type and number of tags that can be written to any vertex is not limited.

  The SES can be implemented as a database using standard database products such as the “Microsoft Access” product version or the “MySQL” product version. Agents that manage communications between the database and other system components can be written in any programming language such as Basic or C ++ known to those skilled in the art.

  In one embodiment, the database is a single table that holds all registration information. The manager communicates with other agents in the control system and relays requests generated to the database. The manager may receive one of four types of requests from any agent: data notification, data retrieval using a data name, data retrieval using a data type, and data retrieval using a location. it can. The notification function captures all relevant data from the requesting agent and registers these data in the database at the correct vertex position. The related data includes a data name, a data type, and a mosaic frequency at which the data is to be registered. The vertex angle is determined by SES according to the pan angle (or azimuth angle) and tilt angle (or elevation angle) at which the data was found. A time stamp is also registered together with related data. The data retrieval function using the data name queries the database using the specified name. This query returns all records in the database that contain the given name. All data is returned to the requesting agent. The data retrieval function using the data type is similar to the previous function, but the query uses the data type instead of the name. The data retrieval function using the position determines the vertex to be queried using the specified position and the adjacent depth to be searched. Once all vertices are determined, a query is made and all records at the specified vertices are returned.

  In another embodiment, the database consists of two tables where the vertex table holds vertex angles and their indexes and the data table holds all registration data. When the SES is created, the manager creates vertices for the projection interface. Each vertex in the vertex table maintains an azimuth angle, an elevation angle, and an index that uniquely identifies each vertex. The manager communicates with an agent outside the control system and relays the request generated to the database. The manager can receive one of four requests from any agent: data notification, data retrieval using data name, data retrieval using data type, and data retrieval using location. . The notification function captures all relevant data from the requesting agent and registers these data in the database at the correct vertex position. The data retrieval function using the data name queries the database using the specified name. This query returns all records in the database that contain the given name. All data is returned to the requesting agent. The data retrieval function using the data type is similar to the data retrieval function using the data name, but the query uses the data type instead of the name. The data retrieval function using position uses an index and an angle stored in the vertex table. The desired position specified in the request is converted to a vertex on the SES. The index for this vertex is localized and all indexes that fall within the desired adjacent position of the first position are collected. The angles that fit these indexes are then used in a query to the main database that holds the registration data. All information at these locations is returned to the requesting component.

  In addition to notification and retrieval agents, other agents can perform functions such as data analysis or data display on information stored in the SES through the use of notification and retrieval agents.

  Since each SPM agent writes to a vertex on the SES, the attention agent searches the vertex list to find the most active vertex called the aiming vertex. High activity at a vertex or group of vertices is a very quick way to aim the robot at events in the environment that may be related to the robot without first processing the information at all vertices of the SES It is. In one embodiment of the invention, the attention agent identifies the aiming vertex by finding the vertex with the most SPM messages.

  In a preferred embodiment, the attention agent weights the information written to the SES, determines the activity value of each message based in part on the currently executing behavior, and as the vertex with the highest activity value. Identify the aiming vertex. If the currently executing behavior ends normally (the post-condition state is satisfied), the attention agent should expect to see the post-condition state, and the SES A portion can be sensitized so that SPM data written to the sensitized portion of the SES is given greater weight or activity. Each SPM can also be biased based on the currently executing behavior from the database combined memory (DBAM) so that more weight is given to the predicted SPM signal.

  For example, the currently executing behavior may have a post-conditional state that predicts viewing a red object 45 ° to the left of the current path. The attention agent may assign these SPM data written to vertices in the area around 45 ° to the left of the current path, for example, to assign an activity 50% higher than the activity at the other vertices. Will be sensitized. Similarly, an SPM that detects a red object in the environment will, for example, write a message having an activity level that is 50% higher than the activity level of other SPMs.

  Events in the environment are thought to stimulate several sensors simultaneously, but messages from various SPMs are written to the SES at different times due to different delays (latency) associated with each particular sensor. Will be. For example, finding moving edges in an image sequence will consume more time than detecting motion with an IR sensor array. The coincidence detection agent addresses different sensor delays using training techniques known to those skilled in the art so that messages received within a certain time interval of SES are identified as responses to a single event. Can be trained.

  In addition to the SPM data written to the vertices, the vertices can include links to behaviors stored in the DBAM. Similarly, the landmark mapping agent can also write to the SES and stores a pointer to the object descriptor at the vertex where the object is predicted. Objects are described in Peters, R., et al., The entire contents of which are incorporated herein by reference. A. II, K.K. E. Hambuchen, K.M. Kawamura, and D.K. M.M. Wilkes' "Sensual Self-Centered Sphere as a Short-Term Memory for Humanoids", "IEEE-RAS Humanoid Robot International Conference" Bulletin, pages 451-459, Waseda University, Tokyo, November 22-24, 2001 It can be tracked on the SES during robot movement using transformations such as

  The ability to place a predicted object on the SES and track the object allows the robot to figure out what to predict and to remember and reproduce where the passed object should be. In addition, the function to reproduce the passed object means that a sudden event may cause the robot's state to shift to a point far from the robot's activity map in front of the event. This allows the robot to backtrack to the previous state when it is lost.

  The function of placing an object on the SES gives the robot the function of self-centered navigation. The placement of the three objects on the SES allows the robot to triangulate its current position, and the ability to place the target state on the SES allows the robot to calculate a target for this current position. enable.

  Also, the object placed in the SES may originate from a source outside this robot, such as another robot, for example. This allows the robot to “know” the position of objects that it cannot directly see.

  Information written to the aiming vertex is vector-encoded into a current state vector and passed to the DBAM. The current state vector is used in the DBAM to terminate or continue the currently executing behavior and further activate subsequent behavior.

  Actuator control is activated by executing a behavior agent retrieved from the DBAM. Each behavior is stored as a record in the DBAM and executed by an independent behavior agent. When the robot is operating in autonomous mode and performing a task, the currently executing behavior agent receives information from the SES. The currently executing behavior agent continues execution if the SES information corresponds to the state predicted by the current behavior, or the current behavior if the SES information corresponds to the post-conditional state of the current behavior. One of them to end. Also, currently running behavior can be terminated by a simple timeout criterion.

  Once the termination condition is identified, the subsequent behavior is selected by propagation of the activation signal during the behavior linked to the currently executing behavior. Limiting the search space to only those behaviors linked to the currently executing behavior, rather than all of the behaviors in the DBAM, significantly reduces subsequent behavior retrieval times, which allows the robot to reduce real-time responsiveness. Show.

  Each of the behaviors linked to the current behavior calculates the vector space distance between the current state and the pre-conditional state of these behaviors themselves. Each behavior propagates a forbidden signal that is inversely proportional to the calculated distance to the other linked behavior (by adding a negative number to the activation term). Propagation of forbidden signals between linked behaviors has the effect that in most cases the behavior with the highest activation term is also the behavior with the pre-condition state that most closely matches the current state of the robot .

  Links between behaviors are created by the SAN agent during task planning, but can also be created by the dream agent during the dream state. These links are task dependent and different behaviors can be linked to each other depending on the assigned goals.

  When a task is imposed on the robot to achieve the goal, an activated diffusion network (SAN) agent backpropagates from the target state to the current state, thereby moving the robot from its current state to the target state within the DBAM. Assemble the behavior sequence (activity map) to be transferred. For each behavior added to the activity map, the SAN agent searches for a behavior having a pre-condition state close to the post-condition state of the added behavior, and links this adjacent behavior to the added behavior. Add An activation term that characterizes the link and is based on the inverse vector space distance between the linked behaviors is also added to the added behavior. The SAN agent can create several paths that connect the current state to the target state.

  The command context agent enables the robot to receive the goal definition task and allows the robot to transition between active mode, dream mode, and training mode.

  The robot can transition to a dream state when not performing or learning a task, or during a mechanical inactivity period when the current task does not use the robot's maximum processing capability. While in the dream state, the robot modifies its behavior based on its most recent activity, or creates a new behavior, and creates a new scenario for prospective execution in future activities (executed in the past by this robot). Create a behavior sequence).

  Each time the robot dreams, the dream agent analyzes R (t) for the most recent activity period from the last dream state by identifying episode boundaries and episodes. Each recent episode is first compared to the existing behavior in the DBAM to see if the recent episode is another instance of the existing behavior. The comparison may be based on the average distance or endpoint distance between recent episodes and existing behavior or any other similar criteria. If this episode is close to this behavior, this behavior can be modified to deal with the new episode.

  If the episode is clearly different from the existing behavior, the dream agent creates a new behavior based on this episode and finds and creates a link to the closest behavior. The default activation link to the closest existing behavior is represented by a typical behavior so that a new behavior generated from a single episode can be assigned a lower activity value than a behavior generated from many episodes. It can be based in part on the number of episodes. New behavior is added to the DBAM for future prospective execution.

  If the robot is limited to behavioral sequences that are learned only through remote control or other known training techniques, the robot may not be able to respond to new situations. In a preferred embodiment, the dream agent is activated during a period of mechanical inactivity, and the robot is willing to react intentionally and positively to unforeseen events that it has not experienced in the past. Create a new valid behavior sequence that can be enabled. The dream agent randomly selects a pair of behaviors from the DBAM and calculates the end point distance between the selected behaviors. The end point distance is a distance between the pre-condition state of one behavior and the post-condition state of the other behavior. This distance can be a vector distance or any suitable measure known to those skilled in the art. If the calculated distance is shorter than the cut-off distance, modify the preceding behavior (behavior having a post-conditional state close to the pre-conditional state of the subsequent behavior) and include a link to the subsequent behavior.

  Pfeifer and Cohen robots need to be trained to identify episodes that lead to task accomplishment. Training typically includes an external handler that observes and rewards robot behavior that advances the robot to task completion. The robot performs either a random move or a best estimate move and receives positive or negative feedback from the handler depending on whether this move has advanced the robot towards the target. This moving feedback cycle must be repeated at each stage toward the goal. The advantage of such a training program is that the robot learns both actions that lead towards the goal and actions that do not achieve the goal. The disadvantage of such a system is that in addition to learning how to accomplish the task, the robot learns much more ways to accomplish the task, so the training time is very long .

  A more efficient way to learn tasks is to teach the robot only those tasks that are needed to achieve the goal. Instead of allowing the robot to perform random movements, the robot is guided to completion of the task through remote control by an external handler. During remote operation, the handler controls all actions of the robot, and at the same time, the robot records its state (sensor and actuator information) during remote operation. The task is repeated several times under slightly different conditions, which allows the formation of episode clusters for later analysis. After one or more training trials, the robot is put into a dream state, where it is recorded to identify episodes, episode boundaries, and create a typical episode for each episode cluster. Information is analyzed.

U.S. Provisional Application No. 60 / 726,033 US Pat. No. 6,697,707 "Learning diffusion mapping expressions based on navigation behavior" by Matric and Brooks, "Cambrian intelligence: the dawn of a new AI", MIT Publishing, 1999 Pfeifer, R.A. And C.I. Scheier, “Sensory-motor coordination: Metaphor and beyond,” Robotics and Autonomous Systems, Special Issue on “Actual and Future of Autonomous Agents,” Volumes 20, 2-4, 157-178, 1997 Albus, J. et al. S. "Introduction to Intelligent Theory", "IEEE Trans. Syst. Man, and Cyber.", Vol. 21, No. 3, 1991 Peters, R.A. A. II, K.K. E. Hambuchen, K.M. Kawamura, and D.K. M.M. Wilkes' “Sensitive Self-Centered Sphere as a Short-Term Memory for Humanoids”, “IEEE-RAS Humanoid Robot International Conference” Bulletin, pages 451-459, Waseda University, Tokyo, November 22-24, 2001 Pfeifer, R.A. Scheier C. "Understanding of Intelligence", (MIT Publishing, 1999) Hambuchen, K.M. A. "Combination of multimodal attention and events in humanoid robots using sensory self-centered spheres", Vanderbilt University, PhD thesis, 2004 Peters, R.A. A. II, Hambuchen, K.M. A. Bodenheimer, R .; E. Papers submitted to the author, “Sensitive self-centered sphere: the interface between sensor and recognition”, “IEEE Transactions on Systems, Humans, and Artificial Brain Engineering”, September 2005 Peters, R.A. A. II, Hambuchen, K.M. A. Kawamura, K .; Wilkes, D .; M.M. Author “Sensitive self-centered sphere as short-term memory in humanoid”, “Conference on IEEE-RAS humanoid robot”, 2001, pp. 451-60 K. R. Cave, "Feature selection gate model for visual selection", "Psychological research", No. 62, pages 182-194 (1999) Shapiro, L.M. , Stockman, G .; C. "Computer vision", (Prentice Hall, 2001) Pratt, W.H. K. "Digital image processing", 454 pages (Wiley-Interscience, 3rd edition, 2001)

  So far, SES has been a poorly concentrated map that can track the position of known objects near the robot. It has been constrained by limited resolution and limited functionality to quickly process the sensory information it receives. The present invention alleviates these problems.

  First, a method for mapping high-density sensory data to SES will be described. Second, a method for finding and ranking related areas in an image that forms a complete visual scene on a SES is described. Further, the image data attention process performs an attention process on each full-size image from the image sequence, maps each attention position to the closest node, and then sums all the attention positions at each node. It has been found that this is done best. Since the attention process is repeated for each image in the sequence, more information is available through this method. Attention points that persist in several adjacent images will have a higher activity value and are therefore considered more prominent than attention points found in only one image. Therefore, the confidence that the position considered significant by the method is a real feature is only performed once on the image where the attention process is restored from the fovea placed on the SES. Higher than with alternative processing methods.

  The above and other objects, features and advantages of the present invention can be more fully understood with reference to the following detailed description.

  FIG. 2 is a schematic diagram illustrating the system architecture of one embodiment of the invention of the 6,697,707 patent. In FIG. 2, the sensory processing module (SPM) 210 provides the “sensory self-centered sphere (SES)” 220 with information about the environment of the robot. The SES 220 functions as a short-term memory of the robot, determines the current state of the robot from the information supplied by the SPM 210, and further determines the aiming area based on the information supplied by the SPM 210, the attention agent 230, and the coincidence agent 240. To do. Vector encoding agent 250 retrieves the data associated with the aiming region from SES 220 and maps this data to a state space region in database coupled memory (DBAM) 260.

  When the robot is in active mode, such as performing a task, the DBAM 260 activates an “activation diffusion network (SAN)” and a series of activities, also referred to as an activity map that the robot performs to achieve the assigned goal. Plan actions. Each action is executed as a behavior stored in the DBAM 260, and the DBAM functions almost like a long-term memory in a robot. The appropriate behavior according to the activity map is retrieved from the DBAM 260 and this behavior is performed by the actuator 270. Actuator 270 includes a controller that controls an actuator in the robot so that the robot acts on the environment through this actuator. The DBAM also supplies the current state information of the robot to the attention agent 230 and the coincidence agent 240.

  The context agent 280 provides information regarding the robot's operational context received from a source external to the robot. In the preferred embodiment, the context agent 280 defines three general operational contexts: task, training, and dream. In the task execution context, the context agent 280 sets a task goal received from an external source. In the training context, the context agent 280 can transmit all remote control commands received from an external source to the actuator through the DBAM. In a dream context, the context agent 280 can stop the actuator and activate the DBAM, modifying and creating behavior based on the most recent robot activity maintained by the SES 220.

  Each SPM 210 consists of one or more agents that function independently of each other, which will now be described in detail.

  Each SPM 210 is associated with a sensor and writes sensor specific information to the SES 220. The robot sensor may be an internal or external sensor. The internal sensor measures the state or state change of the internal device of the robot. Internal sensors include joint position encoders, force-torque sensors, strain gauges, temperature sensors, friction sensors, vibration sensors, inertial induction or vestibular organ sensors such as gyroscopes or accelerometers, current, voltage, resistance, capacitance, or inductance Motor status sensors such as electrical sensors, speedometers, clocks, or other time measuring devices, or other transducers known to those skilled in the art. These sensors can be informational, for example, measuring the status of computer modules, the activity of computer agents, or the communication patterns between them. The success or failure of the task can be “sensed” in information and added to the internal influence measurement.

  The external sensor is an energy converter. These transducers are stimulated by energy incident from outside the robot, which incident energy is sampled and quantized by the robot for abstract representation, or supplied to other sensors or drives actuators. Converted to an internal (relative to the robot) energy source (electrical, mechanical, gravity, or chemical) that can be used either directly. External sensors include either color or monochrome still images, motion video (video) cameras, infrared, visible, ultraviolet, or multispectral non-image generating light sensors sensitive to various wavelengths, microphones, SONAR, RADAR, Or active range finder such as LIDAR, proximity sensor, motion detector, tactile array such as contact sensor in artificial skin, thermometer, single or array contact sensor (feeling means), collision sensor, olfaction Or a chemical sensor, a vibration sensor, a global positioning system (GPS) sensor, a magnetic field sensor (including an azimuth meter), an electric field sensor, and a radiation sensor. Also, the external sensor can be informational to receive communication signals (wireless, TV, data) having a direct internet connection or connection to another robot. The external sensor may have a computer aspect that interprets words, gestures, facial expressions, voice tone, and intonation.

  Each sensor can be associated with one or more SPMs, and each SPM can handle one or more sensors. For example, the SPM can process signals from two microphone sensors to determine the direction of the sound source. In another example, the camera can send a signal to an SPM that identifies only strong contours in the field of view, and the same signal can be sent to another SPM that identifies only red in the field of view.

  Each actuator 270 includes an actuator controller that controls an actuator in the robot. An actuator can be any device that causes a robot to act on its environment or change the relative orientation of any of the parts of the robot. The actuator performs a task and can be driven by any conceivable energy source such as an electrical, atmospheric, hydraulic, thermal, mechanical, atomic, chemical, or gravity source. Actuators include motors, pistons, valves, screws, levers, artificial muscles, or the like known to those skilled in the art. In general, actuators are used for sensor movement, manipulation, or active positioning or scanning. Actuators may refer to a group of actuators that perform coordinated tasks such as arm or leg movements or in an active vision system.

  The actuator controller is typically activated by a robot behavior agent that executes a behavior sequence during a task. During training, the actuator controller can be activated in a process called remote control by a handler outside the robot.

  One of the major open issues in robotics is how to precisely combine different types of sensory information so that the signals are correctly attributed to objects in the environment. Furthermore, “sensory-motor coordination (SMC)” is necessary for animals and robots to perform actions with purpose. This cooperation can also be fundamental to the category. Pfeifer categorizes robot-environment interactions, with SMC data recorded during concurrent actions and sensing by a robot performing a fixed set of tasks in a simple but changing environment It shows that it can self-organize into descriptors. Pfeifer, R.A. Scheier C. See "Understanding Intelligence", (MIT Publishing, 1999). When the robot operates, the SMC requires associating various sensory information with motor activity and further requires sensor coupling despite the different spatio-temporal resolution and different time latency in the processing function. Since resources (sensory, computational, motor) can only be directed to a small subset of environmental features available at any one point in time, the learning SMC also requires attention.

  “Sensory self-centered sphere (SES)” has been proposed as a computer structure that supports both SMC and attention. Hambuchen, K.M. A. See "Multimodal attention and event combination in humanoid robots using sensory self-centered spheres", Vanderbilt University, Doctoral Dissertation, 2004. The self-centered sphere mapping of the SES locale serves as an interface between sensing and recognition. Peters, R.A. A. II, Hambuchen, K.M. A. Bodenheimer, R .; E. See the paper submitted to "Sensory Self-Centered Sphere: Intervention Interface between Sensors and Recognition", "IEEE Transactions on Systems, Humans, and Artificial Brain Engineering", September 2005. SES is close to robots As a short-term memory in humanoids, by Peters, RA II, Hambuchen, KA, Kawamura, K., Wilkes, DM See "Sensitive Self-Centered Sphere", "Conference on IEEE-RAS Humanoid Robot", 2001, 451-60. In an independent parallel SPM, the SES combines the sign sensor data as a result of the geometric structure. See the same literature. SES can also combine attention events detected by different sensors with task and environment specific contexts to produce a set of related areas ranked within the environment. Hambuchen, K.M. A. Please refer to the doctoral dissertation. That is, attention signals can be combined to guide the attention sight. Moreover, it can be made sensitive and customary about attention. See the same literature.

  As used before, SES is a poorly concentrated map. The present invention provides a method for mapping high resolution sensory information (in the form of a series of visual images) onto a SES. The present invention also describes the problem of finding and ranking related areas in an image that forms a complete visual scene on a SES.

  To implement the present invention, a set of 320 × 240 color images was taken with a rotating pan / tilt camera head of a humanoid robot. The image was not pre-processed and did not identify a specific object. While capturing the image, the camera head was swung in the workspace. The result was a complete mapping of the visual scene onto the SES. The camera cannot rotate through 360 degrees, and therefore cannot map to the entire SES, so SES tied in an area of +20 degrees to -60 degrees in tilt and +80 degrees to -80 degrees in pan. The subset was filled with data. This range was chosen because both the camera can cover this range and the ± 80 ° pan range fits the human field of view.

  The task of mapping a complete visual scene onto a “sensory self-centered sphere” was accomplished by first aggregating a list of all SES nodes in the field of view. Next, a sequence of 519 images is generated by taking a picture at each of the pan / tilt positions corresponding to the nodes in this list, and more precisely, the image center corresponds to each of these angle pairs. I let you. A fovea window at the center of each image in the sequence was extracted and placed at the correct node location on the SES. FIG. 3 illustrates this procedure performed to form an image at SES node 1422 having pan and tilt angles of −33.563 and −23.466, respectively.

  The size of the fovea window taken from the center has changed, but in general panning 5 ° and tilting 5 ° is the distance separating most nodes on a geodesic dome with a frequency of 14 in general. Was about 5 °, and the tilt was 5 °. However, since both pentagons and hexagons make up the dome, the edges between the nodes on the geodesic dome do not all have the same length. For precise results, the distance between each node and the four closest nodes (up / down / left / right) of these nodes were calculated in angle and converted to pixel scale. The pixel scale for each angle was determined experimentally. Next, the fovea whose size was appropriately determined was extracted from the image center. Each foveal recording was placed at the node on the SES corresponding to the pan / tilt angle pair of this recording. FIG. 4 shows a visual representation of all foveal images placed on a “sensory self-centered sphere” for a humanoid robot.

  A piecewise continuous image of the visual scene was restored from all foveal images placed on the SES. A node map that associates each pixel in the restored image with a node on the SES was similarly generated. The restored image is illustrated in FIG.

  Attention problems arise when SES is clustered with high density information. If a robot is to interact with a human-centered environment in real time, attention can be directed only to critical areas determined by safety, opportunities, and tasks because of limited computer resources. The problem is how to perform attention processing given a data-filled SES and image input stream. There are at least two possibilities. One is to perform visual attention processing on the entire SES. The other is to detect important points in individual images and combine these points with a series of existing images.

  One model of visual attention is a “feature gate” model. This model is based on Cave's observation that attention appears to suppress objects in the scene based on both the position of the object and its similarity to the target. Which is incorporated herein by reference. R. See Cave, “Characteristic Gate Model of Visual Selection”, “Psychological Studies”, 62, 182-194 (1999). In this model, each position in the visual scene has a basic feature vector, such as direction or color, and an attention gate that adjusts the information flow from each of these positions to the output. This gate restricts the information flow from the location of this information if the information will potentially interfere with information from another location that is more promising or more important for the target currently being processed . Thus, the gated flow depends on the location features and the surrounding location features. The visual scene is divided into areas. Features within a group of areas are scored and compared, and the “excellent” position within each group of areas is passed to the next level. This continues iteratively until only one position remains that is the output of the model. A “feature gate” includes two subsystems to handle bottom-up and top-down mechanisms. Top-down processing is task related. For example, a task may search for a specific person in a scene. In this case, a position having a known characteristic of the target person is supported more than a position having no such characteristic. Specifically, the similarity of the position to the target is scored, and the most similar position is supported over all others. The bottom-up process identifies the most prominent position in the scene that is task independent. In this case, a position having a feature different from the features at the surrounding positions is supported. Specifically, numerical excellence values are calculated for the most prominent features, and the location of these features is supported over other features.

  In the present invention, for this study, a “feature gate” was performed using three separate feature maps for color, brightness, and direction each. Direction processing is performed according to the Frei-Chen standard. Shapiro, L., which is incorporated herein by reference. , Stockman, G .; C. "Computer Vision", (Plentice Hall, 2001), and Pratt, W. et al. K. See “Digital Image Processing”, page 454 (Wiley-Interscience, 3rd edition, 2001). To obtain good results, the incoming image was first blurred using a constant filter. By blurring the image, “feature gate” processing is less susceptible to very small insignificant changes that occur continuously from image to image. According to the bottom-up process of the “feature gate” model, the feature at the position of each pixel was compared with the feature at the eight neighboring positions closest in Euclidean distance, and the results were added and stored in the activation map. When using top-down processing, the location features of each pixel will be compared to the known target features, and the location with the highest activation from the first level is selected as the aim aim. It will be. However, in the experiments conducted so far, no top-down processing was used, and the attention point was selected only by the excellence of this attention point and not by targeting the characteristics of a specific feature.

  In accordance with the present invention, the “feature gate” method is used to perform an attention process on each image in the image sequence, and the result is recorded at the node corresponding to the optical center of the image. In this process, the 12 most prominent positions (row and column positions) and their activity values (or scores) in the excellence array structure were taken. The array also included the pan and tilt angles of the image being processed. Since the number of 12 was found to provide a relatively uniform distribution of attention points across the image, the number of positions returned by the program was arbitrarily set to 12.

  Only a small area (foveal area) is displayed on the graphic SES representation, but a full-size image is taken and processed at the location of each node. For this reason, there is considerable overlap between adjacent node images from the sequence. This overlap means that attention points from different images will often point to the same location in space. In the visual system used in this study, a single image roughly spans 55 ° pan and 45 ° tilt. Therefore, if two images are separated by a pan of 55 ° and a tilt of less than 45 °, these images will overlap. Since only the fovea window is associated with each node, images that are approximately within 30 ° pan and 25 ° tilt will overlap in the fovea. This produces about 30 images that overlap any fovea window. It was necessary that there be one overall attentional excellence value associated with each node of the SES. To calculate a single superior value for a node, combine the excellence of all attention points mapped to this node, whether from the image taken at the node's location or from an adjacent image. It was assumed that attention positions identified in many images are more prominent (and therefore should have higher values) than attention positions found in only one image. The subsequent processing for specifying the position of a scene with high excellence by combining attention points will be described below.

  After the attention data is obtained from the image, each of the 12 salient points in the image is mapped to a SES node corresponding to the position of that point. Correspondence is determined as follows. First, the distance from the attention point to the center of the image is calculated in terms of the number of pixels, and then this is converted into angular displacement using pixel values for each angle determined experimentally. Was about 28 pixels, and a 5 degree range in pan was a range of about 30 pixels.

  With this information in mind, this information is used with the optical center pan / tilt angle so that each attention point can be mapped to the appropriate node, and the actual pan and tilt angles for each of these attention points. Find out. Positional errors may cause attention points from the same feature to be mapped to adjacent nodes. Therefore, we used an attention point clustering algorithm to find all attention positions corresponding to a particular environmental feature. The procedure selects each node ID that has at least 15 attention points and calculates the intermediate pan / tilt values for those points. Next, all attention points in all images that fall within a radius of 2 degrees from the intermediate pan / tilt value were found. All these points were mapped to the same node, which is the node with the most attention points that fall within this radius. A radius of 2 degrees represents approximately one quarter of the average fovea and was chosen because it is compact enough to separate point clusters.

  An example of this is illustrated in Table 1, which shows all the original images (at imgCtrID column) that have attention points mapped to node 1421 (ID column) on the SES, as well as the calculated pan for each attention point. And the tilt angle.

(Table I)

  In order to judge the excellence of this node, the activity values (that is, numerical excellence values) of the attention points arranged in the node were summed. FIG. 6 shows the most prominent positions in the entire top 12 locations in the scene.

  Another way to determine the attention position on all SES is to process an image (eg, the image of FIG. 5) of the visual scene (reconstructed from the foveal image as described above) through a “feature gate”. Conceivable. To do this, the "feature gate" algorithm was modified to include a restored image node map. This makes it possible to record a node ID associated with an attention point for comparison with other attention processing techniques. The result can be seen in FIG.

  FIG. 8 is a graph of the number of nodes versus the activation threshold. This graph represents the number of nodes exceeding the threshold for a threshold ranging from the minimum total activity value to the maximum total activity value for each node calculated in this experiment. There were 672 nodes with attention positions.

  Several thresholds were selected and the percentage of nodes with activation above the threshold level was calculated. The first three columns of Table II list these results. These give a measure of the level of activation required for a node to be a key attention position on the overall SES. For example, to enter the top 10% of attention positions on SES, a node will need to have a total activity value of at least 100,000.

  Another way to determine how important a single attention position is for the overall SES excellence is to calculate the percentage of individual attention positions that are mapped to nodes with activations that exceed a threshold. It is. There were a total of 6228 attention positions on the SES. These calculations were made for several thresholds. For example, if a node with an activity value in the top 10% is selected (threshold 100000), the percentage of individual attention positions mapped to one of these nodes is 41%. In other words, 41% of the individual attention positions are mapped to the top 10% of the node positions on the SES. The percentage calculation results for the different thresholds can be seen in the last column of Table II.

(Table II)

  Another measure of the importance of individual attention positions is the percentage of attention positions that fall within the top N positions (nodes). This is similar to the percentage comparison described above, except that we have chosen a fixed number of nodes that can be useful for comparison. Furthermore, no matter how many attention positions are found in the scene, only a fixed number can be attended and should be attended. For example, it has been found that 19% of the individual attention positions are mapped to the positions of the top 20 nodes on the SES. In other words, the 20 most prominent positions on the sphere represent 19% of all individual attention positions. Table III shows the number of attention positions for several values of N.

(Table III)

  Attention points found in individual images were compared with attention points found over the entire restored scene image. This was done by processing the reconstructed image (single image in FIG. 4) with a “feature gate” to find the N nodes with the highest activation. When attention processing is performed on individual full-size images, some attention positions are mapped to nodes that do not correspond to image pieces arranged in the SES. This occurs in images taken at nodes near the edge of the visual scene. These positions are not represented in the restored visual scene image, and it will not be accurate to compare with the nodes in the restored image. Therefore, the top N positions corresponding to the nodes in the restored scene image were found. Next, the attention position found through the sum of the active values was compared to the position found by processing the reconstructed scene image directly (Table IV).

(Table IV)

  In both the total activation image (first paragraph following Table I) and the restored scene image (second paragraph following Table I), prominent such as pandas, Bernie dolls, trash can, left shelf, and chair Features were detected. Features with clear edges and corners, such as a black frame on the front wall and black wall tape, were also detected in both images.

  The total activation image (first paragraph following Table I) appears to fit better for attention deployment on SES. Contrary to the sequence of overlapping images, only one image determines the most prominent position in the scene, so processing the entire restored scene image gives less information than the total activated image. Furthermore, when a total activation image is implemented, it is easy to update the superiority distribution on the SES as new information becomes available. For example, this update can be done simply by processing a new image and combining a new attention point found with an existing attention point. Activation at each node can be weighted by the age of each attention point, giving newer points greater weight.

  Experiments were conducted to test the robustness of the total activated image processing method. A subset of the original visual scene was selected and an image sequence under different illumination levels of this scene was generated. The number of matching nodes between sequences with different illuminance can be seen in Table V. The illuminance levels of low light illumination and low light spot illumination are very different from the high and medium light illumination levels. This accounts for the low percentage of matching nodes. However, the percentage of matching nodes between high and medium lighting levels is high, indicating that the system behaves similarly when facing different light levels.

(Table V)

  In summary, the image data attention process performs an attention process on each full-size image from the image sequence, maps each attention position to the nearest node, and then sums all the attention positions at each node. It turns out that it is done most appropriately. Since the attention process is repeated for each image in the sequence, more information is available through this method. Attention points that persist in several adjacent images will have a higher activity value and will therefore be considered more prominent than attention points found in only one image. Therefore, the reliability that the position considered significant by the present method is an actual feature is that the attention process is performed only once on the image restored from the fovea placed on the SES. It is higher than when using an alternative processing method.

  Since the preferred embodiments disclosed herein are intended to be illustrative of some aspects of the present invention, the invention described and claimed herein is not to be limited in scope thereby . Any equivalent embodiments are intended to be within the scope of this invention. Indeed, various modifications of the invention in addition to those shown and described herein will become apparent to those skilled in the art from the foregoing description. Such modifications are also intended to fall within the scope of the claims.

  A number of references are cited herein, the entire disclosure of which is hereby incorporated by reference for all purposes. Furthermore, regardless of how these references are characterized above, no priority is granted to the claimed invention of the subject matter claimed herein.

It is an example figure useful for understanding a "sensory self-centered sphere". 1 is a schematic diagram illustrating a system architecture of an exemplary embodiment of a prior art adaptive autonomous robot. FIG. It is a figure which shows the image formation process used in implementing embodiment of this invention. It is a figure which shows 1 set of foveal images arrange | positioned on SES. It is a figure which shows the scene decompress | restored from the foveal image. FIG. 7 identifies the twelve most prominent positions in a scene when identified by summing the scenes in individual images according to one embodiment of the present invention. FIG. 12 identifies the 12 most prominent positions in the scene as identified by processing the entire scene. It is a graph which shows the number of nodes exceeding a specific activation threshold value.

Explanation of symbols

300 SES polyhedron 310 vertex 350 object φ _s azimuth θ _s elevation angle, polar angle

Claims (6)

A sensor,
An actuator,
A camera for receiving image data from an external source;
A database for associating the received image data with a point on a part of a spherical area centered on the robot;
Means for performing attention processing on individual images and specifying the attention position;
Means for mapping each attention position to a closest point on a portion of the spherical region;
An adder for summing all said attention positions mapped to points;
An autonomous adaptive robot characterized by including The sensor is
Means for sensing the internal state of the robot;
Means for sensing one or more characteristics of the environment of the robot;
The robot according to claim 1, comprising:   The robot of claim 2, wherein the database includes a sensory ego sphere for representing an object based on the sensed internal state and the sensed environment. In autonomous adaptive robots,
A sensor,
An actuator,
A camera for receiving image data from an external source;
A database for associating the received image data with a point on a part of a spherical area centered on the robot;
A method of finding and ranking relevant areas of an image from the external source, comprising:
Attention processing is performed on each image, the attention position is specified,
Map each attention position to the closest point on the part of the sphere region;
Summing all the attention positions mapped to points; and
A robot characterized by including: A method for mapping dense sensory data to a robot database,
Editing a list of nodes in a portion of a sphere region centered on the robot;
Generating an image of an area external to the robot at each node;
Extracting a foveal image from the center of each image;
Associating each foveal image with the node in the database;
A method comprising the steps of:   6. The method of claim 5, wherein the portion of the sphere region is a sensory ego sphere.

Images