KR20070121601A - Evaluating visual proto-objects for robot interaction - Google Patents

Abstract

Evaluating visual proto-objects for robot interaction is provided to enhance interaction between a robot and a circumstance around the robot based on visual information about the circumstance around the robot. In evaluating visual proto-objects for robot interaction, an interaction robot includes a visual detecting unit, a control unit, and a computing unit. The computing unit processes output signals from the visual detecting unit to generate proto-objects stored in a memory, wherein the proto-objects indicate interested blobs in an input field of the visual detecting unit based on 3D position label. The computing unit forms object hypothesizes with regard to category of the object based on evaluation of the proto-objects about different behavior property limit. The computing unit selects one among visual tracking motion, motion of the robot, and/or operation of the control unit on the basis of the proto-objects and the hypotheses as a target for motion.

Description

Evaluate visual primitive objects for robotic interaction {EVALUATING VISUAL PROTO-OBJECTS FOR ROBOT INTERACTION}

1 is a schematic diagram of a distribution of work and communication paths of a system according to the present invention;

2 shows that the coordinates of a stereo image acquisition system are converted into axes arranged in parallel.

Research on robots that are close to humans is increasingly focusing on interactions in complex environments, including autonomous decision making and complex coordinated behaviour.

Robots evaluating visual information, particularly stereo information, are known to use the information so obtained about the environment for controlling their behavior.

The present invention relates to robots with degrees-of-freedom that allow them to perform different, simple or complex operations.

Accordingly, the present invention proposes a technique for improving the interaction between the environment and the robot based on visual information about the environment of the robot.

This object is achieved by the features of the independent claims. The dependent claims further develop the central idea of the invention.

The invention can be implemented as a robot capable of interacting with the environment driven by visual perception, for example, using simple decision making and coordinated full body motion.

One aspect proposed by the present invention is to implement, for example, any elongated colored object in order to implement the basic components of an architecture that is easily extensible to handle longer term targets. To build a system that uses the definition of visual target objects, such as

Aspects of embodiments of the present invention are as follows.

Proto-objects can be used in raw form to form stable object assumptions, if necessary for visual tracking or reaching and grasping of these primitive objects, for example in 3d. Recognition information is stored as raw objects in short-term detection memory.

Decision mechanisms to evaluate behavioral and operational alternatives based on sensing information and internal prediction.

A motion control system that can be driven by a wide range of possible target descriptions and ensures smooth, well coordinated overall body motions using a set of cost functions as null space criteria.

The recognition system uses color and stereo based 3d information to detect the associated visual stimulus and maintain this information in the short term sensing memory as primitive objects.

This sensory memory is then used to derive targets for visual tracking and to form stable object assumptions from which motion targets for reach operations can be derived. The prediction based decision system selects an optimal action strategy and executes it in real time. The internal predictions, as well as the actions performed, utilize an integrated control system that achieves well coordinated and smooth overall body motions using the flexible target description of the task space in addition to the cost function of zero space.

Summary of the Invention

One aspect of the invention relates to an interactive robot comprising visual sensing means, manipulation means and computing means. Computing means,

Process output signals from the visual sensing means to generate primitive objects stored in memory and representing blobs of interest in the input field of the visual sensing means by at least a 3d position label,

Forming object assumptions for the category of the object based on the evaluation of the primitive objects for different behavior specific restrictions,

-Based on at least one primitive object and the assumptions as a target for the operation, it is designed to determine at least one of the visual tracking operation of the visual sensing means, the operation of the body of the robot and / or the operation of the manipulation means.

The blobs may also be indicated by at least one of size, orientation, time of detection and accuracy label.

The operation of the manipulation means may comprise at least one of a grasping operation and a poking operation.

The computing means may be designed to only consider primitive objects created in the recent past.

At least one evaluation criterion for the primitive objects may be their elongation.

At least one evaluation criterion for the primitive objects may be their spacing to a behavior-specific reference point.

At least one evaluation criterion for primitive objects is their stability over time.

Another aspect of the invention relates to a method of controlling an interactive robot comprising visual sensing means, grasping means and computing means. The method,

Processing the output signals from the visual sensing means to generate the primitive objects, the primitive objects representing blobs of interest in the input field of the visual sensing means by at least a 3d location label;

Forming assumptions about the category of the object by evaluating the primitive objects,

Determining at least one of a visual tracking operation of the visual sensing means, a motion of the body of the robot and / or an operation of the manipulation means, based on the at least one primitive object and the assumptions as a target for the movement

It includes.

Further objects, features and advantages of the present invention will become apparent from the following detailed description of a non-limiting embodiment of the invention in conjunction with the accompanying drawings.

In general, the present invention seeks to create a system that uses the definition of visual target objects, e.g., any elongated colored object, to implement the basic components of an architecture that is easily extensible to handle longer term targets. Suggest.

The components of one embodiment of the present invention shown in FIG. 1 are as follows:

Primitive objects in raw form in short-term sense memory so that they can be used in raw form to form stable object assumptions, if necessary for visual tracking or reaching and grasping of these primitive objects, for example in 3d. Storing recognition information as

Decision mechanisms to evaluate behavioral and operational alternatives based on sensing information and internal prediction.

A motion control system that can be driven by a wide range of possible target descriptions and ensures smooth, well coordinated overall body motions using a set of cost functions as null space criteria.

The recognition system uses color and stereo based 3d information to detect related visual stimuli and maintains this information as primitive objects in short-term sensing memory.

This sensory memory is then used to derive targets for visual tracking and to form stable object assumptions from which motion targets for reach operations can be derived. The prediction based decision system selects an optimal action strategy and executes it in real time. The internal predictions as well as the operations performed use an integrated control system that achieves well coordinated and smooth overall body motion using the task space's flexible target technology in addition to the zero space cost function.

A. System Overview

1 illustrates the distribution of communication paths and processing of a computing unit designed to control an interactive autonomous robot.

Stereo color images (see FIG. 2) are successively acquired by the image acquisition unit and then processed in two parallel paths. The first passageway is a color discriminating unit designed to extract areas of interest (hereinafter referred to as "blobs").

2 shows how they are transformed to align the initial unaligned coordinates from the left camera (index "1") and the right camera (index "r") in a parallel manner.

The second passageway includes a 3d information extraction unit, which may also be called a stereo calculation block, which calculates the visual distance to the image acquisition unit for each pixel.

The results of the two paths are recombined to form primitive objects in the form of 3d blobs that are stabilized over time in short-term sensing memory.

Using defined criteria, object assumptions are created by evaluating current primitive objects stored in the sense memory. These assumptions can then be used by different acts as targets. Actions are, for example, one or more of the following:

-Search or tracking of heads

-Walking or resting on the legs of the robot

-Reaching, catching or extruding the robot's arms.

Head targets, ie targets for searching or tracking behavior of the head, are selected using a fitness function. The fitness function evaluates the "fitness" of different actions in terms of defined robot-related objects and multiple objects. By applying the fitness function a corresponding scalar fitness value is generated.

Arm and body targets can be generated by selecting the most suitable internal prediction.

Targets for controlling viewing direction, hand position and orientation, and leg motion are supplied to the overall body motion system that generates motor commands.

The whole body motion system can be supported by a collision detection system based on kinematics to prevent damage to the robot.

In order to integrate the visual targets into the behavior, the visual data and the robot poses are time labeled. Mechanisms are provided for accessing the robot posture for a given time. This also requires time synchronization between image acquisition and operating subsystems.

Several components of the system will be described in more detail below.

Vision and control processing is divided into several smaller modules that interact in a data-driven fashion in a real-time environment.

B. Vision

summary

The chain of processing starts with newly acquired color stereo images. These images are fed into two independent parallel passages using color and gray scale images. Color processing consists of color separation of the color image, which in turn constitutes a pixelated mask of color similarity divided into compact regions. Gray scale images can be used to calculate image inconsistencies between left and right images to extract 3D information.

In order to construct a three-dimensional "raw object" from the acquired vision data, each color segment uses, for example, two-dimensional Principal Component Analysis (PCA) of pixel positions to evaluate the principal orientation and the individual sizes. To blobs (eg oriented ellipse). Using the median of discrepancies from the stereo calculations, this blob is transformed into a 3d representation. Blobs that are too small or have a depth outside the given range are ignored.

In order for this 3d blob to stabilize both temporally and spatially, it is converted into world coordinates, since it is possible that the robot has moved since the time of acquisition. To access the pose of the robot in image acquisition, the system uses the time stamp of the vision data and the organized pose buffer as a ring buffer that is constantly updated with the latest poses.

3d blobs in world coordinates are called porto-objects because they are a preliminary approximate representation of what may be physical objects. For stabilization in time, the sense memory compares the current list of primitive object measurements with the prediction of existing primitive objects for a new time step. Using a metric of blob location, size, and orientation, the sense memory updates existing primitive objects, illustrates new primitive objects, or deletes primitive objects that have not been identified after a certain time. The latter ensures that outliers as well as occluded objects do not remain in memory too long.

To the blobs  Based primitive objects field  Detailed description of produce

A detailed description will be given below on the generation of object assumptions from visual raw object data.

The primitive object as described above can remain opaque and enables multiple evaluation methods. However, each object assumption is based on only one particular method of evaluation of this data. In imaginable robotic interaction applications, the system according to the present invention can simultaneously measure various types of distinguishable objects or regions by means of a head movement during capture for one particular stretched object, for example. Evaluate visual primitive object data with respect to

It is proposed to use pairs of color images labeled with the time of acquisition. These images are processed in two parallel paths, as already described above and shown in FIG. 1:

1. Calculate Stereo Mismatch:

Intensity image pairs are generated from the color image pairs. The images are modified, ie deformed and the result corresponds to images captured with two pinhole cameras, for example with collinear image rows (see FIG. 2). Horizontal inconsistencies between corresponding features in the two images are calculated if the features are important enough.

2. Color segmentation:

The color of one of the pairs is modified as above and converted to, for example, HLS (Hue, Luminance, Saturation) color space. All pixels are evaluated for whether they are within a particular volume of HLS space and the result is subjected to morphological operations that remove small areas of one class of pixels. The resulting pixels in the HLS volume are grouped into consecutive regions within the image plane. The maximum resulting groups exceeding the minimum size are selected for further processing.

For each of the groups from the color distinction, the median of the centers X _p , Y _p and inconsistencies d of all those pixels in the image plane is calculated. It is also detected whether the group region touches the image boundaries, and if so, the data is labeled as incorrect because the parts of the real object corresponding to that region are probably outside the field of observation.

Using the principal component analysis (PCA) of the correlation matrix of pixel positions, the orientation of the standard deviations σ _p1 σ _p2 and the fundamental axis ω of the pixels in the image plane is calculated for each group.

Using the geometry of the camera system, the coordinates (x _p , y _p , d) and σ _p1 and σ _p2 are _converted into metric coordinates (x _c , y _c , z _c ) and metric standard deviations σ _c1 and σ _c2 . Is converted.

로 변환하고 기본축 방위 ω를 월드 좌표 내의 방위 벡터

로 변환하는데 이용된다.Using the time labels of the images, the robot pose and position are derived when capturing the image.

And convert the principal axis bearing ω into the bearing vector within the world coordinates.

Used to convert

, 방위

, 표준 편차들 σ_c1 및 σ_c2, 및 데이터의 정확 여부를 나타내는 라벨을 포함하는 데이터의 세트로서 정의될 수 있다.Thus, blobs may have time labels, positions, as described above.

Bearing

, Standard deviations σ _c1 and σ _c2 , and a label indicating the accuracy of the data.

Primitive Objects in Sense Memory field  Save:

Primitive objects are derived from the blob data by taking the incoming blob data and comparing it with the contents of the sense memory.

If the memory is empty, a primitive object is created from the blob data by assigning a new identifier to the new primitive object and inserting incoming blob data into it.

If the sense memory already contains one or more primitive objects, predictions for each primitive object are generated as blob data. This predicted blob data is based on all the blob data contained in the primitive object and is generated for the current time.

Each incoming blob is inserted into a newly created object based on an existing primitive object or the minimum distance between the incoming blob and the predicted blob so that all incoming blobs are assigned unique identifiers.

The metric for calculating the spacing is based on both Euclidean distance and relative rotation angle.

The inserted blob data is also modified so that the azimuth spacing of the new blob is always 90 degrees or less. This is possible because the blob orientation description is ambiguous with respect to 180 degree flips.

Each time a process generates new incoming blob data, the time label is compared with the time labels of the blob data in all the primitive objects and all blob data older than a predetermined threshold are deleted. This is done even if image processing does not find any blobs in the image pairs. If the raw object does not contain any blob data, it is also deleted from the sense memory.

, 방위

, 및 표준 편차들 σ_c1 σ_c2 에 대한 원시 객체 내의 블로브 데이터로부터 로우 패스 필터에 의해 도출된다.Prediction as needed for the comparison is location

Bearing

, And from the blob data in the raw object for the standard deviations σ _c1 σ _c2 .

C. Action Selection

Object assumptions are created by evaluating primitive objects stored in the sense memory against specific criteria. For example, the elongation of primitive objects may be selected as an evaluation criterion for object assumptions, that is, based on the radii of ellipsoids, the elongated primitive objects are evaluated as behavior targets, while the more spherical primitive objects are Ignored The presence of these extended object assumptions is the main criterion in behavior selection mechanisms.

For example, the following two selection mechanisms can be used to control two main action groups:

Search and track actions are described, for example, in 1999 by Neural Networks, T. Bergener, C. Bruckhoff, P. Dahm, H. Janssen, F. Joublin, R. Menzner, A. Steinhage, and W. von Seelen. And may be selected based on a fitness function as described in "Complex behavior by means of dynamical systems for an anthropomorphic robot."

The output of the sense memory can be used to drive two different head actions, for example:

1) search for objects and

2) Stare or track objects or blobs.

Apart from these actions there is always a decision instance or " arbiter " that determines which action should be active. The decision of the coordinator is based solely on the scalar value supplied from the stimulus of the actions, which is the "fitness value". This fitness value always indicates how well the action can be performed. Tracking in this specific case requires at least one incorrect blob location pointing in the gaze direction, but of course may use the full object hypothesis. Thus, the tracking behavior will output fitness of 1 if any blob or object is present and output fitness of 0 otherwise. The fitness is fixed at 1 since the search behavior has no requirements at all. In this very simple behavior setup, arbitration is of course trivial. However, the present invention proposes to use a competitive dynamic system similar to that described in 1999 Bergener et al. (See above) for scalability. Thus, the coordinator uses a vector made from scalar fitness values resulting from the stimulus of all actions as input to the competitive dynamics that calculates the activity value for each action. Competitive dynamics uses pre-specified inhibition matrices that can be used to encode directed inhibition (act A inhibits action B but not vice versa) to specify action prioritization and even cycles of action. In this case tracking can be prioritized to search by the prohibition so indicated.

Retrieval behavior is realized using excessive low (5 by 7) inhibition of the return map with simple relaxation dynamics. When the search behavior becomes active and new vision data becomes available, it will increase the value of the current gaze direction in the map and select the lowest value in the map as the new gaze target. The entire map also suffers from relaxation to zero and less additional noise.

This immediately generates a visual search pattern with a random sequence of fixations that takes into account all visual information and results in an efficient and fast search of related objects. The magnitude of the inhibition of the return map is derived from the field of view of the camera relative to the pan / tilt movement range. Higher resolution will not significantly change the search. The relaxation time constant is set within the second range so that the operation of the robot to effectively invalidate the forbidden map is not a problem.

The tracking behavior can be implemented as multi-tracking of three-dimensional points. The action takes into account all relevant primitive objects and object assumptions and calculates pan / tilt angles to keep them in the field center of observation. A cost function with a trapezoidal shape in pan / tilt coordinates is then used to retrieve the pan / tilt angle to keep the maximum number of objects in the effective viewing field of the cameras, which are sent as pan / tilt commands. The tracking behavior always uses the stabilized output of the sense memory, so the robot will still stare at a particular position even if the blob has disappeared for a while. This greatly improves the performance of the overall system.

Other actions using hard coded criteria based on internal predictions will be discussed in the next section.

Using these selection mechanisms, the system can search for stretched objects or track one or more of them. At the same time, the robot can reach the stretched object using the most appropriate arm with the palms aligned with the object's primary axis, and if the object is too close or too far, it will also select the appropriate walking motion. If no target is available, the robot will stop walking and move its arms to a resting position.

The evaluations are also based on blob data predictions of all primitive objects. The label of this prediction is set to "memorised" if the latest blob data in the primitive object is older than the prediction time. In other cases, it is set to the label of recent blob data in the primitive object.

The minimum criteria already sufficient for the behavior of fixation and tracking is blobs labeled as incorrect. Any blob labeled incorrect may be used for the access act. If we consider more stringent criteria such as stable values σ _c1 and σ _c2 and maximum spacing to avoid relying on insufficient vision data, we extract stable object assumptions. The approximate spherical shape ((σ _c1 − σ _c2 ) / σ _c1 <threshold) and the easiest execution of the action (eg, from the front of the body) to carry out manipulating actions such as “poke baloon”. Additional constraints, such as the minimum interval to behavior specific reference point for poking, may be imposed on stable object assumptions. Actions such as a "power grasp object" will require a minimum elongation ((σ _c1 -σ _c2 ) / σ _c1 > threshold) and an appropriate diameter (threshold <σ _c2 <threshold) for catch stability. .

E. Overall Body Behavior and Prediction

Using targets for the head, arms, and legs, the system is described in 2005 by Humanoid, M. Gienger, H. Janssen, and C. Goerick, "Task oriented whole body motion for humanoid robots." The motor commands can be generated by a full body controller.

The principle of total body motion is to use some flexible criteria in the task space and use zero space to meet some optimization criteria, such as compensation for center of mass movement and avoidance of joint limits.

Since computational costs for overall body motion are low enough, not only can they be used to directly generate robotic motion, but their rapid convergence allows simulation of different behaviors on a faster time scale than real time.

As such, it is used to support behavior selection of walking and arm actions. Four internal simulations attempt to reach the target object from the current pose continuously using either the left or right arm during standing or walking. The metric is then used to select the optimal behavior to run in real time.

F. Collision Detection

In order to ensure the safety of the robot during operation, a real-time collision detection algorithm is used. Collision detection is based on the robot body's relative to spheres and sphere-swept lines used with kinematic information to calculate the spacing between segments (limbs and body parts) of the robot. Use internal hierarchical description. If any of these intervals is less than the threshold, the high level motion control will be disabled so that only mechanical stabilization of bipedal walking will still be active.

Collision detection functions as a last safety measure and is not triggered during normal robot operation.

In addition, simple collision avoidance limits the position of all motion targets, so wrist target positions, for example, within or very close to the body, are never created.

V. Summary

The design and implementation of a system is introduced in which a human-like robot can interact with its visual environment using both legs and arms.

Targets for interaction are based on visually extracted primitive objects. The control system allows the robot to increase the range of its interaction, at the same time achieve multiple targets, and avoid unwanted poses. Several different selection mechanisms are used to switch between different kinds of actions at any time and posture.

According to the present invention, the interaction between the environment and the robot can be improved based on visual information about the environment of the robot.

Claims (16)

An interactive robot comprising visual sensing means, manipulation means and computing means, the computing means comprising: Process output signals from visual sensing means to create pro-objects stored in memory, the primitive objects being blobs of interest in the input field of the visual sensing means by at least a 3d position label; Indicates blobs-, Form object assumptions about the category of the object based on the evaluation of the primitive objects for different behavior specific restrictions, Based on at least one primitive object and the assumptions as a target for the operation, to determine at least one of a visual tracking operation of the visual sensing means, an operation of the body of the robot and / or an operation of the operation means. Interactive robot engineered. The method of claim 1, The blobs are also represented by at least one of size, orientation, time of detection and accuracy labels. The method according to claim 1 or 2, Operation of the manipulation means comprises at least one of grasping and poking operations. The interactive robot of claim 1, wherein the computing means is designed to only consider primitive objects created in the recent past. The interactive robot of claim 1, wherein at least one evaluation criterion for the primitive objects is their elongation. The interactive robot of claim 1, wherein at least one evaluation criterion for the primitive objects is their spacing relative to a behavior specific reference point. The interactive robot of claim 1, wherein at least one evaluation criterion for the primitive objects is their stability over time. A method of controlling an interactive robot comprising visual sensing means, catching means and computing means, Processing output signals from the visual sensing means to generate poroto-objects, wherein the primitive objects represent blobs of interest in the input field of the visual sensing means by at least a 3d location label; Forming assumptions about the category of the object by evaluating the primitive objects, Determining at least one of a visual tracking action of the visual sensing means, an action of the body of the robot and / or an action of the manipulation means based on at least one primitive object and assumptions as a target for the action Interactive robot control method comprising a. The method of claim 7, wherein Discarding the primitive objects after the elapse of a predetermined time period or when the operation of the body of the robot reaches a predetermined threshold. The method according to claim 8 or 9, The blobs are also represented by at least one of size, orientation, time of detection and accuracy label. The method according to any one of claims 8 to 10, wherein the operation of the operation means includes at least one of a catching and extruding operation. The method according to any one of claims 8 to 11, Said computing means being designed to consider only recently created primitive objects. The method according to any one of claims 8 to 12, At least one evaluation criterion for the primitive objects is their height. The method according to any one of claims 8 to 13, At least one evaluation criterion for the primitive objects is their spacing relative to a behavior specific reference point. The method according to any one of claims 8 to 14, At least one evaluation criterion for the primitive objects is their stability over time. A computer software program product supporting the method of any one of claims 8 to 15 when executed in a computing device of an interactive robot.

Images