CN106920250A - Robot target identification and localization method and system based on RGB D videos - Google Patents

Abstract

The invention discloses a robot target recognition and positioning method and system based on RGB-D video, through the steps of target candidate extraction, recognition, confidence estimation based on timing consistency, target segmentation optimization, position estimation, etc., in the scene to determine target category and obtain accurate spatial location positioning. In the present invention, the depth information of the scene is used to enhance the spatial level perception ability of the recognition and positioning algorithm, and the long-sequence target recognition and positioning tasks are ensured while improving the video processing efficiency by adopting the long-short time-spatial-space consistency constraints based on key frames The identity and relatedness of the target. In the localization process, the collaborative object localization in multiple information modalities is realized by accurately segmenting the object in planar space and evaluating the location consistency of the same object in the depth information space. With small amount of calculation, good real-time performance, and high recognition and positioning accuracy, it can be applied to robot tasks based on online visual information analysis and understanding technology.

Description

Robot target recognition and positioning method and system based on RGB-D video

technical field

The invention belongs to the technical field of computer vision, and more specifically relates to a method and system for robot target recognition and positioning based on RGB-D video.

Background technique

In recent years, with the rapid development of robot technology, machine vision technology for robot tasks has also received extensive attention from researchers. Among them, the recognition and precise positioning of the target is an important part of the robot vision problem and a prerequisite for performing subsequent tasks.

Existing object recognition methods generally include two steps: extracting target information to be recognized as the basis for recognition and matching with the scene to be recognized. The traditional expression of the target to be recognized generally includes methods such as geometric shape, target appearance, and extraction of local features. Such methods often have shortcomings such as poor versatility, insufficient stability, and poor target abstraction ability. The defects of the above target expression also bring insurmountable difficulties to the subsequent matching process.

After obtaining the expression of the target to be recognized, target matching refers to comparing the obtained target expression with the features of the scene to be recognized to identify the target. Generally speaking, existing methods include two types of methods based on region matching and feature matching. Region-based matching refers to extracting the information of local sub-regions of the image for comparison, and the calculation amount is proportional to the number of sub-regions to be matched; the feature-based method matches typical features in the image, and the matching accuracy is the same as that of the feature Expression effectiveness is closely related. The above two types of methods put forward high requirements for the acquisition of candidate regions and feature expression, but due to the limitations of two-dimensional plane image information and design features, they are often less effective in robot-oriented complex environment recognition tasks.

Target positioning widely exists in industrial production and life, such as GPS in outdoor sports, military radar monitoring, ship sonar equipment, etc. Such equipment has accurate positioning and a wide range of operating distances, but the price is high. Vision-based positioning system is a new research hotspot in recent years. According to different vision sensors, it can be roughly divided into positioning methods based on monocular vision sensors, binocular and depth sensors, and panoramic vision sensors. The monocular vision sensor is low in price, simple in structure, and easy to calibrate, but its positioning accuracy is often poor; the panoramic vision sensor can obtain complete scene information, and its positioning accuracy is high, but it has a large amount of calculation, poor real-time performance, and complex and expensive equipment; The depth estimation or depth information acquisition equipment of binocular vision has a strong ability to perceive the distance of the scene, and the system is relatively simple, and the real-time performance is easy to realize. In recent years, it has received more and more attention. However, the research in this field is still in its infancy, and there is still a lack of efficient object localization methods that can process RGB-Depth videos in real time.

Due to the high demand for depth information perception capabilities, most existing robot systems collect RGB-Depth videos as a source of visual information. Depth information provides rich information for stereoscopic perception of scenes, hierarchical division and positioning of complex targets. . However, due to the complexity of the robot's working scene, the high computational complexity, and the large amount of calculation, there is no systematic, fast and convenient RGB-Depth video target recognition and precise positioning method. Therefore, the research on indoor robot target recognition and precise positioning algorithm based on RGB-Depth video not only has strong research value, but also has very broad application prospects.

Contents of the invention

In view of the above defects or improvement needs of the prior art, the present invention provides a robot target recognition and positioning method and system based on RGB-D video, by processing the RGB-Depth video obtained from the first perspective of the robot, real-time and accurate Target recognition and precise positioning of the target in the robot working environment, thus assisting complex robot tasks such as target grasping. Therefore, the current technical problem of lacking an efficient target positioning method that can process RGB-Depth video in real time is solved.

In order to achieve the above object, according to one aspect of the present invention, a kind of robot target recognition and localization method based on RGB-D video is provided, comprising:

(1) Obtain the RGB-D video frame sequence of the scene where the positioning target is to be identified;

(2) extract the key video frame in the RGB-D video frame sequence, and extract the target candidate area to the key video frame, filter and screen the target candidate area according to the depth information corresponding to each key video frame;

(3) Identify the filtered and screened target candidate areas based on the deep network, and sort the target recognition results by confidence through long-term time-series spatio-temporal correlation constraints and multi-frame recognition consistency estimation;

(4) Carry out partial fast segmentation to the target candidate area after filtering and screening, according to the confidence degree of the target recognition result and the time sequence interval relationship of each key video frame, select the main key video frame from the key video frame, and segment the area Carry out adjacent frame expansion and collaborative optimization;

(5) Determine the key feature points in the scene as the positioning reference point, and then estimate the camera angle of view and camera motion estimation value. By performing the target feature consistency constraint and target position consistency constraint on the main key video frame recognition and segmentation results, estimate the to-be-identified Coordinate confidence in localizing objects and perform spatially precise localization.

Preferably, said step (2) specifically includes:

(2.1) Determine the key video frame used to identify the target to be identified and positioned with interval sampling or key frame selection method;

(2.2) Using the confidence sorting method based on similarity prior to obtain the target candidate areas in the key video frames to form a target candidate area set, using the depth information corresponding to each key video frame to obtain the interior of each target candidate area and The hierarchical attributes in the neighborhood are used to optimize, screen and reorder the set of target candidate regions.

Preferably, the step (3) specifically includes:

(3.1) Send the target candidate regions screened in step (2) into the trained target recognition depth network, and obtain the target recognition prediction results and the target recognition prediction results of the key video frames corresponding to each filtered target candidate region The first confidence level of

(3.2) According to the time-series spatio-temporal correlation constraints, the target recognition prediction results of the key video frames are evaluated for feature consistency, and the second confidence of each target recognition prediction result is evaluated, and the first confidence and the second The cumulative confidence obtained by the confidence degree is sorted, and the target candidate regions whose cumulative confidence is lower than the preset confidence threshold are further filtered out.

Preferably, said step (4) specifically includes:

(4.1) For the target candidate area and its extended neighborhood obtained in step (3.2), perform a fast target segmentation operation to obtain the initial segmentation of the target and determine the target boundary;

(4.2) take the short-time spatio-temporal consistency as constraints, based on the cumulative confidence ranking result in step (3.2), filter out the main key video frame from the key video frame;

(4.3) Constrained by long-term spatiotemporal consistency, based on the initial segmentation in step (4.1), perform appearance modeling for the target to be identified and locate, construct 3D graphics for the main key video frame and its adjacent frames, and design the maximum a posteriori Probability-Markov random field energy function, the initial segmentation is optimized through the graph cut algorithm, and the target segmentation result of a single frame is segmented, extended and optimized in the adjacent frames before and after the frame.

Preferably, said step (5) specifically includes:

(5.1) for the main key video frame that step (4.2) obtains, according to the adjacency between each main key video frame and the coincidence relation of visual field, extract many groups of point-to-point pairs with the same name as the positioning reference point;

(5.2) Estimate the change of the camera's viewing angle according to the main key video frames whose fields of view overlap, and then use the depth information of the positioning reference point pair to estimate the camera's motion information through the geometric relationship;

(5.3) Evaluate the spatial position consistency of the target to be identified and located in the main key video frame according to the measured depth information, camera angle of view and camera motion information of the target to be identified and located in the main key video frame;

(5.4) According to the result of step (4.3), evaluate the feature consistency of the two-dimensional segmentation area of the target to be identified and positioned;

(5.5) Determine the spatial position of the target to be identified and positioned by comprehensively evaluating the feature consistency and spatial position consistency of the two-dimensional segmented area of the target to be identified and positioned.

According to another aspect of the present invention, there is provided a robot target recognition and positioning system based on RGB-D video, including:

The obtaining module is used to obtain the RGB-D video frame sequence of the scene where the positioning target is to be identified;

Filtering and screening module, used to extract the key video frame in the RGB-D video frame sequence, and extract the target candidate area for the key video frame, and filter the target candidate area according to the depth information corresponding to each key video frame filter;

The confidence ranking module is used to identify the filtered and screened target candidate areas based on the deep network, and sort the target recognition results by confidence through long-term time-series spatio-temporal correlation constraints and multi-frame recognition consistency estimation;

The optimization module is used to perform local rapid segmentation on the filtered target candidate area, and select the main key video frame from the key video frame according to the confidence of the target recognition result and the time sequence interval relationship of each key video frame, and Segment the area for adjacent frame expansion and collaborative optimization;

The positioning module is used to determine key feature points in the scene as positioning reference points, and then estimate the camera angle of view and camera motion estimation value. By performing target feature consistency constraints and target position consistency constraints on the main key video frame recognition and segmentation results, estimate The collaborative confidence of the target to be identified and spatially precise positioning.

Generally speaking, compared with the prior art, the above technical solution conceived by the present invention mainly has the following technical advantages: In the present invention, the depth information of the scene is used to enhance the spatial level perception ability of the recognition and positioning algorithm. The long-short-term spatio-temporal consistency constraints of the key frames not only improve the efficiency of video processing, but also ensure the identity and relevance of targets in long-sequence target recognition and positioning tasks. In the localization process, the collaborative object localization in multiple information modalities is realized by accurately segmenting the target in planar space and evaluating the position consistency of the same target in the depth information space. Small amount of calculation, good real-time performance, high recognition and positioning accuracy, can be applied to robot tasks based on online visual information analysis and understanding technology.

Description of drawings

Fig. 1 is the overall flow diagram of the method of the embodiment of the present invention;

Fig. 2 is a schematic flow chart of target recognition in an embodiment of the present invention;

FIG. 3 is a schematic flow chart of accurate target positioning in an embodiment of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

The method disclosed in the present invention involves technologies such as key frame screening, target recognition based on deep network, segmentation, transfer between marked frames, position estimation based on consistency constraints, and collaborative optimization, and can be directly used to input visual information with RGB-D video In the robot system, the auxiliary robot completes the task of target recognition and target precise positioning.

FIG. 1 is a schematic diagram of the overall flow of the method of the embodiment of the present invention. It can be seen from Figure 1 that this method includes two steps: target recognition and target precise positioning, and target recognition is a prerequisite for precise target positioning. Its specific implementation is as follows:

(1) Obtain the RGB-D video frame sequence of the scene where the positioning target is to be identified;

Preferably, in one embodiment of the present invention, the RGB-D video sequence of the scene where the target to be identified and positioned can be collected by depth vision sensors such as Kinect; RGB image pairs can also be collected by binocular imaging equipment, and estimated by calculating the parallax The scene depth information is used as the depth channel information to synthesize RGB-D video as input.

(2) extract the key video frame in the RGB-D video frame sequence, and extract the target candidate area to the key video frame, filter and screen the target candidate area according to the depth information corresponding to each key video frame;

(3) Identify the filtered and screened target candidate areas based on the deep network, and sort the target recognition results by confidence through long-term time-series spatio-temporal correlation constraints and multi-frame recognition consistency estimation;

(4) Carry out local rapid segmentation on the target candidate area after filtering, select the main key video frame from the key video frame according to the confidence of the target recognition result and the time sequence interval relationship of each key video frame, and carry out the front and back of the segmented area Adjacent frame extension and collaborative optimization;

(5) Determine the key feature points in the scene as the positioning reference point, and then estimate the camera angle of view and camera motion estimation value. By performing the target feature consistency constraint and target position consistency constraint on the main key video frame recognition and segmentation results, estimate the to-be-identified Coordinate confidence in localizing objects and perform spatially precise localization.

Preferably, in one embodiment of the present invention, the above step (1) specifically includes:

(1.1) Use Kinect to collect the RGB-D video sequence of the scene where the target to be identified is located, and fill the depth image hole in a smooth manner with neighborhood sampling, correct it according to the Kinect parameters and convert it into actual depth information, and use the RGB data as input;

(1.2) When a binocular device is used to collect image pairs, camera calibration, stereo matching (image pair feature extraction, corresponding point extraction of the same physical structure, and parallax calculation) steps are performed in sequence, and finally the depth is estimated by the projection model as the depth channel in the video. enter.

Preferably, in one embodiment of the present invention, the above step (2) specifically includes:

(2.1) Determine the key video frame used to identify the target to be identified and positioned with interval sampling or key frame selection method;

Wherein, the step (2.1) specifically includes: using the fast scale-invariant feature transform (Scale-invariant feature transform, SIFT) point matching method to obtain the scene overlap rate of adjacent frames, thereby estimating the scene change rate of the current shooting, which is relatively difficult for shooting scene switching. For fast video frames, increase the sampling frequency, and for slow video frames, reduce the sampling frequency. In addition, when the actual application requirements require high algorithm efficiency, the interval sampling method can be directly used to replace this step.

(2.2) Using the confidence sorting method based on similarity prior to obtain the target candidate areas in the key video frames to form a target candidate area set, using the depth information corresponding to each key video frame to obtain the interior of each target candidate area and The hierarchical attributes in the neighborhood are used to optimize, screen and reorder the set of target candidate regions.

Among them, the confidence ranking method based on the similarity prior can be the BING algorithm or the Edge box algorithm. As shown in Figure 2, the depth information of the corresponding frame is used to obtain the hierarchical attributes of the target candidate area and its neighborhood. According to the high confidence candidate frame, the depth information should be smooth, and the depth information gradient at the inner and outer boundaries of the frame is relatively large. According to the principle, the set of target candidate regions is optimally screened and re-ranked.

Preferably, in one embodiment of the present invention, the above step (3) specifically includes:

(3.1) As shown in Figure 2, send the target candidate areas screened in step (2) into the trained target recognition deep network, and obtain the target recognition prediction results of the key video frames corresponding to each screened target candidate area and the first confidence degree of each target recognition prediction result;

Wherein, the trained target recognition deep network can be deep recognition networks such as SPP-Net, R-CNN, Fast-R-CNN, etc., or can be replaced by other deep recognition networks.

(3.2) According to the time-series spatio-temporal correlation constraints, the feature consistency evaluation of the target recognition prediction results of key video frames is performed, and the second confidence degree of each target recognition prediction result is evaluated, and the first confidence degree and the second confidence degree are obtained. The cumulative confidence is sorted, and the target candidate regions whose cumulative confidence is lower than the preset confidence threshold are further filtered out.

Optionally, in an embodiment of the present invention, by applying recognition instructions to the algorithm, the detection and recognition results of the positioning target to be recognized can be obtained, and the efficiency of the algorithm can be improved by filtering the recognition results with low confidence.

Optionally, in one embodiment of the present invention, the above step (4) specifically includes:

(4.1) As shown in Figure 3, for the target candidate area and its extended neighborhood obtained in step (3.2), perform a fast target segmentation operation to obtain the initial segmentation of the target and determine the target boundary;

Wherein, as an optional implementation, the RGB-D information-based GrabCut segmentation algorithm can be used to perform fast target segmentation operations to obtain the initial segmentation of the target, thereby obtaining the two-dimensional positioning result of the target in the current video frame.

(4.2) In order to further improve the efficiency of video target location, as shown in Figure 3, with short-term spatio-temporal consistency as constraints, based on the cumulative confidence ranking results in step (3.2), a single frame with high confidence and adjacent The strong spatiotemporal consistency of frames is the criterion, and the main key video frames are selected from the key video frames;

(4.3) Constrained by long-term spatiotemporal consistency, based on the initial segmentation in step (4.1), perform appearance modeling for the target to be identified and locate, construct 3D graphics for the main key video frame and its adjacent frames, and design the maximum a posteriori The probability-Markov random field energy function optimizes the initial segmentation through the graph cut algorithm, and the target segmentation result of a single frame is segmented and extended in the adjacent frames before and after the frame, so as to realize the long-short-time spatio-temporal consistency 2D Object Segmentation and Localization Optimization.

Optionally, in one embodiment of the present invention, the above step (5) specifically includes:

(5.1) As shown in Figure 3, for the main key video frame that step (4.2) obtains, according to the adjacency and visual field overlapping relationship between each main key video frame, extract many groups of point pairs with the same name as the positioning reference point;

(5.2) Estimate the change of the camera's viewing angle according to the main key video frames whose fields of view overlap, and then use the depth information of the positioning reference point pair to estimate the camera's motion information through the geometric relationship;

Wherein, the motion information of the camera includes a moving distance and a moving track of the camera.

(5.3) As shown in Figure 3, according to the measured depth information, camera angle of view and camera motion information of the target to be identified and located in the main key video frame, evaluate the spatial position consistency of the target to be identified and located in the main key video frame;

(5.4) According to the result of step (4.3), evaluate the feature consistency of the two-dimensional segmented area of the target to be identified and positioned, generally using a region-based deep network to extract regional depth features for feature distance measurement and feature consistency evaluation;

(5.5) Determine the spatial position of the target to be identified and positioned by comprehensively evaluating the feature consistency and spatial position consistency of the two-dimensional segmented area of the target to be identified and positioned.

In one embodiment of the present invention, a kind of robot target recognition and localization system based on RGB-D video is disclosed, and this system comprises:

The obtaining module is used to obtain the RGB-D video frame sequence of the scene where the positioning target is to be identified;

Filtering and screening module, used to extract the key video frame in the RGB-D video frame sequence, and extract the target candidate area for the key video frame, and filter the target candidate area according to the depth information corresponding to each key video frame filter;

The confidence ranking module is used to identify the filtered and screened target candidate areas based on the deep network, and sort the target recognition results by confidence through long-term time-series spatio-temporal correlation constraints and multi-frame recognition consistency estimation;

The optimization module is used to perform local rapid segmentation on the filtered target candidate area, and select the main key video frame from the key video frame according to the confidence of the target recognition result and the time sequence interval relationship of each key video frame, and Segment the area for adjacent frame expansion and collaborative optimization;

The positioning module is used to determine key feature points in the scene as positioning reference points, and then estimate the camera angle of view and camera motion estimation value. By performing target feature consistency constraints and target position consistency constraints on the main key video frame recognition and segmentation results, estimate The collaborative confidence of the target to be identified and spatially precise positioning.

Wherein, for the specific implementation manner of each module, reference may be made to the description of the method embodiment, and the embodiment of the present invention will not be described again.

Those skilled in the art can easily understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (6)

1. a kind of robot target identification and localization method based on RGB-D videos, it is characterised in that including：

(1) the RGB-D sequence of frames of video of scene where positioning target to be identified is obtained；

(2) the key video sequence frame in the RGB-D sequence of frames of video is extracted, and target candidate area is extracted to the key video sequence frame Domain, filtering screening is carried out according to the corresponding depth information of each key video sequence frame to the object candidate area；

(3) object candidate area after filtering screening is identified based on depth network, is constrained by sequential space time correlation long And multiframe identification Uniform estimates, confidence level sequence is carried out to target identification result；

(4) local Fast Segmentation is carried out to the object candidate area after filtering screening, confidence level according to target identification result and The timing intervals relation of each key video sequence frame, chooses Chief frame of video from the key video sequence frame, and to cut zone Carry out front and rear consecutive frame extension and collaboration optimization；

(5) key feature points are determined in the scene as positioning reference point, and then estimate camera perspective and camera motion estimate, Recognize that segmentation result carries out target signature consistency constraint and target location consistency constraint by Chief frame of video, estimate Count the collaboration confidence level of positioning target to be identified and carry out space and be accurately positioned.

2. method according to claim 1, it is characterised in that the step (2) specifically includes：

(2.1) with interval sampling or key frame extraction method, it is determined that the key video sequence frame for recognizing positioning target to be identified；

(2.2) using based on the object candidate area in the confidence level sort method acquisition key video sequence frame like physical property priori Composition object candidate area set, using the corresponding depth information of each key video sequence frame, obtains the inside of each object candidate area And its hierarchy attributes in neighborhood, screening is optimized to the object candidate area set, is sorted again.

3. method according to claim 2, it is characterised in that the step (3) specifically includes：

(3.1) the target identification depth network that will have been trained by the object candidate area feeding after step (2) screening, obtains The target identification of the corresponding key video sequence frame of object candidate area after each screening predicts the outcome and each target identification predicts the outcome The first confidence level；

(3.2) the space time correlation constraint according to sequential long, the target identification of key video sequence frame is predicted the outcome, and it is consistent to carry out feature Property evaluate, evaluate the second confidence level that each target identification predicts the outcome, will be by first confidence level and second confidence level The accumulation confidence level for obtaining is ranked up, and further filters out the target candidate area that accumulation confidence level is less than default confidence threshold value Domain.

4. method according to claim 3, it is characterised in that the step (4) specifically includes：

(4.1) object candidate area and its extension neighborhood for being obtained for step (3.2), carry out quick Target Segmentation operation, The initial segmentation of target is obtained, object boundary is determined；

(4.2) it is constraint with space-time consistency in short-term, based on the accumulation confidence level ranking results in step (3.2), from the pass Chief frame of video is filtered out in key frame of video；

(4.3) with it is long when space-time consistency be constraint, based on the initial segmentation of step (4.1), positioning target to be identified is carried out Outward appearance is modeled, and carries out 3-D graphic structure to Chief frame of video and its consecutive frame, and design maximum a posteriori probability-Ma Erke Husband's random field energy function, cuts algorithm and initial segmentation is optimized by figure, to the object segmentation result of single frames before the frame Carry out splitting extension and optimization in consecutive frame afterwards.

5. method according to claim 4, it is characterised in that the step (5) specifically includes：

(5.1) the Chief frame of video obtained for step (4.2), according to adjacent between each Chief frame of video and regards Wild coincidence relation, extracts multigroup same place point to as positioning reference point；

(5.2) the Chief frame of video overlapped according to the visual field estimates camera perspective change, and then by geometrical relationship, using fixed The depth information of position reference point point pair estimates the movable information of camera；

(5.3) according to the information that fathoms of positioning target to be identified, camera perspective and camera in Chief frame of video Movable information, evaluates the locus uniformity of positioning target to be identified in Chief frame of video；

(5.4) according to the result of step (4.3), the feature consistency of positioning target two dimension cut zone to be identified is evaluated；

(5.5) feature consistency and locus by overall merit positioning target two dimension cut zone to be identified are consistent Property, determine the locus of positioning target to be identified.

6. a kind of robot target identification and alignment system based on RGB-D videos, it is characterised in that including：

Acquisition module, the RGB-D sequence of frames of video for obtaining scene where positioning target to be identified；

Filtering screening module, for extracting the key video sequence frame in the RGB-D sequence of frames of video, and to the key video sequence frame Object candidate area is extracted, sieves is carried out to the object candidate area according to the corresponding depth information of each key video sequence frame Choosing；

Confidence level order module, for being identified to the object candidate area after filtering screening based on depth network, by length Sequential space time correlation is constrained and multiframe identification Uniform estimates, and confidence level sequence is carried out to target identification result；

Optimization module, for carrying out local Fast Segmentation to the object candidate area after filtering screening, according to target identification result Confidence level and each key video sequence frame timing intervals relation, from the key video sequence frame choose Chief frame of video, and Front and rear consecutive frame extension and collaboration optimization are carried out to cut zone；

Locating module, for determining key feature points in the scene as positioning reference point, and then estimates camera perspective and camera Motion estimated values, recognize that segmentation result carries out target signature consistency constraint and target location one by Chief frame of video The constraint of cause property, estimates the collaboration confidence level of positioning target to be identified and carries out space and be accurately positioned.