CN107818574B - Fish shoal three-dimensional tracking method based on skeleton analysis - Google Patents

Fish shoal three-dimensional tracking method based on skeleton analysis Download PDF

Info

Publication number
CN107818574B
CN107818574B CN201710914852.5A CN201710914852A CN107818574B CN 107818574 B CN107818574 B CN 107818574B CN 201710914852 A CN201710914852 A CN 201710914852A CN 107818574 B CN107818574 B CN 107818574B
Authority
CN
China
Prior art keywords
view
target
skeleton
points
matching
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710914852.5A
Other languages
Chinese (zh)
Other versions
CN107818574A (en
Inventor
钱志明
王志刚
寸天睿
秦海菲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chuxiong Normal University
Original Assignee
Chuxiong Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chuxiong Normal University filed Critical Chuxiong Normal University
Priority to CN201710914852.5A priority Critical patent/CN107818574B/en
Publication of CN107818574A publication Critical patent/CN107818574A/en
Application granted granted Critical
Publication of CN107818574B publication Critical patent/CN107818574B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/285—Analysis of motion using a sequence of stereo image pairs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38—Registration of image sequences
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a fish school three-dimensional tracking method based on skeleton analysis, which relates to the technical field of information. According to the fish school three-dimensional tracking method based on skeleton analysis, effective three-dimensional tracking can be carried out on the fish school target only by two cameras, and the method not only has high accuracy, but also has high tracking speed.

Description

Fish shoal three-dimensional tracking method based on skeleton analysis
Technical Field
The invention relates to the technical field of information, in particular to a fish school three-dimensional tracking method based on skeleton analysis.
Background
Reference Qian Z M, Chen Y Q. feature point based 3D tracking of multiple fish from multiple-view images [ J ]. ploS one, 2017, 12 (6): e0180254 provides a three-dimensional fish school tracking method based on three views. According to the method, firstly, a target in multiple views is simplified into a characteristic point representation by utilizing skeleton analysis, then, according to an obtained characteristic point model, the tracking in a top view direction is taken as a main reference, the tracking in two side view directions is taken as a reference, the target is matched and associated, and finally, the motion track of the target in a three-dimensional space is obtained.
In the tracking method of the reference, when the top view has occlusion, the method uses the tracking results of two side view directions to correlate the targets before and after the occlusion, so that the tracking performance can be improved and the tracking result is more reliable, but at the expense of the tracking efficiency, if the target of the top view has no occlusion, the tracking in the side view is not necessary.
Disclosure of Invention
The invention aims to solve the technical problem of providing a fish school three-dimensional tracking method based on skeleton analysis, which mainly tracks in the top view direction, does not use the detection result in the side view direction for tracking, only uses the detection result in the side view direction for stereo matching with the top view tracking result, reduces the complexity of the tracking method and improves the tracking efficiency.
In order to achieve the above purpose, the present invention provides the following technical solutions:
the fish school three-dimensional tracking method based on skeleton analysis comprises the following steps:
(1) estimating the motion direction of the target by using a plurality of points near the skeleton endpoint, assuming the skeleton endpoint to be p (x, y), and defining the endpoint section es as n skeleton points es { (x) of the adjacent endpointi，yi) I 1., n }, the direction of the end point segment may be calculated according to the least square method,
Figure BSA0000151483460000011
a characteristic point F (p, theta) formed by combining the skeleton endpoint p and the endpoint section direction theta can represent the target;
(2) in the head and tail characteristic points of the target, the tail characteristic point of the target in a top view can be removed according to the asymmetry of the shape, and the head and tail characteristic points are still reserved in the side view direction;
(3) and carrying out data association on the feature points between adjacent frames of the top view to obtain a top view two-dimensional tracking track. Next, stereo matching is carried out by using the top-view tracking track and the characteristic points of the target in the side-view direction, so that the position of the target in a three-dimensional space can be obtained, and the stereo matching is completed by using motion continuity in order to solve the problem of uncertainty of the stereo matching;
suppose that
Figure BSA0000151483460000021
And
Figure BSA0000151483460000022
representing a feature point in top and side views, respectively, if under epipolar constraint,
Figure BSA0000151483460000023
if there are k possible matching candidate feature points in the side view, then the motion continuity constraint is defined as follows:
Figure BSA0000151483460000024
Figure BSA0000151483460000025
wherein the content of the first and second substances,
Figure BSA0000151483460000026
feature points representing time t-1 and top view
Figure BSA0000151483460000027
Characteristic points of the matching side view. pc (personal computer)maxAnd dcmaxRespectively representing the maximum movement distance and the maximum deflection angle of the adjacent frame objects,
Figure BSA0000151483460000028
and
Figure BSA0000151483460000029
respectively represent the feature points
Figure BSA00001514834600000210
And
Figure BSA00001514834600000211
and w and (1-w) represent the weight of the position and the direction in the cost function, respectively. The above expression indicates that, of the k candidate feature points in the side view, a feature point having the best motion continuity with the matching point at the previous time is selected as the matching point at the current time.
The beneficial effect of adopting above technical scheme is: the fish school three-dimensional tracking method based on skeleton analysis mainly tracks in the top view direction, detection results in the side view direction are not used for tracking, and only are used for carrying out three-dimensional matching with top view tracking results, so that the complexity of the tracking method is reduced, and the tracking efficiency is improved. According to the fish school three-dimensional tracking method based on skeleton analysis, effective three-dimensional tracking can be carried out on the fish school target only by two cameras, and the method not only has high accuracy, but also has high tracking speed.
Drawings
The following describes embodiments of the present invention in further detail with reference to the accompanying drawings.
FIG. 1 is a flow chart of fish school three-dimensional tracking based on skeletal analysis;
FIG. 2 is a top view of stereo matching based on motion continuity;
fig. 3 is a perspective matching side view based on motion continuity.
Detailed Description
The following describes a preferred embodiment of the fish school three-dimensional tracking method based on skeleton analysis according to the present invention in detail with reference to the accompanying drawings.
Fig. 1, fig. 2 and fig. 3 show a specific embodiment of the fish school three-dimensional tracking method based on skeleton analysis of the present invention:
the fish school three-dimensional tracking method based on skeleton analysis mainly tracks in the top view direction, detection results in the side view direction are not used for tracking, and only are used for carrying out three-dimensional matching with top view tracking results, so that complexity of the tracking method is reduced, and tracking efficiency is improved. Fig. 1 shows a flow chart of the proposed method. Due to the motion region segmentation and skeleton extraction method and the documents Qian Z M, Chen Y Q.feature point based 3D tracking of multiple fish from multiple-view images [ J ]. ploS one, 2017, 12 (6): e0180254 proposes a three-view-based fish shoal three-dimensional tracking method to keep consistency, and the method is not described here.
The fish school three-dimensional tracking method based on skeleton analysis comprises the following steps:
(1) estimating the motion direction of the target by using a plurality of points near the skeleton endpoint, assuming the skeleton endpoint to be p (x, y), and defining the endpoint section es as n skeleton points es { (x) of the adjacent endpointi，yi) I 1., n }, the direction of the end point segment may be calculated according to the least square method,
Figure BSA0000151483460000031
a characteristic point F (p, theta) formed by combining the skeleton endpoint p and the endpoint section direction theta can represent the target;
this representation has the following advantages: (1) the data volume is small. Targets in different view directions can be effectively represented only by using two points with directions, so that the tracking difficulty is greatly reduced; (2) the shielding processing capacity is strong. Most of the shielding targets can be effectively represented, and the accuracy of shielding tracking is improved.
(2) In the head and tail characteristic points of the target, the tail characteristic point of the target in a top view can be removed according to the asymmetry of the shape, and the head and tail characteristic points are still reserved in the side view direction;
(3) and carrying out data association on the feature points between adjacent frames of the top view to obtain a top view two-dimensional tracking track. Next, stereo matching is carried out by using the top-view tracking track and the characteristic points of the target in the side-view direction, so that the position of the target in a three-dimensional space can be obtained, and the stereo matching is completed by using motion continuity in order to solve the problem of uncertainty of the stereo matching;
suppose that
Figure BSA0000151483460000041
And
Figure BSA0000151483460000042
representing a feature point in top and side views, respectively, if under epipolar constraint,
Figure BSA0000151483460000043
if there are k possible matching candidate feature points in the side view, then the motion continuity constraint is defined as follows:
Figure BSA0000151483460000044
Figure BSA0000151483460000045
wherein the content of the first and second substances,
Figure BSA0000151483460000046
feature points representing time t-1 and top view
Figure BSA0000151483460000047
Characteristic points of the matching side view. pc (personal computer)maxAnd dcmaxRespectively representing the maximum movement distance and the maximum deflection angle of the adjacent frame objects,
Figure BSA0000151483460000048
and
Figure BSA0000151483460000049
respectively represent the feature points
Figure BSA00001514834600000410
And
Figure BSA00001514834600000411
w and (1-w) respectively represent the weight of the position and the direction in the cost function, and the above formula shows that, in the k candidate feature points in the side view, the feature point having the best motion continuity with the matching point at the previous moment is selected as the matching point at the current moment.
Fig. 2 and 3 show an example of stereo matching. The dashed arrows in fig. 2 and 3 indicate polar lines. Target i in the top view of FIG. 2t In the side view of FIG. 3K candidate matching targets exist on the corresponding polar line, and i is selectedtThe target for which the matching point at the previous time has the best motion continuity is taken as the matching target at the current time.
According to the fish school three-dimensional tracking method based on skeleton analysis, effective three-dimensional tracking can be carried out on the fish school target only by two cameras, and the method not only has high accuracy, but also has high tracking speed.
The above is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, many variations and modifications can be made without departing from the inventive concept of the present invention, which falls into the protection scope of the present invention.

Claims (1)

1. A fish school three-dimensional tracking method based on skeleton analysis is characterized by comprising the following steps: the fish school three-dimensional tracking method based on skeleton analysis comprises the following steps:
(1) estimating the motion direction of the target by using a plurality of points near the skeleton endpoint, assuming the skeleton endpoint to be p (x, y), and defining the endpoint section es as n skeleton points es { (x) of the adjacent endpointi,yi) I 1., n }, the direction of the end point segment may be calculated according to the least square method,
Figure FDA0003170048460000011
a characteristic point F (p, theta) formed by combining the skeleton endpoint p and the endpoint section direction theta can represent the target;
(2) in the head and tail characteristic points of the target, the tail characteristic point of the target in a top view can be removed according to the asymmetry of the shape, and the head and tail characteristic points are still reserved in the side view direction;
(3) performing data association on feature points between adjacent frames of the top view to obtain a top view two-dimensional tracking track; next, stereo matching is carried out by using the top-view tracking track and the characteristic points of the target in the side-view direction, so that the position of the target in a three-dimensional space can be obtained, and the stereo matching is completed by using motion continuity in order to solve the problem of uncertainty of the stereo matching;
suppose that
Figure FDA0003170048460000012
And
Figure FDA0003170048460000013
representing a feature point in top and side views, respectively, if under epipolar constraint,
Figure FDA0003170048460000014
if there are k possible matching candidate feature points in the side view, then the motion continuity constraint is defined as follows:
Figure FDA0003170048460000015
Figure FDA0003170048460000016
wherein the content of the first and second substances,
Figure FDA0003170048460000017
feature points representing time t-1 and top view
Figure FDA0003170048460000018
Characteristic points of the matching side view, pcmaxAnd dcmaxRespectively representing the maximum movement distance and the maximum deflection angle of the adjacent frame objects,
Figure FDA0003170048460000019
and
Figure FDA00031700484600000110
respectively represent the feature points
Figure FDA00031700484600000111
And
Figure FDA00031700484600000112
w and (1-w) represent the weight of the position and the direction in the cost function respectively; on the upper partThe formula indicates that, of the k candidate feature points in the side view, a feature point having the best motion continuity with the matching point at the previous time is selected as the matching point at the current time.
CN201710914852.5A 2017-09-21 2017-09-21 Fish shoal three-dimensional tracking method based on skeleton analysis Active CN107818574B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710914852.5A CN107818574B (en) 2017-09-21 2017-09-21 Fish shoal three-dimensional tracking method based on skeleton analysis

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710914852.5A CN107818574B (en) 2017-09-21 2017-09-21 Fish shoal three-dimensional tracking method based on skeleton analysis

Publications (2)

Publication Number Publication Date
CN107818574A CN107818574A (en) 2018-03-20
CN107818574B true CN107818574B (en) 2021-08-27

Family

ID=61607068

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710914852.5A Active CN107818574B (en) 2017-09-21 2017-09-21 Fish shoal three-dimensional tracking method based on skeleton analysis

Country Status (1)

Country Link
CN (1) CN107818574B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111476828B (en) * 2020-03-27 2023-01-10 清华大学 Multi-view animal group tracking method and device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101739568A (en) * 2009-11-04 2010-06-16 北京交通大学 Layered observation vector decomposed hidden Markov model-based method for identifying behaviors
CN102609954A (en) * 2010-12-17 2012-07-25 微软公司 Validation analysis of human target
CN104867135A (en) * 2015-05-04 2015-08-26 中国科学院上海微系统与信息技术研究所 High-precision stereo matching method based on guiding image guidance
CN105225229A (en) * 2015-09-07 2016-01-06 三峡大学 Fish based on vision signal cross dam movement locus locating device and method
CN106875429A (en) * 2017-03-02 2017-06-20 楚雄师范学院 Shoal of fish three-dimensional tracking and system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101739568A (en) * 2009-11-04 2010-06-16 北京交通大学 Layered observation vector decomposed hidden Markov model-based method for identifying behaviors
CN102609954A (en) * 2010-12-17 2012-07-25 微软公司 Validation analysis of human target
CN104867135A (en) * 2015-05-04 2015-08-26 中国科学院上海微系统与信息技术研究所 High-precision stereo matching method based on guiding image guidance
CN105225229A (en) * 2015-09-07 2016-01-06 三峡大学 Fish based on vision signal cross dam movement locus locating device and method
CN106875429A (en) * 2017-03-02 2017-06-20 楚雄师范学院 Shoal of fish three-dimensional tracking and system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
An effective and robust method for tracking multiple fish in video image based on fish head detection;Zhi-Ming Qian等;《BMC Bioinformatics》;20161231;全文 *
半遮挡目标鱼体的识别与跟踪方法研究_毛家发半遮挡目标鱼体的识别与跟踪方法研究;毛家发等;《浙江工业大学学报》;20150430;第43卷(第2期);第1-11页 *

Also Published As

Publication number Publication date
CN107818574A (en) 2018-03-20

Similar Documents

Publication Publication Date Title
US10354129B2 (en) Hand gesture recognition for virtual reality and augmented reality devices
US10366538B2 (en) Method and device for illustrating a virtual object in a real environment
Ttofis et al. A low-cost real-time embedded stereo vision system for accurate disparity estimation based on guided image filtering
US9390560B2 (en) Method and device for illustrating a virtual object in a real environment
US8199977B2 (en) System and method for extraction of features from a 3-D point cloud
CN108776989B (en) Low-texture planar scene reconstruction method based on sparse SLAM framework
CN102902355A (en) Space interaction method of mobile equipment
US8929587B2 (en) Multi-tracking of video objects
Rangesh et al. Ground plane polling for 6dof pose estimation of objects on the road
Li et al. RGBD relocalisation using pairwise geometry and concise key point sets
CN109961461B (en) Multi-moving-object tracking method based on three-dimensional layered graph model
Xu et al. Video-object segmentation and 3D-trajectory estimation for monocular video sequences
CN107818574B (en) Fish shoal three-dimensional tracking method based on skeleton analysis
US20220277595A1 (en) Hand gesture detection method and apparatus, and computer storage medium
Qian et al. Skeleton-based 3D tracking of multiple fish from two orthogonal views
CN103295220A (en) Application method of binocular vision technology in recovery physiotherapy system
Chattopadhyay et al. STAR: a content based video retrieval system for moving camera video shots
CN105631938A (en) Image processing method and electronic equipment
Yoshimoto et al. Cubistic representation for real-time 3D shape and pose estimation of unknown rigid object
JP6910622B2 (en) Image processing system
Wang et al. Local disparity refinement with disparity inheritance
Duanmu et al. A multi-view pedestrian tracking framework based on graph matching
Fan et al. Human-M3: A Multi-view Multi-modal Dataset for 3D Human Pose Estimation in Outdoor Scenes
Marugame et al. Focused object extraction with multiple cameras
Verleysen et al. Wide-baseline foreground object interpolation using silhouette shape prior

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant