CN112906438B - Human body action behavior prediction method and computer equipment - Google Patents

Human body action behavior prediction method and computer equipment Download PDF

Info

Publication number
CN112906438B
CN112906438B CN201911224818.0A CN201911224818A CN112906438B CN 112906438 B CN112906438 B CN 112906438B CN 201911224818 A CN201911224818 A CN 201911224818A CN 112906438 B CN112906438 B CN 112906438B
Authority
CN
China
Prior art keywords
characteristic
action behavior
frame
human
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911224818.0A
Other languages
Chinese (zh)
Other versions
CN112906438A (en
Inventor
李建军
李轲赛
刘慧婷
张宝华
张超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inner Mongolia University of Science and Technology
Original Assignee
Inner Mongolia University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inner Mongolia University of Science and Technology filed Critical Inner Mongolia University of Science and Technology
Priority to CN201911224818.0A priority Critical patent/CN112906438B/en
Publication of CN112906438A publication Critical patent/CN112906438A/en
Application granted granted Critical
Publication of CN112906438B publication Critical patent/CN112906438B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/241Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/46Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462Salient features, e.g. scale invariant feature transforms [SIFT]
    • G06V10/464Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/50Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a prediction method of human body action behaviors and computer equipment, wherein the prediction method comprises the following steps: frame-dividing and sampling the human body action period image information to obtain an image sequence with reduced frame number; sequentially aiming at the images of a plurality of frames of the image sequence according to the ordering of the frames, establishing a skeleton model aiming at each frame of image, and determining the actual values of a plurality of characteristic angles of the skeleton model; according to the reference values of a plurality of characteristic angles corresponding to a plurality of preset human body action behavior categories obtained through pre-training and learning, actual values of a plurality of characteristic angles of the skeleton model are compared with the reference values of a plurality of characteristic angles corresponding to the preset human body action behavior categories; and determining the human action behavior category of each frame of image according to the comparison result. The invention can improve the recognition accuracy of human motion behavior prediction.

Description

人体动作行为的预测方法以及计算机设备Prediction method and computer equipment of human action behavior

技术领域technical field

本发明涉及计算机视觉技术和模式识别领域,尤其涉及一种人体动作行为的预测方法以及计算机设备。The invention relates to the fields of computer vision technology and pattern recognition, in particular to a method for predicting human action behavior and computer equipment.

背景技术Background technique

人工智能的快速发展使计算机视觉技术在视频监控、运动检索、人机交互、智能家居以及医疗保健等众多领域得到了广泛应用。人体行为识别技术已经取得了一定的成果,能够对简单的人类行为,如行走,跑步,弯腰,挥手等动作行为进行准确的识别和分类。在动作行为检测过程中,往往希望能够快速的获得自己所需,减少不必要的时间浪费。因此,视频序列中的动作行为关键帧检索就有着重要的意义。人体动作行为的预测就是其中一个重要的基础。The rapid development of artificial intelligence has made computer vision technology widely used in many fields such as video surveillance, motion retrieval, human-computer interaction, smart home, and healthcare. Human behavior recognition technology has achieved certain results, which can accurately identify and classify simple human behaviors, such as walking, running, bending, waving and other actions. In the process of action behavior detection, it is often hoped to quickly obtain what one needs and reduce unnecessary waste of time. Therefore, the key frame retrieval of action behavior in video sequences is of great significance. The prediction of human action behavior is one of the important foundations.

人体动作行为预测就是对尚未完成整体行为的动作尽可能地推测出该动作的所属类别,较传统的人体动作识别缺少了完整动作的时序结构。Ryoo等通过改进的两种词袋模型,提取动作时序结构相关的中层特征,率先提出动作预测问题。Kong等人提出多时间尺度的概念,将视频进行固定数量的分割,对视频段逐个提取行为的动态特征,通过约束动作的时序关系完成对非完整动作的判别能力。Xu等人利用信息检索中的自动补全思想提出动作自动补全方法,将每个分割的视频段等价于一个句子的单个字符,不完整视频就构成了该语句的前缀,通过相似度进行视频中动作的预测。已知的人体动作行为预测算法识别精度较低。Human action prediction is to guess the category of the action that has not yet completed the overall action as much as possible. Compared with the traditional human action recognition, it lacks the timing structure of the complete action. Ryoo et al. extracted the middle-level features related to the temporal structure of actions through improved two bag-of-words models, and took the lead in proposing the problem of action prediction. Kong et al. proposed the concept of multi-time scales, divided the video into a fixed number of segments, extracted the dynamic features of the behavior one by one from the video segments, and completed the ability to discriminate incomplete actions by constraining the temporal relationship of actions. Xu et al. used the idea of automatic completion in information retrieval to propose an automatic action completion method. Each segmented video segment is equivalent to a single character of a sentence, and the incomplete video constitutes the prefix of the sentence. Prediction of actions in videos. Known human action behavior prediction algorithms have low recognition accuracy.

发明内容Contents of the invention

有鉴于此,本发明提供一种人体动作行为的预测方法以及计算机设备,以提高人体动作行为预测的识别精度。In view of this, the present invention provides a method for predicting human action behavior and computer equipment, so as to improve the recognition accuracy of human action behavior prediction.

一方面,本发明提供一种人体动作行为的预测方法,包括:对人体动作周期图像信息进行分帧采样得到帧数减少后的图像序列;根据各帧的排序依次针对所述图像序列的多个帧的图像进行如下操作:针对每帧图像建立骨架模型,并确定所述骨架模型的多个特征角度的实际值;根据预先训练学习得到的多个预定人体动作行为类别对应的多个特征角度的基准值,将所述骨架模型的多个特征角度的实际值与预定人体动作行为类别对应的多个特征角度的基准值进行比对;根据比对结果确定每帧图像所属的人体动作行为类别。On the one hand, the present invention provides a method for predicting human action behavior, including: performing frame-by-frame sampling on the image information of the human action cycle to obtain an image sequence with a reduced number of frames; The image of frame carries out following operation: set up skeleton model for each frame image, and determine the actual value of a plurality of characteristic angles of described skeleton model; According to the multiple characteristic angles corresponding to a plurality of predetermined human action behavior categories that pre-training learning obtains The reference value is to compare the actual values of the multiple characteristic angles of the skeleton model with the reference values of the multiple characteristic angles corresponding to the predetermined human action category; determine the human action category to which each frame of image belongs according to the comparison result.

进一步地,上述针对每帧图像建立骨架模型,并确定所述骨架模型的多个特征角度的实际值,包括:针对每帧图像,建立骨架模型,所述骨架模型包括上肢部分,下肢部分以及中间部分;所述上肢部分包括:左手、左肘、左肩、右手、右肘、以及右肩;所述下肢部分包括:左脚、左膝、右脚、以及右膝;所述中间部分包括胯部;Further, the above-mentioned establishment of a skeleton model for each frame of image, and determining the actual values of multiple characteristic angles of the skeleton model, includes: establishment of a skeleton model for each frame of image, and the skeleton model includes an upper limb part, a lower limb part and a middle part; the upper body part includes: left hand, left elbow, left shoulder, right hand, right elbow, and right shoulder; the lower body part includes: left foot, left knee, right foot, and right knee; the middle part includes the crotch ;

分别确定左手、左肘及左肩形成的第1个特征角度的实际值,左肘、左肩及胯部形成的第2个特征角度的实际值,右手、右肘及右肩形成的第3个特征角度的实际值,右肘、右肩及胯部形成的第4个特征角度的实际值,左脚、左膝及胯部形成的第5个特征角度的实际值,右脚、右膝及胯部形成的第6个特征角度的实际值,左膝、胯部及右膝形成的第7个特征角度的实际值;其中,所述第1个至第4个特征角度来源于所述上肢部分,所述第5个至第6个特征角度来源于所述下肢部分,第7个特征角度来源于所述中间部分。Determine the actual value of the first characteristic angle formed by the left hand, left elbow and left shoulder, the actual value of the second characteristic angle formed by the left elbow, left shoulder and crotch, and the third characteristic angle formed by the right hand, right elbow and right shoulder The actual value of the angle, the actual value of the fourth characteristic angle formed by the right elbow, right shoulder and hip, the actual value of the fifth characteristic angle formed by the left foot, left knee and hip, the actual value of the right foot, right knee and hip The actual value of the 6th characteristic angle formed by the upper body, the actual value of the 7th characteristic angle formed by the left knee, the crotch and the right knee; wherein, the first to the fourth characteristic angles are derived from the upper limb part , the fifth to sixth characteristic angles originate from the lower limb part, and the seventh characteristic angle originates from the middle part.

进一步地,上述根据预先训练学习得到的多个预定人体动作行为类别对应的多个特征角度的基准值,将所述骨架模型的多个特征角度的实际值与预定人体动作行为类别对应的多个特征角度的基准值进行比对,包括:依次判断第n个特征角度的实际值是否位于预定人体动作行为类别对应的第n个特征角度的基准值的预设偏差范围内,n为1-7之间的整数;对于每帧图像,统计位于特定人体动作行为的第1个至第7个特征角度的基准值的预设偏差范围内的各特征角度的数量;所述数量为所述比对结果;所述特定人体动作行为为预定人体动作行为类别中的任意一种。Further, according to the reference values of the multiple characteristic angles corresponding to the multiple predetermined human action behavior categories obtained through pre-training, the actual values of the multiple characteristic angles of the skeleton model and the multiple predetermined human action behavior categories correspond to Comparing the reference values of the characteristic angles includes: sequentially judging whether the actual value of the nth characteristic angle is within the preset deviation range of the reference value of the nth characteristic angle corresponding to the predetermined human action category, where n is 1-7 Integers between; For each frame of image, count the number of each characteristic angle within the preset deviation range of the reference value of the 1st to the 7th characteristic angle of a specific human action behavior; the number is the comparison Result; the specific human motion behavior is any one of the predetermined human motion behavior categories.

进一步地,上述所述根据比对结果确定每帧图像所属的人体动作行为类别包括:若数量大于或等于5个,则将对应帧的图像所属的人体动作行为类别初步确定为所述特定人体动作行为;对于每帧图像,在未位于特定人体动作行为的各特征角度的基准值的预设偏差范围内的特征角度的数量为两个,且未位于特定人体动作行为的各特征角度的基准值的预设偏差范围内的两个特征角度分别仅来源于上肢部分,下肢部分以及中间部分中的任意一个时,则将对应帧的图像所属的人体动作行为类别最终确定为所述特定人体动作行为。Further, the above-mentioned determination of the human body action category to which each frame of image belongs according to the comparison result includes: if the number is greater than or equal to 5, initially determining the human body action category to which the image of the corresponding frame belongs to the specific human action Behavior; for each frame of image, the number of characteristic angles that are not within the preset deviation range of the reference value of each characteristic angle of a specific human motion behavior is two, and are not located in the reference value of each characteristic angle of a specific human motion behavior When the two characteristic angles within the preset deviation range are only from any one of the upper limb part, the lower limb part and the middle part, the human action behavior category to which the image of the corresponding frame belongs is finally determined as the specific human action behavior .

进一步地,上述所述骨架模型为左手、左肘、左肩、右手、右肘、右肩、左脚、左膝、右脚、右膝以及胯部连接的树状结构模型。Further, the above-mentioned skeleton model is a tree structure model connecting the left hand, left elbow, left shoulder, right hand, right elbow, right shoulder, left foot, left knee, right foot, right knee and crotch.

进一步地,上述所述第n个特征角度的基准值的预设偏差范围为第n个特征角度的基准值度的范围。Further, the above-mentioned preset deviation range of the reference value of the nth characteristic angle is the range of degrees of the reference value of the nth characteristic angle.

进一步地,上述通过如下操作得到预先训练学习得到的预定人体动作行为类别对应的多个特征角度的基准值,具体包括:输入动作行为训练学习数据集;提取所述训练学习数据集的特征信息;将所述特征信息输入支持向量机进行分类,得到各训练学习数据的动作行为类别;所述各训练学习数据的动作行为类别为预定人体动作行为类别;根据每类预定动作行为类别对应的动作行为训练学习数据确定相应预定动作行为类别的多个特征角度的基准值。Further, the above-mentioned benchmark values of multiple characteristic angles corresponding to the predetermined human action behavior categories obtained through pre-training and learning through the following operations specifically include: inputting action behavior training and learning data sets; extracting feature information of the training and learning data sets; Inputting the feature information into a support vector machine for classification to obtain the action behavior category of each training and learning data; the action behavior category of each training and learning data is a predetermined human action behavior category; according to the action behavior corresponding to each type of predetermined action behavior category The training learning data determines the reference values of the plurality of characteristic angles corresponding to the predetermined action category.

进一步地,上述对人体动作周期图像信息进行分帧采样得到帧数减少后的图像序列包括:对人体动作周期图像信息,将首位五帧融合为一帧,每间隔十帧进行采样,将采样到的五帧进行融合,得到帧数减少后的图像序列。Further, the above-mentioned frame-by-frame sampling of the image information of the human body motion cycle to obtain an image sequence with a reduced number of frames includes: for the image information of the human body motion cycle, the first five frames are fused into one frame, and sampling is performed at intervals of ten frames. The five frames are fused to obtain an image sequence with a reduced number of frames.

另一方面,本发明还提供一种计算机设备,包括处理器,所述处理器执行时实现根据上述的人体动作行为的预测方法。On the other hand, the present invention also provides a computer device, which includes a processor, and the processor implements the above-mentioned method for predicting human action behavior when executed.

又一方面,本发明还提供一种存储有计算机程序的计算机可读存储介质,当所述计算机程序在被处理器执行时实现上述的人体动作行为的预测方法。In yet another aspect, the present invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned method for predicting human action behavior is realized.

本发明人体动作行为的预测方法以及计算机设备,通过对人体动作周期图像信息进行分帧采样得到帧数减少后的图像序列,能降低计算负担;根据各帧的排序依次针对所述图像序列的多个帧的图像进行如下操作:针对每帧图像建立骨架模型,并确定所述骨架模型的多个特征角度的实际值;根据预先训练学习得到的多个预定人体动作行为类别对应的多个特征角度的基准值,将所述骨架模型的多个特征角度的实际值与预定人体动作行为类别对应的多个特征角度的基准值进行比对;根据比对结果确定每帧图像所属的人体动作行为类别。通过预先训练学习得到的预定人体动作行为类别对应的多个特征角度的基准值,确定较为精确的多个特征角度的基准值,进而通过与多个特征角度的基准值的比对结果进行特征行为预测,以提高人体动作行为预测的识别精度。The method for predicting human action behavior and the computer equipment of the present invention can reduce the calculation burden by performing frame-by-frame sampling on the image information of the human action cycle to obtain an image sequence with a reduced number of frames; The image of each frame is carried out as follows: set up skeleton model for each frame image, and determine the actual value of a plurality of characteristic angles of described skeleton model; According to a plurality of characteristic angles corresponding to a plurality of predetermined human action behavior categories that pre-training learning obtains Comparing the actual values of multiple characteristic angles of the skeleton model with the reference values of multiple characteristic angles corresponding to the predetermined human action category; determining the human action category to which each frame of image belongs according to the comparison result . The reference values of multiple characteristic angles corresponding to the predetermined human action behavior categories obtained through pre-training and learning are determined to determine more accurate reference values of multiple characteristic angles, and then the characteristic behavior is performed by comparing the results with the reference values of multiple characteristic angles. Prediction to improve the recognition accuracy of human action behavior prediction.

附图说明Description of drawings

为了更清楚地说明本发明实施例的技术方案,下面将对实施例中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其它的附图。In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. Those of ordinary skill in the art can also obtain other drawings based on these drawings without any creative effort.

图1为根据本发明示例性第一实施例的人体动作行为的预测方法的流程图;FIG. 1 is a flowchart of a method for predicting human action behavior according to an exemplary first embodiment of the present invention;

图2为根据本发明示例性第二实施例的人体动作行为的预测方法的流程图;Fig. 2 is a flowchart of a method for predicting human action behavior according to an exemplary second embodiment of the present invention;

图3为图2中利用SVM分类得到各预定动作行为类别的流程图;Fig. 3 is the flow chart that utilizes SVM classification to obtain each predetermined action behavior category in Fig. 2;

图4为图2中预测每帧图像所属的人体动作行为类别的流程图;Fig. 4 is the flow chart of predicting the human action behavior category to which each frame image belongs among Fig. 2;

图5a及图5b分为第15帧以及第25帧的骨架细化示意图;Figure 5a and Figure 5b are divided into schematic diagrams of the skeleton refinement of the 15th frame and the 25th frame;

图6为每帧图像建立的骨架模型示意图;Fig. 6 is the skeleton model schematic diagram that each frame image is established;

图7为根据本发明示例性第三实施例的计算机设备的结构示意图。Fig. 7 is a schematic structural diagram of a computer device according to an exemplary third embodiment of the present invention.

具体实施方式Detailed ways

下面结合附图对本发明实施例进行详细描述。Embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

需说明的是,在不冲突的情况下,以下实施例及实施例中的特征可以相互组合;并且,基于本公开中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本公开保护的范围。It should be noted that, in the case of no conflict, the following embodiments and the features in the embodiments can be combined with each other; and, based on the embodiments in the present disclosure, those of ordinary skill in the art obtained without creative work All other embodiments belong to the protection scope of the present disclosure.

需要说明的是,下文描述在所附权利要求书的范围内的实施例的各种方面。应显而易见,本文中所描述的方面可体现于广泛多种形式中,且本文中所描述的任何特定结构及/或功能仅为说明性的。基于本公开,所属领域的技术人员应了解,本文中所描述的一个方面可与任何其它方面独立地实施,且可以各种方式组合这些方面中的两者或两者以上。举例来说,可使用本文中所阐述的任何数目个方面来实施设备及/或实践方法。另外,可使用除了本文中所阐述的方面中的一或多者之外的其它结构及/或功能性实施此设备及/或实践此方法。It is noted that the following describes various aspects of the embodiments that are within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is illustrative only. Based on the present disclosure one skilled in the art should appreciate that an aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, any number of the aspects set forth herein can be used to implement an apparatus and/or practice a method. In addition, such an apparatus may be implemented and/or such a method practiced using other structure and/or functionality than one or more of the aspects set forth herein.

如图1所示,本发明一种人体动作行为的预测方法,包括:As shown in Fig. 1, a kind of prediction method of human action behavior of the present invention comprises:

步骤101:对人体动作周期图像信息进行分帧采样得到帧数减少后的图像序列;Step 101: Perform frame-by-frame sampling on the image information of the human body movement cycle to obtain an image sequence with a reduced number of frames;

步骤102:根据各帧的排序依次针对所述图像序列的多个帧的图像进行如下操作:Step 102: according to the ordering of each frame, the following operations are performed for the images of a plurality of frames of the image sequence:

步骤102a:针对每帧图像建立骨架模型,并确定所述骨架模型的多个特征角度的实际值;Step 102a: establishing a skeleton model for each frame of image, and determining actual values of multiple characteristic angles of the skeleton model;

步骤102b:根据预先训练学习得到的多个预定人体动作行为类别对应的多个特征角度的基准值,将所述骨架模型的多个特征角度的实际值与预定人体动作行为类别对应的多个特征角度的基准值进行比对;Step 102b: According to the reference values of multiple characteristic angles corresponding to multiple predetermined human action behavior categories obtained in pre-training, compare the actual values of the multiple characteristic angles of the skeleton model with the multiple features corresponding to the predetermined human action behavior categories The benchmark value of the angle is compared;

步骤102c:根据比对结果确定每帧图像所属的人体动作行为类别。Step 102c: According to the comparison result, determine the human action category to which each frame of image belongs.

本实施例通过对人体动作周期图像信息进行分帧采样得到帧数减少后的图像序列,能降低计算负担;根据各帧的排序依次针对所述图像序列的多个帧的图像进行如下操作:针对每帧图像建立骨架模型,并确定所述骨架模型的多个特征角度的实际值;根据预先训练学习得到的多个预定人体动作行为类别对应的多个特征角度的基准值,将所述骨架模型的多个特征角度的实际值与预定人体动作行为类别对应的多个特征角度的基准值进行比对;根据比对结果确定每帧图像所属的人体动作行为类别。通过预先训练学习得到的预定人体动作行为类别对应的多个特征角度的基准值,确定较为精确的多个特征角度的基准值,进而通过与多个特征角度的基准值的比对结果进行特征行为预测,以提高人体动作行为预测的识别精度。This embodiment obtains the image sequence after the number of frames is reduced by performing frame-by-frame sampling on the image information of the human body movement cycle, which can reduce the computational burden; according to the ordering of each frame, the following operations are performed on the images of a plurality of frames of the image sequence: Establish a skeleton model for each frame of image, and determine the actual values of a plurality of characteristic angles of the skeleton model; The actual values of the multiple characteristic angles are compared with the reference values of the multiple characteristic angles corresponding to the predetermined human action category; and the human action category to which each frame of image belongs is determined according to the comparison result. The reference values of multiple characteristic angles corresponding to the predetermined human action behavior categories obtained through pre-training and learning are determined to determine more accurate reference values of multiple characteristic angles, and then the characteristic behavior is performed by comparing the results with the reference values of multiple characteristic angles. Prediction to improve the recognition accuracy of human action behavior prediction.

图2提供了本发明一种人体动作行为的预测方法优选的实施方式,本实施例主要包括两大流程,第一流程是通过预先训练学习得到的预定人体动作行为类别对应的多个特征角度的基准值,具体步骤对应于图2的步骤21-步骤32;第二流程是根据多个特征角度的基准值,对于待预测的图像信息进行具体识别的流程。结合图2-图6所示,本发明实施例另一种人体动作行为的预测方法,包括:Figure 2 provides a preferred implementation of a method for predicting human action behaviors in the present invention. This embodiment mainly includes two major processes. The first process is the multiple characteristic angles corresponding to the predetermined human action behavior categories obtained through pre-training and learning. The reference value, the specific steps correspond to step 21-step 32 in FIG. 2; the second process is a process of specifically identifying the image information to be predicted according to the reference values of multiple characteristic angles. As shown in FIG. 2-FIG. 6, another method for predicting human action behavior in the embodiment of the present invention includes:

首先,结合SVM,建模得到预定人体动作行为类别对应的多个特征角度的基准值。本实施例先通过SVM从大量公开的动作行为数据集中进行人类日常动作行为的识别,即可得到预定人体动作行为类别。由于SVM对各种人类日常动作行为识别的精度较高,根据SVM的识别结果,即可以对每类预定人体动作行为类别进行特征建模,得到预定人体动作行为类别对应的多个特征角度的基准值。Firstly, combined with SVM, the benchmark values of multiple characteristic angles corresponding to predetermined human action categories are obtained by modeling. In this embodiment, SVM is used to identify daily human action behaviors from a large number of public action behavior data sets, so as to obtain predetermined categories of human action actions. Due to the high accuracy of SVM in the recognition of various human daily actions, according to the recognition results of SVM, it is possible to carry out feature modeling for each predetermined human action category, and obtain the benchmark of multiple feature angles corresponding to the predetermined human action category value.

根据SVM识别预定人体动作行为类别的流程具体详见步骤21-步骤24,也可以参见图3所示流程中的步骤301-步骤304;For the process of identifying predetermined human action behavior categories according to SVM, refer to step 21-step 24 for details, and also refer to step 301-step 304 in the process shown in Figure 3;

步骤21:输入人体动作周期图像信息,该人体动作周期图像信息包括但不限于视频;Step 21: Input the image information of human body motion cycle, which includes but not limited to video;

步骤22:预处理,比如滤波及去噪等信号常规处理,在此不在展开;Step 22: Preprocessing, such as filtering and denoising and other signal routine processing, will not be carried out here;

步骤23:对人体动作周期图像信息提取特征信息;具体可以利用HOG(方向梯度直方图,Histogram of Oriented Gradient)方法对动作行为进行特征提取,分别计算图像中像素点的水平方向梯度和垂直方向梯度,求出梯度幅值和梯度方向,利用直方图进行特征提取。如图像中像素点(x,y)的水平方向梯度、垂直方向梯度分别为:Step 23: Extract feature information from the image information of the human motion cycle; specifically, the HOG (Histogram of Oriented Gradient) method can be used to extract the feature of the action behavior, and the horizontal gradient and vertical gradient of the pixels in the image can be calculated respectively , find the gradient magnitude and gradient direction, and use the histogram for feature extraction. For example, the horizontal and vertical gradients of pixels (x, y) in the image are:

Dx(x,y)=H(x+1,y)-H(x-1,y);D x (x,y)=H(x+1,y)-H(x-1,y);

Dy(x,y)=H(x,y+1)-H(x,y-1);D y (x,y)=H(x,y+1)-H(x,y-1);

其中,H(x,y)是输入图像中像素点在(x,y)点的像素值;像素点(x,y)处的梯度幅值和梯度方向分别是:Among them, H(x, y) is the pixel value of the pixel point in the input image at (x, y); the gradient magnitude and gradient direction at the pixel point (x, y) are:

Figure BDA0002301886500000061
Figure BDA0002301886500000061

α(x,y)=tan-1(Gy(x,y)/Gx(x,y));α(x,y)=tan -1 (G y (x,y)/G x (x,y));

步骤24:将所述特征信息输入支持向量机SVM,进行动作行为分类,得到所述预定动作行为类别的分类结果。Step 24: Input the feature information into a support vector machine (SVM) to classify actions, and obtain classification results of the predetermined action categories.

SVM的本质是将最大几何间隔化的问题转化成一个凸函数进行数学求解优化,本实施例使用多个二分类器构造多分类的方法实现人体多种动作行为的分类。SVM在KTH和Weizmann两个动作行为数据集中的识别结果:The essence of SVM is to transform the problem of maximum geometric compartmentalization into a convex function for mathematical solution optimization. This embodiment uses multiple binary classifiers to construct a multi-classification method to realize the classification of various human action behaviors. The recognition results of SVM in the two action behavior datasets of KTH and Weizmann:

Figure BDA0002301886500000062
Figure BDA0002301886500000062

步骤25:对分类正确的动作行为序列进行分帧提取;Step 25: Perform frame-by-frame extraction on correctly classified motion behavior sequences;

考虑到动作训练由于高速采集的原因,往往存在数据冗余现象,本实施例优选采用采样分帧的处理方法重新规整动作序列,减少后续工作的运算量,降低过程的复杂度。可以将首位五帧融合为一帧,每间隔十帧进行采样,将采样到的五帧进行融合,最终将行为动作序列分割为帧数较少的图像序列;例如,将一类动作行为视频分割为750帧的动作序列,按每15帧为一单元进行分割为50个基本单元,分别对每一个基本单元的前5帧进行采样,并将其融合为1帧图像,依次类推,规整后的序列为一个包含50帧图像的新序列。Considering that action training often has data redundancy due to high-speed acquisition, this embodiment preferably adopts the processing method of sampling and dividing into frames to reorganize the action sequence, reduce the amount of calculation of follow-up work, and reduce the complexity of the process. The first five frames can be fused into one frame, sampled every ten frames, and the five sampled frames are fused, and finally the behavior sequence is divided into image sequences with fewer frames; for example, a class of action behavior video segmentation An action sequence of 750 frames is divided into 50 basic units according to each 15-frame unit, and the first 5 frames of each basic unit are sampled and fused into 1 frame of image, and so on, the regularized The sequence is a new sequence containing 50 frames of images.

步骤26:对图像中的人体外观模型(如图5a及图5b所示的模型,其中图5a及图5b中的左图为二维图像,右图为图像骨架)进行骨架细化处理(如图6所示);Step 26: Carry out skeleton thinning processing to the human body appearance model in the image (the model shown in Figure 5a and Figure 5b, wherein the left picture in Figure 5a and Figure 5b is a two-dimensional image, and the right picture is an image skeleton) (such as As shown in Figure 6);

考虑到人体是一个非刚性的结构,对于动作行为的执行,最显著的动作特征主要集中在人体四肢的运动,动作执行过程中,人体四肢运动的角度也随着时间进行变化,同时不同的角度变化之间还具有一定的关联性,比如跑步时膝盖的弯曲角度与手部和肩膀处的角度就有一定的关联性;手部和肩膀处的角度与手肘的弯曲角度也有一定的关联性等。所以本实施例提取人体四肢角度变化作为主要的角度特征。具体地,如图6所示,可以将骨架细化的图像,按照图形结构将人体骨架分为三部分:上肢部分,下肢部分以及中间部分,上肢部分包括:左手、左肘、左肩、右手、右肘、右肩;下肢部分包括:左脚、左膝、右脚、右膝;中间部分为胯部;Considering that the human body is a non-rigid structure, for the execution of action behaviors, the most significant action features mainly focus on the movement of the limbs of the human body. There is also a certain correlation between the changes. For example, when running, the bending angle of the knee has a certain correlation with the angle of the hands and shoulders; the angle of the hands and shoulders has a certain correlation with the bending angle of the elbow wait. Therefore, this embodiment extracts the angle change of the limbs of the human body as the main angle feature. Specifically, as shown in FIG. 6 , the skeletonized image can be divided into three parts according to the graphic structure: the upper limb part, the lower limb part and the middle part. The upper limb part includes: left hand, left elbow, left shoulder, right hand, Right elbow, right shoulder; lower limbs include: left foot, left knee, right foot, right knee; middle part is crotch;

步骤27:计算特征角度,实现建模;Step 27: Calculate the feature angle to realize modeling;

首先,进行独立关节点角度特征角度提取,具体分为左手-左肘-左肩,左肘-左肩-胯部,右手-右肘-右肩,右肘-右肩-胯部;左脚-左膝-胯部,右脚-右膝-胯部;左膝-胯部-右膝七个特征角度;First, extract the angle feature angle of independent joint points, which are divided into left hand-left elbow-left shoulder, left elbow-left shoulder-crotch, right hand-right elbow-right shoulder, right elbow-right shoulder-crotch; left foot-left Knee-crotch, right foot-right knee-crotch; left knee-crotch-right knee seven characteristic angles;

其次,设定特征角度的阈值范围,建立该类动作行为的人体骨架角度特征模型,依次建立多种动作行为的角度特征模型;Secondly, set the threshold range of the characteristic angle, establish the human skeleton angle characteristic model of this type of action behavior, and establish the angle characteristic model of various action behaviors in turn;

经过大量试验,得出选取特征角度的阈值设定以及误差允许的范围,误差大小选取偏差为5℃最佳。如跑的动作行为特征角度阈值设定及允许误差具体如下:After a large number of experiments, it is concluded that the threshold setting of the selected characteristic angle and the allowable range of the error are selected, and the selection deviation of the error size is 5°C. For example, the action behavior characteristic angle threshold setting and allowable error of running are as follows:

Figure BDA0002301886500000071
Figure BDA0002301886500000071

在通过上述步骤21-步骤27建模得到各动作行为的特征角度的基准值之后,对于各个待测试数据通过以下步骤28-210计算各帧图像的特征角度的大小,其中步骤28-步骤30基本对应步骤25-207的处理。步骤28-步骤31的流程可以参见图4的详细流程。After obtaining the reference value of the characteristic angle of each action behavior through the above-mentioned step 21-step 27 modeling, calculate the size of the characteristic angle of each frame image through the following steps 28-210 for each data to be tested, wherein step 28-step 30 basically Corresponding to the processing of steps 25-207. For the flow of steps 28-31, refer to the detailed flow in FIG. 4 .

步骤28:对人体动作周期图像信息进行分帧采样得到帧数减少后的图像序列;Step 28: Perform frame-by-frame sampling on the image information of the human body movement cycle to obtain an image sequence with a reduced number of frames;

步骤29:针对每帧图像建立骨架模型(细化处理);所述骨架模型包括上肢部分,下肢部分以及中间部分;所述上肢部分包括:左手、左肘、左肩、右手、右肘、以及右肩;所述下肢部分包括:左脚、左膝、右脚、以及右膝;所述中间部分包括胯部;Step 29: set up skeleton model (thinning processing) for each frame image; Described skeleton model comprises upper limb part, lower limb part and middle part; Described upper limb part comprises: left hand, left elbow, left shoulder, right hand, right elbow and right hand shoulder; said lower body portion includes: left foot, left knee, right foot, and right knee; said middle portion includes crotch;

步骤30:计算特征角度大小,具体分别确定左手、左肘及左肩形成的第1个特征角度的实际值,左肘、左肩及胯部形成的第2个特征角度的实际值,右手、右肘及右肩形成的第3个特征角度的实际值,右肘、右肩及胯部形成的第4个特征角度的实际值,左脚、左膝及胯部形成的第5个特征角度的实际值,右脚、右膝及胯部形成的第6个特征角度的实际值,左膝、胯部及右膝形成的第7个特征角度的实际值;Step 30: Calculate the size of the characteristic angle, specifically determine the actual value of the first characteristic angle formed by the left hand, left elbow and left shoulder, the actual value of the second characteristic angle formed by the left elbow, left shoulder and crotch, the right hand and right elbow and the actual value of the third characteristic angle formed by the right shoulder, the actual value of the fourth characteristic angle formed by the right elbow, right shoulder and hip, and the actual value of the fifth characteristic angle formed by the left foot, left knee and hip value, the actual value of the sixth characteristic angle formed by the right foot, right knee and crotch, and the actual value of the seventh characteristic angle formed by the left knee, crotch and right knee;

其中,所述第1个至第4个特征角度来源于所述上肢部分,所述第5个至第6个特征角度来源于所述下肢部分,第7个特征角度来源于所述中间部分。Wherein, the first to fourth characteristic angles are derived from the upper limb, the fifth to sixth characteristic angles are derived from the lower limb, and the seventh characteristic angle is derived from the middle part.

步骤31:根据步骤27确定的基准值,确定各特征角度的实际值于基准值之间的误差范围。即:依次判断第n个特征角度的实际值是否位于预定人体动作行为类别对应的第n个特征角度的基准值的预设偏差范围内,n为1-7之间的整数;Step 31: According to the reference value determined in step 27, determine the error range between the actual value of each characteristic angle and the reference value. That is: sequentially determine whether the actual value of the nth characteristic angle is within the preset deviation range of the reference value of the nth characteristic angle corresponding to the predetermined human action category, and n is an integer between 1-7;

对于每帧图像,统计位于特定人体动作行为的第1个至第7个特征角度的基准值的预设偏差范围内的各特征角度的数量;所述数量为所述比对结果;所述特定人体动作行为为预定人体动作行为类别中的任意一种。For each frame of image, count the number of each characteristic angle within the preset deviation range of the reference value of the 1st to the 7th characteristic angle of a specific human action behavior; the number is the comparison result; the specific The human action behavior is any one of predetermined categories of human action actions.

步骤32:根据事先设置的规则,进行动作预判,之后对下一帧图像进行相应人体动作行为的预测。Step 32: Perform action prediction according to the pre-set rules, and then predict the corresponding human action behavior for the next frame of image.

具体可以通过模板匹配的方法进行动作行为预测,具体步骤包括:第一步,粗判断,分别将提取的待测序列的角度特征和模板进行比对,如果7个特征中有5个特征处于阈值变化范围之内,则可以初步预测出该类动作行为;第二步,精细判决,分别在上肢部分,下肢部分以及中心部分三个局部部分进行角度特征匹配,如果有两个局部的角度特征处于阈值变化范围内,则该类动作行为就可以被精准预测。粗判决仅仅进行7个独立角度特征的比对,在粗判决的基础上进行精细判决,即加入局部组合的整体特征角度判决。Specifically, the action behavior prediction can be performed through the template matching method. The specific steps include: the first step, rough judgment, compare the extracted angle features of the sequence to be tested with the template, and if 5 of the 7 features are at the threshold Within the variation range, this type of action behavior can be preliminarily predicted; the second step is fine judgment, and the angle feature matching is performed on the upper limb part, the lower limb part and the central part respectively. If there are two local angle features in the Within the threshold variation range, this type of action behavior can be accurately predicted. The rough judgment only compares 7 independent angle features, and the fine judgment is made on the basis of the rough judgment, that is, the overall feature angle judgment is added to the partial combination.

本实施例以跑的动作行为为例展示预判效果。首先,在动作周期内提取第10帧,第15帧,第25帧,第35帧,第45帧,第50帧的图像,接着对提取图像进行骨架细化处理得出骨架模型。图5a及图5b为关键帧骨架细化图;其中,图5a为第15帧骨架细化图,图5b为第25帧骨架细化图。提取的特征角度分别如下:In this embodiment, the action behavior of running is taken as an example to demonstrate the predictive effect. First, extract the images of the 10th, 15th, 25th, 35th, 45th, and 50th frames in the action cycle, and then perform skeleton thinning processing on the extracted images to obtain the skeleton model. 5a and 5b are skeleton refinement diagrams of key frames; wherein, FIG. 5a is a skeleton refinement diagram of the 15th frame, and FIG. 5b is a skeleton refinement diagram of the 25th frame. The extracted feature angles are as follows:

Figure BDA0002301886500000091
Figure BDA0002301886500000091

精细判决:即局部整体特征角度比对,本实施例共设三个局部整体:将左手-肘-左肩,左肘-左肩-中(即胯部),右手-肘-右肩,右肘-右肩-中作为一个整体,即上肢部分,进行第一个局部整体角度变化比对;将左脚-左膝-中,右脚-右膝-中作为一个整体,即下肢部分,进行第二个局部整体角度变化比对;将左膝-中-右膝作为一个整体,即中心部分,进行胯部角度变化比对。选取在精细判决中,三个局部整体角度变化均处于阈值范围内作为约束条件,符合要求即可实现该类动作行为的预判,同时也可以确定视频序列中的动作关键帧。Fine judgment: that is, the comparison of local overall feature angles. In this embodiment, three local overalls are set up: left hand-elbow-left shoulder, left elbow-left shoulder-middle (i.e. crotch), right hand-elbow-right shoulder, right elbow- Take the right shoulder-middle as a whole, that is, the upper limb part, and perform the first partial overall angle change comparison; take the left foot-left knee-middle, right foot-right knee-middle as a whole, that is, the lower limb part, and perform the second comparison. A local overall angle change comparison; the left knee-middle-right knee is taken as a whole, that is, the central part, and the crotch angle change is compared. In the fine judgment, the three local overall angle changes are all within the threshold range as the constraint condition. If the requirements are met, the prediction of this type of action behavior can be realized, and the key frame of the action in the video sequence can also be determined.

通过公开的动作行为数据集进行人类日常简单动作的筛选,对6类日常行为(跑、走、拍手、弯曲、挥手、向上挥手)进行实验预判,结果如下表。Through the screening of simple human daily actions through the public action behavior data set, the experimental prediction of 6 types of daily actions (running, walking, clapping, bending, waving, and upward waving) is carried out. The results are shown in the following table.

Figure BDA0002301886500000092
Figure BDA0002301886500000092

可以看出跑与行走比其他4类行为预判效果更好,其中跑和行走的动作行为在完成的二分之一周期就可以完全判决出具体的动作行为;而拍手,弯曲,向上挥手以及挥手只可以在行为周期完成的大部分时间后,预判行为的效果才会明显。出现这种现象的原因有两个:第一,跑与行走在运动过程中比其它4类型行为肢体活动范围更大,也就是特征角度变化更明显;第二,跑和行走动作行为预判效果显著区域集中在二分之一周期时刻,是因为在动作执行完成一半的时刻,是行为动作变化最大的时刻,特征角度变化幅度最明显。It can be seen that the prediction effect of running and walking is better than that of the other four types of behaviors. Among them, the specific behaviors of running and walking can be completely judged within one-half cycle of completion; while clapping, bending, waving upwards and The effect of waving on anticipatory behavior was only apparent after most of the behavioral cycle had completed. There are two reasons for this phenomenon: first, running and walking have a larger range of body movement than the other four types of behaviors during exercise, that is, the change of characteristic angle is more obvious; second, the predictive effect of running and walking behaviors The significant area is concentrated at the half-period moment, because the moment when the action execution is half completed is the moment when the behavior changes the most, and the change range of the characteristic angle is the most obvious.

本实施例首先对选取的数据集中人体动作行为进行识别分类,采用HOG特征提取和SVM分类器的方法进行正确的动作行为分类;接着对每一类的动作行为建立骨架模型,具体步骤包括:在该类行为动作周期内进间隔帧采样,重新构建该类行为的数据集,利用骨架细化图像处理方法对图像中的目标进行处理,分别提取7个独立的角度特征,设定特征角度的变化阈值,建立骨架模型,完成模型训练;然后对待测动作行为数据集进行特征角度提取,通过独立角度和局部整体角度两次判决,达到本实施例的预判目的和动作行为关键帧的确定。本实施例提出的人体骨架模型能够以较低的特征维度有效表示人体动作姿态;同时间隔采样融合帧算法能够较好的进行冗余信息的去除,更加有效地描述关键动作,较现有方案中的逐帧提取特征的运算量较少;此外,通过二次角度特征模板匹配,能够进一步提高动作行为的分类准确率和精准的预判动作行为关键帧。In this embodiment, at first, the human action behavior in the selected data set is identified and classified, and the correct action behavior classification is carried out by using the method of HOG feature extraction and SVM classifier; then a skeleton model is established for each type of action behavior, and the specific steps include: This type of behavior is sampled at intervals in the action cycle, and the data set of this type of behavior is reconstructed, and the target in the image is processed by the skeleton thinning image processing method, and 7 independent angle features are extracted respectively, and the change of the feature angle is set. Threshold, establish a skeleton model, and complete model training; then extract feature angles from the action behavior data set to be tested, and make two judgments through independent angles and partial overall angles to achieve the purpose of prediction and the determination of action behavior key frames in this embodiment. The human skeleton model proposed in this embodiment can effectively represent the human body's action posture with a lower feature dimension; at the same time, the interval sampling and fusion frame algorithm can better remove redundant information and describe key actions more effectively, compared with the existing schemes. The frame-by-frame feature extraction requires less computation; in addition, through the secondary angle feature template matching, the classification accuracy of action behavior and the accurate prediction of key frames of action behavior can be further improved.

如图7所示,一种计算机设备实施例包括处理器,所述处理器执行时实现上述图1或图2所示的人体动作行为的预测方法。其中,执行图2所示的方法具体包括:As shown in FIG. 7 , an embodiment of a computer device includes a processor, and when executed, the processor implements the method for predicting human action behavior shown in FIG. 1 or FIG. 2 above. Wherein, executing the method shown in Figure 2 specifically includes:

(1)选取动作行为公开数据集进行动作识别,通过HOG特征提取,SVM分类器进行动作行为的正确分类;(1) Select the public dataset of action behaviors for action recognition, and use the HOG feature extraction and SVM classifier to correctly classify the actions;

(2)首先对分类正确的动作行为序列进行分帧提取,本实施例采用间隔分帧的方法进行分帧处理,将首位五帧融合为一帧,每间隔十帧进行采样,将采样到的五帧进行融合,最终将行为动作序列分割为帧数较少的图像序列;(2) Firstly, frame-by-frame extraction is carried out on the correctly classified actions and behavior sequences. In this embodiment, frame-by-frame processing is carried out by using the interval frame-by-frame method, and the first five frames are fused into one frame, and sampling is performed every ten frames. Five frames are fused, and finally the action sequence is divided into image sequences with fewer frames;

(3)接着对图像中的人体外观模型进行骨架细化处理;(3) Then carry out skeleton thinning processing to the human appearance model in the image;

(4)将骨架细化的图像,按照图形结构将人体骨架分为三部分:上肢部分,下肢部分以及中心部分,上肢部分包括:左手、左肘、左肩、右手、右肘、右肩;下肢部分包括:左脚、左膝、右脚、右膝;中间部分为胯部;(4) The image of the skeleton is refined, and the human skeleton is divided into three parts according to the graphic structure: the upper limb part, the lower limb part and the central part. The upper limb part includes: left hand, left elbow, left shoulder, right hand, right elbow, right shoulder; lower limb Parts include: left foot, left knee, right foot, right knee; the middle part is the crotch;

(5)独立关节点角度特征提取,具体分为左手-肘-左肩,左肘-左肩-中,右手-肘-右肩,右肘-右肩-中;左脚-左膝-中,右脚-右膝-中;左膝-中-右膝七个特征角度;(5) Angle feature extraction of independent joint points, specifically divided into left hand-elbow-left shoulder, left elbow-left shoulder-middle, right hand-elbow-right shoulder, right elbow-right shoulder-middle; left foot-left knee-middle, right Foot-right knee-middle; left knee-middle-right knee seven characteristic angles;

(6)设定特征角度的阈值范围,建立该类动作行为的人体骨架角度特征模型,依次建立多种动作行为的角度特征模型;(6) Set the threshold range of the characteristic angle, establish the human skeleton angle characteristic model of this type of action behavior, and establish the angle characteristic model of various action behaviors in turn;

(7)利用步骤(2)至步骤(6)中的方法对测试序列进行分割和角度特征提取;(7) Utilize the method in step (2) to step (6) to carry out segmentation and angle feature extraction to test sequence;

(8)通过模板匹配的方法进行动作行为预测,具体步骤包括:第一步,粗判断,分别将提取的待测序列的角度特征和模板进行比对,如果7个特征中有5个特征处于阈值变化范围之内,则可以初步预测出该类动作行为;第二步,精细判决,分别在上肢部分,下肢部分以及中心部分三个局部部分进行角度特征匹配,如果有两个局部的角度特征处于阈值变化范围内,则该类动作行为就可以被精准预测;(8) Predict action behavior by template matching method. The specific steps include: the first step, rough judgment, compare the extracted angle features of the sequence to be tested with the template, if 5 of the 7 features are in the If it is within the range of the threshold value change, this type of action behavior can be preliminarily predicted; the second step is fine judgment, and the angle feature matching is performed on the three local parts of the upper body part, the lower body part and the central part respectively. If there are two local angle features If it is within the threshold variation range, this type of action behavior can be accurately predicted;

(9)通过动作行为的预测步骤,确定动作行为的关键帧。(9) Through the prediction step of the action behavior, determine the key frame of the action behavior.

本实施例可以对人类日常简单动作行为进行动作关键帧检索和行为预判分类,具有较高的分类准确度。以从视频序列中提取到的人体骨架信息为基础,通过多角度特征匹配模型,进行动作关键帧检索并对后续时刻的动作行为进行预判。In this embodiment, action key frame retrieval and behavior prediction classification can be performed on simple human daily action behaviors, which has high classification accuracy. Based on the human skeleton information extracted from the video sequence, through the multi-angle feature matching model, the key frame of the action is retrieved and the action behavior at the subsequent moment is predicted.

本发明还提供一种存储有计算机程序的计算机可读存储介质实施例,当所述计算机程序在被处理器执行时实现上述的人体动作行为的预测方法。The present invention also provides an embodiment of a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned method for predicting human action behavior is realized.

本发明存储有计算机程序的计算机可读存储介质实施例具有上述人体动作行为的预测方法实施例相应的技术效果,在此不再赘述。The embodiment of the computer-readable storage medium storing the computer program of the present invention has the corresponding technical effects of the above-mentioned embodiment of the method for predicting human action behavior, which will not be repeated here.

以上所述,仅为本发明的具体实施方式,但本发明的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到的变化或替换,都应涵盖在本发明的保护范围之内。因此,本发明的保护范围应以权利要求的保护范围为准。The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. A method for predicting human action behavior, comprising:
frame-dividing and sampling the human body action period image information to obtain an image sequence with reduced frame number;
sequentially aiming at the images of a plurality of frames of the image sequence according to the ordering of the frames, carrying out the following operations:
establishing a skeleton model for each frame of image, and determining actual values of a plurality of characteristic angles of the skeleton model;
according to the reference values of the plurality of characteristic angles corresponding to the plurality of preset human action behavior categories obtained through pre-training and learning, the actual values of the plurality of characteristic angles of the skeleton model are compared with the reference values of the plurality of characteristic angles corresponding to the preset human action behavior categories, and the method comprises the following steps:
sequentially judging whether the actual value of the nth characteristic angle is within a preset deviation range of a reference value of the nth characteristic angle corresponding to a preset human body action behavior category, wherein n is an integer between 1 and 7;
counting the number of the characteristic angles within a preset deviation range of the reference values of the 1 st to 7 th characteristic angles of the specific human action behaviors for each frame of image; the number is the comparison result; the specific human action behavior is any one of preset human action behavior categories;
determining the human action behavior category of each frame of image according to the comparison result, wherein the human action behavior category comprises:
if the number is greater than or equal to 5, preliminarily determining the human body action behavior category to which the image of the corresponding frame belongs as the specific human body action behavior;
for each frame of image, if the number of the characteristic angles which are not located in the preset deviation range of the reference value of each characteristic angle of the specific human motion behavior is two, and the two characteristic angles which are not located in the preset deviation range of the reference value of each characteristic angle of the specific human motion behavior are respectively sourced from any one of the upper limb part, the lower limb part and the middle part, the human motion behavior category to which the image of the corresponding frame belongs is finally determined as the specific human motion behavior.
2. The method of predicting human action behavior according to claim 1, wherein building a skeleton model for each frame of images in the plurality of frames and determining actual values of a plurality of feature angles of the skeleton model comprises:
establishing a skeleton model for each frame of image, wherein the skeleton model comprises an upper limb part, a lower limb part and a middle part; the upper limb portion includes: left hand, left elbow, left shoulder, right hand, right elbow, and right shoulder; the lower limb portion includes: left foot, left knee, right foot, right knee; the intermediate portion comprising a crotch portion;
respectively determining actual values of 1 st characteristic angles formed by a left hand, a left elbow and a left shoulder, actual values of 2 nd characteristic angles formed by a left elbow, a left shoulder and a crotch, actual values of 3 rd characteristic angles formed by a right hand, a right elbow and a right shoulder, actual values of 4 th characteristic angles formed by a right elbow, a right shoulder and a crotch, actual values of 5 th characteristic angles formed by a left foot, a left knee and a crotch, actual values of 6 th characteristic angles formed by a right foot, a right knee and a crotch, and actual values of 7 th characteristic angles formed by a left knee, a crotch and a right knee;
wherein the 1 st to 4 th characteristic angles originate from the upper limb portion, the 5 th to 6 th characteristic angles originate from the lower limb portion, and the 7 th characteristic angle originates from the intermediate portion.
3. The method of claim 1 or 2, wherein the skeletal model is a tree model of left hand, left elbow, left shoulder, right hand, right elbow, right shoulder, left foot, left knee, right foot, right knee, and crotch connection.
4. The prediction method of human motion behavior according to claim 1, wherein the preset deviation range of the reference value of the nth characteristic angle is a range of ±5 degrees of the reference value of the nth characteristic angle.
5. The method for predicting human action behavior according to claim 1, wherein the reference values of the plurality of feature angles corresponding to the predetermined human action behavior class obtained by training and learning in advance are obtained by:
inputting an action training learning data set;
extracting characteristic information of the training learning data set;
inputting the characteristic information into a support vector machine for classification to obtain action behavior categories of each training learning data; the action behavior type of each training learning data is a preset human action behavior type;
and determining reference values of a plurality of characteristic angles of the corresponding preset action behavior categories according to the action behavior training learning data corresponding to each preset action behavior category.
6. The method for predicting human motion behavior according to claim 1, wherein the step of sampling the human motion cycle image information in frames to obtain the image sequence with reduced frame number comprises:
and merging the first five frames into one frame according to the human motion period image information, sampling every ten frames, and merging the sampled five frames to obtain an image sequence with reduced frame number.
7. A computer device comprising a processor which when executed implements a method of predicting human action behaviour according to any one of claims 1 to 6.
8. A computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the method of predicting human action behavior according to any one of claims 1 to 6.
CN201911224818.0A 2019-12-04 2019-12-04 Human body action behavior prediction method and computer equipment Active CN112906438B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911224818.0A CN112906438B (en) 2019-12-04 2019-12-04 Human body action behavior prediction method and computer equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911224818.0A CN112906438B (en) 2019-12-04 2019-12-04 Human body action behavior prediction method and computer equipment

Publications (2)

Publication Number Publication Date
CN112906438A CN112906438A (en) 2021-06-04
CN112906438B true CN112906438B (en) 2023-05-02

Family

ID=76104735

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911224818.0A Active CN112906438B (en) 2019-12-04 2019-12-04 Human body action behavior prediction method and computer equipment

Country Status (1)

Country Link
CN (1) CN112906438B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113887424A (en) * 2021-09-30 2022-01-04 深圳奇迹智慧网络有限公司 Human behavior recognition method and device, computer equipment and storage medium
CN119206860B (en) * 2024-09-05 2025-04-11 广州越明科技有限公司 A skipping rope counting method and system based on image recognition

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102129551A (en) * 2010-02-16 2011-07-20 微软公司 Gesture detection based on joint skipping
CN105320944A (en) * 2015-10-24 2016-02-10 西安电子科技大学 Human body behavior prediction method based on human body skeleton movement information
CN107122752A (en) * 2017-05-05 2017-09-01 北京工业大学 A kind of human action comparison method and device
CN108416251A (en) * 2018-01-08 2018-08-17 中国矿业大学 Efficient human motion recognition method based on quantum genetic algorithm optimization
CN108764120A (en) * 2018-05-24 2018-11-06 杭州师范大学 A kind of human body specification action evaluation method
CN109101864A (en) * 2018-04-18 2018-12-28 长春理工大学 The upper half of human body action identification method returned based on key frame and random forest
CN109409384A (en) * 2018-09-30 2019-03-01 内蒙古科技大学 Image-recognizing method, device, medium and equipment based on fine granularity image
CN110414453A (en) * 2019-07-31 2019-11-05 电子科技大学成都学院 Human body action state monitoring method under a kind of multiple perspective based on machine vision

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4677046B2 (en) * 2006-12-06 2011-04-27 本田技研工業株式会社 Fast human pose estimation using appearance and motion via multidimensional boost regression
JP6433149B2 (en) * 2013-07-30 2018-12-05 キヤノン株式会社 Posture estimation apparatus, posture estimation method and program
EP3753489B1 (en) * 2018-05-28 2022-01-05 Kaia Health Software GmbH Monitoring the performance of physical exercises

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102129551A (en) * 2010-02-16 2011-07-20 微软公司 Gesture detection based on joint skipping
CN105320944A (en) * 2015-10-24 2016-02-10 西安电子科技大学 Human body behavior prediction method based on human body skeleton movement information
CN107122752A (en) * 2017-05-05 2017-09-01 北京工业大学 A kind of human action comparison method and device
CN108416251A (en) * 2018-01-08 2018-08-17 中国矿业大学 Efficient human motion recognition method based on quantum genetic algorithm optimization
CN109101864A (en) * 2018-04-18 2018-12-28 长春理工大学 The upper half of human body action identification method returned based on key frame and random forest
CN108764120A (en) * 2018-05-24 2018-11-06 杭州师范大学 A kind of human body specification action evaluation method
CN109409384A (en) * 2018-09-30 2019-03-01 内蒙古科技大学 Image-recognizing method, device, medium and equipment based on fine granularity image
CN110414453A (en) * 2019-07-31 2019-11-05 电子科技大学成都学院 Human body action state monitoring method under a kind of multiple perspective based on machine vision

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
2d human pose estimation: New benchmark and state of the art analysis;Mykhaylo Andriluka;《In Computer Vision and Pattern》;20141230;3686-3693 *
PoseTrack: Joint Multi-person Pose Estimation and;Iqbal U;《ComputerVisionandPatternRecognition》;20171230;4654-4663 *
一种基于RGB-D特征融合的人体行为识别框架;毛峡等;《计算机科学》;20180830(第8期);22-28 *
人体行为分析方法的研究与仿真;甘玲等;《计算机仿真》;20130915(第09期);435-439 *

Also Published As

Publication number Publication date
CN112906438A (en) 2021-06-04

Similar Documents

Publication Publication Date Title
Ji et al. Interactive body part contrast mining for human interaction recognition
WO2020108362A1 (en) Body posture detection method, apparatus and device, and storage medium
Tian et al. Densely connected attentional pyramid residual network for human pose estimation
CN113128424B (en) Method for identifying action of graph convolution neural network based on attention mechanism
Tang et al. Facial landmark detection by semi-supervised deep learning
CN110135375A (en) Multi-Person Pose Estimation Method Based on Global Information Integration
CN110781843A (en) Classroom behavior detection method and electronic equipment
Geng et al. Human action recognition based on convolutional neural networks with a convolutional auto-encoder
CN108416266A (en) A kind of video behavior method for quickly identifying extracting moving target using light stream
CN107067413A (en) A kind of moving target detecting method of time-space domain statistical match local feature
CN107633226A (en) A kind of human action Tracking Recognition method and system
CN106570480A (en) Posture-recognition-based method for human movement classification
CN110991274A (en) Pedestrian tumbling detection method based on Gaussian mixture model and neural network
Weiyao et al. Human action recognition using multilevel depth motion maps
CN105373810B (en) A method and system for establishing an action recognition model
Liao et al. A two-stage method for hand-raising gesture recognition in classroom
CN112906520A (en) Gesture coding-based action recognition method and device
Zhou et al. A study on attention-based LSTM for abnormal behavior recognition with variable pooling
Jangade et al. Study on deep learning models for human pose estimation and its real time application
CN112906438B (en) Human body action behavior prediction method and computer equipment
CN117437690A (en) Gesture recognition method, system and medium combining environment adaptation and estimation classification
CN114973305A (en) An Accurate Human Analysis Method for Crowded Crowds
Liu et al. Key algorithm for human motion recognition in virtual reality video sequences based on hidden markov model
Aiouez et al. Real-time Arabic Sign Language Recognition based on YOLOv5.
Liu et al. Fabric defect detection using fully convolutional network with attention mechanism

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant