CN116721468A - Intelligent guided broadcast switching method based on multi-person gesture estimation action amplitude detection - Google Patents

Abstract

The invention provides an intelligent director switching method based on multi-person posture estimation action amplitude detection. First, the video signal is framed into an image, and the image is input into a built and trained multi-person posture estimation model to detect key point coordinates and correlations. Information, the motion range detection module actually considers the problem of different depth of field and character scale. Based on the coordinates and related information of key points, it is normalized through the motion range detection algorithm to calculate the posture feature value, and based on this, the motion range is judged. , the threshold method is used to determine whether it is a large-scale action. If there is a gesture feature whose value exceeds the system-set threshold, the action is determined to be a large-scale action, otherwise it is a small-scale action. Finally, the intelligent guidance system is used to perform shots based on the detection results. switch. The present invention focuses on improving the accuracy of human body key point positioning and movement range detection to achieve automatic or auxiliary production of programs.

Description

An intelligent director switching method based on multi-person posture estimation and motion range detection

Technical field

The invention relates to an intelligent broadcast switching method based on multi-person posture estimation and motion range detection, and belongs to the field of artificial intelligence technology.

Background technique

Performing arts programs usually use multiple cameras to shoot from different angles of the stage. The director selects and switches the shots from different camera positions for broadcast. The director's choice of lens images is crucial to the overall effect and expressiveness of the program. effect. However, the traditional directing production method requires an experienced director and directing team, consumes a lot of manpower and material resources, requires a long production cycle in terms of multi-camera shooting, director switching and post-editing of the program, and the production process is relatively cumbersome. Especially in live performance scenes, the director needs to quickly judge multiple sets of camera images to capture the performers' wonderful moments. The director is particularly important in the selection and switching of camera images. Intelligent broadcasting based on artificial intelligence technology aims to solve the problem of intelligent broadcasting of various forms of performing arts programs, provide automated and intelligent decision-making methods, and conduct switching of broadcasts through real-time status recognition of characters in the scene in the scene. The application of this technology will greatly reduce the production cycle and human and material costs, and improve the production efficiency and quality of programs. Different from action recognition and action detection, action recognition aims to detect the type of action. The action range detection of this method aims to detect the extension change range of human action in a specific scene, which belongs to the application direction of the new artificial intelligence technology field. The size of the movement can affect the actor's performance style and can also help the actor show more emotion and dynamics on stage. Through motion amplitude detection, the video is analyzed frame by frame to realize real-time status identification of the host and guests, and the results are fed back to the intelligent broadcasting system to achieve the effect of automatic production or auxiliary production of the program, which has important research significance and application value.

With the development of deep learning, there are currently two main ideas for motion amplitude detection methods based on deep learning: First, the motion amplitude detection method based on image classification uses image classification technology to identify and detect the actor's movements, which is simply implemented. Range of motion detection. However, there is a lack of posture feature analysis of actions, and there are still challenges in its real-time and multi-person action analysis capabilities. Second, the motion range detection method is based on multi-person pose estimation. Multi-person pose estimation mainly detects the joint points of the human body, such as shoulders, knees and ankles, etc., while the amplitude changes of performing arts movements are mainly based on the human body posture below the head. The joint points are reflected and the relative positions and movements between these joint points are analyzed to realize the range detection of performing arts movements. Therefore, this method can better reduce the interference of the background environment, focus on the character itself, and has the advantages of fast operation and accurate detection.

Chinese patent CN201910530100.8 discloses a method for measuring the amplitude of human action, which includes the steps of parsing the first video and the second video into a frame sequence, extracting the last frame of the first video and the first frame of the second video, using The joint point detection algorithm finds the corresponding joint points in the last frame and the first frame, calculates the mean displacement of each pair of corresponding joint points, and normalizes the mean displacement Dis to obtain the motion amplitude value of the two frames; beneficial effects Yes: Through the method of this invention, we can finally calculate the motion amplitude of the next frame relative to the previous frame. This motion amplitude is also the measurement standard for our frame insertion position. We will crop the two frames with a larger motion amplitude, and crop the two frames with a larger motion amplitude. The two smaller frames are interpolated, and the smoothness of the spliced video after final processing is better.

As stated in the invention patent, current motion amplitude detection mainly relies on the relative displacement of joint points in adjacent video frames. However, in performing arts scenes, due to the wide variety of movements, existing movement range detection algorithms are prone to misjudgments, and the reliability of the algorithms is low. In addition, the current multi-person pose estimation algorithm used in artificial intelligence applications is a top-down and bottom-up two-stage multi-person pose estimation model. The top-down multi-person pose estimation model has large memory requirements. , poor real-time performance and high computational cost. The bottom-up multi-person pose estimation model has shortcomings such as a large impact of complex backgrounds and prone to misjudgments and matching errors. Therefore, the existing multi-person pose estimation technology needs to be improved and improved. . At present, there is still a lack of motion range detection algorithm based on multi-person pose estimation that reaches a high level of speed and accuracy to meet the application of intelligent guidance systems in performing arts scenes.

To sum up, the technical solution for motion range detection has the following shortcomings:

1. Occupies a lot of computing resources:

The model requires manual operations, such as cropping, non-maximum suppression, and grouping operations. The target detection model is used to assist in motion amplitude detection, which consumes a lot of computing resources.

2. The multi-person pose estimation model cannot achieve a balance between speed and accuracy:

Most of the existing action analysis methods based on multi-person pose estimation (including action recognition, action classification, motion range detection, etc.) use top-down and bottom-up two-stage multi-person pose estimation models, which cannot be achieved. Complete end-to-end multi-person pose estimation, achieving a balance between speed and accuracy.

3. The accuracy of motion range detection is low:

Motion range detection is mainly implemented through image classification or the relative displacement of joint points in adjacent video frames. The lack of human posture feature analysis leads to problems such as misjudgment, poor reliability and low accuracy in motion range detection.

Contents of the invention

In order to solve the above technical problems, the present invention provides an intelligent broadcast switching method based on multi-person posture estimation motion range detection, focusing on solving the problem that the multi-person posture estimation model cannot achieve a balance between speed and accuracy, resulting in the system occupying a large amount of computing resources and It solves the problems of low motion range detection accuracy and achieves excellent results. This invention applies the End-to-End Multi-Person Pose Estimation with Transformers (PETR) model based on Transformers to the motion range detection direction, and actually considers the problem of different depth of field and character scale in the picture, based on joints The point coordinates are normalized and calculated through the motion range detection algorithm to obtain the posture characteristic value, and the motion range is judged accordingly to achieve intelligent director switching, achieve the effect of automatic production or auxiliary production of programs, and save manpower and material resources. ,Improve work efficiency.

The technical solution adopted by the present invention is an intelligent broadcast switching method based on multi-person posture estimation and motion range detection, which includes the following steps:

Step 1: Extract frames to obtain images based on the performance video collected by the camera;

Step 2, build and train a multi-person pose estimation model;

Step 3: Input the image from Step 1 into the multi-person pose estimation model in Step 2, and calculate each pose feature through the motion range detection algorithm;

Step 4: Judge the movement range based on the posture features obtained in step 3;

Step 5: Switch lenses based on the judgment result of the movement range.

The step 1 specifically includes:

Multi-person posture and movement range videos are obtained based on multiple cameras, and multi-person posture and movement range images are obtained by extracting frames based on the multi-person posture and movement range videos.

Step 1.1, select the video frames of the multi-person gesture action range video at intervals of n frames, and other video frames are directly discarded due to information redundancy; where n is a positive integer greater than 1, for example, n is 10.

Step 1.2: Name each video frame according to the camera position information to obtain multiple images, each image representing the current program picture of a camera position. For example, 1_01.jpg represents the first frame of image from camera No. 1.

The step 2 specifically includes:

The multi-person pose estimation model, the PETR model, mainly includes a backbone network module, a position encoding module, a visual feature encoder module, a pose decoder module, and a joint point decoder module. Based on the image obtained in step 1, the PETR model can output the coordinates of the human joint points in the image, and draw the human posture in the image according to the joint point association information.

Step 2.1, the backbone network module is used to input the image in step 1 and output it as a multi-scale feature map. The backbone network module is ResNet-50. ResNet-50 is a 50-layer residual network used to extract feature maps of images. The backbone network module is used to extract high-resolution multi-scale feature maps.

Step 2.2, the visual feature encoder generates the position code for each pixel based on the multi-scale feature map obtained in step 2.1 and the position coding module, and generates multi-scale feature tokens and pose queries;

Step 2.3, the pose decoder queries Q _pose ∈R ^N×D based on the multi-scale feature token F obtained in step 2.2 and N randomly initialized poses, and obtains the poses of N bodies Among them/> represents the coordinates of the K joint points of the i-th person, and D represents the dimension of the query key. The pose decoder estimates pose coordinates layer by layer with the help of all decoder layers, with each layer refining the pose based on the predictions of the previous layer.

Step 2.4, the joint point decoder uses the K joint point information of each group of human postures predicted by the posture decoder in step 2.3 as a randomly initialized joint point query, and refines the joint point position information and joint point structure information. The feature information of joint point queries and keys is continuously updated through self-attention and deformable cross-attention. Finally, the joint point query aggregates features through deviation to further refine the joint point position information and joint point structure information.

In step 2.5, the set-based Hungarian loss is used during training to force a unique prediction for each true pose and reduce false detections and false detections of joint points. The classification loss function denoted L _cls is used in the classification header. In order to eliminate the relative error in the prediction results, the PETR model uses OKS loss. OKS loss is the regression loss calculation between the joint point coordinates of the human posture predicted by the model and the real joint point coordinates. The L ₁ loss is recorded as L _reg and the OKS loss is recorded as L _oks , which are used in the loss functions of the attitude regression head and the joint regression head respectively, so that the attitude decoder and joint point decoder can converge better;

Using methods based on heat map regression to assist training can add a directional guidance to the training of the model. The closer it is to the target joint point, the greater the activation value. The model can quickly approach the target joint point according to the guidance direction and achieve rapid convergence. effect. Therefore, the PETR model uses a deformable transformer encoder to generate heat map predictions, and then calculates a variant Focal Loss between the predicted and true heat maps, denoted as L _hm . The heat map branch is only used to assist training and will be removed during the inference phase. To sum up, all the loss functions of the PETR model are expressed as:

L＝L _cls +λ ₁ L _reg +λ ₂ L _oks +λ ₃ L _hm

Among them, λ ₁ , λ ₂ , λ ₃ represent the corresponding loss weights.

Step 2.6, build the COCO joint point data set as a training set, and train and test the PETR model; specifically, the COCO joint point data contains a total of 200,000 images, including a total of 64,355 joint point images, including 250,000 people's joint points. Each person's mark has 17 joint points, such as head, shoulders, arms, hands, knees, ankles and other joint points. The PETR model was trained for a total of 100 iteration cycles. The learning rate is 2.5e-5. In order to make the PETR model training converge better, when the training reaches 80 epochs, the learning rate is reduced to 2.5e-6. By training the PETR model, the human body joint point coordinates are detected for each input image and the human body posture is generated.

The step 3 specifically includes:

The input of this step is the coordinates of the human body joint points in step 2, and the output is the posture feature value of the human body in the image. Actually taking into account the different depth of field of the picture and the scale of the characters, the motion range detection algorithm consists of four detection conditions.

Step 3.1, the posture estimation joint point connection format adopts the COCO human body joint point connection format, as shown in Figure 5, ( _xi , y _i ) is expressed as the x coordinate and y coordinate of the i-th joint point of the human body, and the left shoulder and right shoulder are calculated The coordinates of the midpoint s of the joint point (joint point numbers: 5, 6), the coordinates of the midpoint h of the left and right hip joint points (joint point numbers: 11, 12), the left knee and right knee joint points (joint point numbers :13,14) The coordinates of the midpoint k.

Step 3.2, detection condition 1: Calculate the ratio of the opening and closing length of the hands to the length of the trunk in the center of the body. The opening and closing length of the hands is composed of the joint points of the left and right wrists (joint point numbers: 9, 10), and the length of the body center trunk is composed of s and h. Based on the joint point coordinates, the opening and closing length of the hands d_hand and the length of the body center trunk d_body are calculated. When the ratio of the opening and closing length of the hands to the length of the body center trunk is greater than or equal to the first threshold, it is determined to be a large-scale movement. Otherwise, it is a small-scale movement. According to The first threshold of the experimental results is set to 1.8, and the discrimination formula is as follows:

Step 3.3, detection condition 2: Calculate the tilt angle of the body trunk. The vector x ₁ composed of s and h, the vector x ₂ composed of h and k, the angle of the two vectors is the inclination angle of the body trunk. When the body trunk tilt angle angle_1 is less than or equal to the second threshold, it is determined to be a large-scale movement, otherwise it is a small-scale movement. According to the experimental results, the second threshold is set to 150°. The discrimination formula is as follows:

Step 3.4, detection condition three: Calculate the opening and closing angle of the arm and the torso. The opening and closing angle of the left arm is composed of the vector x ₃ of the left elbow and left shoulder (joint point number: 7, 5) and the vector x ₄ of the left shoulder and left hip (joint point number: 5, 11). The opening and closing angle of the right arm is composed of the right It consists of the vector x ₅ of the elbow and right shoulder (joint point numbers: 8, 6) and the vector x ₆ of the right shoulder and right hip (joint point numbers: 6, 12). When the left arm opening and closing angle angle_2 and the right arm opening and closing angle angle_3 are greater than or equal to the third threshold, it is determined to be a large-scale movement, otherwise it is a small-scale movement. According to the experimental results, the third threshold is set to 90°. The discrimination formula is as follows:

Step 3.5, detection condition four: Calculate the opening and closing angle of the legs and torso. The opening and closing angle of the left leg is composed of the vector x ₇ of the left knee and left hip (joint point numbers: 13, 11) and the vector x ₈ of the left hip and right hip (joint point number: 11, 12). The opening and closing angle of the right leg It consists of the vector x ₉ of the right knee and right hip (joint point numbers: 14, 12) and the vector x ₁₀ of the right hip and left hip (joint point numbers: 11, 12). When the opening and closing angle of the left leg angle_4 and the opening and closing angle of the right leg angle_5 are greater than or equal to the fourth threshold, it is determined to be a large-scale movement, otherwise it is a small-scale movement. According to the experimental results, the fourth threshold is set to 125°. The discrimination formula is as follows:

The step 4 specifically includes:

The input of this step is the posture feature value of the human body in the image output in step 3, and the motion amplitude detection result is output. By calculating the amplitude of changes in posture characteristics, the threshold method is used to determine whether it is a large-scale action.

Step 4.1, the output result of step 3 is compared with the set threshold. When one of the posture characteristic values in the four detection conditions exceeds the set threshold, the performing action is determined to be a large-amplitude action, otherwise it is a small-amplitude action.

The step 5 specifically includes:

The input of this step is the motion amplitude detection result output in step 4. The intelligent broadcasting system controls the camera to switch the lens to the camera screen corresponding to the large motion image.

Step 5.1, as described in step 1.2, each image is named according to the camera position information, so each image represents the current program picture of a camera, for example, 1_01.jpg represents the first frame of camera No. 1. Therefore, the intelligent broadcasting system can switch the lens to the camera screen corresponding to the image with large motion based on the image name and motion amplitude detection results.

The present invention applies a complete end-to-end multi-person posture estimation model to motion range detection to obtain improvements in function and performance, focusing on improving the accuracy of human joint point positioning and motion range detection. By accurately and quickly calculating posture characteristics based on joint point coordinates, and judging the range of movements accordingly, the intelligent broadcasting system achieves the effect of automatic production or auxiliary production of programs, saving a lot of manpower and material resources, and improving work efficiency.

Compared with the prior art, the beneficial effects of the present invention are as follows:

(1) High operation: Compared with the motion amplitude detection method based on image classification and the motion amplitude detection method based on posture estimation, this invention can better reduce the interference of the background environment and focus on the character itself. This model omits manual operation. There is no need to rely on the target detection model, which reduces a large amount of computing resources, has the advantages of high-speed operation and low memory consumption.

(2) Completely end-to-end, speed and accuracy trade off each other: This invention applies the PETR model to the motion range detection direction. Compared with the two-stage model such as top-down and bottom-up multi-person pose estimation, this model integrates the human body Pose estimation is summarized as a hierarchical set prediction problem. Human body instances and fine-grained human joint point coordinates are processed uniformly, achieving complete end-to-end multi-person pose estimation. The speed and accuracy have reached a high level, and the two trade off each other.

(3) High accuracy: Traditional motion amplitude detection and judgment mainly rely on the relative displacement of joint points in adjacent video frames, and there are fewer detected motion categories. The present invention actually considers the problem of different depth of field of the picture and different scales of the characters, proposes a movement amplitude detection algorithm, and normalizes the movement range detection through gesture characteristic analysis. In the performing arts scene, the detection effect of the performing arts movement amplitude is better. High accuracy.

Description of the drawings

Figure 1 is a flow chart of a specific embodiment of the present invention;

Figure 2 is an overall structural diagram of the PETR model of the present invention;

Figure 3 is a structural diagram of the gesture decoder of the present invention;

Figure 4 is a structural diagram of the joint point decoder of the present invention;

Figure 5 is a connection diagram of joint points of the COCO human body according to the present invention.

Detailed ways

The method will be described in detail below with reference to the drawings and examples.

The flow chart of the implementation is shown in Figure 1, which includes the following steps:

Step S10, video frame extraction;

Step S20, build and train a multi-person pose estimation model;

Step S30: Calculate posture characteristics through the motion range detection algorithm;

Step S40, motion range determination;

Step S50: According to the judgment result, the intelligent guidance system switches lenses.

Intelligent broadcasting based on artificial intelligence technology aims to solve problems in the production process of various forms of performing arts programs, including real-time status recognition and switching of directors in different scenarios. The core of this technology is to use multi-person pose estimation to identify dynamic changes in human joint points, thereby realizing automatic switching of multiple programs. The application of this technology will greatly reduce the production cycle and human and material costs, and improve the production efficiency and quality of programs. Action recognition and action detection aim to detect the type of action. Different from the two, action amplitude detection aims to detect the stretching change range of human actions, which belongs to the application direction of new artificial intelligence technology field. The size of the movement can affect the actor's performance style and can also help the actor show more emotion and dynamics on stage. Through motion range detection, the video is analyzed frame by frame to realize real-time status identification of the host and guests, and the results are fed back to the intelligent broadcasting system to achieve the effect of automatic production or auxiliary production of the program.

The video frame extraction step S10 of the embodiment specifically includes:

The input of this step is video from multiple cameras, and the output is the image obtained by extracting frames from the video. The image is obtained by extracting the video frame as the input of the PETR model in step S20.

Step S100, the image naming method is as follows. Each image is named according to the camera position information, so each image represents the current program screen of a camera; for example, 1_01.jpg represents the first frame image of camera No. 1. .

Step S110, the video frame extraction method of step S100: video frames are selected at intervals of n frames, and other video frames are directly discarded due to information redundancy; n is a positive integer greater than 1, for example, n is 10.

Step S20 of building and training a multi-person pose estimation model specifically includes:

The input of this step is the image obtained in step S10, the output is the coordinates of human body joint points, and the human body posture is drawn in the image according to the joint point association information. The PETR model mainly includes a backbone network module, a position encoding module, a visual feature encoder module, a posture decoder module, and a joint point decoder module; the overall structure of the PETR model is shown in Figure 2.

Step S200, the input of this step is the image obtained in step S10, and the output is a multi-scale feature map. ResNet-50 serves as the backbone network of the PETR model. ResNet-50 is a 50-layer residual network that can be used to extract feature maps of images. The model of the present invention is used to extract multi-scale feature maps in the last three stages. input image Where H is the height of the image, W is the width, and the multi-scale feature maps of the last three stages are extracted through the ResNet-50 network; ResNet-50 is a 50-layer residual network that can be used to extract the feature map of the image.

Step S210: Visual feature encoder of PETR model. The input of this step is the multi-scale feature map obtained in step S200 and the position code generated by the position coding module for each pixel point, and the output is multi-scale feature information and posture query key. Since the multi-head attention module has the square complexity of the input scale, the visual feature encoder adopts a deformable multi-head attention module to achieve feature encoding. will contain multi-scale feature information F∈R ^L×256 , where L is the total number of tokens. Finally, F and the pose query key are input into the pose decoder for pose prediction.

Step S220, pose decoder of PETR model. The input of this step is the multi-scale feature F obtained in step S210 and N randomly initialized pose queries Q _pose ∈R ^N×D . The output of the pose decoder is the pose of N bodies. Among them/> represents the coordinates of the K joint points of the i-th person, and D represents the dimension of the query key.

The structure diagram of the pose decoder is shown in Figure 3. First, the pose query is input into the self-attention (SelfAttention) for interaction between pose queries, that is, Pose-To-Pose SelfAttention. Then, each pose query extracts K feature image pixels as keys layer by layer through deformable cross-attention from the multi-scale feature memory F. Aggregate features based on biases as values based on pose queries. The cross attention module outputs K derived joint point coordinates as the initial coordinates of human posture joint points. Subsequently, the pose query with the key's attention feature information is input into the multi-task prediction head, where the classification head predicts the confidence of each target through a linear mapping layer; the pose regression head uses an MLP with a hidden layer number of 256 to Predict the relative position offset of K derivation points.

The gesture decoder can be composed of multiple decoding layers. This is different from other Transformer methods that only use the last decoder layer to predict pose coordinates. PETR's pose decoder uses all decoder layers to estimate pose coordinates layer by layer, and each layer is refined based on the prediction of the previous layer. attitude.

Step S230: Joint point decoder of the PETR model. The input of this step is the K joint point information of each group of human postures predicted by the posture decoder in step S220 as a randomly initialized joint point query, and the output is further refined joint point position information and joint point structure information. The feature information of joint point queries and keys is continuously updated through self-attention and deformable cross-attention. Finally, the joint point query aggregates features through deviation to further refine the joint point position information and joint point structure information. Since each group of human posture joint points are independent of each other, all postures can be processed in parallel, which greatly reduces the time complexity of network prediction and derivation. The structure diagram of the joint point decoder is shown in Figure 4.

Joint point query first uses the self-attention module to interact with the joint point query feature information, that is, Joint-To-Joint attention. Then the visual features are extracted through the deformable cross-attention module, namely Feature-To-Joint. Subsequently, the joint point prediction head uses MLP to predict the relative displacement ΔJ = (Δx, Δy) between 2D joint points. Similar to the pose decoder, the joint point coordinates are also progressively refined.

The above steps S200 to S230 are the building part of the overall framework of the PETR model. The specific steps of PETR model training are described below.

In step S240, the set-based Hungarian loss is used during training to force a unique prediction for each real pose and reduce false detections and false detections of joint points. The classification loss function denoted L _cls is used in the classification header. In order to eliminate the relative error in the prediction results, the PETR model uses OKS loss. OKS loss is the regression loss calculation between the joint point coordinates of the human posture predicted by the model and the real joint point coordinates. The L ₁ loss is recorded as L _reg and the OKS loss is recorded as L _oks , which are used in the loss functions of the attitude regression head and the joint regression head respectively, so that the attitude decoder and joint point decoder can better converge.

Using methods based on heat map regression to assist training can add a directional guidance to the training of the model. The closer it is to the target joint point, the greater the activation value. The model can quickly approach the target joint point according to the guidance direction and achieve rapid convergence. effect. Therefore, the PETR model uses a deformable transformer encoder to generate heat map predictions, and then calculates a variant Focal Loss between the predicted and true heat maps, denoted as L _hm . The heat map branch is only used to assist training and will be removed during the inference phase. To sum up, all the loss functions of the PETR model are expressed as:

L＝L _cls +λ ₁ L _reg +λ ₂ L _oks +λ ₃ L _hm

Among them, λ ₁ , λ ₂ , λ ₃ represent the corresponding loss weights.

Step S250, the COCO joint point data set is used as the training set, and resnet-50 is used as the backbone network. The model was trained for a total of 100 iterations. The learning rate is 2.5e-5. In order to make the model training converge better, when the training reaches 80 epochs, the learning rate is reduced to 2.5e-6. Train and test the PETR model. Through the trained PETR model, the human body joint point coordinates are detected for each input image and the human body posture is generated.

The step S30 of calculating posture features through the motion range detection algorithm specifically includes:

The input of this step is the human body joint point coordinates in step S20, and the output is the posture feature value of the human body in the image. Actually taking into account the different depth of field of the picture and the scale of the characters, the motion range detection algorithm consists of four detection conditions.

Step S300, the posture estimation joint point connection format of the present invention adopts the COCO human body joint point connection format, as shown in Figure 5, ( _xi , y _i ) is expressed as the x coordinate and y coordinate of the i-th joint point of the human body, and the calculation The coordinates of the midpoint s of the left shoulder and right shoulder joint points (joint point numbers: 5, 6), the coordinates of the midpoint h of the left and right hip joint points (joint point numbers: 11, 12), the left knee and right knee joint points The coordinates of the midpoint k of (joint point number: 13, 14);

Step S310, detection condition 1: Calculate the ratio of the opening and closing length of the hands to the length of the central torso of the body. The opening and closing length of the hands is composed of the joint points of the left and right wrists (joint point numbers: 9, 10), and the length of the body center trunk is composed of s and h. Based on the joint point coordinates, the opening and closing length of the hands d_hand and the length of the body center trunk d_body are calculated. When the ratio of the opening and closing length of the hands to the length of the body center trunk is greater than or equal to the first threshold, it is determined to be a large-scale movement. Otherwise, it is a small-scale movement. According to The first threshold of the experimental results is set to 1.8, and the discrimination formula is as follows:

Step S320, detection condition 2: calculate the tilt angle of the body trunk. The vector x ₁ composed of s and h, the vector x ₂ composed of h and k, the angle of the two vectors is the inclination angle of the body trunk. When the body trunk tilt angle angle_1 is less than or equal to the second threshold, it is determined to be a large-scale movement, otherwise it is a small-scale movement. According to the experimental results, the second threshold is set to 150°. The discrimination formula is as follows:

Step S330, detection condition three: calculate the opening and closing angle of the arm and the trunk. The opening and closing angle of the left arm is composed of the vector x ₃ of the left elbow and left shoulder (joint point number: 7, 5) and the vector x ₄ of the left shoulder and left hip (joint point number: 5, 11). The opening and closing angle of the right arm is composed of the right It consists of the vector x ₅ of the elbow and right shoulder (joint point numbers: 8, 6) and the vector x ₆ of the right shoulder and right hip (joint point numbers: 6, 12). When the left arm opening and closing angle angle_2 or the right arm opening and closing angle angle_3 is greater than or equal to the third threshold, it is determined to be a large-scale movement, otherwise it is a small-scale movement. According to the experimental results, the third threshold is set to 90°. The discrimination formula is as follows:

Step S340, detection condition four: calculate the opening and closing angle of the legs and the trunk. The opening and closing angle of the left leg is composed of the vector x ₇ of the left knee and left hip (joint point numbers: 13, 11) and the vector x ₈ of the left hip and right hip (joint point number: 11, 12). The opening and closing angle of the right leg It consists of the vector x ₉ of the right knee and right hip (joint point numbers: 14, 12) and the vector x ₁₀ of the right hip and left hip (joint point numbers: 11, 12). When the left leg opening and closing angle angle_4 or the right leg opening and closing angle angle_5 is greater than or equal to the fourth threshold, it is determined to be a large-scale movement, otherwise it is a small-scale movement. According to the experimental results, the fourth threshold is set to 125°. The discrimination formula is as follows:

The action range determination step S40 specifically includes:

The input of this step is the posture feature value of the human body in the image output in step S30, and the motion range detection result is output. By calculating the amplitude of changes in posture characteristics, the threshold method is used to determine whether it is a large-scale action;

Step S400, the output result of step S30 is compared with the set threshold. When one of the posture characteristic values among the four detection conditions exceeds the threshold set by the system, it is determined that the performing action is a large-amplitude action, otherwise it is a small-amplitude action. .

According to the judgment result, the intelligent broadcasting system switches lenses step S50, which specifically includes:

The input of this step is the motion amplitude detection result output in step S40, and the intelligent broadcasting system switches the lens to the camera screen corresponding to the large motion image.

Step S500, as described in step S100, each image is named according to the camera position information, so each image represents the current program picture of a camera, for example, 1_01.jpg represents the first frame image of camera No. 1. Therefore, the intelligent director system can switch the lens to the camera screen corresponding to the image with large motion based on the image name and motion amplitude detection results.

The experimental results of applying the present invention are given below.

The interval length of key frame images in the intelligent broadcasting system of the present invention is approximately: T ₁ =0.3333s, and the time required to process one key frame image is: T ₂ =0.3024s. Because T ₁ > T ₂ , the system can complete motion amplitude detection when the next image is input, achieving real-time lens switching. Therefore, the detection speed and accuracy of the present invention meet the application requirements of actual scenarios.

Table 1 shows the results of testing using the present invention on 600 performing arts test images.

Table 1 Performance range of motion test results

Image category quantity Accuracy Big acting moves 300 88.0 Small acting moves 300 86.33

The beneficial effects of the present invention are as follows:

The present invention applies a complete end-to-end multi-person posture estimation model to motion range detection to obtain improvements in function and performance, focusing on improving the accuracy of human joint point positioning and motion range detection. Accurately and quickly calculate posture characteristics based on joint point coordinates, and judge the movement range accordingly, achieving the effect of automatic production or auxiliary production of programs, saving a lot of manpower and material resources, and improving work efficiency. In addition, the present invention also has the following beneficial effects:

1. High operation: It can better reduce the interference of the background environment, focus on the character itself, omit manual operations, do not need to rely on target detection models, reduce a large amount of computing resources, and have the advantages of high-speed operation and low memory consumption.

2. Completely end-to-end, speed and accuracy trade off each other: Compared with methods using two-stage models such as top-down and bottom-up multi-person pose estimation, this model summarizes human pose estimation as a hierarchical set prediction problem. Human body instances and fine-grained human body joint point coordinates are processed uniformly, achieving complete end-to-end multi-person pose estimation. The speed and accuracy have reached a high level, and the two trade off each other.

3. High accuracy: Traditional motion amplitude detection and judgment mainly rely on the relative displacement of joint points in adjacent video frames, and there are fewer detected motion categories. The present invention actually considers the problem of different depth of field of the picture and different scales of the characters, proposes a movement amplitude detection algorithm, and normalizes the movement range detection through gesture characteristic analysis. In the performing arts scene, the detection effect of the performing arts movement amplitude is better. High accuracy.

Claims (6)

1. An intelligent guided broadcast switching method based on multi-person gesture estimation motion amplitude detection is characterized by comprising the following steps:

step 1, performing frame extraction based on a performance video acquired by a camera to acquire an image;

step 2, building and training a multi-person gesture estimation model;

step 3, inputting the image in the step 1 into the multi-person gesture estimation model in the step 2, and calculating each gesture characteristic through a motion amplitude detection algorithm;

step 4, judging the action amplitude of the gesture features obtained in the step 3;

and step 5, performing lens switching according to the judgment result of the action amplitude.

2. The intelligent guided broadcast switching method based on the multi-person gesture estimation motion amplitude detection of claim 1, wherein the step 1 specifically includes:

and acquiring multi-person gesture motion amplitude videos based on the plurality of cameras, and obtaining multi-person gesture motion amplitude images based on the multi-person gesture motion amplitude videos.

Step 1.1, selecting video frames of a multi-person gesture motion amplitude video at intervals of n frames, and directly discarding other video frames due to information redundancy; wherein n is a positive integer greater than 1.

And 1.2, naming each video frame according to camera position information to obtain a plurality of images, wherein each image represents a current program picture of a camera position camera.

3. The intelligent guided broadcast switching method based on the multi-person gesture estimation motion amplitude detection of claim 2, wherein the step 2 specifically includes:

the multi-person pose estimation model, PETR model, mainly includes a backbone network module, a position coding module, a visual feature encoder module, a pose decoder module, and a joint decoder module. Based on the image obtained in the step 1, the PETR model can output coordinates of human joints in the image and draw human gestures in the image according to joint related information.

And 2.1, the backbone network module is used for inputting the image in the step 1 and outputting the image as a multi-scale characteristic map. The backbone network module is ResNet-50, and ResNet-50 is a residual network of 50 layers and is used for extracting the characteristic diagram of the image, and the backbone network module is used for extracting the multi-scale characteristic diagram with high resolution.

Step 2.2, the visual feature encoder generates a position code for each pixel point based on the multi-scale feature map and the position code module obtained in the step 2.1, and generates a multi-scale feature token and a gesture query;

step 2.3, the gesture decoder based on the multi-scale feature token F and N randomly initialized gesture queries Q obtained in step 2.2 _pose ∈R ^N×D And acquires N body posesWherein->Represents the coordinates of the K joints of the ith person, D represents the dimension of the query key. The pose decoder estimates the pose coordinates layer by means of all decoder layers, each layer refining the pose based on the predictions of the previous layer.

Step 2.4, the joint point decoder uses the K joint point information of each group of human body gestures predicted by the gesture decoder in step 2.3 as randomly initialized joint point query, and refines the joint point position information and the joint point structure information. The characteristic information of the node query and the key is continuously updated through self-attention and deformable cross-attention, and finally the node query is aggregated through deviation, so that the node position information and the node structure information can be further refined.

Step 2.5, training by adopting a Hungary loss based on the set, and recording a classification loss function as L _cls Is used in the sorting head. In order to eliminate the relative error of the predicted result, the PETR model adopts OKS loss, which is the regression loss calculation of the joint point coordinates of the human body posture predicted by the model and the real joint point coordinates. L (L) ₁ Loss is marked as L _reg And OKS loss is noted as L _oks The loss functions are respectively used for the gesture regression head and the joint regression head;

the PETR model uses a deformable transducer encoder to generate thermodynamic diagram predictions, and calculates a variant FocalLoss, i.e., focus loss, between the predicted and true thermodynamic diagrams, denoted as L _hm . Thermodynamic branches are used only for training assistance and are removed during the inference phase. All loss functions of the PETR model are expressed as:

L＝L _cls +λ ₁ L _reg +λ ₂ L _oks +λ ₃ L _hm

wherein lambda is ₁ ,λ ₂ ,λ ₃ Representing the corresponding loss weight.

Step 2.6, constructing a COCO node data set as a training set, and training and testing a PETR model; the COCO joint point data contains 200000 images in total, wherein the joint point images total 64355, including 250000 human joint points. Each person is marked with 17 nodes. The PETR model is trained for a total of 100 iteration cycles. The learning rate was 2.5e-5, and was reduced to 2.5e-6 when 80 epochs were trained in order to allow better convergence of PETR model training. And detecting the coordinates of the joint points of the human body for each input image through training the PETR model, and generating the human body posture.

4. The intelligent guided broadcast switching method based on the multi-person gesture estimation motion amplitude detection of claim 3, wherein the step 3 specifically includes:

the input is the coordinates of the joint points of the human body in the step 2, and the coordinates are output as the characteristic values of the posture of the human body in the image. The problem that the depth of field of the picture is different from the figure scale is actually considered, and the action amplitude detection algorithm consists of four detection conditions.

Step 3.1, the joint point connection format of the gesture estimation adopts a COCO human body joint point connection format, (x) _i ,y _i ) The coordinate of the midpoint s of the joint points of the left shoulder and the right shoulder, the coordinate of the midpoint h of the joint points of the left hip and the right hip, and the coordinate of the midpoint k of the joint points of the left knee and the right knee are calculated by representing the coordinate of the x coordinate and the y coordinate of the ith joint point of the human body.

Step 3.2, detection condition one: and calculating the ratio of the opening and closing length of the two hands to the length of the trunk at the center of the body. The opening and closing length of the two hands is composed of left wrist joint points and right wrist joint points, and the length of the trunk at the center of the body is composed of s and h. Calculating the opening and closing length d_hand of the two hands and the length d_body of the body center based on the joint point coordinates, judging that the two hands move in a large amplitude mode when the ratio of the opening and closing length of the two hands to the length of the body center and the body is larger than or equal to a first threshold value, otherwise, setting the first threshold value as 1.8 according to an experimental result, and judging that a formula is shown as follows:

step 3.3, detecting a second condition: the body trunk tilt angle is calculated. Vector x composed of s and h ₁ Vector x consisting of h and k ₂ The angle of the two vectors is the body trunk inclination angle. When the body trunk inclination angle angle_1 is smaller than or equal to a second threshold value, judging that the body trunk inclination angle is large-amplitude motion, otherwise, judging that the body trunk inclination angle is small-amplitude motion, setting the second threshold value to be 150 degrees according to an experimental result, and judging that a formula is as follows:

step 3.4, detecting a third condition: and calculating the opening and closing angles of the arms and the trunk. The opening and closing angle of the left arm is defined by the vector x of the left elbow and the left shoulder ₃ Vector x and left shoulder, left hip ₄ The right arm opening and closing angle consists of a vector x of a right elbow and a right shoulder ₅ Vector x for right shoulder and hip ₆ Composition is prepared. When the left arm opening and closing angle angle_2 and the right arm opening and closing angle angle_3 are larger than or equal to a third threshold, the large-amplitude motion is judged, otherwise, the small-amplitude motion is judged, the third threshold is set to 90 degrees according to the experimental result, and the judgment formula is as follows:

step 3.5, detection condition four: and calculating the opening and closing angles of the legs and the trunk. The left leg opening and closing angle is defined by the vector x of the left knee and the left hip ₇ And vector x of left hip and right hip ₈ The right leg opening and closing angle consists of a vector x of a right knee and a right hip ₉ And vector x of right and left hip ₁₀ Composition is prepared. When the left leg opening and closing angle angle_4 and the right leg opening and closing angle angle_5 are larger than or equal to a fourth threshold value, judging that the motion is large-amplitude motion, otherwise, judging that the motion is small-amplitude motion, and according to the experimental result, the fourth threshold valueThe value is set to 125 degrees, and the discrimination formula is as follows:

。

5. the intelligent guided broadcast switching method based on the multi-person gesture estimation motion amplitude detection of claim 4, wherein the step 4 specifically includes:

the input of the step is the characteristic value of the human body in the image output in the step 3, and the detection result of the action amplitude is output. And judging whether the motion is a large-amplitude motion or not by calculating the amplitude of the change of the gesture characteristics and adopting a threshold method.

And 4.1, comparing the output result of the step 3 with a set threshold value, and judging that the performance acts as a large-amplitude action when one of the gesture characteristic values in the four detection conditions exceeds the set threshold value, or else, judging as a small-amplitude action.

6. The intelligent guided broadcast switching method based on the multi-person gesture estimation motion amplitude detection of claim 5, wherein the step 5 specifically includes:

the step of inputting is that the step of 4 outputs the detection result of the movement amplitude, the intelligent guiding and broadcasting system controls the camera to switch the lens to the camera picture corresponding to the large-amplitude movement image.

Step 5.1, as described in step 1.2, each image is named according to camera position information, and each image represents the current program picture of a camera position. And the intelligent guide system switches the lens to a camera picture corresponding to the large-amplitude action image according to the image name and the action amplitude detection result.