CN113312966B - An action recognition method and device based on a first-person perspective - Google Patents

Abstract

The invention discloses an action recognition method and device based on a first-person perspective. The method includes: acquiring RGB video frames to be processed; the RGB video frames to be processed include hand movement image information based on the first-person perspective; The video frame is input to the pre-trained HOPE-Net deep neural network to obtain the corresponding hand joint position information; select a predetermined number of target RGB video frames from all RGB video frames to be processed, and input them into the I3D model for identification, and obtain the corresponding The video frame features; the hand joint point position information is input into the AGCN model to obtain the corresponding position information features; the video frame features and position information features are fused one by one to obtain the probability of recognizing action instructions. The video frames are sequentially extracted from hand bones and joints, RGB and bone action features are extracted, and finally feature fusion is performed to obtain the action command probability, thereby getting rid of the dependence on external hardware devices, and has strong robustness to lighting and scene changes.

Description

An action recognition method and device based on a first-person perspective

technical field

The present invention relates to the technical field of motion recognition, in particular to a method and device for motion recognition based on a first-person perspective.

Background technique

Although robots can understand human behavior intentions well and learn human behavior autonomously by learning actions from human demonstration videos, in practical applications, the learning of human behavior by robots requires a detailed understanding process, especially It is particularly challenging for robots to learn behaviors derived from daily activities. For example, in the first-person video taken by a wearable camera, the robot can only obtain the human hand's operation from a single angle. In this case, it is full of things such as fast hand movement and occlusion when the hand is operated, resulting in a large degree of unpredictability. Therefore, the recognition of subtle differences in human actions by robots, and the process of learning and executing human actions is still a major problem in the field of robotics, especially in the direction of action recognition from the first-person perspective, which is one of the research hotspots. .

Based on the method of first-person perspective action recognition, the current methods mainly include three types: (1) use sensors such as LeapMtion to collect information about hand joints in demonstration videos, and then assist action recognition. This method requires hardware support and requires operation. Personnel need to demonstrate actions in a specific environment; (2) For demonstration videos, use dense trajectories to represent motion features, and use HOG to collect gesture features. (3) Segment the hands of the operator in the demonstration video and then input them into the deep neural network for recognition. Although this method can effectively reduce background interference, most of the original information is missing. Obviously, the existing action recognition methods based on the first-person perspective have certain defects.

In summary, it is of great significance to propose a first-person perspective action recognition scheme that can get rid of the dependence on external hardware devices and is robust to illumination and scene changes.

Contents of the invention

The present invention provides an action recognition method and device based on a first-person perspective, which can get rid of the dependence on external hardware devices, and has strong robustness to illumination and scene changes.

In the first aspect, the present invention provides an action recognition method based on a first-person perspective, including:

Obtain the RGB video frame to be processed; the RGB video frame to be processed contains hand movement image information based on the first-person perspective;

All the RGB video frames to be processed are input to the pre-trained HOPE-Net deep neural network to obtain the corresponding hand joint point position information;

Select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them into the I3D model for identification to obtain corresponding video frame features;

Input the position information of the joint points of the hand into the AGCN model to obtain the corresponding position information features;

The video frame features and the position information features are fused in one-to-one correspondence to obtain the probability of recognizing an action instruction.

Optionally, the HOPE-Net deep neural network includes: a ResNet10 network and an adaptive graph U-Net network; all the RGB video frames are input to the pre-trained HOPE-Net deep neural network to obtain corresponding hand joints point location information, including:

Encoding and predicting all the RGB video frames through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points;

Inputting a plurality of Cartesian coordinate points of the target plane into the adaptive U-Net network to obtain the position information of the hand joint points corresponding to the RGB video frame.

Optionally, all the RGB video frames are encoded and predicted through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points, including:

Encoding all the RGB video frames to obtain encoded video frames;

Predicting all the encoded video frames to obtain corresponding initial plane Cartesian coordinate points;

Convolving all the initial plane Cartesian coordinate points with the corresponding RGB video frames to obtain the target plane Cartesian coordinate points.

Optionally, obtain RGB video frames to be processed, including:

Obtain the video to be processed; the video to be processed contains hand movement image information based on the first-person perspective;

The hand movement image information is converted into the RGB video frame to be processed by OpenCV.

Optionally, the one-to-one correspondence fusion of the video frame features and the position information features to obtain the probability of recognizing action instructions includes:

Analyzing the distance relationship between the video frame feature and the position information feature through the pre-established relational graph convolution network, and creating a connection between each of the video frame features and each position information feature based on the distance relationship;

The video frame feature and the positional information feature are respectively input into a convolutional network to obtain the convolutional video card feature and the convolutional positional information feature;

The convolved video evidence features and the convolved position information features in the same connection are fused, and the fused information features are input into the fully connected layer network to obtain the probability of the recognition action instruction.

In the second aspect, the present invention provides an action recognition device based on a first-person perspective, including:

An acquisition module, configured to acquire an RGB video frame to be processed; the RGB video frame to be processed includes hand movement image information based on a first-person perspective;

The first input module is used to input all the RGB video frames to be processed to the pre-trained HOPE-Net deep neural network to obtain corresponding hand joint point position information;

Selection module is used to select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them to identify in the I3D model to obtain corresponding video frame features;

The second input module is used to input the position information of the joint points of the hand into the AGCN model to obtain corresponding position information features;

The fusion module is used to fuse the video frame features and the position information features one by one to obtain the probability of recognizing the action instruction.

Optionally, the HOPE-Net deep neural network includes: ResNet10 network and adaptive graph U-Net network; the first input module includes:

The encoding submodule is used to encode and predict all the RGB video frames through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points;

The first input sub-module is configured to input a plurality of Cartesian coordinate points of the target plane into the adaptive U-Net network to obtain the position information of the hand joint points corresponding to the RGB video frame.

Optionally, the coding submodule includes:

An encoding unit, configured to encode all the RGB video frames to obtain encoded video frames;

A prediction unit, configured to predict all the encoded video frames to obtain corresponding initial plane Cartesian coordinate points;

The convolution unit is configured to convolve all the initial plane Cartesian coordinate points with the corresponding RGB video frames to obtain the target plane Cartesian coordinate points.

Optionally, the acquisition module includes:

The obtaining sub-module is used to obtain the video to be processed; the video to be processed contains hand movement image information based on the first-person perspective;

The conversion sub-module is used to convert the hand motion image information into the RGB video frame to be processed through OpenCV.

Optionally, the fusion module includes:

The connection sub-module is used to analyze the distance relationship between the video frame feature and the position information feature through the pre-established relational graph convolution network, and create each of the video frame features and each position based on the distance relationship. connection of information features;

The second input submodule is used to input the video frame feature and the position information feature into the convolutional network respectively to obtain the convolved video card feature and the convolved position information feature;

The fusion sub-module is used to fuse the convolved video card features and the convolved position information features in the same connection, and input the fused information features into the fully connected layer network to obtain the probability of the recognition action instruction.

As can be seen from the above technical solutions, the present invention has the following advantages:

The present invention obtains the RGB video frame to be processed; the RGB video frame to be processed contains hand motion image information based on the first-person perspective; all the RGB video frames to be processed are input to the pre-trained HOPE-Net deep neural network, Obtain the corresponding hand joint point position information; Select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them into the I3D model for identification to obtain corresponding video frame features; The position information is input into the AGCN model to obtain the corresponding position information features; the video frame features and the position information features are fused one by one to obtain the probability of recognizing action instructions. The video frames are sequentially extracted from hand bones and joints, RGB and bone action features are extracted, and finally the feature fusion is performed to obtain the action command probability, thereby getting rid of the dependence on external hardware devices, and has strong robustness to lighting and scene changes.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings on the premise of not paying creative labor;

FIG. 1 is a flow chart of the steps of Embodiment 1 of an action recognition method based on a first-person perspective in the present invention;

FIG. 2 is a flow chart of steps in Embodiment 2 of an action recognition method based on a first-person perspective in the present invention;

Fig. 3 is the processing schematic diagram from the video to be processed to the hand joint position information of the present invention;

Fig. 4 is a schematic structural diagram of an adaptive graph convolution module of the present invention;

Fig. 5 is the schematic diagram of the use of the relational graph convolutional network of the present invention;

FIG. 6 is a structural block diagram of an embodiment of an action recognition device based on a first-person perspective in the present invention.

Detailed ways

Embodiments of the present invention provide an action recognition method and device based on a first-person perspective, which can get rid of the dependence on external hardware devices and have strong robustness to illumination and scene changes.

In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the following The described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment 1, please refer to FIG. 1. FIG. 1 is a flow chart of steps in Embodiment 1 of an action recognition method based on a first-person perspective according to the present invention, including:

S101. Obtain an RGB video frame to be processed; the RGB video frame to be processed includes hand movement image information based on a first-person perspective;

S102, inputting all the RGB video frames to be processed into the pre-trained HOPE-Net deep neural network to obtain corresponding hand joint point position information;

S103, selecting a predetermined number of target RGB video frames from all the RGB video frames to be processed, and inputting them into the I3D model for identification to obtain corresponding video frame features;

S104. Input the position information of the hand joint points into the AGCN model to obtain corresponding position information features;

S105, merging the video frame features and the position information features one by one to obtain a probability of recognizing an action instruction.

In the embodiment of the present invention, by obtaining the RGB video frames to be processed; the RGB video frames to be processed include hand motion image information based on the first-person perspective; all the RGB video frames to be processed are input to the pre-trained HOPE- Net deep neural network to obtain the corresponding hand joint point position information; select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them into the I3D model for identification to obtain the corresponding video frame features; The position information of the hand joint points is input into the AGCN model to obtain the corresponding position information features; the video frame features and the position information features are fused one by one to obtain the probability of recognizing action instructions. The video frames are sequentially extracted from hand bones and joints, RGB and bone action features are extracted, and finally feature fusion is performed to obtain the action command probability, thereby getting rid of the dependence on external hardware devices, and has strong robustness to lighting and scene changes.

Embodiment 2, please refer to FIG. 2. FIG. 2 is a flow chart of steps in Embodiment 2 of an action recognition method based on a first-person perspective according to the present invention, which specifically includes:

Step S201, acquiring the video to be processed; the video to be processed contains hand movement image information based on the first-person perspective;

Step S202, converting the hand movement image information into the RGB video frame to be processed through OpenCV;

In the embodiment of the present invention, first the video to be processed is converted into a plurality of RGB video frames to be processed using OpenCV.

It should be noted that OpenCV is a cross-platform computer vision and machine learning software library that can run on Linux, Windows, Android and Mac OS operating systems. OpenCV has light-weight and high-efficiency features—consists of a series of C functions and a small number of C++ classes, and provides interfaces for languages such as Python, Ruby, and MATLAB, thereby implementing many general-purpose algorithms in image processing and computer vision.

Step S203, encoding and predicting all the RGB video frames through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points;

In an optional embodiment, all the RGB video frames are encoded and predicted by the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points, including:

Encoding all the RGB video frames to obtain encoded video frames;

Predicting all the encoded video frames to obtain corresponding initial plane Cartesian coordinate points;

Convolving all the initial plane Cartesian coordinate points with the corresponding RGB video frames to obtain the target plane Cartesian coordinate points.

In the embodiment of the present invention, the ResNet10 network is used to perform feature encoding on all RGB video frames to obtain encoded video frames, and based on the encoded video frames, the initial plane Cartesian coordinate points are obtained by prediction, and then the initial plane Cartesian coordinate points and The RGB video frame is convolved to obtain a more accurate Cartesian coordinate point of the target plane.

Step S204, inputting a plurality of Cartesian coordinate points of the target plane into the adaptive U-Net network to obtain the position information of the hand joint points corresponding to the RGB video frame;

In the embodiment of the present invention, after obtaining the Cartesian coordinate point of the target plane mentioned in step S203, it is input into the adaptive U-Net network to calculate the depth value of the hand joint point, that is, the corresponding hand joint point position information, The position information of the hand joint points is the final three-dimensional rectangular coordinate information of the 21 hand joints, so as to realize the transformation of the hand joint points from the target plane rectangular coordinates to the three-dimensional rectangular coordinates.

Please refer to Figure 3, Figure 3 is a schematic diagram of the processing from the video to be processed to the position information of the hand joints, in which 1 is the HOPE-Net network, and the HOPE-Net network includes ResNet10 network 2 and U-Net network 3, in the ResNet10 network With the assistance of 2, the initial plane rectangular coordinate points are obtained, and then the target plane rectangular coordinate points are obtained again with the assistance of ResNet10 network 2, and then the hand joint point position information corresponding to the RGB video frame is obtained with the assistance of U-Net network 3 .

Step S205, selecting a predetermined number of target RGB video frames from all the RGB video frames to be processed, and inputting them into the I3D model for identification to obtain corresponding video frame features;

In the embodiment of the present invention, in order to extract more feature details from the RGB video frames to be processed, the I3D model is used to identify the target RGB video frames selected from the RGB video frames to be processed. Extend to three-dimensional convolution, that is, increase the time dimension in the convolution kernel and pooling layer, and use three-dimensional convolution to extract the video frame features corresponding to the target RGB video frame.

It should be noted that the filter of the three-dimensional convolution is N*N*N, that is, the filter weight of N*N is repeated N times along the time dimension, and normalized by dividing by N, and except for the last layer In addition to the convolutional layer, the BN function and the Relu activation function are added after each layer of convolution.

In a specific implementation, 32 frames are selected from the RGB video frames to be processed as a set of input to the I3D model, and the corresponding video frame features are generated through the I3D model.

Step S206, input the position information of the joint points of the hand into the AGCN model to obtain the corresponding position information features;

It should be noted that the AGCN model includes a 9-layer adaptive graph convolution combination module, which automatically generates different topology graphs for different GCN units and different samples. Please refer to Figure 4, Figure 4 is a schematic structural diagram of an adaptive graph convolution module of the present invention, including spatial graph convolution 4, temporal graph convolution 5 and an additional dropout layer 6, each graph convolution layer is followed by There are BN layer 7 and Relu layer 8. Through the combination of 5 different types of layers, the corresponding topological structure diagram is generated. In order to obtain a more stable effect, the adaptive graph convolution combination of each layer in the AGCN model is fast as a residual connection. In addition, the features obtained through the AGCN network model are N*256 features, where N is the number of samples.

In the embodiment of the present invention, the AGCN model is used to represent the natural skeleton structure of the hand mainly based on the position information of the joint points of the hand through a topology map. The model is based on a series of hand skeleton diagrams, that is, the position information of hand joint points. Each node of the hand skeleton diagram represents a joint of the hand at a time, expressed in three-dimensional coordinates. There are two types of edges in the graph, one is the spatial edge between the natural connections of the joints of the hand at a certain moment, and the other is the temporal edge connecting the same joints across consecutive time steps. On this basis, a multi-layer spatio-temporal graph convolution is constructed to realize the aggregation of information in both spatial and temporal dimensions.

Step S207, analyze the distance relationship between the video frame feature and the position information feature through the pre-established relational graph convolution network, and create a relationship between each video frame feature and each position information feature based on the distance relationship connect;

Step S208, respectively input the video frame feature and the position information feature into the convolutional network to obtain the convolved video card feature and the convolved position information feature;

In step S209, the convolved video evidence features and the convolved position information features in the same connection are fused, and the fused information features are input into the fully connected layer network to obtain the probability of the recognition action instruction.

Please refer to Figure 5, Figure 5 is a schematic diagram of the use of the relational graph convolution network, in which 9 is the video frame feature, 10 is the position information feature, 11 is the GCN unit, 12 is the information feature after fusion, and 13 is the recognition action instruction probability. The video frame feature 9 and the location information feature 10 are input to multiple GCN units for convolution respectively, and the video card feature after convolution and the location information feature after convolution are obtained, and then they are fused to obtain the fused information feature 12, and then Get the probability of recognition action command 13. Furthermore, it is possible to perform action recognition on the operation video without restricting the demonstration video and the demonstration environment, and without relying on additional auxiliary sensors.

A kind of action recognition method based on the first-person perspective provided in the embodiment of the present invention, by obtaining the RGB video frame to be processed; The RGB video frame to be processed contains the hand movement image information based on the first-person perspective; The RGB video frames to be processed are input to the pre-trained HOPE-Net deep neural network to obtain the corresponding hand joint point position information; a predetermined number of target RGB video frames are selected from all the RGB video frames to be processed, and input to I3D Identify in the model to obtain the corresponding video frame features; input the position information of the hand joint points into the AGCN model to obtain the corresponding position information features; fuse the video frame features and the position information features one by one to obtain recognition The probability of an action command. The video frames are sequentially extracted from hand bones and joints, RGB and bone action features are extracted, and finally feature fusion is performed to obtain the action command probability, thereby getting rid of the dependence on external hardware devices, and has strong robustness to lighting and scene changes.

Please refer to FIG. 6, which shows a structural block diagram of an embodiment of an action recognition device based on a first-person perspective. The device includes:

The obtaining module 101 is used to obtain the RGB video frame to be processed; the RGB video frame to be processed includes hand movement image information based on the first-person perspective;

The first input module 102 is used to input all the RGB video frames to be processed to the pre-trained HOPE-Net deep neural network to obtain corresponding hand joint point position information;

The selection module 103 is used to select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them into the I3D model for identification to obtain corresponding video frame features;

The second input module 104 is used to input the position information of the joint points of the hand into the AGCN model to obtain the corresponding position information features;

The fusion module 105 is configured to fuse the features of the video frame and the features of the location information in a one-to-one correspondence to obtain the probability of recognizing an action instruction.

In an optional embodiment, the HOPE-Net deep neural network includes: a ResNet10 network and an adaptive graph U-Net network; the first input module 102 includes:

The encoding submodule is used to encode and predict all the RGB video frames through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points;

The first input sub-module is configured to input a plurality of Cartesian coordinate points of the target plane into the adaptive U-Net network to obtain the position information of the hand joint points corresponding to the RGB video frame.

In an optional embodiment, the encoding submodule includes:

An encoding unit, configured to encode all the RGB video frames to obtain encoded video frames;

A prediction unit, configured to predict all the encoded video frames to obtain corresponding initial plane Cartesian coordinate points;

The convolution unit is configured to convolve all the initial plane Cartesian coordinate points with the corresponding RGB video frames to obtain the target plane Cartesian coordinate points.

In an optional embodiment, the acquisition module 101 includes:

The obtaining sub-module is used to obtain the video to be processed; the video to be processed contains hand movement image information based on the first-person perspective;

The conversion sub-module is used to convert the hand motion image information into the RGB video frame to be processed through OpenCV.

In an optional embodiment, the fusion module 105 includes:

The connection sub-module is used to analyze the distance relationship between the video frame feature and the position information feature through the pre-established relational graph convolution network, and create each of the video frame features and each position based on the distance relationship. connection of information features;

The second input submodule is used to input the video frame feature and the position information feature into the convolutional network respectively to obtain the convolved video card feature and the convolved position information feature;

The fusion sub-module is used to fuse the convolved video card features and the convolved position information features in the same connection, and input the fused information features into the fully connected layer network to obtain the probability of the recognition action instruction.

As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described devices and modules can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and other media that can store program codes.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions recorded in each embodiment are modified, or some of the technical features are replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (4)

1. An action recognition method based on a first-person perspective, characterized in that, comprising: Obtain the RGB video frame to be processed; the RGB video frame to be processed contains hand movement image information based on the first-person perspective; All the RGB video frames to be processed are input to the pre-trained HOPE-Net deep neural network to obtain the corresponding hand joint point position information; Select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them into the I3D model for identification to obtain corresponding video frame features; Input the position information of the joint points of the hand into the AGCN model to obtain the corresponding position information features; The video frame feature and the position information feature are fused one by one to obtain the probability of recognizing an action instruction; Described HOPE-Net depth neural network comprises: ResNet10 network and self-adaptive figure U-Net network; All described RGB video frames are input to pre-trained HOPE-Net depth neural network, obtain corresponding hand joint point position information, include: Encoding and predicting all the RGB video frames through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points; A plurality of Cartesian coordinate points of the target plane are input into the adaptive U-Net network to obtain the corresponding hand joint point position information of the RGB video frame; All the RGB video frames are encoded and predicted through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points, including: Encoding all the RGB video frames to obtain encoded video frames; Predicting all the encoded video frames to obtain corresponding initial plane Cartesian coordinate points; Carry out convolution with all described initial plane Cartesian coordinate points and corresponding RGB video frame, obtain target plane Cartesian coordinate point; The video frame feature and the position information feature are fused one-to-one to obtain the probability of recognizing an action instruction, including: Analyzing the distance relationship between the video frame feature and the position information feature through the pre-established relational graph convolution network, and creating a connection between each of the video frame features and each position information feature based on the distance relationship; The video frame feature and the positional information feature are respectively input into a convolutional network to obtain the convolutional video card feature and the convolutional positional information feature; The convolved video evidence features and the convolved position information features in the same connection are fused, and the fused information features are input into the fully connected layer network to obtain the probability of the recognition action instruction. 2. The action recognition method based on the first-person perspective according to claim 1, wherein obtaining RGB video frames to be processed comprises: Obtain the video to be processed; the video to be processed contains hand movement image information based on the first-person perspective; The hand movement image information is converted into the RGB video frame to be processed by OpenCV. 3. An action recognition device based on a first-person perspective, characterized in that it comprises: An acquisition module, configured to acquire an RGB video frame to be processed; the RGB video frame to be processed includes hand movement image information based on a first-person perspective; The first input module is used to input all the RGB video frames to be processed to the pre-trained HOPE-Net deep neural network to obtain corresponding hand joint point position information; The selection module is used to select a predetermined number of target RGB video frames from all the RGB video frames to be processed, and input them into the I3D model for identification to obtain corresponding video frame features; The second input module is used to input the position information of the joint points of the hand into the AGCN model to obtain corresponding position information features; The fusion module is used to fuse the video frame features and the position information features one by one to obtain the probability of recognizing action instructions; Described HOPE-Net deep neural network comprises: ResNet10 network and adaptive figure U-Net network; Described first input module comprises: The encoding submodule is used to encode and predict all the RGB video frames through the ResNet network to obtain a plurality of corresponding target plane Cartesian coordinate points; The first input submodule is used to input a plurality of Cartesian coordinate points of the target plane into the adaptive U-Net network to obtain the corresponding hand joint point position information of the RGB video frame; The encoding submodule includes: An encoding unit, configured to encode all the RGB video frames to obtain encoded video frames; A prediction unit, configured to predict all the encoded video frames to obtain corresponding initial plane Cartesian coordinate points; The convolution unit is used to convolve all the initial plane Cartesian coordinate points and corresponding RGB video frames to obtain the target plane Cartesian coordinate points; The fusion module includes: The connection sub-module is used to analyze the distance relationship between the video frame feature and the position information feature through the pre-established relational graph convolution network, and create each of the video frame features and each position based on the distance relationship. connection of information features; The second input submodule is used to input the video frame feature and the position information feature into the convolutional network respectively to obtain the convolved video card feature and the convolved position information feature; The fusion sub-module is used to fuse the convolved video card features and the convolved position information features in the same connection, and input the fused information features into the fully connected layer network to obtain the probability of the recognition action instruction. 4. The action recognition device based on the first-person perspective according to claim 3, wherein the acquisition module comprises: The obtaining sub-module is used to obtain the video to be processed; the video to be processed contains hand movement image information based on the first-person perspective; The conversion sub-module is used to convert the hand motion image information into the RGB video frame to be processed through OpenCV.