KR20210145271A - Motion recognition method and apparatus, electronic device, computer readable storage medium - Google Patents

Abstract

An embodiment of the present invention provides a method and apparatus for recognizing a motion, an electronic device, and a computer-readable storage medium. Here, the present invention determines motion characteristic information using a target frame corresponding to a target target, and does not determine motion characteristic information using an entire frame image, but data amount for performing motion recognition in each frame image. can be effectively lowered, thereby increasing the quantity of images for performing gesture recognition, which is advantageous for improving the accuracy of gesture recognition; In addition, in the present invention, motion classification and recognition are performed using the motion characteristic information of the target object, and scenario characteristic information of a scenario in which the target object is located and the motion of the target object are performed using a video clip and the determined motion characteristic information. Time-series characteristic information associated with , is extracted, and the accuracy of motion recognition can be further improved by combining scenario information and time-series characteristic information based on the motion characteristic information.

Description

Motion recognition method and apparatus, electronic device, computer readable storage medium

Cross-referencing of related applications

The present invention has been filed based on a Chinese patent application with an application number of 202010166148.8 and an application date of March 11, 2020, and claims priority to the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference. do it with

The present invention relates to the field of computer technology and image processing, and more particularly, to a method and apparatus for recognizing a motion, an electronic device, and a computer-readable storage medium.

Motion detection and recognition has a wide range of applications in robotics, safety and health, and more. In the related art, when performing gesture recognition, there is a limitation in the data processing capability of the recognition device, and the data type for performing gesture recognition is a single equal element, so there is a defect in which gesture recognition accuracy is low.

In view of this, the present invention provides at least a gesture recognition method and apparatus, an electronic device, and a computer-readable storage medium.

In a first aspect, the present invention provides a gesture recognition method, the gesture recognition method comprising:

acquiring a video clip;

determining, by the target object, motion characteristic information of the target object based on an object frame in a key frame image in the video clip;

determining scenario characteristic information and time series characteristic information corresponding to the target object based on the video clip and the motion characteristic information; and

and determining an operation type of the target object based on the operation characteristic information, the scenario characteristic information, and the time series characteristic information.

In an embodiment of the present invention, motion characteristic information is determined by using a target frame corresponding to a target target, and motion characteristic information is not determined using an entire frame image, but for performing motion recognition in each frame image. By being able to effectively lower the amount of data, it is possible to increase the quantity of images for performing motion recognition, which is advantageous for improving the accuracy of motion recognition; In addition, in this aspect, motion classification and recognition are performed using the motion characteristic information of the target object, as well as the scenario characteristic information of the scenario in which the target object is located and the motion of the target object using the video clip and the determined motion characteristic information. Time-series characteristic information associated with , is extracted, and the accuracy of motion recognition can be further improved by combining scenario information and time-series characteristic information based on the motion characteristic information.

In a possible embodiment, the gesture recognition method further comprises the step of determining a target frame in a key frame image, said step comprising:

selecting a key frame image from the video clip;

performing object detection on the key frame image obtained by screening, so that the target object determines an initial object bounding box in the key frame image; and

and performing expansion on the initial target bounding box according to preset expansion size information, so that the target target obtains the target frame in the key frame image.

In the embodiment of the present invention, using the method of object detection, the target object determines the frame in the image, performs motion recognition to reduce the amount of data that needs processing, determines a relatively small initial object bounding box, and then for By performing the extension, the target frame for performing motion recognition can include more complete target target information and more environment information, thereby retaining more spatial detail, which is advantageous for improving the accuracy of motion recognition.

In a possible embodiment, the step of determining, by the target object, the motion characteristic information of the target object based on an object frame in a key frame image in the video clip, comprises:

selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;

obtaining a plurality of target target images corresponding to the key frame images by cutting out partial images from at least partially related images corresponding to the key frame images according to the target frame corresponding to the key frame image; and

and determining motion characteristic information of the target target based on a plurality of target target images corresponding to the key frame image.

In an embodiment of the present invention, the target object performs positioning by using the object frame in the key frame image, and cuts the target object image for determining motion characteristic information from the key frame image and a plurality of associative images associated with each other, The precision of an image used for determining the motion characteristic information is improved, and the number of images for determining the motion characteristic information can be increased, thereby improving the accuracy of motion recognition.

In a possible embodiment, the step of selecting a plurality of associated images corresponding to a key frame image in the video clip comprises:

selecting from the video clip a first sub-video clip comprising a key frame image, wherein the first sub video clip further comprises N images contiguous in time series with the key frame image, where N is a positive integer; ; and

and selecting the plurality of related images from the first sub-video clip.

In an embodiment of the present invention, if an image related to a key frame image is selected in a sub video clip similar to the shooting time of the key frame image, an image with the closest degree of association to the key frame image can be selected, and the key frame image and the key frame image are selected. Based on the image having the closest degree of association, the accuracy of the determined motion characteristic information may be improved.

In a possible embodiment, after obtaining a plurality of target object images and prior to determining motion characteristic information of the target object,

The target image is set as an image having a preset image resolution.

In an embodiment of the present invention, after the target target image is cut out, the target target image is set to a preset resolution, so that the quantity of information included in the target target image can be improved, that is, the cropped target target image can be enlarged. Therefore, it is advantageous to obtain fine grained detail of the target object, thereby improving the accuracy of the determined motion characteristic information.

In a possible embodiment, the determining of scenario characteristic information and time series characteristic information corresponding to the target target based on the video clip and the motion characteristic information includes:

selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;

at least partially performing a video scenario feature extraction operation on the associated image to obtain the scenario feature information;

obtaining initial time-series feature information by performing a time-series feature extraction operation on an object other than the target object in the video clip; and

and determining time-series characteristic information corresponding to the target target based on the initial time-series characteristic information and the operation characteristic information.

In an embodiment of the present invention, by extracting the scenario features from the key frame image and the correlated image, it is possible to obtain relatively complete scenario feature information, and improve the accuracy of motion recognition based on the relatively complete scenario feature information; In addition, in an embodiment of the present invention, time series characteristics of other objects except for the target object, that is, the initial time series characteristic information are extracted, and based on the time series characteristics of other objects and operation characteristic information of the target object, time series characteristics associated with the target object are extracted. After determining the information, the accuracy of motion recognition may be further improved by using time-series characteristic information related to the target object.

In a possible embodiment, the step of obtaining initial time-series feature information by performing a time-series feature extraction operation on an object other than the target object in the video clip includes:

selecting a second sub-video clip including a key frame image from the video clip with respect to the key frame image, wherein the second sub video clip further includes P images adjacent to the key frame image in time series wherein P is a positive integer - ; and

and extracting motion characteristics of objects other than the target object from the image in the second sub-video clip, and using the obtained motion characteristics as the initial time series characteristic information.

In an embodiment of the present invention, it is possible to select a sub video clip relatively similar to the shooting time of the key frame image from the video clip to extract the time series feature, thereby reducing the data amount of the extracted time series feature, and the determined time series feature and the key By being able to improve the association of frame images, it is advantageous to improve the accuracy of motion recognition; Further, in the embodiment of the present invention, it is possible to improve the targetability of the sequential feature used for motion recognition by using the motion characteristic of another object as a time series characteristic, which is advantageous for improving the accuracy of motion recognition.

In a possible embodiment, the determining of time-series characteristic information corresponding to the target target based on the initial time-series characteristic information and the operation characteristic information comprises:

performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively;

performing an average pooling operation on the initial time series feature information subjected to dimension reduction; and

and performing a merging operation on the initial time-series characteristic information on which the average pooling operation has been performed and the dimensionally-reduced operation characteristic information to obtain time-series characteristic information corresponding to the target object.

In an embodiment of the present invention, when determining the time series characteristic information based on the initial time series characteristic information and the operation characteristic information, dimensionality reduction processing is performed on the initial time series characteristic information and the operation characteristic information to reduce the amount of data required for processing By being able to, it is advantageous to improve the efficiency of motion recognition; In addition, the embodiment of the present invention may improve the efficiency of motion recognition by performing an average pooling operation on the dimension-reduced initial time series feature information to simplify the operation step of time series feature extraction.

In a possible embodiment, the determining of time-series characteristic information corresponding to the target target based on the initial time-series characteristic information and the operation characteristic information comprises:

The method further includes using the obtained time-series characteristic information corresponding to the target object as new initial time-series characteristic information, and returning to the step of performing dimension reduction processing on the initial time-series characteristic information and the operation characteristic information, respectively.

In an embodiment of the present invention, based on the initial time-series characteristic information and the operation characteristic information, the time-series characteristic extraction operation for determining the time-series characteristic information corresponding to the target object is repeatedly executed, thereby improving the accuracy of the determined time-series characteristic information have.

In a second aspect, the present invention provides a gesture recognition method, the gesture recognition apparatus comprising:

a video acquisition module configured to acquire a video clip;

a motion characteristic determining module, wherein the target object is configured to determine motion characteristic information of the target object based on an object frame in a key frame image in the video clip;

a scenario time-series characteristic determining module, configured to determine, based on the video clip and the motion characteristic information, scenario characteristic information and time-series characteristic information corresponding to the target object; and

and a motion recognition module configured to determine an operation type of the target object based on the motion characteristic information, the scenario characteristic information, and the time series characteristic information.

In a possible embodiment, the motion characteristic determination module further determines a target frame in the key frame image,

select a key frame image from the video clip;

performing object detection on the key frame image obtained by screening, so that the target object determines an initial object bounding box in the key frame image; and

and perform expansion on the initial object bounding box according to preset extension size information, so that the target object obtains the object frame in the key frame image.

In a possible embodiment, the motion characteristic determining module is configured to: when the target object determines motion characteristic information of the target object based on a target frame in a key frame image in the video clip,

selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;

cutting out partial images from at least partially related images corresponding to the key frame images according to the target frame corresponding to the key frame image, respectively, to obtain a plurality of target target images corresponding to the key frame image;

and determining motion characteristic information of the target target based on a plurality of target target images corresponding to the key frame image.

In a possible embodiment, when the motion characteristic determining module selects a plurality of related images corresponding to a key frame image in the video clip,

select from the video clip a first sub-video clip comprising a key frame image, wherein the first sub video clip further comprises N images contiguous in time series with the key frame image, where N is a positive integer;

and select the plurality of associated images from the first sub-video clip.

In a possible embodiment, after obtaining a plurality of target object images and before determining the operation characteristic information of the target object, the operation characteristic determining module further comprises:

and set the target target image to an image having a preset image resolution.

In a possible embodiment, when the scenario time series feature determination module determines the scenario feature information and time series feature information corresponding to the target object based on the video clip and the motion feature information,

selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;

at least partially performing a video scenario feature extraction operation on the associated image to obtain the scenario feature information;

performing a time-series feature extraction operation on an object other than the target object in the video clip to obtain initial time-series feature information;

and determining time-series characteristic information corresponding to the target target based on the initial time-series characteristic information and the operation characteristic information.

In a possible embodiment, the scenario time series feature determination module performs a time series feature extraction operation on objects other than the target object in the video clip to obtain initial time series feature information,

selecting a second sub-video clip including a key frame image from the video clip with respect to the key frame image, wherein the second sub video clip further includes P images adjacent to the key frame image in time series wherein P is a positive integer - ; and

and extracting motion characteristics of objects other than the target object from the image in the second sub-video clip, and using the obtained motion characteristics as the initial time-series characteristic information.

In a possible embodiment, when the scenario time series characteristic determination module determines time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,

performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively;

performing an average pooling operation on the initial time series feature information that has been dimensionally reduced;

and a merging operation is performed on the initial time-series feature information on which the average pooling operation has been performed and the dimension-reduced operation feature information to obtain time-series feature information corresponding to the target object.

In a possible embodiment, when the scenario time series characteristic determination module determines the time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,

and use the obtained time-series characteristic information corresponding to the target object as new initial time-series characteristic information, and return to the step of performing dimension reduction processing on the initial time-series characteristic information and the operation characteristic information, respectively.

In a third aspect, the present invention provides an electronic device, wherein the electronic device includes a processor and a storage medium connected to each other, wherein the storage medium stores machine-readable instructions executable by the processor, the electronic device When activated, the processor executes the steps of the method for recognizing the motion by executing the machine readable instructions.

In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, the same steps as the operation recognition method when the computer program is operated by a processor run

In a fifth aspect, the present invention further provides a computer program comprising computer readable code, wherein when the computer readable code is operated in an electronic device, a processor in the electronic device performs the steps of the method for recognizing the motion. implement

The device, the electronic device and the computer-readable storage medium of the present invention include at least the technical features substantially the same as or similar to the technical features of any one embodiment of the method or any one aspect of the present invention. , therefore, for the description of the effect of the device, the electronic device and the computer-readable storage medium, reference may be made to the description of the effect of the method content, which is not repeated herein any further.

In order to more clearly explain the technical solutions of the embodiments of the present invention, the drawings that will be used in the embodiments below will be briefly described and understood, since the drawings below only show some embodiments of the present invention, It should not be construed as a limitation in scope, and a person skilled in the art may obtain other related drawings according to these drawings, even on the premise that creative labor is not imparted.
1A is a flowchart illustrating a gesture recognition method provided in an embodiment of the present invention.
1B shows an exemplary diagram of a network architecture provided in an embodiment of the present invention.
2 is a flowchart of determining motion characteristic information of a target object in another motion recognition method provided in an embodiment of the present invention.
FIG. 3 is a flowchart of determining scenario characteristic information and time series characteristic information corresponding to the target object in another motion recognition method provided in an embodiment of the present invention.
4 is a diagram illustrating the structure of a simplified time series feature extraction module in an embodiment of the present invention.
5 illustrates another method for recognizing a motion provided in an embodiment of the present invention.
6 is a structural exemplary diagram of another gesture recognition apparatus provided in an embodiment of the present invention.
7 is a schematic structural diagram of another electronic device provided in an embodiment of the present invention.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the drawings in the embodiments of the present invention are combined below to give a clear and complete description of the technical solutions in the embodiments of the present invention, It should be understood that, in the present invention, the drawings are used only for the purpose of description and description, and are not intended to limit the protection scope of the present invention. Also, it should be understood that the exemplary drawings are not drawn to scale. The flowchart used in the present invention illustrates the operations implemented in accordance with some embodiments of the present invention. It should be understood that the sequence of operations in the flowchart need not be implemented, and steps without a logical contextual relationship may be performed in the reverse order or concurrently. In addition, a person skilled in the art may add one or a plurality of other operations to the flowchart, and may remove one or more operations from the flowchart under the guidance of the present invention.

In addition, the described embodiments are merely some embodiments of the present invention, not all embodiments. Typically, the components of the embodiments of the invention described and illustrated in the drawings herein are arranged and designed in a variety of different configurations. Accordingly, the detailed description of the embodiments of the present invention provided in the drawings below is not intended to limit the scope of the present invention for which protection is requested, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art under the premise that creative labor is not given fall within the protection scope of the present invention.

It should be noted that in the embodiments of the present invention, the term “comprising” is used, and is intended to indicate the existence of a later stated feature, but does not exclude incrementing other features.

The present invention provides a gesture recognition method and apparatus, an electronic device, and a computer-readable storage medium for a technical problem of low recognition precision existing in gesture recognition. Here, the present invention determines motion characteristic information using a target frame corresponding to a target target, and does not determine motion characteristic information using an entire frame image, but data amount for performing motion recognition in each frame image. can be effectively lowered, thereby increasing the quantity of images for performing gesture recognition, which is advantageous for improving the accuracy of gesture recognition; In addition, in the present invention, motion classification and recognition are performed using the motion characteristic information of the target object, and scenario characteristic information of a scenario in which the target object is located and the motion of the target object are performed using a video clip and the determined motion characteristic information. Time-series characteristic information associated with , is extracted, and the accuracy of motion recognition can be further improved by combining scenario information and time-series characteristic information based on the motion characteristic information.

Hereinafter, a method and apparatus for recognizing a motion of the present invention, an electronic device, and a computer-readable storage medium will be described through specific embodiments.

An embodiment of the present invention provides a method for recognizing a motion, the method for recognizing a motion is applied to a hardware device such as a terminal device for performing motion recognition, and the method for recognizing a motion may be implemented by a processor executing a computer program. have. Specifically, as shown in FIG. 1A , the gesture recognition method provided in the embodiment of the present invention includes the following steps.

In step S110, a video clip is acquired.

Here, the video clip is a video clip for performing motion recognition, and includes a plurality of images, the image includes a target object requiring motion recognition, and the target object may be a person or an animal.

The video clip may be photographed by the terminal device performing motion recognition using its own camera, such as a recording device, may be photographed by another photographing device, or may be captured by another photographing device and then the video clip It may be transmitted to a terminal device that performs motion recognition.

In step S120, the target object determines motion characteristic information of the target object based on the object frame in the key frame image in the video clip.

Here, the target frame is a bounding box surrounding the target target, and when determining the motion characteristic information of the target target using image information in the bounding box, the amount of data processed by the terminal device may be reduced.

Before determining the motion characteristic information based on the target frame, key frame images in a video clip should be selected first, and the target target should determine the target frame in each key frame image.

In a specific implementation, a predetermined time interval may be used to select a key frame image from a video clip, and of course, other methods may be used to select a key frame image from a video clip, for example, a plurality of video clips After dividing into two sub clips, one frame image is extracted from each sub clip and used as a key frame image. The present invention is not limited to the method of selecting key frame images in a video clip.

After selecting a plurality of key frame images from a video clip, the target frame in each key frame image can be used to determine the motion characteristic information of the target target, and of course, partial key frames in the selected multiple key frame images The motion characteristic information of the target object may be determined by using the target frame in the image. When determining the motion characteristic information of the target object by using the target frame in the partial key frame image, the target frame in the partial key frame image is extracted or determined, and then the motion characteristic information of the target object is obtained by using the extracted or determined frame again. have to decide

In a specific implementation, the method of object detection may be used, for example, a human body detector may be used, the method of human body detection may be used to determine the object frame, and of course, other methods may be used to determine the object frame. Also, the present invention does not limit the method of determining the target frame.

When specifically implemented, a target frame detected and obtained by the human body detector may be used as a final target frame for determining motion characteristic information. However, since the target frame detected and obtained by the human body detector may be a relatively small frame including the target target, in order to obtain more complete target target information and more environmental information, the human body detector detects and obtains the target frame, and then According to the set extension size information, each of the target frames detected and obtained by each human body detector may be expanded, so that the target object may obtain a final target frame in each of the key frame images. Next, motion characteristic information of the target target is determined using the determined final target frame.

The extended size information for performing the expansion on the target frame is preset, for example, the extended size information includes a first extension length of the target frame in the longitudinal direction and a second extension length of the target frame in the width direction of the target frame. include According to the first extension length, it extends on both sides with respect to the length of the target frame, and extends on both sides in the longitudinal direction by half of the first extension length, respectively. The second extension length extends to both sides with respect to the width of the target frame, and extends to both sides in the width direction by half the second extension length, respectively.

The first extension length and the second extension length may be predetermined specific values, or may be values determined based on the length and width of the target frame obtained by being directly detected by the human body detector. For example, the first extended length may be equal to the length of the target frame obtained by being directly detected by the human body detector, and the second extended length may be equal to the width of the target frame obtained by being directly detected by the human body detector.

Through the above form, using the method of object detection, the target object determines the frame in the image, performs motion recognition to reduce the amount of data that needs processing, determines a relatively small initial object bounding box, and then performs motion recognition It is advantageous to improve the accuracy of motion recognition by enabling the target frame configured to include more complete target target information and more environmental information.

The motion characteristic information is information extracted from an image in a video clip and may indicate motion characteristics of a target object.

In step S130, based on the video clip and the motion characteristic information, scenario characteristic information and time series characteristic information corresponding to the target object are determined.

Here, the scenario feature information is to indicate the scenario feature of a scenario in which the target object is located, and may be obtained by performing scenario feature extraction from at least partially related images associated with the key frame image.

The time series characteristic information is characteristic information related to the motion of the target object in time series, and may be, for example, motion characteristic information of an object other than the target object in a video clip. It may be determined based on the operation characteristic information.

In step S140, an operation type of the target is determined based on the operation characteristic information, the scenario characteristic information, and the time series characteristic information.

After determining the operation characteristic information, the scenario characteristic information, and the time series characteristic information, the three pieces of information may be merged. For example, by concatenating and then merging information obtained by classifying the information obtained, the motion type of the target target is obtained, and motion recognition of the target target is implemented.

In an embodiment of the present invention, motion characteristic information is determined by using a target frame corresponding to a target target, and motion characteristic information is not determined using an entire frame image, but for performing motion recognition in each frame image. By effectively lowering the amount of data, it is possible to increase the quantity of images for performing gesture recognition, which is advantageous for improving the accuracy of gesture recognition; In addition, in the embodiment of the present invention, motion classification and recognition are performed using the motion characteristic information of the target object, and the scenario characteristic information and the target of the scenario in which the target object is located using the video clip and the determined motion characteristic information. Time-series characteristic information related to the motion of the target is extracted, and the accuracy of motion recognition can be further improved by combining scenario information and time-series characteristic information based on the motion characteristic information.

In an embodiment of the present invention, motion recognition for a target is performed through a network architecture as shown in FIG. 1B, and FIG. 1B shows an exemplary diagram of a network architecture provided in an embodiment of the present invention. The network architecture includes a user terminal 201 , a network 202 , and a terminal device 203 of gesture recognition. Example application To support the user terminal 201 and the terminal device 203 of gesture recognition to establish a communication connection through the network 202, the user terminal 201 acquires the operation type of the target object When to do, first, send the request information configured to determine the operation type to the terminal device 203 of the gesture recognition via the network 202; The motion recognition terminal device 203 determines motion characteristic information by using a target frame corresponding to the target target through acquiring a video clip; extracting scenario characteristic information of a scenario in which the target object is located and time series characteristic information associated with the motion of the target object by using the video clip and the determined motion characteristic information; Finally, the operation type of the target is determined with higher accuracy by comprehensively considering the scenario information and the time series characteristic information, and the determined operation type is fed back to the user terminal 201 .

As an example, the user terminal 201 may include a device having data processing capability, and the gesture recognition terminal device 203 may include an image collection device and a processing device or remote server having data processing capability. have. The network 202 may use a wired connection or a wireless connection type. Here, when the gesture recognition terminal device is a processing device having data processing capability, the user terminal 201 may communicate with the processing device through a wired connection, for example, perform data communication through a bus, ; When the gesture recognition terminal device 203 is a remote server, the user terminal may perform data interaction with the remote server through a wireless network.

In some embodiments, as shown in FIG. 2 , the determining of the motion characteristic information of the target object according to the target frame in the key frame image in the video clip by the target object includes: It can be implemented using the same steps as

In step S210, a plurality of related images corresponding to the key frame image are selected from the video clip with respect to the key frame image.

Here, the related image associated with the key frame image may be an image similar to an image characteristic of the key frame image, and may be, for example, an image similar to a shooting time of the key frame image.

When specifically implemented, a related image corresponding to the key frame image may be selected using the following sub-steps.

A first sub-video clip according to sub-step 1, comprising a key frame image in the video clip, wherein the first sub video clip further comprises N images adjacent in time series to the key frame image, where N is positive Integer - Select .

In the first sub-video clip, the key frame image may be located in a clip in the first half of the first sub-video clip, or in a clip in the second half of the first sub-video clip, of course, in the first sub video clip It may have been located in the middle or close to the center of

In a possible embodiment, a sub-video clip containing a key frame image may be truncated from the video clip, eg a sub-video clip of 64 frames may be truncated. In the sub video clip, the key frame image is in the middle or close to the middle of the sub video clip. For example, the sub video clip includes a 32 frame image before a key frame image, a key frame image and a 31 frame image after the key frame image; Also for example, in the sub video clip, the key frame image is in the clip of the first part of the sub video clip, and the sub video clip is a 10 frame image before the key frame image, a key frame image and 53 after the key frame image Contains frame images. Also for example, in the sub video clip, the key frame image is in the clip of the latter part of the sub video clip, and the sub video clip is a 50 frame image before the key frame image, a key frame image and 13 after the key frame image. Contains frame images.

Also, in the first sub video clip, the key frame image may be located at both ends of the first sub video clip, that is, the N images adjacent to the key frame image in time series are the N images before the key frame image. or back N images. The present invention does not limit the position of the key frame image in the first sub video clip.

In sub-step 2, the plurality of related images are selected from the first sub-video clip.

In a possible implementation form, the associated image may be selected from the first sub-video clip based on a preset time interval, for example, sparsely sampled with a period τ in the first sub-video clip to obtain the T frame associated image. The selected related image may include a key frame image or may not include a key frame image, and has a certain randomness, and the present invention does not limit whether or not the key frame image is included in the related image.

Based on a preset time interval, if an image related to the key frame image is selected from a sub video clip similar to the shooting time of the key frame image, an image that is the closest to the key frame image can be selected, and the key frame image It is possible to improve the accuracy of the determined motion characteristic information based on the image having the closest correlation to .

In addition, other methods may be used to select key frame images and related images associated with each other. For example, in the first sub video clip, the image similarity between each frame image and the key frame image is first calculated, and then a plurality of images having the highest image similarity may be selected and used as the key frame image and the related image associated with each other.

In step S220, according to the target frame corresponding to the key frame image, partial images are cut out from at least partially related images corresponding to the key frame image, respectively, to obtain a plurality of target target images corresponding to the key frame image .

Here, by using the target frame corresponding to the key image, the partial image is cut out from the partial or all related images related to the key frame image. If the target target image is cut from the partially related image, specifically, the target target image may be cut by selecting the partially related image most similar to the shooting time of the key frame image from all related images, and of course using another method It may be cutting out the target target image by selecting a partially related image. For example, according to a certain time interval, a partially associative image is selected from all associative images.

When cutting the target target image according to the target frame corresponding to the key frame image, the specific process includes first copying the target frame from all the related images or the partial related images according to the chronological order. Here, when copying the target frame in the associative image, the target frame implements frame copying in the associative image by using the coordinate information in the key frame image, for example, the target frame in the coordinate information in the key frame image. Accordingly, the frame position offset is performed according to the temporal sequence or the frame position is directly copied to obtain the target frame in the associated image. Next, after the target frame is copied, the related image is cropped according to the target frame to obtain a target target image, that is, an image in the target frame from the related image is cut out and used as the target target image.

The action of the key frame image is to implement the positioning of the target target image, and is not necessarily for directly determining the motion characteristic information. For example, when the key frame image is not included in the associated image, the target target image for determining the motion characteristic information from the key frame image is not cut out.

In step S230, motion characteristic information of the target target is determined based on a plurality of target target images corresponding to key frames.

After the target target image is cut, motion feature extraction may be performed on each of a plurality of target target images, and specifically, the target target image may be processed using a three-dimensional (3D) convolutional neural network. , by extracting the motion characteristics from the target target image, it is possible to obtain motion characteristic information of the target target.

In addition, in an embodiment of the present invention, after obtaining a plurality of target target images, before determining the motion characteristic information of the target target, the following steps may be used to process the target target images.

and set the target target image to an image having a preset image resolution. The preset image resolution is higher than the original image resolution of the target image. In concrete implementation, an existing method or tool may be used to set the image resolution of the target image, for example, an interpolation method or the like is used to adjust the image resolution of the target image.

Here, the target target image is cut out, and then the target target image is set to a preset resolution, so that the quantity of information included in the target target image can be improved, that is, by enlarging the cut target target image, more By making it possible to maintain fine-grained detail, the accuracy of the determined motion characteristic information can be improved.

When specifically implemented, the preset image resolution can be set to H×W, the number of target target images cut from each frame key frame image is T, and the number of channels of each frame target target image is 3, 3D convolve It is an image block of T×H×W×3 to perform motion feature extraction by inputting the lution neural network. After global average pooling is performed on the input image block through the 3D convolutional neural network, a 2048-dimensional feature vector may be obtained, and the feature vector is the motion feature information.

In an embodiment of the present invention, the target object performs positioning by using the object frame in the key frame image, and cuts the target object image for determining motion characteristic information from a plurality of associated images associated with each other with the key frame image, The precision of an image used for determining the motion characteristic information is improved, and the number of images for determining the motion characteristic information can be increased, thereby improving the accuracy of motion recognition.

In some embodiments, as shown in FIG. 3 , the determining of scenario characteristic information and time series characteristic information corresponding to the target target based on the video clip and the motion characteristic information includes:

In step S310, a plurality of related images corresponding to the key frame image are selected from the video clip with respect to the key frame image; A video scenario feature extraction operation is performed on at least partially related images to obtain the scenario feature information.

Here, by performing video scenario feature extraction and global average pooling on part or all of the associated images using a 3D convolutional neural network specifically, a 2048-dimensional feature vector can be obtained, and the feature vector is the scenario feature information. am.

In step S320, a time series feature extraction operation is performed on objects other than the target object in the video clip to obtain initial time series feature information.

Here, the initial time-series characteristic information is, for example, a time-series characteristic of another object other than the target object, such as an operation characteristic of another object, and when specifically implemented, may be determined through the following sub-steps.

The second sub-video clip according to sub-step 1, which includes a key frame image in the video clip with respect to the key frame image, wherein the second sub video clip is P images adjacent to the key frame image in time series , where P is a positive integer.

In the second sub video clip, the key frame image may be located in a clip in the first half of the second sub video clip, or in a clip in the second half in the second sub video clip, of course, in the second sub video clip It may have been located in the middle or close to the center of

Also, in the second sub video clip, key frame images may be located at both ends of the second sub video clip, that is, the P images adjacent to the key frame image in time series are the P images before the key frame image. or back P images. The present invention does not limit the position of the key frame image in the second sub video clip.

In a possible implementation form, a sub-video clip comprising a key frame image may be cut out from the video clip, and a sub-video clip of 2 seconds may be cut out, wherein the time of the sub video is for determining a time series characteristic of a relatively long time series .

In sub-step 2, motion characteristics of objects other than the target object are extracted from each image in the second sub-video clip, and the obtained motion characteristics are used as the initial time-series characteristic information.

Here, in detail, motion features of objects other than the target object can be extracted from the sub-video clip using a 3D convolutional neural network, and the obtained initial time-series feature information is a video time-series feature library (long-term feature bank, LFB). ) can be stored and used in the form of

In an embodiment of the present invention, by selecting a sub-video clip relatively similar to the shooting time of the key frame image from the video clip to extract the time series feature, the amount of data of the extracted time series feature can be reduced, and the determined time series feature and the key By being able to improve the association of frame images, it is advantageous to improve the accuracy of motion recognition; In addition, in an embodiment of the present invention, it is possible to improve the targetability of the sequential feature used for motion recognition by using the motion characteristics of other objects as time series characteristics, which is advantageous for improving the accuracy of motion recognition.

In step S330, based on the initial time series characteristic information and the operation characteristic information, time series characteristic information corresponding to the target object is determined.

Here, time-series feature extraction may be performed on the initial time-series feature information and the operation feature information to obtain time-series feature information corresponding to the target object.

In a possible implementation form, time-series feature information corresponding to a target object can be obtained by performing time-series feature extraction on initial time-series feature information and operation feature information using the following sub-steps.

In sub-step 1, dimension reduction processing is performed on the initial time series characteristic information and the operation characteristic information, respectively.

It is possible to reduce the amount of data required for processing by obtaining initial time-series characteristic information and operation characteristic information of the target object other than the target object, and then performing dimensionality reduction processing on the initial time-series characteristic information and operation characteristic information. It is advantageous for improving the efficiency of motion recognition.

In a possible implementation form, after obtaining initial time-series characteristic information and operation characteristic information, random dropout processing may be performed on the initial time-series characteristic information and operation characteristic information, and the dropout processing is performed on the initial time-series characteristic information and operation characteristic information. It may be implemented in the last network layer of the neural network for extracting information, and may be implemented in each network layer of the neural network for extracting initial time series characteristic information and operation characteristic information.

In sub-step 2, an average pooling operation is performed on the initial time series feature information that has been dimensionally reduced.

In sub-step 3, a merging operation is performed on the initial time-series feature information on which the average pooling operation has been performed and the dimension-reduced operation feature information to obtain time-series feature information corresponding to the target object. The merging operation may specifically implement channel merging, that is, adding a channel of one feature information to a channel of another feature information and then merging; The merging operation may be an addition operation, that is, an operation of adding initial time series characteristic information subjected to an average pooling operation and operation characteristic information subjected to dimensionality reduction.

Sub-steps 2 and 3 are substantially to perform a time-series feature extraction operation on the initial time-series feature information and operation feature information, and can be specifically implemented using a simplified time-series feature extraction module as shown in FIG. 4 . . The simplified time series feature extraction module 400 as shown in FIG. 4 is for extracting the time series feature information, and specifically, a linear layer 401, an average pooling (Average) layer 402, standardization and It may include an activation function (LN+ReLU) layer 403 and a random dropout layer 404 . In the sub-step 2, the time series feature extraction operation is simplified, the average pooling operation is performed on the initial time series feature information that has been dimensionally reduced using only the average pooling Average layer, and the normalization (softmax) operation is not performed, and the time series The operation step of feature extraction is simplified, that is, the existing time-series feature extraction module is simplified to improve the efficiency of motion recognition. Here, the existing time series feature extraction module does not include an average pooling layer, but includes a classification normalization softmax layer, and the processing complexity performed in the softmax layer is higher than the average pooling operation. In addition, since the existing time series feature extraction module further includes a linear layer before the dropout layer, and the simplified time series feature extraction module in the present invention does not include the linear layer, the efficiency of motion recognition can be further improved. have.

Specifically, the time series feature information output by the time series feature extraction module may be a 512-dimensional feature vector, and the 512-dimensional feature vector is time series feature information of the target object.

In an embodiment of the present invention, by extracting scenario features from a partially or all related image associated with the key frame image, relatively complete scenario feature information can be obtained, and the accuracy of motion recognition can be improved based on the relatively complete scenario feature information. can In addition, in an embodiment of the present invention, time series characteristics of other objects except for the target object, that is, the initial time series characteristic information are extracted, and time series characteristics associated with the target object are extracted based on the time series characteristics of other objects and the operation characteristic information of the target object After determining the information, the accuracy of motion recognition may be further improved by using time-series characteristic information related to the target object.

In order to further improve the accuracy of the extracted time-series feature information, a plurality of time-series feature extraction modules are serially connected to extract the time-series feature information, and the time-series feature information extracted by the time-series feature extraction module is obtained by extracting the time-series feature information from other time-series feature extraction modules. can be used as input. Specifically, using the time series characteristic information corresponding to the target object extracted by the previous time series characteristic extraction module as new initial time series characteristic information, and performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively can be returned as

When specifically implemented, the final time series feature information can be determined by connecting three simplified time series feature extraction modules in series.

A method for recognizing a motion of the present invention will be described below through specific embodiments.

As shown in FIG. 5 , in the embodiment of the present invention, motion recognition is performed by targeting a person. Specifically, the method for recognizing a motion according to an embodiment of the present invention may include the following steps.

In step 1, a video clip 501 is obtained, and a key frame image is selected from the video clip.

In step 2, by using the human body detector 502, a person positioning is performed on each key frame image to obtain an initial object bounding box of a person, that is, a target object.

In step 3, the initial target bounding box is expanded according to preset expansion size information to obtain a final target frame; By using the target frame, partial image cutting is performed on the key frame image and the correlated image to obtain a target target image corresponding to each key image.

In step 4, the target target images corresponding to all the obtained key images are input to the 3D convolutional neural network 503, the motion characteristics of the target are extracted using the 3D convolutional neural network 503, and the target target to obtain operation characteristic information corresponding to .

In step 5, the key frame image and the related image associated with each other are input to the 3D convolutional neural network 503, and the video scenario feature of the scenario in which the target object is located is extracted using the 3D convolutional neural network 503. , to obtain scenario characteristic information.

In step 6, time-series feature extraction is performed on the video clip using another 3D convolutional neural network 503, that is, motion features of objects other than the target object are extracted to obtain initial time-series feature information; The initial time series feature information may exist in the form of a time series feature library; Here, when time-series feature extraction is performed, it may be extracted from the entire video clip as well as from a relatively long sub-video clip including a key frame image in the video clip.

In step 7, a time-series feature extraction operation is performed on the initial time-series feature information and the operation feature information using the simplified time-series feature extraction module 504 to obtain time-series feature information corresponding to a target object.

In step 8, the time series characteristic information, the operation characteristic information, and the scenario characteristic information are spliced, and the information obtained by splicing is classified using the motion classifier 505 to obtain a target motion type.

Corresponding to the gesture recognition method, the present invention further provides a gesture recognition apparatus, wherein the gesture recognition apparatus is applied to a hardware device such as a terminal device for performing gesture recognition on a target object, each module having the same method as the method Since the same beneficial effects can be obtained by implementing the steps, the present invention will not be further described for the same parts.

Specifically, as shown in Figure 6, the operating device provided by the present invention,

a video acquisition module 610 configured to acquire a video clip;

a motion characteristic determining module (620), configured for the target object to determine motion characteristic information of the target object based on an object frame in a key frame image in the video clip;

a scenario time series characteristic determining module 630, configured to determine scenario characteristic information and time series characteristic information corresponding to the target object based on the video clip and the motion characteristic information; and

and a motion recognition module 640 configured to determine the motion type of the target target based on the motion characteristic information, the scenario characteristic information, and the time series characteristic information.

In some embodiments, the operation characteristic determining module 620 is further configured to determine a target frame in the key frame image;

select a key frame image from the video clip;

performing object detection on the key frame image obtained by screening, so that the target object determines an initial object bounding box in the key frame image;

and perform expansion on the initial object bounding box according to preset extension size information, so that the target object obtains the object frame in the key frame image.

In some embodiments, the motion characteristic determining module 620 is configured to: when the target object determines motion characteristic information of the target object based on a target frame in a key frame image in the video clip,

selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;

cutting out partial images from at least partially related images corresponding to the key frame images according to the target frame corresponding to the key frame image, respectively, to obtain a plurality of target target images corresponding to the key frame image;

and determine motion characteristic information of the target target based on a plurality of target target images corresponding to the key frame images.

In some embodiments, when the operation characteristic determining module 620 selects a plurality of related images corresponding to a key frame image in the video clip,

select from the video clip a first sub-video clip comprising a key frame image, wherein the first sub video clip further comprises N images contiguous in time series with the key frame image, where N is a positive integer;

and select the plurality of associated images from the first sub-video clip.

In some embodiments, after obtaining a plurality of target object images and before determining the operation characteristic information of the target object, the operation characteristic determining module 620 is further configured to:

and set the target target image to an image having a preset image resolution.

In some embodiments, the scenario time series feature determination module 630 determines the scenario feature information and time series feature information corresponding to the target target based on the video clip and the motion feature information,

selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;

at least partially performing a video scenario feature extraction operation on the associated image to obtain the scenario feature information;

performing a time-series feature extraction operation on an object other than the target object in the video clip to obtain initial time-series feature information;

and determine time-series characteristic information corresponding to the target target based on the initial time-series characteristic information and the operation characteristic information.

In some embodiments, the scenario time series feature determination module 630 performs a time series feature extraction operation on an object other than the target object in the video clip to obtain initial time series feature information,

for the key frame image, select a second sub video clip including a key frame image in the video clip, the second sub video clip further comprising P images adjacent in time series to the key frame image and P is a positive integer - ;

and extracting motion characteristics of objects other than the target object from the image in the second sub-video clip, and using the obtained motion characteristics as the initial time-series characteristic information.

In some embodiments, when the scenario time series characteristic determination module 630 determines time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,

performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively;

performing an average pooling operation on the initial time series feature information that has been dimensionally reduced;

and a merging operation is performed on the initial time-series feature information on which the average pooling operation has been performed and the dimension-reduced operation feature information to obtain time-series feature information corresponding to the target object.

In some embodiments, the scenario time series characteristic determination module 630 is further configured to determine time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,

and use the obtained time-series characteristic information corresponding to the target object as new initial time-series characteristic information, and return to the step of performing dimension reduction processing on the initial time-series characteristic information and the operation characteristic information, respectively.

An embodiment of the present invention discloses an electronic device, and as shown in FIG. 7 , includes a processor 701 and a storage medium 702 connected to each other, wherein the storage medium includes machine-readable instructions executable by the processor stored, and when the electronic device is operated, the processor executes the step of the method for recognizing the motion by executing the machine-readable instruction. Specifically, the processor 701 and the storage medium 702 may be connected via a bus 703 .

When the machine-readable instructions are executed by the processor 701, the following steps of the action recognition method are executed.

acquire a video clip;

the target object determines motion characteristic information of the target object based on an object frame in a key frame image in the video clip;

determine scenario characteristic information and time series characteristic information corresponding to the target object based on the video clip and the motion characteristic information;

An operation type of the target is determined based on the operation characteristic information, the scenario characteristic information, and the time series characteristic information.

In addition, when the machine-readable instructions are executed by the processor 701 , the method contents in any one embodiment described in the method section may be executed, which is not repeated herein any more.

An embodiment of the present invention further provides a computer program product corresponding to the method and apparatus, comprising a computer readable storage medium storing the program code, and the instructions included in the program code include the method in the previous method embodiment. for implementation, and specific implementations may refer to method embodiments, which are not repeated herein any further. The computer-readable storage medium may be a volatile or non-volatile storage medium.

The description of each embodiment above focuses on the differences between the embodiments, and the same or similar parts of the embodiments may refer to each other, and will not be described further for the sake of brevity.

Those skilled in the art can clearly understand that for the convenience and conciseness of the description, the specific working process of the system and apparatus described above may refer to the corresponding process in the method embodiment, and in the present invention, do not explain In the several embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods may be embodied in other forms. The device embodiment described above is merely exemplary, for example, the division of the module is only logical function division, and there may be other division forms when actually implemented in practice, and for example, a plurality of modules or components It may be combined or integrated into other systems, or some features may be ignored or not implemented. Further, the mutual or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some communication interface, device or module, and may be in the form of electrical, mechanical, or other forms.

The module described as the separation member may or may not be physically separated, and the member posted as a module may or may not be a physical unit, that is, it may be located in one place or may be distributed in a plurality of network units. . According to actual needs, some or all of them may be selected to implement the purpose of the embodiments of the present application.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist alone physically, and two or two or more units may be integrated into one unit. .

When the above functions are implemented in the form of a software function unit and sold or used as a standalone product, the function may be stored in a non-volatile computer-readable storage medium executable by the processor. Based on this understanding, the technical solution of the present invention essentially or a part contributing to the existing technology, but a part of the technical solution may be implemented in the form of a software product, and the computer software product is stored in one storage medium and includes several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or partial steps of the method according to each embodiment of the present invention. The above-described storage medium includes various media capable of storing a program code, such as a USB memory, an external hard drive, a read-only memory (ROM), a random access memory (RAM), a diskette, or a CD.

The above is only a specific implementation method of the present invention, the protection scope of the present invention is not limited thereto, and changes or replacements within the technical scope disclosed in the present invention are all within the protection scope of the present invention for those skilled in the art. It will be easy to see that it should belong within. Accordingly, the protection scope of the present invention should be based on the protection scope of the claims.

The present invention provides a motion recognition method and apparatus, an electronic device, and a computer readable storage medium, wherein the motion recognition method includes: acquiring a video clip; determining, by the target object, motion characteristic information of the target object based on an object frame in a key frame image in the video clip; determining scenario characteristic information and time series characteristic information corresponding to the target object based on the video clip and the motion characteristic information; and determining an operation type of the target object based on the operation characteristic information, the scenario characteristic information, and the time series characteristic information.

Claims (21)

A gesture recognition method comprising:
acquiring a video clip;
determining, by the target object, motion characteristic information of the target object based on an object frame in a key frame image in the video clip;
determining scenario characteristic information and time series characteristic information corresponding to the target object based on the video clip and the motion characteristic information; and
and determining an operation type of the target object based on the operation characteristic information, the scenario characteristic information, and the time series characteristic information. According to claim 1,
Further comprising the step of determining a target frame in the key frame image, the step of determining the target frame in the key frame image,
selecting a key frame image from the video clip;
performing object detection on the key frame image obtained by screening, so that the target object determines an initial object bounding box in the key frame image; and
and performing expansion on the initial target bounding box according to preset expansion size information, so that the target target obtains the target frame in the key frame image. 3. The method of claim 1 or 2,
determining, by the target object, motion characteristic information of the target object based on an object frame in a key frame image in the video clip,
selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;
obtaining a plurality of target target images corresponding to the key frame images by cutting out partial images from at least partially related images corresponding to the key frame images according to the target frame corresponding to the key frame image; and
and determining motion characteristic information of the target target based on a plurality of target target images corresponding to the key frame image. 4. The method of claim 3,
The step of selecting a plurality of related images corresponding to the key frame image in the video clip comprises:
selecting from the video clip a first sub-video clip comprising a key frame image, wherein the first sub video clip further comprises N images contiguous in time series with the key frame image, where N is a positive integer; ; and
and selecting the plurality of related images from the first sub-video clip. 4. The method of claim 3,
After obtaining a plurality of target target images, before determining the motion characteristic information of the target target,
The method further comprising the step of setting the target target image as an image having a preset image resolution. 6. The method according to any one of claims 1 to 5,
Determining scenario characteristic information and time series characteristic information corresponding to the target target based on the video clip and the motion characteristic information,
selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;
at least partially performing a video scenario feature extraction operation on the associated image to obtain the scenario feature information;
obtaining initial time-series feature information by performing a time-series feature extraction operation on an object other than the target object in the video clip; and
and determining time-series characteristic information corresponding to the target object based on the initial time-series characteristic information and the operation characteristic information. 7. The method of claim 6,
The step of obtaining initial time-series feature information by performing a time-series feature extraction operation on an object other than the target object in the video clip,
for the key frame image, select a second sub video clip including a key frame image in the video clip, the second sub video clip further comprising P images adjacent in time series to the key frame image and P is a positive integer - ;
and extracting a motion characteristic of an object other than the target object from the image in the second sub-video clip, and using the obtained motion characteristic as the initial time-series characteristic information. 8. The method of claim 6 or 7,
The step of determining time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,
performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively;
performing an average pooling operation on the initial time series feature information subjected to dimension reduction; and
and performing a merging operation on the initial time series feature information on which the average pooling operation has been performed and the dimension reduction-processed motion feature information to obtain time series feature information corresponding to the target object. 9. The method of claim 8,
The step of determining time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,
Using the obtained time-series characteristic information corresponding to the target object as new initial time-series characteristic information, and returning to the step of performing dimension reduction processing on the initial time-series characteristic information and the operation characteristic information, respectively motion recognition method. A gesture recognition device comprising:
a video acquisition module configured to acquire a video clip;
a motion characteristic determining module, wherein the target object is configured to determine motion characteristic information of the target object based on an object frame in a key frame image in the video clip;
a scenario time-series characteristic determining module, configured to determine, based on the video clip and the motion characteristic information, scenario characteristic information and time-series characteristic information corresponding to the target object; and
and a motion recognition module configured to determine a motion type of the target object based on the motion characteristic information, the scenario characteristic information, and the time series characteristic information. 11. The method of claim 10,
the operation characteristic determining module is further configured to determine a target frame in the key frame image;
select a key frame image from the video clip;
performing object detection on the key frame image obtained by screening, so that the target object determines an initial object bounding box in the key frame image;
and performing expansion on the initial target bounding box according to preset expansion size information, so that the target target obtains the target frame in the key frame image. 12. The method of claim 10 or 11,
The motion characteristic determining module is configured to: when the target object determines motion characteristic information of the target object based on the target frame in the key frame image in the video clip,
selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;
cutting out partial images from at least partially related images corresponding to the key frame images according to the target frame corresponding to the key frame image, respectively, to obtain a plurality of target target images corresponding to the key frame image;
and determine motion characteristic information of the target target based on a plurality of target target images corresponding to the key frame images. 13. The method of claim 12,
When the operation characteristic determining module selects a plurality of related images corresponding to the key frame image in the video clip,
select from the video clip a first sub-video clip comprising a key frame image, wherein the first sub video clip further comprises N images contiguous in time series with the key frame image, where N is a positive integer;
and select the plurality of associated images from the first sub-video clip. 13. The method of claim 12,
After obtaining a plurality of target object images, and before determining the operation characteristic information of the target object, the operation characteristic determining module is further configured to:
and set the target target image to an image having a preset image resolution. 15. The method according to any one of claims 10 to 14,
When the scenario time series characteristic determination module determines the scenario characteristic information and time series characteristic information corresponding to the target target based on the video clip and the motion characteristic information,
selecting a plurality of related images corresponding to the key frame image from the video clip by using the key frame image;
at least partially performing a video scenario feature extraction operation on the associated image to obtain the scenario feature information;
performing a time-series feature extraction operation on an object other than the target object in the video clip to obtain initial time-series feature information;
and to determine time-series characteristic information corresponding to the target object based on the initial time-series characteristic information and the motion characteristic information. 16. The method of claim 15,
When the scenario time series feature determination module performs a time series feature extraction operation on an object other than the target object in the video clip to obtain initial time series feature information,
for the key frame image, select a second sub video clip including a key frame image in the video clip, the second sub video clip further comprising P images adjacent in time series to the key frame image and P is a positive integer - ;
and extracting a motion characteristic of an object other than the target object from the image in the second sub-video clip, and using the obtained motion characteristic as the initial time-series characteristic information. 17. The method of claim 15 or 16,
When the scenario time series characteristic determination module determines time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,
performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively;
performing an average pooling operation on the initial time series feature information subjected to dimension reduction; and
The apparatus for recognizing motion, characterized in that it is configured to obtain time-series characteristic information corresponding to the target object by performing a merging operation on the initial time-series characteristic information on which the average pooling operation has been performed and the dimensionally-reduced operation characteristic information. 18. The method of claim 17,
When the scenario time series characteristic determination module determines the time series characteristic information corresponding to the target target based on the initial time series characteristic information and the operation characteristic information,
Operation characterized in that it is configured to use the obtained time series characteristic information corresponding to the target object as new initial time series characteristic information, and return to the step of performing dimension reduction processing on the initial time series characteristic information and the operation characteristic information, respectively. recognition device. As an electronic device,
A processor and a storage medium connected to each other, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is operated, the processor executes the machine-readable instructions, An electronic device for executing the gesture recognition method according to claim 9 . A computer readable storage medium comprising:
A computer program is stored in the computer readable storage medium, and when the computer program is operated by a processor, the method for recognizing a motion according to any one of claims 1 to 9 is executed. storage medium. A computer program comprising:
10. A method comprising computer readable code, wherein when the computer readable code is operated in an electronic device, a processor in the electronic device implements the method for recognizing a motion according to any one of claims 1 to 9. computer program to do.

Images