WO2020010979A1 - 手部关键点的识别模型训练方法、识别方法及设备 - Google Patents

手部关键点的识别模型训练方法、识别方法及设备 Download PDF

Info

Publication number
WO2020010979A1
WO2020010979A1 PCT/CN2019/090542 CN2019090542W WO2020010979A1 WO 2020010979 A1 WO2020010979 A1 WO 2020010979A1 CN 2019090542 W CN2019090542 W CN 2019090542W WO 2020010979 A1 WO2020010979 A1 WO 2020010979A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
hand
dimensional
recognition
layer
Prior art date
Application number
PCT/CN2019/090542
Other languages
English (en)
French (fr)
Inventor
易阳
赵世杰
李峰
左小祥
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP19833424.5A priority Critical patent/EP3742332B1/en
Priority to JP2020558521A priority patent/JP7130057B2/ja
Publication of WO2020010979A1 publication Critical patent/WO2020010979A1/zh
Priority to US17/000,844 priority patent/US11989350B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/143—Segmentation; Edge detection involving probabilistic approaches, e.g. Markov random field [MRF] modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/174—Segmentation; Edge detection involving the use of two or more images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • Embodiments of the present application relate to the technical field of gesture recognition, and in particular, to a training method, a recognition method, and a device for a recognition model of a key point of a hand.
  • the key to gesture interaction is the recognition of key points in the hand.
  • a hand keypoint recognition method a developer first constructs a virtual three-dimensional model of the hand through a computer, and uses the virtual image of the virtual three-dimensional model and the three-dimensional coordinate data corresponding to the keypoint of the hand as training samples. Point recognition model.
  • subsequent gesture recognition after inputting the real image of the gesture including the gesture through the camera into the hand keypoint recognition model, the three-dimensional coordinates of each hand keypoint in the real image can be obtained, thereby identifying the gesture therein.
  • the recognition model trained based on the virtual image is used.
  • the accuracy of gesture recognition is low, which affects the accuracy of gesture interaction.
  • the embodiments of the present application provide a recognition model training method, recognition method, and equipment for key points of the hand.
  • a method for training a key point recognition model of a hand includes:
  • a sample virtual image is converted into a simulation image.
  • the sample virtual image is an image generated by 3D modeling, and the sample virtual image includes key point coordinates corresponding to key points of the hand.
  • the simulation image is used for For imitating images collected under real scenes;
  • the hand keypoint recognition model is used to output the hand of the hand in the real image according to the input real image Key point coordinates.
  • a method for identifying key points of a hand includes:
  • the hand keypoint recognition model is obtained by training based on a simulation image and three-dimensional annotation data of the hand image in the simulation image, where
  • the simulation image is generated by a Cycle-GAN model based on a sample virtual image, and the Cycle-GAN model is generated based on training of a sample real image and the sample virtual image.
  • the sample virtual image is an image generated by three-dimensional modeling, and the sample virtual The image contains key point coordinates corresponding to the key points of the hand.
  • a training model recognition device for key points of the hand includes:
  • a conversion module for converting a sample virtual image into a simulation image through a Cycle-GAN model the sample virtual image is an image generated by three-dimensional modeling, and the sample virtual image includes key point coordinates corresponding to key points of the hand,
  • the simulation image is used to imitate an image collected in a real scene;
  • a first extraction module for extracting a hand image in the simulation image
  • a first training module configured to train a hand keypoint recognition model based on the hand image in the simulation image and the keypoint coordinates, and the hand keypoint recognition model is configured to output the handpoint according to an input real image Coordinates of key points of the hand in the real image.
  • a device for identifying key points of a hand includes:
  • An acquisition module configured to acquire a real image, where the real image includes a hand image
  • a second extraction module for extracting the hand image in the real image
  • a recognition module configured to input the hand image into a hand keypoint recognition model to obtain hand keypoint coordinates, and the hand keypoint recognition model is based on a simulation image and three-dimensional annotation data of the hand image in the simulation image
  • the simulation image is generated by the Cycle-GAN model according to the sample virtual image
  • the Cycle-GAN model is generated by training based on the sample real image and the sample virtual image.
  • the sample virtual image is an image generated by three-dimensional modeling.
  • the sample virtual image includes the key point coordinates corresponding to the key points of the hand.
  • a model training device includes a processor and a memory, and the memory stores computer-readable instructions.
  • the processor causes the processor to perform the following steps. :
  • a sample virtual image is converted into a simulation image.
  • the sample virtual image is an image generated by 3D modeling, and the sample virtual image includes key point coordinates corresponding to key points of the hand.
  • the simulation image is used for For imitating images collected under real scenes;
  • the hand keypoint recognition model is used to output the hand of the hand in the real image according to the input real image Key point coordinates.
  • a hand keypoint recognition device includes a processor and a memory, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the The processor performs the following steps:
  • the hand keypoint recognition model is obtained by training based on a simulation image and three-dimensional annotation data of the hand image in the simulation image, where
  • the simulation image is generated by a Cycle-GAN model based on a sample virtual image, and the Cycle-GAN model is generated based on training of a sample real image and the sample virtual image.
  • the sample virtual image is an image generated by three-dimensional modeling, and the sample virtual The image contains key point coordinates corresponding to the key points of the hand.
  • a non-transitory computer-readable storage medium stores computer-readable instructions.
  • the one or more processors execute the following steps:
  • a sample virtual image is converted into a simulation image.
  • the sample virtual image is an image generated by three-dimensional modeling.
  • the sample virtual image includes key point coordinates corresponding to key points of the hand.
  • the hand keypoint recognition model is used to output the hand of the hand in the real image according to the input real image Key point coordinates.
  • a non-transitory computer-readable storage medium stores computer-readable instructions.
  • the one or more processors execute the following steps:
  • the hand keypoint recognition model is obtained by training based on a simulation image and three-dimensional annotation data of the hand image in the simulation image, where
  • the simulation image is generated by a Cycle-GAN model based on a sample virtual image, and the Cycle-GAN model is generated based on training of a sample real image and the sample virtual image.
  • the sample virtual image is an image generated by three-dimensional modeling, and the sample virtual The image contains key point coordinates corresponding to the key points of the hand.
  • FIG. 1 shows a schematic diagram of an implementation environment provided by an embodiment of the present application
  • FIG. 2 is a schematic diagram of training a model based on a virtual image in the related art
  • FIG. 3 is a schematic diagram of a training process of a gesture keypoint recognition model according to an embodiment of the present application
  • FIG. 4 shows a flowchart of a recognition model training method provided by an embodiment of the present application
  • FIG. 5 shows a flowchart of a recognition model training method according to another embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a generator in a Cycle-GAN model according to an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a discriminator in a Cycle-GAN model according to an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a Cycle-GAN model according to an embodiment of the present application.
  • FIG. 9 shows a schematic diagram of using a gesture segmentation neural network to determine a hand confidence region according to an embodiment of the present application.
  • FIG. 10 shows a flowchart of a method for determining a hand confidence region using a gesture segmentation neural network according to an embodiment of the present application
  • FIG. 11 is a schematic diagram of implementing a binarization process according to an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of a hand keypoint recognition model according to an embodiment of the present application.
  • FIG. 13 shows a flowchart of a method for training a keypoint recognition model of a hand according to an embodiment of the present application
  • FIG. 14 shows a flowchart of a method for identifying a key point of a hand according to an embodiment of the present application
  • FIG. 15 is a schematic diagram of the use of the key point recognition result of the hand in different application scenarios.
  • FIG. 16 shows a block diagram of a hand model for identifying key points of a hand provided by an embodiment of the present application
  • FIG. 17 shows a block diagram of a hand keypoint recognition device provided by an embodiment of the present application.
  • FIG. 18 is a schematic structural diagram of a model training device according to an embodiment of the present application.
  • FIG. 19 shows a structural block diagram of a hand keypoint recognition device provided by an exemplary embodiment of the present application.
  • Cycle-GAN model Cycle-Generation Adversarial Networks (Cycle-GAN) model, a deep learning model used to transform data from the original data domain to the target data domain, which can be used for image processing Style conversion.
  • the Cycle-GAN model can be used to output cartoon pictures based on input pictures (pictures are cartoonized), or it can be used to convert A objects in the input pictures to B objects (such as cats to dogs).
  • the Cycle-GAN model in the embodiment of the present application is used to convert a virtual image into a real image, or a real image into a virtual image. Wherein, the virtual image is generated by computer modeling and carries key point coordinates.
  • the Cycle-GAN model is composed of a generator and a discriminator.
  • the generator is used to perform style conversion on the image
  • the discriminator is used to discriminate the original image and the image output by the generator.
  • Convolution layer A layer used to extract features in a convolutional neural network. It is used to extract low-dimensional features from high-dimensional data. It is divided into three operations: convolution operation, activation operation, and pooling operation. section. Among them, when performing a convolution operation, feature extraction is performed using a convolution kernel that has been trained and learned in advance, and when performing an activation operation, an activation function is used to perform activation processing on the feature map obtained from the convolution. Common activation functions include Linear rectification (Rectified, Linear, Unit, ReLU) function, S-shaped (Sigmoid) function and hyperbolic tangent (Tanh) function.
  • convolution operation when performing a convolution operation, feature extraction is performed using a convolution kernel that has been trained and learned in advance
  • an activation function is used to perform activation processing on the feature map obtained from the convolution.
  • Common activation functions include Linear rectification (Rectified, Linear, Unit, ReLU) function, S-shaped (Sigmoid)
  • the feature vector of the convolutional activation output is reduced by the pooling operation (that is, the size of the feature map is reduced), and the overfitting problem is improved at the same time.
  • pooling methods include mean-pooling, max-pooling, and stochastic-pooling.
  • Deconvolution layer In contrast to the role of the convolution layer, the deconvolution layer is used to map low-dimensional features into high-dimensional outputs. In the embodiment of the present application, the deconvolution layer is a layer for upsampling the feature vector, that is, for increasing the size of the feature map.
  • stitching refers to the fusion process of two multi-channel feature maps of the same size, where the number of channels of the stitched feature map is equal to the sum of the number of channels of the two feature maps.
  • Residual layer A layer structure for feature extraction through a residual network (Residual Network, ResNet) structure.
  • ResNet residual Network
  • the residual layer is used to perform feature extraction on the extracted hand image to determine the hand image. Key points.
  • Fully connected layer also known as inner layer.
  • the convolutional neural network after feature extraction, the picture is mapped into the feature space of the hidden layer, and the fully connected layer is used to map the learning and training features to the sample classification space.
  • FIG. 1 illustrates a schematic diagram of an implementation environment provided by an embodiment of the present application.
  • the implementation environment includes at least one terminal 110 and a server 120.
  • the terminal 110 is an electronic device with an image acquisition function.
  • the electronic device may be a smartphone, a tablet computer, a somatosensory game device, or a personal computer equipped with a camera.
  • the camera 111 may be a front camera (located on a side of a mobile phone screen) or a rear camera (located on a side opposite to the mobile phone screen).
  • the terminal 110 when a preset application is started (supporting gesture interaction function), the terminal 110 captures red, green and blue (RGB) pictures through a camera, or captures continuous RGB pictures through a camera Framed video. For example, when a somatosensory game supporting gesture interaction is started, the terminal 110 enables a camera for image acquisition.
  • RGB red, green and blue
  • the terminal 110 and the server 120 are connected through a wired or wireless network.
  • the server 120 is a server cluster or a cloud computing center composed of one server and several servers. In a possible implementation manner, the server 120 is a background server of a preset application program in the terminal 110.
  • the terminal 110 collects the RGB pictures through the camera 111, it sends the RGB pictures to the server 120.
  • the server 120 uses a pre-trained gesture segmentation neural network 121 to extract a hand image from an RGB picture, and then uses a pre-trained hand key point recognition model 122 to identify the key point coordinates of the hand key point in the hand image, so that The key point coordinates determine the gesture indicated by the hand image, and then feed back the gesture recognition result to the terminal 110.
  • the terminal 110 when the terminal 110 has a gesture recognition function, for the collected RGB pictures, the terminal 110 performs hand image extraction and hand key point recognition locally without using the server 120.
  • the terminal 110 or the server 120 implements a gesture recognition function through a neural network chip.
  • the aforementioned wireless network or wired network uses standard communication technologies and / or protocols.
  • the network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, wired, or wireless Network, private network, or any combination of virtual private networks).
  • technologies and / or formats including HyperText Markup Language (HTML), Extensible Markup Language (XML), and the like are used to represent data exchanged over a network.
  • SSL Secure Socket Layer
  • TLS Transport Layer Security
  • VPN Virtual Private Network
  • IPsec Internet Protocol Security
  • customized and / or dedicated data communication technologies may also be used in place of or in addition to the aforementioned data communication technologies.
  • the training method for identifying the key points of the hand may be performed by a model training device, and the model training device may be a computer or a server with strong computing power;
  • the identification method can be executed by a hand key point identification device, and the hand key point identification device can be the terminal 110 or the server 120 in FIG. 1.
  • Three-dimensional gesture recognition based on RGB images refers to determining the key points of the hand in the two-dimensional RGB image, and further obtaining the three-dimensional coordinates of the key points of each hand.
  • hand keypoint recognition models based on deep neural networks are usually used for hand keypoint recognition.
  • the acquisition of hand keypoint coordinates in a real scene requires a large number of sensing devices. Therefore, it is difficult to collect early training samples, which leads to higher difficulty in training the hand keypoint recognition model.
  • a training method for hand keypoint recognition model is proposed.
  • a three-dimensional model of the hand is automatically modeled by means of a computer's graphic rendering function, so that the virtual image 21 and the annotation information 22 (the coordinates automatically generated during modeling) are corresponding to the three-dimensional model of the hand.
  • Data as training samples, respectively training gesture segmentation model 23 (for identifying hand regions in the image), 2D keypoint recognition model 24 (for obtaining heat maps of two-dimensional hand keypoints in the image), and 3D keypoints Recognition model 25 (for obtaining three-dimensional coordinates of three-dimensional hand key points in an image).
  • the labeling information is automatically generated by the computer during the modeling phase, it is possible to reduce the difficulty in obtaining training samples, thereby improving the training efficiency of the model.
  • due to the data distribution gap between the virtual image and the real image collected in the real scene there is a large difference in the representation of details
  • a Cycle-GAN model 27 is trained according to the virtual image 21 and the real image 26 in advance, and Cycle is used.
  • the GAN model 27 converts the virtual image 21 into a simulation image 28, thereby training the gesture segmentation model 23 using the simulation image 28 and the corresponding annotation information 22 as training samples; and, by using the 2D keypoint recognition model and the 3D keypoint recognition model Coupling into a 2D-3D key point coupling recognition model 29 improves the accuracy of the model output.
  • the data of the virtual scene domain is migrated to the real scene data domain, thereby alleviating the problem of the data distribution gap, thereby improving the test results of the model in the real image, and improving the hand key point recognition in the actual application scene Accuracy.
  • the method for recognizing the key points of the hand provided in the embodiments of the present application can be used for other scenarios involving gesture recognition, such as a sign language recognition scene, a gesture interaction scene, and a hand special effect scene.
  • gesture recognition such as a sign language recognition scene, a gesture interaction scene, and a hand special effect scene.
  • the following describes it in combination with different application scenarios.
  • the method of recognizing the key points of the hand can be implemented as sign language recognition software, and the sign language recognition software is installed in the terminal. After the terminal starts the sign language recognition software, the terminal collects the sign language image through the camera, and recognizes the meaning expressed by the sign language image, and displays the recognition result in the form of text, so that it can not be used to communicate with people with disabilities.
  • the hand key point recognition method can be implemented into smart home control software and installed on a smart home device that supports gesture control.
  • the camera collects images containing user gestures and recognizes the gestures in the camera to determine the control instructions corresponding to the user gestures, and then performs corresponding operations according to the control instructions, which is convenient for users to quickly control intelligence through gestures. Household equipment.
  • the hand key point recognition method When the hand key point recognition method is applied to the hand special effect scene, the hand key point recognition method can be implemented into image processing software and installed in the terminal. After the terminal enables the image processing software, it collects hand images through the camera, and superimposes hand special effects on corresponding hand key points according to the hand special effects selected by the user (such as eagle claws, bear paws, etc.) to achieve Hand effects are displayed in real time.
  • the terminal enables the image processing software, it collects hand images through the camera, and superimposes hand special effects on corresponding hand key points according to the hand special effects selected by the user (such as eagle claws, bear paws, etc.) to achieve Hand effects are displayed in real time.
  • FIG. 4 illustrates a flowchart of a recognition model training method provided by an embodiment of the present application.
  • This embodiment is exemplified by applying the training method for identifying key points of a hand to a model training device.
  • the method may include the following steps:
  • a sample virtual image is converted into a simulation image through a Cycle-GAN model.
  • the sample virtual image is an image generated by three-dimensional modeling, and the sample virtual image includes key point coordinates corresponding to the key points of the hand.
  • the model training device trains the Cycle-GAN model according to the sample real image and the sample virtual image.
  • the sample real image is an RGB image containing a hand image collected in a real scene
  • the sample virtual image is an RGB image of a three-dimensional hand model in a virtual environment, which is three-dimensionally modeled by a computer in a virtual environment Constructed, and the sample virtual image carries the key point coordinates of the key points of the hand in the three-dimensional hand model (automatically generated during modeling).
  • the trained Cycle-GAN model is used to transform the sample virtual image into a simulation image, or it is used to transform a real image into a pseudo-virtual image.
  • the key points of the hand include fingertips, distal knuckles (knuckles near the fingertips), proximal knuckles (knuckles near the palm), palm joints, and other key points, such as 21 key points Point; key point coordinates can be three-dimensional coordinates, and / or two-dimensional coordinates.
  • the model training device collects images that include gestures in the real scene As the sample real image set, an existing virtual image carrying label information is used as the sample virtual image set.
  • the model training device inputs the sample virtual image into the Cycle-GAN model, so as to obtain the simulated image output from the Cycle-GAN model. Data domain migration in the scenario.
  • step S402 a hand image in the simulation image is extracted.
  • the model training device further extracts the hand image in the simulation image.
  • the model training device pre-trains a gesture segmentation neural network, and uses the gesture segmentation neural network to extract a hand image in a simulation image.
  • the gesture segmentation neural network can be generated by training on several real images stored with label information (labeling the hand region and the non-hand region).
  • Step S403 training a hand keypoint recognition model according to the hand image and keypoint coordinates in the simulation image, and the hand keypoint recognition model is used to output hand keypoint coordinates of the hand in the real image according to the input real image.
  • the model training device uses the extracted hand image and corresponding key point coordinates as training samples to train a hand key point recognition model. Since the simulation image used as the training sample has the same style (belonging to the same data domain) as the real image, the accuracy of the test using the hand keypoint recognition model of the real image is higher, and the hand keypoint recognition model is used to identify the real image Gestures are more accurate.
  • the hand keypoint recognition model includes a two-dimensional recognition branch and a three-dimensional recognition branch. After the collected real image is input to the hand keypoint recognition model, the two-dimensional recognition branch output two-dimensional is obtained. Key point heat map (high-response points in the heat map are two-dimensional key points), and three-dimensional key point coordinates output by three-dimensional recognition branches.
  • the sample virtual image is converted into a simulation image by using the Cycle-GAN model, so that the hand keypoint recognition model is trained based on the hand image and keypoint coordinates in the simulation image;
  • the simulation image obtained from the GAN model can simulate the style of the real image of the sample, that is, the training sample is closer to the collected real image. Therefore, the accuracy of gesture recognition of the real image using the hand keypoint recognition model is higher;
  • the hand keypoint recognition model is trained based on the sample virtual image carrying the keypoint coordinates, avoiding manual labeling of keypoint coordinates, and further improving model training efficiency and accuracy.
  • the key to training the Cycle-GAN model is the training of the generator and discriminator.
  • the generator in the embodiment of the present application is used to perform style conversion on the image, and the discriminator is used to discriminate the original image and the image output by the generator.
  • the GAN model is a process in which the generator and the discriminator oppose each other, so that the style of the output image of the generator is consistent with the style of the target image (the output approximates the target distribution), and the discriminator is in a random discrimination state (discrimination of the original image and the output image of the generator Equal probability).
  • the following describes the training process of the Cycle-GAN model using a schematic embodiment.
  • FIG. 5 illustrates a flowchart of a recognition model training method according to another embodiment of the present application.
  • This embodiment is exemplified by applying the training method for identifying key points of a hand to a model training device.
  • the method may include the following steps:
  • Step S501 a first generator and a second generator are constructed.
  • the first generator is used to generate a simulation image
  • the second generator is used to generate a pseudo virtual image
  • the pseudo sample virtual image is used to imitate the style of the sample virtual image.
  • the training part of the Cycle-GAN model is completed by the generator.
  • the Cycle-GAN model in the embodiment of the present application is used to perform style conversion on an image, and includes two generators, each of which is a first generator for generating a simulated image based on an input image, and a pseudo-virtual based on the input image.
  • the second generator of the image The input image of the first generator is a (original) virtual image or a pseudo-virtual image output by the second generator, and the input image of the second generator is a (original) real image or a simulation image output by the first generator.
  • the generator may be composed of three parts, which are an encoder 61 and a converter 62 and a decoder 63.
  • the encoder 61 includes several convolutional layers 611 (three convolutional layers are taken as an example in the figure), which are respectively used to perform convolution processing on the input image to extract features of the input image.
  • the convolution layer 611 includes several convolution kernels with a step size of 2.
  • an activation layer (ReLU) or a batch normalization (BN) layer can be added, which is not limited in this embodiment of the present application.
  • the input image After passing through the encoder 61, the input image is compressed into the feature vector of the data domain to which the input images belong, and then the feature vector is converted into the feature vector of the data domain to which the output image belongs through the converter 62.
  • the size of the input image is 256 ⁇ 256 ⁇ 3 (width ⁇ height ⁇ number of channels)
  • a feature map with a size of 64 ⁇ 64 ⁇ 256 is obtained after passing through the encoder 61.
  • the converter 62 is a residual network composed of a plurality of residual blocks 621 (three residual blocks are taken as an example in the figure).
  • each residual block 621 is composed of at least two convolutional layers, and a part of the input of the residual block 621 is directly added to the output, that is, the output of the residual block 621 is composed of the output and input of the convolutional layer. It can be ensured that the input of the previous network layer directly acts on the subsequent network layer, so that the deviation of the corresponding output from the original input is reduced, so that the features of the original image are retained in the output and the output result.
  • the converter 62 is configured to perform data domain conversion on the feature vector output by the encoder 61.
  • the output size of the converter 62 is the same as the input size.
  • the converter 62 converts the 64 ⁇ 64 ⁇ 256 feature map (feature of the virtual image) output by the encoder 61, and outputs the 64 ⁇ 64 ⁇ 256 feature map (feature of the real image) to realize the virtual image data domain to Conversion of real image data fields.
  • the decoding process is used to restore low-level features from the feature vector, and finally obtain a style-transformed image.
  • the decoder 63 includes several layers of deconvolution layers 631 (three layers of deconvolution layers are taken as an example in the figure), which are respectively used to perform deconvolution processing on the feature vectors output by the converter 62 to restore Low-level features.
  • the deconvolution layer 631 includes several convolution kernels with a step size of 1/2.
  • the size of the final output image of the decoder 63 is the same as the size of the input image. For example, when the size of an input image is 256 ⁇ 256 ⁇ 3, the decoder 63 finally outputs an image with a size of 256 ⁇ 256 ⁇ 3 and undergoing style conversion.
  • step S502 a first discriminator and a second discriminator are constructed.
  • the first discriminator is used to discriminate the sample virtual image and the pseudo-virtual image
  • the second discriminator is used to discriminate the sample real image and the simulated image.
  • the adversarial training part of the Cycle-GAN model is done by the discriminator.
  • the Cycle-GAN model in the embodiment of the present application includes two discriminators, a first discriminator for discriminating a sample virtual image and a pseudo-virtual image, and a second discriminator for discriminating a sample real image and a simulated image.
  • the pseudo-virtual image is generated by the second generator according to the sample image
  • the simulation image is generated by the first generator according to the sample virtual image.
  • the discriminator uses a convolutional neural network structure, which contains several convolutional layers for extracting image features and a convolutional layer for generating a one-dimensional output.
  • the convolutional layer 71, the convolutional layer 72, the convolutional layer 73, and the convolutional layer 74 in the discriminator are used to sequentially perform feature extraction on the input image to obtain a feature map 75 of the input image.
  • the convolutional layer 76 is used to generate a one-dimensional output according to the feature map 75, so as to determine whether the input image belongs to a real image or a simulated image according to the one-dimensional output, or whether the input image belongs to a virtual image or a pseudo-virtual image.
  • the discriminator in the embodiment of the present application uses 0/1 discrimination. For example, when the output of the discriminator is 1, it indicates that the input image is a real image, and when the output of the discriminator is 0, it indicates that the input image is a simulation image.
  • the discriminator in the embodiment of the present application does not use 0/1 discrimination, but divides the input image into several patch blocks of the same size, and performs discrimination based on the patch blocks. Compared with 0/1 discrimination, using the patch block-based discrimination method can reduce the amount of training parameters, thereby improving training efficiency.
  • step S503 a Cycle-GAN model is constructed according to the first generator, the second generator, the first discriminator, and the second discriminator.
  • the model training device constructs a Cycle-GAN model.
  • the Cycle-GAN model shown in FIG. 8 includes a first generator 81, a second generator 82, a first discriminator 83, and a second discriminator 84.
  • Step S504 Calculate the loss of the Cycle-GAN model according to the sample real image and the sample virtual image.
  • the loss of the Cycle-GAN model includes a generator loss, a discriminator loss, and a cycle loss.
  • the model training device takes the sample virtual image and the sample real image as the input images of the corresponding generator, the generator performs style conversion on the input image, and the discriminator determines the image output by the generator and The original input image is discriminated, and the generator loss, discriminator loss, and cyclic loss are calculated, so that the model is trained based on the above-mentioned loss and back propagation algorithm.
  • the generator loss is the loss generated when the generator converts the image
  • the discriminator loss is used to indicate the error between the discriminator's discrimination result and the expected result
  • the cyclic loss or Cycle Consistency Loss
  • Reconstruc reconstruction error
  • the input sample real image and sample virtual image need not be paired, that is, the gestures in the sample real image and the sample virtual image need not be consistent.
  • the model training device When training the generator in the Cycle-GAN model shown in FIG. 8, the model training device inputs the sample virtual image A into the first generator 81, and the first generator 81 converts the sample virtual image A into a simulation image B1. Further, the first generator 81 inputs the converted simulation image B1 into the second generator 82, and the second generator 82 converts the simulation image B1 into a pseudo-virtual image A2, and calculates the sample virtual image A and the pseudo-image
  • the reconstruction error RA between the virtual images A2 determines the cyclic loss.
  • the reconstruction error RA is used to indicate the image difference between the sample virtual image A and the original sample virtual image A after the two generator style conversions. The smaller the reconstruction error RA is, the more similar the sample virtual image A and the pseudo-virtual image A2 are.
  • the model training device inputs the sample real image B into the second generator 82, and the second generator 82 converts the sample real image B into a pseudo-virtual image A1. Further, the second generator 82 inputs the converted virtual image A1 into the first generator 81, and the first generator 81 converts the virtual image A1 into a simulated image B2, and calculates the sample real image B and The reconstruction error RB between the simulated images B2 determines the cyclic loss.
  • the reconstruction error RB is used to indicate the image difference between the sample real image B and the original sample real image B after two generator style conversions. Among them, the smaller the reconstruction error RB, the more similar the sample real image B and the simulated image B2 are.
  • the model training device When training the discriminator in the Cycle-GAN model shown in FIG. 8, the model training device inputs the sample virtual image A and the pseudo-virtual image A1 into the first discriminator 83, and the first discriminator 83 performs discrimination, and according to the actual discrimination result And the target discrimination result to calculate the discriminator loss of the first discriminator 83 and the generator loss of the second generator 82; similarly, the model training device inputs the sample real image B and the simulation image B1 to the second discriminator 84, and the second discriminator 84 The discriminator 84 performs discrimination, and calculates a discriminator loss of the second discriminator 84 and a generator loss of the first generator 81 according to the actual discrimination result and the target discrimination result.
  • the model training device constructs a loss function of the Cycle-GAN model according to the generator loss, the discriminator loss, and the loop loss.
  • the multiplication factor (weight) of the cyclic loss in the loss function is greater than the multiplication factor of the discriminator loss.
  • step S505 the Cycle-GAN model is back-trained according to the loss of the Cycle-GAN model.
  • the model training device uses the back-propagation algorithm to train the Cycle-GAN model according to the generator-loss, discriminator loss, and cyclic loss of the Cycle-GAN model.
  • the embodiment of this application does not specifically describe the training of the Cycle-GAN model based on the loss. Way to achieve.
  • the loss function of the Cycle-GAN model finally trained is the smallest.
  • the model training device obtains a Cycle-GAN model for style conversion according to the sample virtual image and the sample real image.
  • the model training device uses the model pair to carry The sample virtual image labeled with information is style converted.
  • step S506 the sample virtual image is converted into a simulation image by a first generator in the Cycle-GAN model.
  • the model training device inputs the sample virtual image into the first generator of the Cycle-GAN model, and the first generator converts the sample virtual image into a simulation image.
  • the model training device first locates the hand confidence region, and then crops out the hand and a small part of the surrounding area according to the hand confidence region, and scales it as the input of the next level. As shown in FIG. 5, the model training device further extracts a hand image from the simulation image through the following steps S507 to S508.
  • step S507 the simulated image is input into a gesture segmentation neural network, and a hand confidence region is output.
  • the gesture segmentation neural network is obtained by training according to an image labeled with a hand region.
  • the gesture segmentation neural network is a Convolutional Neural Networks (CNN) using a U-NET structure, including n-layer convolutional layers and n-layer deconvolutional layers .
  • CNN Convolutional Neural Networks
  • the n-layer convolution layer is used to perform feature extraction at different levels on the image
  • the n-layer deconvolution layer is used to decode and output the extracted features.
  • the gesture segmentation neural network includes three layers of convolutional layers and three layers of deconvolutional layers.
  • determining a hand confidence region through a gesture segmentation neural network includes the following steps.
  • step S507A the n-layer convolution layer is used to process the simulated image to obtain an intermediate feature map corresponding to the simulated image.
  • the model training device inputs the simulated image into the gesture neural segmentation network, and sequentially performs convolution processing on the simulated image from the n convolutional layers, thereby extracting the features in the simulated image and obtaining the intermediate feature map corresponding to the simulated image.
  • the input is a simulation image
  • the input is the feature map output from the i-1th convolution layer.
  • each convolution layer in addition to using a convolution kernel to perform convolution processing on the input image or feature map (the output of the previous convolution layer), each convolution layer also activates the convolution result and performs pooling processing.
  • the ReLU function can be used to activate the convolution result.
  • the embodiment of the present application does not limit the specific activation function used.
  • the convolutional layer After the activation process, the convolutional layer performs pooling on the activation result (feature map), and inputs the pooled feature map to the next convolutional layer.
  • the pooling process is used to reduce the size of the feature map and retain important information in the feature map.
  • the pooling layers at all levels perform maximum pooling processing on the input feature map. For example, when the feature map is pooled with 2 ⁇ 2 blocks and 2 as the step size, the maximum value in the 2 ⁇ 2 blocks in the feature map is extracted to reduce the size of the feature map to a quarter.
  • average pooling or random pooling may also be used for pooling processing, which is not limited in this embodiment.
  • the convolution-pooling process is performed on the first convolution layer to obtain a feature map 92, and the second convolution layer is input.
  • the feature map 92 is obtained after the second convolution layer 92 is subjected to convolution-pooling processing to obtain the feature map 93 and input to the third convolution layer; the feature map 93 is subjected to convolution-pooling at the third convolution layer
  • a further convolution operation is performed to obtain a generated feature map 94, and an n-layer deconvolution layer is input.
  • Step S507B the intermediate feature map is processed by the n-layer deconvolution layer to obtain a target feature map corresponding to the simulation image.
  • the gesture segmentation neural network further deconvolves the intermediate feature map through n layers of deconvolution layers, and finally obtains the target feature map corresponding to the simulation image.
  • the size of the target feature map is equal to the size of the simulation image.
  • this step may include the following steps.
  • the model training device For the feature map output by the m-th deconvolution layer, the model training device performs channel stitching between the feature map and the feature map output by the nm-th convolution layer to obtain a stitched feature map, where the stitched feature map
  • the number of channels is equal to the sum of the number of channels in the feature map before stitching.
  • the feature map output by the nm-th convolution layer is larger than the size of the feature map output by the m-th deconvolution layer, the feature map output by the m-th deconvolution layer also needs to undergo upsampling processing, thereby increasing Feature map size.
  • the feature map is further convolved using a 1 ⁇ 1 convolution check, thereby reducing the dimension of the feature map, reducing the number of channels of the feature map after stitching, and thus reducing the amount of calculation.
  • a 3 ⁇ 3 convolution kernel can also be used for further convolution processing to expand the receptive field of the feature map to eliminate inaccurate edges during the upsampling process. Impact.
  • the stitched feature map is input to the m + 1th layer deconvolution layer, and the m + 1th layer deconvolution layer performs deconvolution activation processing on it.
  • the feature map 94 is stitched with the feature map 92 generated by the second convolution layer to generate a feature map 95 And input the second deconvolution layer; the feature map 95 is stitched with the feature map 91 generated by the first convolution layer after deconvolution-upsampling processing to generate the feature map 96.
  • step S507C a probability matrix is generated according to the target feature map, and the probability matrix is used to represent the probability that each pixel in the simulation image is a hand.
  • the model training device determines a hand confidence region in the simulation image.
  • the model training device performs pixel-level recognition and classification according to the target feature map, and determines the probability that each pixel point corresponds to the hand region, where the higher the probability that the pixel point corresponds to, the pixel point is The higher the probability of the hand area.
  • step S507D the probability matrix is binarized, and a heat map corresponding to the hand region is output.
  • the heat map is used to indicate the hand confidence region in the simulation image.
  • the model training device compares the magnitude relationship between each probability in the probability matrix and the probability threshold. If the probability is greater than the probability threshold, the probability is set to 1, If the probability is less than the probability threshold, the probability is set to 0.
  • the model training device performs a binarization process on the probability matrix 1101 according to a probability threshold of 0.6, and sets a region with a probability greater than 0.6 to 1 and a region with a probability less than 0.6 to 0 to obtain a heat map. 1102.
  • the region with a value of 1 is the hand confidence region.
  • the white area is the hand confidence region
  • the black area is the non-hand confidence region.
  • step S508 a hand image is extracted from the simulation image according to the hand confidence region.
  • the model training device extracts the hand image from the simulation image.
  • step S507D when the hand confidence region is obtained through the binarization process, pixels of some hand regions may be incorrectly divided into non-hand regions, and pixels of some non-hand regions may be divided into hand regions. These error divisions are usually located at the edge of the actual hand region. Therefore, in a possible implementation, the model training device extracts a size slightly larger than the hand from the XY plane of the simulation image according to the coordinates of the confidence region of the hand. Hand image of confidence region.
  • the model training device performs scaling processing on the extracted hand image to ensure that the size of the hand image is consistent with the input size of the hand keypoint recognition model.
  • Step S509 a hand keypoint recognition model is constructed, and the hand keypoint recognition model includes a two-dimensional recognition branch and a three-dimensional recognition branch.
  • the model training device uses a deep learning method in which a two-dimensional heat map is coupled with three-dimensional key point coordinates to train a hand key point recognition model. Because this coupling method is closer to the mechanism by which the human eye processes visual images (three-dimensional Predicts deep stereo information and is located at the apparent information), so the accuracy of key point recognition using the trained model is higher.
  • the hand keypoint recognition model constructed by the model training device includes a two-dimensional recognition branch and a three-dimensional recognition branch, wherein the two-dimensional recognition branch includes an i-layer, a two-dimensional residual layer, and a convolution layer, and the three-dimensional recognition branch includes an i-layer.
  • the 3D residual layer and the fully connected layer, and the 2D residual layer of the previous j layer is coupled to the 3D residual layer of the previous j layer, 2 ⁇ j ⁇ i-1, i and j are integers.
  • the hand keypoint recognition model constructed by the model training device includes a two-dimensional recognition branch 1210 and a three-dimensional recognition branch 1220.
  • the 2D recognition branch 1210 is composed of 5 2D residual blocks (each 2D residual block corresponds to a 2D residual layer) and a fully connected layer 1216.
  • the 3D recognition branch 1220 is composed of 5 3D residual blocks ( Each 3D residual block corresponds to a 3D residual layer) and a convolution layer 1226, and the first 3 layers of the 2D recognition branch 1210 and the 3D recognition branch 1220 are coupled to each other.
  • the residual blocks in the two-dimensional recognition branch and the three-dimensional recognition branch may use commonly used residual blocks, and in some residual blocks, a convolution kernel with a step size of 2 may be used to reduce the feature map dimension to reduce The small calculation comes to expand the receptive field of the feature map, which is not limited in this application.
  • the inputs of the first two-dimensional residual block 1211 and the first three-dimensional residual block 1221 are extracted hand images; the input of the second two-dimensional residual block 1212 is the first two-dimensional residual block 1211. And the output of the first three-dimensional residual block 1221; the input of the second three-dimensional residual block 1222 is the output of the first two-dimensional residual block 1211 and the first three-dimensional residual block 1221; the input of the third two-dimensional residual block 1213 Is the output of the second two-dimensional residual block 1212 and the second three-dimensional residual block 1222; the input of the third three-dimensional residual block 1223 is the output of the second two-dimensional residual block 1212 and the second three-dimensional residual block 1222.
  • Step S510 Calculate the 2D recognition loss and 3D recognition loss of the hand keypoint recognition model according to the hand image and the keypoint coordinates.
  • the 2D recognition loss is the recognition loss of the 2D recognition branch
  • the 3D recognition loss is the 3D recognition branch. Identify losses.
  • the model training device inputs the hand image into the two-dimensional recognition branch and the three-dimensional recognition branch, respectively, and calculates the two-dimensional recognition loss according to the key point coordinates and the two-dimensional recognition branch output recognition result, and recognizes the key point coordinates and the three-dimensional recognition branch output recognition. As a result, three-dimensional recognition loss is calculated.
  • this step includes the following steps.
  • Step S510A input the hand image into a two-dimensional recognition branch and a three-dimensional recognition branch, respectively.
  • the model training device inputs hand images into a first two-dimensional residual block 1211 and a first three-dimensional residual block 1221, respectively.
  • step S510B an addition operation is performed on the feature map output from the k-th two-dimensional residual layer and the feature map output from the k-th three-dimensional residual layer, 1 ⁇ k ⁇ j-1.
  • the model training device For the two-dimensional residual layer and three-dimensional residual layer in which the previous j layers are mutually coupled, the model training device performs an addition operation on the feature map of the k-th layer of the two-dimensional residual layer and the feature map of the k-th layer of the three-dimensional residual layer.
  • the added feature map is used as the input of the k + 1th two-dimensional residual layer and the k + 1th three-dimensional residual layer.
  • the dimensions of the feature maps subjected to the addition operation are the same, and the dimensions of the feature maps remain the same before and after the addition.
  • the model training device performs an element-level addition operation on the feature map output by the first two-dimensional residual block 1211 and the first three-dimensional residual block 1221; and the second two-dimensional residual block 1212 Element-level addition operation is performed with the feature map output by the second three-dimensional residual block 1222.
  • Step S510C input the added feature map into the k + 1th two-dimensional residual layer and the k + 1th three-dimensional residual layer.
  • the model training device inputs the added feature maps into the k + 1th two-dimensional residual layer and the k + 1th three-dimensional residual layer, respectively, and the k + 1th two-dimensional residual layer and the kth layer
  • the +1 layer 3D residual layer performs residual processing on the added feature maps.
  • the model training device repeats the above steps S510B and S510C until the added feature map is input to the j-th layer of the two-dimensional residual layer. And the j-th layer of the three-dimensional residual layer.
  • step S510D the feature map output by the j-th layer of the two-dimensional residual layer is processed by the j + 1th to the i-th two-dimensional residual layers and the convolution layer to obtain a two-dimensional recognition result.
  • the model training device For the feature maps output by the two-dimensional residual layer of the j-th layer, the model training device sequentially performs the residual processing on the feature maps through the two-dimensional residual layers of the j + 1th to the i-th layers, and then the second layer of the i-th layer through the convolution layer.
  • the feature map output by the dimensional residual layer is processed to obtain a two-dimensional recognition result, and the following step S510F is performed.
  • the two-dimensional recognition result is a heat map of two-dimensional key points in the hand image, and the size of the heat map is consistent with the hand image, and the point with high response in the heat image is the hand key point.
  • the two-dimensional recognition branch superimposes and displays the hand key points indicated by the two-dimensional recognition result on the original simulation image, and uses different colors to identify the hand key points corresponding to different fingers.
  • step S510E the feature map output from the j-th three-dimensional residual layer is processed through the j + 1th to i-th three-dimensional residual layers and the fully connected layer to obtain a three-dimensional recognition result.
  • the model training device sequentially performs the residual processing on the feature map through the j + 1-th layer of the three-dimensional residual layer, and then through the fully connected layer
  • the feature map output from the i-th three-dimensional residual layer is processed to obtain a three-dimensional recognition result, and the following step S510G is performed.
  • the three-dimensional recognition result is three-dimensional coordinates of key points of each hand in the hand image, and the three-dimensional coordinates are relative coordinates, that is, preset key points in the hand are used as coordinate origins, and other key points and presets are calculated.
  • Step S510F calculating a two-dimensional recognition loss according to the two-dimensional recognition result and the key point coordinates.
  • the model training device calculates the two-dimensional recognition loss between the two-dimensional coordinates and the key point coordinates (that is, the ground truth) according to the two-dimensional coordinates of the key point of the hand indicated by the two-dimensional recognition result.
  • the key point coordinates are two-dimensional coordinates of the key points of the hand in the sample virtual image, and the two-dimensional recognition loss may be identified by a European distance. For example, when the two-dimensional coordinates of key point A in the two-dimensional recognition result are (100px, 150px) and the actual two-dimensional coordinates of key point A are (90px, 140px), the two-dimensional recognition loss of key point A is
  • Step S510G calculating a three-dimensional recognition loss according to the three-dimensional recognition result and the key point coordinates.
  • the model training device calculates the three-dimensional recognition loss between the three-dimensional coordinate and the key point coordinate (ie, ground truth) according to the three-dimensional coordinates of the key point of the hand indicated by the three-dimensional recognition result.
  • the key point coordinates are three-dimensional coordinates of the key points of the hand in the sample virtual image, and the three-dimensional recognition loss may be identified by using a European distance. For example, when the two-dimensional coordinates of key point A in the three-dimensional recognition result are (100px, 150px, 100px) and the actual three-dimensional coordinates of key point A are (90px, 140px, 100px), the three-dimensional recognition loss of key point A is
  • step S511 the hand keypoint recognition model is back-trained according to the two-dimensional recognition loss and the three-dimensional recognition loss.
  • the model training device is trained by a back-propagation algorithm and the two-dimensional recognition branch of the keypoint recognition model of the hand is trained; according to the calculated three-dimensional recognition loss, the model training device is reversed by The propagation algorithm is used to train the 3D recognition branch of the keypoint recognition model of the hand.
  • the embodiment of the present application does not limit the specific manner of training the hand keypoint recognition model based on the recognition loss.
  • the two-dimensional recognition branch and the three-dimensional recognition branch are coupled, so that the trained hand keypoint recognition model is closer to the mechanism of human eyes processing visual images, thereby improving The accuracy rate of hand keypoint recognition is improved.
  • FIG. 14 shows a flowchart of an identification method provided by an embodiment of the present application.
  • the hand keypoint identification method is applied to the hand keypoint identification device as an example.
  • the method may include the following steps:
  • step S1401 a real image is collected, and the real image includes a hand image.
  • the hand keypoint recognition device is a terminal provided with a camera, and the terminal collects a real image including a hand image through the camera.
  • Step S1402 Extract a hand image from a real image.
  • the hand keypoint recognition device uses the gesture segmentation neural network in the embodiment shown in FIG. 5 to extract a hand image in a real image. This embodiment will not repeat the process of extracting a hand image.
  • Step S1403 input the hand image into the hand keypoint recognition model to obtain the hand keypoint coordinates, and the hand keypoint recognition model is trained based on the simulated image and the 3D annotation data of the hand image in the simulated image, and the simulated image is obtained by Cycle-
  • the GAN model is generated based on the sample virtual image, and the Cycle-GAN model is generated based on the training of the sample real image and the sample virtual image.
  • the sample virtual image is an image generated by 3D modeling, and the sample virtual image contains the key point coordinates corresponding to the key points of the hand.
  • the hand keypoint recognition device inputs the extracted hand image into the hand keypoint recognition model trained in the above embodiment, so as to obtain the hand keypoint coordinates output by the model.
  • the key point coordinates of the hand are two-dimensional coordinates and / or three-dimensional coordinates.
  • the hand key point recognition device when the coordinates of the key points of the hand are two-dimensional coordinates, the hand key point recognition device superimposes and displays the key points of the hand on the two-dimensional coordinates. On the real image, different colors are used to identify key points on the same finger; as shown in (b) of Figure 15, when the hand key point coordinates are three-dimensional coordinates, the hand key point recognition device is based on the three-dimensional coordinates. Construct a three-dimensional hand model in a three-dimensional coordinate system, and use different colors to identify key points on the same finger.
  • the terminal displays a sign language translation result “OK” in the interface according to the OK gesture recognized by a key point of the hand; when applied to a hand special effect scene, the terminal The identified OK gesture simulates a bear's paw to make an OK gesture in the hand area of the interface; when applied to a gesture interaction scenario, the smart home device performs a determination operation based on the identified OK gesture.
  • the Cycle-GAN model is trained according to the sample real image and the sample virtual image in advance, and the sample virtual image is converted into a simulation image by using the Cycle-GAN model, so that according to the hand image in the simulation image And key point coordinates to train the hand key point recognition model; since the simulation image transformed by the Cycle-GAN model can simulate the style of the sample real image, that is, the training sample is closer to the collected real image, so the hand is subsequently used
  • the key point recognition model has a higher accuracy rate for hand gesture recognition on real images; and, the hand key point recognition model is trained based on the sample virtual image carrying the key point coordinates, avoiding manual labeling of key point coordinates, and further improving model training efficiency And accuracy.
  • FIG. 16 shows a block diagram of a recognition model training apparatus provided by an embodiment of the present application.
  • the device has a function of performing the above method example, and the function may be implemented by hardware, and may also be implemented by hardware executing corresponding software.
  • the device may include a conversion module 1610, a first extraction module 1620, and a first training module 1630, where:
  • a conversion module 1610 is configured to convert a sample virtual image into a simulation image through a Cycle-GAN model, the sample virtual image is an image generated by 3D modeling, and the sample virtual image includes key point coordinates corresponding to key points of the hand
  • the simulation image is used to simulate an image collected in a real scene.
  • the first extraction module 1620 is configured to extract a hand image in the simulation image.
  • a first training module 1630 is configured to train a hand keypoint recognition model based on the hand image in the simulation image and the keypoint coordinates, and the hand keypoint recognition model is used to output the The hand keypoint coordinates of the hand in the real image are described.
  • the apparatus further includes: a second training module, configured to train the Cycle-GAN model according to a sample real image and the sample virtual image, where the sample real image is an image collected in a real scene.
  • a second training module configured to train the Cycle-GAN model according to a sample real image and the sample virtual image, where the sample real image is an image collected in a real scene.
  • the second training module is further configured to construct a first generator and a second generator, the first generator is used to generate the simulation image, and the second generator is used to generate a pseudo-virtual image, so The pseudo-virtual image is used to imitate the style of the sample virtual image; a first discriminator and a second discriminator are constructed, the first discriminator is used to discriminate the sample virtual image and the pseudo-virtual image, and the second A discriminator for discriminating the sample real image and the simulation image; constructing the Cycle-GAN model according to the first generator, the second generator, the first discriminator, and the second discriminator ; Calculating the loss of the Cycle-GAN model according to the sample real image and the sample virtual image, the loss of the Cycle-GAN model includes a generator loss, a discriminator loss, and a cycle loss; and according to the Cycle-GAN The loss of the model inversely trains the Cycle-GAN model.
  • the conversion module 1610 is further configured to: convert the sample virtual image into the simulation image through the first generator in the Cycle-GAN model.
  • the first extraction module 1620 is further configured to: input the simulated image into a gesture segmentation neural network and output a hand confidence region, and the gesture segmentation neural network is obtained based on the image labeled with the hand region; and A hand image is extracted from the simulation image according to the hand confidence region.
  • the gesture segmentation neural network includes n layers of convolutional layers and n layers of deconvolutional layers, where n ⁇ 2, where n is an integer; the first extraction module 1620 is further configured to pass the n-layer volume The layer processes the simulation image to obtain an intermediate feature map corresponding to the simulation image; and processes the intermediate feature map through the n-layer deconvolution layer to obtain a target feature map corresponding to the simulation image; Generating a probability matrix according to the target feature map, where the probability matrix is used to represent a probability that each pixel point in the simulation image is a hand; performing a binarization process on the probability matrix, and outputting a corresponding value of the hand region A heat map, which is used to indicate the hand confidence region in the simulation image.
  • the first extraction module 1620 is further configured to: The feature map output by the convolution layer is stitched with the feature map output by the nm-th layer convolution layer, 1 ⁇ m ⁇ n-1; and the stitched feature map is processed by the m + 1th deconvolution layer.
  • the first training module 1630 is further configured to construct the hand keypoint recognition model, where the hand keypoint recognition model includes a two-dimensional recognition branch and a three-dimensional recognition branch, and the two-dimensional recognition branch includes i-layer two-dimensional residual layer and convolution layer, the three-dimensional recognition branch includes i-layer three-dimensional residual layer and fully connected layer, and the former j-layer two-dimensional residual layer is coupled with the former j-layer three-dimensional residual layer, 2 ⁇ j ⁇ i-1, i and j are integers; based on the hand image and the key point coordinates, calculate a two-dimensional recognition loss and a three-dimensional recognition loss of the hand key point recognition model, and the two-dimensional recognition loss Is the recognition loss of the two-dimensional recognition branch, the three-dimensional recognition loss is the recognition loss of the three-dimensional recognition branch; and training the hand key point recognition in reverse according to the two-dimensional recognition loss and the three-dimensional recognition loss. model.
  • the first training module 1630 is further configured to: The hand image is input to the two-dimensional recognition branch and the three-dimensional recognition branch, respectively; and the feature map output by the k-th layer of the two-dimensional residual layer and the feature map output by the k-th layer of the three-dimensional residual layer are added, 1 ⁇ k ⁇ j-1; input the added feature map into the k + 1th two-dimensional residual layer and the k + 1th three-dimensional residual layer; through the j + 1th to i-th two-dimensional residuals Layer and the convolution layer process the feature map output by the j-th layer of the two-dimensional residual layer to obtain a two-dimensional recognition result; through the j + 1 to the i-th three-dimensional residual layer and the fully connected layer Process the feature map output from the j-th 3D residual layer to obtain a 3D recognition result; calculate the 2D recognition loss according to:
  • FIG. 17 illustrates a block diagram of an identification device provided by an embodiment of the present application.
  • the device has a function of performing the above method example, and the function may be implemented by hardware, and may also be implemented by hardware executing corresponding software.
  • the device may include a collection module 1710, a second extraction module 1720, and a recognition module 1730, where:
  • a collection module 1710 is configured to collect a real image, where the real image includes a hand image.
  • a second extraction module 1720 is configured to extract the hand image in the real image.
  • a recognition module 1730 configured to input the hand image into a hand keypoint recognition model to obtain hand keypoint coordinates, and the hand keypoint recognition model is based on a simulation image and a three-dimensional annotation of the hand image in the simulation image Obtained from data training, the simulation image is generated by the Cycle-GAN model based on the sample virtual image, and the Cycle-GAN model is generated based on the training of the sample real image and the sample virtual image.
  • the sample virtual image is an image generated by 3D modeling.
  • the sample virtual image includes key point coordinates corresponding to key points of the hand.
  • FIG. 18 shows a schematic structural diagram of a model training device provided by an embodiment of the present application.
  • the model training device is configured to implement a method for identifying a key point of a hand provided by the foregoing embodiment. Specifically:
  • the model training device 1800 includes a central processing unit (CPU) 1801, a system memory 1804 including a random access memory (RAM) 1802 and a read-only memory (ROM) 1803, and a system bus connecting the system memory 1804 and the central processing unit 1801. 1805.
  • the model training device 1800 also includes a basic input / output system (I / O system) 1806 that helps transfer information between various devices in the computer, and a large-scale storage system for storing the operating system 1813, application programs 1814, and other program modules 1815.
  • the basic input / output system 1806 includes a display 1808 for displaying information and an input device 1809 such as a mouse, a keyboard, or the like for a user to input information.
  • the display 1808 and the input device 1809 are both connected to the central processing unit 1801 through an input-output controller 1810 connected to the system bus 1805.
  • the basic input / output system 1806 may further include an input / output controller 1810 for receiving and processing input from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus.
  • the input-output controller 1810 also provides output to a display screen, printer, or other type of output device.
  • the mass storage device 1807 is connected to the central processing unit 1801 through a mass storage controller (not shown) connected to the system bus 1805.
  • the mass storage device 1807 and its associated computer-readable medium provide non-volatile storage for the model training device 1800. That is, the mass storage device 1807 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.
  • the computer-readable media may include computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.
  • Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory, or other solid-state storage technologies, CD-ROM, DVD or other optical storage, tape cartridges, magnetic tape, disk storage, or other magnetic storage devices.
  • RAM random access memory
  • ROM read-only memory
  • EPROM Erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • flash memory or other solid-state storage technologies
  • CD-ROM, DVD or other optical storage CD-ROM, DVD or other optical storage
  • tape cartridges magnetic tape
  • disk storage or other magnetic storage devices.
  • the above-mentioned system memory 1804 and mass storage device 1807 may be collectively referred to as a memory.
  • the model training device 1800 may also be operated by a remote computer connected to a network through a network such as the Internet. That is, the model training device 1800 may be connected to the network 1812 through the network interface unit 1811 connected to the system bus 1805, or the network interface unit 1811 may also be used to connect to other types of networks or remote computer systems.
  • the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is configured to be executed by one or more processors to implement the above.
  • FIG. 19 shows a structural block diagram of a hand keypoint recognition device 1000 provided by an exemplary embodiment of the present application.
  • the hand keypoint identification device 1000 may be a portable mobile terminal, such as: a smartphone, a tablet, an MP3 player (Moving Picture Experts Group, Audio Layer III, Motion Picture Expert Compression Standard Audio Level 3), and MP4 (Moving Picture Experts Group) Audio Layer IV, Motion Picture Expert Compression Standard Audio Level 4) Player.
  • the hand keypoint recognition device 1000 includes a processor 1001 and a memory 1002.
  • the processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like.
  • the processor 1001 may use at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). achieve.
  • the processor 1001 may also include a main processor and a co-processor.
  • the main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit).
  • the co-processor is Low-power processor for processing data in standby.
  • the processor 1001 may be integrated with a GPU (Graphics Processing Unit).
  • the GPU is responsible for rendering and drawing content needed to be displayed on the display screen.
  • the processor 1001 may further include an AI (Artificial Intelligence) processor, and the AI processor is configured to process computing operations related to machine learning.
  • AI Artificial Intelligence
  • the AI processor is a neural network processor (chip) having a gesture recognition function.
  • the memory 1002 may include one or more computer-readable storage media, which may be tangible and non-transitory.
  • the memory 1002 may further include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash storage devices.
  • the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction for execution by the processor 1001 to implement the key points of the hand provided in this application. recognition methods.
  • the hand keypoint identification device 1000 may optionally include a peripheral device interface 1003 and at least one peripheral device.
  • the peripheral device includes at least one of a radio frequency circuit 1004, a touch display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008, and a power supply 1009.
  • the peripheral device interface 1003 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002.
  • the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1001, the memory 1002, and the peripheral device interface 1003 or Both can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
  • the radio frequency circuit 1004 is used to receive and transmit an RF (Radio Frequency) signal, also called an electromagnetic signal.
  • the radio frequency circuit 1004 communicates with a communication network and other communication devices through electromagnetic signals.
  • the radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals.
  • the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.
  • the radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol.
  • the wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, mobile communication networks (2G, 3G, 4G, and 5G) of various generations, wireless local area networks, and / or WiFi (Wireless Fidelity) networks.
  • the radio frequency circuit 1004 may further include NFC (Near Field Communication) circuits, which are not limited in this application.
  • the touch display screen 1005 is used to display a UI (User Interface).
  • the UI may include graphics, text, icons, videos, and any combination thereof.
  • the touch display screen 1005 also has the ability to collect touch signals on or above the surface of the touch display screen 1005.
  • the touch signal can be input to the processor 1001 as a control signal for processing.
  • the touch display 1005 is used to provide a virtual button and / or a virtual keyboard, which is also called a soft button and / or a soft keyboard.
  • the touch display screen 1005 may be one, and the front panel of the hand key point identification device 1000 is provided; in other embodiments, the touch display screen 1005 may be at least two, respectively provided at the hand key point.
  • the different surfaces of the identification device 1000 may be in a folded design; in still other embodiments, the touch display screen 1005 may be a flexible display disposed on a curved surface or a folded surface of the hand keypoint identification device 1000. Furthermore, the touch display screen 1005 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen.
  • the touch display 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
  • the camera assembly 1006 is used for capturing images or videos.
  • the camera assembly 1006 includes a front camera and a rear camera.
  • a front camera is used to implement a video call or a selfie
  • a rear camera is used to implement a photo or video shooting.
  • the camera assembly 1006 may further include a flash.
  • the flash can be a monochrome temperature flash or a dual color temperature flash.
  • a dual color temperature flash is a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
  • the audio circuit 1007 is used to provide an audio interface between the user and the hand keypoint identification device 1000.
  • the audio circuit 1007 may include a microphone and a speaker.
  • the microphone is used for collecting sound waves of the user and the environment, and converting the sound waves into electrical signals and inputting them to the processor 1001 for processing, or inputting to the radio frequency circuit 1004 to implement voice communication.
  • the microphone can also be an array microphone or an omnidirectional acquisition microphone.
  • the speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves.
  • the speaker can be a traditional film speaker or a piezoelectric ceramic speaker.
  • the speaker When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for ranging purposes.
  • the audio circuit 1007 may further include a headphone jack.
  • the positioning component 1008 is used to locate the current geographic position of the hand keypoint identification device 1000 to implement navigation or LBS (Location Based Service).
  • the positioning component 1008 may be a positioning component based on a US-based GPS (Global Positioning System), a Beidou system in China, or a Galileo system in Russia.
  • the power supply 1009 is used to power various components in the hand keypoint identification device 1000.
  • the power source 1009 may be AC power, DC power, a disposable battery, or a rechargeable battery.
  • the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery.
  • the wired rechargeable battery is a battery charged through a wired line
  • the wireless rechargeable battery is a battery charged through a wireless coil.
  • the rechargeable battery can also be used to support fast charging technology.
  • the hand keypoint recognition device 1000 further includes one or more sensors 1010.
  • the one or more sensors 1010 include, but are not limited to, an acceleration sensor 1011, a gyro sensor 1012, a pressure sensor 1013, a fingerprint sensor 1014, an optical sensor 1015, and a proximity sensor 1016.
  • the acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the hand keypoint recognition device 1000.
  • the acceleration sensor 1011 may be used to detect components of the acceleration of gravity on three coordinate axes.
  • the processor 1001 may control the touch display screen 1005 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1011.
  • the acceleration sensor 1011 may also be used for collecting motion data of a game or a user.
  • the gyro sensor 1012 can detect the body direction and rotation angle of the hand key point identification device 1000, and the gyro sensor 1012 can cooperate with the acceleration sensor 1011 to collect the 3D motion of the user's hand key point identification device 1000. Based on the data collected by the gyro sensor 1012, the processor 1001 can implement the following functions: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
  • motion sensing such as changing the UI according to the user's tilt operation
  • image stabilization during shooting
  • game control game control
  • inertial navigation inertial navigation
  • the pressure sensor 1013 may be disposed on a side frame of the hand keypoint recognition device 1000 and / or a lower layer of the touch display screen 1005.
  • the pressure sensor 1013 When the pressure sensor 1013 is disposed on the side frame of the hand key point recognition device 1000, the user can detect a grip signal of the hand key point recognition device 1000, and perform left-right hand recognition or quick operation according to the grip signal.
  • the pressure sensor 1013 When the pressure sensor 1013 is disposed on the lower layer of the touch display screen 1005, it is possible to control the operable controls on the UI interface according to the pressure operation of the user on the touch display screen 1005.
  • the operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
  • the fingerprint sensor 1014 is used to collect a user's fingerprint to identify the user's identity based on the collected fingerprint. When the user's identity is identified as trusted, the processor 1001 authorizes the user to perform related sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, paying and changing settings.
  • the fingerprint sensor 1014 may be provided on the front, back, or side of the hand keypoint recognition device 1000. When a physical key or a manufacturer's logo is set on the hand keypoint recognition device 1000, the fingerprint sensor 1014 can be integrated with the physical key or the manufacturer's logo.
  • the optical sensor 1015 is used to collect ambient light intensity.
  • the processor 1001 may control the display brightness of the touch display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the intensity of ambient light is high, the display brightness of touch screen 1005 is increased; when the intensity of ambient light is low, the display brightness of touch screen 1005 is reduced.
  • the processor 1001 may also dynamically adjust the shooting parameters of the camera component 1006 according to the ambient light intensity collected by the optical sensor 1015.
  • the proximity sensor 1016 also called a distance sensor, is usually disposed on the front of the hand keypoint recognition device 1000.
  • the proximity sensor 1016 is used to collect the distance between the user and the front of the hand keypoint recognition device 1000.
  • the processor 1001 controls the touch display screen 1005 to switch from the bright screen state to the closed screen state;
  • the processor 1001 controls the touch display screen 1005 to switch from the on-screen state to the on-screen state.
  • FIG. 10 does not constitute a limitation of the key point recognition device 1000 of the opponent, and may include more or fewer components than shown in the figure, or combine some components, or use different Component placement.
  • a model training device which includes a memory and a processor.
  • the memory stores a computer program.
  • the processor is caused to execute the steps of the above-mentioned recognition model training method.
  • the steps of the recognition model training method herein may be the steps in the recognition model training method of each of the above embodiments.
  • a computer-readable storage medium which stores a computer program, and when the computer program is executed by a processor, the processor causes the processor to execute the steps of the foregoing recognition model training method.
  • the steps of the recognition model training method herein may be the steps in the recognition model training method of each of the above embodiments.
  • a hand keypoint recognition device which includes a memory and a processor.
  • the memory stores a computer program.
  • the processor is caused to execute the steps of the foregoing identification method.
  • the steps of the identification method herein may be the steps in the identification methods of the foregoing embodiments.
  • a computer-readable storage medium storing a computer program.
  • the processor is caused to execute the steps of the foregoing identification method.
  • the steps of the identification method herein may be the steps in the identification methods of the foregoing embodiments.
  • the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), solid state hard drives (SSD, Solid State Drives), or optical discs.
  • the random access memory may include resistive random access memory (ReRAM, Resistance, Random Access Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory).
  • ReRAM resistive random access memory
  • DRAM Dynamic Random Access Memory

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

一种手部关键点的识别模型训练方法，包括：通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，样本虚拟图像是三维建模生成的图像，且样本虚拟图像包含手部关键点对应的关键点坐标，仿真图像用于模仿真实场景下采集的图像；提取仿真图像中的手部图像；及根据仿真图像中的手部图像以及关键点坐标，训练手部关键点识别模型，手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。

Description

手部关键点的识别模型训练方法、识别方法及设备
本申请要求于2018年07月10日提交中国专利局，申请号为201810752953.1、发明名称为“手部关键点的识别模型训练方法、识别方法及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及手势识别技术领域，特别涉及一种手部关键点的识别模型训练方法、识别方法及设备。
背景技术
随着人工智能技术的不断成熟，越来越多的应用场景开始支持人机交互，而手势交互则是一种常见的人机交互方式。
手势交互的关键在于手部关键点识别。在一种手部关键点识别方式中，开发人员首先通过计算机构建手部的虚拟三维模型，并将虚拟三维模型的虚拟图像以及对应手部关键点的三维坐标数据作为训练样本，训练手部关键点识别模型。后续进行手势识别时，将通过摄像头采集的包含手势的真实图像输入手部关键点识别模型后，即可得到真实图像中各个手部关键点的三维坐标，从而识别出其中的手势。
然而，采用上述方法进行手势识别时，由于虚拟图像与真实图像的细节存在较大差异，比如，虚拟图像与真实图像中的手部皮肤以及背景存在差异，因此利用基于虚拟图像训练出的识别模型进行手势识别的准确率较低，影响手势交互的准确性。
发明内容
本申请实施例提供了一种手部关键点的识别模型训练方法、识别方法及 设备。
一种手部关键点的识别模型训练方法，所述方法包括：
通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像；
提取所述仿真图像中的手部图像；及
根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
一种手部关键点的识别方法，所述方法包括：
采集真实图像，所述真实图像中包含手部图像；
提取所述真实图像中的所述手部图像；及
将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
一种手部关键点的识别模型训练装置，所述装置包括：
转化模块，用于通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像；
第一提取模块，用于提取所述仿真图像中的手部图像；及
第一训练模块，用于根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
一种手部关键点的识别装置，所述装置包括：
采集模块，用于采集真实图像，所述真实图像中包含手部图像；
第二提取模块，用于提取所述真实图像中的所述手部图像；及
识别模块，用于将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
一种模型训练设备，所述模型训练设备包括处理器和存储器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行以下步骤：
通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像；
提取所述仿真图像中的手部图像；及
根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
一种手部关键点识别设备，所述手部关键点识别设备包括处理器和存储器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行以下步骤：
采集真实图像，所述真实图像中包含手部图像；
提取所述真实图像中的所述手部图像；及
将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注 数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
一种非易失性的计算机可读存储介质，存储有计算机可读指令，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行以下步骤：
通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像；
提取所述仿真图像中的手部图像；及
根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
一种非易失性的计算机可读存储介质，存储有计算机可读指令，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行以下步骤：
采集真实图像，所述真实图像中包含手部图像；
提取所述真实图像中的所述手部图像；及
将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请 的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1示出了本申请一个实施例提供的实施环境的示意图；
图2是相关技术中根据虚拟图像训练模型的原理示意图；
图3是本申请实施例提供的手势关键点识别模型训练过程的原理示意图；
图4示出了本申请一个实施例提供的识别模型训练方法的流程图；
图5示出了本申请另一个实施例提供的识别模型训练方法的流程图；
图6示出了本申请一个实施例提供的Cycle-GAN模型中生成器的结构示意图；
图7示出了本申请一个实施例提供的Cycle-GAN模型中判别器的结构示意图；
图8示出了本申请一个实施例提供的Cycle-GAN模型的结构示意图；
图9示出了本申请一个实施例提供的利用手势分割神经网络确定手部置信区域的原理示意图；
图10示出了本申请一个实施例提供的利用手势分割神经网络确定手部置信区域过程的方法流程图；
图11示出了本申请一个实施例提供的二值化处理的实施示意图；
图12示出了本申请一个实施例提供的手部关键点的识别模型的结构示意图；
图13示出了本申请一个实施例提供的训练手部关键点的识别模型的方法流程图；
图14示出了本申请一个实施例提供的手部关键点的识别方法的流程图；
图15是不同应用场景下手部关键点识别结果用途的示意图；
图16示出了本申请一个实施例提供的手部关键点的识别模型训练装置的框图；
图17示出了本申请一个实施例提供的手部关键点的识别装置的框图；
图18示出了本申请一个实施例提供的模型训练设备的结构示意图；
图19示出了本申请一个示例性实施例提供的手部关键点识别设备的结构框图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
为了方便理解，下面对本申请实施例中涉及的名词进行说明。
Cycle-GAN模型：也就是循环生成对抗网络(Cycle Generative Adversarial Networks，Cycle-GAN)模型，一种用于将数据从原始数据域转化到目标数据域的深度学习模型，可以被用于对图像进行风格转换。比如，Cycle-GAN模型可以用于根据输入图片输出卡通图片(图片卡通化)，或者，用于将输入图片中的A物体转化为B物体(比如将猫转化成狗)。本申请实施例中的Cycle-GAN模型用于将虚拟图像转化为真实图像，或者，将真实图像转化为虚拟图像。其中，该虚拟图像由计算机建模生成，且携带有关键点坐标。
Cycle-GAN模型由生成器(Generator)和判别器(Discriminator)构成，其中，生成器用于对图像进行风格转换，判别器用于判别原始图像和生成器输出的图像。
卷积(convolution)层：卷积神经网络中用于提取特征的层，用于对高维数据进行低维特征提取，分为卷积操作、激活(activation)操作、池化(pooling)操作三部分。其中，进行卷积操作时，使用预先经过训练学习得 到的卷积核进行特征提取，进行激活操作时，使用激活函数(activation function)对卷积得到的特征图进行激活处理，常用的激活函数包括线性整流(Rectified Linear Unit，ReLU)函数、S型(Sigmoid)函数和双曲正切(Tanh)函数。经过卷积激活层之后，通过池化操作降低卷积激活输出的特征向量(即缩小特征图的尺寸)，同时改善过拟合问题。常用的池化方式包括平均池化(mean-pooling)、最大池化(max-pooling)和随机池化(stochastic-pooling)。
反卷积(deconvolution)层：与卷积层的作用相反，反卷积层用于将低维特征映射成高维输出。本申请实施例中，反卷积层是用于对特征向量进行上采样的层，即用于增大特征图的尺寸。
拼接：本申请实施例中，拼接是指将两张同尺寸的多通道特征图进行融合过程，其中，拼接后特征图的通道数等于拼接两张特征图的通道数之和。
残差层：一种通过残差网络(Residual Network，ResNet)结构进行特征提取的层结构，本申请实施例中，残差层用于对提取出的手部图像进行特征提取，确定手部图像中的关键点。
全连接层：又称为内积层。在卷积神经网络中，经过特征提取后，图片被映射到隐藏层的特征空间中，全连接层则是用于将学习训练得到的特征映射到样本分类空间中。
请参考图1，其示出了本申请一个实施例提供的实施环境的示意图。该实施环境中包括至少一个终端110和服务器120。
终端110是具有图像采集功能的电子设备，该电子设备可以为配置有摄像头的智能手机、平板电脑、体感游戏设备或个人计算机等等。比如，如图1所示，当终端110为配置有摄像头111的手机时，该摄像头111可以为前置摄像头(位于手机屏幕一侧)或后置摄像头(位于手机屏幕背对侧)。
在一种可能的使用场景下，当启动预设应用程序(支持手势交互功能)时，终端110即通过摄像头采集红绿蓝(Red Green Blue，RGB)图片，或，通过摄像头采集由连续RGB图片帧构成的视频。比如，当启动支持手势交互 的体感游戏时，终端110即启用摄像头进行图像采集。
终端110与服务器120之间通过有线或无线网络相连。
服务器120是一台服务器、若干台服务器构成的服务器集群或云计算中心。在一种可能的实施方式中，服务器120是终端110中预设应用程序的后台服务器。
在一种可能的应用场景下，如图1所示，终端110通过摄像头111采集到RGB图片后，即将RGB图片发送至服务器120。服务器120通过预先训练的手势分割神经网络121，提取RGB图片中的手部图像，然后通过预先训练的手部关键点识别模型122，识别手部图像中手部关键点的关键点坐标，从而根据关键点坐标确定手部图像指示的手势，进而将手势识别结果反馈给终端110。
在另一种可能的应用场景下，当终端110具备手势识别功能时，对于采集到的RGB图片，终端110即在本地进行手部图像提取和手部关键点识别，而无需借助服务器120。
可选的，终端110或服务器120通过神经网络芯片实现手势识别功能。
可选地，上述的无线网络或有线网络使用标准通信技术和/或协议。网络通常为因特网、但也可以是任何网络，包括但不限于局域网(Local Area Network，LAN)、城域网(Metropolitan Area Network，MAN)、广域网(Wide Area Network，WAN)、移动、有线或者无线网络、专用网络或者虚拟专用网络的任何组合)。在一些实施例中，使用包括超文本标记语言(Hyper Text Mark-up Language，HTML)、可扩展标记语言(Extensible Markup Language，XML)等的技术和/或格式来代表通过网络交换的数据。此外还可以使用诸如安全套接字层(Secure Socket Layer，SSL)、传输层安全(Transport Layer Security，TLS)、虚拟专用网络(Virtual Private Network，VPN)、网际协议安全(Internet Protocol Security，IPsec)等常规加密技术来加密所有或者一些链路。在另一些实施例中，还可以使用定制和/或专用数据通信技术取代或者补充上述数据通信技术。
本申请各个实施例提供的手部关键点的识别模型训练方法可以由模型训练设备执行，该模型训练设备可以为具有较强计算能力的计算机或服务器；本申请各个实施例提供的手部关键点的识别方法可以由手部关键点识别设备执行，该手部关键点识别设备可以为图1中的终端110或服务器120。
基于RGB图像的三维手势识别是指确定二维RGB图像中的手部关键点，并进一步获取各个手部关键点的三维坐标。相关技术中，通常采用基于深度神经网络的手部关键点识别模型进行手部关键点识别。然而，在手部关键点识别模型训练阶段，由于需要大量携带标注信息(手部关键点的坐标)的RGB图像作为训练样本，而在真实场景下采集手部关键点坐标需要使用大量传感设备，因此前期训练样本的采集难度较高，进而导致手部关键点识别模型的训练难度较高。
为了降低手部关键点识别模型的训练难度，提出了一种手部关键点识别模型训练方法。如图2所示，在该训练方法中，首先借助计算机的图形渲染功能自动建模手部三维模型，从而将手部三维模型对应的虚拟图像21以及标注信息22(建模时自动生成的坐标数据)作为训练样本，分别训练手势分割模型23(用于识别图像中的手部区域)、2D关键点识别模型24(用于得到图像中二维手部关键点的热图)、3D关键点识别模型25(用于得到图像中三维手部关键点的三维坐标)。
采用上述方法，由于标注信息在建模阶段由计算机自动生成，因此能够降低训练样本的获取难度，从而提高模型的训练效率。但是由于虚拟图像与真实场景下采集的真实图像之间存在数据分布鸿沟(细节表现方面存在较大差异)，比如，真实图像与虚拟图像中的手部皮肤细节以及背景细节等存在较大差异，因此利用真实图像26对模型(虚拟图像训练得到)进行测试的测试效果较差；并且，由于实际应用场景下，该模型的输入均为真实图像，导致采用该模型进行手部关键点识别的准确率较低。
为了解决数据分布鸿沟带来的问题，如图3所示，本申请实施例提供的 手势关键点识别模型训练方法中，预先根据虚拟图像21和真实图像26训练Cycle-GAN模型27，并利用Cycle-GAN模型27将虚拟图像21转化为仿真图像28，从而将仿真图像28以及对应的标注信息22作为训练样本，训练手势分割模型23；并且，通过将2D关键点识别模型和3D关键点识别模型耦合成2D-3D关键点耦合识别模型29，提高模型输出的准确率。借助Cycle-GAN模型27将虚拟场景域的数据迁移至真实场景数据域，从而缓解数据分布鸿沟的问题，从而提高模型在真实图像下的测试结果，并提高了实际应用场景下手部关键点识别的准确率。
本申请实施例中提供的手部关键点的识别方法可以用于手语识别场景、手势交互场景、手部特效场景等其他涉及手势识别的场景。下面结合不同的应用场景进行说明。
手语识别场景：
当手部关键点的识别方法应用于手语识别场景时，可以将手部关键点的识别方法实现成为手语识别软件，并将手语识别软件安装在终端中。终端启动手语识别软件后，终端即通过摄像头采集手语图像，并对手语图像所表达的含义进行识别，以文字的形式展示识别结果，方便不会手语的用于与残障人士进行沟通。
手势交互场景：
当手部关键点的识别方法应用于手势交互场景时，可以将手部关键点的识别方法实现成为智能家居控制软件并安装在支持手势控制的智能家居设备。智能家居设备运行过程中，即通过摄像头采集包含用户手势的图像，并对其中的手势进行识别，从而确定用户手势对应的控制指令，进而根据控制指令执行相应的操作，方便用户通过手势快速控制智能家居设备。
手部特效场景：
当手部关键点的识别方法应用于手部特效场景时，可以将手部关键点的识别方法实现成为图像处理软件并安装在终端中。终端启用该图像处理软件 后，通过摄像头采集用于手部图像，并根据用户选择的手部特效(比如鹰爪、熊掌等等)，将手部特效叠加在相应的手部关键点，实现手部特效实时显示。
请参考图4，其示出了本申请一个实施例提供的识别模型训练方法的流程图。本实施例以该手部关键点的识别模型训练方法应用于模型训练设备来举例说明，该方法可以包括以下几个步骤：
步骤S401，通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，样本虚拟图像是三维建模生成的图像，且样本虚拟图像包含手部关键点对应的关键点坐标，仿真图像用于模仿真实场景下采集的图像。
可选的，在使用Cycle-GAN模型进行图像转化前，模型训练设备根据样本真实图像和样本虚拟图像训练Cycle-GAN模型。
其中，样本真实图像是在真实场景下采集的包含手部图像的RGB图像；样本虚拟图像是虚拟环境中三维手部模型的RGB图像，该三维手部模型由计算机通过三维建模在虚拟环境中构建，且样本虚拟图像携带有三维手部模型中手部关键点的关键点坐标(建模时自动生成)。训练得到的Cycle-GAN模型用于将样本虚拟图像转化为仿真图像，或者，用于将真实图像转化为仿虚拟图像。
可选的，该手部关键点包括指尖、远端指关节(靠近指尖的指关节)、近端指关节(靠近手掌的指关节)、掌关节等多个关键点，比如21个关键点；关键点坐标可以为三维坐标，和/或，二维坐标。
由于训练Cycle-GAN模型时无需使用一一对应的图像对，即样本真实图像与样本虚拟图像无需一一对应，因此在一种可能的实施方式中，模型训练设备采集现实场景中包含手势的图像作为样本真实图像集，采用已有的携带标注信息的虚拟图像作为样本虚拟图像集。
为了缓解真实图像与虚拟图像之间数据分布鸿沟的问题，模型训练设备将样本虚拟图像输入Cycle-GAN模型，从而得到Cycle-GAN模型输出的仿真图像，实现数据由虚拟场景下的数据域向真实场景下的数据域迁移。
其中，样本虚拟图像经过风格转化得到仿真图像后，其对应的关键点坐标仍旧保留。
步骤S402，提取仿真图像中的手部图像。
由于样本虚拟图像中除了包含手部图像外，还可能包含躯干图像、背景图像等干扰因素，为了提高构建模型的质量，模型训练设备进一步提取仿真图像中的手部图像。
在一种可能的实施方式中，模型训练设备预先训练手势分割神经网络，并利用该手势分割神经网络提取仿真图像中的手部图像。其中，该手势分割神经网络可以根据若干张存储有标签信息(标注手部区域和非手部区域)的真实图像训练生成。
步骤S403，根据仿真图像中的手部图像以及关键点坐标，训练手部关键点识别模型，手部关键点识别模型用于根据输入的真实图像输出真实图像中手部的手部关键点坐标。
进一步的，模型训练设备将提取到的手部图像以及对应的关键点坐标作为训练样本，训练手部关键点识别模型。由于作为训练样本的仿真图像与真实图像的风格相同(属于相同数据域)，因此利用真实图像对手部关键点识别模型进行测试的准确性更高，且利用该手部关键点识别模型识别真实图像中的手势的准确性更高。
在一种可能的实施方式中，该手部关键点识别模型包括二维识别分支和三维识别分支，采集到的真实图像输入手部关键点识别模型后，即得到二维识别分支输出的二维关键点热图(热图中响应高的点即为二维关键点)，以及三维识别分支输出的三维关键点坐标。
综上所述，本实施例中，通过利用Cycle-GAN模型将样本虚拟图像转化为仿真图像，从而根据仿真图像中的手部图像以及关键点坐标训练手部关键点识别模型；由于经过Cycle-GAN模型转化得到的仿真图像能够模拟出样本真实图像的风格，即训练样本更加趋近于采集的真实图像，因此后续利用该手部关键点识别模型对真实图像进行手势识别的准确率更高；并且，基于携 带有关键点坐标的样本虚拟图像训练手部关键点识别模型，避免了人工标注关键点坐标，进一步提高了模型训练效率及准确性。
训练Cycle-GAN模型的关键在于生成器和判别器的训练，本申请实施例中的生成器用于对图像进行风格转换，判别器则用于判别原始图像和生成器输出的图像，而训练Cycle-GAN模型即为生成器与判别器相互对抗的过程，使得生成器输出图像的风格与目标图像风格一致(输出逼近目标分布)，判别器处于随机判别状态(对原始图像和生成器输出图像的判别概率均等)。下面采用示意性的实施例对Cycle-GAN模型的训练过程进行说明。
请参考图5，其示出了本申请另一个实施例提供的识别模型训练方法的流程图。本实施例以该手部关键点的识别模型训练方法应用于模型训练设备来举例说明，该方法可以包括以下几个步骤：
步骤S501，构建第一生成器和第二生成器，第一生成器用于生成仿真图像，第二生成器用于生成仿虚拟图像，仿样本虚拟图像用于模仿样本虚拟图像的风格。
Cycle-GAN模型的生成训练部分由生成器完成。本申请实施例中的Cycle-GAN模型用于对图像进行风格转换，其中包括两个生成器，分别是用于根据输入图像生成仿真图像的第一生成器，以及用于根据输入图像生成仿虚拟图像的第二生成器。第一生成器的输入图像为(原始)虚拟图像或者第二生成器输出的仿虚拟图像，第二生成器的输入图像为(原始)真实图像或第一生成器输出的仿真图像。
在一种可能的实施方式中，如图6所示，生成器可以由三部分组成，分别为编码器61、转换器62个解码器63。
其中，编码器61包含若干层卷积层611(图中以3层卷积层为例)，分别用于对输入图像进行卷积处理，从而提取出输入图像的特征。比如，卷积层611中包含若干个步长为2的卷积核。可选的，卷积层611后还可以添加激活层(ReLU)或批归一化(Batch Normalization，BN)层，本申请实施例 对此不做限定。
经过编码器61后，输入图像被压缩成若干个输入图像所属的数据域特征向量，后续即通过转换器62将该特征向量转化为输出图像所属数据域的特征向量。比如，当输入图像的尺寸为256×256×3(宽×高×通道数)，经过编码器61后得到尺寸为64×64×256的特征图。
转换器62是由若干个(图中以3个残差块为例)残差块621构成的残差网络。可选的，每个残差块621由至少两个卷积层构成，且残差块621的部分输入直接添加到输出，即残差块621的输出其卷积层的输出和输入构成，这样可以确保先前网络层的输入直接作用于后面的网络层，使得相应输出与原始输入的偏差缩小，使得原始图像的特征被保留在输出中且输出结果中。
本申请实施例中，转换器62用于对编码器61输出的特征向量进行数据域转换。其中，转换器62的输出尺寸与输入尺寸相同。比如，转换器62对编码器61输出的64×64×256的特征图(虚拟图像的特征)进行转换，输出64×64×256的特征图(真实图像的特征)，实现虚拟图像数据域到真实图像数据域的转换。
与编码过程相反，解码过程用于从特征向量中还原出低级特征，最终得到风格转换后的图像。如图6所示，解码器63包含若干层反卷积层631(图中以3层反卷积层为例)，分别用于对转换器62输出的特征向量进行反卷积处理，还原出低级特征。比如，反卷积层631中包含若干个步长为1/2的卷积核。
其中，解码器63最终输出图像的尺寸与输入图像的尺寸相同。比如，当输入图像的尺寸为256×256×3时，解码器63最终输出尺寸为256×256×3且经过风格转换的图像。
步骤S502，构建第一判别器和第二判别器，第一判别器用于判别样本虚拟图像和仿虚拟图像，第二判别器用于判别样本真实图像和仿真图像。
Cycle-GAN模型的对抗训练部分由判别器完成。本申请实施例中的Cycle-GAN模型包括两个判别器，分别是用于判别样本虚拟图像和仿虚拟图 像的第一判别器，以及用于判别样本真实图像和仿真图像的第二判别器。其中，仿虚拟图像由第二生成器根据样本图像生成，仿真图像由第一生成器根据样本虚拟图像生成。
判别器采用卷积神经网络结构，其中包含若干个用于提取图像特征的卷积层以及一个用于产生一维输出的卷积层。如图7所示，判别器中的卷积层71、卷积层72、卷积层73和卷积层74用于依次对输入图像进行特征提取，得到输入图像的特征图75，卷积层76则用于根据特征图75产生一维输出，从而根据一维输出确定输入图像属于真实图像还是仿真图像，或，该输入图像属于虚拟图像还是仿虚拟图像。
在一种可能的实施方式中，本申请实施例中的判别器采用0/1判别。比如，当判别器的输出为1时，指示输入图像为真实图像，当判别器的输出为0时，指示输入图像为仿真图像。
在另一种可能的实施方式中，本申请实施例中的判别器并非采用0/1判别，而是将输入图像划分为若干个相同尺寸的补丁(Patch)块，并基于Patch块进行判别。相较于0/1判别，采用基于Patch块的判别方式，能够减少训练的参数量，从而提高训练效率。
步骤S503，根据第一生成器、第二生成器、第一判别器和第二判别器构建Cycle-GAN模型。
进一步的，根据上述步骤中构建得到的生成器和判别器，模型训练设备构建Cycle-GAN模型。
示意性的，图8所示的Cycle-GAN模型中包含第一生成器81、第二生成器82、第一判别器83和第二判别器84
步骤S504，根据样本真实图像和样本虚拟图像计算Cycle-GAN模型的损失，Cycle-GAN模型的损失包括生成器损失、判别器损失和循环损失。
在训练构建得到的Cycle-GAN模型时，模型训练设备将样本虚拟图像和样本真实图像作为相应生成器的输入图像，由生成器对输入图像进行风格转换，由判别器对生成器输出的图像以及原始输入图像进行判别，并计算生成 器损失、判别器损失和循环损失，以便后续根据上述损失，并通过反向传播算法进行模型训练。其中，生成器损失是生成器转换图像时产生的损失，判别器损失用于指示判别器判别结果与预计结果之间误差，循环损失(或称为循环一致性损失(Cycle Consistency Loss))用于指示输入图像与经过两个生成器转换后输出图像之间的重构误差(Reconstruc)。
其中，输入的样本真实图像和样本虚拟图像并不需要成对，即样本真实图像中的手势与样本虚拟图像中的手势无需保持一致。
在训练图8所示Cycle-GAN模型中的生成器时，模型训练设备将样本虚拟图像A输入到第一生成器81中，由第一生成器81将样本虚拟图像A转换为仿真图像B1。进一步的，第一生成器81将转换得到的仿真图像B1输入到第二生成器82中，由第二生成器82将仿真图像B1转换为仿虚拟图像A2，并通过计算样本虚拟图像A与仿虚拟图像A2之间的重构误差RA确定循环损失。重构误差RA即用于指示样本虚拟图像A经过两个生成器风格转换后，与原始样本虚拟图像A之间的图像差异。其中，重构误差RA越小，表明样本虚拟图像A与仿虚拟图像A2越相似。
模型训练设备将样本真实图像B输入到第二生成器82中，由第二生成器82将样本真实图像B转换为仿虚拟图像A1。进一步的，第二生成器82将转换得到的仿虚拟图像A1输入到第一生成器81中，由第一生成器81将仿虚拟图像A1转换为仿真图像B2，并通过计算样本真实图像B与仿真图像B2之间的重构误差RB确定循环损失。重构误差RB即用于指示样本真实图像B经过两个生成器风格转换后，与原始样本真实图像B之间的图像差异。其中，重构误差RB越小，表明样本真实图像B与仿真图像B2越相似。
在训练图8所示Cycle-GAN模型中的判别器时，模型训练设备将样本虚拟图像A和仿虚拟图像A1输入第一判别器83，由第一判别器83进行判别，并根据实际判别结果和目标判别结果计算第一判别器83的判别器损失以及第二生成器82的生成器损失；类似的，模型训练设备将样本真实图像B和仿真图像B1输入第二判别器84，由第二判别器84进行判别，并根据实际判别 结果和目标判别结果计算第二判别器84的判别器损失以及第一生成器81的生成器损失。
可选的，模型训练设备根据生成器损失、判别器损失和循环损失构建Cycle-GAN模型的损失函数。其中，损失函数中，循环损失的乘法因子(权重)大于判别器损失的乘法因子。
步骤S505，根据Cycle-GAN模型的损失反向训练Cycle-GAN模型。
可选的，模型训练设备根据Cycle-GAN模型的生成器损失、判别器损失和循环损失，采用反向传播算法训练Cycle-GAN模型，本申请实施例并不对根据损失训练Cycle-GAN模型的具体方式进行实现。
可选的，最终训练得到的Cycle-GAN模型的损失函数最小。
最终训练得到的Cycle-GAN模型中，生成器与判别器之间实现了纳什均衡(Nash equilibrium)，即成器输出图像的风格与目标图像风格一致(输出逼近目标分布)，判别器处于随机判别状态(对原始图像和生成器输出图像的判别概率均等)。
通过上述步骤S501至S505，模型训练设备根据样本虚拟图像和样本真实图像训练得到用于风格转换的Cycle-GAN模型，后续进行手部关键点识别模型训练时，模型训练设备即使用该模型对携带标注信息的样本虚拟图像进行风格转换。
步骤S506，通过Cycle-GAN模型中的第一生成器，将样本虚拟图像转化为仿真图像。
为了使训练样本更加接近真实场景下采集到的真实图像，模型训练设备将样本虚拟图像输入Cycle-GAN模型的第一生成器，由第一生成器将样本虚拟图像转化为仿真图像。
由于仿真图像中手部可能只占较少的空间，如果直接对仿真图像缩放后进行训练或者测试，将导致手部图像的面积被进一步压缩。除此之外，手部关键点分布密集，且存在自遮挡等问题，在分辨率较小、图像模糊的情况下识别效果会显著降低。为了缓解这些问题，本申请实施例中，模型训练设备 首先先定位手部置信区域，再根据手部置信区域将手部及其周围少部分区域裁剪出来，并缩放作为下一级的输入。如图5所示，模型训练设备进一步通过下述步骤S507至S508从仿真图像中提取手部图像。
步骤S507，将仿真图像输入手势分割神经网络，输出手部置信区域，手势分割神经网络根据标注有手部区域的图像训练得到。
在一种可能的实施方式中，手势分割神经网络是采用U型网络(U-NET)结构的卷积神经网络(Convolutional Neural Networks，CNN)，包括n层卷积层和n层反卷积层。其中，n层卷积层用于对图像进行不同层次的特征提取，n层反卷积层用于对提取到的特征进行解码输出。示意性的，如图9所示，该手势分割神经网络包括3层卷积层和3层反卷积层。
可选的，如图10所示，通过手势分割神经网络确定手部置信区域包括如下步骤。
步骤S507A，通过n层卷积层对仿真图像进行处理，得到仿真图像对应的中间特征图。
模型训练设备将仿真图像输入手势神经分割网络，由n层卷积层依次对仿真图像进行卷积处理，从而提取出仿真图像中的特征，得到仿真图像对应的中间特征图。其中，对于第一层卷积层，其输入为仿真图像，而对于第i层卷积层，其输入则为第i-1层卷积层输出的特征图。
可选的，每层卷积层除了使用卷积核对输入图像或特征图(上一层卷积层输出)进行卷积处理外，还对进行卷积结果进行激活以及池化处理。
其中，可以使用ReLU函数对卷积结果进行激活处理。本申请实施例并不对使用的具体激活函数进行限定。
经过激活处理后，卷积层即对激活结果(特征图)进行池化处理，并将池化处理后的特征图输入下一层卷积层。其中，池化处理用于缩小特征图的尺寸，并保留特征图中的重要信息。可选的，各级池化层对输入的特征图进行最大池化处理。比如，当以2×2块，2为步长对特征图进行池化处理时，即提取特征图中2×2块中的最大值，将特征图的尺寸缩小至四分之一。
可选的，还可以采用平均池化或随机池化进行池化处理，本实施例并不对此构成限定。
示意性的，如图9所示，仿真图像91输入第一层卷积层后，在第一层卷积层经过卷积-池化处理，得到特征图92，并输入第二层卷积层；特征图92在第二层卷积层92经过卷积-池化处理后，得到特征图93，并输入第三层卷积层；特征图93在第三层卷积层经过卷积-池化处理后，进一步经过卷积操作后得到生成特征图94，并输入n层反卷积层。
步骤S507B，通过n层反卷积层对中间特征图进行处理，得到仿真图像对应的目标特征图。
手势分割神经网络进一步通过n层反卷积层，对中间特征图进行反卷积处理，最终得到仿真图像对应的目标特征图。其中，目标特征图的尺寸等于仿真图像的尺寸。
由于底层特征图具有更好的空间信息，而高层特征图具有更好的语义信息，为了使特征图能够更好的体现特征，在一种可能的实施方式中，在对中间特征图进行反卷积操作过程中，将高层特征图与先前生成的底层特征图进行拼接融合。相应的，本步骤可以包括如下步骤。
一、对第m层反卷积层输出的特征图与第n-m层卷积层输出的特征图进行拼接，1≤m≤n-1。
对于第m层反卷积层输出的特征图，模型训练设备将该特征图与第n-m层卷积层输出的特征图进行通道拼接，从而得到拼接后的特征图，其中，拼接后的特征图的通道数等于拼接前特征图的通道数之和。
由于第n-m层卷积层输出的特征图的尺寸大于第m层反卷积层输出的特征图的尺寸，因此，第m层反卷积层输出的特征图还需要经过上采样处理，从而增加特征图的尺寸。
可选的，在拼接前，使用1×1的卷积核对特征图进一步卷积处理，从而降低特征图的维度，减少拼接后特征图的通道数，从而降低计算量。除了使用1×1的卷积核进行卷积处理外，还可以使用3×3的卷积核对特征图进一步 卷积处理，从而扩大扩大特征图的感受野，以消除上采样过程中边缘不准确的影响。
二、通过第m+1层反卷积层对拼接后的特征图进行处理。
进一步的，拼接后的特征图被输入第m+1层反卷积层，由第m+1层反卷积层对其进行反卷积激活处理。
示意性的，如图9所示，特征图94在第一层反卷积层经过反卷积-上采样处理后，与第二层卷积层生成的特征图92进行拼接，生成特征图95，并输入第二层反卷积层；特征图95在第二层反卷积层经过反卷积-上采样处理后，与第一层卷积层生成的特征图91进行拼接，生成特征图96。
上述实施例仅以包含3层卷积层和3层反卷积层为例进行说明，并不对手势分割神经网络的结构构成限定。
步骤S507C，根据目标特征图生成概率矩阵，概率矩阵用于表征仿真图像中各个像素点是手部的概率。
进一步的，根据输出的目标特征图，模型训练设备确定仿真图像中的手部置信区域。在一种可能的实施方式中，模型训练设备根据目标特征图进行像素级的识别和分类，确定各个像素点对应手部区域的概率，其中，像素点对应的概率越高，表明该像素点为手部区域的概率越高。
步骤S507D，对概率矩阵进行二值化处理，输出手部区域对应的热图，热图用于指示仿真图像中的手部置信区域。
关于进行二值化处理的方式，在一种可能的实施方式中，模型训练设备比较概率矩阵中各个概率与概率阈值之间的大小关系，若概率大于概率阈值，则将该概率设置为1，若概率小于概率阈值，则将该概率设置为0。
示意性的，如图11所示，模型训练设备根据概率阈值0.6对概率矩阵1101进行二值化处理，将概率大于0.6的区域设置为1，将概率小于0.6的区域设置为0，得到热图1102，热图1102中，值为1的区域即为手部置信区域。
比如，如图9所示，经过二值化处理得到的热图97中，白色区域为手部置信区域，黑色区域为非手部置信区域。
步骤S508，根据手部置信区域，从仿真图像中提取手部图像。
根据手势分割神经网络输出的手部置信区域的坐标，模型训练设备从仿真图像中提取手部图像。
上述步骤S507D中，通过二值化处理得到手部置信区域时，可能会将部分手部区域的像素点错误划分为非手部区域，将部分非手部区域的像素点划分为手部区域，而这些错误划分通常位于实际手部区域的边缘位置，因此，在一种可能的实施方式中，模型训练设备根据手部置信区域的坐标，从仿真图像的XY平面上提取出尺寸略大于手部置信区域的手部图像。
可选的，模型训练设备对提取到的手部图像进行缩放处理，确保手部图像的尺寸与手部关键点识别模型的输入尺寸一致。
步骤S509，构建手部关键点识别模型，手部关键点识别模型包括二维识别分支和三维识别分支。
利用图2中训练得到的模型进行手部关键点识别时，首先利用2D关键点识别模型23得到二维关键点热图，再将二维关键点输入3D关键点识别模型24，最终得到手部关键点的三维坐标。然而，采用这种方式，二维和三维手部关键点采用独立的模型进行识别，忽略了二维和三维感知的内在联系性，导致关键点识别的准确性较低。
而本申请实施例中，模型训练设备采用二维热图与三维关键点坐标相耦合的深度学习方法训练手部关键点识别模型，由于这种耦合方式更加接近人眼处理视觉图像的机制(三维预测深度立体信息，而位于观测表观信息)，因此利用训练得到的模型进行关键点识别的准确率更高。
可选的，模型训练设备构建的手部关键点识别模型包括二维识别分支和三维识别分支，其中，二维识别分支包括i层二维残差层和卷积层，三维识别分支包括i层三维残差层和全连接层，且前j层二维残差层与前j层三维残差层耦合，2≤j≤i-1，i和j为整数。可选的，
示意性的，如图12所示，模型训练设备构建的手部关键点识别模型中包含二维识别分支1210和三维识别分支1220。二维识别分支1210由5个二维 残差块(每个二维残差块对应一层二维残差层)和一个全连接层1216构成，三维识别分支1220由5个三维残差块(每个三维残差块对应一层三维残差层)和一个卷积层1226构成，并且二维识别分支1210和三维识别分支1220的前3层相互耦合。可选的，二维识别分支和三维识别分支中的残差块可以采用常用的Residual残差块，且部分残差块中可以使用步长为2的卷积核进行特征图降维，以减小计算来那个并扩大特征图的感受野，本申请并对此不进行限定。
图12中，第一二维残差块1211和第一三维残差块1221的输入均为提取到的手部图像；第二二维残差块1212的输入为第一二维残差块1211以及第一三维残差块1221的输出；第二三维残差块1222的输入为第一二维残差块1211以及第一三维残差块1221的输出；第三二维残差块1213的输入为第二二维残差块1212以及第二三维残差块1222的输出；第三三维残差块1223的输入为第二二维残差块1212以及第二三维残差块1222的输出。
步骤S510，根据手部图像和关键点坐标，计算手部关键点识别模型的二维识别损失和三维识别损失，二维识别损失为二维识别分支的识别损失，三维识别损失是三维识别分支的识别损失。
进一步的，模型训练设备将手部图像分别输入二维识别分支和三维识别分支，并根据关键点坐标和二维识别分支输出识别结果计算二维识别损失，根据关键点坐标和三维识别分支输出识别结果计算三维识别损失。在一种可能的实施方式中，如图13所示，本步骤包括如下步骤。
步骤S510A，将手部图像分别输入二维识别分支和三维识别分支。
如图12所示，模型训练设备将手部图像分别输入第一二维残差块1211和第一三维残差块1221。
步骤S510B，对第k层二维残差层输出的特征图和第k层三维残差层输出的特征图进行相加操作，1≤k≤j-1。
对于前j层相互耦合的二维残差层和三维残差层，模型训练设备对第k层二维残差层的特征图和第k层三维残差层输出的特征图进行相加操作，并 将相加后得到的特征图作为第k+1层二维残差层和第k+1层三维残差层的输入。
其中，进行相加操作的特征图的维度相同，且相加前后特征图的维度保持一致。
示意性的，如图12所示，模型训练设备将第一二维残差块1211与第一三维残差块1221输出的特征图进行元素级相加操作；将第二二维残差块1212与第二三维残差块1222输出的特征图进行元素级相加操作。
步骤S510C，将相加后的特征图输入第k+1层二维残差层和第k+1层三维残差层。
进一步的，模型训练设备将相加后的特征图分别输入第k+1层二维残差层和第k+1层三维残差层，由第k+1层二维残差层和第k+1层三维残差层对相加后的特征图进行残差处理。
对于前j-1层二维残差层与前j-1层三维残差层，模型训练设备重复执行上述步骤S510B和S510C，直至将相加后的特征图输入第j层二维残差层和第j层三维残差层。
步骤S510D，通过第j+1至第i层二维残差层以及卷积层对第j层二维残差层输出的特征图进行处理，得到二维识别结果。
对于第j层二维残差层输出的特征图，模型训练设备依次通过第j+1至第i层二维残差层对特征图进行残差处理，进而通过卷积层对第i层二维残差层输出的特征图进行处理，得到二维识别结果，并执行下述步骤S510F。
可选的，该二维识别结果是手部图像中二维关键点的热图，该热图的尺寸与手部图像一致，且热图中响应高的点即为手部关键点。
可选的，二维识别分支将二维识别结果指示的手部关键点叠加显示在原始仿真图像上，且用不同的颜色对不同手指对应的手部关键点进行标识。
步骤S510E，通过第j+1至第i层三维残差层以及全连接层对第j层三维残差层输出的特征图进行处理，得到三维识别结果。
与二维识别相似的，对于第j层三维残差层输出的特征图，模型训练设 备依次通过第j+1至第i层三维残差层对特征图进行残差处理，进而通过全连接层对第i层三维残差层输出的特征图进行处理，得到三维识别结果，并执行下述步骤S510G。
可选的，该三维识别结果是手部图像中各个手部关键点的三维坐标，且该三维坐标为相对坐标，即以手部中预设关键点为坐标原点，计算其它关键点与预设关键点之间的归一化距离确定其它关键点的三维坐标。比如，当包含21个手部关键点时，三维识别分支中的全连接层的神经元数目即为21×3＝63个。
步骤S510F，根据二维识别结果和关键点坐标计算二维识别损失。
模型训练设备根据二维识别结果所指示手部关键点的二维坐标，计算该二维坐标与关键点坐标(即真值(Ground Truth))之间的二维识别损失。可选的，关键点坐标为样本虚拟图像中手部关键点的二维坐标，且二维识别损失可以是采用欧式距离标识。比如，二维识别结果中关键点A的二维坐标为(100px，150px)，且关键点A的实际二维坐标为(90px，140px)时，关键点A的二维识别损失即为
Figure PCTCN2019090542-appb-000001
步骤S510G，根据三维识别结果和关键点坐标计算三维识别损失。
与计算二维识别损失相似的，模型训练设备根据三维识别结果所指示手部关键点的三维坐标，计算该三维坐标与关键点坐标(即真值(Ground Truth))之间的三维识别损失。可选的，关键点坐标为样本虚拟图像中手部关键点的三维坐标，且三维识别损失可以是采用欧式距离标识。比如，三维识别结果中关键点A的二维坐标为(100px，150px，100px)，且关键点A的实际三维坐标为(90px，140px，100px)时，关键点A的三维识别损失即为
Figure PCTCN2019090542-appb-000002
步骤S511，根据二维识别损失和三维识别损失反向训练手部关键点识别模型。
可选的，根据计算得到的二维识别损失，模型训练设备通过反向传播算法，对手部关键点识别模型的二维识别分支进行训练；根据计算得到的三维 识别损失，模型训练设备通过反向传播算法，对手部关键点识别模型的三维识别分支进行训练。本申请实施例并不对根据识别损失训练手部关键点识别模型的具体方式进行限定。
本实施例中，通过构建包含两个生成器和两个判别器的Cycle-GAN模型，并根据生成器损失、判别器损失和循环损失反向训练Cycle-GAN模型，提高了生成得到的Cycle-GAN模型的质量。
本实施例中，在训练手部关键点识别模型时，通过将二维识别分支和三维识别分支进行耦合，使得训练出的手部关键点识别模型更加接近人眼处理视觉图像的机制，从而提高了手部关键点识别的准确率。
请参考图14，其示出了本申请一个实施例提供的识别方法的流程图。本实施例以该手部关键点的识别方法应用于手部关键点识别设备来举例说明，该方法可以包括以下几个步骤：
步骤S1401，采集真实图像，真实图像中包含手部图像。
在一种可能的实施方式中，手部关键点识别设备是设置有摄像头的终端，终端即通过摄像头采集包含手部图像的真实图像。
步骤S1402，提取真实图像中的手部图像。
在一种可能的实施方式中，手部关键点识别设备利用图5所示实施例中的手势分割神经网络，提取出真实图像中的手部图像。本实施例不再赘述提取手部图像的过程。
步骤S1403，将手部图像输入手部关键点识别模型，得到手部关键点坐标，手部关键点识别模型根据仿真图像以及仿真图像中手部图像的三维标注数据训练得到，仿真图像由Cycle-GAN模型根据样本虚拟图像生成，Cycle-GAN模型根据样本真实图像和样本虚拟图像训练生成，样本虚拟图像是三维建模生成的图像，且样本虚拟图像包含手部关键点对应的关键点坐标。
手部关键点识别设备将提取到的手部图像输入上述实施例中训练出的手部关键点识别模型，从而得到模型输出的手部关键点坐标。其中，该手部关 键点坐标为二维坐标和/或三维坐标。
在一种可能的实施方式中，如图15中的(a)所示，当手部关键点坐标为二维坐标时，手部关键点识别设备根据二维坐标将手部关键点叠加显示在真实图像上，并使用不同颜色对同一手指上的关键点进行标识；如图15中的(b)所示，当手部关键点坐标为三维坐标时，手部关键点识别设备根据三维坐标在三维坐标系中构建三维手部模型，并使用不同颜色对同一手指上的关键点进行标识。
如图15所示，当上述方法应用于手语识别场景时，终端根据手部关键点识别出的OK手势，在界面中显示手语翻译结果“OK”；当应用于手部特效场景时，终端根据识别出的OK手势，在界面中的手部区域模拟熊掌摆出OK手势；当应用于手势交互场景，智能家居设备根据识别出的OK手势执行确定操作。
综上所述，本实施例中，通过预先根据样本真实图像和样本虚拟图像训练Cycle-GAN模型，并利用Cycle-GAN模型将样本虚拟图像转化为仿真图像，从而根据仿真图像中的手部图像以及关键点坐标训练手部关键点识别模型；由于经过Cycle-GAN模型转化得到的仿真图像能够模拟出样本真实图像的风格，即训练样本更加趋近于采集的真实图像，因此后续利用该手部关键点识别模型对真实图像进行手势识别的准确率更高；并且，基于携带有关键点坐标的样本虚拟图像训练手部关键点识别模型，避免了人工标注关键点坐标，进一步提高了模型训练效率及准确性。
下述为本申请装置实施例，可以用于执行本申请方法实施例。对于本申请装置实施例中未披露的细节，请参照本申请方法实施例。
请参考图16，其示出了本申请一个实施例提供的识别模型训练装置的框图。该装置具有执行上述方法示例的功能，功能可以由硬件实现，也可以由硬件执行相应的软件实现。该装置可以包括转化模块1610、第一提取模块1620和第一训练模块1630，其中：
转化模块1610，用于通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像。
第一提取模块1620，用于提取所述仿真图像中的手部图像。
第一训练模块1630，用于根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
可选的，所述装置还包括：第二训练模块，用于根据样本真实图像和所述样本虚拟图像训练所述Cycle-GAN模型，所述样本真实图像是真实场景下采集的图像。
可选的，所述第二训练模块还用于构建第一生成器和第二生成器，所述第一生成器用于生成所述仿真图像，所述第二生成器用于生成仿虚拟图像，所述仿虚拟图像用于模仿所述样本虚拟图像的风格；构建第一判别器和第二判别器，所述第一判别器用于判别所述样本虚拟图像和所述仿虚拟图像，所述第二判别器用于判别所述样本真实图像和所述仿真图像；根据所述第一生成器、所述第二生成器、所述第一判别器和所述第二判别器构建所述Cycle-GAN模型；根据所述样本真实图像和所述样本虚拟图像计算所述Cycle-GAN模型的损失，所述Cycle-GAN模型的损失包括生成器损失、判别器损失和循环损失；及根据所述Cycle-GAN模型的损失反向训练所述Cycle-GAN模型。
可选的，所述转化模块1610还用于：通过所述Cycle-GAN模型中的所述第一生成器，将所述样本虚拟图像转化为所述仿真图像。
可选的，所述第一提取模块1620还用于：将所述仿真图像输入手势分割神经网络，输出手部置信区域，所述手势分割神经网络根据标注有手部区域的图像训练得到；及根据所述手部置信区域，从所述仿真图像中提取手部图像。
可选的，所述手势分割神经网络包括n层卷积层和n层反卷积层，n≥2，n为整数；所述第一提取模块1620，还用于：通过所述n层卷积层对所述仿真图像进行处理，得到所述仿真图像对应的中间特征图；通过所述n层反卷积层对所述中间特征图进行处理，得到所述仿真图像对应的目标特征图；根据所述目标特征图生成概率矩阵，所述概率矩阵用于表征所述仿真图像中各个像素点是手部的概率；对所述概率矩阵进行二值化处理，输出所述手部区域对应的热图，所述热图用于指示所述仿真图像中的所述手部置信区域。
可选的，通过所述n层反卷积层对所述中间特征图进行处理，得到所述仿真图像对应的目标特征图时，所述第一提取模块1620还用于：对第m层反卷积层输出的特征图与第n-m层卷积层输出的特征图进行拼接，1≤m≤n-1；及通过第m+1层反卷积层对拼接后的特征图进行处理。
可选的，所述第一训练模块1630还用于：构建所述手部关键点识别模型，所述手部关键点识别模型包括二维识别分支和三维识别分支，所述二维识别分支包括i层二维残差层和卷积层，所述三维识别分支包括i层三维残差层和全连接层，且前j层二维残差层与前j层三维残差层耦合，2≤j≤i-1，i和j为整数；根据所述手部图像和所述关键点坐标，计算所述手部关键点识别模型的二维识别损失和三维识别损失，所述二维识别损失为所述二维识别分支的识别损失，所述三维识别损失是所述三维识别分支的识别损失；及根据所述二维识别损失和所述三维识别损失反向训练所述手部关键点识别模型。
可选的，根据所述手部图像和所述关键点坐标，计算所述手部关键点识别模型的二维识别损失和三维识别损失时，所述第一训练模块1630，还用于：将所述手部图像分别输入所述二维识别分支和所述三维识别分支；对第k层二维残差层输出的特征图和第k层三维残差层输出的特征图进行相加操作，1≤k≤j-1；将相加后的特征图输入第k+1层二维残差层和第k+1层三维残差层；通过第j+1至第i层二维残差层以及所述卷积层对所述第j层二维残差层输出的特征图进行处理，得到二维识别结果；通过第j+1至第i层三维残差层以及所述全连接层对所述第j层三维残差层输出的特征图进行处理，得到三维识 别结果；根据所述二维识别结果和所述关键点坐标计算所述二维识别损失；及根据所述三维识别结果和所述关键点坐标计算所述三维识别损失。
请参考图17，其示出了本申请一个实施例提供的识别装置的框图。该装置具有执行上述方法示例的功能，功能可以由硬件实现，也可以由硬件执行相应的软件实现。该装置可以包括采集模块1710、第二提取模块1720和识别模块1730，其中：
采集模块1710，用于采集真实图像，所述真实图像中包含手部图像。
第二提取模块1720，用于提取所述真实图像中的所述手部图像。
识别模块1730，用于将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
请参考图18，其示出了本申请一个实施例提供的模型训练设备的结构示意图。该模型训练设备用于实施上述实施例提供的手部关键点的识别模型训练方法。具体来讲：
所述模型训练设备1800包括中央处理单元(CPU)1801、包括随机存取存储器(RAM)1802和只读存储器(ROM)1803的系统存储器1804，以及连接系统存储器1804和中央处理单元1801的系统总线1805。所述模型训练设备1800还包括帮助计算机内的各个器件之间传输信息的基本输入/输出系统(I/O系统)1806，和用于存储操作系统1813、应用程序1814和其他程序模块1815的大容量存储设备1807。
所述基本输入/输出系统1806包括有用于显示信息的显示器1808和用于用户输入信息的诸如鼠标、键盘之类的输入设备1809。其中所述显示器1808 和输入设备1809都通过连接到系统总线1805的输入输出控制器1810连接到中央处理单元1801。所述基本输入/输出系统1806还可以包括输入输出控制器1810以用于接收和处理来自键盘、鼠标、或电子触控笔等多个其他设备的输入。类似地，输入输出控制器1810还提供输出到显示屏、打印机或其他类型的输出设备。
所述大容量存储设备1807通过连接到系统总线1805的大容量存储控制器(未示出)连接到中央处理单元1801。所述大容量存储设备1807及其相关联的计算机可读介质为模型训练设备1800提供非易失性存储。也就是说，所述大容量存储设备1807可以包括诸如硬盘或者CD-ROM驱动器之类的计算机可读介质(未示出)。
不失一般性，所述计算机可读介质可以包括计算机存储介质和通信介质。计算机存储介质包括以用于存储诸如计算机可读指令、数据结构、程序模块或其他数据等信息的任何方法或技术实现的易失性和非易失性、可移动和不可移动介质。计算机存储介质包括RAM、ROM、EPROM、EEPROM、闪存或其他固态存储其技术，CD-ROM、DVD或其他光学存储、磁带盒、磁带、磁盘存储或其他磁性存储设备。当然，本领域技术人员可知所述计算机存储介质不局限于上述几种。上述的系统存储器1804和大容量存储设备1807可以统称为存储器。
根据本申请的各种实施例，所述模型训练设备1800还可以通过诸如因特网等网络连接到网络上的远程计算机运行。也即模型训练设备1800可以通过连接在所述系统总线1805上的网络接口单元1811连接到网络1812，或者说，也可以使用网络接口单元1811来连接到其他类型的网络或远程计算机系统。
所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、至少一段程序、代码集或指令集经配置以由一个或者一个以上处理器执行，以实现上述手部关键点的识别模型训练方法中各个步骤的功能。
图19示出了本申请一个示例性实施例提供的手部关键点识别设备1000的结构框图。该手部关键点识别设备1000可以是便携式移动终端，比如：智能手机、平板电脑、MP3播放器(Moving Picture Experts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving Picture Experts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器。
通常，手部关键点识别设备1000包括有：处理器1001和存储器1002。
处理器1001可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1001可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1001也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1001可以在集成有GPU(Graphics Processing Unit，图像处理器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1001还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。可选的，本申请实施例中，该AI处理器为具有手势识别功能的神经网络处理器(芯片)。
存储器1002可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是有形的和非暂态的。存储器1002还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1002中的非暂态的计算机可读存储介质用于存储至少一个指令，该至少一个指令用于被处理器1001所执行以实现本申请中提供的手部关键点的识别方法。
在一些实施例中，手部关键点识别设备1000还可选包括有：外围设备接口1003和至少一个外围设备。具体地，外围设备包括：射频电路1004、触摸显示屏1005、摄像头1006、音频电路1007、定位组件1008和电源1009 中的至少一种。
外围设备接口1003可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1001和存储器1002。在一些实施例中，处理器1001、存储器1002和外围设备接口1003被集成在同一芯片或电路板上；在一些其他实施例中，处理器1001、存储器1002和外围设备接口1003中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
射频电路1004用于接收和发射RF(Radio Frequency，射频)信号，也称电磁信号。射频电路1004通过电磁信号与通信网络以及其他通信设备进行通信。射频电路1004将电信号转换为电磁信号进行发送，或者，将接收到的电磁信号转换为电信号。可选地，射频电路1004包括：天线系统、RF收发器、一个或多个放大器、调谐器、振荡器、数字信号处理器、编解码芯片组、用户身份模块卡等等。射频电路1004可以通过至少一种无线通信协议来与其它终端进行通信。该无线通信协议包括但不限于：万维网、城域网、内联网、各代移动通信网络(2G、3G、4G及5G)、无线局域网和/或WiFi(Wireless Fidelity，无线保真)网络。在一些实施例中，射频电路1004还可以包括NFC(Near Field Communication，近距离无线通信)有关的电路，本申请对此不加以限定。
触摸显示屏1005用于显示UI(User Interface，用户界面)。该UI可以包括图形、文本、图标、视频及其它们的任意组合。触摸显示屏1005还具有采集在触摸显示屏1005的表面或表面上方的触摸信号的能力。该触摸信号可以作为控制信号输入至处理器1001进行处理。触摸显示屏1005用于提供虚拟按钮和/或虚拟键盘，也称软按钮和/或软键盘。在一些实施例中，触摸显示屏1005可以为一个，设置手部关键点识别设备1000的前面板；在另一些实施例中，触摸显示屏1005可以为至少两个，分别设置在手部关键点识别设备1000的不同表面或呈折叠设计；在再一些实施例中，触摸显示屏1005可以是柔性显示屏，设置在手部关键点识别设备1000的弯曲表面上或折叠面上。 甚至，触摸显示屏1005还可以设置成非矩形的不规则图形，也即异形屏。触摸显示屏1005可以采用LCD(Liquid Crystal Display，液晶显示器)、OLED(Organic Light-Emitting Diode,有机发光二极管)等材质制备。
摄像头组件1006用于采集图像或视频。可选地，摄像头组件1006包括前置摄像头和后置摄像头。通常，前置摄像头用于实现视频通话或自拍，后置摄像头用于实现照片或视频的拍摄。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能，主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能。在一些实施例中，摄像头组件1006还可以包括闪光灯。闪光灯可以是单色温闪光灯，也可以是双色温闪光灯。双色温闪光灯是指暖光闪光灯和冷光闪光灯的组合，可以用于不同色温下的光线补偿。
音频电路1007用于提供用户和手部关键点识别设备1000之间的音频接口。音频电路1007可以包括麦克风和扬声器。麦克风用于采集用户及环境的声波，并将声波转换为电信号输入至处理器1001进行处理，或者输入至射频电路1004以实现语音通信。出于立体声采集或降噪的目的，麦克风可以为多个，分别设置在手部关键点识别设备1000的不同部位。麦克风还可以是阵列麦克风或全向采集型麦克风。扬声器则用于将来自处理器1001或射频电路1004的电信号转换为声波。扬声器可以是传统的薄膜扬声器，也可以是压电陶瓷扬声器。当扬声器是压电陶瓷扬声器时，不仅可以将电信号转换为人类可听见的声波，也可以将电信号转换为人类听不见的声波以进行测距等用途。在一些实施例中，音频电路1007还可以包括耳机插孔。
定位组件1008用于定位手部关键点识别设备1000的当前地理位置，以实现导航或LBS(Location Based Service，基于位置的服务)。定位组件1008可以是基于美国的GPS(Global Positioning System，全球定位系统)、中国的北斗系统或俄罗斯的伽利略系统的定位组件。
电源1009用于为手部关键点识别设备1000中的各个组件进行供电。电 源1009可以是交流电、直流电、一次性电池或可充电电池。当电源1009包括可充电电池时，该可充电电池可以是有线充电电池或无线充电电池。有线充电电池是通过有线线路充电的电池，无线充电电池是通过无线线圈充电的电池。该可充电电池还可以用于支持快充技术。
在一些实施例中，手部关键点识别设备1000还包括有一个或多个传感器1010。该一个或多个传感器1010包括但不限于：加速度传感器1011、陀螺仪传感器1012、压力传感器1013、指纹传感器1014、光学传感器1015以及接近传感器1016。
加速度传感器1011可以检测以手部关键点识别设备1000建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器1011可以用于检测重力加速度在三个坐标轴上的分量。处理器1001可以根据加速度传感器1011采集的重力加速度信号，控制触摸显示屏1005以横向视图或纵向视图进行用户界面的显示。加速度传感器1011还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器1012可以检测手部关键点识别设备1000的机体方向及转动角度，陀螺仪传感器1012可以与加速度传感器1011协同采集用户对手部关键点识别设备1000的3D动作。处理器1001根据陀螺仪传感器1012采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器1013可以设置在手部关键点识别设备1000的侧边框和/或触摸显示屏1005的下层。当压力传感器1013设置在手部关键点识别设备1000的侧边框时，可以检测用户对手部关键点识别设备1000的握持信号，根据该握持信号进行左右手识别或快捷操作。当压力传感器1013设置在触摸显示屏1005的下层时，可以根据用户对触摸显示屏1005的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。
指纹传感器1014用于采集用户的指纹，以根据采集到的指纹识别用户的 身份。在识别出用户的身份为可信身份时，由处理器1001授权该用户执行相关的敏感操作，该敏感操作包括解锁屏幕、查看加密信息、下载软件、支付及更改设置等。指纹传感器1014可以被设置手部关键点识别设备1000的正面、背面或侧面。当手部关键点识别设备1000上设置有物理按键或厂商Logo时，指纹传感器1014可以与物理按键或厂商Logo集成在一起。
光学传感器1015用于采集环境光强度。在一个实施例中，处理器1001可以根据光学传感器1015采集的环境光强度，控制触摸显示屏1005的显示亮度。具体地，当环境光强度较高时，调高触摸显示屏1005的显示亮度；当环境光强度较低时，调低触摸显示屏1005的显示亮度。在另一个实施例中，处理器1001还可以根据光学传感器1015采集的环境光强度，动态调整摄像头组件1006的拍摄参数。
接近传感器1016，也称距离传感器，通常设置在手部关键点识别设备1000的正面。接近传感器1016用于采集用户与手部关键点识别设备1000的正面之间的距离。在一个实施例中，当接近传感器1016检测到用户与手部关键点识别设备1000的正面之间的距离逐渐变小时，由处理器1001控制触摸显示屏1005从亮屏状态切换为息屏状态；当接近传感器1016检测到用户与手部关键点识别设备1000的正面之间的距离逐渐变大时，由处理器1001控制触摸显示屏1005从息屏状态切换为亮屏状态。
本领域技术人员可以理解，图10中示出的结构并不构成对手部关键点识别设备1000的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
在一个实施例中，提供了一种模型训练设备，包括存储器和处理器，存储器存储有计算机程序，计算机程序被处理器执行时，使得处理器执行上述识别模型训练方法的步骤。此处识别模型训练方法的步骤可以是上述各个实施例的识别模型训练方法中的步骤。
在一个实施例中，提供了一种计算机可读存储介质，存储有计算机程序，计算机程序被处理器执行时，使得处理器执行上述识别模型训练方法的步骤。 此处识别模型训练方法的步骤可以是上述各个实施例的识别模型训练方法中的步骤。
在一个实施例中，提供了一种手部关键点识别设备，包括存储器和处理器，存储器存储有计算机程序，计算机程序被处理器执行时，使得处理器执行上述识别方法的步骤。此处识别方法的步骤可以是上述各个实施例的识别方法中的步骤。
在一个实施例中，提供了一种计算机可读存储介质，存储有计算机程序，计算机程序被处理器执行时，使得处理器执行上述识别方法的步骤。此处识别方法的步骤可以是上述各个实施例的识别方法中的步骤。
可选地，该计算机可读存储介质可以包括：只读存储器(ROM，Read Only Memory)、随机存取记忆体(RAM，Random Access Memory)、固态硬盘(SSD，Solid State Drives)或光盘等。其中，随机存取记忆体可以包括电阻式随机存取记忆体(ReRAM,Resistance Random Access Memory)和动态随机存取存储器(DRAM，Dynamic Random Access Memory)。上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。

Claims (20)

  1. 一种手部关键点的识别模型训练方法，由模型训练设备执行，所述方法包括：
    通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像；
    提取所述仿真图像中的手部图像；及
    根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
  2. 根据权利要求1所述的方法，其特征在于，所述通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像之前，所述方法还包括：
    根据样本真实图像和所述样本虚拟图像训练所述Cycle-GAN模型，所述样本真实图像是真实场景下采集的图像。
  3. 根据权利要求2所述的方法，其特征在于，所述根据样本真实图像和所述样本虚拟图像训练所述Cycle-GAN模型，包括：
    构建第一生成器和第二生成器，所述第一生成器用于生成所述仿真图像，所述第二生成器用于生成仿虚拟图像，所述仿虚拟图像用于模仿所述样本虚拟图像的风格；
    构建第一判别器和第二判别器，所述第一判别器用于判别所述样本虚拟图像和所述仿虚拟图像，所述第二判别器用于判别所述样本真实图像和所述仿真图像；
    根据所述第一生成器、所述第二生成器、所述第一判别器和所述第二判别器构建所述Cycle-GAN模型；
    根据所述样本真实图像和所述样本虚拟图像计算所述Cycle-GAN模型的损失，所述Cycle-GAN模型的损失包括生成器损失、判别器损失和循环损失；及
    根据所述Cycle-GAN模型的损失反向训练所述Cycle-GAN模型。
  4. 根据权利要求3所述的方法，其特征在于，所述通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，包括：
    通过所述Cycle-GAN模型中的所述第一生成器，将所述样本虚拟图像转化为所述仿真图像。
  5. 根据权利要求1至4任一所述的方法，其特征在于，所述提取所述仿真图像中的手部图像，包括：
    将所述仿真图像输入手势分割神经网络，输出手部置信区域，所述手势分割神经网络根据标注有手部区域的图像训练得到；及
    根据所述手部置信区域，从所述仿真图像中提取手部图像。
  6. 根据权利要求5所述的方法，其特征在于，所述手势分割神经网络包括n层卷积层和n层反卷积层，n≥2，n为整数；
    所述将所述仿真图像输入手势分割神经网络，输出手部置信区域，包括：
    通过所述n层卷积层对所述仿真图像进行处理，得到所述仿真图像对应的中间特征图；
    通过所述n层反卷积层对所述中间特征图进行处理，得到所述仿真图像对应的目标特征图；
    根据所述目标特征图生成概率矩阵，所述概率矩阵用于表征所述仿真图像中各个像素点是手部的概率；及
    对所述概率矩阵进行二值化处理，输出所述手部区域对应的热图，所述热图用于指示所述仿真图像中的所述手部置信区域。
  7. 根据权利要求6所述的方法，其特征在于，所述通过所述n层反卷积层对所述中间特征图进行处理，得到所述仿真图像对应的目标特征图，包括：
    对第m层反卷积层输出的特征图与第n-m层卷积层输出的特征图进行拼接，1≤m≤n-1；及
    通过第m+1层反卷积层对拼接后的特征图进行处理。
  8. 根据权利要求1至4任一所述的方法，其特征在于，所述根据所述仿 真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，包括：
    构建所述手部关键点识别模型，所述手部关键点识别模型包括二维识别分支和三维识别分支，所述二维识别分支包括i层二维残差层和卷积层，所述三维识别分支包括i层三维残差层和全连接层，且前j层二维残差层与前j层三维残差层耦合，2≤j≤i-1，i和j为整数；
    根据所述手部图像和所述关键点坐标，计算所述手部关键点识别模型的二维识别损失和三维识别损失，所述二维识别损失为所述二维识别分支的识别损失，所述三维识别损失是所述三维识别分支的识别损失；及
    根据所述二维识别损失和所述三维识别损失反向训练所述手部关键点识别模型。
  9. 根据权利要求8所述的方法，其特征在于，所述根据所述手部图像和所述关键点坐标，计算所述手部关键点识别模型的二维识别损失和三维识别损失，包括：
    将所述手部图像分别输入所述二维识别分支和所述三维识别分支；
    对第k层二维残差层输出的特征图和第k层三维残差层输出的特征图进行相加操作，1≤k≤j-1；
    将相加后的特征图输入第k+1层二维残差层和第k+1层三维残差层；
    通过第j+1至第i层二维残差层以及所述卷积层对所述第j层二维残差层输出的特征图进行处理，得到二维识别结果；
    通过第j+1至第i层三维残差层以及所述全连接层对所述第j层三维残差层输出的特征图进行处理，得到三维识别结果；
    根据所述二维识别结果和所述关键点坐标计算所述二维识别损失；及
    根据所述三维识别结果和所述关键点坐标计算所述三维识别损失。
  10. 一种手部关键点的识别方法，由手部关键点识别设备执行，所述方法包括：
    采集真实图像，所述真实图像中包含手部图像；
    提取所述真实图像中的所述手部图像；及
    将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
  11. 一种手部关键点的识别模型训练装置，其特征在于，所述装置包括：
    转化模块，用于通过Cycle-GAN模型，将样本虚拟图像转化为仿真图像，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标，所述仿真图像用于模仿真实场景下采集的图像；
    第一提取模块，用于提取所述仿真图像中的手部图像；及
    第一训练模块，用于根据所述仿真图像中的手部图像以及所述关键点坐标，训练手部关键点识别模型，所述手部关键点识别模型用于根据输入的真实图像输出所述真实图像中手部的手部关键点坐标。
  12. 根据权利要求11所述的装置，其特征在于，所述装置还包括：
    第二训练模块，用于根据样本真实图像和所述样本虚拟图像训练所述Cycle-GAN模型，所述样本真实图像是真实场景下采集的图像。
  13. 根据权利要求11所述的装置，其特征在于，所述第二训练模块还用于构建第一生成器和第二生成器，所述第一生成器用于生成所述仿真图像，所述第二生成器用于生成仿虚拟图像，所述仿虚拟图像用于模仿所述样本虚拟图像的风格；构建第一判别器和第二判别器，所述第一判别器用于判别所述样本虚拟图像和所述仿虚拟图像，所述第二判别器用于判别所述样本真实图像和所述仿真图像；根据所述第一生成器、所述第二生成器、所述第一判别器和所述第二判别器构建所述Cycle-GAN模型；根据所述样本真实图像和所述样本虚拟图像计算所述Cycle-GAN模型的损失，所述Cycle-GAN模型的 损失包括生成器损失、判别器损失和循环损失；及根据所述Cycle-GAN模型的损失反向训练所述Cycle-GAN模型。
  14. 根据权利要求13所述的装置，其特征在于，所述转化模块还用于通过所述Cycle-GAN模型中的所述第一生成器，将所述样本虚拟图像转化为所述仿真图像。
  15. 根据权利要求11至14任一项所述的装置，其特征在于，所述第一提取模块还用于将所述仿真图像输入手势分割神经网络，输出手部置信区域，所述手势分割神经网络根据标注有手部区域的图像训练得到；及根据所述手部置信区域，从所述仿真图像中提取手部图像。
  16. 根据权利要求15所述的装置，其特征在于，所述手势分割神经网络包括n层卷积层和n层反卷积层，n≥2，n为整数；所述第一提取模块还用于通过所述n层卷积层对所述仿真图像进行处理，得到所述仿真图像对应的中间特征图；通过所述n层反卷积层对所述中间特征图进行处理，得到所述仿真图像对应的目标特征图；根据所述目标特征图生成概率矩阵，所述概率矩阵用于表征所述仿真图像中各个像素点是手部的概率；及对所述概率矩阵进行二值化处理，输出所述手部区域对应的热图，所述热图用于指示所述仿真图像中的所述手部置信区域。
  17. 一种手部关键点的识别装置，其特征在于，所述装置包括：
    采集模块，用于采集真实图像，所述真实图像中包含手部图像；
    第二提取模块，用于提取所述真实图像中的所述手部图像；及
    识别模块，用于将所述手部图像输入手部关键点识别模型，得到手部关键点坐标，所述手部关键点识别模型根据仿真图像以及所述仿真图像中手部图像的三维标注数据训练得到，所述仿真图像由Cycle-GAN模型根据样本虚拟图像生成，所述Cycle-GAN模型根据样本真实图像和所述样本虚拟图像训练生成，所述样本虚拟图像是三维建模生成的图像，且所述样本虚拟图像包含手部关键点对应的关键点坐标。
  18. 一种模型训练设备，其特征在于，所述模型训练设备包括处理器和存储器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如权利要求1至9中任一项所述的方法的步骤。
  19. 一种手部关键点识别设备，其特征在于，所述手部关键点识别设备包括处理器和存储器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如权利要求10所述的方法的步骤。
  20. 一种非易失性的计算机可读存储介质，存储有计算机可读指令，所述计算机可读指令被一个或多个处理器执行时，使得所述一个或多个处理器执行如权利要求1至10中任一项所述的方法的步骤。
PCT/CN2019/090542 2018-07-10 2019-06-10 手部关键点的识别模型训练方法、识别方法及设备 WO2020010979A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP19833424.5A EP3742332B1 (en) 2018-07-10 2019-06-10 Method and apparatus for training model for recognizing key points of hand, and method and apparatus for recognizing key points of hand
JP2020558521A JP7130057B2 (ja) 2018-07-10 2019-06-10 手部キーポイント認識モデルの訓練方法及びその装置、手部キーポイントの認識方法及びその装置、並びにコンピュータプログラム
US17/000,844 US11989350B2 (en) 2018-07-10 2020-08-24 Hand key point recognition model training method, hand key point recognition method and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810752953.1A CN110163048B (zh) 2018-07-10 2018-07-10 手部关键点的识别模型训练方法、识别方法及设备
CN201810752953.1 2018-07-10

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/000,844 Continuation US11989350B2 (en) 2018-07-10 2020-08-24 Hand key point recognition model training method, hand key point recognition method and device

Publications (1)

Publication Number Publication Date
WO2020010979A1 true WO2020010979A1 (zh) 2020-01-16

Family

ID=67645012

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/090542 WO2020010979A1 (zh) 2018-07-10 2019-06-10 手部关键点的识别模型训练方法、识别方法及设备

Country Status (5)

Country Link
US (1) US11989350B2 (zh)
EP (1) EP3742332B1 (zh)
JP (1) JP7130057B2 (zh)
CN (1) CN110163048B (zh)
WO (1) WO2020010979A1 (zh)

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310616A (zh) * 2020-02-03 2020-06-19 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备和存储介质
CN111340694A (zh) * 2020-02-07 2020-06-26 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机可读存储介质和计算机设备
CN111680758A (zh) * 2020-06-15 2020-09-18 杭州海康威视数字技术股份有限公司 图像训练样本生成方法和装置
CN111832612A (zh) * 2020-06-03 2020-10-27 北京百度网讯科技有限公司 动物识别模型的训练方法、装置、设备及存储介质
CN111832383A (zh) * 2020-05-08 2020-10-27 北京嘀嘀无限科技发展有限公司 姿态关键点识别模型的训练方法、姿态识别方法及装置
CN111862046A (zh) * 2020-07-21 2020-10-30 江苏省人民医院（南京医科大学第一附属医院） 一种心脏冠脉剪影中导管位置判别系统和方法
CN111967353A (zh) * 2020-07-31 2020-11-20 北京金山云网络技术有限公司 图片识别方法、装置、电子设备以及介质
CN112115799A (zh) * 2020-08-24 2020-12-22 青岛小鸟看看科技有限公司 基于标记点的三维手势的识别方法、装置及设备
CN112330711A (zh) * 2020-11-26 2021-02-05 北京奇艺世纪科技有限公司 模型生成方法、信息提取方法、装置及电子设备
CN112561782A (zh) * 2020-12-15 2021-03-26 哈尔滨工程大学 一种提高海上场景仿真图片真实度的方法
CN112836597A (zh) * 2021-01-15 2021-05-25 西北大学 基于级联并行卷积神经网络的多手姿态关键点估计方法
CN113011403A (zh) * 2021-04-30 2021-06-22 恒睿(重庆)人工智能技术研究院有限公司 手势识别方法、系统、介质及设备
CN113111886A (zh) * 2021-04-19 2021-07-13 太原科技大学 一种基于双层残差网络的交通场景图像语义分割方法
CN113177606A (zh) * 2021-05-20 2021-07-27 上海商汤智能科技有限公司 图像处理方法、装置、设备及存储介质
CN113468924A (zh) * 2020-03-31 2021-10-01 北京沃东天骏信息技术有限公司 关键点检测模型训练方法和装置、关键点检测方法和装置
CN113869217A (zh) * 2021-09-29 2021-12-31 北京复数健康科技有限公司 一种用于获取图像识别数据的方法和系统
CN114077891A (zh) * 2020-08-07 2022-02-22 北京达佳互联信息技术有限公司 风格转换模型的训练方法及虚拟建筑检测模型的训练方法
CN114186632A (zh) * 2021-12-10 2022-03-15 北京百度网讯科技有限公司 关键点检测模型的训练方法、装置、设备、存储介质
JP2022522551A (ja) * 2020-02-03 2022-04-20 ベイジン センスタイム テクノロジー ディベロップメント カンパニー リミテッド 画像処理方法及び装置、電子機器並びに記憶媒体
CN117420917A (zh) * 2023-12-19 2024-01-19 烟台大学 基于手部骨架的虚拟现实控制方法、系统、设备及介质
CN117807782A (zh) * 2023-12-29 2024-04-02 南京仁高隆软件科技有限公司 一种实现三维仿真模型的方法
CN118196910A (zh) * 2024-05-17 2024-06-14 江西求是高等研究院 一种手势交互方法、系统、计算机及存储介质

Families Citing this family (67)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102061408B1 (ko) * 2017-03-24 2019-12-31 (주)제이엘케이인스펙션 가상 3차원 심층 신경망을 이용하는 영상 분석 장치 및 방법
CN110490213B (zh) * 2017-09-11 2021-10-29 腾讯科技（深圳）有限公司 图像识别方法、装置及存储介质
CN108882025B (zh) * 2018-08-07 2019-12-10 北京字节跳动网络技术有限公司 视频帧处理方法和装置
CN110070063B (zh) * 2019-04-29 2020-06-30 北京字节跳动网络技术有限公司 目标对象的动作识别方法、装置和电子设备
KR20210030147A (ko) * 2019-09-09 2021-03-17 삼성전자주식회사 3d 렌더링 방법 및 장치
CN110705611B (zh) * 2019-09-17 2024-06-28 平安科技（深圳）有限公司 眼底图像样本扩展方法、装置、介质及电子设备
CN110930354B (zh) * 2019-10-09 2020-07-10 西安交通大学 用于图像大数据的平滑过渡的视频画面内容分析系统
CN112767300B (zh) * 2019-10-18 2024-07-09 宏达国际电子股份有限公司 自动生成手部的标注数据的方法和计算骨骼长度的方法
CN111222401B (zh) * 2019-11-14 2023-08-22 北京华捷艾米科技有限公司 一种手部关键点三维坐标的识别方法及装置
WO2021098543A1 (zh) * 2019-11-20 2021-05-27 Oppo广东移动通信有限公司 一种姿势识别方法及装置、存储介质
CN111062261B (zh) * 2019-11-25 2023-07-07 维沃移动通信（杭州）有限公司 一种图像处理方法及装置
CN110991319B (zh) * 2019-11-29 2021-10-19 广州市百果园信息技术有限公司 手部关键点检测方法、手势识别方法及相关装置
CN113033256B (zh) * 2019-12-24 2024-06-11 武汉Tcl集团工业研究院有限公司 一种指尖检测模型的训练方法和设备
CN111161239B (zh) * 2019-12-27 2024-02-27 上海联影智能医疗科技有限公司 医学图像分析方法、装置、存储介质及计算机设备
CN111222486B (zh) * 2020-01-15 2022-11-04 腾讯科技（深圳）有限公司 手部姿态识别模型的训练方法、装置、设备及存储介质
CN111368668B (zh) * 2020-02-25 2023-05-23 北京字节跳动网络技术有限公司 三维手部识别方法、装置、电子设备及存储介质
CN112742031B (zh) * 2020-03-18 2022-08-02 腾讯科技（深圳）有限公司 模型训练方法、游戏测试方法、ai角色训练方法及装置
CN111462234B (zh) * 2020-03-27 2023-07-18 北京华捷艾米科技有限公司 一种位置确定方法及装置
CN111488824B (zh) * 2020-04-09 2023-08-08 北京百度网讯科技有限公司 运动提示方法、装置、电子设备和存储介质
CN111401318B (zh) * 2020-04-14 2022-10-04 支付宝(杭州)信息技术有限公司 动作识别方法及装置
CN113674182B (zh) * 2020-05-15 2023-12-05 北京罗克维尔斯科技有限公司 图像的生成方法及装置
CN111598075B (zh) * 2020-05-25 2024-09-20 深圳前海微众银行股份有限公司 图片生成方法、设备及可读存储介质
CN111695628B (zh) * 2020-06-11 2023-05-05 北京百度网讯科技有限公司 关键点标注方法、装置、电子设备及存储介质
CN111754478A (zh) * 2020-06-22 2020-10-09 怀光智能科技（武汉）有限公司 一种基于生成对抗网络的无监督域适应系统及方法
CN111783626B (zh) * 2020-06-29 2024-03-26 北京字节跳动网络技术有限公司 图像识别方法、装置、电子设备及存储介质
CN111985414B (zh) * 2020-08-21 2024-02-23 成都数字天空科技有限公司 一种关节点位置确定方法及装置
CN112101362A (zh) * 2020-08-25 2020-12-18 中国科学院空间应用工程与技术中心 用于空间科学实验数据的语义分割方法及系统
CN112115894B (zh) * 2020-09-24 2023-08-25 北京达佳互联信息技术有限公司 手部关键点检测模型的训练方法、装置及电子设备
CN112215112A (zh) * 2020-09-30 2021-01-12 幻境虚拟现实(广州)智能科技研究院有限公司 一种可用于手部动作识别的神经网络模型的生成方法和系统
CN112232183B (zh) * 2020-10-14 2023-04-28 抖音视界有限公司 虚拟佩戴物匹配方法、装置、电子设备和计算机可读介质
CN112256589B (zh) * 2020-11-11 2022-02-01 腾讯科技（深圳）有限公司 一种仿真模型的训练方法、点云数据的生成方法及装置
CN114756115A (zh) * 2020-12-28 2022-07-15 阿里巴巴集团控股有限公司 交互控制方法、装置及设备
CN112767478B (zh) * 2021-01-08 2022-07-19 北京航空航天大学 一种基于表观指导的六自由度位姿估计方法
CN112699837A (zh) * 2021-01-13 2021-04-23 新大陆数字技术股份有限公司 一种基于深度学习的手势识别方法及设备
CN112905006B (zh) * 2021-01-21 2022-12-02 清华大学 基于物理仿真的虚拟现实人手交互方法和装置
CN113033300B (zh) * 2021-02-07 2022-11-08 广东省科学院智能制造研究所 一种基于计算机视觉的扶梯安全自动监控方法及系统
CN112818929B (zh) * 2021-02-26 2023-04-18 济南博观智能科技有限公司 一种人员斗殴检测方法、装置、电子设备及存储介质
CN113034361B (zh) * 2021-03-03 2022-10-14 深圳市数字城市工程研究中心 一种基于改进esrgan的遥感影像超分重建方法
CN115082978A (zh) * 2021-03-10 2022-09-20 佳能株式会社 面部姿态的检测装置、方法、图像处理系统及存储介质
CN112967180B (zh) * 2021-03-17 2023-12-22 福建库克智能科技有限公司 一种生成对抗网络的训练方法、图像风格转换方法和装置
CN113139441A (zh) * 2021-04-07 2021-07-20 青岛以萨数据技术有限公司 一种图像处理方法及系统
CN113192175A (zh) * 2021-04-14 2021-07-30 武汉联影智融医疗科技有限公司 模型训练方法、装置、计算机设备和可读存储介质
CN113762969B (zh) * 2021-04-23 2023-08-08 腾讯科技（深圳）有限公司 信息处理方法、装置、计算机设备和存储介质
CN113128520B (zh) * 2021-04-28 2022-11-11 北京市商汤科技开发有限公司 图像特征提取方法、目标重识别方法、装置及存储介质
CN113238705A (zh) * 2021-05-10 2021-08-10 青岛小鸟看看科技有限公司 虚拟键盘交互方法及系统
CN113326755A (zh) * 2021-05-21 2021-08-31 华南理工大学 一种照明系统监测手部位置实现照明区域控制的方法
CN113077383B (zh) * 2021-06-07 2021-11-02 深圳追一科技有限公司 一种模型训练方法及模型训练装置
CN113610102A (zh) * 2021-06-23 2021-11-05 浙江大华技术股份有限公司 分割网络的训练及目标分割方法、相关设备
CN113487738B (zh) * 2021-06-24 2022-07-05 哈尔滨工程大学 基于虚拟知识迁移的建筑物及其遮挡区域单体化提取方法
CN113706463B (zh) * 2021-07-22 2024-04-26 杭州键嘉医疗科技股份有限公司 基于深度学习的关节影像关键点自动检测方法、装置
CN113706497B (zh) * 2021-08-24 2024-04-09 南京金盾公共安全技术研究院有限公司 一种违禁品智能识别装置和系统
CN113808184A (zh) * 2021-08-30 2021-12-17 中科尚易健康科技(北京)有限公司 基于点云识别的插值方法和装置、设备及存储介质
CN113723283A (zh) * 2021-08-30 2021-11-30 河南牧原智能科技有限公司 个体关键点检测模型获取方法、检测方法及系统
CN114066814A (zh) * 2021-10-19 2022-02-18 杭州易现先进科技有限公司 一种ar设备的手势3d关键点检测方法、电子设备
CN114185429B (zh) * 2021-11-11 2024-03-26 杭州易现先进科技有限公司 手势关键点定位或姿态估计的方法、电子装置和存储介质
CN114330433B (zh) * 2021-12-24 2023-05-05 南京理工大学 基于虚拟惯性测量信号生成模型的动作识别方法及系统
CN116453204B (zh) * 2022-01-05 2024-08-13 腾讯科技（深圳）有限公司 动作识别方法和装置、存储介质及电子设备
CN114882168B (zh) * 2022-04-08 2023-04-18 清华大学 一种基于视觉的触觉传感器的数字孪生方法及装置
CN115167673A (zh) * 2022-07-06 2022-10-11 中科传媒科技有限责任公司 虚拟手势同步的实现方法、装置、设备及存储介质
CN114973424A (zh) * 2022-08-01 2022-08-30 深圳市海清视讯科技有限公司 特征提取模型训练、手部动作识别方法、装置及电子设备
US20240193866A1 (en) * 2022-12-09 2024-06-13 Yannick VERDIE Methods and systems for 3d hand pose estimation from rgb images
CN116704427B (zh) * 2023-04-19 2024-01-26 广东建设职业技术学院 一种基于3d cnn循环施工过程监测方法
CN116148311B (zh) * 2023-04-24 2023-06-27 苏州太湖雪丝绸股份有限公司 一种纺织品接触凉感性能检测装置及其检测方法
CN116820251B (zh) * 2023-08-28 2023-11-07 中数元宇数字科技(上海)有限公司 一种手势轨迹交互方法、智能眼镜及存储介质
CN117523645B (zh) * 2024-01-08 2024-03-22 深圳市宗匠科技有限公司 一种人脸关键点检测方法、装置、电子设备及存储介质
CN117908678B (zh) * 2024-01-19 2024-07-16 广州维启通讯科技有限公司 一种耳机及其充电盒显示屏的手势识别控制系统
CN118379441B (zh) * 2024-06-27 2024-08-23 江西财经大学 面向真实场景的实时3d手部网格重建方法与系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170137350A (ko) * 2016-06-03 2017-12-13 (주)싸이언테크 신경망 생성 모델을 이용한 객체 움직임 패턴 학습장치 및 그 방법
CN108133220A (zh) * 2016-11-30 2018-06-08 北京市商汤科技开发有限公司 模型训练、关键点定位及图像处理方法、系统及电子设备
CN108230232A (zh) * 2016-12-21 2018-06-29 腾讯科技（深圳）有限公司 一种图像处理的方法以及相关装置

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6147678A (en) * 1998-12-09 2000-11-14 Lucent Technologies Inc. Video hand image-three-dimensional computer interface with multiple degrees of freedom
TW201322058A (zh) * 2011-11-16 2013-06-01 Chunghwa Picture Tubes Ltd 手勢辨識系統及方法
EP2980728A1 (en) * 2014-08-01 2016-02-03 Imersivo, S.L. Procedure for identifying a hand gesture
US10078780B2 (en) * 2015-03-27 2018-09-18 Intel Corporation Gesture recognition mechanism
US10157309B2 (en) * 2016-01-14 2018-12-18 Nvidia Corporation Online detection and classification of dynamic gestures with recurrent convolutional neural networks
CN105787439B (zh) * 2016-02-04 2019-04-05 广州新节奏智能科技股份有限公司 一种基于卷积神经网络的深度图像人体关节定位方法
JP6723061B2 (ja) * 2016-04-15 2020-07-15 キヤノン株式会社 情報処理装置、情報処理装置の制御方法およびプログラム
US10916001B2 (en) * 2016-11-28 2021-02-09 Adobe Inc. Facilitating sketch to painting transformations
CN107168527B (zh) * 2017-04-25 2019-10-18 华南理工大学 基于区域卷积神经网络的第一视角手势识别与交互方法
CN107808143B (zh) * 2017-11-10 2021-06-01 西安电子科技大学 基于计算机视觉的动态手势识别方法
CN108229318A (zh) * 2017-11-28 2018-06-29 北京市商汤科技开发有限公司 手势识别和手势识别网络的训练方法及装置、设备、介质
CN108229324B (zh) * 2017-11-30 2021-01-26 北京市商汤科技开发有限公司 手势追踪方法和装置、电子设备、计算机存储介质
CN108256431B (zh) * 2017-12-20 2020-09-25 中车工业研究院有限公司 一种手部位置标识方法及装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170137350A (ko) * 2016-06-03 2017-12-13 (주)싸이언테크 신경망 생성 모델을 이용한 객체 움직임 패턴 학습장치 및 그 방법
CN108133220A (zh) * 2016-11-30 2018-06-08 北京市商汤科技开发有限公司 模型训练、关键点定位及图像处理方法、系统及电子设备
CN108230232A (zh) * 2016-12-21 2018-06-29 腾讯科技（深圳）有限公司 一种图像处理的方法以及相关装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
HE JIAN-HUA; LONG FA-NING; ZHU XIAO-SHU: "Unpaired Image-to-Image Translation Using Improved Cycle-Consistent Adversarial Networks", JOURNAL OF YULIN NORMAL UNIVERSITY (NATURAL SCIENCE), vol. 39, no. 2, 30 April 2018 (2018-04-30), pages 122 - 126, XP009522308, DOI: 10.13792/j.cnki.cn45-1300/z.2018.02.025 *
See also references of EP3742332A4 *

Cited By (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111310616A (zh) * 2020-02-03 2020-06-19 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备和存储介质
US11417078B2 (en) 2020-02-03 2022-08-16 Beijing Sensetime Technology Development Co., Ltd. Image processing method and apparatus, and storage medium
JP2022522551A (ja) * 2020-02-03 2022-04-20 ベイジン センスタイム テクノロジー ディベロップメント カンパニー リミテッド 画像処理方法及び装置、電子機器並びに記憶媒体
CN111310616B (zh) * 2020-02-03 2023-11-28 北京市商汤科技开发有限公司 图像处理方法及装置、电子设备和存储介质
CN111340694A (zh) * 2020-02-07 2020-06-26 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机可读存储介质和计算机设备
CN111340694B (zh) * 2020-02-07 2023-10-27 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机可读存储介质和计算机设备
CN113468924A (zh) * 2020-03-31 2021-10-01 北京沃东天骏信息技术有限公司 关键点检测模型训练方法和装置、关键点检测方法和装置
CN111832383B (zh) * 2020-05-08 2023-12-08 北京嘀嘀无限科技发展有限公司 姿态关键点识别模型的训练方法、姿态识别方法及装置
CN111832383A (zh) * 2020-05-08 2020-10-27 北京嘀嘀无限科技发展有限公司 姿态关键点识别模型的训练方法、姿态识别方法及装置
CN111832612A (zh) * 2020-06-03 2020-10-27 北京百度网讯科技有限公司 动物识别模型的训练方法、装置、设备及存储介质
CN111832612B (zh) * 2020-06-03 2023-06-23 北京百度网讯科技有限公司 动物识别模型的训练方法、装置、设备及存储介质
CN111680758B (zh) * 2020-06-15 2024-03-05 杭州海康威视数字技术股份有限公司 图像训练样本生成方法和装置
CN111680758A (zh) * 2020-06-15 2020-09-18 杭州海康威视数字技术股份有限公司 图像训练样本生成方法和装置
CN111862046B (zh) * 2020-07-21 2023-11-17 江苏省人民医院（南京医科大学第一附属医院） 一种心脏冠脉剪影中导管位置判别系统和方法
CN111862046A (zh) * 2020-07-21 2020-10-30 江苏省人民医院（南京医科大学第一附属医院） 一种心脏冠脉剪影中导管位置判别系统和方法
CN111967353B (zh) * 2020-07-31 2024-05-14 北京金山云网络技术有限公司 图片识别方法、装置、电子设备以及介质
CN111967353A (zh) * 2020-07-31 2020-11-20 北京金山云网络技术有限公司 图片识别方法、装置、电子设备以及介质
CN114077891A (zh) * 2020-08-07 2022-02-22 北京达佳互联信息技术有限公司 风格转换模型的训练方法及虚拟建筑检测模型的训练方法
CN112115799B (zh) * 2020-08-24 2023-12-26 青岛小鸟看看科技有限公司 基于标记点的三维手势的识别方法、装置及设备
CN112115799A (zh) * 2020-08-24 2020-12-22 青岛小鸟看看科技有限公司 基于标记点的三维手势的识别方法、装置及设备
CN112330711B (zh) * 2020-11-26 2023-12-05 北京奇艺世纪科技有限公司 模型生成方法、信息提取方法、装置及电子设备
CN112330711A (zh) * 2020-11-26 2021-02-05 北京奇艺世纪科技有限公司 模型生成方法、信息提取方法、装置及电子设备
CN112561782B (zh) * 2020-12-15 2023-01-03 哈尔滨工程大学 一种提高海上场景仿真图片真实度的方法
CN112561782A (zh) * 2020-12-15 2021-03-26 哈尔滨工程大学 一种提高海上场景仿真图片真实度的方法
CN112836597B (zh) * 2021-01-15 2023-10-17 西北大学 基于级联并行卷积神经网络的多手姿态关键点估计方法
CN112836597A (zh) * 2021-01-15 2021-05-25 西北大学 基于级联并行卷积神经网络的多手姿态关键点估计方法
CN113111886A (zh) * 2021-04-19 2021-07-13 太原科技大学 一种基于双层残差网络的交通场景图像语义分割方法
CN113111886B (zh) * 2021-04-19 2023-03-24 太原科技大学 一种基于双层残差网络的交通场景图像语义分割方法
CN113011403B (zh) * 2021-04-30 2023-11-24 恒睿(重庆)人工智能技术研究院有限公司 手势识别方法、系统、介质及设备
CN113011403A (zh) * 2021-04-30 2021-06-22 恒睿(重庆)人工智能技术研究院有限公司 手势识别方法、系统、介质及设备
CN113177606A (zh) * 2021-05-20 2021-07-27 上海商汤智能科技有限公司 图像处理方法、装置、设备及存储介质
CN113177606B (zh) * 2021-05-20 2023-11-28 上海商汤智能科技有限公司 图像处理方法、装置、设备及存储介质
CN113869217A (zh) * 2021-09-29 2021-12-31 北京复数健康科技有限公司 一种用于获取图像识别数据的方法和系统
CN114186632B (zh) * 2021-12-10 2023-04-18 北京百度网讯科技有限公司 关键点检测模型的训练方法、装置、设备、存储介质
CN114186632A (zh) * 2021-12-10 2022-03-15 北京百度网讯科技有限公司 关键点检测模型的训练方法、装置、设备、存储介质
CN117420917A (zh) * 2023-12-19 2024-01-19 烟台大学 基于手部骨架的虚拟现实控制方法、系统、设备及介质
CN117420917B (zh) * 2023-12-19 2024-03-08 烟台大学 基于手部骨架的虚拟现实控制方法、系统、设备及介质
CN117807782A (zh) * 2023-12-29 2024-04-02 南京仁高隆软件科技有限公司 一种实现三维仿真模型的方法
CN117807782B (zh) * 2023-12-29 2024-06-07 南京仁高隆软件科技有限公司 一种实现三维仿真模型的方法
CN118196910A (zh) * 2024-05-17 2024-06-14 江西求是高等研究院 一种手势交互方法、系统、计算机及存储介质

Also Published As

Publication number Publication date
EP3742332B1 (en) 2023-06-28
JP2021518022A (ja) 2021-07-29
CN110163048A (zh) 2019-08-23
US20200387698A1 (en) 2020-12-10
EP3742332A4 (en) 2021-08-18
CN110163048B (zh) 2023-06-02
EP3742332A1 (en) 2020-11-25
JP7130057B2 (ja) 2022-09-02
US11989350B2 (en) 2024-05-21

Similar Documents

Publication Publication Date Title
WO2020010979A1 (zh) 手部关键点的识别模型训练方法、识别方法及设备
CN110348543B (zh) 眼底图像识别方法、装置、计算机设备及存储介质
WO2020224479A1 (zh) 目标的位置获取方法、装置、计算机设备及存储介质
CN109947886B (zh) 图像处理方法、装置、电子设备及存储介质
CN108594997B (zh) 手势骨架构建方法、装置、设备及存储介质
CN110135336B (zh) 行人生成模型的训练方法、装置及存储介质
CN111541907B (zh) 物品显示方法、装置、设备及存储介质
US20220309836A1 (en) Ai-based face recognition method and apparatus, device, and medium
CN107833219B (zh) 图像识别方法及装置
CN111476783B (zh) 基于人工智能的图像处理方法、装置、设备及存储介质
CN111091166A (zh) 图像处理模型训练方法、图像处理方法、设备及存储介质
CN110570460B (zh) 目标跟踪方法、装置、计算机设备及计算机可读存储介质
CN110544272A (zh) 脸部跟踪方法、装置、计算机设备及存储介质
CN112749613B (zh) 视频数据处理方法、装置、计算机设备及存储介质
CN111062981A (zh) 图像处理方法、装置及存储介质
CN112036331A (zh) 活体检测模型的训练方法、装置、设备及存储介质
CN110796005A (zh) 在线教学监控的方法、装置、电子设备及介质
CN108961375A (zh) 一种根据二维图像生成三维图像的方法及装置
CN113516665A (zh) 图像分割模型的训练方法、图像分割方法、装置、设备
CN113570645A (zh) 图像配准方法、装置、计算机设备及介质
CN116129526A (zh) 合拍控制方法、装置、电子设备及存储介质
CN110991445A (zh) 竖排文字识别方法、装置、设备及介质
CN110956571A (zh) 基于slam进行虚实融合的方法及电子设备
KR20220124432A (ko) 2차원 이미지에 기초한 3차원 가상 의류 착용방법 및 그 시스템
CN112257594A (zh) 多媒体数据的显示方法、装置、计算机设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19833424

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2019833424

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2020558521

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE