CN112306220A - Control method and device based on limb identification, electronic equipment and storage medium - Google Patents
Control method and device based on limb identification, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112306220A CN112306220A CN201910704604.7A CN201910704604A CN112306220A CN 112306220 A CN112306220 A CN 112306220A CN 201910704604 A CN201910704604 A CN 201910704604A CN 112306220 A CN112306220 A CN 112306220A
- Authority
- CN
- China
- Prior art keywords
- limb
- user
- video stream
- action gesture
- gesture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 230000009471 action Effects 0.000 claims abstract description 205
- 230000002452 interceptive effect Effects 0.000 claims abstract description 57
- 238000003062 neural network model Methods 0.000 claims abstract description 40
- 230000003993 interaction Effects 0.000 claims abstract description 18
- 238000012549 training Methods 0.000 claims description 27
- 238000012545 processing Methods 0.000 claims description 26
- 238000004590 computer program Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 10
- 230000036544 posture Effects 0.000 abstract description 43
- 230000006870 function Effects 0.000 abstract description 29
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000002618 waking effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The embodiment of the disclosure discloses a control method, a control device, electronic equipment and a storage medium based on limb identification, wherein the method comprises the following steps: acquiring a video stream comprising a limb of a user; inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb; and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture. According to the technical scheme of the embodiment, the limb postures are accurately recognized, and the purpose of controlling different system functions through multiple different limb postures is achieved, so that the user and the equipment have sufficient space ductility during interaction, and the interaction experience of the user is improved.
Description
Technical Field
The embodiments of the present disclosure relate to computer processing technologies, and in particular, to a control method and apparatus based on limb identification, an electronic device, and a storage medium.
Background
At present, most of common human-computer interaction modes are interaction modes based on a touch screen, such as a mobile phone, a self-service express cabinet and the like. In the field of television, the method is basically an interactive mode based on a remote controller. No matter the mobile phone, the self-service express cabinet or the television, the user is required to contact with an interactive tool (such as a remote controller and a touch screen) of the equipment, and once a gap exists between the user and the interactive tool of the equipment, the interaction with the equipment cannot be realized. Therefore, the existing man-machine interaction mode limits the space extensibility of the user to interact with the device and brings inconvenience to the user.
In order to solve the above problems, gesture recognition technology using human hands as a direct human-computer interaction input means is becoming mature. At present, a commonly used gesture recognition method is to roughly recognize a gesture based on an infrared induction principle, and the recognition precision is low, so that the abundant gesture gestures cannot be fully utilized to carry out interactive control of multiple functions.
Disclosure of Invention
The embodiment of the disclosure provides a control method and device based on limb recognition, electronic equipment and a storage medium, so as to realize accurate recognition of limb postures and further realize control of different system functions through various different limb postures.
In a first aspect, an embodiment of the present disclosure provides a control method based on limb identification, where the method includes:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
In a second aspect, an embodiment of the present disclosure further provides a control device based on limb identification, where the device includes:
the acquisition module is used for acquiring a video stream comprising limbs of a user;
the recognition module is used for inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and the execution module is used for executing the interactive operation associated with the target action gesture if the action gesture is the set action gesture and the target action gesture of the user limb is identified within the preset time length after the set action gesture is obtained.
In a third aspect, an embodiment of the present disclosure further provides an apparatus, where the apparatus includes:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a limb identification-based control method as in any of the embodiments of the present disclosure.
In a fourth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions, which when executed by a computer processor, are used to execute the limb identification-based control method according to any one of the embodiments of the present disclosure.
According to the technical scheme of the embodiment of the disclosure, the action posture of the user limb is obtained by acquiring the video stream comprising the user limb and inputting the video stream into a pre-trained neural network model; if the action gesture is the set action gesture and the target action gesture of the user limb is recognized within the preset time after the set action gesture is obtained, the technical means of interactive operation associated with the target action gesture is executed, accurate recognition of the limb gesture is achieved, the purpose of controlling different system functions through various different limb gestures is achieved, the interaction between the user and the equipment has sufficient space ductility, and the interaction experience of the user is improved.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.
Fig. 1 is a schematic flowchart of a control method based on limb identification according to a first embodiment of the present disclosure;
FIG. 2 is a schematic diagram of a hand set point provided in accordance with an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a control device based on limb identification according to a second embodiment of the disclosure;
fig. 4 is a schematic structural diagram of an electronic device according to a third embodiment of the disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
Example one
Fig. 1 is a schematic flowchart of a control method based on limb identification according to an embodiment of the present disclosure, where the present embodiment is applicable to a human-computer interaction scenario, and the method may be executed by a control device based on limb identification, where the control device may be implemented in a form of software and/or hardware and is integrated in a terminal device, and typically, the terminal device includes a smartphone or a server. As shown in fig. 1, the control method based on limb identification disclosed in this embodiment includes:
The user limb may be a hand of the user, or an arm, a leg, a head, or a foot of the user.
Illustratively, obtaining a video stream including a limb of a user comprises:
and acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment.
Or, the video stream including the user's limb is obtained from a separate device from the terminal device.
Specifically, if the terminal device does not have a video acquisition hardware system, a communication connection can be established with a special video acquisition device, the special video acquisition device is responsible for acquiring a video stream including the limb of the user, and then the special video acquisition device sends the acquired video stream to the terminal device or a server corresponding to the terminal device, so that the limb action posture of the user in the video stream is identified through the terminal device or the server. The limb action gesture is, for example, a circular, heart-shaped, square or triangular gesture made by the user through a hand, or an action gesture of sliding the user's finger up or down, and the like.
And 120, inputting the video stream to a pre-trained neural network model to obtain the action posture of the limb of the user.
The pre-trained neural network model specifically comprises: CNN (Convolutional Neural Network) model. The CNN is a feedforward neural network, comprises a convolutional layer and a pooling layer, and has wide application in the field of image recognition. Theoretically, as long as there are enough training samples, which refer to raw data labeled with a recognition result, for example, in a user gesture recognition scene, the training samples are video streams or images labeled with a user gesture, a CNN model of a corresponding recognition function can be obtained.
Further, in order to improve the recognition accuracy and the convergence rate of the model, the training samples may be preprocessed before being input to the model to be trained, specifically, the training samples may be subjected to preliminary feature extraction, so as to improve the convergence rate of the model by reducing the calculation amount. Taking a scene of recognizing the gesture of the user as an example, the preprocessing the training sample specifically comprises:
labeling the set point position information of the user hand in the training sample, wherein the set point of the user hand specifically comprises the joint point of the user hand, and of course, the set point of the user hand can also comprise the non-joint point of the user hand. It can be understood that the more the number of labeled set points, the higher the recognition accuracy of the trained model, and the finer the gesture recognized. Schematic hand set points as can be seen in fig. 2, only A, B and C are shown in fig. 2, the set points being selected at joint points generally on the hand contour.
Illustratively, inputting the video stream into a pre-trained neural network model to obtain the motion posture of the user limb includes:
preprocessing the video stream to label the position information of the user limb set point included in the video stream;
inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the user limb;
the pre-trained neural network model is obtained by training based on training samples, and the training samples comprise video streams marked with the position information of the limb set point and limb action posture information included in the video streams. The motion gesture information includes specific gestures, such as a circle, a heart, a square, or a triangle, etc., which are put out by hand.
And step 130, if the action gesture is a set action gesture and the target action gesture of the user limb is recognized within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
Wherein the interaction comprises: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out. For example, when the user puts out an O-shaped target action gesture in space by hand, the control application is opened; when the user puts out a C-shaped target action gesture in the space by hand, the application is controlled to be closed; when the hand of the user continuously makes a target action gesture of sliding downwards (a single finger can make the sliding downwards action, or the whole hand can make the sliding downwards action), controlling the current display page to turn downwards; when the user continuously makes a target action gesture of sliding upwards, controlling the current display page to turn upwards; and when the user makes a target action gesture of sliding left and right, controlling the page to turn left and right, and the like.
Specifically, each target action posture and the controlled interactive operation thereof can be stored correspondingly in advance, and when the target action posture of the user is identified, the control function which needs to be executed currently is determined by searching the interactive operation associated with the target action posture.
Illustratively, performing the interactive operation associated with the target action gesture includes:
acquiring interactive operation associated with the target action gesture based on a prestored action gesture and an associated interactive operation list thereof;
and controlling the corresponding function module to execute the interactive operation associated with the target action gesture through the function control module.
The set action gesture is used for waking up a function control module of the equipment, if the set action gesture is made before the target action gesture is made by the user, the user is informed of the terminal equipment that the user needs to interact with the terminal equipment, and if the target action gesture made by the user is identified within the preset time period, the function control module controls a corresponding function module to execute the interactive operation associated with the target action gesture. If the user does not make the set action gesture before making the target action gesture, the requirement that the user does not interact with the terminal equipment is indicated, and at the moment, even if the target action gesture made by the user is identified, the corresponding function module is not controlled, namely, the target action gesture is not responded, so that the false triggering of the user on the terminal equipment is avoided. For example, a gesture which is accidentally put out by the user in the process of communicating with other people is recognized as a target action gesture by the terminal device, and further interactive operation associated with the target action gesture is executed, so that a situation that the user triggers the terminal device by mistake occurs.
Optionally, the device may be awakened in a voice manner, for example, when the user wants to interact with the terminal device, a set awakening word, such as "xiaozhi", "ducu", "small Q", or the like, may be spoken by voice, and when the terminal device recognizes the set awakening word, the control system of the terminal device is awakened to start to perform the steps of acquiring a video stream including the limb of the user, inputting the video stream to a pre-trained neural network model, obtaining a target motion posture of the limb of the user, and performing an interactive operation associated with the target motion posture.
Before the user and the terminal equipment start to interact, the terminal equipment is awakened by setting the action gesture or the awakening word, so that the problem of false triggering of the terminal equipment by the user is avoided, and the user experience is improved; because the terminal equipment does not need to be in a working state all the time, the power consumption of the terminal equipment is greatly reduced.
According to the technical scheme of the embodiment of the disclosure, the action posture of the user limb is obtained by acquiring the video stream comprising the user limb and inputting the video stream into a pre-trained neural network model; if the action gesture is the set action gesture and the target action gesture of the user limb is recognized within the preset time after the set action gesture is obtained, the interactive operation associated with the target action gesture is executed, the accurate recognition of the limb gesture is realized, the purpose of controlling different system functions through various different limb gestures is further realized, the interaction between the user and the equipment has sufficient space ductility, and the interaction experience of the user is improved.
Example two
Fig. 3 is a control device based on limb identification according to a second embodiment of the present disclosure, where the device includes: an acquisition module 410, an identification module 420 and an execution module 430;
the obtaining module 410 is configured to obtain a video stream including a limb of a user; the recognition module 420 is configured to input the video stream to a pre-trained neural network model to obtain a motion posture of a limb of the user; and the executing module 430 is configured to execute the interactive operation associated with the target action gesture if the action gesture is the set action gesture and the target action gesture of the user limb is identified within a preset time period after the set action gesture is obtained.
On the basis of the above technical solutions, the apparatus is integrated in a terminal device, and the corresponding obtaining module 410 is specifically configured to:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment;
or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device.
On the basis of the above technical solutions, the identification module 420 includes:
the preprocessing unit is used for preprocessing the video stream so as to label the position information of the user limb set point included in the video stream;
the recognition unit is used for inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the limb of the user;
the pre-trained neural network model is obtained by training based on training samples, and the training samples comprise video streams marked with the position information of the limb set point and limb action posture information included in the video streams.
On the basis of the above technical solutions, the executing module 430 includes:
the acquisition unit is used for acquiring the interactive operation associated with the target action gesture based on a prestored action gesture and an associated interactive operation list thereof;
and the control unit is used for controlling the corresponding functional module to execute the interactive operation associated with the target action gesture through the functional control module.
On the basis of the above technical solutions, the user limb includes: the target action gesture comprises a circular, heart-shaped, square or triangular gesture.
On the basis of the above technical solutions, the interactive operation includes: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out.
According to the technical scheme of the embodiment of the disclosure, the action posture of the user limb is obtained by acquiring the video stream comprising the user limb and inputting the video stream into a pre-trained neural network model; if the action gesture is the set action gesture and the target action gesture of the user limb is recognized within the preset time after the set action gesture is obtained, the interactive operation associated with the target action gesture is executed, the accurate recognition of the limb gesture is realized, the purpose of controlling different system functions through various different limb gestures is further realized, the interaction between the user and the equipment has sufficient space ductility, and the interaction experience of the user is improved.
The control device based on limb identification provided by the embodiment of the disclosure can execute the control method based on limb identification provided by any embodiment of the disclosure, and has corresponding functional modules and beneficial effects of the execution method.
It should be noted that, the units and modules included in the apparatus are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only used for distinguishing one functional unit from another, and are not used for limiting the protection scope of the embodiments of the present disclosure.
EXAMPLE III
Referring now to fig. 4, a schematic diagram of an electronic device (e.g., the terminal device or the server of fig. 4) 400 suitable for implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 4, the electronic device 400 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 401 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from a storage device 406 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for the operation of the electronic apparatus 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.
Generally, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 406 including, for example, magnetic tape, hard disk, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate wirelessly or by wire with other devices to exchange data. While fig. 4 illustrates an electronic device 400 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 409, or from the storage means 406, or from the ROM 402. The computer program performs the above-described functions defined in the methods of the embodiments of the present disclosure when executed by the processing device 401.
The terminal provided by the embodiment of the present disclosure and the control method based on limb identification provided by the embodiment of the present disclosure belong to the same inventive concept, and technical details that are not described in detail in the embodiment of the present disclosure may be referred to the embodiment of the present disclosure, and the embodiment of the present disclosure have the same beneficial effects.
Example four
The embodiments of the present disclosure provide a computer storage medium, on which a computer program is stored, which when executed by a processor implements the limb identification-based control method provided by the above embodiments.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a cell does not in some cases constitute a limitation on the cell itself, for example, an editable content display cell may also be described as an "editing cell".
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Some example embodiments of the present disclosure are listed below.
In one aspect, there is provided a limb recognition-based control method, the method comprising: acquiring a video stream comprising a limb of a user; inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb; and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
In some embodiments, the method is applied to a terminal device for acquiring a video stream including a limb of a user, and comprises the following steps:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment;
or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device.
In some embodiments, inputting the video stream to a pre-trained neural network model to obtain the motion posture of the user limb includes:
preprocessing the video stream to label the position information of the user limb set point included in the video stream;
inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the user limb;
the pre-trained neural network model is obtained by training based on training samples, and the training samples comprise video streams marked with the position information of the limb set point and limb action posture information included in the video streams.
In some embodiments, performing the interactive operation associated with the target action gesture includes:
acquiring interactive operation associated with the target action gesture based on a prestored action gesture and an associated interactive operation list thereof;
and controlling the corresponding function module to execute the interactive operation associated with the target action gesture through the function control module.
In some embodiments, the user limb comprises: the target action gesture comprises a circular, heart-shaped, square or triangular gesture.
In some embodiments, the interaction comprises: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out.
In one aspect, there is provided a limb recognition-based control device, the device comprising:
the acquisition module is used for acquiring a video stream comprising limbs of a user;
the recognition module is used for inputting the video stream to a pre-trained neural network model to obtain a target action posture of the user limb;
and the execution module is used for executing the interactive operation associated with the target action gesture if the action gesture is the set action gesture and the target action gesture of the user limb is identified within the preset time length after the set action gesture is obtained.
In some embodiments, the apparatus is integrated in a terminal device, and the corresponding obtaining module 410 is specifically configured to:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment;
or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device.
In some embodiments, the identification module 420 comprises:
the preprocessing unit is used for preprocessing the video stream so as to label the position information of the user limb set point included in the video stream;
the recognition unit is used for inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the limb of the user;
the pre-trained neural network model is obtained by training based on training samples, and the training samples comprise video streams marked with the position information of the limb set point and limb action posture information included in the video streams.
In some embodiments, the execution module 430 includes:
the acquisition unit is used for acquiring the interactive operation associated with the target action gesture based on a prestored action gesture and an associated interactive operation list thereof;
and the control unit is used for controlling the corresponding functional module to execute the interactive operation associated with the target action gesture through the functional control module.
In some embodiments, the user limb comprises: the target action gesture comprises a circular, heart-shaped, square or triangular gesture.
In some embodiments, the interaction comprises: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out.
In one aspect, there is provided a terminal device, the device comprising: one or more processing devices; storage means for storing one or more programs; when executed by the one or more processing devices, cause the one or more processing devices to implement a limb recognition-based control method that:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
In some embodiments, the one or more programs, when executed by the one or more processing devices, cause the one or more processing devices to implement the limb recognition based control method of:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment; or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
In some embodiments, the one or more programs, when executed by the one or more processing devices, cause the one or more processing devices to implement the limb recognition based control method of:
acquiring a video stream comprising a limb of a user;
preprocessing the video stream to label the position information of the user limb set point included in the video stream;
inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the user limb;
if the action gesture is a set action gesture and a target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture;
the pre-trained neural network model is obtained by training based on a training sample, wherein the training sample comprises a video stream marked with the position information of the limb set point and the posture information of the limb action included in the video stream;
in some embodiments, the one or more programs, when executed by the one or more processing devices, cause the one or more processing devices to implement the limb recognition based control method of:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
if the action gesture is a set action gesture and a target action gesture of the limb of the user is identified within a preset time after the set action gesture is obtained, acquiring an interactive operation associated with the target action gesture based on a pre-stored action gesture and an interactive operation list associated with the pre-stored action gesture;
and controlling the corresponding function module to execute the interactive operation associated with the target action gesture through the function control module.
In some embodiments, the one or more programs, when executed by the one or more processing devices, cause the one or more processing devices to implement the limb recognition based control method of:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
if the action gesture is a set action gesture and a target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture;
the user limb includes: a user hand, the target action gesture comprising a circular, heart, square, or triangular gesture;
the interactive operation comprises the following steps: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out.
In one aspect, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processing apparatus, implements a limb recognition-based control method as follows:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
In some embodiments, the program when executed by the processing device implements a limb recognition based control method as follows:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment; or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
In some embodiments, the program when executed by the processing device implements a limb recognition based control method as follows:
acquiring a video stream comprising a limb of a user;
preprocessing the video stream to label the position information of the user limb set point included in the video stream;
inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the user limb;
if the action gesture is a set action gesture and a target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture;
the pre-trained neural network model is obtained by training based on a training sample, wherein the training sample comprises a video stream marked with the position information of the limb set point and the posture information of the limb action included in the video stream.
In some embodiments, the program when executed by the processing device implements a limb recognition based control method as follows:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
if the action gesture is a set action gesture and a target action gesture of the limb of the user is identified within a preset time after the set action gesture is obtained, acquiring an interactive operation associated with the target action gesture based on a pre-stored action gesture and an interactive operation list associated with the pre-stored action gesture;
and controlling the corresponding function module to execute the interactive operation associated with the target action gesture through the function control module.
In some embodiments, the program when executed by the processing device implements a limb recognition based control method as follows:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
if the action gesture is a set action gesture and a target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture;
the user limb includes: a user hand, the target action gesture comprising a circular, heart, square, or triangular gesture;
the interactive operation comprises the following steps: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
Claims (10)
1. A control method based on limb identification is characterized by comprising the following steps:
acquiring a video stream comprising a limb of a user;
inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and if the action gesture is a set action gesture and the target action gesture of the user limb is identified within a preset time length after the set action gesture is obtained, executing interactive operation associated with the target action gesture.
2. The method according to claim 1, wherein the method is applied to a terminal device, and the obtaining a video stream including a limb of a user comprises:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment;
or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device.
3. The method of claim 1, wherein inputting the video stream into a pre-trained neural network model to obtain the motion posture of the user's limb comprises:
preprocessing the video stream to label the position information of the user limb set point included in the video stream;
inputting the preprocessed video stream into a pre-trained neural network model to obtain the action posture of the user limb;
the pre-trained neural network model is obtained by training based on a training sample, wherein the training sample comprises a video stream marked with the position information of the limb set point and the posture information of the limb action included in the video stream.
4. The method of any of claims 1-3, wherein performing the interactive operation associated with the target action gesture comprises:
acquiring interactive operation associated with the target action gesture based on a prestored action gesture and an associated interactive operation list thereof;
and controlling the corresponding function module to execute the interactive operation associated with the target action gesture through the function control module.
5. The method of any of claims 1-3, wherein the user limb comprises: the target action gesture comprises a circular, heart-shaped, square or triangular gesture.
6. The method of any of claims 1-3, wherein the interaction comprises: at least one of opening an application, closing an application, turning pages, page zooming in, or page zooming out.
7. A limb identification based control device, comprising:
the acquisition module is used for acquiring a video stream comprising limbs of a user;
the recognition module is used for inputting the video stream to a pre-trained neural network model to obtain the action posture of the user limb;
and the execution module is used for executing the interactive operation associated with the target action gesture if the action gesture is the set action gesture and the target action gesture of the user limb is identified within the preset time length after the set action gesture is obtained.
8. The apparatus according to claim 7, wherein the apparatus is integrated in a terminal device, and correspondingly, the obtaining module is specifically configured to:
acquiring the video stream comprising the limbs of the user based on a camera carried by the terminal equipment;
or, the video stream including the user limb is acquired from a video acquisition device independent of the terminal device.
9. A terminal device, characterized in that the device comprises:
one or more processing devices;
storage means for storing one or more programs;
when executed by the one or more processing devices, cause the one or more processing devices to implement the limb identification based control method of any one of claims 1-6.
10. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processing device, carries out a limb identification-based control method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910704604.7A CN112306220A (en) | 2019-07-31 | 2019-07-31 | Control method and device based on limb identification, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910704604.7A CN112306220A (en) | 2019-07-31 | 2019-07-31 | Control method and device based on limb identification, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112306220A true CN112306220A (en) | 2021-02-02 |
Family
ID=74485755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910704604.7A Pending CN112306220A (en) | 2019-07-31 | 2019-07-31 | Control method and device based on limb identification, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112306220A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113014815A (en) * | 2021-03-03 | 2021-06-22 | 南京地平线机器人技术有限公司 | Method and device for controlling equipment in closed space, medium and equipment |
| CN113569688A (en) * | 2021-07-21 | 2021-10-29 | 上海健指树健康管理有限公司 | Body fitness testing method and device based on limb recognition technology and storage medium |
| CN113849065A (en) * | 2021-09-17 | 2021-12-28 | 支付宝(杭州)信息技术有限公司 | Method and device for triggering client operation instruction by using body-building action |
| CN113900521A (en) * | 2021-09-30 | 2022-01-07 | 上海千丘智能科技有限公司 | Interactive method and system for multi-person behavior training |
| CN114115536A (en) * | 2021-11-22 | 2022-03-01 | 北京字节跳动网络技术有限公司 | Interaction method, interaction device, electronic equipment and storage medium |
| CN114327062A (en) * | 2021-12-28 | 2022-04-12 | 深圳Tcl新技术有限公司 | Man-machine interaction method, device, electronic equipment, storage medium and program product |
| CN115530855A (en) * | 2022-09-30 | 2022-12-30 | 先临三维科技股份有限公司 | Control method and device of three-dimensional data acquisition equipment and three-dimensional data acquisition equipment |
| CN118264858A (en) * | 2024-05-29 | 2024-06-28 | 深圳爱图仕创新科技股份有限公司 | Data processing method, device, computer equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108616712A (en) * | 2018-04-18 | 2018-10-02 | 深圳中电数码显示有限公司 | A kind of interface operation method, device, equipment and storage medium based on camera |
| CN108629306A (en) * | 2018-04-28 | 2018-10-09 | 北京京东金融科技控股有限公司 | Human posture recognition method and device, electronic equipment, storage medium |
| CN109951628A (en) * | 2017-12-21 | 2019-06-28 | 广东欧珀移动通信有限公司 | Model building method, photographic method, device, storage medium and terminal |
| WO2019127271A1 (en) * | 2017-12-28 | 2019-07-04 | 深圳市锐明技术股份有限公司 | Warning method, device, storage medium and server regarding physical conflict behavior |
-
2019
- 2019-07-31 CN CN201910704604.7A patent/CN112306220A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109951628A (en) * | 2017-12-21 | 2019-06-28 | 广东欧珀移动通信有限公司 | Model building method, photographic method, device, storage medium and terminal |
| WO2019127271A1 (en) * | 2017-12-28 | 2019-07-04 | 深圳市锐明技术股份有限公司 | Warning method, device, storage medium and server regarding physical conflict behavior |
| CN108616712A (en) * | 2018-04-18 | 2018-10-02 | 深圳中电数码显示有限公司 | A kind of interface operation method, device, equipment and storage medium based on camera |
| CN108629306A (en) * | 2018-04-28 | 2018-10-09 | 北京京东金融科技控股有限公司 | Human posture recognition method and device, electronic equipment, storage medium |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113014815A (en) * | 2021-03-03 | 2021-06-22 | 南京地平线机器人技术有限公司 | Method and device for controlling equipment in closed space, medium and equipment |
| CN113014815B (en) * | 2021-03-03 | 2022-06-17 | 南京地平线机器人技术有限公司 | Method and device for controlling equipment in closed space, medium and equipment |
| CN113569688A (en) * | 2021-07-21 | 2021-10-29 | 上海健指树健康管理有限公司 | Body fitness testing method and device based on limb recognition technology and storage medium |
| CN113849065A (en) * | 2021-09-17 | 2021-12-28 | 支付宝(杭州)信息技术有限公司 | Method and device for triggering client operation instruction by using body-building action |
| WO2023040449A1 (en) * | 2021-09-17 | 2023-03-23 | 支付宝(杭州)信息技术有限公司 | Triggering of client operation instruction by using fitness action |
| CN113900521A (en) * | 2021-09-30 | 2022-01-07 | 上海千丘智能科技有限公司 | Interactive method and system for multi-person behavior training |
| CN114115536A (en) * | 2021-11-22 | 2022-03-01 | 北京字节跳动网络技术有限公司 | Interaction method, interaction device, electronic equipment and storage medium |
| CN114327062A (en) * | 2021-12-28 | 2022-04-12 | 深圳Tcl新技术有限公司 | Man-machine interaction method, device, electronic equipment, storage medium and program product |
| CN115530855A (en) * | 2022-09-30 | 2022-12-30 | 先临三维科技股份有限公司 | Control method and device of three-dimensional data acquisition equipment and three-dimensional data acquisition equipment |
| WO2024067027A1 (en) * | 2022-09-30 | 2024-04-04 | 先临三维科技股份有限公司 | Control method and apparatus for three-dimensional data acquisition device, and three-dimensional data acquisition device |
| CN118264858A (en) * | 2024-05-29 | 2024-06-28 | 深圳爱图仕创新科技股份有限公司 | Data processing method, device, computer equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112306220A (en) | Control method and device based on limb identification, electronic equipment and storage medium | |
| US20180329209A1 (en) | Methods and systems of smart eyeglasses | |
| US11720814B2 (en) | Method and system for classifying time-series data | |
| CN112154431B (en) | Man-machine interaction method and electronic equipment | |
| CN112860169B (en) | Interaction method and device, computer readable medium and electronic equipment | |
| CN111107278B (en) | Image processing method and device, electronic equipment and readable storage medium | |
| WO2022095674A1 (en) | Method and apparatus for operating mobile device | |
| CN111968647A (en) | Voice recognition method, device, medium and electronic equipment | |
| CN112306235A (en) | Gesture operation method, device, equipment and storage medium | |
| CN117615200A (en) | Video generation method, device, computer readable storage medium and electronic equipment | |
| CN111160047A (en) | Data processing method and device and data processing device | |
| CN111443801A (en) | Man-machine interaction method, device, equipment and storage medium | |
| CN111968641B (en) | Voice assistant awakening control method and device, storage medium and electronic equipment | |
| CN112270302A (en) | Limb control method and device and electronic equipment | |
| CN112036174B (en) | Punctuation marking method and device | |
| CN110555329A (en) | Sign language translation method, terminal and storage medium | |
| CN111382748A (en) | Image translation method, device and storage medium | |
| CN113345452B (en) | Voice conversion method, training method, device and medium of voice conversion model | |
| CN110851032A (en) | Display style adjustment method and device for target device | |
| CN112309384A (en) | Voice recognition method, device, electronic equipment and medium | |
| CN112818841B (en) | Method and related device for identifying emotion of user | |
| CN112000766B (en) | Data processing method, device and medium | |
| CN111128131B (en) | Voice recognition method and device, electronic equipment and computer readable storage medium | |
| CN112242143B (en) | Voice interaction method and device, terminal equipment and storage medium | |
| CN112382292A (en) | Voice-based control method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210202 |