CN109032358B - Control method and device of AR interaction virtual model based on gesture recognition - Google Patents
Control method and device of AR interaction virtual model based on gesture recognition Download PDFInfo
- Publication number
- CN109032358B CN109032358B CN201810980628.0A CN201810980628A CN109032358B CN 109032358 B CN109032358 B CN 109032358B CN 201810980628 A CN201810980628 A CN 201810980628A CN 109032358 B CN109032358 B CN 109032358B
- Authority
- CN
- China
- Prior art keywords
- gesture
- control command
- group
- gestures
- gesture group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention provides a control method and a device of an AR interaction virtual model based on gesture recognition, wherein the method comprises the following steps: acquiring continuous video images by shooting gestures of a user; identifying a first gesture group from the continuous video images, the first gesture group comprising one or more gestures of the user; acquiring a first control command corresponding to a first gesture by searching a mapping relation between a preset gesture group and the control command; and controlling the virtual model in the AR scene to execute corresponding operation according to the first control command. The control method and device of the AR interaction virtual model based on gesture recognition can determine the control command according to the gesture group of the user and control the operation corresponding to the virtual model in the AR scene according to the control command corresponding to the gesture group, so that the complexity of the operation device and the operation steps when the AR interacts with the user is reduced, and the AR interaction efficiency is improved.
Description
Technical Field
The invention relates to the technical field of Augmented Reality (AR), in particular to a method and a device for controlling an AR interaction virtual model based on gesture recognition.
Background
Currently, augmented Reality (AR) is a technology for increasing the perception of a user to the real world through information provided by a computer system. The AR technology can apply virtual information to the real world and superimpose computer-generated virtual object, scene or system prompt information onto the real scene that a user can see through AR display devices such as AR glasses, thereby achieving enhancement of the user's realistic perception.
In the conventional AR technology, in order to improve the user experience, in addition to simply superimposing a virtual AR scene on an AR display device, a user may operate a control device such as a handle of the AR display device to move a specific AR virtual model in the AR scene. For example, the user controls the AR virtual character in the AR scene to move in the AR scene by operating the direction key of the handle.
However, in the prior art, a control device such as a handle is usually required to be arranged outside the AR display device to enable the user to operate the AR virtual model in the AR scene. When a user watches an AR scene through an AR display device, the user needs to operate the control device, so that the operation device and the operation steps are complex when the AR interacts with the user, and the efficiency of AR interaction is low.
Disclosure of Invention
The invention provides a control method and device of an AR interaction virtual model based on gesture recognition, which aim to reduce the complexity of an operation device and operation steps when an AR interacts with a user and improve the efficiency of AR interaction.
The invention provides a control method of an AR interaction virtual model based on gesture recognition, which comprises the following steps:
acquiring continuous video images by shooting gestures of a user;
identifying a first set of gestures from the continuous video image, the first set of gestures including one or more gestures of the user;
acquiring a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
and controlling a virtual model in the AR scene to execute corresponding operation according to the first control command.
In an embodiment of the first aspect of the present invention, the mapping relationship between the gesture group and the control command includes:
mapping between the changing characteristics of a plurality of gestures in the gesture group and the control command.
In an embodiment of the first aspect of the present invention, the controlling, according to the first control command, a virtual model in an AR scene to perform a corresponding operation includes:
controlling movement of the virtual model in the AR scene according to the first control command.
In an embodiment of the first aspect of the present invention, the controlling, according to the first control command, a virtual model in an AR scene to perform a corresponding operation includes:
and controlling the scaling of the virtual model in the AR scene according to the first control command.
In an embodiment of the first aspect of the present invention, the changing characteristics of the plurality of gestures in the gesture group include at least one or more of the following:
a characteristic of variation of one finger in each gesture of the group of gestures, a characteristic of variation of a plurality of fingers in each gesture of the group of gestures, a characteristic of variation of joints of a single finger in each gesture of the group of gestures, and a characteristic of variation of joints of a plurality of fingers in each gesture of the group of gestures.
In an embodiment of the first aspect of the present invention, the method further includes:
acquiring a mapping relation between at least one gesture group and a control command;
and determining the mapping relation between the preset gesture group and the control command according to the mapping relation between at least one gesture group and the control command.
In an embodiment of the first aspect of the present invention, after acquiring the continuous video images by capturing the gesture of the user, the method further includes:
judging whether the continuous video images comprise the gesture of the user or not;
if so, acquiring a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
if not, acquiring continuous video images again by shooting the gestures of the user.
In an embodiment of the first aspect of the present invention, the acquiring consecutive video images by capturing gestures of a user includes:
acquiring shot video data within a preset time range;
the successive video images are acquired from the video data.
In an embodiment of the first aspect of the present invention, the obtaining a first control command corresponding to the first gesture by searching for a mapping relationship between a preset gesture group and a control command includes:
sending the first gesture group to a server so that the server obtains a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
and receiving the first control command sent by the server.
The second aspect of the present invention provides a control apparatus for an AR interaction virtual model based on gesture recognition, including:
the camera is used for acquiring continuous video images by shooting gestures of a user;
a processor configured to identify a first set of gestures from the continuous video image, the first set of gestures including one or more gestures of the user;
the processor is further used for obtaining a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
and the AR display is used for controlling the virtual model in the AR scene to execute corresponding operation according to the first control command.
In an embodiment of the second aspect of the present invention, the mapping relationship between the gesture group and the control command includes:
mapping between the changing characteristics of a plurality of gestures in the gesture group and the control command.
In an embodiment of the second aspect of the present invention, the AR display is specifically configured to control the movement of the virtual model in the AR scene according to the first control command.
In an embodiment of the second aspect of the present invention, the AR display is specifically configured to control zooming of the virtual model in the AR scene according to the first control command.
In an embodiment of the second aspect of the present invention, the changing characteristics of the plurality of gestures in the gesture group include at least one or more of the following:
a change characteristic of one finger in each gesture of the group of gestures, a change characteristic of a plurality of fingers in each gesture of the group of gestures, a change characteristic of joints of a single finger in each gesture of the group of gestures, and a change characteristic of joints of a plurality of fingers in each gesture of the group of gestures.
In an embodiment of the second aspect of the present invention, the method further includes:
the receiver is used for acquiring a mapping relation between at least one gesture group and the control command;
the processor is further configured to determine a mapping relationship between the preset gesture group and the control command according to a mapping relationship between at least one gesture group and the control command.
In an embodiment of the second aspect of the present invention, the processor is further configured to,
judging whether the continuous video images comprise the gestures of the user or not;
if yes, the processor obtains a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
and if not, the camera acquires continuous video images again by shooting the gestures of the user.
In an embodiment of the second aspect of the present invention, the camera is specifically configured to,
acquiring shot video data within a preset time range;
the successive video images are acquired from the video data.
In an embodiment of the second aspect of the present invention, the method further includes:
the transmitter is used for transmitting the first gesture group to a server so that the server can obtain a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and a control command;
the receiver is further configured to receive the first control command sent by the server.
In a third aspect, an embodiment of the present application provides a control apparatus for an AR interaction virtual model based on gesture recognition, including: a processor and a memory; the memory is used for storing programs; the processor is configured to call a program stored in the memory to perform the method according to any one of the first aspect of the present application.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing program code, which when executed, performs the method according to any one of the first aspect of the present application.
In summary, the present invention provides a method and an apparatus for controlling an AR interaction virtual model based on gesture recognition, wherein the method comprises: acquiring continuous video images by shooting gestures of a user; identifying a first set of gestures from the continuous video image, the first set of gestures including one or more gestures of the user; acquiring a first control command corresponding to a first gesture by searching a mapping relation between a preset gesture group and the control command; and controlling the virtual model in the AR scene to execute corresponding operation according to the first control command. The control method and device of the AR interaction virtual model based on gesture recognition can determine the control command according to the gesture group of the user and control the operation corresponding to the virtual model in the AR scene according to the control command corresponding to the gesture group, so that the complexity of the operation device and the operation steps when the AR interacts with the user is reduced, and the AR interaction efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic flowchart illustrating an embodiment of a method for controlling an AR interaction virtual model based on gesture recognition according to the present invention;
FIG. 2 is a schematic diagram illustrating an application of the control method for the AR interaction virtual model based on gesture recognition according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a mapping relationship between gesture groups and control commands according to the present application;
FIG. 4 is a schematic diagram illustrating a virtual model in an AR scene according to the present application;
FIG. 5 is a schematic structural diagram of an embodiment of an AR display device according to the present application;
FIG. 6 is a schematic structural diagram of an embodiment of an AR display device according to the present application;
FIG. 7 is a schematic structural diagram of an embodiment of an AR display device according to the present application;
FIG. 8 is a schematic structural diagram of an embodiment of a control apparatus for an AR interaction virtual model based on gesture recognition according to the present invention;
FIG. 9 is a schematic structural diagram of an embodiment of a control apparatus for an AR interaction virtual model based on gesture recognition according to the present invention;
FIG. 10 is a schematic structural diagram of an embodiment of a control apparatus for an AR interaction virtual model based on gesture recognition according to the present invention;
fig. 11 is a schematic structural diagram of a control apparatus of an AR interaction virtual model based on gesture recognition according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical means of the present invention will be described in detail with reference to specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 1 is a schematic flowchart illustrating a control method of an AR interactive virtual model based on gesture recognition according to an embodiment of the present invention. As shown in fig. 1, the method for controlling an AR interaction virtual model based on gesture recognition provided in this embodiment includes:
s101: continuous video images are acquired by photographing the gestures of the user.
S102: a first set of gestures is identified from the continuous video images, the first set of gestures including one or more gestures of the user.
S103: and acquiring a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command.
S104: and controlling the virtual model in the AR scene to execute corresponding operation according to the first control command.
Specifically, fig. 2 is a schematic application diagram of an embodiment of the control method of the AR interaction virtual model based on gesture recognition. The method in fig. 1 is described below with reference to the application scenario diagram shown in fig. 2.
As shown in fig. 2, in the application scenario of the present embodiment, the execution subject of the embodiment of the method is an AR display apparatus 1, and the AR display apparatus may be any electronic device with an AR display function, for example: the user can view the virtual model in the AR scene through the AR display device 1, such as a mobile phone, a tablet computer, a desktop computer, or AR glasses. Taking the AR display device 1 as AR glasses as an example in the example shown in fig. 2, the user 2 can view an AR scene in the AR display device 1 through the AR display device 1 worn.
For example, the first AR scene 11 as shown in fig. 2 is a default AR scene that the AR display device provides to the user 2. The user 2 is at the sea, the actual picture that the user can see is the sea surface, and the AR display device 1 loads the first image 101 of the virtual model ship to the lower left corner of the actual picture that the user can see. So that the user can see the first AR scene as a virtual ship superimposed on the actual sea surface view through the AR display device 1.
The position of the virtual model provided by the AR display apparatus 1 and some attributes (size, color, brightness, etc.) are generally not changed by default, and if the user wants to adjust the position or the attributes of the virtual model in the AR scene, in the prior art, some AR display apparatuses provide a special operation apparatus, and the user operates the apparatus to implement the operation on the virtual model in the AR scene. In this embodiment, the method for controlling the virtual model in the AR scene to execute the corresponding operation by the AR display device itself according to the gesture of the user is provided.
First, in S101 of the present embodiment, the AR display device 1 may acquire continuous video images by capturing gestures of the user 2. The AR display device 1 may capture the hand of the user 2 through a camera or a camera module, etc. to obtain a continuous video image. The continuous video image here may be a video of the hand of the user 2 taken for a period of time, or the continuous video image may be one or more images of the hand of the user 2 taken at intervals of a preset time. Alternatively, the AR display device 1 may execute S101 at a specific time or under a specific condition, for example, execute S101 at intervals to capture the gesture of the user 2, or execute S101 when the head lowering motion of the user 2 is detected, or execute S101 when the user presses a switch on the AR display device 1.
Alternatively, the continuous video image acquired by the AR display device in S101 is shot video data in a preset time range, and the AR display device first acquires the video data in the preset time range and then extracts the framed video image in the video from the video data. The preset time here may be a delay time set by a user, or a time specified in an AR scene.
In S102, the AR display device 1 identifies a first gesture group from the continuous video images acquired in S101 by means of gesture recognition, wherein the first gesture group includes one or more gestures of the user. Alternatively, in S102, the processor of the AR display apparatus 1 identifies each of the images of the consecutive videos, or identifies the image of each frame of the consecutive videos. And recognizing the gestures of all images in the continuous video images through an image processing technology, and performing induction and de-duplication to recognize the gestures continuously made by the user. And the gesture of the user can be represented by the simultaneous change of more fingers and the movement of more finger joints of the hand of the user in the embodiment shown in the application. Wherein the change characteristic of the gesture group may be a change characteristic of one finger in each gesture of the gesture group, a change characteristic of a plurality of fingers in each gesture of the gesture group, a change characteristic of joints of a single finger in each gesture of the gesture group, and a change characteristic of joints of a plurality of fingers in each gesture of the gesture group. Optionally, the recognition mode in S102 may be an image processing mode based on deep learning, and the gesture in the video image is determined according to the calculation result of the deep learning by using the video image as input.
For example: in the example shown in fig. 2, the first gesture group 3 comprises a first gesture 31, a second gesture 32 and a third gesture 33. The number of consecutive video images captured by the AR display device may be greater than 3, for example, 10 consecutive video images. The AR display device 1 recognizes 10 images of gestures included in all of the 10 video images, and obtains a first gesture group including 3 of the three gestures described above in the 10 images of gestures. In the first gesture group shown in fig. 2, the state in which the hand of the user changes from the unfolded state to the fist-opened state is defined as the first gesture group 3, that is, the first gesture group 3 can identify the change of the gesture of the user by the first gesture 31, the second gesture 32, and the third gesture 33 arranged in this order. It should be noted that, as shown in fig. 2, the first gesture group 3 is only a left hand of a user, and three gestures in the gesture group are only examples, and likewise, the first gesture group 3 may include one or more gestures, or the first gesture group may include a left-hand gesture, a right-hand gesture, or a two-hand gesture, and specific implementation is only changes in number and is not described again.
In S103, the AR display device 1 obtains a first control command corresponding to the first gesture by searching a mapping relationship between a preset gesture group and the control command according to the first gesture group of the user identified in S102. The AR display apparatus 1 stores a mapping relationship between a preset gesture group and a control command, and specifically, the mapping relationship refers to a mapping relationship between a change characteristic of a plurality of gestures in the gesture group and the control command.
For example, fig. 3 is a schematic diagram illustrating a mapping relationship between a gesture group and a control command according to the present application. Four groups of gestures as shown in FIG. 3: gesture group 1, gesture group 2, gesture group 3, and gesture group 4 may be exemplarily used as a mapping relationship between a preset gesture group stored in the AR display device and a control command. Wherein, 3 gestures exist in the gesture group 1, which sequentially presents the state from the unfolding to the fist making of the left hand of the user, that is, when the change characteristic of the left hand of the user is the state from the unfolding to the fist making, the control command corresponding to the change characteristic is the reduction of the virtual model. In gesture group 2, when the change feature of the right hand of the user is from fist to the state that the thumb extends to the right side, the change feature corresponds to the right shift of the virtual model. The change characteristic of the left hand of the user in the gesture group 3 is that 1-4 fingers are sequentially extended, which corresponds to the change of the serial number of the virtual model, and each gesture in the gesture group 3 can correspond to the serial number of one virtual model respectively. The gestures in the gesture group 4 are the two hands of the user, the thumb and the forefinger of the left hand of the user extend out, the brightness of the virtual model corresponds to, and the brightness of the right hand decreases from the fist making state to the thumb extending state correspondingly. It should be noted that the preset gesture group may further include a left hand gesture, a right hand gesture, or a two-hand gesture of the user, and the specific implementation is only the change in quantity and is not described any more.
In the embodiment shown in fig. 2, after the first gesture group is compared with the preset gesture group in S103, it is determined that the first gesture group in fig. 2 is the gesture group 1 in the preset gesture group, and it is determined that the first control command corresponding to the first gesture group is the control command "virtual model zoom-out" corresponding to the gesture group 1. And, optionally, the number of gesture groups specifically included in the preset gesture groups stored by the AR display device as shown in fig. 3 may be different from the number of specific gestures in the first gesture group recognized in S102. For example, the corresponding gesture in the gesture group 1 includes three gestures from the state of spreading out to the state of making a fist, and if only two gestures of the spreading out state and the state of making a fist are included in the first gesture group recognized in S102, the changed state that can also be recognized as the corresponding gesture is the gesture group 1. The other gesture groups are the same as the above embodiments, and are not described again.
In S104, the AR display apparatus controls the virtual model in the AR scene to perform a corresponding operation according to the first control command determined in S103. For example, in the above embodiment, the control command corresponding to the first gesture group determined in S103 is "virtual model reduction". Then applied to the embodiment shown in fig. 2, the user can see through the AR display device 1 that the first AR scene 11 is a virtual model of a vessel superimposing the first image 101 for a picture of the actual sea surface. The AR display apparatus reduces the virtual model of the first image 101 according to the first control command, resulting in the second AR scene 12 as shown in fig. 2, and it can be seen that the size of the virtual model in the AR scene presented to the user in the second AR scene 12 is reduced.
Meanwhile, fig. 4 is a schematic diagram illustrating operations performed by the virtual model in the AR scene according to the present application, and illustrates an operation manner of the virtual model corresponding to the control command in another 3. As shown in fig. 4, when the control command is a virtual model enlargement, the AR display device enlarges the virtual model of the first image 101, resulting in the second AR scene 12 shown in fig. 2. When the control command is to move the virtual model to the right, the AR display device moves the vessel of the first image 101 to the rightmost side of the AR scene to the right to get the second AR scene 12. When the control command is a virtual model number, the AR display device may replace the virtual model 101 in the first AR scene 11 with the virtual model 102, showing a different vessel in the second AR scene 12 than the virtual model in the first AR scene 11.
Therefore, in summary, in the control method for the AR interaction virtual model based on gesture recognition provided in this embodiment, the control command can be determined according to the gesture group of the user, and the operation corresponding to the virtual model in the AR scene is controlled according to the control command corresponding to the gesture group. Compared with the prior art, the method and the device have the advantages that the device can be controlled less, and the user can operate the virtual model according to simple and direct gesture changes. Therefore, the complexity of the operation device and the operation steps when the AR interacts with the user is reduced, and the efficiency of AR interaction is improved.
Further, in the above embodiment, in order to enable the AR display device to determine the first control command corresponding to the first gesture group according to the mapping relationship between the preset gesture group and the control command. For the AR display device, a specific gesture group may also be added in a learning entry manner, for example, when the user operates the AR display device, the entered control instruction is selected to be shutdown, and a gesture is made, for example, two hands are closed. The AR display device can acquire a gesture group with two hands closing the gesture through a camera, and a mapping relation is established between the gesture group and the control instruction light machine. The AR display device stores the acquired mapping relationship between all the at least one gesture group and the control command in a memory of the AR display device to obtain the relationship between the preset gesture group and the control command in the above embodiment.
Fig. 5 is a schematic structural diagram of an AR display device according to an embodiment of the present application. Fig. 5 shows a manner of providing the camera 10 on the AR display device 1, in which the AR display device itself may employ AR glasses in the prior art, and fig. 5 shows an existing AR glasses. And this application can set up the camera and point to user's sight direction at AR display device edge on current AR glasses basis. Fig. 6 is a schematic structural diagram of an embodiment of an AR display device according to the present application, where the AR display device in the embodiment shown in fig. 6 further includes a processor for executing the steps of S102 and S103, and a transceiver for connecting to the internet through a wireless network or a wired network.
In the above embodiment, the user sometimes cannot aim the camera at both hands completely to capture continuous video images. After the AR display apparatus acquires the continuous video images in S101, it may be further determined whether the acquired continuous images include a gesture image of the user. If yes, the mapping relationship between the preset gesture group and the control command is searched in S102, and the first control command corresponding to the group of gestures is obtained. If not, the AR display device needs to perform S101 to acquire continuous video images by capturing the gesture of the user, and optionally prompts, by means of a screen or an indicator light or the like, that the user does not acquire a video image containing the gesture and needs to acquire again.
Further, fig. 7 is a schematic structural diagram of an embodiment of an AR display device according to the present application. In the embodiment shown in fig. 7, another way of executing S103 in the above embodiment shown in fig. 1 by a server is shown. Specifically, as shown in fig. 7, after the AR display device 1 acquires the first gesture group, the first gesture group is transmitted to the server from a wireless network or a wired network through the transceiver. The server specially used for recognizing the gestures obtains a first control command corresponding to a group of gestures by searching the mapping relation between a preset gesture group and a control command stored in the server, and sends the determined first control command to the AR display device. The AR display device receives the first control instruction sent by the server through the transceiver and then performs the relevant operation in S104. Therefore, the embodiment prevents the server from executing the specific gesture comparison and analysis step, so as to reduce the performance requirement and the storage requirement on the AR display device, and the specially-arranged server specially determines the first control command corresponding to the first gesture group, so that the processing efficiency of the recognition and determination can be improved to a certain extent.
Fig. 8 is a schematic structural diagram of a control apparatus of an AR interaction virtual model based on gesture recognition according to an embodiment of the present invention. As shown in fig. 8, the control device based on the AR interaction virtual model of another gesture provided in this embodiment includes: an acquisition module 801, a processing module 802 and a display module 803. Wherein, the first and the second end of the pipe are connected with each other,
the acquiring module 801 is configured to acquire continuous video images by capturing gestures of a user;
the processing module 802 is configured to identify a first gesture group from the continuous video images, where the first gesture group includes one or more gestures of the user;
the processing module 802 is further configured to obtain a first control command corresponding to the first gesture by searching for a mapping relationship between a preset gesture group and the control command;
the display module 803 is configured to control the virtual model in the AR scene to perform a corresponding operation according to the first control command.
The control device of the AR interaction virtual model based on gesture recognition provided in this embodiment may be used to execute the control method of the AR interaction virtual model based on gesture recognition shown in fig. 1, and the specific implementation manner and principle thereof are the same, and are not described again.
Optionally, in the above embodiment, the mapping relationship between the gesture group and the control command includes:
and mapping relation between the change characteristics of the plurality of gestures in the gesture group and the control command.
Optionally, in the above embodiment, the display module 803 is specifically configured to control the movement of the virtual model in the AR scene according to the first control command.
Optionally, in the foregoing embodiment, the display module 803 is specifically configured to control, according to the first control command, scaling of the virtual model in the AR scene.
Optionally, in the above embodiment, the variation characteristics of the plurality of gestures in the gesture group include at least one or more of the following:
the change characteristic of one finger in each gesture of the gesture group, the change characteristic of a plurality of fingers in each gesture of the gesture group, the change characteristic of joints of a single finger in each gesture of the gesture group, and the change characteristic of joints of a plurality of fingers in each gesture of the gesture group.
The control device of the AR interaction virtual model based on gesture recognition provided in this embodiment may be used to execute the control method of the AR interaction virtual model based on gesture recognition shown in the foregoing embodiments, and a specific implementation manner and a principle thereof are the same, and are not described again.
Fig. 9 is a schematic structural diagram of a control apparatus of an AR interaction virtual model based on gesture recognition according to an embodiment of the present invention. The apparatus shown in fig. 9 is based on the apparatus shown in fig. 8, and further comprises: a receiving module 901 and a sending module 902. Wherein, the first and the second end of the pipe are connected with each other,
the receiving module 901 is configured to obtain a mapping relationship between at least one gesture group and a control command;
the processing module 802 is further configured to determine a mapping relationship between a preset gesture group and a control command according to the mapping relationship between at least one gesture group and the control command.
Optionally, in the above-described embodiment, the processing module 802 is further configured to,
judging whether the continuous video images comprise gestures of the user or not;
if yes, the processing module 802 obtains a first control command corresponding to the first gesture by searching a mapping relationship between a preset gesture group and the control command;
if not, the obtaining module 801 obtains continuous video images again by capturing gestures of the user.
Optionally, in the above embodiment, the obtaining module 801 is specifically configured to,
acquiring shot video data within a preset time range;
successive video images are acquired from the video data.
Optionally, in the foregoing embodiment, the sending module 902 is configured to send the first gesture group to the server, so that the server obtains the first control command corresponding to the first gesture by searching a mapping relationship between a preset gesture group and the control command;
the receiving module 901 is further configured to receive a first control command sent by the server.
The control device of the AR interaction virtual model based on gesture recognition provided in this embodiment may be used to execute the control method of the AR interaction virtual model based on gesture recognition shown in the foregoing embodiments, and a specific implementation manner and a principle thereof are the same, and are not described again.
It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and another division manner may be available in actual implementation. Each functional module in the embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.
Fig. 10 is a schematic structural diagram of a control apparatus of an AR interaction virtual model based on gesture recognition according to an embodiment of the present invention. As shown in fig. 10, the control device based on the AR interaction virtual model of another gesture provided in this embodiment includes: a camera 1001, a processor 1002 and an AR display 1003. Wherein the content of the first and second substances,
the camera 1001 is used for acquiring continuous video images by shooting gestures of a user;
the processor 1002 is configured to identify a first set of gestures from the continuous video image, the first set of gestures including one or more gestures of the user;
the processor 1002 is further configured to obtain a first control command corresponding to the first gesture by searching for a mapping relationship between a preset gesture group and the control command;
the AR display 1003 is configured to control the virtual model in the AR scene to perform a corresponding operation according to the first control command.
The control device of the AR interaction virtual model based on gesture recognition provided in this embodiment may be used to execute the control method of the AR interaction virtual model based on gesture recognition shown in fig. 1, and a specific implementation manner and a principle thereof are the same and are not described again.
Optionally, in the above embodiment, the mapping relationship between the gesture group and the control command includes:
and mapping relation between the change characteristics of the plurality of gestures in the gesture group and the control command.
Optionally, in the above embodiment, the AR display 1003 is specifically configured to control the movement of the virtual model in the AR scene according to the first control command.
Optionally, in the above embodiment, the AR display 1003 is specifically configured to control, according to the first control command, zooming of the virtual model in the AR scene.
Optionally, in the above embodiment, the variation characteristics of the plurality of gestures in the gesture group include at least one or more of the following:
the change characteristic of one finger in each gesture of the gesture group, the change characteristic of a plurality of fingers in each gesture of the gesture group, the change characteristic of joints of a single finger in each gesture of the gesture group, and the change characteristic of joints of a plurality of fingers in each gesture of the gesture group.
The control device of the AR interaction virtual model based on gesture recognition provided in this embodiment may be used to execute the control method of the AR interaction virtual model based on gesture recognition shown in the foregoing embodiments, and a specific implementation manner and a principle thereof are the same, and are not described again.
Fig. 11 is a schematic structural diagram of a control apparatus of an AR interaction virtual model based on gesture recognition according to an embodiment of the present invention. The apparatus shown in fig. 11 is based on the apparatus shown in fig. 10, and further includes: a receiver 1004 and a transmitter 1005. Wherein, the first and the second end of the pipe are connected with each other,
the receiver 1004 is configured to obtain a mapping relationship between at least one gesture group and a control command;
the processor 1002 is further configured to determine a mapping relationship between a preset gesture group and a control command according to the mapping relationship between at least one gesture group and the control command.
Optionally, in the above embodiment, the processor 1002 is further configured to,
judging whether the continuous video images comprise gestures of the user or not;
if yes, the processor 1002 obtains a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
if not, the camera 1001 acquires continuous video images again by shooting the gestures of the user.
Alternatively, in the above-described embodiment, the camera 1001 is specifically used for,
acquiring shot video data within a preset time range;
successive video images are acquired from the video data.
Optionally, in the above embodiment, the transmitter 1005 is configured to transmit the first gesture group to the server, so that the server obtains the first control command corresponding to the first gesture by searching a mapping relationship between a preset gesture group and the control command;
the receiver 1004 is further configured to receive a first control command sent by the server.
The control device of the AR interaction virtual model based on gesture recognition provided in this embodiment may be used to execute the control method of the AR interaction virtual model based on gesture recognition shown in the foregoing embodiments, and a specific implementation manner and a principle thereof are the same, and are not described again.
The present invention also provides an electronic device readable storage medium, which includes a program, and when the program runs on an electronic device, the program causes the electronic device to execute the control method of the AR interaction virtual model based on gesture recognition according to any of the above embodiments.
An embodiment of the present invention further provides an electronic device, including: a processor; and a memory for storing executable instructions for the processor; wherein the processor is configured to execute the control method of the AR interaction virtual model based on gesture recognition in any one of the above embodiments via execution of the executable instructions.
An embodiment of the present invention also provides a program product, including: a computer program (i.e., executing instructions) stored in a readable storage medium. The computer program may be read from a readable storage medium by at least one processor of the encoding device, and execution of the computer program by the at least one processor causes the encoding device to implement the control method of the AR interaction virtual model based on gesture recognition provided by the various embodiments described above.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and any simple modification, equivalent change and modification made to the above embodiment according to the technical spirit of the present invention are still within the scope of the technical solution of the present invention.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (16)
1. A control method of an Augmented Reality (AR) interaction virtual model based on gesture recognition is applied to an AR display device, and is characterized in that at least one preset gesture group is stored in the AR display device, and the method comprises the following steps:
acquiring continuous video images by shooting gestures of a user;
identifying a first gesture group from the continuous video images, the first gesture group comprising a plurality of gestures of the user;
acquiring a first control command corresponding to the first gesture group by searching a mapping relation between the at least one preset gesture group and the control command; each gesture group in the at least one preset gesture group consists of a plurality of gestures;
controlling a virtual model in the AR scene to execute corresponding operation according to the first control command;
the mapping relation between the gesture group and the control command comprises the following steps:
mapping relationships between varying characteristics of a plurality of gestures in the gesture group and the control commands;
the changing characteristics of a plurality of gestures in the gesture group at least comprise one or more of the following:
a change characteristic of one finger in each gesture of the group of gestures, a change characteristic of a plurality of fingers in each gesture of the group of gestures, a change characteristic of joints of a single finger in each gesture of the group of gestures, and a change characteristic of joints of a plurality of fingers in each gesture of the group of gestures;
the obtaining a first control command corresponding to the first gesture group by searching for a mapping relationship between the at least one preset gesture group and the control command includes:
selecting a gesture group containing all gesture states in the first gesture group from the at least one preset gesture group as a target gesture group;
determining the first control command according to the control command corresponding to the target gesture group;
wherein the number of gestures in the first gesture group is less than or equal to the number of gestures in the target gesture group;
before the obtaining of the first control command corresponding to the first gesture group by searching the mapping relationship between the at least one preset gesture group and the control command, the method further includes:
for each control command corresponding to a preset gesture group, receiving a plurality of gestures input by the user after the user selects the control command;
and obtaining a mapping relation between a preset gesture group and the control command according to the control command and the plurality of gestures input by the user.
2. The method of claim 1, wherein the controlling, according to the first control command, a virtual model in an AR scene to perform a corresponding operation comprises:
controlling movement of the virtual model in the AR scene according to the first control command.
3. The method of claim 1, wherein the controlling, according to the first control command, a virtual model in an AR scene to perform a corresponding operation comprises:
and controlling the scaling of the virtual model in the AR scene according to the first control command.
4. The method according to any one of claims 1-3, further comprising:
acquiring a mapping relation between at least one gesture group and a control command;
and determining the mapping relation between the preset gesture group and the control command according to the mapping relation between at least one gesture group and the control command.
5. The method according to any one of claims 1-3, wherein after acquiring the continuous video images by capturing the gesture of the user, further comprising:
judging whether the continuous video images comprise the gestures of the user or not;
if yes, obtaining a first control command corresponding to the first gesture group by searching a mapping relation between a preset gesture group and the control command;
if not, acquiring continuous video images again by shooting the gestures of the user.
6. The method according to any one of claims 1-3, wherein said acquiring successive video images by capturing gestures of a user comprises:
acquiring shot video data within a preset time range;
the successive video images are acquired from the video data.
7. The method according to any one of claims 1 to 3, wherein obtaining the first control command corresponding to the first gesture by searching a mapping relationship between a preset gesture group and a control command comprises:
sending the first gesture group to a server so that the server obtains a first control command corresponding to the first gesture group by searching a mapping relation between a preset gesture group and a control command;
and receiving the first control command sent by the server.
8. The utility model provides a controlling means of mutual virtual model of augmented reality AR based on gesture recognition, is applied to AR display device, its characterized in that, storage has at least one predetermined gesture group in the AR display device, controlling means includes:
the camera is used for acquiring continuous video images by shooting gestures of a user;
a processor configured to identify a first set of gestures from the continuous video image, the first set of gestures including a plurality of gestures of the user;
the processor is further configured to obtain a first control command corresponding to the first gesture group by searching for a mapping relationship between the at least one preset gesture group and the control command; each gesture group in the at least one preset gesture group consists of a plurality of gestures;
the AR display is used for controlling a virtual model in the AR scene to execute corresponding operation according to the first control command;
the mapping relation between the gesture group and the control command comprises the following steps:
mapping relationships between varying characteristics of a plurality of gestures in the gesture group and the control commands;
the changing characteristics of a plurality of gestures in the gesture group at least comprise one or more of the following:
a change characteristic of one finger in each gesture of the group of gestures, a change characteristic of a plurality of fingers in each gesture of the group of gestures, a change characteristic of joints of a single finger in each gesture of the group of gestures, and a change characteristic of joints of a plurality of fingers in each gesture of the group of gestures;
the processor is specifically configured to select, from the at least one preset gesture group, a gesture group including all gesture states in the first gesture group as a target gesture group, and determine the first control command according to a control command corresponding to the target gesture group, where a number of gestures in the first gesture group is less than or equal to a number of gestures in the target gesture group;
the processor is further configured to, before the first control command corresponding to the first gesture group is obtained by searching for the mapping relationship between the at least one preset gesture group and the control command, receive, for each control command corresponding to the preset gesture group, after the user selects the control command, the plurality of gestures entered by the user, and obtain, according to the control command and the plurality of gestures entered by the user, the mapping relationship between one preset gesture group and the control command.
9. The apparatus of claim 8, wherein the AR display is specifically configured to control movement of the virtual model in the AR scene according to the first control command.
10. The apparatus of claim 8, wherein the AR display is specifically configured to control a scaling of the virtual model in the AR scene according to the first control command.
11. The apparatus of any one of claims 8-10, further comprising:
the receiver is used for acquiring a mapping relation between at least one gesture group and the control command;
the processor is further configured to determine a mapping relationship between the preset gesture group and the control command according to a mapping relationship between at least one gesture group and the control command.
12. The apparatus of any of claims 8-10, wherein the processor is further configured to,
judging whether the continuous video images comprise the gesture of the user or not;
if so, the processor obtains a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
and if not, the camera acquires continuous video images again by shooting the gestures of the user.
13. The device according to any of claims 8 to 10, wherein the camera is particularly adapted to,
acquiring shot video data within a preset time range;
the successive video images are acquired from the video data.
14. The apparatus of any one of claims 8-10, further comprising:
the transmitter is used for transmitting the first gesture group to a server so that the server can obtain a first control command corresponding to the first gesture by searching a mapping relation between a preset gesture group and the control command;
a receiver, configured to receive the first control command sent by the server.
15. A control device of an Augmented Reality (AR) interaction virtual model based on gesture recognition is characterized by comprising: a processor, a memory, and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor, the computer program comprising instructions for performing the method of any of claims 1-7.
16. A computer-readable storage medium, characterized in that it stores a computer program that causes a server to execute the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810980628.0A CN109032358B (en) | 2018-08-27 | 2018-08-27 | Control method and device of AR interaction virtual model based on gesture recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810980628.0A CN109032358B (en) | 2018-08-27 | 2018-08-27 | Control method and device of AR interaction virtual model based on gesture recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109032358A CN109032358A (en) | 2018-12-18 |
| CN109032358B true CN109032358B (en) | 2023-04-07 |
Family
ID=64625083
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810980628.0A Active CN109032358B (en) | 2018-08-27 | 2018-08-27 | Control method and device of AR interaction virtual model based on gesture recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109032358B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710066B (en) * | 2018-12-19 | 2022-03-25 | 平安普惠企业管理有限公司 | Interaction method and device based on gesture recognition, storage medium and electronic equipment |
| CN111383345B (en) * | 2018-12-29 | 2022-11-22 | 广东虚拟现实科技有限公司 | Virtual content display method and device, terminal equipment and storage medium |
| CN109814717B (en) * | 2019-01-29 | 2020-12-25 | 珠海格力电器股份有限公司 | Household equipment control method and device, control equipment and readable storage medium |
| CN110321008B (en) * | 2019-06-28 | 2023-10-24 | 北京百度网讯科技有限公司 | Interaction method, device, equipment and storage medium based on AR model |
| CN111694427A (en) * | 2020-05-13 | 2020-09-22 | 北京农业信息技术研究中心 | AR virtual honey shake interactive experience system, method, electronic equipment and storage medium |
| CN111627097B (en) * | 2020-06-01 | 2023-12-01 | 上海商汤智能科技有限公司 | Virtual scene display method and device |
| CN111640200B (en) * | 2020-06-10 | 2024-01-09 | 浙江商汤科技开发有限公司 | AR scene special effect generation method and device |
| CN111766947A (en) * | 2020-06-30 | 2020-10-13 | 歌尔科技有限公司 | Display method, display device, wearable device and medium |
| CN112799507B (en) * | 2021-01-15 | 2022-01-04 | 北京航空航天大学 | Human body virtual model display method and device, electronic equipment and storage medium |
| CN113696850A (en) * | 2021-08-27 | 2021-11-26 | 上海仙塔智能科技有限公司 | Vehicle control method and device based on gestures and storage medium |
| CN114339039A (en) * | 2021-12-24 | 2022-04-12 | 北京百度网讯科技有限公司 | Virtual photographing method and device based on gesture recognition, electronic equipment and medium |
| CN114760440A (en) * | 2022-03-22 | 2022-07-15 | 浙江大学 | Visual monitoring glasses |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885316A (en) * | 2016-09-29 | 2018-04-06 | 阿里巴巴集团控股有限公司 | A kind of exchange method and device based on gesture |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885317A (en) * | 2016-09-29 | 2018-04-06 | 阿里巴巴集团控股有限公司 | A kind of exchange method and device based on gesture |
| CN107562198A (en) * | 2016-11-14 | 2018-01-09 | 上海闻泰电子科技有限公司 | VR apparatus control methods |
| CN106502424A (en) * | 2016-11-29 | 2017-03-15 | 上海小持智能科技有限公司 | Based on the interactive augmented reality system of speech gestures and limb action |
| CN106886285A (en) * | 2017-01-20 | 2017-06-23 | 西安电子科技大学 | A kind of historical relic interactive system and operating method based on virtual reality |
| CN107390881A (en) * | 2017-09-14 | 2017-11-24 | 西安领讯卓越信息技术有限公司 | A kind of gestural control method |
| CN108345387A (en) * | 2018-03-14 | 2018-07-31 | 百度在线网络技术（北京）有限公司 | Method and apparatus for output information |
-
2018
- 2018-08-27 CN CN201810980628.0A patent/CN109032358B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885316A (en) * | 2016-09-29 | 2018-04-06 | 阿里巴巴集团控股有限公司 | A kind of exchange method and device based on gesture |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109032358A (en) | 2018-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109032358B (en) | Control method and device of AR interaction virtual model based on gesture recognition | |
| JP6400197B2 (en) | Wearable device | |
| US9298266B2 (en) | Systems and methods for implementing three-dimensional (3D) gesture based graphical user interfaces (GUI) that incorporate gesture reactive interface objects | |
| IL261580A (en) | System and method for deep learning based hand gesture recognition in first person view | |
| US8379098B2 (en) | Real time video process control using gestures | |
| US20170192500A1 (en) | Method and electronic device for controlling terminal according to eye action | |
| US11782514B2 (en) | Wearable device and control method thereof, gesture recognition method, and control system | |
| CN111580652B (en) | Video playing control method and device, augmented reality equipment and storage medium | |
| CN107479691B (en) | Interaction method, intelligent glasses and storage device thereof | |
| US20160165170A1 (en) | Augmented reality remote control | |
| CN112954210B (en) | Photographing method and device, electronic equipment and medium | |
| US9607394B2 (en) | Information processing method and electronic device | |
| CN103581544A (en) | Dynamic region of interest adaptation and image capture device providing same | |
| EP2659336A1 (en) | User interface, apparatus and method for gesture recognition | |
| EP3153945A1 (en) | Method and device for implementing virtual button through camera, and computer-readable recording medium | |
| EP2939411B1 (en) | Image capture | |
| US20160147294A1 (en) | Apparatus and Method for Recognizing Motion in Spatial Interaction | |
| CN103873759B (en) | A kind of image pickup method and electronic equipment | |
| CN111913674A (en) | Virtual content display method, device, system, terminal equipment and storage medium | |
| US20210342624A1 (en) | System and method for robust image-query understanding based on contextual features | |
| CN107357424B (en) | Gesture operation recognition method and device and computer readable storage medium | |
| JP5558899B2 (en) | Information processing apparatus, processing method thereof, and program | |
| CN107340962B (en) | Input method and device based on virtual reality equipment and virtual reality equipment | |
| US9396405B2 (en) | Image processing apparatus, image processing method, and image processing program | |
| US10877641B2 (en) | Image adjustment method, apparatus, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |