CN110794959A - Gesture interaction AR projection method and device based on image recognition - Google Patents
Gesture interaction AR projection method and device based on image recognition Download PDFInfo
- Publication number
- CN110794959A CN110794959A CN201910909648.3A CN201910909648A CN110794959A CN 110794959 A CN110794959 A CN 110794959A CN 201910909648 A CN201910909648 A CN 201910909648A CN 110794959 A CN110794959 A CN 110794959A
- Authority
- CN
- China
- Prior art keywords
- gesture
- image
- user
- model
- model image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
Abstract
The invention relates to the technical field of AR projection, and discloses a gesture interaction AR projection method based on image recognition, which comprises the following steps: acquiring a 3D model image to be displayed from a 3D model image library and performing projection display; acquiring a user gesture video, and sequentially extracting a plurality of user gesture images from the user gesture video; respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline; generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information. The invention has the technical effect of high gesture interaction real-time performance.
Description
Technical Field
The invention relates to the technical field of AR projection, in particular to a gesture interaction AR projection method and device based on image recognition.
Background
AR (augmented reality), AR technology can apply virtual information to the real world. At present, AR technology is already applied to the fields of medical rehabilitation, games and the like. The integration of gesture recognition and AR technology can realize the interaction between the user and the virtual information, enhance the rehabilitation effect, the game experience and the like. However, at present, when performing gesture interaction based on image recognition, only some simple gesture interactions can be usually realized, and when performing complex gesture interaction, a large amount of image operations are usually required for support, which results in poor real-time performance and influences on interaction effect.
Disclosure of Invention
The invention aims to overcome the technical defects, provides a gesture interaction AR projection method and device based on image recognition, and solves the technical problem of poor real-time performance in complex gesture interaction in the prior art.
In order to achieve the technical purpose, the technical scheme of the invention provides a gesture interaction AR projection method based on image recognition, which comprises the following steps:
acquiring a 3D model image to be displayed from a 3D model image library and performing projection display;
acquiring a user gesture video, and sequentially extracting a plurality of user gesture images from the user gesture video;
respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline;
generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
The invention also provides a gesture interaction AR projection device based on image recognition, which comprises an AR projector, a camera and a 3D resource server;
the 3D resource server is used for establishing a 3D model image library;
the AR projector is used for acquiring a 3D model image to be displayed from the 3D model image library and performing projection display;
the camera is used for collecting a user gesture video;
the AR projector is also used for sequentially extracting a plurality of user gesture images from the user gesture video; respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline; generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
The invention also provides a computer storage medium on which a computer program is stored, wherein the computer program is executed by a processor to realize the gesture interaction AR projection method based on image recognition.
Compared with the prior art, the invention has the beneficial effects that: according to the method, a plurality of user gesture images in the user gesture video are extracted, and the extracted user gesture images are subjected to operation processing, so that the problems of large operation amount and complex operation caused by the operation processing of the whole user gesture video are solved. The gesture contour is extracted from the gesture image of the user, the gesture vector is generated according to the gesture contour, the gesture contour is replaced by the gesture vector, the gesture motion information of the user can be acquired according to the change of the gesture vector in the gesture images of the multiple users, the gesture vector not only can reflect the position change of the gesture of the user, but also can reflect the direction change of the gesture of the user, so that the motion information identified according to the gesture vector is richer, more and more complex gestures can be identified, and richer interaction functions are realized. Meanwhile, compared with a gesture outline, the gesture vector has a simple data structure, small data volume and convenience in operation, and is beneficial to improving the real-time performance of gesture interaction.
Drawings
FIG. 1 is a flowchart of an embodiment of a gesture interaction AR projection method based on image recognition provided by the present invention;
fig. 2 is a schematic structural diagram of an embodiment of a gesture interaction AR projection apparatus based on image recognition provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Example 1
As shown in fig. 1, embodiment 1 of the present invention provides an image recognition-based gesture interaction AR projection method, including the following steps:
s1, acquiring a 3D model image to be displayed from the 3D model image library and performing projection display;
s2, collecting a user gesture video, and sequentially extracting a plurality of user gesture images from the user gesture video;
s3, respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline;
and S4, generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
According to the gesture interaction AR projection method based on image recognition, AR projection display is carried out on a 3D model image to be displayed, then a user gesture video is obtained, the user gesture video is obtained through a camera, a plurality of user gesture images are extracted from the user gesture video, preferably at equal intervals, and subsequent gesture movement information calculation, such as gesture movement speed calculation, is facilitated. Only part of the user gesture images are extracted for operation to realize gesture interaction, so that the image operation amount is greatly reduced, and the real-time performance of gesture interaction is ensured. The gesture outline is extracted from the gesture image of the user, so that a gesture vector is generated, the gesture vector has directionality and can describe the shape and size of the gesture of the user to a certain extent, and a relatively simple gesture center of mass contains more gesture information, so that more and richer gestures can be distinguished and identified, and richer interaction functions are realized; and compared with the gesture outline, the gesture vector operation is simple, the operation amount is small, the operation complexity is reduced, and the real-time performance of gesture interaction is improved, so that the effect of rich gesture description identification is realized through simple data operation.
Preferably, the method for displaying the 3D model image includes obtaining the 3D model image to be displayed from the 3D model image library and performing projection display, and specifically includes:
establishing a 3D model image library, and setting label information for each 3D model image in the 3D model image library;
acquiring a static image input by a user, and identifying the static image to obtain image information;
searching a 3D model image corresponding to the label information with the highest similarity with the image information in the 3D model image library as the 3D model image to be displayed;
and converting the 3D model image to be displayed into an optical signal and performing projection display.
The 3D model image to be displayed is screened by identifying the static image input by the user, which can further improve the interest and the interactivity of the AR projection. For example, the static image input by the user is an image of a sheep, the static image is identified to obtain image information 'sheep', and a 3D model image about the sheep is found in a 3D model image library according to the image information to be displayed in a projection mode. It should be understood that the image information may also include other types of attribute information to improve the similarity of the matching, for example, the attribute information may also include color, size, shape, etc. The similarity between the image information and the label information can be obtained according to a weighted average of the similarities of the attribute information.
Preferably, the extracting the gesture outline in the user gesture image specifically includes:
acquiring a histogram of the user gesture image, and calculating a segmentation threshold according to the histogram;
and segmenting the user gesture image according to the segmentation threshold value, and then extracting the outline to obtain the gesture outline.
The histogram can be obtained by adopting the prior art, namely, the binarization processing is firstly carried out on the user gesture image to obtain a binarization image, and the histogram is formed according to the binarization image. And calculating a segmentation threshold according to the histogram, segmenting the user gesture image according to the segmentation threshold to obtain a characteristic picture, namely the gesture picture of the hand of the user, removing the background picture, and extracting the outline of the gesture picture to obtain a gesture outline.
Preferably, generating a gesture vector according to the gesture outline specifically includes:
and acquiring a gesture center of mass and a gesture feature point according to the gesture outline, and establishing the gesture vector by taking the gesture center of mass as a starting point and the gesture feature point as an end point.
The centroid of the gesture describes the location of the gesture, and the feature points of the gesture can be used as identification points for recognizing the direction of the gesture. The characteristic points of the gesture can be arbitrarily set according to the requirement, for example, the characteristic points can be set as a thumb tip, a middle finger tip, a little finger tip, an intersection point of adjacent fingers, a point farthest from the centroid on the gesture outline, a point nearest to the centroid on the gesture outline, and the like. It should be understood that the number of the gesture vectors may also be multiple, that is, a plurality of gesture feature points are selected by using a gesture center of mass as a starting point to establish a plurality of gesture vectors, so that according to changes of the plurality of gesture vectors, not only the position change and the rotation angle change of the gesture can be obtained, but also the change of the gesture shape can be obtained. For example, a gesture center of mass is used as a starting point, five finger fingertips are respectively used as gesture feature points, five gesture vectors are established, a fist making gesture and a palm gesture can be recognized according to the five gesture vectors, a five-finger closing gesture and a five-finger separating gesture can also be recognized, and therefore a richer gesture recognition function is achieved.
Preferably, the gesture motion information includes a motion speed, a motion track, a rotation angle, and a rotation speed.
The gesture centroid describes the position of the gesture, and the movement speed, the movement track and the like of the user gesture can be obtained according to the change of the gesture centroid of the gesture images of the users. The gesture feature points can be used as identification points for identifying the direction of the gesture, tracking the gesture feature points, and combining the gesture mass center and the gesture feature points, the rotation angle, the rotation speed and the like of the gesture can be obtained. If a plurality of gesture feature points are set, namely a plurality of gesture vectors are set, the shape of the gesture can be judged according to the angle relation among the plurality of gesture vectors and the lengths of the plurality of vectors. Because the gesture vector contains more gesture information than a simple gesture center of mass, more abundant gesture motion information can be acquired, and in this embodiment, only a few kinds of more common gesture motion information are listed, but not limited thereto.
Preferably, the 3D model image is controlled to move according to the gesture movement information, specifically;
and establishing a motion model library, searching a motion model with the highest similarity to the gesture motion information in the motion model library as a matching motion model, and controlling the 3D model image to move by using the matching motion model.
And controlling the 3D model image to move according to the gesture movement information of the user, thereby realizing the purpose of gesture interaction. The motion model includes a plurality of motion attributes, and the motion attributes correspond to the motion attributes of the gesture motion information, specifically, the motion attributes of the motion model in this embodiment also include motion attributes of a motion speed, a motion trajectory, a rotation angle, and a rotation speed. And calculating the similarity of the gesture motion information and the motion model, and setting according to specific requirements. For example, if the 3D model image movement is required to follow the user gesture to make the same movement, it is defined that the smaller the difference between the movement speeds, the higher the similarity, the closer the movement trajectory is to the similarity, the higher the similarity, the smaller the difference between the rotation angles, and the smaller the difference between the rotation speeds, the higher the similarity. If the 3D model image and the gesture of the user are required to do opposite-direction actions, the opposite definition is carried out, the larger the difference of the defined motion speeds is, the higher the similarity is, the larger the difference of the motion tracks is, the higher the similarity is, the larger the difference of the rotation angles is, and the higher the similarity is, and the larger the difference of the rotation speeds is.
Example 2
As shown in fig. 2, embodiment 2 of the present invention provides a system including an AR projector 1, a camera 2, and a 3D resource server 3;
the 3D resource server 3 is used for establishing a 3D model image library;
the AR projector 1 is used for obtaining a 3D model image to be displayed from the 3D model image library and performing projection display, and the AR projector 1 projects the image to the projection wall 4.
The camera 2 is used for acquiring a user gesture video;
the AR projector 1 is further used for sequentially extracting a plurality of user gesture images from the user gesture video; respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline; generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
The gesture interaction AR projection device based on image recognition is used for realizing the gesture interaction AR projection method based on image recognition, so that the technical effect of the gesture interaction AR projection method based on image recognition is also achieved by the gesture interaction AR projection device based on image recognition, and the details are not repeated herein.
Example 3
Embodiment 3 of the present invention provides a computer storage medium on which a computer program is stored, which, when executed by a processor, implements the above embodiment gesture interaction AR projection method based on image recognition.
The gesture interaction AR projection method based on image recognition specifically comprises the following steps:
performing projection display on a 3D model image to be displayed;
sequentially extracting a plurality of user gesture images from a user gesture video;
respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline;
generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
The computer storage medium provided by the invention is used for realizing the gesture interaction AR projection method based on the image recognition, so that the technical effects of the gesture interaction AR projection method based on the image recognition are also achieved by the computer storage medium, and the description is omitted.
The above-described embodiments of the present invention should not be construed as limiting the scope of the present invention. Any other corresponding changes and modifications made according to the technical idea of the present invention should be included in the protection scope of the claims of the present invention.
Claims (8)
1. A gesture interaction AR projection method based on image recognition is characterized by comprising the following steps:
acquiring a 3D model image to be displayed from a 3D model image library and performing projection display;
acquiring a user gesture video, and sequentially extracting a plurality of user gesture images from the user gesture video;
respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline;
generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
2. The image recognition-based gesture interaction AR projection method according to claim 1, wherein the 3D model image to be displayed is obtained from a 3D model image library and is subjected to projection display, specifically:
establishing a 3D model image library, and setting label information for each 3D model image in the 3D model image library;
acquiring a static image input by a user, and identifying the static image to obtain image information;
searching a 3D model image corresponding to the label information with the highest similarity with the image information in the 3D model image library as the 3D model image to be displayed;
and converting the 3D model image to be displayed into an optical signal and performing projection display.
3. The image recognition-based gesture interaction AR projection method according to claim 1, wherein extracting a gesture contour in the user gesture image specifically comprises:
acquiring a histogram of the user gesture image, and calculating a segmentation threshold according to the histogram;
and segmenting the user gesture image according to the segmentation threshold value, and then extracting the outline to obtain the gesture outline.
4. The image recognition-based gesture interaction AR projection method according to claim 1, wherein a gesture vector is generated according to the gesture contour, specifically:
and acquiring a gesture center of mass and a gesture feature point according to the gesture outline, and establishing the gesture vector by taking the gesture center of mass as a starting point and the gesture feature point as an end point.
5. The image recognition-based gesture interaction AR projection method according to claim 1, wherein the gesture motion information comprises motion speed, motion trajectory, rotation angle and rotation speed.
6. The image recognition-based gesture interaction AR projection method according to claim 1, wherein the 3D model image is controlled to move according to the gesture movement information, specifically;
and establishing a motion model library, searching a motion model with the highest similarity to the gesture motion information in the motion model library as a matching motion model, and controlling the 3D model image to move by using the matching motion model.
7. A gesture interaction AR projection device based on image recognition is characterized by comprising an AR projector, a camera and a 3D resource server;
the 3D resource server is used for establishing a 3D model image library;
the AR projector is used for acquiring a 3D model image to be displayed from the 3D model image library and performing projection display;
the camera is used for collecting a user gesture video;
the AR projector is also used for sequentially extracting a plurality of user gesture images from the user gesture video; respectively extracting a gesture outline in each user gesture image, and generating a gesture vector according to the gesture outline; generating gesture motion information according to the gesture vectors of the plurality of user gesture images, and controlling the 3D model image to move according to the gesture motion information.
8. A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image recognition based gesture interaction AR projection method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910909648.3A CN110794959A (en) | 2019-09-25 | 2019-09-25 | Gesture interaction AR projection method and device based on image recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910909648.3A CN110794959A (en) | 2019-09-25 | 2019-09-25 | Gesture interaction AR projection method and device based on image recognition |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110794959A true CN110794959A (en) | 2020-02-14 |
Family
ID=69439706
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910909648.3A Pending CN110794959A (en) | 2019-09-25 | 2019-09-25 | Gesture interaction AR projection method and device based on image recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110794959A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150220776A1 (en) * | 2012-08-03 | 2015-08-06 | Crunchfish Ab | Identification of a gesture |
| CN105353873A (en) * | 2015-11-02 | 2016-02-24 | 深圳奥比中光科技有限公司 | Gesture manipulation method and system based on three-dimensional display |
| CN107016733A (en) * | 2017-03-08 | 2017-08-04 | 北京光年无限科技有限公司 | Interactive system and exchange method based on augmented reality AR |
| US20180173315A1 (en) * | 2016-12-20 | 2018-06-21 | Getgo, Inc. | Augmented reality user interface |
| CN109190516A (en) * | 2018-08-14 | 2019-01-11 | 东北大学 | A kind of static gesture identification method based on volar edge contour vectorization |
| CN109685907A (en) * | 2017-10-18 | 2019-04-26 | 深圳市掌网科技股份有限公司 | Image combination method and system based on augmented reality |
| CN109961005A (en) * | 2019-01-28 | 2019-07-02 | 山东大学 | A kind of dynamic gesture identification method and system based on two-dimensional convolution network |
-
2019
- 2019-09-25 CN CN201910909648.3A patent/CN110794959A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150220776A1 (en) * | 2012-08-03 | 2015-08-06 | Crunchfish Ab | Identification of a gesture |
| CN105353873A (en) * | 2015-11-02 | 2016-02-24 | 深圳奥比中光科技有限公司 | Gesture manipulation method and system based on three-dimensional display |
| WO2017075932A1 (en) * | 2015-11-02 | 2017-05-11 | 深圳奥比中光科技有限公司 | Gesture-based control method and system based on three-dimensional displaying |
| US20180173315A1 (en) * | 2016-12-20 | 2018-06-21 | Getgo, Inc. | Augmented reality user interface |
| CN107016733A (en) * | 2017-03-08 | 2017-08-04 | 北京光年无限科技有限公司 | Interactive system and exchange method based on augmented reality AR |
| CN109685907A (en) * | 2017-10-18 | 2019-04-26 | 深圳市掌网科技股份有限公司 | Image combination method and system based on augmented reality |
| CN109190516A (en) * | 2018-08-14 | 2019-01-11 | 东北大学 | A kind of static gesture identification method based on volar edge contour vectorization |
| CN109961005A (en) * | 2019-01-28 | 2019-07-02 | 山东大学 | A kind of dynamic gesture identification method and system based on two-dimensional convolution network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Memo et al. | Head-mounted gesture controlled interface for human-computer interaction | |
| US20180088663A1 (en) | Method and system for gesture-based interactions | |
| Betancourt et al. | The evolution of first person vision methods: A survey | |
| Hackenberg et al. | Lightweight palm and finger tracking for real-time 3D gesture control | |
| US8897491B2 (en) | System for finger recognition and tracking | |
| US8660303B2 (en) | Detection of body and props | |
| US20220066569A1 (en) | Object interaction method and system, and computer-readable medium | |
| AU2012268589A1 (en) | System for finger recognition and tracking | |
| CN114651284A (en) | Lightweight multi-branch and multi-scale heavy person identification | |
| US11372518B2 (en) | Systems and methods for augmented or mixed reality writing | |
| Yousefi et al. | 3D gesture-based interaction for immersive experience in mobile VR | |
| Liu et al. | Holoscopic 3D micro-gesture database for wearable device interaction | |
| Caputo et al. | SFINGE 3D: A novel benchmark for online detection and recognition of heterogeneous hand gestures from 3D fingers’ trajectories | |
| Huang et al. | Egocentric hand detection via dynamic region growing | |
| CN110069125B (en) | Virtual object control method and device | |
| Li et al. | Hand gesture tracking and recognition based human-computer interaction system and its applications | |
| Conly et al. | An integrated RGB-D system for looking up the meaning of signs | |
| WO2024078088A1 (en) | Interaction processing method and apparatus | |
| Ueng et al. | Vision based multi-user human computer interaction | |
| Alcoverro et al. | Gesture control interface for immersive panoramic displays | |
| CN110794959A (en) | Gesture interaction AR projection method and device based on image recognition | |
| Song et al. | Real-time single camera natural user interface engine development | |
| CN110069126B (en) | Virtual object control method and device | |
| Zhang et al. | Dynamic gesture tracking and recognition algorithm based on deep learning | |
| Prabhakar et al. | AI And Hand Gesture Recognition Based Virtual Mouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |