CN107450714A - Man-machine interaction support test system based on augmented reality and image recognition - Google Patents

Man-machine interaction support test system based on augmented reality and image recognition Download PDF

Info

Publication number
CN107450714A
CN107450714A CN201610369666.3A CN201610369666A CN107450714A CN 107450714 A CN107450714 A CN 107450714A CN 201610369666 A CN201610369666 A CN 201610369666A CN 107450714 A CN107450714 A CN 107450714A
Authority
CN
China
Prior art keywords
gesture
image
mouse
module
hand
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201610369666.3A
Other languages
Chinese (zh)
Inventor
宋立新
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Datang Telecommunication Science & Technology Co Ltd
Datang Telecom Technology Co Ltd
Original Assignee
Datang Telecommunication Science & Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Datang Telecommunication Science & Technology Co Ltd filed Critical Datang Telecommunication Science & Technology Co Ltd
Priority to CN201610369666.3A priority Critical patent/CN107450714A/en
Publication of CN107450714A publication Critical patent/CN107450714A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The present invention relates to augmented reality and the human-computer interaction technique field of image recognition, the man-machine interaction support test system of augmented reality and image recognition is based particularly on.The invention obtains the image information of real scene using video camera, and detection and matching is marked to the image of acquisition, obtains feature and identity information corresponding to the mark.According to the feature and identity information of mark, the attitude information of the video camera is calculated.The virtual objects corresponding with mark identity information are generated, and accurately " are alignd " with real scene, finally the image of final virtual reality fusion is showed by display device.It is responsible for the collection of video image by camera during man-machine interaction, each two field picture is judged, the identification of gesture, position locating and tracking, the calculating of gesture pose, dummy model loading, virtual reality fusion, the process such as display output, to complete a completely augmented reality system interactive process based on gesture identification.

Description

Man-machine interaction support test system based on augmented reality and image recognition
Technical field
The present invention relates to augmented reality and the human-computer interaction technique field of image recognition, augmented reality and figure are based particularly on As the man-machine interaction support test system of identification.
Background technology
Augmented reality is a caused new technology recently as the development of virtual reality technology, and it utilizes meter Calculation machine renders generation virtual scene, and this virtual scene is accurately merged with real world, and final utilize regards Scene after virtual fusion is presented to user by frequency display device, so as to substantially increase the visual experience of the mankind.Enhancing is existing Real technology can produce real-time feeling of immersion, allow people by it is a kind of it is more natural in a manner of interact with computer, be a kind of The human-computer interaction technology of " human beings " is emphasized, has been widely used in the fields such as military, medical treatment, education, amusement at present.
In current augmented reality systematic research and development process, man-machine interaction is used as using the handheld device of entity mostly Primary interface, although this method development cost is relatively low, its interactive function is relatively easy, interactive mode not enough naturally, and The feeling of immersion of man-machine interaction in augmented reality greatly reduces.In order to provide a kind of more natural friendship with intelligence to user By way of image recognition, the man-machine interaction of virtual scene is completed instead of handheld device using human hand by mutual mode, the present invention, Compared with the handheld device of solid type, its ease for use is it will be apparent that being completed and virtual field using human hand as input interface The interaction of scape, dummy model is tied to operator, it is more naturally operated virtual objects, is greatly improved people The feeling of immersion of machine interaction, has expanded the mode of augmented reality Method of Man-computer Interaction.
The content of the invention
The problem of present invention exists for prior art, has invented the man-machine interaction branch based on augmented reality and image recognition Support test system.The invention obtains the image information of real scene using video camera, and detection is marked to the image of acquisition And matching, obtain feature and identity information corresponding to the mark.According to the feature and identity information of mark, the shooting is calculated The attitude information of machine.The virtual objects corresponding with mark identity information are generated, and accurately " are alignd " with real scene, finally The image of final virtual reality fusion is showed by display device.Video image is responsible for by camera during man-machine interaction Collection, each two field picture is judged, the identification of gesture, position locating and tracking, gesture pose calculates, dummy model adds The processes such as load, virtual reality fusion, display output, to complete a completely augmented reality system man-machine interaction based on gesture identification Process.
The system includes four functional modules:Tracking module, gesture tracking module, mouse control module and track is marked to know Other module.Gesture tracking is again comprising image preprocessing, target following, target detection, target information fusion.Gesture recognition module has Three feature extraction, cluster, track identification sub-block compositions.
Described mark tracking module, the video image that processing camera gathers in real time.By marking detection and identification, obtain The id information of mark is taken, the attitude information of video camera is calculated, obtains the transformed matrix from true coordinate to two-dimensional screen.
Described gesture tracking module, it is the tracking with detection.The module judges to whether there is in single frames picture Certain defined gesture, if there is then determining its position in the picture.When tracking runs into violent variation or trace point When being disappeared from camera watch region, target can be detected again and it is tracked.
Described defined gesture is four kinds of static gestures, including forefinger hand-type, scissors hand-type, palm hand-type and fist Hand-type.
Described mouse control is that the click and roaming of mouse left, center, right are realized by four hand-type detecting and trackings.Wherein Palm is defined to the relation that fist is left mouse button;Relation of the palm to scissors for right mouse button;Palm is to forefinger in mouse The relation of keyboard.The change of hand-type is mapped on mouse event, when non-detecting and tracking is to any hand-type, only starts detection hand Type zero, hand-type zero represent human hand and start to hold mouse, and start to do the action of mobile mouse.This is to start other tertiary target inspections Measuring and calculating, the instruction for waiting similar mouse left, center, right key to press and discharge.
Described pattern recognition module, the features such as position, direction and speed are clustered, discretized features vector conduct HMMs input, probability is verified after being exported using Viterbi algorithm to observation sequence symbol, output probability the maximum is gesture institute Belong to classification.
The one or more technical schemes provided in the embodiment of the present application, have at least the following technical effects or advantages:Should System realizes the seamless connection of virtual information and real world, reaches the purpose virtually merged, user is supplied on sense organ Brand-new experience.For man-machine interaction support test system, not merely go to show virtual information, and allow user and system Interacted in real time.Realize the man-machine interaction support test system that having based on augmented reality touches perceptional function System.
Brief description of the drawings
The present invention is further detailed explanation with reference to the accompanying drawings and detailed description.
Fig. 1 is the system block diagram of the present invention
Fig. 2 is implementation process figure of the present invention
Embodiment
As shown in figure 1, the man-machine interaction support test system of the invention based on augmented reality and image recognition, there is provided one It is individual to use single-hand movement, by gathering image from monocular, common camera, carry out the man-machine interaction support test system of image recognition System, this system include three functional modules:Image tagged module, gesture tracking module, mouse control module and track identification mould Block.Include image preprocessing, target following, target detection, target information fusion inside gesture tracking;Gesture recognition module has Three feature extraction, cluster, track identification sub-block compositions.Fig. 2 is the flow chart that the present invention realizes.
In system initialisation phase, it is necessary first to open video path and search window size, then initialize video camera Parameter is simultaneously loaded into marker data, finally opens image window and obtains frame of video.
During mark analyte detection identification, connected domain analysis should be carried out to the video image of acquisition, then first Line identifier matches.During matching is identified, it is also necessary to the calculating of transformation matrix is carried out to the mark that each is matched.
System carries out dummy object modeling by read module information, improves the complexity of model and the efficiency of modeling, It is that the dummy object of man-machine interaction support test system has more the sense of reality.
The present invention is analyzed by the type to marker and state to change the attribute of single or multiple dummy objects, Such as size, color, illumination and animation, blending of the real-world object to dummy object is reached with this.
Realize that actual situation is mutually melted by the judgement to marker type.Judgement to marker type using connected domain analysis and The method of mark matching is realized.Different types of marker will produce different influences to the attribute of dummy object.
Realize that actual situation is mutually melted by the judgement to marker physical spatial location.Judgement to marker object space position It can be changed and realized by the coordinates matrix of mark.The locus of marker will have an impact to the attribute of dummy object.
Realize that actual situation is mutually melted by the judgement the space length marker.The space length of marker will be to virtual object The attribute of body has an impact.
System carries out real-time image rendering using Cg to dummy object, utilizes Cg illumination and atomization and other effects.Afterwards The synthesis of actual situation scene is carried out, matrix format is changed, dummy object is drawn or imports model information, dummy object is folded It is added in true picture.
Although the foregoing describing the embodiment of the present invention, those skilled in the art should be appreciated that this Be merely illustrative of, various changes or modifications can be made to present embodiment, without departing from the present invention principle and essence, Protection scope of the present invention is only limited by the claims that follow.

Claims (7)

1. the man-machine interaction support test system based on augmented reality and image recognition, it is characterised in that:The system includes four Functional module:Mark tracking module, gesture tracking module, mouse control module and track identification module.
2. according to the system described in claim 1, it is characterised in that described mark tracking module, processing camera are adopted in real time The video image of collection, detect and identify by marking, obtain the id information of mark, the attitude information of video camera is calculated, obtains Obtain the transformed matrix from true coordinate to two-dimensional screen.
3. according to the system described in claim 1, it is characterised in that described gesture tracking module, the module are judged in single frames Whether there is certain defined gesture in picture, if there is then determining its position in the picture, when tracking run into it is violent When variation or trace point disappear from camera watch region, target can be detected again and it is tracked.
4. according to the gesture tracking module described in claim 3, it is characterised in that described defined gesture is four kinds of static hands Gesture, including forefinger hand-type, scissors hand-type, palm hand-type and fist hand-type.
5. according to the system described in claim 1, it is characterised in that described mouse control is to pass through four hand-type detecting and trackings Realize the click and roaming of mouse left, center, right.
6. according to the system described in claim 1, it is characterised in that define palm to the relation that fist is left mouse button;Palm To the relation that scissors is right mouse button;Relation of the palm to forefinger for middle button of mouse disk.The change of hand-type is mapped to mouse thing On part, the instruction pressed and discharged similar to mouse left, center, right key is waited.
7. according to the system described in claim 1, it is characterised in that described pattern recognition module, by position, direction and speed Clustered, input of the discretized features vector as HMMs, observation sequence symbol is exported using Viterbi algorithm etc. feature After verify probability, output probability the maximum is gesture generic.
CN201610369666.3A 2016-05-31 2016-05-31 Man-machine interaction support test system based on augmented reality and image recognition Pending CN107450714A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610369666.3A CN107450714A (en) 2016-05-31 2016-05-31 Man-machine interaction support test system based on augmented reality and image recognition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610369666.3A CN107450714A (en) 2016-05-31 2016-05-31 Man-machine interaction support test system based on augmented reality and image recognition

Publications (1)

Publication Number Publication Date
CN107450714A true CN107450714A (en) 2017-12-08

Family

ID=60484858

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610369666.3A Pending CN107450714A (en) 2016-05-31 2016-05-31 Man-machine interaction support test system based on augmented reality and image recognition

Country Status (1)

Country Link
CN (1) CN107450714A (en)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108509026A (en) * 2018-02-06 2018-09-07 西安电子科技大学 Tele-Maintenance Support System and method based on enhancing interactive mode
CN108804330A (en) * 2018-06-12 2018-11-13 Oppo(重庆)智能科技有限公司 Test method, device, storage medium and electronic equipment
CN109656364A (en) * 2018-08-15 2019-04-19 亮风台(上海)信息科技有限公司 It is a kind of for the method and apparatus of augmented reality content to be presented on a user device
CN110120060A (en) * 2018-02-06 2019-08-13 广东虚拟现实科技有限公司 Recognition methods, device and the recognition and tracking system of marker
WO2019154169A1 (en) * 2018-02-06 2019-08-15 广东虚拟现实科技有限公司 Method for tracking interactive apparatus, and storage medium and electronic device
CN110599603A (en) * 2019-09-20 2019-12-20 上海大学 Mechanical equipment visual interaction and equipment state monitoring system and method based on augmented reality
CN110597446A (en) * 2018-06-13 2019-12-20 北京小鸟听听科技有限公司 Gesture recognition method and electronic equipment
CN110619674A (en) * 2019-08-15 2019-12-27 重庆特斯联智慧科技股份有限公司 Three-dimensional augmented reality equipment and method for accident and alarm scene restoration
CN111103969A (en) * 2018-10-26 2020-05-05 广东虚拟现实科技有限公司 Information identification method and device, electronic equipment and computer readable storage medium
CN111167115A (en) * 2018-11-09 2020-05-19 致伸科技股份有限公司 Interactive game system
CN111931605A (en) * 2020-07-23 2020-11-13 中核核电运行管理有限公司 Intelligent monitoring system and method for high-risk operation of nuclear power plant
CN111950521A (en) * 2020-08-27 2020-11-17 深圳市慧鲤科技有限公司 Augmented reality interaction method and device, electronic equipment and storage medium
CN112017304A (en) * 2020-09-18 2020-12-01 北京百度网讯科技有限公司 Method, apparatus, electronic device, and medium for presenting augmented reality data
CN112561953A (en) * 2019-09-26 2021-03-26 北京外号信息技术有限公司 Method and system for target recognition and tracking in real scenes
CN112633145A (en) * 2020-12-21 2021-04-09 武汉虚世科技有限公司 WebAR processing method based on 3D detection and identification and moving target tracking
CN112929750A (en) * 2020-08-21 2021-06-08 海信视像科技股份有限公司 Camera adjusting method and display device
CN113923501A (en) * 2021-10-09 2022-01-11 深圳市中渤光电有限公司 LED screen panoramic display method and system based on VR virtual reality
CN116258655A (en) * 2022-12-13 2023-06-13 合肥工业大学 Real-time image enhancement method and system based on gesture interaction
CN116912950A (en) * 2023-09-12 2023-10-20 湖北星纪魅族科技有限公司 Identification method, head-mounted device and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101799717A (en) * 2010-03-05 2010-08-11 天津大学 Man-machine interaction method based on hand action catch
CN102467905A (en) * 2010-10-28 2012-05-23 鸿富锦精密工业(深圳)有限公司 Gesture recognition appparatus and method
CN104460951A (en) * 2013-09-12 2015-03-25 天津智树电子科技有限公司 Human-computer interaction method
CN104820497A (en) * 2015-05-08 2015-08-05 东华大学 A 3D interaction display system based on augmented reality

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101799717A (en) * 2010-03-05 2010-08-11 天津大学 Man-machine interaction method based on hand action catch
CN102467905A (en) * 2010-10-28 2012-05-23 鸿富锦精密工业(深圳)有限公司 Gesture recognition appparatus and method
CN104460951A (en) * 2013-09-12 2015-03-25 天津智树电子科技有限公司 Human-computer interaction method
CN104820497A (en) * 2015-05-08 2015-08-05 东华大学 A 3D interaction display system based on augmented reality

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108509026A (en) * 2018-02-06 2018-09-07 西安电子科技大学 Tele-Maintenance Support System and method based on enhancing interactive mode
CN110120060A (en) * 2018-02-06 2019-08-13 广东虚拟现实科技有限公司 Recognition methods, device and the recognition and tracking system of marker
WO2019154169A1 (en) * 2018-02-06 2019-08-15 广东虚拟现实科技有限公司 Method for tracking interactive apparatus, and storage medium and electronic device
CN108509026B (en) * 2018-02-06 2020-04-14 西安电子科技大学 Remote maintenance support system and method based on enhanced interaction mode
CN108804330A (en) * 2018-06-12 2018-11-13 Oppo(重庆)智能科技有限公司 Test method, device, storage medium and electronic equipment
CN110597446A (en) * 2018-06-13 2019-12-20 北京小鸟听听科技有限公司 Gesture recognition method and electronic equipment
CN109656364B (en) * 2018-08-15 2022-03-29 亮风台(上海)信息科技有限公司 Method and device for presenting augmented reality content on user equipment
CN109656364A (en) * 2018-08-15 2019-04-19 亮风台(上海)信息科技有限公司 It is a kind of for the method and apparatus of augmented reality content to be presented on a user device
CN111103969B (en) * 2018-10-26 2023-09-01 广东虚拟现实科技有限公司 Information identification method, information identification device, electronic equipment and computer readable storage medium
CN111103969A (en) * 2018-10-26 2020-05-05 广东虚拟现实科技有限公司 Information identification method and device, electronic equipment and computer readable storage medium
CN111167115A (en) * 2018-11-09 2020-05-19 致伸科技股份有限公司 Interactive game system
CN110619674A (en) * 2019-08-15 2019-12-27 重庆特斯联智慧科技股份有限公司 Three-dimensional augmented reality equipment and method for accident and alarm scene restoration
CN110599603A (en) * 2019-09-20 2019-12-20 上海大学 Mechanical equipment visual interaction and equipment state monitoring system and method based on augmented reality
CN112561953A (en) * 2019-09-26 2021-03-26 北京外号信息技术有限公司 Method and system for target recognition and tracking in real scenes
CN111931605A (en) * 2020-07-23 2020-11-13 中核核电运行管理有限公司 Intelligent monitoring system and method for high-risk operation of nuclear power plant
CN111931605B (en) * 2020-07-23 2024-05-14 中核核电运行管理有限公司 Nuclear power plant high-risk operation intelligent monitoring system and method
CN112929750A (en) * 2020-08-21 2021-06-08 海信视像科技股份有限公司 Camera adjusting method and display device
CN111950521A (en) * 2020-08-27 2020-11-17 深圳市慧鲤科技有限公司 Augmented reality interaction method and device, electronic equipment and storage medium
CN112017304A (en) * 2020-09-18 2020-12-01 北京百度网讯科技有限公司 Method, apparatus, electronic device, and medium for presenting augmented reality data
CN112017304B (en) * 2020-09-18 2023-12-22 北京百度网讯科技有限公司 Method, apparatus, electronic device and medium for presenting augmented reality data
CN112633145A (en) * 2020-12-21 2021-04-09 武汉虚世科技有限公司 WebAR processing method based on 3D detection and identification and moving target tracking
CN112633145B (en) * 2020-12-21 2024-04-26 武汉虚世科技有限公司 WebAR processing method based on 3D detection recognition and moving target tracking
CN113923501A (en) * 2021-10-09 2022-01-11 深圳市中渤光电有限公司 LED screen panoramic display method and system based on VR virtual reality
CN116258655A (en) * 2022-12-13 2023-06-13 合肥工业大学 Real-time image enhancement method and system based on gesture interaction
CN116258655B (en) * 2022-12-13 2024-03-12 合肥工业大学 Real-time image enhancement method and system based on gesture interaction
CN116912950A (en) * 2023-09-12 2023-10-20 湖北星纪魅族科技有限公司 Identification method, head-mounted device and storage medium

Similar Documents

Publication Publication Date Title
CN107450714A (en) Man-machine interaction support test system based on augmented reality and image recognition
Cheng et al. Survey on 3D hand gesture recognition
CN103941866B (en) Three-dimensional gesture recognizing method based on Kinect depth image
Erol et al. Vision-based hand pose estimation: A review
JP6079832B2 (en) Human computer interaction system, hand-to-hand pointing point positioning method, and finger gesture determination method
Kang et al. Color based hand and finger detection technology for user interaction
CN103809733B (en) Man-machine interactive system and method
Zhu et al. Vision based hand gesture recognition
US20140232631A1 (en) Model-based multi-hypothesis target tracker
CN108509026B (en) Remote maintenance support system and method based on enhanced interaction mode
Kulshreshth et al. Poster: Real-time markerless kinect based finger tracking and hand gesture recognition for HCI
Sun et al. Magichand: Interact with iot devices in augmented reality environment
Wu et al. Robust fingertip detection in a complex environment
JP2017505965A (en) Real-time 3D gesture recognition and tracking system for mobile devices
She et al. A real-time hand gesture recognition approach based on motion features of feature points
Plouffe et al. Natural human-computer interaction using static and dynamic hand gestures
Boruah et al. Development of a learning-aid tool using hand gesture based human computer interaction system
Dhore et al. Human Pose Estimation And Classification: A Review
Che et al. Detection-guided 3D hand tracking for mobile AR applications
Abdallah et al. An overview of gesture recognition
CN108108648A (en) A kind of new gesture recognition system device and method
Usabiaga et al. Global hand pose estimation by multiple camera ellipse tracking
Otberdout et al. Hand pose estimation based on deep learning depth map for hand gesture recognition
Yousaf et al. Virtual keyboard: real-time finger joints tracking for keystroke detection and recognition
Raees et al. Thumb inclination-based manipulation and exploration, a machine learning based interaction technique for virtual environments

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20171208

RJ01 Rejection of invention patent application after publication