CN116301551A - Touch identification method, touch identification device, electronic equipment and medium - Google Patents
Touch identification method, touch identification device, electronic equipment and medium Download PDFInfo
- Publication number
- CN116301551A CN116301551A CN202211478103.XA CN202211478103A CN116301551A CN 116301551 A CN116301551 A CN 116301551A CN 202211478103 A CN202211478103 A CN 202211478103A CN 116301551 A CN116301551 A CN 116301551A
- Authority
- CN
- China
- Prior art keywords
- gesture
- determining
- point
- touch
- motion information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention provides a touch identification method, a touch identification device, electronic equipment and a medium, wherein the touch identification method is applied to a touch identification system, and the touch identification system comprises the following steps: an infrared light source, an infrared camera, and a projection device; the method comprises the following steps: acquiring gesture action information of a user based on the infrared light source and the infrared camera; and under the condition that the gesture motion information is matched with a preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information. The touch identification method, the touch identification device, the electronic equipment and the medium provided by the invention can capture gesture action information of the user in an infrared environment, can feed back gesture actions of the user in real time, and realize interaction between the user and the projection equipment.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a touch recognition method, a touch recognition device, an electronic apparatus, and a medium.
Background
The development and improvement of the multi-touch interaction system provide a good test and development platform for the research and gesture design of the two-hand interaction technology. The prior multi-touch interactive system mainly comprises:
1. an interactive system in which a projector located above a desktop projects downward. The system supports multi-user multi-point input, and has the defect that the multi-point input of a single user is easily confused;
2. a desktop aware interactive system based on sensor technology. The system can obtain clear detection results of multiple input points and shapes, but cannot identify multi-user input;
3. a desktop interactive system that generates an under projection based on total reflection. The system supports the projection from bottom to top, but the implementation environment is difficult to build, the total reflection requirement is high, and the implementation in the actual working environment is difficult;
desktop interaction system based on lower projection. The system supports simultaneous interaction between four users and the desktop, but does not support shared screen interaction;
5. a desktop interactive system with an underlying infrared light source projected under. The system supports the limitation of size and appearance, is more suitable for families and is not suitable for large-scale meeting occasions.
Disclosure of Invention
The invention provides a touch identification method, a touch identification device, electronic equipment and a medium, which are used for solving the problem that the interaction modes in a touch interaction system in the prior art are limited to different degrees.
The invention provides a touch identification method, which is applied to a touch identification system, wherein the touch identification system comprises the following steps: an infrared light source, an infrared camera, and a projection device;
the method comprises the following steps:
acquiring gesture action information of a user based on the infrared light source and the infrared camera;
and under the condition that the gesture motion information is matched with a preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information.
In some embodiments, the preset gesture template is constructed by:
image segmentation is carried out on an image sequence corresponding to a standard gesture template, and an interested region in the image sequence is determined;
identifying a point outline corresponding to the region of interest based on the region of interest;
determining a contour circumscribed rectangle corresponding to the point contour based on the point contour;
and determining the preset gesture template based on the coordinates of the central point of the outline circumscribed rectangle and the image sequence corresponding to the image sequence.
In some embodiments, the acquiring gesture information of the user based on the infrared light source and the infrared camera includes:
determining the number of tracking points corresponding to the user based on the infrared light source and the infrared camera;
under the condition that the number of the tracking points is larger than 0, determining the number of detection points corresponding to the user;
determining that the gesture state of the user is a leaving state under the condition that the number of the detection points is 0;
and under the condition that the number of the detection points is larger than 0, determining the gesture state based on the distance between the tracking point and the detection points.
In some embodiments, the determining the gesture state based on a distance between the tracking point and the detection point comprises:
judging whether a minimum distance value is larger than a distance threshold value or not based on the distance between the tracking point and the detection point;
determining that the gesture state is a moving state if the distance minimum is greater than the distance threshold;
and determining that the gesture state is a pressing state under the condition that the minimum distance value is smaller than or equal to the distance threshold value.
In some embodiments, before the gesture motion information is converted into the control instruction, if the gesture motion information matches a preset gesture template, the method further includes:
determining the gesture motion information based on the gesture state;
under the condition that the number of the tracking points exceeds a threshold value of the number of the tracking points, matching the tracking points with the preset template gestures to obtain a matching result;
and under the condition that the matching result exceeds a matching result threshold value, determining that the gesture motion information is matched with the preset gesture template.
In some embodiments, before the acquiring gesture information of the user based on the infrared light source and the infrared camera, the method further includes:
determining pixel coordinates of a calibration point in the calibration image based on the calibration image acquired by the infrared camera;
determining screen coordinates of the calibration point based on the projection image corresponding to the calibration image;
and determining affine transformation matrix parameters based on the coordinate relationship between the pixel coordinates of the calibration point and the screen coordinates of the calibration point, wherein the affine transformation matrix parameters are used for converting the pixel coordinates of the image shot by the external camera into the screen coordinates of the image projected by the projection device.
The invention also provides a touch recognition device which is applied to a touch recognition system, wherein the touch recognition system comprises: an infrared light source, an infrared camera, and a projection device;
the device comprises:
the acquisition module is used for acquiring gesture action information of a user based on the infrared light source and the infrared camera;
the conversion module is used for converting the gesture motion information into a control instruction under the condition that the gesture motion information is matched with a preset gesture template, and the control instruction is used for controlling the projection equipment.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the touch identification method according to any one of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a touch recognition method as described in any of the above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements a touch recognition method as described in any one of the above.
According to the touch identification method, the touch identification device, the electronic equipment and the medium, the gesture action information of the user is captured in the infrared environment, and the gesture action information is matched with the preset gesture template, so that the gesture action of the user can be fed back in real time and converted into the control instruction of the projection equipment, and interaction between the user and the projection equipment is realized.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a touch recognition method according to the present invention;
FIG. 2 is a second flow chart of the touch recognition method according to the present invention;
FIG. 3 is a third flow chart of the touch recognition method according to the present invention;
fig. 4 is a schematic diagram of a software architecture of a system of the touch recognition method provided by the present invention;
fig. 5 is a schematic structural diagram of a touch recognition device provided by the present invention;
fig. 6 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In the related art, the development and improvement of the multi-touch interactive system provides a good test and development platform for the research and gesture design of the two-hand interactive technology. The prior multi-touch interactive system mainly comprises:
1. diamond Touch. The display of the desktop is formed by downward projection of a projector positioned above the desktop, so as to support multi-point input of multiple people. The system can identify multiple contacts and can determine from which user each contact comes.
2. Smart Skin. The desktop perception interaction system is based on the sensor technology. The system can provide a more complete desktop outstanding image, can obtain clear detection results of multiple input points and shapes, and is also used for describing some hand shapes and multiple-finger inputs.
3. FTIR. According to the desktop interaction system based on the projection below the total reflection, an infrared lamp is arranged around the device to provide an active light source, so that the total internal reflection propagation is realized. The system has the advantages of being capable of being composed of common ready-made parts, low in cost and supporting bottom-up projection, and is widely used in the multi-touch research field at present.
4. Lumisight Table. It is also a desktop interaction system based on lower projection, and is different from the way that the system is characterized in that an infrared light source is placed at the bottom of the device to irradiate upwards instead of total reflection, and four users are supported to interact with the desktop at the same time,
5. microsoft Surface. Also a desktop interactive system with the underlying infrared light source projected. The system can recognize gesture input, and the size and the appearance of the system are limited, so that the system is more suitable for families and is not suitable for large-scale meeting occasions.
The disadvantage of the Diamond Touch system is that the single user's multi-point input is ambiguous, confusing among the points, and the suggestion is not to input simultaneously. Smart Skin is unable to recognize multi-user inputs. The FTIR environment is difficult to build, has high total reflection requirements, and is difficult to realize in the actual working environment. Lumisight Table does not support shared screen interactions. Microsoft Surface system can recognize gesture inputs whose size and appearance limit it to be more suitable for home use than for large meeting situations.
In recent years, with the gradual decrease in cost and the gradual decrease in shape of digital cameras and projectors, projection apparatuses can be regarded as programmable light sources, so that detailed information of scenes is obtained by the cameras. Conversely, a camera can also be regarded as a sensor, with the aid of which the projection device can project a high-quality image on an arbitrary surface. The combination of the projection device and the camera in a common space provides both input and output capabilities, creating a new human-machine interaction.
Fig. 1 is a schematic flow chart of a touch recognition method provided by the invention. Referring to fig. 1, the touch recognition method provided by the present invention is applied to a touch recognition system, the touch recognition system includes: an infrared light source, an infrared camera, and a projection device, the method comprising: step 110 and step 120.
The execution subject of the touch identification method provided by the invention can be electronic equipment, a component in the electronic equipment, an integrated circuit or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. By way of example, the mobile electronic device may be a cell phone, tablet computer, notebook computer, palm computer, vehicle mounted electronic device, wearable device, ultra-mobile personal computer (ultra-mobile personal computer, UMPC), netbook or personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device may be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., without limitation of the present invention.
The following describes the technical scheme of the invention in detail by taking a computer to execute the touch recognition method provided by the invention as an example.
In a touch recognition system constructed based on an infrared light source, an infrared camera and a projection device, the following aspects need to be described:
establishing a geometric corresponding relation between a virtual scene of a computer and an actual interaction environment of a user;
the infrared light source is arranged in the environment and used for providing infrared lamp light, users interact in the interaction environment, and the infrared light at the corresponding position becomes bright or dark;
the infrared camera can record the change of infrared light in the interactive environment in real time and transmit the change to the computer;
and the computer performs corresponding processing and projects a processing result to the interactive interface.
Thus, the location of the infrared light source, infrared camera, and projection device are related to the implementation of the system.
In practical implementation, the embodiment of the invention adopts a wall surface direct projection mode, and the hardware comprises an infrared camera, four infrared lamps and a projection device.
The system can track physical objects entering the interactive environment in real time, and can detect the finger tip position of the finger to track more accurately if the interactive physical objects are human hands. The system has low environmental requirements, any wall surface can be projected, the advantages of high identification precision and the like can be achieved, and simultaneous operation of multiple people is supported. The method can adopt an infrared laser to capture gesture actions, and is combined with a preset gesture template matching technology, predefine the gesture actions, shoot the gesture actions by using an infrared camera, analyze and feed back the result shot by the infrared camera in real time, and give an identification result. And after the result is obtained, interacting with a projection interface.
In one embodiment, the system operating environment is a PC, WIN10 (64-bit) operating system is used, the processor is Intel Core i7-8700K, and the memory is 16G. The integrated development environment used is Visual Studio2017 and the programming language used is c++. By recognizing the dynamic gesture in real time, the system can convert the gesture into a control instruction so as to control the image projected by the projection device in real time.
FIG. 2 is a second flow chart of the touch recognition method according to the present invention;
as shown in fig. 2, in the embodiment of the present invention, the gesture motion of the user may be captured by the infrared light source, and then the gesture motion information of the user is obtained by the infrared camera, and the gesture motion information is sent to the projection device.
Under the condition that the projection equipment determines that the gesture motion information is matched with the preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information in real time. The preset function may be an operation of enlarging, reducing, moving, or restoring the projected image.
According to the touch identification method, the gesture action information of the user is captured in the infrared environment, and the gesture action information is matched with the preset gesture template, so that the gesture action of the user can be fed back in real time and converted into the control instruction of the projection equipment, and interaction between the user and the projection equipment is realized.
In some embodiments, the preset gesture template is constructed by:
image segmentation is carried out on an image sequence corresponding to the standard gesture template, and an interested region in the image sequence is determined;
identifying a point profile corresponding to the region of interest based on the region of interest;
determining a contour circumscribed rectangle corresponding to the point contour based on the point contour;
and determining a preset gesture template based on the coordinates of the central point of the outline circumscribed rectangle and the image sequence corresponding to the image sequence.
In actual implementation, the preset gesture template comprises collection of standard gesture version images, extraction of feature points and maximum and minimum positions of all the feature points. The specific flow is as follows:
step a: image segmentation is carried out on an image sequence corresponding to the standard gesture template, an interested region in the image sequence is selected, and the interested region is stored;
step b: extracting features of the region of interest, and identifying the point profile of the identified feature points in the template picture sequence, namely the minimum position of the corresponding feature points;
step c: calculating a contour circumscribed rectangle for the point contour of the feature point, solving the center point coordinate of the circumscribed rectangle, and reserving the coordinate; the circumscribed rectangle is the smallest circumscribed rectangle of a certain outline, namely the smallest rectangle which can contain the outline and corresponds to the largest position of the feature point;
step d: determining an image of a preset gesture template based on the point outline of the image feature point and the outline circumscribed rectangle; based on the coordinates of the selected center point, storing images of a preset gesture template according to the sequence of the images corresponding to the standard gesture template, namely generating an image sequence of the preset gesture template, and marking the image sequence of the preset gesture template according to the predefined gesture name.
The touch recognition method provided by the invention can realize the self definition of the preset gesture template, and is convenient for realizing projection touch recognition by combining with feature point tracking.
In some embodiments, acquiring gesture motion information of a user based on an infrared light source and an infrared camera includes:
determining the number of tracking points corresponding to a user based on an infrared light source and an infrared camera;
under the condition that the number of tracking points is greater than 0, determining the number of detection points corresponding to a user;
under the condition that the number of detection points is 0, determining that the gesture state of the user is a leaving state;
when the number of detection points is greater than 0, the gesture state is determined based on the distance between the tracking point and the detection point.
FIG. 3 is a third flow chart of the touch recognition method according to the present invention;
as shown in fig. 3, in the embodiment of the present invention, as can be seen from fig. 3, the present invention considers several gesture states that may occur in actual situations, namely: an away state, a move state, and a press state. According to the continuous multi-frame images acquired by the infrared camera, the characteristic points detected in the continuous multi-frame images are tracking points (track) and detection points (detect), and the target points are characteristic points corresponding to gesture actions of the user.
The tracking point is used for tracking the motion state of the gesture feature point of the user between the continuous multi-frame images, and the gesture motion change can be estimated according to the relation between the frames. The detection points are characteristic points which are recognized under a single frame image and related to gesture actions of a user or hand positioning.
The following is considered according to three cases where the number of tracking points is 0, 1, and 2.
When the number of track points is 0, judging whether a target point is detected currently, and if the target point is detected, creating a new point for the track points.
Under the condition that the number of track points is larger than 0, determining the number of detect points corresponding to the user gesture;
when the track point number is 1, the number of detect points is judged.
When the number of detect points is 0, the leave state is returned.
When the detect point is greater than 0, the gesture state is determined based on the distance between the tracking point and the detection point.
According to the touch identification method, the characteristic points are tracked, clear logic judgment is adopted for the detection points and the algorithm realization of the tracking points, the time consumption of the algorithm is low, gesture motion information of a user can be fed back in real time, and a favorable item is provided for the practical application of the algorithm.
In some embodiments, determining the gesture state based on the distance between the tracking point and the detection point includes:
judging whether the minimum distance value is larger than a distance threshold value or not based on the distance between the tracking point and the detection point;
determining that the gesture state is a moving state when the distance minimum value is greater than the distance threshold value;
and when the minimum distance value is smaller than or equal to the distance threshold value, determining that the gesture state is a pressing state.
In actual execution, when the number of the detect points is 1, the distance between the current position of the track point and the detect points is calculated, when the distance dis is greater than the distance threshold T, the moving state is returned, and otherwise, the pressing state is returned;
when the number of detection points is 2, first, a distance determination is made with the track point, and the detection point which is closer to the track point is used as the next track point, and the distance is calculated. Likewise, when the distance dis is greater than the distance threshold T, the moving state is returned, otherwise the pressing state is returned. And for another detect point, a new track point is created for that point.
When the number of track points is 2, the number of detect points is determined.
When the number of detect points is 0, two track points are set to the away state.
When the number of the detect points is 1, calculating the distance between the two track points and the detect points, and comparing the two distances dis 1 And dis 2 Wherein the detect point yielding the smaller distance value is the next track point; judging whether the smaller distance value is larger than a distance threshold value T, if so, returning to a moving state, otherwise, returning to a pressing state; the state of the other track point is the away state.
When the number of the detected points is 2, the distances between the two detected points and the two track points are respectively judged, wherein the detected point and track point pairs with smaller distances are generated as the current points of the track, so that two groups of track and detected point pairs are generated. And respectively calculating the moving distance of each group of points, and returning to a moving state when the distance dis is larger than a threshold value T, or returning to a pressing state.
Through the above steps, 3 states are generated in total, which are respectively: a state of departure, a state of pressing, a state of movement. The 3 states are used for next analysis of motion information of the gesture.
According to the touch identification method provided by the invention, the distance between the tracking point and the detection point is calculated through the characteristic point tracking, and the gesture state is determined, so that gesture action information can be obtained; the method has the advantages that clear logic judgment is adopted for the algorithm implementation of the detection points and the tracking points, the time consumption of the algorithm is low, gesture motion information of a user can be fed back in real time, and a favorable lifting part is provided for the practical application of the algorithm.
In some embodiments, before converting the gesture motion information into the control instruction, in a case where the gesture motion information matches the preset gesture template, the method further includes:
determining gesture motion information based on the gesture state;
under the condition that the number of the tracking points exceeds the threshold value of the number of the tracking points, matching the tracking points with the preset template gestures to obtain a matching result;
and under the condition that the matching result exceeds the matching result threshold value, determining that the gesture motion information is matched with the preset gesture template.
Based on the above embodiments, the obtained gesture states are analyzed, and classified into one-hand motions and two-hand motions.
Wherein, the single hand action has: single click, double click and move; the actions of the hands are as follows: two-hand movement, two-hand pressing and two-hand lifting.
Specific gesture motion information is generated by combining the obtained gesture states.
When the number of the tracking points exceeds the threshold T of the number of the tracking points TrackNum When the user calls the preset handMatching the tracking points with a preset template gesture to obtain a matching result; if the obtained matching result is greater than the matching result threshold T result And when the gesture action information is judged to be a preset action, namely the matching is successful, the user-defined function corresponding to the gesture action information can be realized.
According to the touch identification method provided by the invention, through combining the preset gesture template matching and the feature point tracking, on the basis of adding the user-defined gesture function, the capture of all actions of gesture features under the condition of being in an infrared environment is realized.
In some embodiments, before acquiring gesture motion information of the user based on the infrared light source and the infrared camera, the method further comprises:
determining pixel coordinates of a calibration point in a calibration image based on the calibration image acquired by the infrared camera;
determining screen coordinates of the calibration point based on the projection image corresponding to the calibration image;
based on the coordinate relationship between the pixel coordinates of the calibration point and the screen coordinates of the calibration point, affine transformation matrix parameters for converting the pixel coordinates of the image photographed by the external camera into the screen coordinates of the image projected by the projection device are determined.
Since the system needs to perform feedback according to the actual operation of the user, the coordinate transformation between the projected image and the scene image captured by the infrared camera is important.
Because the system adopts the infrared camera, the visible light in the environment can not be sensed, namely the picture shot by the infrared camera can not change along with the change of the projection content, but change according to the change of the infrared light entering the camera lens, therefore, a calibration object is placed according to the projection content of the projection equipment, and the infrared light entering the camera lens is changed by reflecting or absorbing the infrared light through the calibration object.
Setting a calibration object at the edge position of the wall surface, acquiring a calibration image when the infrared camera shoots, and determining pixel coordinates of a calibration point corresponding to the calibration object in the calibration image; determining screen coordinates of the calibration point based on the projection image corresponding to the calibration image; and determining the coordinate relation between the pixel coordinates of the calibration point and the screen coordinates of the calibration point to obtain affine transformation matrix parameters.
After affine transformation matrix parameters are obtained, pixel coordinates of identification target points in images shot by an infrared camera can be converted into screen coordinates in projection images in real time, so that the effect of real-time interaction of users is achieved.
According to the touch identification method provided by the invention, the conversion between the pixel coordinates of the image shot by the infrared camera and the screen coordinates corresponding to the projection image can be realized, so that the touch identification efficiency is improved.
In some embodiments, as shown in fig. 4, the software architecture of the touch recognition system may be mainly divided into four layers, including a system layer, an engine layer, a target tracking recognition layer, and an application layer. The bottom layer is a system layer which is the basis of the whole system architecture and comprises input and output equipment, an operating system interface and a graphical interface of the bottom layer. The engine layer divides the upper layer application into modules, divides the operation flow in the modules into migration between states, provides interfaces for the modules for the bottom layer call, and supports multithreading. The target tracking and identifying layer provides functional support for the upper multi-point touch and identifying system application through the program state interface so as to complete the tracking and detection of hands. The application layer provides service for users, has a network transmission user interface function and is mainly responsible for communication between the users and application programs or between the application programs on the network. The application layer may comprise a user interaction environment, in particular may be used to provide a user interaction environment for interactions between virtual objects and user behavior.
The touch recognition method provided by the invention has the main advantages that direct interaction, multi-point touch, multi-user experience and object recognition are realized, a direct interaction user directly interacts with an application interface without using input tools such as a mouse, a keyboard and the like to carry out multi-point touch, the multi-point touch interface is not provided with a plurality of input response points, unlike the mouse, the keyboard and the like, only one response point is used, a plurality of users can stand at different positions of the application interface and interact with the application program to recognize the object recognition marked object.
The touch recognition device provided by the invention is described below, and the touch recognition device described below and the touch recognition method described above can be correspondingly referred to each other.
Fig. 5 is a schematic structural diagram of a touch recognition device provided by the present invention. Referring to fig. 5, the touch recognition device provided by the present invention includes: an acquisition module 510 and a conversion module 520.
The acquiring module 510 is configured to acquire gesture information of a user based on the infrared light source and the infrared camera;
the conversion module 520 is configured to convert the gesture motion information into a control instruction, where the control instruction is used to control the projection device, where the gesture motion information matches with a preset gesture template.
According to the touch recognition device provided by the invention, the gesture action information of the user is captured in the infrared environment, and the gesture action information is matched with the preset gesture template, so that the gesture action of the user can be fed back in real time and converted into the control instruction of the projection equipment, and the interaction between the user and the projection equipment is realized.
In some embodiments, the preset gesture template is constructed by:
image segmentation is carried out on an image sequence corresponding to a standard gesture template, and an interested region in the image sequence is determined;
identifying a point outline corresponding to the region of interest based on the region of interest;
determining a contour circumscribed rectangle corresponding to the point contour based on the point contour;
and determining the preset gesture template based on the coordinates of the central point of the outline circumscribed rectangle and the image sequence corresponding to the image sequence.
In some embodiments, the obtaining module 510 is further configured to:
determining the number of tracking points corresponding to the user based on the infrared light source and the infrared camera;
under the condition that the number of the tracking points is larger than 0, determining the number of detection points corresponding to the user;
determining that the gesture state of the user is a leaving state under the condition that the number of the detection points is 0;
and under the condition that the number of the detection points is larger than 0, determining the gesture state based on the distance between the tracking point and the detection points.
In some embodiments, the obtaining module 510 is further configured to:
judging whether a minimum distance value is larger than a distance threshold value or not based on the distance between the tracking point and the detection point;
determining that the gesture state is a moving state if the distance minimum is greater than the distance threshold;
and determining that the gesture state is a pressing state under the condition that the minimum distance value is smaller than or equal to the distance threshold value.
In some embodiments, the apparatus further comprises:
the first determining module is used for determining the gesture action information based on the gesture state;
the matching module is used for matching the tracking points with the preset template gestures under the condition that the number of the tracking points exceeds the threshold value of the number of the tracking points, so as to obtain a matching result;
and the second determining module is used for determining that the gesture motion information is matched with the preset gesture template under the condition that the matching result exceeds a matching result threshold value.
In some embodiments, the apparatus further comprises:
the third determining module is used for determining pixel coordinates of a calibration point in the calibration image based on the calibration image acquired by the infrared camera;
a fourth determining module, configured to determine screen coordinates of the calibration point based on the projection image corresponding to the calibration image;
and a fifth determining module, configured to determine affine transformation matrix parameters based on a coordinate relationship between the pixel coordinates of the calibration point and the screen coordinates of the calibration point, where the affine transformation matrix parameters are used to convert the pixel coordinates of the image captured by the external camera into the screen coordinates of the image projected by the projection device.
Fig. 6 illustrates a physical schematic diagram of an electronic device, as shown in fig. 6, which may include: processor 610, communication interface (Communications Interface) 620, memory 630, and communication bus 640, wherein processor 610, communication interface 620, and memory 630 communicate with each other via communication bus 640. The processor 610 may invoke logic instructions in the memory 630 to perform a touch recognition method that includes acquiring gesture motion information of a user based on the infrared light source and the infrared camera;
and under the condition that the gesture motion information is matched with a preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information.
Further, the logic instructions in the memory 630 may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute the touch recognition method provided by the above methods, and the method includes:
acquiring gesture action information of a user based on the infrared light source and the infrared camera;
and under the condition that the gesture motion information is matched with a preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information.
In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the touch recognition method provided by the above methods, the method comprising:
acquiring gesture action information of a user based on the infrared light source and the infrared camera;
and under the condition that the gesture motion information is matched with a preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The touch identification method is characterized by being applied to a touch identification system, and the touch identification system comprises: an infrared light source, an infrared camera, and a projection device;
the method comprises the following steps:
acquiring gesture action information of a user based on the infrared light source and the infrared camera;
and under the condition that the gesture motion information is matched with a preset gesture template, converting the gesture motion information into a control instruction, wherein the control instruction is used for controlling the projection equipment to execute a preset function corresponding to the gesture motion information.
2. The touch recognition method according to claim 1, wherein the preset gesture template is constructed by:
image segmentation is carried out on an image sequence corresponding to a standard gesture template, and an interested region in the image sequence is determined;
identifying a point outline corresponding to the region of interest based on the region of interest;
determining a contour circumscribed rectangle corresponding to the point contour based on the point contour;
and determining the preset gesture template based on the coordinates of the central point of the outline circumscribed rectangle and the image sequence corresponding to the image sequence.
3. The touch recognition method according to claim 1, wherein the acquiring gesture information of the user based on the infrared light source and the infrared camera includes:
determining the number of tracking points corresponding to the user based on the infrared light source and the infrared camera;
under the condition that the number of the tracking points is larger than 0, determining the number of detection points corresponding to the user;
determining that the gesture state of the user is a leaving state under the condition that the number of the detection points is 0;
and under the condition that the number of the detection points is larger than 0, determining the gesture state based on the distance between the tracking point and the detection points.
4. The touch recognition method of claim 3, wherein the determining the gesture state based on the distance between the tracking point and the detection point comprises:
judging whether a minimum distance value is larger than a distance threshold value or not based on the distance between the tracking point and the detection point;
determining that the gesture state is a moving state if the distance minimum is greater than the distance threshold;
and determining that the gesture state is a pressing state under the condition that the minimum distance value is smaller than or equal to the distance threshold value.
5. The touch recognition method according to claim 4, wherein, in the case that the gesture motion information matches a preset gesture template, before converting the gesture motion information into a control instruction, the method further comprises:
determining the gesture motion information based on the gesture state;
under the condition that the number of the tracking points exceeds a threshold value of the number of the tracking points, matching the tracking points with the preset template gestures to obtain a matching result;
and under the condition that the matching result exceeds a matching result threshold value, determining that the gesture motion information is matched with the preset gesture template.
6. The touch recognition method according to any one of claims 1-5, wherein before the acquiring gesture information of the user based on the infrared light source and the infrared camera, the method further comprises:
determining pixel coordinates of a calibration point in the calibration image based on the calibration image acquired by the infrared camera;
determining screen coordinates of the calibration point based on the projection image corresponding to the calibration image;
and determining affine transformation matrix parameters based on the coordinate relationship between the pixel coordinates of the calibration point and the screen coordinates of the calibration point, wherein the affine transformation matrix parameters are used for converting the pixel coordinates of the image shot by the external camera into the screen coordinates of the image projected by the projection device.
7. The utility model provides a touch-control recognition device, is characterized in that is applied to touch-control recognition system, touch-control recognition system includes: an infrared light source, an infrared camera, and a projection device;
the device comprises:
the acquisition module is used for acquiring gesture action information of a user based on the infrared light source and the infrared camera;
the conversion module is used for converting the gesture motion information into a control instruction under the condition that the gesture motion information is matched with a preset gesture template, and the control instruction is used for controlling the projection equipment.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the touch recognition method of any one of claims 1 to 6 when the program is executed by the processor.
9. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the touch recognition method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the touch recognition method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211478103.XA CN116301551A (en) | 2022-11-23 | 2022-11-23 | Touch identification method, touch identification device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211478103.XA CN116301551A (en) | 2022-11-23 | 2022-11-23 | Touch identification method, touch identification device, electronic equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116301551A true CN116301551A (en) | 2023-06-23 |
Family
ID=86794770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211478103.XA Pending CN116301551A (en) | 2022-11-23 | 2022-11-23 | Touch identification method, touch identification device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116301551A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116631065A (en) * | 2023-07-25 | 2023-08-22 | 银河航天科技(南通)有限公司 | Gesture recognition method and device |
-
2022
- 2022-11-23 CN CN202211478103.XA patent/CN116301551A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116631065A (en) * | 2023-07-25 | 2023-08-22 | 银河航天科技(南通)有限公司 | Gesture recognition method and device |
| CN116631065B (en) * | 2023-07-25 | 2023-09-26 | 银河航天科技(南通)有限公司 | Gesture recognition method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Shen et al. | Vision-based hand interaction in augmented reality environment | |
| EP2790089A1 (en) | Portable device and method for providing non-contact interface | |
| US9195345B2 (en) | Position aware gestures with visual feedback as input method | |
| JP5201096B2 (en) | Interactive operation device | |
| US20110102570A1 (en) | Vision based pointing device emulation | |
| US20140161309A1 (en) | Gesture recognizing device and method for recognizing a gesture | |
| JP2015212849A (en) | Image processor, image processing method and image processing program | |
| CN114138121B (en) | User gesture recognition method, device and system, storage medium and computing equipment | |
| CN112068698A (en) | Interaction method and device, electronic equipment and computer storage medium | |
| CN112506340A (en) | Device control method, device, electronic device and storage medium | |
| CN101869484A (en) | Medical diagnosis device having touch screen and control method thereof | |
| US20140369559A1 (en) | Image recognition method and image recognition system | |
| Lee et al. | TiPoint: Detecting fingertip for mid-air interaction on computational resource constrained smartglasses | |
| Iannizzotto et al. | Hand tracking for human-computer interaction with graylevel visualglove: turning back to the simple way | |
| US20160140762A1 (en) | Image processing device and image processing method | |
| CN116301551A (en) | Touch identification method, touch identification device, electronic equipment and medium | |
| KR101426378B1 (en) | System and Method for Processing Presentation Event Using Depth Information | |
| EP2618237B1 (en) | Gesture-based human-computer interaction method and system, and computer storage media | |
| CN111782041A (en) | Typing method and device, equipment and storage medium | |
| Khan et al. | Computer vision based mouse control using object detection and marker motion tracking | |
| Xu et al. | Bare hand gesture recognition with a single color camera | |
| Park et al. | Interactive display of image details using a camera-coupled mobile projector | |
| JP6699406B2 (en) | Information processing device, program, position information creation method, information processing system | |
| KR20130078496A (en) | Apparatus and method for controlling electric boards using multiple hand shape detection and tracking | |
| CN111949134A (en) | Human-computer interaction method, device and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |