CN102799318B - A kind of man-machine interaction method based on binocular stereo vision and system - Google Patents
A kind of man-machine interaction method based on binocular stereo vision and system Download PDFInfo
- Publication number
- CN102799318B CN102799318B CN201210284934.3A CN201210284934A CN102799318B CN 102799318 B CN102799318 B CN 102799318B CN 201210284934 A CN201210284934 A CN 201210284934A CN 102799318 B CN102799318 B CN 102799318B
- Authority
- CN
- China
- Prior art keywords
- image
- finger tip
- coordinate
- point
- infrared light
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The present invention relates to human-computer interaction technique field, provide a kind of man-machine interaction method based on binocular stereo vision and system, the method comprises: to projecting plane projection screen uncalibrated image, the uncalibrated image on acquired projections face carries out system calibrating; To projecting plane projected image and transmitting infrared light, infrared light forms staff profile infrared light spot after running into staff; The image of the band staff profile infrared light spot on acquired projections face, calculates the finger tip coordinate of staff according to system calibrating; Be screen coordinate according to system calibrating by finger tip coordinate conversion, perform the operation of the contact corresponding with screen coordinate.The present invention is by system calibrating and infrared detection, obtain fingertip location and coordinate, realize user based on the touch control operation of finger on common projecting plane, energy is convenient and carry out man-machine interaction efficiently, without the need to installing any special panel and auxiliary locator on the projection surface, install and easy to use, cost is lower.
Description
Technical field
The present invention relates to human-computer interaction technique field, particularly relate to a kind of man-machine interaction method based on binocular stereo vision and system.
Background technology
The application of touch-screen is comparatively extensive, and application comprises smart mobile phone, video conference, interactive electric whiteboard, product interactive advertisement displaying etc.Touch screen technology roughly can be divided into touch screen class touching technique and vision class touching technique two kinds principle.Touch screen class touching technique adopts the screen of special substance, generally cover at display surface the transparent panel that one deck contains sensing wire rod, the location of finger contact position coordinates is realized by transducing signal, the realization of this touching technique needs the screen of special substance, hardware cost is higher, and screen needs installation fixing, cannot move.Vision class touching technique is by special infraluminescence pen, with the movement locus of cameras capture infraluminescence pen, or installation infrared laser instrument on operation planar, produce infrared light spot when a finger is touching the screen, catch infrared light spot by video camera and realize contact coordinate location, this touching technique needs special infraluminescence pen, or the fixed pan being provided with infrared laser just can operate, and installing and using all has inconvenience.A kind of mode of operation of being carried out man-machine interaction by projection pattern is on the projection surface also had in vision class touching technique, but this type of touch manner needs the projecting plane by special substance mostly, such as transparent or semitransparent glass, and adopt rear-projection mode, coordinate infrared laser to realize touching location again, this mode is also not too convenient in installation and use.
Summary of the invention
The present invention is intended to convenient and carries out man-machine interaction efficiently, and without the need to installing any special panel and auxiliary locator on display interface, to install and easy to use, and cost is lower.
The present invention adopts following technical scheme:
Based on a man-machine interaction method for binocular stereo vision, described method comprises the steps:
To projecting plane Projection surveying image, the uncalibrated image gathered on described projecting plane carries out system calibrating;
To projecting plane projection screen image and transmitting infrared light, described infrared light forms staff profile infrared light spot after running into staff;
Gather the image of the band staff profile infrared light spot on described projecting plane, calculate the finger tip coordinate of staff according to system calibrating;
Be screen coordinate according to system calibrating by finger tip coordinate conversion, perform the operation of the contact corresponding with described screen coordinate.
Preferably, described to projecting plane Projection surveying image, the step that the uncalibrated image gathered on described projecting plane carries out system calibrating specifically comprises:
To projecting plane projection screen uncalibrated image, form Projection surveying image;
Gather described Projection surveying image and carry out stereo calibration;
Gather described Projection surveying image and carry out Planar Mapping demarcation.
Preferably, the step that the described Projection surveying image of described collection carries out stereo calibration specifically comprises:
By the Projection surveying image of two image acquisition device many groups different angles and diverse location;
Detect the calibration point in described Projection surveying image;
Calculate inner parameter and the external parameter of described two image collecting devices;
Three-dimensional correction is carried out to described inner parameter and external parameter;
Calculate stereo calibration matrix.
Preferably, the step that the described Projection surveying image of described collection carries out Planar Mapping demarcation specifically comprises:
By image acquisition device Projection surveying image;
Extract the calibration point position in Projection surveying image by calibration point detection algorithm and calibration point is sorted;
According to the positional information of calibration point corresponding with screen uncalibrated image in the calibration point in screen uncalibrated image and Projection surveying image, calculate projective transformation matrix.
Preferably, described uncalibrated image is cross-hatch pattern picture.
Preferably, the image of the band staff profile infrared light spot on the described projecting plane of described collection, the step calculating the finger tip coordinate of staff according to system calibrating specifically comprises:
By the image image as a setting not with staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes;
By the image of the band staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes as current frame image;
Calculate the difference image of background image and current frame image;
The finger tip coordinate of the staff in difference image is calculated according to system calibrating.
Preferably, the described step calculating the finger tip coordinate of the staff in difference image according to system calibrating specifically comprises:
In difference image, extract palm profile by rim detection;
Obtain convex closure and convex defect based on described palm profile, obtain the recessed degree of depth of concave vertex, convex defect point and convex defect point, tentatively determine finger tip candidate point;
Calculate the angle of concave vertex and convex defect point, threshold decision is carried out to described angle thus screens finger tip candidate point further;
Calculate focus point according to described palm profile, and calculate the distance of finger tip candidate point and focus point, threshold decision is carried out to described distance thus determines finger tip point, merge continuous print or adjacent point in finger tip point, obtain the location of pixels coordinate of finger tip point;
According to stereo calibration matrix, by two image acquisition device to band staff profile infrared light spot image in finger tip point location of pixels coordinate calculate the three-dimensional coordinate of finger tip.
Preferably, the image of the band staff profile infrared light spot on the described projecting plane of described collection, calculate according to system calibrating in the step of the finger tip coordinate of staff, described finger tip coordinate is finger tip three-dimensional coordinate;
Described is screen coordinate according to system calibrating by finger tip coordinate conversion, also comprises before performing the step of the operation of the contact corresponding with described screen coordinate:
According to the finger tip three-dimensional coordinate calculated, judge whether finger tip is positioned at default projection plane;
When finger tip is positioned at default projection plane, performing described is screen coordinate according to system calibrating by finger tip coordinate conversion, performs the step of the touch control operation of the contact corresponding with described screen coordinate.
Preferably, at the finger tip three-dimensional coordinate that described basis calculates, judge finger tip also comprises before whether being positioned at the step of default projection plane:
Calculate the three-dimensional coordinate of calibration point, calculated the space plane equation on projecting plane by planar fit method;
The described step judging whether finger tip is positioned at default projection plane is specially: judge whether described finger tip coordinate meets space plane equation.
Preferably, described is screen coordinate according to system calibrating by finger tip coordinate conversion, and perform in the step of the operation of the contact corresponding with described screen coordinate, described finger tip coordinate is finger tip two-dimensional coordinate;
Described is screen coordinate according to system calibrating by finger tip coordinate conversion, and the step performing the operation of the contact corresponding with described screen coordinate specifically comprises:
Finger tip two-dimensional coordinate in the image of the band staff profile infrared light spot any one in two image collecting devices collected is converted to screen coordinate by projective transformation matrix;
When described finger tip is single finger tip, the contact information corresponding with described screen coordinate is converted to mouse message and executable operations; When described finger tip is multiple finger tip, activates multi-point touch operation pattern, the contact information corresponding with described screen coordinate is converted to multiple touch point information and executable operations.
Present invention also offers a kind of man-machine interactive system based on binocular stereo vision, described system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, described electronic equipment is connected with described projection arrangement and image collecting device respectively, described image collecting device is with infrared fileter, described projection arrangement is to projecting plane projected image, described infrared light supply launches infrared light to projecting plane, image on described image acquisition device projecting plane, described system adopts preceding method to carry out man-machine interaction.
Preferably, described image collector is set to two video cameras with infrared fileter or the binocular camera with infrared fileter; The infrared fileter of described video camera is removed when carrying out system calibrating.
Man-machine interaction method based on binocular stereo vision disclosed by the invention and system, by system calibrating and infrared detection, draw fingertip location and coordinate, realize user based on the touch control operation of finger on common projecting plane, man-machine interaction can be carried out convenient and efficiently, without the need to installing any special panel and auxiliary locator on the projection surface, to install and easy to use, and cost is lower.
Accompanying drawing explanation
Fig. 1 is the man-machine interaction method process flow diagram of the embodiment of the present invention 1 based on binocular stereo vision;
Fig. 2 is the detail flowchart of system calibrating in the embodiment of the present invention 1;
Fig. 3 is the detail flowchart that the embodiment of the present invention 1 neutral body is demarcated;
Fig. 4 is the detail flowchart of the embodiment of the present invention 1 midplane mapping calibrating;
Fig. 5 adopts cross-hatch pattern picture to carry out the schematic diagram of Planar Mapping demarcation in the embodiment of the present invention 1;
Fig. 6 is the detail flowchart calculating finger tip coordinate in the embodiment of the present invention 1;
Fig. 7 is the detail flowchart of step S34 in Fig. 6;
Fig. 8 is the man-machine interaction method process flow diagram of the embodiment of the present invention 2 based on binocular stereo vision;
Fig. 9 is the man-machine interactive system schematic diagram of the embodiment of the present invention 3 based on binocular stereo vision.
Embodiment
In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.
Embodiments provide a kind of man-machine interaction method based on binocular stereo vision, the method comprises the steps:
To projecting plane Projection surveying image, the uncalibrated image on acquired projections face carries out system calibrating;
To projecting plane projection screen image and transmitting infrared light, infrared light forms staff profile infrared light spot after running into staff;
The image of the band staff profile infrared light spot on acquired projections face, calculates the finger tip coordinate of staff according to system calibrating;
Be screen coordinate according to system calibrating by finger tip coordinate conversion, perform the operation of the contact corresponding with screen coordinate.
The embodiment of the present invention additionally provides a kind of man-machine interactive system based on binocular stereo vision, this system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, electronic equipment is connected with projection arrangement and image collecting device respectively, image collecting device is with infrared fileter, projection arrangement is to projecting plane projected image, infrared light supply launches infrared light to projecting plane, the image on image acquisition device projecting plane, and this system adopts preceding method to carry out man-machine interaction.
The embodiment of the present invention is by system calibrating and infrared detection, draw fingertip location and coordinate, realize user based on the touch control operation of finger on common projecting plane, man-machine interaction can be carried out convenient and efficiently, without the need to installing any special panel and auxiliary locator on the projection surface, install and easy to use, and cost is lower.
embodiment 1:
Refer to shown in Fig. 1, for the embodiment of the present invention 1 is based on the man-machine interaction method process flow diagram of binocular stereo vision.The method comprises the steps:
Step S1: Projection surveying image, carries out system calibrating.
This step is to projecting plane Projection surveying image, and the uncalibrated image on acquired projections face carries out system calibrating.Adopt the method for the present embodiment to carry out man-machine interaction, to the material on projecting plane without particular/special requirement, various ordinary desktop, wall can use as projecting plane.By projector uncalibrated image, uncalibrated image can adopt cross-hatch pattern picture.The net result of system calibrating obtains calculating the stereo calibration matrix of three-dimensional coordinate and finger tip coordinate (namely finger tip is arranged in the coordinate of the projected image on projecting plane) can be converted to the projective transformation matrix of screen coordinate.
Because need to carry out stereo calibration, so need the image collecting device of two acquired projections images, the video camera of such as binocular camera or the right placement of two the first from left, the present embodiment adopts the mode of the video camera of the right placement of two the first from left.When carrying out system calibrating, obtained inner parameter and the external parameter of two video cameras by stereoscopic vision scaling method, and then calculate stereo calibration matrix; After camera position is fixing, also need the corresponding relation calculating projected image coordinate and screen coordinate, the calculating of this corresponding relation realizes by the method projecting uncalibrated image Calculation Plane mapping matrix.In the process of carrying out stereo calibration, in order to make video camera photograph visible images, now video camera is not with infrared fileter.
Step S2: projected image and transmitting infrared light.
This step is to projecting plane projection screen image and launch infrared light, and infrared light forms staff profile infrared light spot after running into staff.The present embodiment mid-infrared light is sent by LED infrared lamp, vertical irradiation on the projection surface, produces the sightless illuminated area of human eye, when staff is placed in projecting plane, infrared light is reflected by staff, makes the shooting function with infrared fileter collect the image of band staff profile infrared light spot.
Step S3: calculate finger tip coordinate.
The image of this step collection band staff profile infrared light spot, calculates the finger tip coordinate of staff according to system calibrating.By the image of two video camera Real-time Obtaining projections, and calculate finger tip coordinate in inputting electronic equipment (such as computing machine), carry out finger detection and localization.Video camera has now installed infrared fileter additional, does not produce induction to the visible ray in environment, therefore, enters the shooting area of video camera as long as no new object, even if the content of projected image constantly changes, the image of video camera shooting also can not change.When staff enters view field, video camera just can capture finger infrared light according under image, detect fingertip location, and according to the stereo calibration matrix of two video cameras, calculate the three-dimensional coordinate of finger tip.
Step S4: coordinate transform, performs operating of contacts.
Finger tip coordinate conversion is screen coordinate according to system calibrating by this step, performs the operating of contacts corresponding with described screen coordinate.According to the finger tip three-dimensional coordinate that step S3 calculates, finger tip two-dimensional coordinate is on the projection surface converted to screen coordinate and mouse movement information by Planar Mapping demarcation.Finger tip two-dimensional coordinate in the image of the band staff profile infrared light spot that this step is collected by any one in two video cameras is converted to screen coordinate by projective transformation matrix.When the finger tip detected is single finger tip, the contact information corresponding with screen coordinate is converted to mouse message and executable operations; When the finger tip detected is multiple finger tip, activates multi-point touch operation pattern, the contact information corresponding with screen coordinate is converted to multiple touch point information and executable operations.
As shown in Figure 2, it comprises the steps: the detailed implementation of step S1 further
Step S11: Projection surveying image.
This step, to projecting plane projection screen uncalibrated image, forms Projection surveying image.The present embodiment adopts plane cross-hatch pattern picture as uncalibrated image.
Step S12: stereo calibration.
This step acquired projections uncalibrated image carries out stereo calibration.The fundamental purpose of this step is inner parameter and the external parameter of acquisition two cameras, and then can obtain the spatial positional information on finger tip and projecting plane.After the position of two video cameras is relatively fixing, stereo calibration only needs to carry out once.
Step S13: Planar Mapping is demarcated.
This step acquired projections uncalibrated image carries out Planar Mapping demarcation.The fundamental purpose of this step is the mapping relations obtaining finger tip coordinate and screen coordinate when pointing enforcement touch control operation.Planar Mapping is demarcated only needs a video camera just can realize.
As shown in Figure 3, the step of stereo calibration comprises the detailed implementation of step S12 further:
Step S121: gather many group Projection surveying images.
This step is by the cross-hatch pattern picture of the many groups different angles on two camera acquisition projecting planes and diverse location.
Step S122: detect calibration point.
This step detects the angle point in cross-hatch pattern picture.
Step S123: calculate inner parameter and external parameter.
This step calculates inner parameter and the external parameter of two video cameras.
Step S124: three-dimensional correction.
This step carries out three-dimensional correction to inner parameter and external parameter.
Step S125: calculate stereo calibration matrix.
This step calculates stereo calibration matrix (being also three-dimensional measurement matrix) Q.
As shown in Figure 4, the step that Planar Mapping is demarcated comprises the detailed implementation of step S13 further:
Step S131: acquired projections uncalibrated image.
This step is by the cross-hatch pattern picture on camera acquisition projecting plane.Referring to shown in Fig. 5, carrying out the schematic diagram of Planar Mapping demarcation for adopting cross-hatch pattern picture in the embodiment of the present invention 1.The cross-hatch pattern of N × M has multiple angle point 10 in 1.Cross-hatch pattern picture on projecting plane, by the impact of video camera putting position, can produce certain distortion, such as, deformation pattern shown in Fig. 5.
Step S132: detect the calibration point in Projection surveying image and sort.
Extracted the corner location in the cross-hatch pattern picture on projecting plane by calibration point detection algorithm and angle point is sorted.
Step S133: calculate projective transformation matrix.
According to the positional information of angle point corresponding with screen cross-hatch pattern picture in the angle point in screen cross-hatch pattern picture and the cross-hatch pattern picture on projecting plane, calculate projective transformation matrix H, specifically can adopt Homography matrix computational approach.As shown in Figure 5, deformation pattern can be corrected as correcting image according to this projective transformation matrix H, thus obtain the position of pixel 11 pixel 12 of correspondence in correcting image (being equivalent to screen picture) in deformation pattern (being equivalent to projected image).
As shown in Figure 6, it comprises the steps: the detailed implementation of step S3 further
Step S31: gather background image.
First namely this step, by the image as a setting image of two video camera synchronous acquisitions not with staff profile infrared light spot, also take the image image I as a setting that a width does not have finger manipulation
band preserve.
Step S32: acquisition frame image.
This step passes through the image of two video camera synchronous acquisition band staff profile infrared light spot as current frame image, and definition current frame image is I
c.
Step S33: calculate difference image.
This step calculates the difference image I of background image and current frame image, I=I
b-I
c, then the image of follow-up required process is the difference image I of two images.
Step S34: calculate finger tip coordinate.
This step calculates the finger tip coordinate of the staff in difference image I according to system calibrating, and as shown in Figure 7, step S34 comprises the steps: detailed step further
Step S341: extract palm profile.
This step, in difference image I, extracts palm profile by rim detection, such as, go out palm profile by Threshold segmentation and Canny contours extract operator extraction.
Step S342: the recessed degree of depth obtaining concave vertex, convex defect point and convex defect point, tentatively determines finger tip candidate point.
This step obtains convex closure and convex defect based on the palm profile that step S341 extracts, and obtains the recessed degree of depth of concave vertex, convex defect point and convex defect point.
If T [x], x=1 ..., N is all convex defect point detected, N is the sum of all convex defect points.Shown in the structure C vConvexityDefect of each convex defect point T [x] is defined as follows, wherein CvPoint*start and CvPoint*end is concave vertex, CvPoint*depth_point is convex defect point, and float depth is the recessed degree of depth of convex defect point.
Because the degree of depth of webs is obviously larger, therefore can according to the recessed degree of depth determination finger tip candidate point of convex defect point, namely when the recessed degree of depth is in certain threshold range, then think finger tip candidate point, if depth1 and depth2 is the threshold value of the recessed degree of depth, the decision method of finger tip candidate point as shown in Equation 1:
Step S343: the angle calculating concave vertex and convex defect point, further screening finger tip candidate point.
This step calculates the angle of concave vertex and convex defect point, carries out threshold decision thus screen finger tip candidate point further to described angle.Angle can utilize the vector product of 3 to try to achieve.If T [i] is one of them finger tip candidate point, define vector v 1 and v2 according to formula 2 and formula 3:
According to the angle α of formula 3 compute vector v1 and v2, when angle α region [0,90) in time, then think that T [i] is finger tip candidate point, the threshold decision of angle α undertaken by formula 4.
Step S344: the location of pixels coordinate calculating finger tip point.
The palm profile that this step is extracted according to step S341 calculates focus point, and the distance of the finger tip candidate point that filters out of calculation procedure S343 and focus point, threshold decision is carried out to this distance thus determines finger tip point, merge continuous print or adjacent point in finger tip point, obtain the location of pixels coordinate of finger tip point.
If T [i] is one of them finger tip candidate point, O point is the focus point utilizing profile to obtain, and the distance d of finger tip candidate point and focus point defines as shown in Equation 5:
If d is between threshold value d1 and d2, then T [i] .start and T [i] .end is finger tip point.Merge continuous print or adjacent very near point in finger tip point, the point of finally trying to achieve is the location of pixels coordinate of finger tip point.Leave this location of pixels coordinate in P [i], i=1 ..., in M, M is the number of finger tip, M ∈ [0,10].
Step S345: the three-dimensional coordinate calculating finger tip.
This step according to stereo calibration matrix, by two camera acquisitions to band staff profile infrared light spot image in finger tip point location of pixels coordinate calculate the three-dimensional coordinate of finger tip.
First, according to two camera acquisitions to image in corresponding finger tip coordinate, calculate the three-dimensional coordinate P of single or multiple finger tip
i(X
w, Y
w, Z
w).If P
l[i] is the finger tip point of in left image, P
r[i] is finger tip point corresponding in right image, by the stereo calibration matrix Q for calculating three-dimensional coordinate obtained after stereo calibration, can calculate finger tip three-dimensional coordinate P according to formula 6 and formula 7
i(X
w, Y
w, Z
w).
embodiment 2:
The difference of the present embodiment and embodiment 1 is, the present embodiment is after calculating the three-dimensional coordinate of finger tip, be before screen coordinate by finger tip coordinate conversion, first judge whether finger tip is positioned at default projection plane, when finger tip is positioned at default projection plane, just think that staff have issued touch-control order, now finger tip coordinate is converted to screen coordinate and mouse movement information by projective transformation matrix H.Refer to shown in Fig. 8, for the embodiment of the present invention 2 is based on the man-machine interaction method process flow diagram of binocular stereo vision.For convenience of explanation, only list the step from acquisition frame image in the drawings, step is before this similar to Example 1, no longer describes in detail herein.The method mainly comprises the steps:
Steps A 1: acquisition frame image.
This step passes through the image of two video camera synchronous acquisition band staff profile infrared light spot as current frame image, and definition current frame image is I
c.
Steps A 2: calculate difference image.
This step calculates background image I
bwith current frame image I
cdifference image I, I=I
b-I
c, then the image of follow-up required process is the difference image I of two images.
Steps A 3: calculate finger tip coordinate.
This step calculates the finger tip coordinate of the staff in difference image according to system calibrating, and the step S34 of specific implementation and embodiment 1 is similar, repeats no more herein.
Steps A 4: judge whether finger tip is positioned at default projection plane, if so, then performs steps A 5, otherwise returns execution steps A 1.
The finger tip coordinate that steps A 3 calculates is finger tip three-dimensional coordinate P
i, the finger tip three-dimensional coordinate P that this step calculates according to steps A 3
i, judge whether finger tip is positioned at default projection plane.Default projection plane comprises multiple, and such as presetting projection plane is exactly projecting plane, when staff touches projecting plane, when also namely finger tip planimetric coordinates is identical with projecting plane planimetric coordinates, just thinks that finger has touched projecting plane, so just can send touch-control order; Or default projection plane comprises projecting plane and from the space within the scope of projecting plane up certain distance, as long as staff is in this distance range preset, all thinks that finger contact has arrived projecting plane, can send touch-control order.Certainly, in order to avoid maloperation, this distance range should be the distance shorter apart from projecting plane, it is a given height error scope that this distance range also can be understood as, make finger tip planimetric coordinates and projecting plane planimetric coordinates close to when overlapping, just carry out location, contact, contact coordinate is converted to screen coordinate, realize corresponding touch control operation.
When default projection plane is exactly projecting plane, need in the system calibrating stage first according to the three-dimensional coordinate of stereo calibration matrix Q calculating calibration point, according to the space plane equation L of the calibration point three-dimensional coordinate calculated by planar fit method calculating projecting plane.Judge whether finger tip three-dimensional coordinate meets space plane equation L in this step, if meet, then illustrate that this finger tip point is positioned on projecting plane, touch-control order can be sent.
Steps A 5: coordinate transform, performs operating of contacts.
Finger tip coordinate conversion, when finger tip is positioned at default projection plane, is screen coordinate according to projective transformation matrix by this step, and performs the touch control operation of the contact corresponding with screen coordinate.
If finger tip is positioned at default projection plane, also namely judge that finger has touched projecting plane, then select the image (shooting of arbitrary video camera, be described for left image) of video camera shooting below, according to formula 8 and formula 9 by finger tip coordinate P
l[i], by projective transformation matrix H, is converted to screen coordinate (x
p, y
p), if single finger manipulation is then converted to mouse message, if multiple finger tip detected, then activate multi-point touch operation pattern, realize the interactive operation of finger.
embodiment 3:
Refer to shown in Fig. 9, for the embodiment of the present invention 3 is based on the man-machine interactive system process flow diagram of binocular stereo vision, this system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, in the present embodiment, electronic equipment adopts computing machine 1, and projection arrangement adopts micro projector 2, and image collecting device adopts two video cameras 3, infrared light supply adopts two LED infrared lamps 4, and projecting plane 5 generates on common desktop 6.Projection pattern is vertical projection, and micro projector 2, two video cameras 3 and two LED infrared lamps 4 are all vertically mounted on the top on projecting plane 5.Computing machine 1 is connected with micro projector 2 and two video cameras 3 respectively; Two video cameras 3 are common video camera, but all need band infrared fileter (not shown); Micro projector 2 is to the screen picture of projecting plane 5 projecting computer 1; Two LED infrared lamps 4 launch infrared light to projecting plane 5; Image on two video camera 3 acquired projections faces 5.This system adopts the method for embodiment 1 or 2 to carry out man-machine interaction.The infrared fileter removing video camera 3 is needed when carrying out system calibrating.
In the present embodiment, electronic equipment adopts independently computing machine 1; In addition, electronic equipment can also adopt Embedded mode, forms a whole with miscellaneous part, and such system can depart from external computing machine and work alone; Electronic equipment also can be that other possess the terminal device of data-handling capacity, as smart mobile phone, panel computer etc.Micro projector 2 is common projector, and adopt LCOS chip or DLP chip, brightness is more than 50 lumens.The resolution of video camera 3 is 640 × 480 pixels, and speed is 30 frames/second.The centre wavelength that infrared light launched by LED infrared lamp 4 is 850nm, and also can adopt the LED infrared lamp of other wave band and coordinate corresponding infrared fileter to realize, fan angle launched by LED infrared lamp 4 is 120 degree, and power is 100mw.The infrared fileter installed additional before video camera 3 is operated in 850nm infrared band, in order to improve the anti-interference of whole system light to external world, the present embodiment adopts arrowband infrared fileter, can greatly reduce the interference of extraneous light like this, improves positioning precision and image processing speed.Desktop 6 is smooth, does not have other particular/special requirements, for strengthening drop shadow effect, recommends light desktop, and the flat surfaces such as blank sheet of paper, metope or ground can certainly be used to replace desktop 6.In the present embodiment, micro projector 2, two video cameras 3 and two LED infrared lamps 4 are packaged into a single unit system, be installed on the top on projecting plane 5, such mode makes whole device small and exquisite portable, also can adopt other modes, such as, micro projector 2 and two video cameras 3 are packaged into an entirety, and using infrared LED 4 lamp as external separate unit, coupled system work.
The workflow of this system is as follows:
1, computing machine 1 passes through micro projector 2 to projecting plane 5 Projection surveying image, remove the infrared fileter of two video cameras 3, uncalibrated image taken by two video cameras 3, be transferred to computing machine 1, carry out system calibrating (comprising stereo calibration and plane mapping calibrating) by computing machine 1, obtain stereo calibration matrix and projective transformation matrix.
2, LED infrared lamp vertical irradiation on the projection surface, two video camera 3 Real-time Obtaining projected images, and input in computing machine 1 and carry out finger detection and localization; When staff enters in region, projecting plane 5, two, left and right video camera 3 just can capture the image of finger under infrared light shines.
3, computing machine 1 goes out the finger tip three-dimensional coordinate in the image of video camera 3 transmission according to stereo calibration matrix computations.
4, judge whether this finger tip three-dimensional coordinate is positioned at default projection plane, then finger tip three-dimensional coordinate is converted to screen coordinate by projective transformation matrix, perform the operation of the contact corresponding with screen coordinate.
The embodiment of the present invention can carry out man-machine interaction easily and efficiently, structure is simple, available finger direct control, do not need special infrared interaction pen, also without the need to installing any panel and auxiliary locator (such as infrared laser) on existing projecting plane, install and use simple and convenient, cost is low.By system calibrating and finger tip detection algorithm, realize high-precision finger tip contacts and detect and finger tip location.Owing to adopting infrared technique, system works is stablized, and not by ambient light (as extraneous visible ray) interference, system accuracy is high, fast response time, and Stability and adaptability is strong.To projecting plane material without particular/special requirement, various ordinary desktop, wall all normally can use, there is the mode of operation of portability and hommization.Adopt the multiple interaction function that the method for the embodiment of the present invention or system can realize on the table, as played chess, the input of desktop game, dummy keyboard and computation etc.
Through many experiments, and test under various illumination condition, the method and system of the embodiment of the present invention is workable, scaling method simple and fast, if video camera and projector position change, only need the extremely short time (as less than 1 second) again to demarcate with regard to feasible system, and enter duty at once.Finger positioning precision is high, uses 300,000 pixel common camera, and the error of finger tip three-dimensional coordinate location, within 2 millimeters, meets the demand of application completely.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, all any amendments done within the spirit and principles in the present invention, equivalent replacement and improvement etc., all should be included within protection scope of the present invention.
Claims (9)
1. based on a man-machine interaction method for binocular stereo vision, it is characterized in that, described method comprises the steps:
To projecting plane Projection surveying image, the uncalibrated image gathered on described projecting plane carries out system calibrating;
To projecting plane projection screen image and transmitting infrared light, described infrared light forms staff profile infrared light spot after running into staff;
Gather the image of the band staff profile infrared light spot on described projecting plane, the finger tip coordinate of staff is calculated according to system calibrating, specifically comprise: by the image image as a setting not with staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes, by the image of the band staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes as current frame image, calculate the difference image of background image and current frame image, calculate the finger tip coordinate of the staff in difference image according to system calibrating;
Calculate the three-dimensional coordinate of calibration point, calculated the space plane equation on projecting plane by planar fit method; According to the finger tip three-dimensional coordinate calculated, judge whether described finger tip coordinate meets space plane equation, judge whether finger tip is positioned at default projection plane; Described default projection plane comprises projecting plane and apart from the space within the scope of the certain distance of described projecting plane;
When finger tip is positioned at default projection plane, is screen coordinate according to system calibrating by finger tip coordinate conversion, performs the operation of the contact corresponding with described screen coordinate.
2. method according to claim 1, is characterized in that, described to projecting plane Projection surveying image, the step that the uncalibrated image gathered on described projecting plane carries out system calibrating specifically comprises:
To projecting plane projection screen uncalibrated image, form Projection surveying image;
Gather described Projection surveying image and carry out stereo calibration;
Gather described Projection surveying image and carry out Planar Mapping demarcation.
3. method according to claim 2, is characterized in that, the step that the described Projection surveying image of described collection carries out stereo calibration specifically comprises:
By the Projection surveying image of two image acquisition device many groups different angles and diverse location;
Detect the calibration point in described Projection surveying image;
Calculate inner parameter and the external parameter of described two image collecting devices;
Three-dimensional correction is carried out to described inner parameter and external parameter;
Calculate stereo calibration matrix.
4. method according to claim 2, is characterized in that, the step that the described Projection surveying image of described collection carries out Planar Mapping demarcation specifically comprises:
By image acquisition device Projection surveying image;
Extract the calibration point position in Projection surveying image by calibration point detection algorithm and calibration point is sorted;
According to the positional information of calibration point corresponding with screen uncalibrated image in the calibration point in screen uncalibrated image and Projection surveying image, calculate projective transformation matrix.
5. the method according to any one of Claims 1 to 4, described uncalibrated image is cross-hatch pattern picture.
6. method according to claim 1, is characterized in that, the described step calculating the finger tip coordinate of the staff in difference image according to system calibrating specifically comprises:
In difference image, extract palm profile by rim detection;
Obtain convex closure and convex defect based on described palm profile, obtain the recessed degree of depth of concave vertex, convex defect point and convex defect point, tentatively determine finger tip candidate point;
Calculate the angle of concave vertex and convex defect point, threshold decision is carried out to described angle thus screens finger tip candidate point further;
Calculate focus point according to described palm profile, and calculate the distance of finger tip candidate point and focus point, threshold decision is carried out to described distance thus determines finger tip point, merge continuous print or adjacent point in finger tip point, obtain the location of pixels coordinate of finger tip point;
According to stereo calibration matrix, by two image acquisition device to band staff profile infrared light spot image in finger tip point location of pixels coordinate calculate the three-dimensional coordinate of finger tip.
7. method according to claim 4, is characterized in that, described is screen coordinate according to system calibrating by finger tip coordinate conversion, and perform in the step of the operation of the contact corresponding with described screen coordinate, described finger tip coordinate is finger tip two-dimensional coordinate;
Described is screen coordinate according to system calibrating by finger tip coordinate conversion, and the step performing the operation of the contact corresponding with described screen coordinate specifically comprises:
Finger tip two-dimensional coordinate in the image of the band staff profile infrared light spot any one in two image collecting devices collected is converted to screen coordinate by projective transformation matrix;
When described finger tip is single finger tip, the contact information corresponding with described screen coordinate is converted to mouse message and executable operations; When described finger tip is multiple finger tip, activates multi-point touch operation pattern, the contact information corresponding with described screen coordinate is converted to multiple touch point information and executable operations.
8. the man-machine interactive system based on binocular stereo vision, it is characterized in that, described system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, described electronic equipment is connected with described projection arrangement and image collecting device respectively, described image collecting device is with infrared fileter, described projection arrangement is to projecting plane projected image, described infrared light supply launches infrared light to projecting plane, image on described image acquisition device projecting plane, described system adopts the method described in any one of claim 1 ~ 7 to carry out man-machine interaction.
9. system according to claim 8, is characterized in that, described image collector is set to two video cameras with infrared fileter or the binocular camera with infrared fileter; The infrared fileter of described video camera is removed when carrying out system calibrating.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210284934.3A CN102799318B (en) | 2012-08-13 | 2012-08-13 | A kind of man-machine interaction method based on binocular stereo vision and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210284934.3A CN102799318B (en) | 2012-08-13 | 2012-08-13 | A kind of man-machine interaction method based on binocular stereo vision and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102799318A CN102799318A (en) | 2012-11-28 |
| CN102799318B true CN102799318B (en) | 2015-07-29 |
Family
ID=47198437
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210284934.3A Active CN102799318B (en) | 2012-08-13 | 2012-08-13 | A kind of man-machine interaction method based on binocular stereo vision and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102799318B (en) |
Families Citing this family (55)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102945637A (en) * | 2012-11-29 | 2013-02-27 | 河海大学 | Augmented reality based embedded teaching model and method |
| CN103033145B (en) * | 2013-01-08 | 2015-09-02 | 天津锋时互动科技有限公司 | For identifying the method and system of the shape of multiple object |
| CN105138194A (en) * | 2013-01-11 | 2015-12-09 | 海信集团有限公司 | Positioning method and electronic device |
| CN103148802B (en) * | 2013-01-16 | 2015-03-11 | 吉林大学 | Overhead size binocular panoramic visual inspection system for commercial automobile |
| JP6037900B2 (en) * | 2013-03-11 | 2016-12-07 | 日立マクセル株式会社 | Operation detection device and operation detection method |
| CN103150063A (en) * | 2013-04-09 | 2013-06-12 | 北京紫霄科技有限公司 | LCD (Liquid Crystal Display)-based optical multi-touch equipment |
| CN104123552B (en) * | 2013-04-28 | 2017-08-22 | 浙江大学 | It is a kind of can be with the translation glasses of automatic identification |
| CN103345301B (en) * | 2013-06-18 | 2016-08-10 | 华为技术有限公司 | A kind of depth information acquisition method and device |
| CN103336575B (en) * | 2013-06-27 | 2016-06-29 | 深圳先进技术研究院 | The intelligent glasses system of a kind of man-machine interaction and exchange method |
| KR101438636B1 (en) * | 2013-07-25 | 2014-09-05 | 현대자동차 주식회사 | Touch point recognition method of touch screen and the system performing this |
| CN103488972B (en) * | 2013-09-09 | 2016-07-06 | 西安交通大学 | Fingertip Detection based on depth information |
| CN104598015B (en) * | 2013-10-31 | 2017-07-18 | 深圳先进技术研究院 | A kind of information sharing system and method based on intelligent glasses |
| CN104657568B (en) * | 2013-11-21 | 2017-10-03 | 深圳先进技术研究院 | Many people's moving game system and methods based on intelligent glasses |
| CN103970367A (en) * | 2013-11-21 | 2014-08-06 | 郭轩 | Stereoscopic vision based multi-point touch screen system |
| CN103713737B (en) * | 2013-12-12 | 2017-01-11 | 中国科学院深圳先进技术研究院 | Virtual keyboard system used for Google glasses |
| CN103728902A (en) * | 2013-12-24 | 2014-04-16 | 华南农业大学 | Man-machine interaction device suitable for paddy field power-driven operation machine and application of man-machine interaction device |
| CN104750286B (en) * | 2013-12-26 | 2017-12-29 | 联想(北京)有限公司 | A kind of data capture method and electronic equipment |
| CN103777754B (en) * | 2014-01-10 | 2017-01-11 | 上海大学 | Hand motion tracking device and method based on binocular infrared vision |
| CN104656902A (en) * | 2015-02-26 | 2015-05-27 | 华南理工大学 | Method and system for visual processing human-machine interactive projection based on Android |
| CN106152937B (en) * | 2015-03-31 | 2019-10-25 | 深圳超多维科技有限公司 | Space positioning apparatus, system and method |
| CN105373266A (en) * | 2015-11-05 | 2016-03-02 | 上海影火智能科技有限公司 | Novel binocular vision based interaction method and electronic whiteboard system |
| CN105547188A (en) * | 2015-12-07 | 2016-05-04 | 中国科学院地球环境研究所 | Three-dimensional scanning system and three-dimensional scanning method for measuring volume of loose sediment block sample |
| US10948994B2 (en) | 2016-02-29 | 2021-03-16 | Huawei Technologies Co., Ltd. | Gesture control method for wearable system and wearable system |
| CN107180406B (en) * | 2016-03-09 | 2019-03-15 | 腾讯科技（深圳）有限公司 | Image processing method and equipment |
| CN106204604B (en) * | 2016-04-29 | 2019-04-02 | 北京仁光科技有限公司 | Project touch control display apparatus and its exchange method |
| CN106095276A (en) * | 2016-06-20 | 2016-11-09 | 联想(北京)有限公司 | A kind of electronic equipment and control method thereof |
| CN105955554A (en) * | 2016-06-23 | 2016-09-21 | 苏州和氏设计营造股份有限公司 | Multi-ocular video camera-based projection multipoint touch table apparatus |
| CN106373085A (en) * | 2016-09-20 | 2017-02-01 | 福州大学 | Intelligent terminal 3D watch try-on method and system based on augmented reality |
| CN106373086A (en) * | 2016-09-29 | 2017-02-01 | 福州大学 | Augmented reality-based ring 2D online real-time trying method and system |
| CN106709952B (en) * | 2016-11-18 | 2018-11-13 | 南京仁光电子科技有限公司 | A kind of automatic calibration method of display screen |
| CN107133984A (en) * | 2017-03-24 | 2017-09-05 | 深圳奥比中光科技有限公司 | The scaling method and system of depth camera and main equipment |
| CN108230383B (en) * | 2017-03-29 | 2021-03-23 | 北京市商汤科技开发有限公司 | Hand three-dimensional data determination method and device and electronic equipment |
| CN106873789B (en) * | 2017-04-20 | 2020-07-07 | 歌尔科技有限公司 | Projection system |
| CN107122083B (en) * | 2017-04-25 | 2020-02-18 | 上海唱风信息科技有限公司 | Touch detection method of projection surface |
| CN107301650A (en) * | 2017-06-28 | 2017-10-27 | 湖南瑞森可机器人科技有限公司 | Four connect chess checkerboard image processing method, system and man-machine chess's system |
| CN107506023B (en) * | 2017-07-20 | 2020-12-08 | 武汉秀宝软件有限公司 | Wall surface image infrared ray light spot tracking method and system |
| CN107656637B (en) * | 2017-08-28 | 2018-06-05 | 哈尔滨拓博科技有限公司 | A kind of automation scaling method using the projected keyboard for choosing manually at 4 points |
| CN107633537B (en) * | 2017-08-28 | 2021-08-27 | 深圳市德赛微电子技术有限公司 | Camera calibration method based on projection |
| CN107577376A (en) * | 2017-08-30 | 2018-01-12 | 努比亚技术有限公司 | A kind of control method and terminal |
| CN107632744A (en) * | 2017-09-09 | 2018-01-26 | 深圳大学 | A kind of desktop alternative projection system and control method |
| CN107818585B (en) * | 2017-09-27 | 2020-05-29 | 歌尔科技有限公司 | Method and device for determining finger position information of user, projector and projection system |
| CN107798700B (en) * | 2017-09-27 | 2019-12-13 | 歌尔科技有限公司 | Method and device for determining finger position information of user, projector and projection system |
| CN107818584B (en) * | 2017-09-27 | 2020-03-17 | 歌尔科技有限公司 | Method and device for determining finger position information of user, projector and projection system |
| CN107943351B (en) * | 2017-11-22 | 2021-01-05 | 苏州佳世达光电有限公司 | Projection surface touch identification system and method |
| CN108335664B (en) * | 2018-01-19 | 2019-11-05 | 长春希达电子技术有限公司 | The method that camera image curved surface is demarcated using two-dimentional relative movement mode |
| CN110471576B (en) * | 2018-08-16 | 2023-11-17 | 中山叶浪智能科技有限责任公司 | Single-camera near-screen touch method, system, platform and storage medium |
| CN110471577B (en) * | 2018-08-17 | 2023-08-22 | 中山叶浪智能科技有限责任公司 | 360-degree omnibearing virtual touch control method, system, platform and storage medium |
| CN109343766B (en) * | 2018-08-23 | 2021-03-23 | 视联动力信息技术股份有限公司 | Touch operation method and device |
| CN109409351B (en) * | 2018-10-29 | 2021-11-19 | 北京西潼科技有限公司 | Amusement equipment based on projection |
| CN110069131B (en) * | 2019-03-01 | 2022-07-22 | 中国人民解放军军事科学院国防科技创新研究院 | Multi-fingertip positioning method based on near-infrared light circular spot detection |
| CN113424524B (en) | 2019-03-27 | 2023-02-14 | Oppo广东移动通信有限公司 | Three-dimensional modeling using hemispherical or spherical visible depth images |
| CN110347007B (en) * | 2019-05-05 | 2021-06-04 | 青岛小鸟看看科技有限公司 | Method and device for calibrating laser in projection lamp |
| CN111739359A (en) * | 2020-06-30 | 2020-10-02 | 上海乂学教育科技有限公司 | Augmented reality courseware generation system |
| CN112509138B (en) * | 2020-12-11 | 2023-07-18 | 电子科技大学 | LCOS-based high-precision three-dimensional reconstruction system for indoor plastering robot |
| CN112672122B (en) * | 2020-12-15 | 2022-05-24 | 深圳市普汇智联科技有限公司 | Method and system for calibrating projection and camera mapping relation errors |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1912816A (en) * | 2005-08-08 | 2007-02-14 | 北京理工大学 | Virtus touch screen system based on camera head |
| CN101639747A (en) * | 2009-08-31 | 2010-02-03 | 广东威创视讯科技股份有限公司 | Spatial three-dimensional positioning method |
| CN101807114A (en) * | 2010-04-02 | 2010-08-18 | 浙江大学 | Natural interactive method based on three-dimensional gestures |
| CN101907954A (en) * | 2010-07-02 | 2010-12-08 | 中国科学院深圳先进技术研究院 | Interactive projection system and interactive projection method |
| CN102184021A (en) * | 2011-05-27 | 2011-09-14 | 华南理工大学 | Television man-machine interaction method based on handwriting input and fingertip mouse |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101393497A (en) * | 2008-10-30 | 2009-03-25 | 上海交通大学 | Multi-point touch method based on binocular stereo vision |
| US20140002367A1 (en) * | 2010-12-30 | 2014-01-02 | Jesper Glückstad | System and device with three-dimensional image display |
-
2012
- 2012-08-13 CN CN201210284934.3A patent/CN102799318B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1912816A (en) * | 2005-08-08 | 2007-02-14 | 北京理工大学 | Virtus touch screen system based on camera head |
| CN101639747A (en) * | 2009-08-31 | 2010-02-03 | 广东威创视讯科技股份有限公司 | Spatial three-dimensional positioning method |
| CN101807114A (en) * | 2010-04-02 | 2010-08-18 | 浙江大学 | Natural interactive method based on three-dimensional gestures |
| CN101907954A (en) * | 2010-07-02 | 2010-12-08 | 中国科学院深圳先进技术研究院 | Interactive projection system and interactive projection method |
| CN102184021A (en) * | 2011-05-27 | 2011-09-14 | 华南理工大学 | Television man-machine interaction method based on handwriting input and fingertip mouse |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102799318A (en) | 2012-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102799318B (en) | A kind of man-machine interaction method based on binocular stereo vision and system | |
| US20170351324A1 (en) | Camera-based multi-touch interaction apparatus, system and method | |
| CN102323868B (en) | Man-machine multipoint interaction method and device based on infrared image | |
| US20120249422A1 (en) | Interactive input system and method | |
| CN104067209B (en) | Interactive pointer detection with image frame processing | |
| CN102722254B (en) | Method and system for location interaction | |
| CN103279225B (en) | Projection type man-machine interactive system and touch control identification method | |
| CN102508578B (en) | Projection positioning device and method as well as interaction system and method | |
| WO2016168786A1 (en) | Augmented interface authoring | |
| US20120293555A1 (en) | Information-processing device, method thereof and display device | |
| Katz et al. | A multi-touch surface using multiple cameras | |
| CN100478860C (en) | Electronic plane display positioning system and positioning method | |
| CN104423721A (en) | Frameless multipoint touch man-machine interaction method and system based on radar eye | |
| CN102566827A (en) | Method and system for detecting object in virtual touch screen system | |
| CN103905865A (en) | Touchable intelligent television and method for achieving touching | |
| CN102193687A (en) | Multipoint touch screen interactive system based on LABVIEW | |
| CN101807115B (en) | Interactive stereo display system and distance calculating method | |
| KR102158613B1 (en) | Method of space touch detecting and display device performing the same | |
| CN203606780U (en) | Multi-touch and gesture recognition fusion system | |
| CN1674042A (en) | Contact screen information inputting positioning method based on image recognition | |
| CN109871178A (en) | A kind of virtual touch screen system based on image recognition | |
| CN102799344A (en) | Virtual touch screen system and method | |
| CN102221938A (en) | Touch positioning method and system as well as display | |
| Candela et al. | HumanTop: A multi-object tracking tabletop | |
| Cheng et al. | Fingertip-based interactive projector–camera system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |