CN102799318B - A kind of man-machine interaction method based on binocular stereo vision and system - Google Patents

A kind of man-machine interaction method based on binocular stereo vision and system Download PDF

Info

Publication number
CN102799318B
CN102799318B CN201210284934.3A CN201210284934A CN102799318B CN 102799318 B CN102799318 B CN 102799318B CN 201210284934 A CN201210284934 A CN 201210284934A CN 102799318 B CN102799318 B CN 102799318B
Authority
CN
China
Prior art keywords
image
described
finger tip
coordinate
point
Prior art date
Application number
CN201210284934.3A
Other languages
Chinese (zh)
Other versions
CN102799318A (en
Inventor
宋展
郗瑶颖
马天驰
刘晶
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Priority to CN201210284934.3A priority Critical patent/CN102799318B/en
Publication of CN102799318A publication Critical patent/CN102799318A/en
Application granted granted Critical
Publication of CN102799318B publication Critical patent/CN102799318B/en

Links

Abstract

The present invention relates to human-computer interaction technique field, provide a kind of man-machine interaction method based on binocular stereo vision and system, the method comprises: to projecting plane projection screen uncalibrated image, the uncalibrated image on acquired projections face carries out system calibrating; To projecting plane projected image and transmitting infrared light, infrared light forms staff profile infrared light spot after running into staff; The image of the band staff profile infrared light spot on acquired projections face, calculates the finger tip coordinate of staff according to system calibrating; Be screen coordinate according to system calibrating by finger tip coordinate conversion, perform the operation of the contact corresponding with screen coordinate.The present invention is by system calibrating and infrared detection, obtain fingertip location and coordinate, realize user based on the touch control operation of finger on common projecting plane, energy is convenient and carry out man-machine interaction efficiently, without the need to installing any special panel and auxiliary locator on the projection surface, install and easy to use, cost is lower.

Description

A kind of man-machine interaction method based on binocular stereo vision and system

Technical field

The present invention relates to human-computer interaction technique field, particularly relate to a kind of man-machine interaction method based on binocular stereo vision and system.

Background technology

The application of touch-screen is comparatively extensive, and application comprises smart mobile phone, video conference, interactive electric whiteboard, product interactive advertisement displaying etc.Touch screen technology roughly can be divided into touch screen class touching technique and vision class touching technique two kinds principle.Touch screen class touching technique adopts the screen of special substance, generally cover at display surface the transparent panel that one deck contains sensing wire rod, the location of finger contact position coordinates is realized by transducing signal, the realization of this touching technique needs the screen of special substance, hardware cost is higher, and screen needs installation fixing, cannot move.Vision class touching technique is by special infraluminescence pen, with the movement locus of cameras capture infraluminescence pen, or installation infrared laser instrument on operation planar, produce infrared light spot when a finger is touching the screen, catch infrared light spot by video camera and realize contact coordinate location, this touching technique needs special infraluminescence pen, or the fixed pan being provided with infrared laser just can operate, and installing and using all has inconvenience.A kind of mode of operation of being carried out man-machine interaction by projection pattern is on the projection surface also had in vision class touching technique, but this type of touch manner needs the projecting plane by special substance mostly, such as transparent or semitransparent glass, and adopt rear-projection mode, coordinate infrared laser to realize touching location again, this mode is also not too convenient in installation and use.

Summary of the invention

The present invention is intended to convenient and carries out man-machine interaction efficiently, and without the need to installing any special panel and auxiliary locator on display interface, to install and easy to use, and cost is lower.

The present invention adopts following technical scheme:

Based on a man-machine interaction method for binocular stereo vision, described method comprises the steps:

To projecting plane Projection surveying image, the uncalibrated image gathered on described projecting plane carries out system calibrating;

To projecting plane projection screen image and transmitting infrared light, described infrared light forms staff profile infrared light spot after running into staff;

Gather the image of the band staff profile infrared light spot on described projecting plane, calculate the finger tip coordinate of staff according to system calibrating;

Be screen coordinate according to system calibrating by finger tip coordinate conversion, perform the operation of the contact corresponding with described screen coordinate.

Preferably, described to projecting plane Projection surveying image, the step that the uncalibrated image gathered on described projecting plane carries out system calibrating specifically comprises:

To projecting plane projection screen uncalibrated image, form Projection surveying image;

Gather described Projection surveying image and carry out stereo calibration;

Gather described Projection surveying image and carry out Planar Mapping demarcation.

Preferably, the step that the described Projection surveying image of described collection carries out stereo calibration specifically comprises:

By the Projection surveying image of two image acquisition device many groups different angles and diverse location;

Detect the calibration point in described Projection surveying image;

Calculate inner parameter and the external parameter of described two image collecting devices;

Three-dimensional correction is carried out to described inner parameter and external parameter;

Calculate stereo calibration matrix.

Preferably, the step that the described Projection surveying image of described collection carries out Planar Mapping demarcation specifically comprises:

By image acquisition device Projection surveying image;

Extract the calibration point position in Projection surveying image by calibration point detection algorithm and calibration point is sorted;

According to the positional information of calibration point corresponding with screen uncalibrated image in the calibration point in screen uncalibrated image and Projection surveying image, calculate projective transformation matrix.

Preferably, described uncalibrated image is cross-hatch pattern picture.

Preferably, the image of the band staff profile infrared light spot on the described projecting plane of described collection, the step calculating the finger tip coordinate of staff according to system calibrating specifically comprises:

By the image image as a setting not with staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes;

By the image of the band staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes as current frame image;

Calculate the difference image of background image and current frame image;

The finger tip coordinate of the staff in difference image is calculated according to system calibrating.

Preferably, the described step calculating the finger tip coordinate of the staff in difference image according to system calibrating specifically comprises:

In difference image, extract palm profile by rim detection;

Obtain convex closure and convex defect based on described palm profile, obtain the recessed degree of depth of concave vertex, convex defect point and convex defect point, tentatively determine finger tip candidate point;

Calculate the angle of concave vertex and convex defect point, threshold decision is carried out to described angle thus screens finger tip candidate point further;

Calculate focus point according to described palm profile, and calculate the distance of finger tip candidate point and focus point, threshold decision is carried out to described distance thus determines finger tip point, merge continuous print or adjacent point in finger tip point, obtain the location of pixels coordinate of finger tip point;

According to stereo calibration matrix, by two image acquisition device to band staff profile infrared light spot image in finger tip point location of pixels coordinate calculate the three-dimensional coordinate of finger tip.

Preferably, the image of the band staff profile infrared light spot on the described projecting plane of described collection, calculate according to system calibrating in the step of the finger tip coordinate of staff, described finger tip coordinate is finger tip three-dimensional coordinate;

Described is screen coordinate according to system calibrating by finger tip coordinate conversion, also comprises before performing the step of the operation of the contact corresponding with described screen coordinate:

According to the finger tip three-dimensional coordinate calculated, judge whether finger tip is positioned at default projection plane;

When finger tip is positioned at default projection plane, performing described is screen coordinate according to system calibrating by finger tip coordinate conversion, performs the step of the touch control operation of the contact corresponding with described screen coordinate.

Preferably, at the finger tip three-dimensional coordinate that described basis calculates, judge finger tip also comprises before whether being positioned at the step of default projection plane:

Calculate the three-dimensional coordinate of calibration point, calculated the space plane equation on projecting plane by planar fit method;

The described step judging whether finger tip is positioned at default projection plane is specially: judge whether described finger tip coordinate meets space plane equation.

Preferably, described is screen coordinate according to system calibrating by finger tip coordinate conversion, and perform in the step of the operation of the contact corresponding with described screen coordinate, described finger tip coordinate is finger tip two-dimensional coordinate;

Described is screen coordinate according to system calibrating by finger tip coordinate conversion, and the step performing the operation of the contact corresponding with described screen coordinate specifically comprises:

Finger tip two-dimensional coordinate in the image of the band staff profile infrared light spot any one in two image collecting devices collected is converted to screen coordinate by projective transformation matrix;

When described finger tip is single finger tip, the contact information corresponding with described screen coordinate is converted to mouse message and executable operations; When described finger tip is multiple finger tip, activates multi-point touch operation pattern, the contact information corresponding with described screen coordinate is converted to multiple touch point information and executable operations.

Present invention also offers a kind of man-machine interactive system based on binocular stereo vision, described system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, described electronic equipment is connected with described projection arrangement and image collecting device respectively, described image collecting device is with infrared fileter, described projection arrangement is to projecting plane projected image, described infrared light supply launches infrared light to projecting plane, image on described image acquisition device projecting plane, described system adopts preceding method to carry out man-machine interaction.

Preferably, described image collector is set to two video cameras with infrared fileter or the binocular camera with infrared fileter; The infrared fileter of described video camera is removed when carrying out system calibrating.

Man-machine interaction method based on binocular stereo vision disclosed by the invention and system, by system calibrating and infrared detection, draw fingertip location and coordinate, realize user based on the touch control operation of finger on common projecting plane, man-machine interaction can be carried out convenient and efficiently, without the need to installing any special panel and auxiliary locator on the projection surface, to install and easy to use, and cost is lower.

Accompanying drawing explanation

Fig. 1 is the man-machine interaction method process flow diagram of the embodiment of the present invention 1 based on binocular stereo vision;

Fig. 2 is the detail flowchart of system calibrating in the embodiment of the present invention 1;

Fig. 3 is the detail flowchart that the embodiment of the present invention 1 neutral body is demarcated;

Fig. 4 is the detail flowchart of the embodiment of the present invention 1 midplane mapping calibrating;

Fig. 5 adopts cross-hatch pattern picture to carry out the schematic diagram of Planar Mapping demarcation in the embodiment of the present invention 1;

Fig. 6 is the detail flowchart calculating finger tip coordinate in the embodiment of the present invention 1;

Fig. 7 is the detail flowchart of step S34 in Fig. 6;

Fig. 8 is the man-machine interaction method process flow diagram of the embodiment of the present invention 2 based on binocular stereo vision;

Fig. 9 is the man-machine interactive system schematic diagram of the embodiment of the present invention 3 based on binocular stereo vision.

Embodiment

In order to make object of the present invention, technical scheme and advantage clearly understand, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, be not intended to limit the present invention.

Embodiments provide a kind of man-machine interaction method based on binocular stereo vision, the method comprises the steps:

To projecting plane Projection surveying image, the uncalibrated image on acquired projections face carries out system calibrating;

To projecting plane projection screen image and transmitting infrared light, infrared light forms staff profile infrared light spot after running into staff;

The image of the band staff profile infrared light spot on acquired projections face, calculates the finger tip coordinate of staff according to system calibrating;

Be screen coordinate according to system calibrating by finger tip coordinate conversion, perform the operation of the contact corresponding with screen coordinate.

The embodiment of the present invention additionally provides a kind of man-machine interactive system based on binocular stereo vision, this system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, electronic equipment is connected with projection arrangement and image collecting device respectively, image collecting device is with infrared fileter, projection arrangement is to projecting plane projected image, infrared light supply launches infrared light to projecting plane, the image on image acquisition device projecting plane, and this system adopts preceding method to carry out man-machine interaction.

The embodiment of the present invention is by system calibrating and infrared detection, draw fingertip location and coordinate, realize user based on the touch control operation of finger on common projecting plane, man-machine interaction can be carried out convenient and efficiently, without the need to installing any special panel and auxiliary locator on the projection surface, install and easy to use, and cost is lower.

embodiment 1:

Refer to shown in Fig. 1, for the embodiment of the present invention 1 is based on the man-machine interaction method process flow diagram of binocular stereo vision.The method comprises the steps:

Step S1: Projection surveying image, carries out system calibrating.

This step is to projecting plane Projection surveying image, and the uncalibrated image on acquired projections face carries out system calibrating.Adopt the method for the present embodiment to carry out man-machine interaction, to the material on projecting plane without particular/special requirement, various ordinary desktop, wall can use as projecting plane.By projector uncalibrated image, uncalibrated image can adopt cross-hatch pattern picture.The net result of system calibrating obtains calculating the stereo calibration matrix of three-dimensional coordinate and finger tip coordinate (namely finger tip is arranged in the coordinate of the projected image on projecting plane) can be converted to the projective transformation matrix of screen coordinate.

Because need to carry out stereo calibration, so need the image collecting device of two acquired projections images, the video camera of such as binocular camera or the right placement of two the first from left, the present embodiment adopts the mode of the video camera of the right placement of two the first from left.When carrying out system calibrating, obtained inner parameter and the external parameter of two video cameras by stereoscopic vision scaling method, and then calculate stereo calibration matrix; After camera position is fixing, also need the corresponding relation calculating projected image coordinate and screen coordinate, the calculating of this corresponding relation realizes by the method projecting uncalibrated image Calculation Plane mapping matrix.In the process of carrying out stereo calibration, in order to make video camera photograph visible images, now video camera is not with infrared fileter.

Step S2: projected image and transmitting infrared light.

This step is to projecting plane projection screen image and launch infrared light, and infrared light forms staff profile infrared light spot after running into staff.The present embodiment mid-infrared light is sent by LED infrared lamp, vertical irradiation on the projection surface, produces the sightless illuminated area of human eye, when staff is placed in projecting plane, infrared light is reflected by staff, makes the shooting function with infrared fileter collect the image of band staff profile infrared light spot.

Step S3: calculate finger tip coordinate.

The image of this step collection band staff profile infrared light spot, calculates the finger tip coordinate of staff according to system calibrating.By the image of two video camera Real-time Obtaining projections, and calculate finger tip coordinate in inputting electronic equipment (such as computing machine), carry out finger detection and localization.Video camera has now installed infrared fileter additional, does not produce induction to the visible ray in environment, therefore, enters the shooting area of video camera as long as no new object, even if the content of projected image constantly changes, the image of video camera shooting also can not change.When staff enters view field, video camera just can capture finger infrared light according under image, detect fingertip location, and according to the stereo calibration matrix of two video cameras, calculate the three-dimensional coordinate of finger tip.

Step S4: coordinate transform, performs operating of contacts.

Finger tip coordinate conversion is screen coordinate according to system calibrating by this step, performs the operating of contacts corresponding with described screen coordinate.According to the finger tip three-dimensional coordinate that step S3 calculates, finger tip two-dimensional coordinate is on the projection surface converted to screen coordinate and mouse movement information by Planar Mapping demarcation.Finger tip two-dimensional coordinate in the image of the band staff profile infrared light spot that this step is collected by any one in two video cameras is converted to screen coordinate by projective transformation matrix.When the finger tip detected is single finger tip, the contact information corresponding with screen coordinate is converted to mouse message and executable operations; When the finger tip detected is multiple finger tip, activates multi-point touch operation pattern, the contact information corresponding with screen coordinate is converted to multiple touch point information and executable operations.

As shown in Figure 2, it comprises the steps: the detailed implementation of step S1 further

Step S11: Projection surveying image.

This step, to projecting plane projection screen uncalibrated image, forms Projection surveying image.The present embodiment adopts plane cross-hatch pattern picture as uncalibrated image.

Step S12: stereo calibration.

This step acquired projections uncalibrated image carries out stereo calibration.The fundamental purpose of this step is inner parameter and the external parameter of acquisition two cameras, and then can obtain the spatial positional information on finger tip and projecting plane.After the position of two video cameras is relatively fixing, stereo calibration only needs to carry out once.

Step S13: Planar Mapping is demarcated.

This step acquired projections uncalibrated image carries out Planar Mapping demarcation.The fundamental purpose of this step is the mapping relations obtaining finger tip coordinate and screen coordinate when pointing enforcement touch control operation.Planar Mapping is demarcated only needs a video camera just can realize.

As shown in Figure 3, the step of stereo calibration comprises the detailed implementation of step S12 further:

Step S121: gather many group Projection surveying images.

This step is by the cross-hatch pattern picture of the many groups different angles on two camera acquisition projecting planes and diverse location.

Step S122: detect calibration point.

This step detects the angle point in cross-hatch pattern picture.

Step S123: calculate inner parameter and external parameter.

This step calculates inner parameter and the external parameter of two video cameras.

Step S124: three-dimensional correction.

This step carries out three-dimensional correction to inner parameter and external parameter.

Step S125: calculate stereo calibration matrix.

This step calculates stereo calibration matrix (being also three-dimensional measurement matrix) Q.

As shown in Figure 4, the step that Planar Mapping is demarcated comprises the detailed implementation of step S13 further:

Step S131: acquired projections uncalibrated image.

This step is by the cross-hatch pattern picture on camera acquisition projecting plane.Referring to shown in Fig. 5, carrying out the schematic diagram of Planar Mapping demarcation for adopting cross-hatch pattern picture in the embodiment of the present invention 1.The cross-hatch pattern of N × M has multiple angle point 10 in 1.Cross-hatch pattern picture on projecting plane, by the impact of video camera putting position, can produce certain distortion, such as, deformation pattern shown in Fig. 5.

Step S132: detect the calibration point in Projection surveying image and sort.

Extracted the corner location in the cross-hatch pattern picture on projecting plane by calibration point detection algorithm and angle point is sorted.

Step S133: calculate projective transformation matrix.

According to the positional information of angle point corresponding with screen cross-hatch pattern picture in the angle point in screen cross-hatch pattern picture and the cross-hatch pattern picture on projecting plane, calculate projective transformation matrix H, specifically can adopt Homography matrix computational approach.As shown in Figure 5, deformation pattern can be corrected as correcting image according to this projective transformation matrix H, thus obtain the position of pixel 11 pixel 12 of correspondence in correcting image (being equivalent to screen picture) in deformation pattern (being equivalent to projected image).

As shown in Figure 6, it comprises the steps: the detailed implementation of step S3 further

Step S31: gather background image.

First namely this step, by the image as a setting image of two video camera synchronous acquisitions not with staff profile infrared light spot, also take the image image I as a setting that a width does not have finger manipulation band preserve.

Step S32: acquisition frame image.

This step passes through the image of two video camera synchronous acquisition band staff profile infrared light spot as current frame image, and definition current frame image is I c.

Step S33: calculate difference image.

This step calculates the difference image I of background image and current frame image, I=I b-I c, then the image of follow-up required process is the difference image I of two images.

Step S34: calculate finger tip coordinate.

This step calculates the finger tip coordinate of the staff in difference image I according to system calibrating, and as shown in Figure 7, step S34 comprises the steps: detailed step further

Step S341: extract palm profile.

This step, in difference image I, extracts palm profile by rim detection, such as, go out palm profile by Threshold segmentation and Canny contours extract operator extraction.

Step S342: the recessed degree of depth obtaining concave vertex, convex defect point and convex defect point, tentatively determines finger tip candidate point.

This step obtains convex closure and convex defect based on the palm profile that step S341 extracts, and obtains the recessed degree of depth of concave vertex, convex defect point and convex defect point.

If T [x], x=1 ..., N is all convex defect point detected, N is the sum of all convex defect points.Shown in the structure C vConvexityDefect of each convex defect point T [x] is defined as follows, wherein CvPoint*start and CvPoint*end is concave vertex, CvPoint*depth_point is convex defect point, and float depth is the recessed degree of depth of convex defect point.

Because the degree of depth of webs is obviously larger, therefore can according to the recessed degree of depth determination finger tip candidate point of convex defect point, namely when the recessed degree of depth is in certain threshold range, then think finger tip candidate point, if depth1 and depth2 is the threshold value of the recessed degree of depth, the decision method of finger tip candidate point as shown in Equation 1:

T [ x ] = T [ x ] depth 1 < T [ x ] . depth < depth 2 NULL otherwise , x = 1 , . . . , N - - - ( 1 )

Step S343: the angle calculating concave vertex and convex defect point, further screening finger tip candidate point.

This step calculates the angle of concave vertex and convex defect point, carries out threshold decision thus screen finger tip candidate point further to described angle.Angle can utilize the vector product of 3 to try to achieve.If T [i] is one of them finger tip candidate point, define vector v 1 and v2 according to formula 2 and formula 3:

v 1 = T [ i ] . start - T [ i ] . depth _ point v 2 = T [ i ] . end - T [ i ] . depth _ point - - - ( 2 )

According to the angle α of formula 3 compute vector v1 and v2, when angle α region [0,90) in time, then think that T [i] is finger tip candidate point, the threshold decision of angle α undertaken by formula 4.

cos &alpha; = v 1 &CenterDot; v 2 | v 1 | | v 2 | - - - ( 3 )

T [ x ] = T [ x ] 0 < cos &alpha; < 1 NULL otherwise - - - ( 4 )

Step S344: the location of pixels coordinate calculating finger tip point.

The palm profile that this step is extracted according to step S341 calculates focus point, and the distance of the finger tip candidate point that filters out of calculation procedure S343 and focus point, threshold decision is carried out to this distance thus determines finger tip point, merge continuous print or adjacent point in finger tip point, obtain the location of pixels coordinate of finger tip point.

If T [i] is one of them finger tip candidate point, O point is the focus point utilizing profile to obtain, and the distance d of finger tip candidate point and focus point defines as shown in Equation 5:

d = ( T [ i ] . depth _ point . x - O . x ) 2 + ( T [ i ] . depth _ point . y - O . y ) 2 - - - ( 5 )

If d is between threshold value d1 and d2, then T [i] .start and T [i] .end is finger tip point.Merge continuous print or adjacent very near point in finger tip point, the point of finally trying to achieve is the location of pixels coordinate of finger tip point.Leave this location of pixels coordinate in P [i], i=1 ..., in M, M is the number of finger tip, M ∈ [0,10].

Step S345: the three-dimensional coordinate calculating finger tip.

This step according to stereo calibration matrix, by two camera acquisitions to band staff profile infrared light spot image in finger tip point location of pixels coordinate calculate the three-dimensional coordinate of finger tip.

First, according to two camera acquisitions to image in corresponding finger tip coordinate, calculate the three-dimensional coordinate P of single or multiple finger tip i(X w, Y w, Z w).If P l[i] is the finger tip point of in left image, P r[i] is finger tip point corresponding in right image, by the stereo calibration matrix Q for calculating three-dimensional coordinate obtained after stereo calibration, can calculate finger tip three-dimensional coordinate P according to formula 6 and formula 7 i(X w, Y w, Z w).

x y z w = Q P L [ i ] . x P L [ i ] . y P R [ i ] . x - P L [ i ] . x 1 - - - ( 6 )

X W = x / w Y W = y / w Z W = z / w - - - ( 7 )

embodiment 2:

The difference of the present embodiment and embodiment 1 is, the present embodiment is after calculating the three-dimensional coordinate of finger tip, be before screen coordinate by finger tip coordinate conversion, first judge whether finger tip is positioned at default projection plane, when finger tip is positioned at default projection plane, just think that staff have issued touch-control order, now finger tip coordinate is converted to screen coordinate and mouse movement information by projective transformation matrix H.Refer to shown in Fig. 8, for the embodiment of the present invention 2 is based on the man-machine interaction method process flow diagram of binocular stereo vision.For convenience of explanation, only list the step from acquisition frame image in the drawings, step is before this similar to Example 1, no longer describes in detail herein.The method mainly comprises the steps:

Steps A 1: acquisition frame image.

This step passes through the image of two video camera synchronous acquisition band staff profile infrared light spot as current frame image, and definition current frame image is I c.

Steps A 2: calculate difference image.

This step calculates background image I bwith current frame image I cdifference image I, I=I b-I c, then the image of follow-up required process is the difference image I of two images.

Steps A 3: calculate finger tip coordinate.

This step calculates the finger tip coordinate of the staff in difference image according to system calibrating, and the step S34 of specific implementation and embodiment 1 is similar, repeats no more herein.

Steps A 4: judge whether finger tip is positioned at default projection plane, if so, then performs steps A 5, otherwise returns execution steps A 1.

The finger tip coordinate that steps A 3 calculates is finger tip three-dimensional coordinate P i, the finger tip three-dimensional coordinate P that this step calculates according to steps A 3 i, judge whether finger tip is positioned at default projection plane.Default projection plane comprises multiple, and such as presetting projection plane is exactly projecting plane, when staff touches projecting plane, when also namely finger tip planimetric coordinates is identical with projecting plane planimetric coordinates, just thinks that finger has touched projecting plane, so just can send touch-control order; Or default projection plane comprises projecting plane and from the space within the scope of projecting plane up certain distance, as long as staff is in this distance range preset, all thinks that finger contact has arrived projecting plane, can send touch-control order.Certainly, in order to avoid maloperation, this distance range should be the distance shorter apart from projecting plane, it is a given height error scope that this distance range also can be understood as, make finger tip planimetric coordinates and projecting plane planimetric coordinates close to when overlapping, just carry out location, contact, contact coordinate is converted to screen coordinate, realize corresponding touch control operation.

When default projection plane is exactly projecting plane, need in the system calibrating stage first according to the three-dimensional coordinate of stereo calibration matrix Q calculating calibration point, according to the space plane equation L of the calibration point three-dimensional coordinate calculated by planar fit method calculating projecting plane.Judge whether finger tip three-dimensional coordinate meets space plane equation L in this step, if meet, then illustrate that this finger tip point is positioned on projecting plane, touch-control order can be sent.

Steps A 5: coordinate transform, performs operating of contacts.

Finger tip coordinate conversion, when finger tip is positioned at default projection plane, is screen coordinate according to projective transformation matrix by this step, and performs the touch control operation of the contact corresponding with screen coordinate.

If finger tip is positioned at default projection plane, also namely judge that finger has touched projecting plane, then select the image (shooting of arbitrary video camera, be described for left image) of video camera shooting below, according to formula 8 and formula 9 by finger tip coordinate P l[i], by projective transformation matrix H, is converted to screen coordinate (x p, y p), if single finger manipulation is then converted to mouse message, if multiple finger tip detected, then activate multi-point touch operation pattern, realize the interactive operation of finger.

x 1 y 1 z 1 = H P L [ i ] . x P L [ i ] . y 1 - - - ( 8 )

x p = x 1 / z 1 y p = y 1 / z 1 - - - ( 9 )

embodiment 3:

Refer to shown in Fig. 9, for the embodiment of the present invention 3 is based on the man-machine interactive system process flow diagram of binocular stereo vision, this system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, in the present embodiment, electronic equipment adopts computing machine 1, and projection arrangement adopts micro projector 2, and image collecting device adopts two video cameras 3, infrared light supply adopts two LED infrared lamps 4, and projecting plane 5 generates on common desktop 6.Projection pattern is vertical projection, and micro projector 2, two video cameras 3 and two LED infrared lamps 4 are all vertically mounted on the top on projecting plane 5.Computing machine 1 is connected with micro projector 2 and two video cameras 3 respectively; Two video cameras 3 are common video camera, but all need band infrared fileter (not shown); Micro projector 2 is to the screen picture of projecting plane 5 projecting computer 1; Two LED infrared lamps 4 launch infrared light to projecting plane 5; Image on two video camera 3 acquired projections faces 5.This system adopts the method for embodiment 1 or 2 to carry out man-machine interaction.The infrared fileter removing video camera 3 is needed when carrying out system calibrating.

In the present embodiment, electronic equipment adopts independently computing machine 1; In addition, electronic equipment can also adopt Embedded mode, forms a whole with miscellaneous part, and such system can depart from external computing machine and work alone; Electronic equipment also can be that other possess the terminal device of data-handling capacity, as smart mobile phone, panel computer etc.Micro projector 2 is common projector, and adopt LCOS chip or DLP chip, brightness is more than 50 lumens.The resolution of video camera 3 is 640 × 480 pixels, and speed is 30 frames/second.The centre wavelength that infrared light launched by LED infrared lamp 4 is 850nm, and also can adopt the LED infrared lamp of other wave band and coordinate corresponding infrared fileter to realize, fan angle launched by LED infrared lamp 4 is 120 degree, and power is 100mw.The infrared fileter installed additional before video camera 3 is operated in 850nm infrared band, in order to improve the anti-interference of whole system light to external world, the present embodiment adopts arrowband infrared fileter, can greatly reduce the interference of extraneous light like this, improves positioning precision and image processing speed.Desktop 6 is smooth, does not have other particular/special requirements, for strengthening drop shadow effect, recommends light desktop, and the flat surfaces such as blank sheet of paper, metope or ground can certainly be used to replace desktop 6.In the present embodiment, micro projector 2, two video cameras 3 and two LED infrared lamps 4 are packaged into a single unit system, be installed on the top on projecting plane 5, such mode makes whole device small and exquisite portable, also can adopt other modes, such as, micro projector 2 and two video cameras 3 are packaged into an entirety, and using infrared LED 4 lamp as external separate unit, coupled system work.

The workflow of this system is as follows:

1, computing machine 1 passes through micro projector 2 to projecting plane 5 Projection surveying image, remove the infrared fileter of two video cameras 3, uncalibrated image taken by two video cameras 3, be transferred to computing machine 1, carry out system calibrating (comprising stereo calibration and plane mapping calibrating) by computing machine 1, obtain stereo calibration matrix and projective transformation matrix.

2, LED infrared lamp vertical irradiation on the projection surface, two video camera 3 Real-time Obtaining projected images, and input in computing machine 1 and carry out finger detection and localization; When staff enters in region, projecting plane 5, two, left and right video camera 3 just can capture the image of finger under infrared light shines.

3, computing machine 1 goes out the finger tip three-dimensional coordinate in the image of video camera 3 transmission according to stereo calibration matrix computations.

4, judge whether this finger tip three-dimensional coordinate is positioned at default projection plane, then finger tip three-dimensional coordinate is converted to screen coordinate by projective transformation matrix, perform the operation of the contact corresponding with screen coordinate.

The embodiment of the present invention can carry out man-machine interaction easily and efficiently, structure is simple, available finger direct control, do not need special infrared interaction pen, also without the need to installing any panel and auxiliary locator (such as infrared laser) on existing projecting plane, install and use simple and convenient, cost is low.By system calibrating and finger tip detection algorithm, realize high-precision finger tip contacts and detect and finger tip location.Owing to adopting infrared technique, system works is stablized, and not by ambient light (as extraneous visible ray) interference, system accuracy is high, fast response time, and Stability and adaptability is strong.To projecting plane material without particular/special requirement, various ordinary desktop, wall all normally can use, there is the mode of operation of portability and hommization.Adopt the multiple interaction function that the method for the embodiment of the present invention or system can realize on the table, as played chess, the input of desktop game, dummy keyboard and computation etc.

Through many experiments, and test under various illumination condition, the method and system of the embodiment of the present invention is workable, scaling method simple and fast, if video camera and projector position change, only need the extremely short time (as less than 1 second) again to demarcate with regard to feasible system, and enter duty at once.Finger positioning precision is high, uses 300,000 pixel common camera, and the error of finger tip three-dimensional coordinate location, within 2 millimeters, meets the demand of application completely.

The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, all any amendments done within the spirit and principles in the present invention, equivalent replacement and improvement etc., all should be included within protection scope of the present invention.

Claims (9)

1. based on a man-machine interaction method for binocular stereo vision, it is characterized in that, described method comprises the steps:
To projecting plane Projection surveying image, the uncalibrated image gathered on described projecting plane carries out system calibrating;
To projecting plane projection screen image and transmitting infrared light, described infrared light forms staff profile infrared light spot after running into staff;
Gather the image of the band staff profile infrared light spot on described projecting plane, the finger tip coordinate of staff is calculated according to system calibrating, specifically comprise: by the image image as a setting not with staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes, by the image of the band staff profile infrared light spot on two image collecting device synchronous acquisition projecting planes as current frame image, calculate the difference image of background image and current frame image, calculate the finger tip coordinate of the staff in difference image according to system calibrating;
Calculate the three-dimensional coordinate of calibration point, calculated the space plane equation on projecting plane by planar fit method; According to the finger tip three-dimensional coordinate calculated, judge whether described finger tip coordinate meets space plane equation, judge whether finger tip is positioned at default projection plane; Described default projection plane comprises projecting plane and apart from the space within the scope of the certain distance of described projecting plane;
When finger tip is positioned at default projection plane, is screen coordinate according to system calibrating by finger tip coordinate conversion, performs the operation of the contact corresponding with described screen coordinate.
2. method according to claim 1, is characterized in that, described to projecting plane Projection surveying image, the step that the uncalibrated image gathered on described projecting plane carries out system calibrating specifically comprises:
To projecting plane projection screen uncalibrated image, form Projection surveying image;
Gather described Projection surveying image and carry out stereo calibration;
Gather described Projection surveying image and carry out Planar Mapping demarcation.
3. method according to claim 2, is characterized in that, the step that the described Projection surveying image of described collection carries out stereo calibration specifically comprises:
By the Projection surveying image of two image acquisition device many groups different angles and diverse location;
Detect the calibration point in described Projection surveying image;
Calculate inner parameter and the external parameter of described two image collecting devices;
Three-dimensional correction is carried out to described inner parameter and external parameter;
Calculate stereo calibration matrix.
4. method according to claim 2, is characterized in that, the step that the described Projection surveying image of described collection carries out Planar Mapping demarcation specifically comprises:
By image acquisition device Projection surveying image;
Extract the calibration point position in Projection surveying image by calibration point detection algorithm and calibration point is sorted;
According to the positional information of calibration point corresponding with screen uncalibrated image in the calibration point in screen uncalibrated image and Projection surveying image, calculate projective transformation matrix.
5. the method according to any one of Claims 1 to 4, described uncalibrated image is cross-hatch pattern picture.
6. method according to claim 1, is characterized in that, the described step calculating the finger tip coordinate of the staff in difference image according to system calibrating specifically comprises:
In difference image, extract palm profile by rim detection;
Obtain convex closure and convex defect based on described palm profile, obtain the recessed degree of depth of concave vertex, convex defect point and convex defect point, tentatively determine finger tip candidate point;
Calculate the angle of concave vertex and convex defect point, threshold decision is carried out to described angle thus screens finger tip candidate point further;
Calculate focus point according to described palm profile, and calculate the distance of finger tip candidate point and focus point, threshold decision is carried out to described distance thus determines finger tip point, merge continuous print or adjacent point in finger tip point, obtain the location of pixels coordinate of finger tip point;
According to stereo calibration matrix, by two image acquisition device to band staff profile infrared light spot image in finger tip point location of pixels coordinate calculate the three-dimensional coordinate of finger tip.
7. method according to claim 4, is characterized in that, described is screen coordinate according to system calibrating by finger tip coordinate conversion, and perform in the step of the operation of the contact corresponding with described screen coordinate, described finger tip coordinate is finger tip two-dimensional coordinate;
Described is screen coordinate according to system calibrating by finger tip coordinate conversion, and the step performing the operation of the contact corresponding with described screen coordinate specifically comprises:
Finger tip two-dimensional coordinate in the image of the band staff profile infrared light spot any one in two image collecting devices collected is converted to screen coordinate by projective transformation matrix;
When described finger tip is single finger tip, the contact information corresponding with described screen coordinate is converted to mouse message and executable operations; When described finger tip is multiple finger tip, activates multi-point touch operation pattern, the contact information corresponding with described screen coordinate is converted to multiple touch point information and executable operations.
8. the man-machine interactive system based on binocular stereo vision, it is characterized in that, described system comprises electronic equipment, projection arrangement, image collecting device, infrared light supply and projecting plane, described electronic equipment is connected with described projection arrangement and image collecting device respectively, described image collecting device is with infrared fileter, described projection arrangement is to projecting plane projected image, described infrared light supply launches infrared light to projecting plane, image on described image acquisition device projecting plane, described system adopts the method described in any one of claim 1 ~ 7 to carry out man-machine interaction.
9. system according to claim 8, is characterized in that, described image collector is set to two video cameras with infrared fileter or the binocular camera with infrared fileter; The infrared fileter of described video camera is removed when carrying out system calibrating.
CN201210284934.3A 2012-08-13 2012-08-13 A kind of man-machine interaction method based on binocular stereo vision and system CN102799318B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201210284934.3A CN102799318B (en) 2012-08-13 2012-08-13 A kind of man-machine interaction method based on binocular stereo vision and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201210284934.3A CN102799318B (en) 2012-08-13 2012-08-13 A kind of man-machine interaction method based on binocular stereo vision and system

Publications (2)

Publication Number Publication Date
CN102799318A CN102799318A (en) 2012-11-28
CN102799318B true CN102799318B (en) 2015-07-29

Family

ID=47198437

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201210284934.3A CN102799318B (en) 2012-08-13 2012-08-13 A kind of man-machine interaction method based on binocular stereo vision and system

Country Status (1)

Country Link
CN (1) CN102799318B (en)

Families Citing this family (40)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102945637A (en) * 2012-11-29 2013-02-27 河海大学 Augmented reality based embedded teaching model and method
CN103033145B (en) * 2013-01-08 2015-09-02 天津锋时互动科技有限公司 For identifying the method and system of the shape of multiple object
CN105138194A (en) * 2013-01-11 2015-12-09 海信集团有限公司 Positioning method and electronic device
CN103148802B (en) * 2013-01-16 2015-03-11 吉林大学 Overhead size binocular panoramic visual inspection system for commercial automobile
JP6037900B2 (en) * 2013-03-11 2016-12-07 日立マクセル株式会社 Operation detection device and operation detection method
CN103150063A (en) * 2013-04-09 2013-06-12 北京紫霄科技有限公司 LCD (Liquid Crystal Display)-based optical multi-touch equipment
CN104123552B (en) * 2013-04-28 2017-08-22 浙江大学 It is a kind of can be with the translation glasses of automatic identification
CN103345301B (en) * 2013-06-18 2016-08-10 华为技术有限公司 A kind of depth information acquisition method and device
CN103336575B (en) * 2013-06-27 2016-06-29 深圳先进技术研究院 The intelligent glasses system of a kind of man-machine interaction and exchange method
KR101438636B1 (en) * 2013-07-25 2014-09-05 현대자동차 주식회사 Touch point recognition method of touch screen and the system performing this
CN103488972B (en) * 2013-09-09 2016-07-06 西安交通大学 Fingertip Detection based on depth information
CN104598015B (en) * 2013-10-31 2017-07-18 深圳先进技术研究院 A kind of information sharing system and method based on intelligent glasses
CN103970367A (en) * 2013-11-21 2014-08-06 郭轩 Stereoscopic vision based multi-point touch screen system
CN104657568B (en) * 2013-11-21 2017-10-03 深圳先进技术研究院 Many people's moving game system and methods based on intelligent glasses
CN103713737B (en) * 2013-12-12 2017-01-11 中国科学院深圳先进技术研究院 Virtual keyboard system used for Google glasses
CN103728902A (en) * 2013-12-24 2014-04-16 华南农业大学 Man-machine interaction device suitable for paddy field power-driven operation machine and application of man-machine interaction device
CN104750286B (en) * 2013-12-26 2017-12-29 联想(北京)有限公司 A kind of data capture method and electronic equipment
CN103777754B (en) * 2014-01-10 2017-01-11 上海大学 Hand motion tracking device and method based on binocular infrared vision
CN104656902A (en) * 2015-02-26 2015-05-27 华南理工大学 Method and system for visual processing human-machine interactive projection based on Android
CN106152937B (en) * 2015-03-31 2019-10-25 深圳超多维科技有限公司 Space positioning apparatus, system and method
CN105373266A (en) * 2015-11-05 2016-03-02 上海影火智能科技有限公司 Novel binocular vision based interaction method and electronic whiteboard system
CN105547188A (en) * 2015-12-07 2016-05-04 中国科学院地球环境研究所 Three-dimensional scanning system and three-dimensional scanning method for measuring volume of loose sediment block sample
CN107180406B (en) * 2016-03-09 2019-03-15 腾讯科技(深圳)有限公司 Image processing method and equipment
CN106204604B (en) * 2016-04-29 2019-04-02 北京仁光科技有限公司 Project touch control display apparatus and its exchange method
CN106095276A (en) * 2016-06-20 2016-11-09 联想(北京)有限公司 A kind of electronic equipment and control method thereof
CN105955554A (en) * 2016-06-23 2016-09-21 苏州和氏设计营造股份有限公司 Multi-ocular video camera-based projection multipoint touch table apparatus
CN106373085A (en) * 2016-09-20 2017-02-01 福州大学 Intelligent terminal 3D watch try-on method and system based on augmented reality
CN106373086A (en) * 2016-09-29 2017-02-01 福州大学 Augmented reality-based ring 2D online real-time trying method and system
CN106709952B (en) * 2016-11-18 2018-11-13 南京仁光电子科技有限公司 A kind of automatic calibration method of display screen
CN107133984A (en) * 2017-03-24 2017-09-05 深圳奥比中光科技有限公司 The scaling method and system of depth camera and main equipment
CN108230383A (en) * 2017-03-29 2018-06-29 北京市商汤科技开发有限公司 Hand three-dimensional data determines method, apparatus and electronic equipment
CN106873789A (en) * 2017-04-20 2017-06-20 歌尔科技有限公司 A kind of optical projection system
CN107122083A (en) * 2017-04-25 2017-09-01 上海唱风信息科技有限公司 The touch control detecting method on perspective plane
CN107656637B (en) * 2017-08-28 2018-06-05 哈尔滨拓博科技有限公司 A kind of automation scaling method using the projected keyboard for choosing manually at 4 points
CN107577376A (en) * 2017-08-30 2018-01-12 努比亚技术有限公司 A kind of control method and terminal
CN107632744A (en) * 2017-09-09 2018-01-26 深圳大学 A kind of desktop alternative projection system and control method
CN107818584A (en) * 2017-09-27 2018-03-20 歌尔科技有限公司 Determination method and device, projecting apparatus, the optical projection system of user's finger positional information
CN107818585A (en) * 2017-09-27 2018-03-20 歌尔科技有限公司 Determination method and device, projecting apparatus, the optical projection system of user's finger positional information
CN107798700B (en) * 2017-09-27 2019-12-13 歌尔科技有限公司 Method and device for determining finger position information of user, projector and projection system
CN108335664B (en) * 2018-01-19 2019-11-05 长春希达电子技术有限公司 The method that camera image curved surface is demarcated using two-dimentional relative movement mode

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1912816A (en) * 2005-08-08 2007-02-14 北京理工大学 Virtus touch screen system based on camera head
CN101639747A (en) * 2009-08-31 2010-02-03 广东威创视讯科技股份有限公司 Spatial three-dimensional positioning method
CN101807114A (en) * 2010-04-02 2010-08-18 浙江大学 Natural interactive method based on three-dimensional gestures
CN101907954A (en) * 2010-07-02 2010-12-08 中国科学院深圳先进技术研究院 Interactive projection system, interactive projection method
CN102184021A (en) * 2011-05-27 2011-09-14 华南理工大学 Television man-machine interaction method based on handwriting input and fingertip mouse

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101393497A (en) * 2008-10-30 2009-03-25 上海交通大学 Multi-point touch method based on binocular stereo vision
WO2012089576A1 (en) * 2010-12-30 2012-07-05 Danmarks Tekniske Universitet System and device with three-dimensional image display

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1912816A (en) * 2005-08-08 2007-02-14 北京理工大学 Virtus touch screen system based on camera head
CN101639747A (en) * 2009-08-31 2010-02-03 广东威创视讯科技股份有限公司 Spatial three-dimensional positioning method
CN101807114A (en) * 2010-04-02 2010-08-18 浙江大学 Natural interactive method based on three-dimensional gestures
CN101907954A (en) * 2010-07-02 2010-12-08 中国科学院深圳先进技术研究院 Interactive projection system, interactive projection method
CN102184021A (en) * 2011-05-27 2011-09-14 华南理工大学 Television man-machine interaction method based on handwriting input and fingertip mouse

Also Published As

Publication number Publication date
CN102799318A (en) 2012-11-28

Similar Documents

Publication Publication Date Title
JP5926184B2 (en) Remote control of computer equipment
US9658765B2 (en) Image magnification system for computer interface
CA2748881C (en) Gesture recognition method and interactive input system employing the same
JP5887775B2 (en) Human computer interaction system, hand-to-hand pointing point positioning method, and finger gesture determination method
CN101971123B (en) Interactive surface computer with switchable diffuser
US9207773B1 (en) Two-dimensional method and system enabling three-dimensional user interaction with a device
US7256772B2 (en) Auto-aligning touch system and method
JP2004500657A (en) Data input method and apparatus using virtual input device
US20120326995A1 (en) Virtual touch panel system and interactive mode auto-switching method
US6512838B1 (en) Methods for enhancing performance and data acquired from three-dimensional image systems
KR20100055516A (en) Optical touchscreen with improved illumination
US20110199335A1 (en) Determining a Position of an Object Using a Single Camera
Dohse et al. Enhancing multi-user interaction with multi-touch tabletop displays using hand tracking
JP2012515966A (en) Device and method for monitoring the behavior of an object
TWI454968B (en) Three-dimensional interactive device and operation method thereof
US8971565B2 (en) Human interface electronic device
EP2645303A2 (en) Gesture recognition inrterface system
US20120169671A1 (en) Multi-touch input apparatus and its interface method using data fusion of a single touch sensor pad and an imaging sensor
US8723789B1 (en) Two-dimensional method and system enabling three-dimensional user interaction with a device
EP2353069B1 (en) Stereo optical sensors for resolving multi-touch in a touch detection system
JP2013054661A (en) Information display system, information display method and program for information display
JP2016509394A (en) Method and system for projecting digital information on a real object in a real environment
JP5950130B2 (en) Camera-type multi-touch interaction device, system and method
CN101419513B (en) An infrared laser pointer virtual virtual point touch system
TW201135558A (en) Projecting system with touch controllable projecting picture

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant