CN101673161A - Visual, operable and non-solid touch screen system - Google Patents

Abstract

The invention belongs to the technical field of human-computer interaction, and is a visual, operable and non-solid touch screen system which comprises a computer, two network cameras, a head-mounteddisplay and a calibration reference object, wherein the calibration reference object calibrates a virtual touch screen. A user can observe a real world containing the virtual touch screen, and directly operates the non-solid touch screen by hand through the head-mounted display. The invention uses two cameras for real-time image collection. The computer system carries out spatial calibration and the identification of the action of fingertip positions according to the image input by the cameras and synthesizes a corresponding image. The head-mounted display outputs the image containing the virtual touch screen to the user, and the user directly observes the feedback of the virtual touch screen through the head-mounted display. The invention can be used in the field of human-computer interaction, and is particularly suitable to be used as the equipment of human-computer interaction of a portable computer or a multimedia interaction experience system.

Description

A kind of visual, operable and non-solid touch screen system

Technical field

The invention belongs to human-computer interaction technique field, be specifically related to a kind of visual exercisable touch-screen system.

Background technology

In a lot of virtual reality systems, the user uses data glove, stylus, 3D mouse to replace mouse and keyboard to operate.They can adapt to sterically defined requirement in the virtual reality system, have expanded the form of operation.But they have common shortcoming: use not nature.When the user used these equipment, what have will wear heavy equipment, the data line of the restraint scope of will ining succession that has.

With the virtual reality the personalizing and be microminiaturization, carry-onization, the embeddingization of the computing machine of representative of computer system of representative with Hand Personal Computer, smart mobile phone, be two important development trends of current computer, and be that the GUI technology of representative is the bottleneck that influences their development with mouse and keyboard.So, separation instrumentation, end user's sensation and action are imported and more and more are subjected to people's attention.

On the other hand, in the man-machine interaction of virtual reality, people singly do not need the input of nature, also need the output of nature.People wish the virtual world observed just around them, and are not limited only in the screen.People could watch and obtain the information that needs from the angle of oneself liking in the mode of usual observation object like this.The fixing display that the researcher begins to discard tradition has proposed the method exported with projection and head mounted display, or uses the 3D hologram display device to show virtual object to people.

The CN1912816A patent announcement has been announced a kind of virtual touch screen system based on multi-cam.This system has 2 or above fixing camera to take finger moving and clicking operation on virtual touch screen, utilizes HSV color space background segment method to extract the hand profile, uses particle filter to follow the trail of and carries out the finger tip location.This invention can solve the problem of utilizing direct staff to import, but does not consider but how the user observes virtual output, only is to use common display screen to export, and does not realize virtual in the aspect of exporting.

Summary of the invention

The object of the present invention is to provide a kind of touch-screen system that does not have entity that the user is observed simultaneously and operate, this system combines based on the finger tip detection technique of computer vision and augmented reality technology, the finger location map in the augmented reality world, the virtual touch screen that the user is seen can be touched in correct position and operate, and can realize virtual in the output aspect.

Touch-screen system of the present invention comprises a computing machine, two IP Camera, and a head mounted display, a calibrated reference, this calibrated reference have been demarcated a virtual touch screen for operator's hand touch operation.Two IP Camera are connected with computing machine, and computing machine is connected with head mounted display again.Head mounted display is worn on user's (being the operator) at the moment, and two IP Camera are arranged at top about head mounted display respectively, are used to take user's hand and calibrated reference, and captured in real time obtains video image.Two IP Camera are taken direction place ray and are intersected, and become the angle of 30 to 90 degree on same surface level.This system does not have strict requirement to the angle angle.The first video captured frame of network shooting is sent to computing machine; Be provided with the software for calculation system in the computing machine, these software systems calculate the orientation at current camera place according to the image of calibrated reference in frame of video, thereby determine the virtual touch screen position; According to the image of staff in frame of video, calculate the position of staff, differentiate staff and whether contact with virtual touch screen; According to above information, it is synthetic that the software for calculation system carries out image again; Synthetic image is exported by head mounted display, and the user observes directly the feedback of virtual touch screen by head mounted display.

Various piece of the present invention further describes as follows:

1, virtual touch screen, and the user is to its operation

Virtual touch screen among the present invention is actually a rectangular area in the realistic space.This panel region uses an object of reference to demarcate.The operation of conventional touch screen such as the user can click in this panel region with finger (any finger that stretches out when clenching fist state), towing, multi-point touch.The present invention is not provided with any touch-control perception hardware device and display device in this zone, but carry out staff touch-control sensing based on vision by two IP Camera, the stack by touch-screen virtual image in the head mounted display and real world image shows output.In the view of other people, the user is at the sightless object of operation.

2, two cameras are used for gathering in real time video image

The present invention uses two IP Camera to take user's hand and calibrated reference simultaneously.

System mainly is to carry out the previous work that camera is caught image when initialization, comprises the data of reading in pick-up lens, sets the optical characteristics and the picture size of camera.In addition, the shape data of object of reference also can be read in system, for coupling use afterwards.

System catches the image of two cameras respectively in the approaching time, and waits until later step and handle.Simultaneously, the image that captures is done a backup, the background image that uses when synthesizing as the later stage.

The picture format that captures is RGBA form (R-is red, and G-is green, B-indigo plant, A-transparency).Why keeping A channel, is because will use A channel to carry out transparent judgement in the synthetic processing of image afterwards, and then the synthesis mode of decision virtual image and real world images.

System requires approaching as far as possible the opportunity of catching at left order camera and right order camera, be to judge whether this moment finger tip has the touching plane because will rely on the position of two observed finger tips of camera.If it is far away that capture time differs, the image that two cameras capture has not just had related each other.

3, computing machine is handled input picture

Core of the present invention is computer software module.This module is handled the image of camera collection, obtains necessary human-machine interactive information, and then carries out the synthetic of virtual touch screen image.This computer software module comprises the following module:

1) virtual touch screen locating module

This module is determined the locus of virtual touch screen with respect to camera according to the transformational relation between the coordinate system of the coordinate system of object of reference and camera.After virtual touch screen positioned, system can obtain the corresponding relation of the point on the image that point above the virtual touch screen and camera photograph, can obtain the virtual touch screen image and synthesize position and angle at the real world image.

ARToolKit is an augmented reality development library of increasing income that is proposed by people such as Hirokazu Kato.System uses ARToolKit to carry out the augmented reality correlation computations.This storehouse also can be replaced by other similar development libraries of increasing income.Use the built-in function arGetTransMat of ARToolkit,, can obtain the transformation matrix Tm that is tied to the camera coordinate system from the mark coordinate by the size and the angle of object of reference in image.The built-in function gluProject of coupled computer graphical development bag OpenGL can calculate the corresponding point of 4 angle points on the camera coordinate system of rectangle virtual touch screen.

Behind the position that has obtained the virtual touch screen plane, system has determined the position of virtual touch screen in images acquired.This independent intercepting in piece zone is come out to proceed to handle finger tip identification as sensitizing range (zone that finger tip location and motion detection block are concerned about), give up all the other zones simultaneously.

2) finger tip location and motion detection block

This module is at first carried out Face Detection, and the staff Region Segmentation is come out, and obtains the position of staff finger tip (as forefinger) in camera image with the morphological method of Computer Image Processing again.Then,, calculate finger position thereon, and use the motion tracking algorithm to add stiff stability according to the position of virtual touch screen.At last, use the projective geometry method, judge whether finger tip contacts with virtual touch screen.This module is divided into following several submodule again:

I) Face Detection submodule

Native system uses a kind of background segment method based on the Face Detection algorithm of propositions such as Wu Yueming.This algorithm is based on such statistics: when the color of staff projects on the Cb-Cr plane in the distribution on YCbCr (Y-luminance component, Cb-chroma blue component, and the Cr-red color) color space, can be similar to and accumulate in the elliptic region.Can judge that thus the color in this elliptic region scope is the colour of skin, obtain the zone at colour of skin place, i.e. the zone at staff place.

Before using this segmenting Background, need adjust according to specific camera.Finish in case adjust, these data just can be used always.

The R that below mentions, G, B and Y, Cb, Cr will be normalized in the 0-255 scope.

At first, need be to the YCbCr color space the RGBA color space transformation that collects picture.Conversion formula is as shown in Equation 1:

$\left[\begin{array}{c}Y\\ \mathrm{Cb}\\ \mathrm{Cr}\end{array}\right]=\left[\begin{array}{c}16\\ 128\\ 128\end{array}\right]+\left[\begin{array}{ccc}0.263& 0.5021& 0.0975\\ -0.1476& -0.29& 0.4375\\ 0.4375& -0.3663& -0.329\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right]$ (formula 1)

Formula 1 is the normalized form of RGBA color space transformation to the YCbCr color space.

Use concrete camera collection picture, staff zone is wherein extracted, its color is added up, draw its color distribution on the Cb-Cr plane.With the approximate color distribution scope of surrounding of an elliptic region, obtain this oval major semi-axis length a and minor semi-axis length b, the coordinate (x of central point _c, y _c), and major axis counter clockwise direction and x axle angulation

This ellipse is carried out standardization, and corresponding transformation for mula is suc as formula 2, shown in the formula 3:

[x ', y ', 1]=[x, y, 1] M (formula 3)

Wherein M is a transformation matrix, and (x y) is point in the former ellipse, and (x ', y ') is the point after the standardization.

When point (x ', y ') when standard ellipse is inner, see formula 4, think that the color that this point is represented is the colour of skin.

$\frac{{x^{\′}}^2}{a^2}+\frac{{y^{\′}}^2}{b^2}\≤1$ (formula 4)

After carrying out background segment, obtain all zones, i.e. the zone at staff place near the colour of skin.

Ii) finger tip position detection sub-module in camera image

In this module, system will obtain finger tip position in camera image.

At first, system uses connection district algorithm to obtain all connection districts to area of skin color, judges that hand region is the connection district of maximum in the picture.

After carrying out denoising to this connection district, extract the profile of finger, fill finger areas complete.

Then, determine fingertip location from it.Use a kind of Takao algorithm to orient the position of finger tip: to calculate the angle of inclination (direction that colour of skin point distributes) that hand is communicated with the district earlier, decide the bottom of finger then according to the close situation on hand and border, sensitizing range, then calculate from the finger bottom and find the solstics along the hand vergence direction.This point is fingertip location.

It should be noted that situation, support that two fingers touch simultaneously for multi-point touch.In identification, get maximum and time big two and be communicated with the district, and think that the finger tip on the left side, plane is the left hand finger tip as hand region, the right, plane be right hand finger tip.

Keep the backup of a hand images, so that the later stage uses when synthetic.

After finishing this section operation, can allow camera obtain the next frame image, employ one's time to the best advantage.

The iii) position calculation submodule of finger tip in virtual touch screen

Before, by the virtual touch screen locating module, 4 angle points of 4 angle points correspondence on the camera photographic images of virtual touch screen have been obtained.In this module, system finds out the position of finger tip on virtual touch screen by approximate projective transformation.

Its conversion requires the point above the irregular quadrilateral is mapped on the rectangle.

Here use the method for two-dimentional projective transformation.Suppose on the camera imaging plane a bit (X, Y) corresponding point on the virtual touch screen plane be (x, y), then (X, Y) can obtain by projective transformation (x, y).Here, the virtual touch screen zone of camera imaging plane is an irregular quadrilateral, and the touch screen zone on the virtual touch screen plane is a rectangle.

Below 8 yuan of 1 equation of n th order n group formula 5, band numbering (X, Y)---(X1, Y1) (X2, Y2) (X3, Y3) (X4 Y4) is 4 corresponding on camera photographic images angle points, (x, y)---(x1, y1) (x2, y2) (x3, y3) (x4 y4) is 4 angle points of virtual touch screen correspondence.

$\left[\begin{array}{cccccccc}X1& Y1& 1& 0& 0& 0& -x1\×Y1& -x1\\ X2& Y2& 1& 0& 0& 0& -x2\×Y2& -x2\\ X3& Y3& 1& 0& 0& 0& -x2\×Y3& -x3\\ X4& Y4& 1& 0& 0& 0& -x4\×Y4& -x4\\ 0& 0& 0& X1& Y1& 1& -y1\×Y1& -y1\\ 0& 0& 0& X2& Y2& 1& -y2\×Y2& -y2\\ 0& 0& 0& X3& Y3& 1& -y3\×Y3& -y3\\ 0& 0& 0& X4& Y4& 1& -y4\×Y4& -y4\end{array}\right]\left[\begin{array}{c}a1\\ a2\\ a3\\ b1\\ b2\\ b3\\ c2\\ c3\end{array}\right]=\left[\begin{array}{c}x1\×X1\\ x2\×X2\\ x3\×X3\\ x4\×X4\\ y1\×X1\\ y2\×X2\\ y3\×X3\\ y4\×X4\end{array}\right]$ (formula 5)

As long as 4 groups of corresponding point substitutions, just can utilize Gaussian processes to solve 8 parameter a1, a2, a3, b1, b2, b3, c2, c3.

Afterwards, obtain the corresponding point of point on rectangle on the irregular quadrilateral with formula 6 and formula 7.

$x=\frac{a_1\×X+a_2\×Y+a_3}{X+c_2\×Y+c_3}$ (formula 6)

$y=\frac{b_1\×X+b_2\×Y+b_3}{X+c_2\×Y+c_3}$ (formula 7)

Obtain fingertip location on the virtual touch screen plane (x, y) after, use the Condensation algorithm to come fingertip location is followed the trail of, determine probability by the normal distribution of measurement point and estimation point inverse distance.

Iv) finger tip and virtual touch screen contact detection submodule

This module is used to judge whether finger tip contacts with virtual touch screen.Its information that obtains can allow operations such as system judges whether finger tip is clicked, towing.

Utilize the method for projective geometry to judge whether finger tip contacts virtual screen plane.When finger tip does not drop on the virtual plane, to observe by imaging plane, its projection on virtual plane does not overlap.

When finger tip dropped on the virtual plane, it overlapped in the projection of virtual plane.

As long as set 2 of projections apart from threshold values, and think and represent that less than this threshold values the user touches the plane, just can know the user whether point click, drag motions.The big I of this threshold values is used in reality and is adjusted according to user's request.

Because the position of two cameras is not fixed, under the finger situation identical apart from the planimetric position, 2 distance of projection is not necessarily identical.Therefore, provide the threshold parameter of camera to adjust, in order to the degree of tightness of judgement of decision button to the user.

3) virtual touch screen video image synthesis module

This module is according to the positional information of the virtual touch screen that obtains previously and the position action message of finger tip, and synthetic corresponding virtual image is to provide suitable operational feedback.

Obtained position and the direction of camera in front with respect to calibrated reference.Use ARToolkit built-in function argConvGlpara to obtain the parameter of camera lens, compose the setting of carrying out camera position of giving OpenGL, just can obtain correct 3-D view.

According to the finger tip information that obtains before, can draw corresponding feedback image.

Virtual image and real world images are synthesized, and are steps very important in the augmented reality.Have only correct composograph, the user can think that just virtual object really is present in the real world.

Here, being divided into two parts carries out:

I) dummy object is superimposed upon on the background image

According to the touch control operation information that collects above, corresponding virtual touch screen feedback is drawn by system, is superimposed upon then on the real world images of camera shooting.

Ii) handle is superimposed upon the image the top

The user judges a whether real major criterion of virtual touch screen, is whether his hand and the position relation of touch-screen can correctly be represented.User's hand should be operated above virtual touch screen, so, be superimposed upon the user's who preserves before hand images the top of picture.

4, synthetic dummy object image and outputing on the head mounted display

Image after synthetic is outputed on the head mounted display, and the user just can observe virtual touch screen.

When system's continuous service, the user just can watch continuous images, when the frame number of camera is enough high, just with observe real world and be as good as.

Compared with prior art, beneficial effect of the present invention is embodied in:

1, adopted the background segment technology that is applicable to the augmented reality environment, in conjunction with finish the method that the location map of finger is arrived plane, the augmented reality world and its touch action of perception by two cameras, higher adaptability is provided, has realized the function that expensive multi-point touch touch-screen has with low-cost equipment simultaneously.

2, because the finger location map in the augmented reality world, the virtual touch screen that the user sees by wear-type can be touched in correct position, has saved the entity display, has increased portability of equipment.

3, the relative object of reference with touch location of the viewing location of virtual touch screen is fixed, and its physical location has unchangeability in the space, strengthened the sense of reality.

Description of drawings

Fig. 1 is the system architecture vertical view.

Fig. 2 is the overview flow chart of system.

Number in the figure 1 is a computing machine, and 2 is head mounted display, and 3 is IP Camera, and 4 is calibrated reference, and 5 is virtual touch screen.

Embodiment

The invention will be further described with enforcement below in conjunction with accompanying drawing:

Embodiment 1: being implemented in and operating the Windows program on the virtual touch screen is example.

Present embodiment has showed how the present invention substitutes common touch screen operation computing machine.

When the user used, two IP Camera wearing head mounted display and be fixed thereon had been put calibrated reference simultaneously.After the system on the operation computing machine, the user can see that a virtual touch-screen has appearred in the calibrated reference next door, has the interface of Windows program above by head mounted display.When the user clicked this virtual touch screen with forefinger, touch-screen can be made corresponding operation.For example, lift when the user presses icon with forefinger again, finish single-click operation one time, icon is selected; When the user presses icon, finger tip remains on the plane of virtual touch screen, moves, and icon just can be drawn to corresponding position by finger, lifts finger constipation bundle drag operation.

Realize that this example need carry out following configuration:

1, hardware device

This example use two LogitechS5500 IP Camera (Fig. 1-3) carry out the video input, use i-glasses head mounted display (Fig. 1-2) to export, use a computing machine (Fig. 1-1) that disposes Intel Core263001.86GHz*2 processor, 2GB internal memory, NVIDIA GeForce 7300GT display card to handle.

2, initialization

Before concrete the use, the program calib_camera2 of use ARToolkit obtains the deformation parameter of camera earlier.When initialization, directly call this deformation parameter.Read in the data of calibrated reference, use the patt.sample2 of ARToolkit in this example.Calibrated reference is 80 millimeters a square paperboard sheet.

3, use camera to gather video image in real time

Two cameras are apart from 15 centimetres, and the angle of taking between the direction is 40 degree.

4, colour of skin identification

When carrying out colour of skin identification, formula 2 is used following parameter:

x _c=110.0, y _c=120.5 (formulas 8)

When ellipse was carried out standardization, the transformation for mula of corresponding abbreviation was suc as formula 9, shown in the formula 10

C ' _b=c _b* 0.716-c _r* 0.699+ (x _c* 0.716+y _c* 0.699) (formula 9)

C ' _r=c _b* 0.699+c _r* 0.716+ (x _c* 0.699-y _c* 0.716) (formula 10)

(c wherein _b, c _r) be (x, concrete value y) in the formula 3, formula 4 above.

When point (cb ', cr ') is 23.478 at major semi-axis, minor semi-axis is 11.434 standard ellipse when inner, sees formula 11, thinks that the color that this point is represented is the colour of skin.

$\frac{{c_b}^{{\′}^2}}{{23.478}^2}+\frac{{c_r}^{{\′}^2}}{1{1.434}^2}\≤1$ (formula 11)

In addition, this example has also also been done restriction to area of skin color brightness, and the too low color of brightness is not thought the colour of skin.Adjust by test, this example sets a threshold values 30 for the Y value of YCbCr color space, does not think then that less than this value this color is the colour of skin.Can adjust voluntarily according to the camera situation in actual use.

5, finger tip location and contact detection

The step that the hand profile is handled all realizes with the built-in function of computer vision development library OpenCV.

When using the Condensation function tracking of OpenCV, population is 500, and each measurement is only once predicted.

Adjust according to test, that judges in this example whether finger tip click virtual touch screen is made as 10 pixels apart from threshold values.When distance threshold values during, think left mouse button pressing less than 10 pixels.When distance threshold values during, think left mouse button unclamping greater than 10 pixels.Mouse event by heavily loaded Windows system just can be mapped the operation of virtual touch screen and Windows mouse action, just can make user's direct control Windows program on virtual touch screen.

6, synthetic virtual touch screen video image

When synthetic virtual touch screen video image, the Windows windows content is intercepted, project on the virtual touch screen, the user just can directly see the operation feedback of Windows program on virtual touch screen.

7, output to head mounted display

Embodiment 2: carry out the multi-point touch photo and browse on virtual touch screen

This example is showed multi-point touch function of the present invention.

User's wearable device and start computer system after, he can see and show multiple pictures on the virtual touch screen.Towing about the user carries out with right hand forefinger comparison film center, photo can switch; When the user drags simultaneously the upper left side and the corner, lower right of photo with right-hand man's forefinger, photo can carry out convergent-divergent according to the position of two fingers.

Implement this example, carry out finger tip when identification, get maximum and time big two and be communicated with the district, and think that the finger tip on the left side, plane is the left hand finger tip as hand region, the right, plane be right hand finger tip.

To each photo, set photo centre one square area and be the towing zone, when about the user carries out this zone, dragging, switch a last photo respectively and next opens photo on virtual touch screen, on virtual touch screen, draw new photo; Setting the photo upper left corner and the lower right corner is the convergent-divergent control area, and when user's left index finger and right hand forefinger dragged the upper left corner and the lower right corner respectively, photo redrew on virtual touch screen simultaneously, and the user is fed back in real time.

Claims (2)

1. visual, operable and non-solid touch screen system, it is characterized in that comprising a computing machine, two IP Camera, a head mounted display, a calibrated reference, this calibrated reference have been demarcated a virtual touch screen for user's hand touch operation; Two IP Camera are connected with computing machine, and computing machine is connected with head mounted display again; Head mounted display is worn on the user at the moment, and two IP Camera are arranged at top about head mounted display respectively, are used to take user's hand and calibrated reference, and captured in real time obtains video image; Two IP Camera are taken direction place ray and are intersected, and become the angle of 30 to 90 degree on same surface level; The first video captured frame of network shooting is sent to computing machine; Be provided with the software for calculation system in the computing machine, these software systems are according to the image of calibrated reference in frame of video, calculate the orientation at current camera place, thereby determine the virtual touch screen position, and according to the image of staff in frame of video, calculate the position of staff, differentiate staff and whether contact with virtual touch screen; According to above information, it is synthetic that the software for calculation system carries out image again; Synthetic image is exported by head mounted display, and the user observes directly the feedback of virtual touch screen by head mounted display.

2. touch-screen system according to claim 1 is characterized in that described software for calculation system is made up of following software module:

1) virtual touch screen locating module

This module is determined the locus of virtual touch screen with respect to camera according to the transformational relation between the coordinate system of the coordinate system of object of reference and camera; After virtual touch screen positioned, system can obtain the corresponding relation of the point on the image that point above the virtual touch screen and camera photograph, comprise 4 angle points of virtual touch screen corresponding 4 points on the camera photographic images, obtain the virtual touch screen image and synthesize position and angle at the real world image;

2) finger tip location and motion detection block

This module is at first carried out Face Detection, and the staff Region Segmentation is come out, and obtains the position of staff finger tip in camera image with the morphological method of Computer Image Processing again; Then,, calculate finger position thereon, and use the motion tracking algorithm to add stiff stability according to the position of virtual touch screen; At last, use the projective geometry method, judge whether finger tip contacts with virtual touch screen; This module is divided into following several submodule again:

I) Face Detection submodule

Native system uses a kind of background segment method based on the Face Detection algorithm; This algorithm is based on such statistics: when the distribution of the color of staff on the YCbCr color space projects on the Cb-Cr plane, can approximate accumulate in the elliptic region, judge that thus the color in this elliptic region scope is the colour of skin, obtain the zone at colour of skin place, i.e. the zone at staff place;

Ii) finger tip position detection sub-module in camera image

In this module, system will obtain finger tip position in camera image;

At first, system uses connection district algorithm to obtain all connection districts to area of skin color, judges that hand region is the connection district of maximum in the picture;

After carrying out denoising to this connection district, extract the profile of finger, fill finger areas complete;

Then, determine fingertip location from it: the position of using a kind of improved Takao algorithm to orient finger tip: calculate the angle of inclination that hand is communicated with the district earlier, decide the bottom of finger then according to the close situation on hand and border, sensitizing range, then calculate from the finger bottom and find the solstics along the hand vergence direction; This point is fingertip location;

The iii) position calculation submodule of finger tip in virtual touch screen

In this module, system finds out the position of finger tip on virtual touch screen by approximate projective transformation;

Iv) finger tip and virtual touch screen contact detection submodule

This module is used to judge whether finger tip contacts with virtual touch screen, and its information that obtains can allow operations such as system judges whether finger tip is clicked, towing;

Utilize the method for projective geometry to judge whether finger tip contacts virtual screen plane, when finger tip does not drop on the virtual plane, observe by imaging plane, its projection on virtual plane does not overlap; When finger tip dropped on the virtual plane, it overlapped in the projection of virtual plane;

3) virtual touch screen video image synthesis module

This module is according to the positional information of the virtual touch screen that obtains previously and the position action message of finger tip, and synthetic corresponding virtual image is to provide suitable operational feedback; Specifically be divided into two parts:

I) dummy object is superimposed upon on the background image

According to the touch control operation information that collects above, corresponding virtual touch screen feedback is drawn by system, is superimposed upon then on the real world images of camera shooting;

Ii) handle is superimposed upon the image the top

Image after synthetic is outputed on the head mounted display, and the user just can observe virtual touch screen.