CN102541417A - Multi-object tracking method and system in virtual touch screen system - Google Patents
Multi-object tracking method and system in virtual touch screen system Download PDFInfo
- Publication number
- CN102541417A CN102541417A CN2010106158966A CN201010615896A CN102541417A CN 102541417 A CN102541417 A CN 102541417A CN 2010106158966 A CN2010106158966 A CN 2010106158966A CN 201010615896 A CN201010615896 A CN 201010615896A CN 102541417 A CN102541417 A CN 102541417A
- Authority
- CN
- China
- Prior art keywords
- patch
- point sequence
- new
- imported
- existing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Position Input By Displaying (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a multi-object tracking method and a multi-object tracking system in a virtual touch screen system, which comprises: projecting an image onto a projection surface; continuously acquiring the image of the environment of the projection surface; detecting the candidate patch of at least one object within a preset distance before the projection surface from the acquired frames of images; and classifying the patches into corresponding sequences of points according to the time and spatial relationships between the centroid points of the acquired patches in two adjacent frames of images.
Description
Technical field
The present invention relates to a kind of field of human-computer interaction, especially the design digital image processing field.Particularly, the present invention relates to follow the tracks of in a kind of virtual touch screen system a plurality of object method and system.
Background technology
Touchscreen technology is used for portable equipment (for example smart phone) and the PC (for example Desktop PC) as HMI equipment now more and more widely.Through touch screen, the user can operate this equipment more comfortable and easily and bring favorable experience.Although touchscreen technology is extremely successful in handheld device; But the touch screen for large-sized monitor but still exists some problems and chance to belong to Canesta; Which key assignments the Inc title selects be designated as the method for current key assignments for the U.S. Pat 7151530B2 of " System and Method for Determining an InputSelected By a User through a Virtual Interface (confirming the system and method for user-selected input through virtual interface) " has proposed a kind of being used at one group of key assignments, therefore be able to provide with virtual interface in the object that intersects of zone.This virtual interface can be implemented in the single key assignments of selection in the key assignments group and use depth transducer to confirm that position, this depth transducer can confirm the degree of depth of the position relevant with the position of depth transducer.In addition, the style characteristic of the placement property of object or object one of at least can be able to confirm.Positional information can be similar to the degree of depth of object with respect to position transducer or other RPs.When the existing of the pixel denoted object of sufficient amount in the pel array of camera, just think to detect this object.Confirm to compare with the shape of the object of the surface crosswise of virtual input region and with multiple known shape (for example finger or stylus).
Belong to Canesta equally; The Inc title comprises two optical system OS1 and OS2 for the U.S. Pat 6710770B2 of " Quasi-Three-Dimensional Method AndApparatus To Detect And Localize Interaction Of User-Obj ect And VirtualTransfer Device mutual standard three method for position and the equipment of consumer positioning-object and virtual conversion equipment (be used to detect and) " has disclosed a kind of virtual bench input or transmission information of adopting to the system of auxiliary equipment.In light structure embodiment, OS1 is on virtual bench and be parallel to the luminous energy on this virtual bench emission fan beam plane ().When user object penetrated interested beam plane, OS2 registered this incident.Triangulation method can be located virtual contact, and with the consumer premise information transmission to subsystem.In non-structure active light structure, OS1 is preferably a kind of digital camera, and its visual field has defined interested plane, and this plane is illuminated by an active luminous energy source.
The title that belongs to Apple company is a kind of for " identifying contacts on a touch surface (the lip-deep contact of recognizing touch operation) " U.S. Pat 7619618B2 has disclosed to be used for equipment and the method that, contact approaching at hand simultaneously followed the tracks of a plurality of fingers and palm contact point near the many touch-surfaces of sensing or when sliding above that.It is unprecedentedly integrated that the detection of hand structure and action and classification intuitively realized that the key entry in multi-usage ergonomics computer output device, static, indication, rolling, 3D handle.
The title that the belongs to Matsushita Electric company system and method that is used for many touch sensitive surface that a kind of two independent array through using orthogonal linear tolerance limit sensor can the detection and tracking multiple touch points for the U.S. Patent application US20100073318A1 of " Multi-touch surface providingdetection and tracking of multiple touch points (many touch-surfaces of multiple touch points detection and tracking are provided) " has disclosed.
It seems that from top these prior aries of mentioning most of large scale touch screen all are based on magnetic board (such as electronic whiteboard), IR border (such as interactive large-sized monitor) etc.For the current technical scheme of large scale touch screen, still had a lot of problems at that time, for example: in general, the equipment of these types is big and heavy owing to its volume that its hardware causes usually, therefore is difficult to carry, and does not have portability.And the device screen size of these types receives restriction and the fixed size of hardware and can not freely regulate according to the needs of environment, but also needs a kind of special time writer or a kind of IR pen to operate.
For some virtual whiteboard projector, the user must control the on/off switch of laser pen, and this is very loaded down with trivial details, therefore has the unmanageable problem of laser pen.In addition, in this virtual whiteboard projector,, just be difficult to accurately laser point navigated to next position, therefore have the difficult problem in laser point location in case laser pen is closed.Adopt finger mouse to substitute laser pen in some virtual whiteboard projector, still, adopted the virtual whiteboard projector of finger mouse can not detect touch beginning (touch on) or touch end (touch up).
Summary of the invention
For these problems of the prior art of being mentioned above solving, the present invention proposes and follow the tracks of a plurality of object method and system in a kind of virtual touch screen system.Particularly, following the tracks of a plurality of object method in this virtual touch screen system comprises: with image projection to in the projection surface; Obtain the image of the environment of said projection surface continuously; From the every two field picture that is obtained, detect the alternative patch of at least one object that is positioned at the preceding preset distance of said projection surface; And each patch is included into corresponding point sequence according to the relation of centroid point on time and space of the patch that obtains in adjacent two two field pictures in front and back.
According to method of in virtual touch screen system, following the tracks of a plurality of objects of the present invention, the said relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back comprises the step that each patch is included into corresponding point sequence: receive the centroid point that from a frame new images, obtains a plurality of new patches; Retrieve a plurality of existing point sequences, to each existing point sequence, the searching distance should have the centroid point that point sequence closes on, and the pairing new patch of nearest centroid point is had the nearest new patch of sequence of points as being somebody's turn to do; And the new patch nearest apart from this existing point sequence be included into this existing point sequence.
According to method of in virtual touch screen system, following the tracks of a plurality of objects of the present invention; The said relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back also comprises the step that each patch is included into corresponding point sequence: to any one existing point sequence; If there is not any new patch that closes on it, then provide notice and this existing point sequence of deletion that this existing point sequence has finished.
According to method of in virtual touch screen system, following the tracks of a plurality of objects of the present invention; The said relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back also comprises the step that each patch is included into corresponding point sequence: be directed against any new patch in a plurality of new patch in the two field picture of being imported; If there is not any existing point sequence that closes on it; Then providing this new patch is the notice of the starting point of a new point sequence, and creates a new point sequence.
According to method of in virtual touch screen system, following the tracks of a plurality of objects of the present invention; Said step of being somebody's turn to do the nearest centroid point of existing point sequence to each existing point sequence searching distance comprises: import an existing point sequence, and seek in a plurality of new patch from a two field picture of being imported apart from having the new patch that point sequence closes on; Do not find under the situation of the new patch that the existing point sequence imported of distance closes at a plurality of new patch from a two field picture of being imported, then provide the notice that the existing point sequence imported will be deleted;
The new patch that the existing point sequence that has found distance to be imported at a plurality of new patch from a two field picture of being imported closes on; And the new patch that closes on that is found only is the new patch that closes on of the existing point sequence imported, confirms that then this new patch belongs to the existing point sequence of being imported; And the new patch that closes on of the existing point sequence that has found distance to be imported at a plurality of new patch from a two field picture of being imported; And the new patch that closes on that is found still is the new patch that closes on of other existing point sequences; If the distance of the new patch that closes on and the existing point sequence of being imported that are found is than little with the distance of other existing point sequences; Confirm that then this new patch belongs to and the nearest new patch of being imported of existing point sequence; Otherwise, provide the notice that the existing point sequence imported will be deleted; Examine and whether all existing sequence of points have been carried out above-mentioned steps.
According to method of in virtual touch screen system, following the tracks of a plurality of objects of the present invention; The step of seeking the new patch that distance should existing point sequence closes in the said a plurality of new patch from a two field picture of being imported comprises: import a new patch, calculate new patch of being imported and the distance of being imported between the point sequence that has; Calculated import new patch and the existing point sequence imported between the situation of distance less than a predetermined distance threshold Td under; The new patch of import is inserted into the candidate who belongs to the existing track of being imported to be closed in the new patch tabulation; Otherwise, confirm whether all new patches have been carried out above-mentioned steps; Confirming that the candidate who belongs to the existing point sequence of being imported closes under the situation of size less than a predetermined size threshold value Tsize of new patch tabulation; Confirm whether all new patches have been carried out above-mentioned steps; Otherwise; The candidate that deletion belongs to the existing point sequence of being imported close in the new patch tabulation with the existing point sequence of being imported new patch farthest after, confirm whether all new patches have been carried out above-mentioned steps; Confirming that new patch has been carried out under the situation of above-mentioned steps to all; Close on the tabulation of new patch for empty if belong to the candidate of the existing point sequence of being imported, then the candidate from said tabulation closes on the recently newly patch of the nearest new patch of the existing point sequence finding out in the new patch and imported as the existing point sequence of being imported.
According to method of in virtual touch screen system, following the tracks of a plurality of objects of the present invention, also comprise: adopt following formula to carry out coordinate optimization carrying out smoothing processing to the point sequence of final acquisition,
Wherein
is the point in the point sequence; K is the iteration sign; N is the point sequence sign, and m is the radix of iteration point.
According to another aspect of the present invention, a kind of system that in virtual touch screen system, follows the tracks of a plurality of objects is provided, has comprised: projector, with image projection to in the projection surface; Depth cameras obtains the image of the environment of said projection surface continuously; Subject detecting unit from the every two field picture that is obtained, detects candidate's patch of at least one object that is positioned at the preceding preset distance of said projection surface; Tracking cell is included into corresponding point sequence according to the relation of centroid point on time and space of the patch that obtains in adjacent two two field pictures in front and back with each patch.
According to the system that in virtual touch screen system, follows the tracks of a plurality of objects of the present invention, said tracking cell comprises: receiver module receives the centroid point that from a frame new images, obtains a plurality of new patches; Patch is sought module recently; Retrieve a plurality of existing point sequences; To each existing point sequence; The searching distance should have the centroid point that point sequence closes on, and the pairing new patch of nearest centroid point is had the nearest new patch of sequence of points as being somebody's turn to do, and the new patch nearest apart from this existing point sequence is included into this existing point sequence.
According to the system that in virtual touch screen system, follows the tracks of a plurality of objects of the present invention, also comprise the smoothing processing unit, adopt following formula to carry out coordinate optimization carrying out smoothing processing to the point sequence of final acquisition,
Wherein
is the point in the point sequence; K is the iteration sign; N is the point sequence sign, and m is the radix of iteration point.
Description of drawings
Shown in Figure 1 is according to the synoptic diagram of the framework of virtual touch screen of the present invention system.
Shown in Figure 2 is the overview flow chart of performed object detection of control module according to the present invention and object tracking processing.
Shown in Fig. 3 (a)-(c) is the synoptic diagram of from current depth map, disposing background depth map
Shown in Figure 4 is carries out the synoptic diagram of binary conversion treatment with the patch that obtains candidate target to the depth map of the current scene imported.
Shown in Fig. 5 (a) is the synoptic diagram that is used for patch is added the connected domain of numbering
Shown in Fig. 5 (b) is the synoptic diagram of the bianry image of the patch that has the connected domain numbering that generates according to depth map.
Shown in Fig. 6 (a)-(d) is the synoptic diagram of enhancing process of the bianry image of patch.
Shown in Figure 7 is detects the synoptic diagram of the process of the coordinate of the centroid point of patch in the binary image of the patch shown in Fig. 6 (d).
The synoptic diagram of the track that shown in Figure 8 is user's finger or stylus moves on the screen of virtual touch screen.
The process flow diagram that is used to follow the tracks of the object that is detected shown in Figure 9.
Shown in Figure 10 is to seek the process flow diagram of the recently new spot of every existing track according to the present invention is directed to all existing tracks.
Shown in Figure 11 is to be directed against the existing track of being imported to seek the process flow diagram apart from its nearest new patch.
The point sequence of shown in Figure 12 the is a kind of motion track of detected object on virtual touch screen that a strip adoption the present invention is obtained carries out the method for smoothing processing.
Shown in Figure 13 (a) is the synoptic diagram of the motion track of detected object on virtual touch screen that obtain of a kind of the present invention of employing.
Shown in Figure 13 (b) is the synoptic diagram through the object motion track after the smoothing processing.
Shown in Figure 14 is the synoptic diagram of the concrete configuration of control module.
Embodiment
Below, will specify specific embodiment of the present invention with reference to accompanying drawing.
Shown in Figure 1 is according to the synoptic diagram of the framework of virtual touch screen of the present invention system.As shown in Figure 1, comprise projector equipment 1, optical device 2, control module 3 and projection surface (following also can be referred to as projection screen or virtual screen) 4 according to virtual touch screen of the present invention system.In embodiment of the present invention, projector equipment can be a projector, and it is used for the needs images displayed is projected in the projection surface 4 as a kind of virtual screen, so that the user is at the enterprising line operate of this virtual screen.Optical device 2 for example can be any equipment that can obtain image, depth cameras for example, and it obtains the depth information of the environment of projection surface 4, and generates depth map based on this depth information.Control module 3 detects at least one object in the said surperficial preset distance of distance and follows the tracks of detected object along the direction away from said surface, to generate level and smooth point sequence.Interactive command or the like is for example painted, made up to said point sequence to be used for further interactive task on virtual screen.
Shown in Figure 14 is the synoptic diagram of the concrete configuration of control module.Said control module 3 comprises depth map processing unit 31, subject detecting unit 32, image enhancing unit 33 and coordinate Calculation and converter unit 34 and tracking cell 35 and smooth unit 36 usually.Depth map processing unit 31, and is numbered the connected domain on this depth map as input and handle this depth map so that background is disposed from this depth map with its depth map of being caught that depth cameras was sent subsequently.Subject detecting unit 32 is based on the depth information from the said depth map of said depth map processing unit 31; Based on a predetermined depth threshold; Depth map is carried out binary conversion treatment, form a plurality of patches, subsequently as candidate target; Based on the size of relation between each patch and the connected domain and plaque area, confirm patch as object.Coordinate Calculation and converter unit 34 calculate the centroid point coordinate of the patch of the object of confirming as, and target coordinate system, the i.e. coordinate system of virtual interacting screen are arrived in its centroid point coordinate transform.Follow the tracks of with smooth unit 35 and follow the tracks of detected a plurality of patches in the multiple image of shooting continuously, generating the coordinate point sequence after a plurality of centroid point conversion, and subsequently the coordinate point sequence that is generated is carried out smoothing processing.
Control module 3 performed object detection that shown in Figure 2 is are handled and to the overview flow chart of image tracing.As shown in Figure 2.At step S21 place; Depth map processing unit 31 receives the depth map that depth cameras 2 is obtained; This depth map obtains through following mode, and promptly depth cameras 2 is taken the current environment image, and when taking, measures the distance of each pixel apart from depth cameras; Constitute with 16 (can be 8 or 32 according to actual needs) depth informations that value was write down, 16 bit depth values of these each pixels have constituted described depth map.For follow-up treatment step, can be before obtaining the depth map of current scene, obtain the background depth map that has no detected object before the projection screen in advance.At step S22 place, depth map processing unit 31 is handled received depth map so that background is disposed from this depth map, and only keeps the depth information of foreground object, and the connected domain in the depth map that is kept is numbered subsequently.Shown in Fig. 3 (a)-(c) is the synoptic diagram of from current depth map, disposing background depth map.The depth map that this employing 16 bit value of institute's diagrammatic sketch show is the convenience in order to explain, is not in the process of embodiment of the present invention, must demonstrate.Shown in Fig. 3 (a) is a kind of synoptic diagram of instance of depth map of background, only is background depth map in the depth map shown in wherein, i.e. the depth map of projection surface, and it does not comprise the depth image of any foreground object (being object).A kind of mode that obtains the depth map of background is; In the starting stage of the method for virtual touch screen system implementation of the present invention virtual touch screen of the present invention function; The snapshots in time of at first obtaining the depth map of current scene and preserving this depth map, thereby the depth map of acquisition background by optical device 2.When obtaining the depth map of this background, in current scene, the place ahead of projection surface 4 (between optical device 2 and the projection surface 4) can not have any dynamic object that is used to touch projection surface.The another kind of mode that obtains the depth map of background is not to use the transient state photo and is to use a series of continuous transient state photos to generate a kind of average background depth map.
Shown in Fig. 3 (b) is the instance of the depth map of being caught of a frame current scene, wherein has an object (for example user's hand or stylus) to touch in projection surface.
Shown in Fig. 3 (c) is the wherein instance of the depth map of background after being eliminated of a frame.A kind of possible mode of removing background depth is to use the depth map of the depth map subtracting background of current scene, and another kind of mode is to scan the depth map of current scene and the every bit and the corresponding point in the background depth map of this depth map are carried out depth value relatively.If the absolute value of the degree of depth difference of the pixel that these are paired is similar and within a predetermined threshold value; Then that the absolute value of the degree of depth difference in the current scene is similar respective point is disposed from the depth map of current scene, does not carry out any change otherwise reservation is put accordingly.Subsequently the connected domain of removing in the background depth map current depth map is afterwards numbered.Connected domain of the present invention refers to such zone: supposing that depth cameras is captured has two 3D points; If their the XY plane (shot picture) that is projected in is gone up adjacent one another are and difference its depth value is not more than given threshold value D, then claim their each other D-be communicated with.If have the D-communication path between any 2 in one group of 3D point, then claim this group 3D point D-connection.If for each the some P in the 3D point that one group of D-is communicated with, each some P does not exist adjacent point under the situation of not interrupting this connection condition, to add in said group on the XY plane, claim that then the 3D point that this group D-is communicated with is communicated with for maximum D-.Connected domain of the present invention is that one group of D-connectivity points and its in the depth map is that maximum D-is communicated with.The connected domain of said depth map is corresponding with continuous blocks (mass) zone that said depth cameras is caught, and connected domain is that the D-connected set of points in the said depth map merges and its maximum D is associated.Therefore, in fact connected domain is numbered is exactly to having 3D point that above-mentioned D-the is communicated with identical numbering of annotating, that is to say, for the pixel that belongs to same connected domain add pour down with numbering, therefore, the numbering matrix of generation connected domain.The continuous block (mass) that the connected domain of said depth map is caught corresponding to said depth cameras.
The numbering matrix of said connected domain is whether a kind of which point in can the said depth map of mark is the data structure of connected domain.Each element in said numbering matrix is corresponding to a point in the depth map, and the value of this element is exactly the numbering (numbering of a connected domain) of the connected domain under this point.
Then, at step S23 place, each point in the depth map is carried out binary conversion treatment, thereby generate some patches, and the pixel of the patch that belongs to same connected domain is added the connected domain numbering as alternative objects based on a depth conditions.To describe concrete binary conversion treatment process below in detail.Shown in Figure 4 is carries out the synoptic diagram of binary conversion treatment with the patch that obtains alternative objects to the depth map of the current scene imported.。As shown in Figure 4, the relative depth information that the present invention is based between the corresponding pixel points of each pixel and background depth map in the depth map in the current scene is carried out the binary conversion treatment process.In an embodiment of the present invention, the depth value of a pixel of retrieval, i.e. distance between the object-point of depth cameras and the pixel representative retrieved from the depth map of said current scene.In Fig. 4; To travel through the mode of all pixels; The depth d of a pixel of retrieval from the depth map of the current scene imported; The depth value b of the pixel that the retrieval from background depth map afterwards is corresponding with the pixel of being retrieved from the depth map of current scene calculates difference (subtraction value) s, the i.e. s=b-d of degree of depth b of depth d and the background pixel point of two target points then.If the difference that is obtained is greater than zero and less than a predetermined distance threshold t, i.e. 0＜s＜t, then the gray-scale value of the pixel that is retrieved in the current scene is set to 255, otherwise then is made as 0.Certainly, this binaryzation also can be directly with two kinds of these positions 0 or 1 of situation difference, as long as can these the two kinds binaryzation modes that make a distinction can be adopted.Through above-mentioned binaryzation mode, can obtain the patch shown in Fig. 5 (b) with a plurality of alternative objects.The precision that the controlled amount system of threshold value t detects object.The threshold value t also hardware specification with depth cameras is relevant.The value of threshold value t is generally the thickness size of a finger or the diameter of common stylus, for example 0.2-1.5 centimetre, is preferably 0.3 centimetre, 0.4 centimetre, 0.7 centimetre, 1.0 centimetres.Threshold value t can adjust according to the environment that wherein uses virtual touch screen system; For distance threshold t, the setting of concrete size can be according to the present invention in application process the concrete size of the diameter of the people's before the screen finger thickness or employed stylus carry out corresponding setting.Usually the possible values of this threshold value t is 1 centimetre.
Shown in Fig. 5 (a) is the synoptic diagram that is used for patch is added the connected domain of numbering.Behind the bianry image that obtains patch, scan search contains the pixel of connected domain numbering, and adds this connected domain numbering to pixel corresponding in the binaryzation patch image, thereby makes some patch have the connected domain numbering, shown in Fig. 5 (b).Patch in the said bianry image (white portion or point) is the candidate that touches the possible destination object in projection surface.According to noted earlier, the binaryzation patch that has connected domain numbering among Fig. 5 (b) possess following two conditions: 1. patch belongs to connected domain.2. difference (subtraction value) s of the depth d of corresponding each pixel of patch and background depth b must be less than threshold value t, i.e. s=b-d＜t.
Then,, the binaryzation patch image of acquisition depth map is carried out enhancement process, to reduce noise unnecessary in the binaryzation patch image and to make the shape of patch become clearer and stable at step S24 place.This step has image enhancing unit 33 to carry out.Particularly, through following steps carry out said enhancement process.
At first, remove the patch that does not belong to connected domain, just with the patch that does not add connected domain numbering at step S23 place directly with its gray-scale value from the highest vanishing, for example with its gray values of pixel points from 255 vanishing.In another kind of mode, become 0 with 1.Thereby obtain the patch binary image shown in Fig. 6 (a).
Secondly, remove and belong to the patch of its area S less than the connected domain of an area threshold value Ts.In an embodiment of the present invention, patch belong to a certain connected domain mean this patch at least one the point in connected domain.If the area S of the connected domain under this patch is less than an area threshold value Ts, then corresponding patch then is regarded as noise and from the bianry image of patch, is removed.Otherwise patch then is considered to the candidate of destination object.Area threshold Ts can regulate according to the employed environment of virtual touch screen system.Area threshold Ts is generally 200 pixels.Thus, obtain the binary image of the patch shown in Fig. 6 (b).
Then, exactly the patch in the binary image of the patch shown in Fig. 6 (b) that obtained is carried out some morphology (morphology) operation.In this embodiment, adopted expansion (dilation) to operate and close (close) operation.At first carrying out an expansive working is to carry out closed operation iteratively then.The iterations that carries out closed operation is a predetermined value, and this predetermined value can be regulated according to the employed environment of virtual touch screen system.This iterations for example can be set to 6 times.Finally obtain the binary image of the patch shown in Fig. 6 (c).
At last, if there are a plurality of patches belong to same connected domain, promptly these patches have identical connected domain numbering, then keep to have a patch of maximum area in the patch with identical connected domain numbering and remove other patches.In an embodiment of the present invention, a connected domain can include a plurality of patches.In these patches, the patch that only has maximum area just is considered to destination object, and other patch then is the noise that need be removed.Finally obtain the binary image of the patch shown in Fig. 6 (d).
At step S25 place, detect the profile of the patch that obtains, the coordinate of the centroid point of calculating patch also arrives coordinates of targets with the centroid point coordinate transform.This detection, calculating and map function are carried out by coordinate Calculation and converter unit 34.Shown in Figure 7 is detects the synoptic diagram of the process of the coordinate of the centroid point of patch in the binary image of the patch shown in Fig. 6 (d).Referring to Fig. 7, calculate the coordinate of the centre of form of patch according to the geological information of patch.This computation process comprises: the profile of detection of plaque, calculate the Hu square of this profile and use said Hu square to calculate the coordinate of centroid point.In an embodiment of the present invention, can use multiple known mode to come the profile of detection of plaque.Also can use known algorithm to calculate the Hu square.After the Hu square that obtains said profile, can calculate the coordinate of centroid point through following formula:
(x
0，y
0)＝(m
10/m
00，m
01/m
00)
(x wherein
0, y
0) be the coordinate of centroid point, and m
10, m
01, m
00It is exactly the Hu square.
Coordinate transform is exactly with the coordinate of the centroid point coordinate system from the coordinate system transformation of the bianry image of patch to user interface.The conversion of coordinate system can be adopted known method.
In order to obtain the continuous motion track of touch point; Can be through the touch point in the successive frame depth map captured in the continuous detecting virtual touch screen system of the present invention; Thereby follow the tracks of detected a plurality of patch to produce the sequence of a plurality of points, obtain the movement locus of touch point thus.
Particularly; Be exactly at step S26 place; Centroid point coordinate on user interface to obtaining the patch of every two field picture behind every frame depth map execution in step S21-S25 of continuous shooting is followed the tracks of, and generates centroid point sequence (being track), and acquisition centroid point sequence is carried out smoothing processing.This tracking and smooth operation are undertaken by tracking cell 35.
The synoptic diagram of the track that shown in Figure 8 is user's finger or stylus moves on the screen of virtual touch screen.The movement locus that has wherein shown two objects (finger).This only is an instance.In other cases, can a plurality of objects, for example 3,4,5 objects are decided according to the actual requirements.
The process flow diagram that is used to follow the tracks of the object that is detected shown in Figure 9.Carry out trace flow as shown in Figure 9 repeatedly, finally can obtain the preceding motion of objects track arbitrarily of screen.Particularly, carrying out tracking operation is exactly the centroid point coordinate of the patch in the new detected depth map in user interface to be included into before obtain in any track.
According to the centroid point coordinate of a plurality of institutes detection of plaque in user interface, follow the tracks of a plurality of new detected patches, thereby produce many tracks and trigger relevant touch event about these tracks.To follow the tracks of patch, just need classify and the centroid point coordinate of patch is placed in one have a few on time and space in the relevant point sequence patch.Have only the point in the identical sequence just can merge into a track.As shown in Figure 8, if virtual touch screen system is supported painting function, then the point in the sequence shown in Fig. 8 is just represented the paint command on projection screen, and the point in same sequence just can couple together and form a curve as shown in Figure 8 so.
In the present invention, can follow the tracks of three kinds of touch events: touch beginning, touch to move and touch and finish.Touch object touch projection screen that beginning just is meant that institute will detect and track is begun.Touch to move and to refer to the detected object of wanting and just touching on the projection screen and during track just continuing in projection surface.And touch to finish to refer to surface and the motion track end that object that institute will detect leaves projection screen.
As shown in Figure 9, at step S91, receive based on the centroid point coordinate of new patch in user interface of a frame depth map according to the detected object of step S21-S25, this is the output of coordinate Calculation and converter unit 34.
Subsequently, at step S92 place, to before the patch of each frame depth map carried out tracking processing after each point sequence in all point sequences that obtain (i.e. all existing tracks, below all become existing track), this has the nearest new patch of track computed range.All tracks that touch the object on touch screen (being projection screen) all are retained in the virtual touch screen system.Each track keeps the patch that quilt is followed the tracks of, and this patch of being followed the tracks of is the last patch that is endowed this track.The distance of new patch of the present invention and existing track refers to the distance between the last patch in a new patch and the existing track.
Then,, new patch is included into apart from its nearest existing track, and triggers the touch moving event at step S93 place.
Then, at step S94 place, if to an existing track; Do not exist any new patch approaching with it, in other words, if all new patches have been included into other existing tracks respectively; Then delete the existing track of this bar, and trigger touch End Event to this existing track.
At last, at step S95 place, if to each new patch; Do not exist any existing track approaching with it; In other words, obtain all existing tracks before and all deleted owing to trigger the touch End Event, the distance of perhaps new patch and all existing tracks is not within certain distance threshold scope; Then confirm the starting point that this new patch is new track, and trigger the beginning incident that touches.
Carry out above-mentioned steps S91-S95 repeatedly, realize tracking, thereby will belong to the track that consists of a little of same point sequence centroid point coordinate in user interface of the patch in the successive frame depth map.
When having many existing tracks, will be to every existing track execution in step S92 repeatedly.The particular flow sheet of tracking cell of the present invention 35 execution in step S92 shown in Figure 10.
At first, at step S101 place, examine and traveled through all existing tracks.This can just can realize through a simple counter.If all carried out step S92, then end step S92 to all existing tracks.If no, then advance to step S102.
At step S102 place, import the existing track of next bar.At step S103 place,, seek subsequently apart from its new patch that closes on to the existing track of being imported.Get into step S104 then.
At step S104 place, determine whether to be directed against the existing track of being imported and found the new patch that closes on apart from it.If the new patch that the existing track that has found distance to be imported closes on then advances to step S105, otherwise, step S108 then got into.
At step S108 place, there is not the new patch that closes on owing to be directed against the existing track of being imported, therefore, be " the existing track that will delete " with this existing Trajectories Toggle of importing.Turn back to step S101 subsequently.Thus, will trigger the touch End Event to this " the existing track that will delete " at step S94 place.
At step S105 place, whether the new patch that the existing track of confirming apart from imported closes on also is the new patch that closes on apart from other existing tracks.In other words, confirm this new patch whether belong to simultaneously apart from two or more existing track close on new patch.If judge this new patch belong to simultaneously two or more existing track close on new patch, then handle getting into step S106, otherwise, handle getting into step S109.
At step S109 place, because this new patch only is the new patch that closes on of the existing track imported, therefore, this new patch is included into the existing track imported as its nearest new patch, promptly become one of point in the point sequence of this existing track.Handle afterwards and turn back to step S102.
At step S106 place and since this new patch belong to simultaneously two or more existing track close on new patch, then calculate this new patch and under the distance of every existing track in many existing tracks.Then at step S107 place; Compare size in step S106 place calculated distance; And confirm between this new patch and the existing track of being imported distance whether in institute's calculated distance for minimum, promptly confirm this new patch and the existing track imported between apart from whether than and other existing tracks between distance all little.If distance for minimum, then handles getting into step S109 between the existing track of confirming this new patch and being imported in the calculated distance of step S106 place, otherwise, handle getting into step S108.
Carry out above-mentioned steps S101-109 repeatedly, thus the processing that performing step S92 is carried out, thus can travel through the patch of the new detection of all existing tracks and input.
Shown in Figure 11 is to be directed against the existing track of being imported to seek the process flow diagram apart from its new patch that closes on.Shown in figure 11, at step S111 place, examine the distance of closing on that whether is directed against between the existing track that all new patches of being imported have all calculated and imported.If close on distance between the existing track that has all calculated and imported to all new patches, then handle getting into step S118, otherwise, handle getting into step S112.
At step S118 place, confirm whether the tabulation as the new patch that closes on of the existing track of being imported is empty.If empty, end process then, otherwise, get into step S119.At step S119 place, in all the new patches that close on tabulations, find the nearest new patch with this existing track of importing, and will this nearest new patch be included into the point sequence of the existing track of being imported.End step S103 afterwards.
At step S112 place, the next new patch of input.Subsequently, at step S113 place, calculate next new patch and the existing track imported between distance.Then, at step S114 place, confirm that whether distance between the next new patch calculated and the existing track of being imported is less than a predetermined threshold value.Less than a predetermined distance threshold Td, then handle entering step S115 if confirm distance between the next new patch calculated and the existing track of being imported, otherwise, turn back to step S111.Distance threshold Td is set to the distance of 10-20 pixel usually here, is preferably the distance of 15 pixels.This threshold value Td adjusts according to the employed environment of virtual touch screen system.In the present invention, if the distance between a new patch and the existing track less than described distance threshold Td, then is referred to as this new patch and should closes on by existing track.
At step S115 place, the new patch of the said next one is inserted in the new patch tabulation of the candidate who belongs to the existing track of being imported.Subsequently at step S116 place, whether the size of the new patch tabulation of candidate of confirming to belong to the existing track of being imported is less than a predetermined size threshold value Tsize.If the size of confirming to belong to the new patch tabulation of the candidate of the existing track of being imported is then handled and is turned back to step S111 less than a predetermined size threshold value Tsize, otherwise, handle getting into step S117.At step S117 place, the longest new patch of candidate of distance that will belong between the existing track in the new patch tabulation of the candidate of the existing track of being imported and that imported is deleted from said tabulation, turns back to step S111 afterwards.
Carry out the step shown in Figure 11 repeatedly, thus completing steps S103.
The above flow process of following the tracks of patch coordinate in user interface to successive image frame of having described with reference to accompanying drawing 9-11.Through above-mentioned tracking operation, the touch that triggers institute's detected object begins incident, touches moving event or touches End Event.Thus, obtain the motion track of institute's detected object on virtual touch screen at last.Shown in Figure 13 (a) is the synoptic diagram of the motion track of detected object on virtual touch screen that obtain of a kind of the present invention of employing.
Obviously, this motion track of detected object on virtual touch screen shown in Figure 13 (a) that tentatively obtains seems comparatively mixed and disorderly.This track also need carry out smoothing processing, so that obtain level and smooth object motion track.Shown in Figure 13 (b) is the synoptic diagram through the object motion track after the smoothing processing.The point sequence of shown in Figure 12 the is a kind of motion track of detected object on virtual touch screen that a strip adoption the present invention is obtained carries out the method for smoothing processing.
The point sequence smoothing processing is exactly that coordinate to the point in this sequence is optimized so that point sequence is level and smooth.Shown in figure 12; Input constitutes the first round input of the original point sequence
(n is a positive integer) of a track as iteration, the i.e. output of patch tracking.In accompanying drawing 12, original point sequence
is first row that sit up.Formula below using then is according to the sequence of calculating the next round iteration from the result of last round of iteration
Wherein
is the point in the point sequence; K is the iteration sign; N is the point sequence sign, and m is the radix of iteration point.
Repeat this iterative computation, up to reaching predetermined iteration threshold.In an embodiment of the present invention, the m parameter can be 3-7, is set to 3 in an embodiment of the present invention, and this means this, and each next stage point has 3 some iteration of upper level to obtain.This iteration threshold is 3.
Through above-mentioned iterative computation, finally obtaining is through the object motion track after the smoothing processing shown in Figure 13 (b).
Here, in this manual, the processing of being carried out by computing machine according to program not needs according to carrying out with time series like the order of flowchart text.That is the processing of, being carried out by computing machine according to program comprises processing (for example parallel processing and target processing) parallel or that carry out separately.
Likewise, program can go up at a computing machine (processor) and carry out, and perhaps can be carried out by many computer distribution types.In addition, program can be transferred to the remote computer at the there executive routine.
Will be understood by those skilled in the art that,,, various modifications, combination, part combination and alternative can occur as long as it falls in the scope of accompanying claims or its equivalent according to designing requirement and other factors.
Claims (10)
1. method of in virtual touch screen system, following the tracks of a plurality of objects comprises:
With image projection to in the projection surface;
Obtain the image of the environment of said projection surface continuously;
From the every two field picture that is obtained, detect candidate's patch of at least one object that is positioned at the preceding preset distance of said projection surface;
The relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back is included into corresponding point sequence with each patch.
2. method of in virtual touch screen system, following the tracks of a plurality of objects according to claim 1, the said relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back comprises the step that each patch is included into corresponding point sequence:
Reception obtains the centroid point of a plurality of new patches from a frame new images;
Retrieve a plurality of existing point sequences, to each existing point sequence, the searching distance should have the centroid point that point sequence closes on, and the pairing new patch of nearest centroid point is had the nearest new patch of sequence of points as being somebody's turn to do;
The new patch nearest apart from this existing point sequence is included into this existing point sequence.
3. method of in virtual touch screen system, following the tracks of a plurality of objects according to claim 2, the said relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back also comprises the step that each patch is included into corresponding point sequence:
To any one existing point sequence,, then provide notice and this existing point sequence of deletion that this existing point sequence has finished if there is not any new patch that closes on it.
4. according to claim 2 or 3 described methods of in virtual touch screen system, following the tracks of a plurality of objects, the said relation of centroid point on time and space according to the patch that obtains in adjacent two two field pictures in front and back also comprises the step that each patch is included into corresponding point sequence:
To any new patch in a plurality of new patch in the two field picture of being imported, if there is not any existing point sequence that closes on it, then providing this new patch is the notice of the starting point of a new point sequence, and creates a new point sequence.
5. method of in virtual touch screen system, following the tracks of a plurality of objects according to claim 4, said step of being somebody's turn to do the nearest centroid point of existing point sequence to each existing point sequence searching distance comprises:
Import an existing point sequence, and seek in a plurality of new patch from a two field picture of being imported apart from having the new patch that point sequence closes on;
Do not find under the situation of the new patch that the existing point sequence imported of distance closes at a plurality of new patch from a two field picture of being imported, then provide the notice that the existing point sequence imported will be deleted;
The new patch that the existing point sequence that has found distance to be imported at a plurality of new patch from a two field picture of being imported closes on; And the new patch that closes on that is found only is the new patch that closes on of the existing point sequence imported, confirms that then this new patch belongs to the existing point sequence of being imported; And
The new patch that the existing point sequence that has found distance to be imported at a plurality of new patch from a two field picture of being imported closes on; And the new patch that closes on that is found still is the new patch that closes on of other existing point sequences; If the distance of the new patch that closes on and the existing point sequence of being imported that are found is than little with the distance of other existing point sequences; Confirm that then this new patch belongs to and the nearest new patch of being imported of existing point sequence; Otherwise, provide the notice that the existing point sequence imported will be deleted;
Examine and whether all existing sequence of points have been carried out above-mentioned steps.
6. method of in virtual touch screen system, following the tracks of a plurality of objects according to claim 5, the step of seeking the new patch that distance should existing point sequence closes in the said a plurality of new patch from a two field picture of being imported comprises:
Import a new patch, calculate the distance between the new patch imported and the existing point sequence of being imported;
Calculated import new patch and the existing point sequence imported between the situation of distance less than a predetermined distance threshold Td under; The new patch of import is inserted into the candidate who belongs to the existing track of being imported to be closed in the new patch tabulation; Otherwise, confirm whether all new patches have been carried out above-mentioned steps;
Confirming that the candidate who belongs to the existing point sequence of being imported closes under the situation of size less than a predetermined size threshold value Tsize of new patch tabulation; Confirm whether all new patches have been carried out above-mentioned steps; Otherwise; The candidate that deletion belongs to the existing point sequence of being imported close in the new patch tabulation with the existing point sequence of being imported new patch farthest after, confirm whether all new patches have been carried out above-mentioned steps;
Confirming that new patch has been carried out under the situation of above-mentioned steps to all; Close on the tabulation of new patch for empty if belong to the candidate of the existing point sequence of being imported, then the candidate from said tabulation closes on the recently newly patch of the nearest new patch of the existing point sequence finding out in the new patch and imported as the existing point sequence of being imported.
7. method of in virtual touch screen system, following the tracks of a plurality of objects according to claim 1 also comprises:
Adopt following formula to carry out coordinate optimization carrying out smoothing processing to the point sequence of final acquisition,
Wherein
is the point in the point sequence; K is the iteration sign; N is the point sequence sign, and m is the radix of iteration point.
8. system that in virtual touch screen system, follows the tracks of a plurality of objects comprises:
Projector, with image projection to in the projection surface;
Depth cameras obtains the image of the environment of said projection surface continuously;
Subject detecting unit from the every two field picture that is obtained, detects candidate's patch of at least one object that is positioned at the preceding preset distance of said projection surface;
Tracking cell is included into corresponding point sequence according to the relation of centroid point on time and space of the patch that obtains in adjacent two two field pictures in front and back with each patch.
9. the system that in virtual touch screen system, follows the tracks of a plurality of objects according to claim 8, said tracking cell comprises:
Receiver module receives the centroid point that from a frame new images, obtains a plurality of new patches;
Patch is sought module recently; Retrieve a plurality of existing point sequences; To each existing point sequence; The searching distance should have the centroid point that point sequence closes on, and the pairing new patch of nearest centroid point is had the nearest new patch of sequence of points as being somebody's turn to do, and the new patch nearest apart from this existing point sequence is included into this existing point sequence.
10. the system that in virtual touch screen system, follows the tracks of a plurality of objects according to claim 9 also comprises: the smoothing processing unit, and adopt following formula to carry out coordinate optimization carrying out smoothing processing to the point sequence of final acquisition,
Wherein
is the point in the point sequence; K be the iteration sign and; N is the point sequence sign, and m is the radix of iteration point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010615896.6A CN102541417B (en) | 2010-12-30 | 2010-12-30 | Multi-object tracking method and system in virtual touch screen system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010615896.6A CN102541417B (en) | 2010-12-30 | 2010-12-30 | Multi-object tracking method and system in virtual touch screen system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102541417A true CN102541417A (en) | 2012-07-04 |
| CN102541417B CN102541417B (en) | 2014-02-26 |
Family
ID=46348423
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010615896.6A Expired - Fee Related CN102541417B (en) | 2010-12-30 | 2010-12-30 | Multi-object tracking method and system in virtual touch screen system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102541417B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103677240A (en) * | 2012-09-24 | 2014-03-26 | 株式会社理光 | Virtual touch interaction method and equipment |
| CN104867158A (en) * | 2015-06-03 | 2015-08-26 | 武汉理工大学 | Monocular vision-based indoor water surface ship precise positioning system and method |
| CN106022266A (en) * | 2016-05-20 | 2016-10-12 | 北京格灵深瞳信息技术有限公司 | Target tracking method and target tracking apparatus |
| CN106648235A (en) * | 2016-12-20 | 2017-05-10 | 美的智慧家居科技有限公司 | Projection touch system and integrated cabinet |
| CN113632055A (en) * | 2019-03-26 | 2021-11-09 | 曹卓祺 | Distinguishing and tracking multiple objects when placed on a capacitive touch screen |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH04260979A (en) * | 1990-11-28 | 1992-09-16 | Hitachi Ltd | Detecting and tracking system for mobile objection |
| US20010002843A1 (en) * | 1999-12-03 | 2001-06-07 | Kunio Yata | Automatic following device |
| CN101840062A (en) * | 2009-08-21 | 2010-09-22 | 深圳先进技术研究院 | Interactive projection system and interactive method |
-
2010
- 2010-12-30 CN CN201010615896.6A patent/CN102541417B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH04260979A (en) * | 1990-11-28 | 1992-09-16 | Hitachi Ltd | Detecting and tracking system for mobile objection |
| US20010002843A1 (en) * | 1999-12-03 | 2001-06-07 | Kunio Yata | Automatic following device |
| CN101840062A (en) * | 2009-08-21 | 2010-09-22 | 深圳先进技术研究院 | Interactive projection system and interactive method |
Non-Patent Citations (2)
| Title |
|---|
| 付晓薇 等: "一种基于特征的多目标跟踪算法", 《计算机工程与应用》, no. 35, 11 December 2004 (2004-12-11) * |
| 张嫣 等: "运动图像序列中多目标跟踪的研究与实现", 《计算机应用研究》, no. 1, 28 January 2002 (2002-01-28) * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103677240A (en) * | 2012-09-24 | 2014-03-26 | 株式会社理光 | Virtual touch interaction method and equipment |
| CN103677240B (en) * | 2012-09-24 | 2017-08-25 | 株式会社理光 | Virtual touch exchange method and virtual touch interactive device |
| CN104867158A (en) * | 2015-06-03 | 2015-08-26 | 武汉理工大学 | Monocular vision-based indoor water surface ship precise positioning system and method |
| CN104867158B (en) * | 2015-06-03 | 2017-09-29 | 武汉理工大学 | Indoor above water craft Precise Position System and method based on monocular vision |
| CN106022266A (en) * | 2016-05-20 | 2016-10-12 | 北京格灵深瞳信息技术有限公司 | Target tracking method and target tracking apparatus |
| CN106648235A (en) * | 2016-12-20 | 2017-05-10 | 美的智慧家居科技有限公司 | Projection touch system and integrated cabinet |
| CN113632055A (en) * | 2019-03-26 | 2021-11-09 | 曹卓祺 | Distinguishing and tracking multiple objects when placed on a capacitive touch screen |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102541417B (en) | 2014-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102841733B (en) | Virtual touch screen system and method for automatically switching interaction modes | |
| US10638117B2 (en) | Method and apparatus for gross-level user and input detection using similar or dissimilar camera pair | |
| US9117274B2 (en) | System and method for interactive markerless paper documents in 3D space with mobile cameras and projectors | |
| US20120274550A1 (en) | Gesture mapping for display device | |
| US20170024017A1 (en) | Gesture processing | |
| Shen et al. | Vision-based hand interaction in augmented reality environment | |
| EP2790089A1 (en) | Portable device and method for providing non-contact interface | |
| CN102566827A (en) | Method and system for detecting object in virtual touch screen system | |
| US20110298708A1 (en) | Virtual Touch Interface | |
| CN104254817A (en) | Rapid gesture re-engagement | |
| CN103294401A (en) | Icon processing method and device for electronic instrument with touch screen | |
| US20150058782A1 (en) | System and method for creating and interacting with a surface display | |
| US20120319945A1 (en) | System and method for reporting data in a computer vision system | |
| CN102541417B (en) | Multi-object tracking method and system in virtual touch screen system | |
| Dai et al. | Touchscreen everywhere: On transferring a normal planar surface to a touch-sensitive display | |
| Katz et al. | A multi-touch surface using multiple cameras | |
| CN102799344B (en) | Virtual touch screen system and method | |
| KR101461145B1 (en) | System for Controlling of Event by Using Depth Information | |
| CN106796462A (en) | Determine the position of input object | |
| CN109871178A (en) | A kind of virtual touch screen system based on image recognition | |
| CN116301551A (en) | Touch identification method, touch identification device, electronic equipment and medium | |
| CN103543824B (en) | Gesture input system and method | |
| KR101289883B1 (en) | System and method for generating mask image applied in each threshold in region | |
| Cheng et al. | Fingertip-based interactive projector–camera system | |
| Jaiswal et al. | Creative exploration of scaled product family 3D models using gesture based conceptual computer aided design (C-CAD) tool |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20140226 Termination date: 20151230 |
|
| EXPY | Termination of patent right or utility model |