CN102541417B - Multi-object tracking method and system in virtual touch screen system - Google Patents
Multi-object tracking method and system in virtual touch screen system Download PDFInfo
- Publication number
- CN102541417B CN102541417B CN201010615896.6A CN201010615896A CN102541417B CN 102541417 B CN102541417 B CN 102541417B CN 201010615896 A CN201010615896 A CN 201010615896A CN 102541417 B CN102541417 B CN 102541417B
- Authority
- CN
- China
- Prior art keywords
- point sequence
- patch
- new
- inputted
- new patch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The invention provides a multi-object tracking method and a multi-object tracking system in a virtual touch screen system, which comprises: projecting an image onto a projection surface; continuously acquiring the image of the environment of the projection surface; detecting the candidate patch of at least one object within a preset distance before the projection surface from the acquired frames of images; and classifying the patches into corresponding sequences of points according to the time and spatial relationships between the centroid points of the acquired patches in two adjacent frames of images.
Description
Technical field
The present invention relates to a kind of field of human-computer interaction, especially design digital image processing field.Particularly, the present invention relates to follow the tracks of in a kind of virtual touch screen system a plurality of object method and system.
Background technology
Touchscreen technology is now by more and more widely for example, for example, for the portable equipment as HMI equipment (smart phone) and PC (Desktop PC).By touch screen, user can operate more comfortable and easily this equipment and bring good experience.Although touchscreen technology is extremely successful in handheld device, but the touch screen for large-sized monitor but still exists some problems and chance to belong to Canesta, the US Patent No. 7151530B2 that Inc title is " System and Method for Determining an InputSelected By a User through a Virtual Interface (determining the system and method for user-selected input by virtual interface) " has proposed a kind of for select which key assignments to be designated as the method for current key assignments at one group of key assignments, therefore provided with virtual interface in the object that intersects of region.This virtual interface can be realized and in key assignments group, selects single key assignments and determine position with depth transducer, and this depth transducer can be determined the degree of depth of the position relevant to the position of depth transducer.In addition, the placement property of object or the style characteristic of object one of at least can be determined.Positional information can be similar to object with respect to the degree of depth of position transducer or other reference point.In the pel array of camera, during the existing of the pixel denoted object of sufficient amount, just think and this object detected.Determine with the shape of the object of the surface crosswise of virtual input region and for example, compare with multiple known shape (finger or stylus).
Belong to equally Canesta, Inc title for " Quasi-Three-Dimensional Method AndApparatus To Detect And Localize Interaction Of User-Obj ect And VirtualTransfer Device (for detection of and mutual standard three method for position and the equipment of consumer positioning-object and virtual conversion equipment) " US Patent No. 6710770B2 disclosed a kind of virtual bench input or transmission information of adopting to the system of auxiliary equipment, comprise two optical system OS1 and OS2.In light structure embodiment, OS1 is on virtual bench and be parallel to the luminous energy of this virtual bench transmitting fan beam plane ().When user object penetrates interested beam plane, OS2 registers this event.Triangulation method can be located virtual contact, and user's predetermined information is transferred to subsystem.In non-structure active light structure, OS1 is preferably a kind of digital camera, and its visual field has defined interested plane, and this plane is illuminated by an active luminous energy source.
The title that belongs to Apple company is a kind of equipment and method of simultaneously following the tracks of a plurality of fingers and palm contact point when approaching, approach the many touch-surfaces of sensing or slide at hand thereon for " identifying contacts on a touch surface (contact on identification touch-surface) " US Patent No. 7619618B2 has disclosed.The detection of hand structure and action and classification intuitively realized unprecedentedly integrated that key entry in multi-usage ergonomics computer output device, static, indication, rolling, 3D handle.
The U.S. Patent application US20100073318A1 that the title that belongs to Matsushita Electric company is " Multi-touch surface providingdetection and tracking of multiple touch points (many touch-surfaces of multiple touch points detection and tracking are provided) " has disclosed a kind of by using the system and method for many touch sensitive surface that two independent array of orthogonal linear tolerance limit sensor can detection and tracking multiple touch points.
From these prior aries of mentioning above, it seems, most of large scale touch screen are all based on magnetic board (such as electronic whiteboard), IR border (such as interactive large-sized monitor) etc.For the current technical scheme of large scale touch screen, still there are a lot of problems at that time, for example: in general, the equipment of these types is conventionally large and heavy due to its volume that its hardware causes, and is therefore difficult to carry, and does not have portability.And the device screen size of these types is subject to the restriction of hardware and size is fixed and can not be according to the needs of environment and freely regulate, but also need a kind of special time writer or a kind of IR pen to operate.
For some virtual whiteboard projector, user must control the on/off switch of laser pen, and this is very loaded down with trivial details, therefore has the unmanageable problem of laser pen.In addition, in this virtual whiteboard projector, once laser pen is closed, be just difficult to accurately laser spots be navigated to next position, therefore exist laser spots to locate difficult problem.In some virtual whiteboard projector, adopt finger mouse to substitute laser pen, still, adopted the virtual whiteboard projector of finger mouse touch beginning (touch on) can not be detected or touch end (touch up).
Summary of the invention
In order to solve these problems of the prior art of mentioning above, the present invention proposes and in a kind of virtual touch screen system, follow the tracks of a plurality of object method and system.Particularly, in this virtual touch screen system, following the tracks of a plurality of object method comprises: project image onto in Yi Yi projection surface; Obtain continuously the image of the environment of described projection surface; From obtained every two field picture, detect the alternative patch of at least one object that is positioned at the front preset distance of described projection surface; And the relation on time and space is included into corresponding point sequence by each patch according to the centroid point of the patch that obtains in adjacent two two field pictures in front and back.
According to method of following the tracks of a plurality of objects in virtual touch screen system of the present invention, the described step that relation on time and space is included into corresponding point sequence by each patch according to the centroid point of the patch that obtains in adjacent two two field pictures in front and back comprises: receive and from a frame new images, obtain the centroid point of a plurality of new patches; Retrieve a plurality of existing point sequences, for each existing point sequence, find the centroid point closing on apart from this existing point sequence, using the corresponding new patch of nearest centroid point as the nearest new patch of this existing sequence of points; And the new patch nearest apart from this existing point sequence is included into this existing point sequence.
According to method of following the tracks of a plurality of objects in virtual touch screen system of the present invention, the described step that relation on time and space is included into corresponding point sequence by each patch according to the centroid point of the patch that obtains in adjacent two two field pictures in front and back also comprises: for any one existing point sequence, if not there is not any new patch closing on it, provide the notice that this existing point sequence finished and delete this existing point sequence.
According to method of following the tracks of a plurality of objects in virtual touch screen system of the present invention, the described step that relation on time and space is included into corresponding point sequence by each patch according to the centroid point of the patch that obtains in adjacent two two field pictures in front and back also comprises: for any one the new patch in a plurality of new patch in an inputted two field picture, if there is no any existing point sequence closing on it, providing this new patch is the notice of the starting point of a new point sequence, and creates a new point sequence.
According to method of following the tracks of a plurality of objects in virtual touch screen system of the present invention, the described step for each existing point sequence searching centroid point nearest apart from this existing point sequence comprises: input an existing point sequence, and find the new patch closing on apart from this existing point sequence in a plurality of new patch from an inputted two field picture; In the situation that the new patch that the existing point sequence that a plurality of new patch from an inputted two field picture does not find distance to input closes on provides the notice that inputted existing point sequence will be deleted;
A plurality of new patch the two field picture from inputted has found the new patch closing on apart from inputted existing point sequence, and the new patch closing on finding is only the new patch closing on of inputted existing point sequence, determine that this new patch belongs to inputted existing point sequence; And a plurality of new patch the two field picture from inputted has found the new patch closing on apart from inputted existing point sequence, and the new patch closing on finding or the new patch closing on of other existing point sequences, if the new patch closing on finding with the distance of inputted existing point sequence than little with the distance of other existing point sequences, determine that this new patch belongs to the new patch nearest with inputted existing point sequence, otherwise, provide the notice that inputted existing point sequence will be deleted; Examine and whether all existing sequence of points have been carried out to above-mentioned steps.
According to method of following the tracks of a plurality of objects in virtual touch screen system of the present invention, the step of finding the new patch close on apart from this existing point sequence in described a plurality of new patch from an inputted two field picture comprises: input a new patch, calculate the distance between the new patch of inputting and the existing point sequence of inputting; In the situation that the distance between calculated the new patch of inputting and the existing point sequence inputted is less than a predetermined distance threshold Td, the candidate that inputted new patch is inserted into the existing track that belongs to inputted closes in new patch list, otherwise, be confirmed whether all new patches to have been carried out to above-mentioned steps; In the situation that determine that the size that the candidate of the existing point sequence belong to inputted closes on new patch list is less than a predetermined size threshold value Tsize, be confirmed whether all new patches to have been carried out to above-mentioned steps, otherwise, the candidate that deletion belongs to inputted existing point sequence closes on after in new patch list and inputted existing point sequence new patch farthest, is confirmed whether all new patches to have been carried out to above-mentioned steps; In the situation that confirming all new patches to have been carried out to above-mentioned steps, if belong to the candidate of inputted existing point sequence, close on new patch list for empty, the candidate from described list closes in new patch and finds out the new patch nearest with inputted existing point sequence as the recently new patch of inputted existing point sequence.
According to method of following the tracks of a plurality of objects in virtual touch screen system of the present invention, also comprise: to the point sequence of final acquisition, adopt following formula to carry out coordinate optimizing to carry out smoothing processing,
Wherein
be the point in point sequence, k is iteration sign, and n is point sequence sign, the radix that m is iteration point.
According to another aspect of the present invention, provide a kind of system of following the tracks of a plurality of objects in virtual touch screen system, having comprised: projector, has projected image onto in Yi Yi projection surface; Depth cameras, obtains the image of the environment of described projection surface continuously; Subject detecting unit, from obtained every two field picture, detects candidate's patch of at least one object that is positioned at the front preset distance of described projection surface; Tracking cell, the relation according to the centroid point of the patch that obtains in adjacent two two field pictures in front and back on time and space is included into corresponding point sequence by each patch.
According to system of following the tracks of a plurality of objects in virtual touch screen system of the present invention, described tracking cell comprises: receiver module, receives and from a frame new images, obtain the centroid point of a plurality of new patches; Patch is found module recently, retrieve a plurality of existing point sequences, for each existing point sequence, the centroid point that searching closes on apart from this existing point sequence, using the corresponding new patch of nearest centroid point as the nearest new patch of this existing sequence of points, and the new patch nearest apart from this existing point sequence is included into this existing point sequence.
According to system of following the tracks of a plurality of objects in virtual touch screen system of the present invention, also comprise smoothing processing unit, to the point sequence of final acquisition, adopt following formula to carry out coordinate optimizing to carry out smoothing processing,
Wherein
be the point in point sequence, k is iteration sign, and n is point sequence sign, the radix that m is iteration point.
Accompanying drawing explanation
Shown in Fig. 1 is according to the schematic diagram of the framework of virtual touch screen system of the present invention.
Shown in Fig. 2 is the performed object detection of control module according to the present invention and the overview flow chart to image tracing processing.
Shown in Fig. 3 (a)-(c) is from current depth map, to dispose the schematic diagram of background depth map
Shown in Fig. 4 is that the depth map of inputted current scene is carried out to binary conversion treatment to obtain the schematic diagram of the patch of candidate target.
Shown in Fig. 5 (a) is for patch being added to the schematic diagram of the connected domain of numbering
Shown in Fig. 5 (b) is according to the schematic diagram of the bianry image of the patch with connected domain numbering of depth map generation.
Shown in Fig. 6 (a)-(d) is the schematic diagram of enhancing process of the bianry image of patch.
Shown in Fig. 7 is the schematic diagram that detects the process of the coordinate of the centroid point of patch in the binary image of the patch as shown in Fig. 6 (d).
Shown in Fig. 8 is user's finger or the schematic diagram of the track that stylus moves on the screen of virtual touch screen.
Shown in Fig. 9 for following the tracks of the process flow diagram of detected object.
Shown in Figure 10 is according to the process flow diagram that the present invention is directed to the recently new spot of every existing track of all existing tracks searchings.
Shown in Figure 11 is to find the process flow diagram apart from its nearest new patch for inputted existing track.
Shown in Figure 12 is the method that the point sequence of the motion track of a kind of detected object that a strip adoption the present invention is obtained on virtual touch screen carries out smoothing processing.
Shown in Figure 13 (a) is the schematic diagram of the motion track of the detected object that obtains of a kind of the present invention of employing on virtual touch screen.
Shown in Figure 13 (b) is the schematic diagram of the object motion track after smoothing processing.
Shown in Figure 14 is the schematic diagram of the concrete configuration of control module.
Embodiment
Below, describe with reference to the accompanying drawings specific embodiments of the invention in detail.
Shown in Fig. 1 is according to the schematic diagram of the framework of virtual touch screen system of the present invention.As shown in Figure 1, virtual touch screen system according to the present invention comprises projector equipment 1, optical device 2, control module 3 and projection surface (also can be referred to as projection screen or virtual screen below) 4.In the specific embodiment of the present invention, projector equipment can be projector, and it projects in projection surface 4 as a kind of virtual screen, so that user is at the enterprising line operate of this virtual screen for the image that needs are shown.Optical device 2 can be for example any equipment that can obtain image, depth cameras for example, and it obtains the depth information of the environment of projection surface 4, and based on this depth information generating depth map.Control module 3 detects at least one object in the described surperficial preset distance of distance and follows the tracks of detected object along the direction away from described surface, to generate level and smooth point sequence.Interactive command etc. is painted, combined to described point sequence for further interactive task, for example, on virtual screen.
Projector equipment 1 projects image onto in projection surface 4 as a kind of virtual screen, for example, so that user, at the enterprising line operate of this virtual screen, draws or combination interactive command.Optical device 2 is caught the environment of the object (such as the user's finger or the stylus that touch in Gai projection surface 4) that includes projection virtual screen and be positioned at the place ahead of projection surface 4.This optical device 2 obtains the depth information of the environment of projection surface 4, and based on this depth information generating depth map.So-called depth map is exactly, depth cameras is positioned at the environment before camera lens by shooting, and calculate in captured environment each pixel apart from the distance of depth cameras, and adopt for example numerical value of 16 to record the object of each pixel representative and the distance between depth cameras, thereby there are 16 bit value of the incidental expression distance of these each pixels to form the figure that a width represents distance between each pixel and camera.Depth map is sent to control module 3 subsequently, and control module 3 detects at least one object in the described surperficial preset distance of distance along the direction away from described projection surface 4.When this object being detected, follow the tracks of the touch action of this object in projection surface to form touch point sequence.Subsequently, 3 pairs of formed touch point sequences of control module are carried out smoothing processing, thereby realize the painting function on this virtual interacting screen.And, these touch point sequences generation interactive command that can be combined, thus realize the interactive function of virtual touch screen, and final virtual touch screen can change according to generated interactive command.The present invention also can adopt other ordinary cameras and other common foreground object detection systems to carry out.For tracking mode of the present invention is convenient to understand, below the testing process of some foreground object of paper, but this testing process is not to realize multi-object to follow the tracks of necessary enforcement means, and is only the prerequisite of following the tracks of a plurality of objects.Namely, the detection of object does not belong to the content to image tracing.
Shown in Figure 14 is the schematic diagram of the concrete configuration of control module.Described control module 3 conventionally comprises depth map processing unit 31, subject detecting unit 32, image enhancing unit 33 and coordinate and calculates and converter unit 34 and tracking cell 35 and smooth unit 36.Depth map processing unit 31 using its depth map of being caught that depth cameras was sent as inputting and process this depth map to background is disposed from this depth map, and subsequently to the connected domain numbering on this depth map.The depth information of the described depth map of subject detecting unit 32 based on from described depth map processing unit 31, based on a predetermined depth threshold, depth map is carried out to binary conversion treatment, formation is as a plurality of patches of candidate target, subsequently, relation based between each patch and connected domain and the size of plaque area, determine the patch as object.Coordinate calculates and converter unit 34 calculates the centroid point coordinate of the patch of the object that is defined as, and its centroid point coordinate transform is arrived to target coordinate system, the i.e. coordinate system of virtual interacting screen.Follow the tracks of and smooth unit 35 tracking detected a plurality of patches in the multiple image of taking continuously, to generate the coordinate point sequence after a plurality of centroid points conversion, and subsequently generated coordinate point sequence is carried out to smoothing processing.
Shown in Fig. 2 is that the performed object detection of control module 3 is processed and the overview flow chart to image tracing.As shown in Figure 2.At step S21 place, depth map processing unit 31 receives the depth map that depth cameras 2 is obtained, this depth map obtains in the following way, be that depth cameras 2 is taken current environment image, and measure each pixel apart from the distance of depth cameras when taking, 16 (can be 8 or 32 the according to actual needs) depth informations that value was recorded of take form, and 16 bit depth values of these each pixels have formed described depth map.For follow-up treatment step, can, before obtaining the depth map of current scene, obtain in advance the front background depth map without any detected object of projection screen.At step S22 place, depth map processing unit 31 is processed received depth map to background is disposed from this depth map, and only retains the depth information of foreground object, and the connected domain in retained depth map is numbered subsequently.Shown in Fig. 3 (a)-(c) is from current depth map, to dispose the schematic diagram of background depth map.The depth map that this employing 16 bit value of institute's diagram show is the convenience in order to illustrate, is not must demonstrate in implementing process of the present invention.Is a kind of schematic diagram of example of depth map of background as shown in Fig. 3 (a), is only the depth map of background depth map ，Ji projection surface in the depth map shown in it, and it does not comprise the depth image of any foreground object (being object).A kind of mode that obtains the depth map of background is, in the starting stage of the method for virtual touch screen System Implementation of the present invention virtual touch screen of the present invention function, first the snapshots in time of obtaining the depth map of current scene and preserving this depth map by optical device 2, thereby the depth map of acquisition background.When obtaining the depth map of this background, in the place ahead of current scene Zhong， projection surface 4 (between optical device 2 and projection surface 4), can not have any for touching the dynamic object of projection surface.The another kind of mode that obtains the depth map of background is not with transient state photo but generates a kind of average background depth map with a series of continuous transient state photos.
Shown in Fig. 3 (b) is the example of the depth map of catching of a frame current scene, wherein has an object (for example user's hand or stylus) to touch in projection surface.
Shown in Fig. 3 (c) is the wherein example of the depth map of background after being eliminated of a frame.The possible mode of removing background depth is to use a depth map for the depth map subtracting background of current scene, and another kind of mode is the depth map of scanning current scene and the corresponding point in the every bit of this depth map and background depth map are carried out to depth value comparison.If the absolute value of the degree of depth difference of the pixel that these are paired is similar and within a predetermined threshold value, the similar respective point of the absolute value of the degree of depth difference in current scene is disposed from the depth map of current scene, otherwise reservation is put accordingly, do not carried out any change.Subsequently the connected domain of removing in background depth map current depth map is afterwards numbered.Connected domain of the present invention refers to such region: supposing that depth cameras is captured has two 3D points, if their the upper difference adjacent one another are and its depth value of XY plane (captured picture) that is projected in is not more than given threshold value D, claim their D-connections each other.If there is D-communication path between any two points in one group of 3D point, claim this group 3D point D-to be communicated with.If for each the some P in the 3D point being communicated with for one group of D-, each some P does not exist adjacent point to add in described group in XY plane, claim that the 3D point that this group D-is communicated with is that maximum D-is communicated with in the situation that not interrupting this connection condition.Connected domain of the present invention is that one group of D-connectivity points in depth map and its are that maximum D-is communicated with.Continuous blocks (mass) region that the connected domain of described depth map is caught with described depth cameras is corresponding, and connected domain is that the D-connected set of points in described depth map merges and its maximum D is associated.Therefore, in fact connected domain is numbered is exactly to having 3D point that above-mentioned D-the is communicated with identical numbering of annotating, that is to say, for the pixel that belongs to same connected domain add pour down with numbering, therefore, the numbering matrix of generation connected domain.The continuous block (mass) that the connected domain of described depth map is caught corresponding to described depth cameras.
The numbering matrix of described connected domain be a kind of can mark described in which point in depth map whether be the data structure of connected domain.Each element in described numbering matrix is corresponding to a point in depth map, and the value of this element is exactly the numbering (numbering of a connected domain) of the connected domain under this point.
Then, at step S23 place, based on a depth conditions, each point in depth map is carried out to binary conversion treatment, thereby generate some as the patch of alternative objects, and add connected domain numbering to belonging to the pixel of the patch of same connected domain.To describe concrete binary conversion treatment process below in detail.Shown in Fig. 4 is that the depth map of inputted current scene is carried out to binary conversion treatment to obtain the schematic diagram of the patch of alternative objects.。As shown in Figure 4, the present invention is based on each pixel in the depth map in current scene and the relative depth information between the corresponding pixel points of background depth map and carry out binary conversion treatment process.In an embodiment of the present invention, from the depth map of described current scene, retrieve the depth value of a pixel, i.e. distance between depth cameras and the object-point of the pixel representative retrieved.In Fig. 4, to travel through the mode of all pixels, from the depth map of inputted current scene, retrieve the depth d of a pixel, the depth value b of the pixel that the retrieval from background depth map is afterwards corresponding with the pixel of retrieving depth map from current scene, then calculate difference (subtraction value) s, the i.e. s=b-d of the depth d of two target points and the degree of depth b of background pixel point.If the difference obtaining is greater than zero and be less than a predetermined distance threshold t, i.e. 0 < s < t, the gray-scale value of the pixel retrieving in current scene is set to 255, otherwise is made as 0.Certainly, this binaryzation also can be directly by two kinds of these positions 0 or 1 of situations difference, as long as these the two kinds binaryzation modes that make a distinction can be able to be adopted.By above-mentioned binaryzation mode, can obtain the patch with a plurality of alternative objects as shown in Fig. 5 (b).The size of threshold value t can be controlled the precision that object is detected.Threshold value t is also relevant to the hardware specification of depth cameras.The value of threshold value t is generally the thickness size of a finger or the diameter of common stylus, and for example 0.2-1.5 centimetre, is preferably 0.3 centimetre, 0.4 centimetre, 0.7 centimetre, 1.0 centimetres.Threshold value t can be according to wherein using the environment of virtual touch screen system to adjust, for distance threshold t, the setting of concrete size can be according to the present invention in application process the concrete size of the diameter of the finger thickness of the people before screen or the stylus used arrange accordingly.Conventionally the possible values of this threshold value t is 1 centimetre.
Shown in Fig. 5 (a) is for patch being added to the schematic diagram of the connected domain of numbering.After obtaining the bianry image of patch, the pixel that scan search contains connected domain numbering, and add this connected domain numbering to pixel corresponding in binaryzation patch image, thus some patch is numbered with connected domain, as shown in Fig. 5 (b).Patch in described bianry image (white portion or point) is the candidate that touches the possible destination object in projection surface.According to noted earlier, the binaryzation patch with connected domain numbering in Fig. 5 (b) possess following two conditions: 1. patch belongs to connected domain.2. the corresponding depth d of each pixel of patch and the difference of background depth b (subtraction value) s must be less than threshold value t, i.e. s=b-d < t.
Then, at step S24 place, the binaryzation patch image of obtained depth map is strengthened to processing, to reduce noise unnecessary in binaryzation patch image and to make the shape of patch become clearer and stable.This step has image enhancing unit 33 to carry out.Particularly, as follows carry out described enhancing processing.
First, remove the patch that does not belong to connected domain, namely by the patch that does not add connected domain numbering at step S23 place directly by its gray-scale value from the highest vanishing, for example by the gray-scale value of its pixel from 255 vanishing.In another kind of mode, by 1, become 0.Thereby obtain the patch binary image as shown in Fig. 6 (a).
Secondly, remove and belong to the patch that its area S is less than the connected domain of an area threshold Ts.In an embodiment of the present invention, patch belong to a certain connected domain mean this patch at least one point in connected domain.If the area S of the connected domain under this patch is less than an area threshold Ts, corresponding patch is regarded as noise and is removed the bianry image from patch.Otherwise patch is considered to the candidate of destination object.Area threshold Ts is that the environment that can use according to virtual touch screen system regulates.Area threshold Ts is generally 200 pixels.Thus, obtain the binary image of the patch as shown in Fig. 6 (b).
Then, exactly the patch in the binary image of the obtained patch as shown in Fig. 6 (b) is carried out to some morphology (morphology) operation.In the present embodiment, adopted expansion (dilation) to operate and close (close) operation.First carrying out an expansive working is then to carry out iteratively closed operation.The iterations that carries out closed operation is a predetermined value, and the environment that this predetermined value can be used according to virtual touch screen system regulates.This iterations for example can be set to 6 times.Finally obtain the binary image of the patch as shown in Fig. 6 (c).
Finally, if there are a plurality of patches belong to same connected domain, these patches have identical connected domain numbering, retain in the patch with identical connected domain numbering, to have a patch of maximum area and remove other patches.In an embodiment of the present invention, a connected domain can include a plurality of patches.In these patches, the patch only with maximum area is just considered to destination object, and other patch is the noise that need to be removed.Finally obtain the binary image of the patch as shown in Fig. 6 (d).
At step S25 place, detect the profile of the patch that obtains, the coordinate of the centroid point of calculating patch also arrives coordinates of targets by centroid point coordinate transform.This detection, calculating and map function are calculated by coordinate and converter unit 34 is carried out.Shown in Fig. 7 is the schematic diagram that detects the process of the coordinate of the centroid point of patch in the binary image of the patch as shown in Fig. 6 (d).Referring to Fig. 7, according to the geological information of patch, calculate the coordinate of the centre of form of patch.This computation process comprises: the profile of detection of plaque, calculate the Hu square of this profile and the coordinate that calculates centroid point with described Hu square.In an embodiment of the present invention, can carry out by multiple known mode the profile of detection of plaque.Also can calculate with known algorithm Hu square.After obtaining the Hu square of described profile, can calculate by following formula the coordinate of centroid point:
(x
0，y
0)＝(m
10/m
00，m
01/m
00)
(x wherein
0, y
0) be the coordinate of centroid point, and m
10, m
01, m
00it is exactly Hu square.
Coordinate transform is exactly the coordinate of centroid point to be transformed to the coordinate system of user interface from the coordinate system of the bianry image of patch.The conversion of coordinate system can adopt known method.
In order to obtain the continuous moving track of touch point, can be by the touch point in successive frame depth map captured in continuous detecting virtual touch screen system of the present invention, thereby follow the tracks of detected a plurality of patch to produce the sequence of a plurality of points, obtain thus the movement locus of touch point.
Particularly, be exactly at step S26 place, to obtaining the centroid point coordinate in user interface of the patch of every two field picture after every frame depth map execution step S21-S25 of continuous shooting, follow the tracks of, generate centroid point sequence (being track), and obtained centroid point sequence is carried out to smoothing processing.This tracking and smooth operation are undertaken by tracking cell 35.
Shown in Fig. 8 is user's finger or the schematic diagram of the track that stylus moves on the screen of virtual touch screen.The movement locus that has wherein shown two objects (finger).This is only an example.In other cases, can a plurality of objects, for example 3,4,5 objects, are decided according to the actual requirements.
Shown in Fig. 9 for following the tracks of the process flow diagram of detected object.Repeatedly carry out trace flow as shown in Figure 9, finally can obtain the movement locus of the front any object of screen.Particularly, carry out following the tracks of operation is exactly that the centroid point coordinate in user interface obtains in any track before being included into by the patch in the depth map newly detecting.
Centroid point coordinate according to a plurality of institutes detection of plaque in user interface, follows the tracks of a plurality of patches that newly detect, thereby produces many tracks and trigger the relevant touch event about these tracks.To follow the tracks of patch, just need to patch classify and by the centroid point coordinate of patch be placed in one a little on time and space in relevant point sequence.Only have the point in identical sequence just can merge into a track.As shown in Figure 8, if virtual touch screen system is supported painting function, the point in the sequence shown in Fig. 8 just represents the paint command on projection screen, and the point in same sequence just can couple together and form a curve as shown in Figure 8 so.
In the present invention, can follow the tracks of three kinds of touch events: touch and start, touch to move and touch and finish.Touch and start just to refer to that the object that will detect touches projection screen and track is started.Touch mobile refer to wanted detected object just touching on projection screen and track Zheng projection surface on continue.And touch to finish to refer to surface and the motion track that the object that will detect leaves projection screen, finish.
As shown in Figure 9, at step S91, receive based on a frame depth map according to the new patch of the detected object of step S21-S25 the centroid point coordinate in user interface, this is that coordinate calculates and the output of converter unit 34.
Subsequently, at step S92 place, for before the patch of each frame depth map is followed the tracks of process after each point sequence in all point sequences that obtain (i.e. all existing tracks, below all become existing track), calculate the new patch nearest apart from this existing track.The track of the object of all touches in touch screen (being projection screen) is all retained in virtual touch screen system.Each track keeps a tracked patch, and this tracked patch is the last patch that is endowed this track.The distance of new patch of the present invention and existing track refers to the distance between the last patch in a new patch and an existing track.
Then, at step S93 place, new patch is included into apart from its nearest existing track, and triggers touch moving event.
Then, at step S94 place, if for an existing track, do not exist any new patch and its to approach, in other words, if all new patches have been included into respectively other existing tracks, delete this existing track, and trigger the touch End Event for this existing track.
Finally, at step S95 place, if for each new patch, do not exist any existing track and its to approach, in other words, obtain all existing tracks all deleted owing to triggering touch End Event, or the distance of new patch and all existing tracks is not within certain distance threshold scope before, determine the starting point that this new patch is new track, and trigger the beginning event that touches.
Repeatedly carry out above-mentioned steps S91-S95, realize the tracking to the centroid point of the patch in successive frame depth map coordinate in user interface, thereby consist of a little a track by what belong to same point sequence.
When there is many existing tracks, will repeatedly perform step S92 for every existing track.The particular flow sheet of tracking cell of the present invention 35 execution step S92 shown in Figure 10.
First, at step S101 place, examine and traveled through all existing tracks.This can just can realize by a simple counter.If all carried out step S92 for all existing tracks, end step S92.If no, advance to step S102.
At step S102 place, input next existing track.At step S103 place, for inputted existing track, find the new patch closing on apart from it subsequently.Then enter step S104.
At step S104 place, determine whether to have found for inputted existing track the new patch closing on apart from it.If found the new patch closing on apart from inputted existing track, advance to step S105, otherwise, enter step S108.
At step S108 place, because the existing track for inputted does not exist the new patch closing on, therefore, the existing Trajectories Toggle that this is inputted is " the existing track that will delete ".Turn back to subsequently step S101.Thus, will, at step S94 place, for this " the existing track that will delete ", trigger and touch End Event.
At step S105 place, determine whether the new patch closing on apart from the existing track of inputting is also the new patch closing on apart from other existing tracks.In other words, determine this new patch whether belong to simultaneously apart from two or more existing track close on new patch.If judge this new patch belong to simultaneously two or more existing track close on new patch, process and enter step S106, otherwise, process and enter step S109.
At step S109 place, because this new patch is only the new patch closing on of inputted existing track, therefore, this new patch is included into inputted existing track as its nearest new patch, become one of point in the point sequence of this existing track.Process and turn back to step S102 afterwards.
At step S106 place, due to this new patch belong to simultaneously two or more existing track close on new patch, calculate the distance of every existing track in this new patch and affiliated many existing tracks.Then at step S107 place, the size of the distance of relatively calculating in step S106 place, and determine between this new patch and the existing track of inputting distance whether in calculated distance for minimum, definite this new patch and the existing track inputted between distance whether than and other existing tracks between distance all little.If for minimum, process and enter step S109 in the distance that between the existing track of determining this new patch and inputting, distance is calculated in step S106 place, otherwise, process and enter step S108.
Repeatedly carry out above-mentioned steps S101-109, thus the processing that performing step S92 carries out, thus can travel through the patch of the new detection of all existing tracks and input.
Shown in Figure 11 is for inputted existing track, to find the process flow diagram of the new patch closing on apart from it.As shown in figure 11, at step S111 place, examine the distance of closing between the existing track that whether has all calculated and inputted for inputted all new patch.If close on distance between the existing track that has all calculated and inputted for all new patches, process and enter step S118, otherwise, process and enter step S112.
At step S118 place, whether the list of definite new patch closing on as inputted existing track is empty.If empty, end process, otherwise, enter step S119.At step S119 place, in all new patch lists that close on, find the nearest new patch of the existing track of inputting with this, and this nearest new patch is included into the point sequence of inputted existing track.End step S103 afterwards.
At step S112 place, the next new patch of input.Subsequently, at step S113 place, calculate next new patch and the existing track inputted between distance.Then, at step S114 place, determine whether the distance between the new patch of the next one calculating and the existing track of inputting is less than a predetermined threshold value.If determine that the distance between the new patch of the next one calculating and the existing track of inputting is less than a predetermined distance threshold Td, process and enter step S115, otherwise, turn back to step S111.Distance threshold Td is set to the distance of 10-20 pixel conventionally herein, is preferably the distance of 15 pixels.The environment that this threshold value Td is used according to virtual touch screen system is adjusted.In the present invention, if the distance between new patch and an existing track is less than described distance threshold Td, be referred to as this new patch and this existing track closes on.
At step S115 place, the new patch of the described next one is inserted in the new patch list of candidate of the existing track that belongs to inputted.At step S116 place, determine whether the size of the new patch list of candidate of the existing track that belongs to inputted is less than a predetermined size threshold value Tsize subsequently.If determine that the size of the new patch list of candidate of the existing track that belongs to inputted is less than a predetermined size threshold value Tsize, process and turn back to step S111, otherwise, process and enter step S117.At step S117 place, the longest new patch of candidate of distance belonging between existing track in the new patch list of candidate of inputted existing track and that input is deleted from described list, turn back to afterwards step S111.
Repeatedly carry out the step shown in Figure 11, thus completing steps S103.
The above flow process of following the tracks of patch coordinate in user interface for successive image frame of having described with reference to accompanying drawing 9-11.By above-mentioned tracking, operate, the touch that triggers institute's detected object starts event, touches moving event or touches End Event.Thus, finally obtain the motion track of institute's detected object on virtual touch screen.Shown in Figure 13 (a) is the schematic diagram of the motion track of the detected object that obtains of a kind of the present invention of employing on virtual touch screen.
Obviously, the motion track of this detected object as shown in Figure 13 (a) tentatively obtaining on virtual touch screen seems comparatively mixed and disorderly.This track also needs to carry out smoothing processing, to obtain level and smooth object motion track.Shown in Figure 13 (b) is the schematic diagram of the object motion track after smoothing processing.Shown in Figure 12 is the method that the point sequence of the motion track of a kind of detected object that a strip adoption the present invention is obtained on virtual touch screen carries out smoothing processing.
Point sequence smoothing processing is exactly that the coordinate of the point in this sequence is optimized so that point sequence is level and smooth.As shown in figure 12, input forms the original point sequence of a track
(n is positive integer) is as the first round input of iteration, the output that patch is followed the tracks of.In accompanying drawing 12, original point sequence
for the first row that sits up.Then use formula below according to the result from last round of iteration, to calculate the sequence of next round iteration
Wherein
be the point in point sequence, k is iteration sign, and n is point sequence sign, the radix that m is iteration point.
Repeat this iterative computation, until reach predetermined iteration threshold.In an embodiment of the present invention, m parameter can be 3-7, is set in an embodiment of the present invention 3, and this means this, and each next stage point has 3 some iteration of upper level to obtain.This iteration threshold is 3.
By above-mentioned iterative computation, finally obtain is the object motion track after smoothing processing as shown in Figure 13 (b).
Herein, in this manual, the processing of being carried out by computing machine according to program does not need according to carrying out with time series as the order of flowchart text.That is the processing of, being carried out by computing machine according to program comprises processing parallel or that carry out separately (for example parallel processing and target are processed).
Similarly, program can above be carried out at a computing machine (processor), or can be carried out by many computer distribution types.In addition, program can be transferred at the remote computer of executive routine there.
Will be understood by those skilled in the art that, according to designing requirement and other factors, as long as it falls in the scope of claims or its equivalent, can occur various modifications, combination, part combination and substitute.
Claims (7)
1. in virtual touch screen system, follow the tracks of a method for a plurality of objects, comprising:
Project image onto in a projection surface;
Obtain continuously the image of the environment of described projection surface;
From obtained every two field picture, detect candidate's patch of at least one object that is positioned at the front preset distance of described projection surface; And
Input an existing point sequence, and candidate's patch that searching closes on apart from this existing point sequence from detected candidate's patch, and find for existing point sequence the centroid point that distance has the nearest candidate's patch of point sequence, and the new patch nearest apart from this existing point sequence is included into this existing point sequence, wherein
The described step of finding the candidate's patch closing on apart from this existing point sequence from detected candidate's patch comprises:
Input, as new candidate's plate of patch, is calculated the distance between the new patch of inputting and the existing point sequence of inputting;
In the situation that the distance between calculated the new patch of inputting and the existing point sequence inputted is less than a predetermined distance threshold Td, the candidate that inputted new patch is inserted into the existing track that belongs to inputted closes in new patch list, otherwise, be confirmed whether all new patches to have been carried out to above-mentioned steps;
In the situation that determine that the size that the candidate of the existing point sequence belong to inputted closes on new patch list is less than a predetermined size threshold value Tsize, be confirmed whether all new patches to have been carried out to above-mentioned steps, otherwise, the candidate that deletion belongs to inputted existing point sequence closes on after in new patch list and inputted existing point sequence new patch farthest, is confirmed whether all new patches to have been carried out to above-mentioned steps;
In the situation that confirming all new patches to have been carried out to above-mentioned steps, if belong to the candidate of inputted existing point sequence, close on new patch list for empty, the candidate from described list closes in new patch and finds out the new patch nearest with inputted existing point sequence as the recently new patch of inputted existing point sequence.
2. method of following the tracks of a plurality of objects in virtual touch screen system according to claim 1, also comprises:
For any one existing point sequence, if there is no any new patch closing on it, provides the notice that this existing point sequence finished and deletes this existing point sequence.
3. method of following the tracks of a plurality of objects in virtual touch screen system according to claim 1 and 2, also comprises:
For any one the new patch in a plurality of new patch in an inputted two field picture, if there is no any existing point sequence closing on it, providing this new patch is the notice of the starting point of a new point sequence, and creates a new point sequence.
4. method of following the tracks of a plurality of objects in virtual touch screen system according to claim 3, the described step of finding apart from the centroid point of the nearest candidate's patch of this existing point sequence for each existing point sequence also comprises:
In the situation that the new patch that the existing point sequence that a plurality of new patch from an inputted two field picture does not find distance to input closes on provides the notice that inputted existing point sequence will be deleted;
A plurality of new patch the two field picture from inputted has found the new patch closing on apart from inputted existing point sequence, and the new patch closing on finding is only the new patch closing on of inputted existing point sequence, determine that this new patch belongs to inputted existing point sequence; And
A plurality of new patch the two field picture from inputted has found the new patch closing on apart from inputted existing point sequence, and the new patch closing on finding or the new patch closing on of other existing point sequences, if the new patch closing on finding with the distance of inputted existing point sequence than little with the distance of other existing point sequences, determine that this new patch belongs to the new patch nearest with inputted existing point sequence, otherwise, provide the notice that inputted existing point sequence will be deleted;
Examine and whether all existing sequence of points have been carried out to above-mentioned steps.
5. method of following the tracks of a plurality of objects in virtual touch screen system according to claim 1, also comprises:
To the point sequence of final acquisition, adopt following formula to carry out coordinate optimizing to carry out smoothing processing,
Wherein
be the point in point sequence, k is iteration sign, and n is point sequence sign, the radix that m is iteration point.
6. in virtual touch screen system, follow the tracks of a system for a plurality of objects, comprising:
Projector, projects image onto in a projection surface;
Depth cameras, obtains the image of the environment of described projection surface continuously;
Subject detecting unit, from obtained every two field picture, detects candidate's patch of at least one object that is positioned at the front preset distance of described projection surface;
Tracking cell, comprising:
Receiver module: it receives an existing point sequence and the new candidate's plate of patch of conduct,
Patch is found module recently, calculates the distance between the new patch of inputting and the existing point sequence of inputting; In the situation that the distance between calculated the new patch of inputting and the existing point sequence inputted is less than a predetermined distance threshold Td, the candidate that inputted new patch is inserted into the existing track that belongs to inputted closes in new patch list, otherwise, be confirmed whether all new patches to have been carried out to above-mentioned steps; In the situation that determine that the size that the candidate of the existing point sequence belong to inputted closes on new patch list is less than a predetermined size threshold value Tsize, be confirmed whether all new patches to have been carried out to above-mentioned steps, otherwise, the candidate that deletion belongs to inputted existing point sequence closes on after in new patch list and inputted existing point sequence new patch farthest, is confirmed whether all new patches to have been carried out to above-mentioned steps; In the situation that confirming all new patches to have been carried out to above-mentioned steps, if belong to the candidate of inputted existing point sequence, close on new patch list for empty, the candidate from described list closes in new patch and finds out the new patch nearest with inputted existing point sequence as the recently new patch of inputted existing point sequence, and find for existing point sequence the centroid point that distance has the nearest candidate's patch of point sequence, and the new patch nearest apart from this existing point sequence is included into this existing point sequence.
7. system of following the tracks of a plurality of objects in virtual touch screen system according to claim 6, also comprises: smoothing processing unit, and to the point sequence of final acquisition, adopt following formula to carry out coordinate optimizing to carry out smoothing processing,
Wherein
be the point in point sequence, k is iteration sign, and n is point sequence sign, the radix that m is iteration point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010615896.6A CN102541417B (en) | 2010-12-30 | 2010-12-30 | Multi-object tracking method and system in virtual touch screen system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010615896.6A CN102541417B (en) | 2010-12-30 | 2010-12-30 | Multi-object tracking method and system in virtual touch screen system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102541417A CN102541417A (en) | 2012-07-04 |
| CN102541417B true CN102541417B (en) | 2014-02-26 |
Family
ID=46348423
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010615896.6A Expired - Fee Related CN102541417B (en) | 2010-12-30 | 2010-12-30 | Multi-object tracking method and system in virtual touch screen system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102541417B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103677240B (en) * | 2012-09-24 | 2017-08-25 | 株式会社理光 | Virtual touch exchange method and virtual touch interactive device |
| CN104867158B (en) * | 2015-06-03 | 2017-09-29 | 武汉理工大学 | Indoor above water craft Precise Position System and method based on monocular vision |
| CN106022266A (en) * | 2016-05-20 | 2016-10-12 | 北京格灵深瞳信息技术有限公司 | Target tracking method and target tracking apparatus |
| CN106648235A (en) * | 2016-12-20 | 2017-05-10 | 美的智慧家居科技有限公司 | Projection touch system and integrated cabinet |
| US11036333B2 (en) * | 2019-03-26 | 2021-06-15 | Jacky Cho | Distinguishing and tracking multiple objects when placed on capacitive touchscreen |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840062A (en) * | 2009-08-21 | 2010-09-22 | 深圳先进技术研究院 | Interactive projection system and interactive method |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB9025797D0 (en) * | 1990-11-28 | 1991-01-09 | Hitachi Europ Ltd | Motion target tracking system |
| US6661450B2 (en) * | 1999-12-03 | 2003-12-09 | Fuji Photo Optical Co., Ltd. | Automatic following device |
-
2010
- 2010-12-30 CN CN201010615896.6A patent/CN102541417B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840062A (en) * | 2009-08-21 | 2010-09-22 | 深圳先进技术研究院 | Interactive projection system and interactive method |
Non-Patent Citations (5)
| Title |
|---|
| JP特开平4-260979A 1992.09.16 |
| 一种基于特征的多目标跟踪算法;付晓薇 等;《计算机工程与应用》;20041211(第35期);第2.2节，表1 * |
| 付晓薇 等.一种基于特征的多目标跟踪算法.《计算机工程与应用》.2004,(第35期),第2.2节，表1. |
| 张嫣 等.运动图像序列中多目标跟踪的研究与实现.《计算机应用研究》.2002,(第1期),全文. |
| 运动图像序列中多目标跟踪的研究与实现;张嫣 等;《计算机应用研究》;20020128(第1期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102541417A (en) | 2012-07-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102841733B (en) | Virtual touch screen system and method for automatically switching interaction modes | |
| US10638117B2 (en) | Method and apparatus for gross-level user and input detection using similar or dissimilar camera pair | |
| US9117274B2 (en) | System and method for interactive markerless paper documents in 3D space with mobile cameras and projectors | |
| US9405182B2 (en) | Image processing device and image processing method | |
| EP2790089A1 (en) | Portable device and method for providing non-contact interface | |
| US20120274550A1 (en) | Gesture mapping for display device | |
| CN102566827A (en) | Method and system for detecting object in virtual touch screen system | |
| CN103294401A (en) | Icon processing method and device for electronic instrument with touch screen | |
| CN102541417B (en) | Multi-object tracking method and system in virtual touch screen system | |
| EP3284078A1 (en) | Augmented interface authoring | |
| CN103914143A (en) | Control method of electronic device | |
| US20120319945A1 (en) | System and method for reporting data in a computer vision system | |
| JP2016091457A (en) | Input device, fingertip-position detection method, and computer program for fingertip-position detection | |
| Katz et al. | A multi-touch surface using multiple cameras | |
| CN102799344B (en) | Virtual touch screen system and method | |
| KR101461145B1 (en) | System for Controlling of Event by Using Depth Information | |
| CN106796462A (en) | Determine the position of input object | |
| CN109871178A (en) | A kind of virtual touch screen system based on image recognition | |
| WO2022123929A1 (en) | Information processing device and information processing method | |
| CN116301551A (en) | Touch identification method, touch identification device, electronic equipment and medium | |
| KR101289883B1 (en) | System and method for generating mask image applied in each threshold in region | |
| US20130187893A1 (en) | Entering a command | |
| JP2018185563A (en) | Information processing apparatus, information processing method, computer program and storage medium | |
| Cheng et al. | Fingertip-based interactive projector–camera system | |
| CN104423560A (en) | Information processing method and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20140226 Termination date: 20151230 |
|
| EXPY | Termination of patent right or utility model |