US20120129605A1 - Method and device for detecting and tracking non-rigid objects in movement, in real time, in a video stream, enabling a user to interact with a computer system - Google Patents

Method and device for detecting and tracking non-rigid objects in movement, in real time, in a video stream, enabling a user to interact with a computer system Download PDF

Info

Publication number
US20120129605A1
US20120129605A1 US13/300,509 US201113300509A US2012129605A1 US 20120129605 A1 US20120129605 A1 US 20120129605A1 US 201113300509 A US201113300509 A US 201113300509A US 2012129605 A1 US2012129605 A1 US 2012129605A1
Authority
US
United States
Prior art keywords
interest
image
points
region
movement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/300,509
Other languages
English (en)
Inventor
Nicolas Livet
Thomas Pasquier
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Total Immersion
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Total Immersion filed Critical Total Immersion
Assigned to TOTAL IMMERSION reassignment TOTAL IMMERSION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Pasquier, Thomas, LIVET, NICOLAS
Publication of US20120129605A1 publication Critical patent/US20120129605A1/en
Assigned to QUALCOMM CONNECTED EXPERIENCES, INC. reassignment QUALCOMM CONNECTED EXPERIENCES, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: TOTAL IMMERSION, SA
Assigned to QUALCOMM INCORPORATED reassignment QUALCOMM INCORPORATED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: QUALCOMM CONNECTED EXPERIENCES, INC.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/002—Specific input/output arrangements not covered by G06F3/01 - G06F3/16
    • G06F3/005—Input arrangements through a video camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present invention concerns the detection of objects by the analysis of images, and their tracking, in a video stream representing a sequence of images and more particularly a method and a device for detecting and tracking non-rigid objects in movement, in real time, in a video stream, enabling a user to interact with a computer system.
  • Augmented reality in particular seeks to insert one or more virtual objects in images of a video stream representing a sequence of images.
  • the position and orientation of those virtual objects may be determined by data that are external to the scene represented by the images, for example coordinates obtained directly from a game scenario, or by data linked to certain elements of that scene, for example coordinates of a particular point in the scene such as the hand of a player.
  • data linked to certain elements of that scene for example coordinates of a particular point in the scene such as the hand of a player.
  • the operations of tracking elements and embedding virtual objects in the real images may be executed by different computers or by the same computer.
  • OCETRE ODETRE
  • HLONICS HOLONICS
  • An example of such approaches is in particular described in the document entitled “ Holographic and action capture techniques ”, T. Rodriguez, A. Cabo de Leon, B. Uzzan, N. Livet, E. Boyer, F. Geffray, T. Balogh, Z. Megyesi and A. Barsi, August 2007, SIGGRAPH '07, ACM SIGGRAPH 2007, Emerging Technologies. It is to be noted that these applications may enable the geometry of the real scene to be reproduced but do not currently enable precise movements to be identified. Furthermore, to meet real time constraints, it is necessary to set up complex and costly hardware architectures.
  • an image is typically captured from a webcam type video camera connected to a computer or to a console.
  • this image is generally analyzed by an object tracking algorithm, also referred to as blobs tracking, to compute in real time the contours of certain elements of the user who is moving in the image by using, in particular, an optical flow algorithm.
  • object tracking algorithm also referred to as blobs tracking
  • the position of those shapes in the image enables certain parts of the displayed image to be modified or deformed. This solution thus enables the disturbance in a zone of the image to be located in two degrees of freedom.
  • the invention enables at least one of the problems set forth above to be solved.
  • the number of degrees of freedom defining the movements of each tracked object may be set for each region of interest.
  • said step of determining a movement comprises a step of determining and matching at least one pair of points of interest in said at least one first and second images, at least one point of said at least one pair of points of interest belonging to said mask of interest.
  • Said transformation preferably implements a weighting function based on a distance between two points of interest from the same pairs of points of interest of said plurality of pairs of points of interest in order to improve the estimation of the movement of the tracked object.
  • Said step of comparing said at least one first and second regions of interest comprises a step of performing subtraction, point by point, of values of corresponding points of said at least one first and second regions of interest and a step of comparing a result of said subtraction to a predetermined threshold.
  • the method further comprises a step of estimating at least one modified second region of interest in said at least one second image, said at least one modified second region of interest of said at least one second image being estimated according to said at least one first region of interest of said at least one first image and of said at least one second region of interest of said at least one second image.
  • the method according to the invention thus makes it possible to anticipate the processing of the following image for the object tracking.
  • Said estimation of said at least one modified second region of interest of said at least one second image for example implements an object tracking algorithm of KLT type.
  • a scale factor may, for example, characterize a mouse click.
  • the movements of at least two objects situated in the field of said image sensor are determined, whether or not said predetermined action is triggered being determined according to a combination of the movements associated with said at least two objects. It is thus possible to determine a movement of an object on the basis of movements of other objects, in particular other objects subjected to constraints of relative position.
  • the invention is also directed to a computer program comprising instructions adapted to the implementation of each of the steps of the method described earlier when said program is executed on a computer as well as a device comprising means adapted to the implementation of each of the steps of the method described earlier.
  • a computer program comprising instructions adapted to the implementation of each of the steps of the method described earlier when said program is executed on a computer as well as a device comprising means adapted to the implementation of each of the steps of the method described earlier.
  • FIG. 2 comprising FIGS. 2 a to 2 d , illustrates examples of variation in a region of interest of an image with the corresponding region of interest of a following image;
  • FIG. 3 is a diagrammatic illustration of the determination of a movement of an object of which at least one part is represented in a region and in a mask of interest of two consecutive images;
  • FIG. 4 is a diagrammatic illustration of certain steps implemented in accordance with the invention to identify, in continuous operation, variations in position of objects between two consecutive (or close) images of a sequence of images;
  • FIG. 5 illustrates certain aspects of the invention when four parameters characterize a movement of an object tracked in consecutive (or close) images of a sequence of images;
  • FIG. 6 illustrates an example of implementation of the invention in the context of a driving simulation game in which two regions of interest enable the tracking of a user's hands in real time, characterizing a vehicle steering wheel movement, in a sequence of images;
  • the regions of interest are, preferably, defined as two-dimensional shapes, in an image. These shapes are, for example, rectangles or circles. They are preferably constant and predetermined.
  • the regions of interest may be characterized by points of interest, that is to say singular points, such as points having a high luminance gradient, and the initial position of the regions of interest may be predetermined, be determined by a user, by an event such as the appearance of a shape or a color or according to predefined features, for example using key images. These regions may also be moved depending on the movement of tracked objects or have a fixed position and orientation in the image.
  • the use of several regions of interest makes it possible, for example, to observe several concomitant interactions of a user (a region of interest may correspond to each of his hands) and/or several concomitant interactions of several users.
  • the points of interest are used in order to find the variation of the regions of interest, in a stream of images, from one image to a following (or close) image, according to techniques of tracking points of interest based, for example, on algorithms known under the name of FAST, for the detection, and KLT (initials of Kanade, Lucas and Tomasi), for tracking in the following image.
  • the points of interest of a region of interest may vary over the images analyzed, in particular according to the distortion of the objects tracked and their movements which may mask parts of the scene represented in the images and/or make parts of those objects leave the zones of interest.
  • the objects whose movements may create an interaction are tracked in each region of interest according to a mechanism for tracking points of interest in masks defined in the regions of interest.
  • FIG. 1 comprising FIGS. 1 a and 1 b , illustrates two successive images of a stream of images that may be used to determine the movement of objects and the interaction of a user.
  • Image 100 - 2 of FIG. 1 b represents an image following the image 100 - 1 of FIG. 1 a in a sequence of images. It is possible to define, in the image 100 - 2 , a region of interest 105 - 2 , corresponding to the position and to the dimensions of the region of interest 105 - 1 defined in the preceding image, in which disturbances may be estimated.
  • the region of interest 105 - 1 is thus compared to the region of interest 105 - 2 of FIG. 1 b , for example by subtracting those image parts one from another, pixel by pixel (pixel being an acronym for PICture ELement), in order to extract therefrom a map of pixels that are considered to be in movement. These pixels in movement constitute a mask of pixels of interest (presented in FIG. 2 ).
  • Points of interest may be determined in the image 100 - 1 , in particular in the region of interest 105 - 1 , according to standard algorithms for image analysis. These points of interest may be advantageously detected at positions in the region of interest which belong to the mask of pixels of interest.
  • the points of interest 110 defined in the region of interest 105 - 1 are tracked in the image 100 - 2 , preferably in the region of interest 105 - 2 , for example using the KLT tracking principles by comparing portions of the images 100 - 1 and 100 - 2 that are associated with the neighborhoods of the points of interest.
  • the determination of points of interest in an image is, preferably, limited to the zone corresponding to the corresponding region of interest as located on the current image or to a zone comprising all or part thereof when a mask of interest of pixels in movement is defined in that region of interest.
  • estimation is made of information characterizing the relative positions and orientations of the objects to track (for example the hand referenced 120 - 1 in FIG. 1 a ) in relation to a reference linked to the video camera from which the images come.
  • information is, for example two-dimensional position information (x, y), orientation information ( ⁇ ) and information on distance to the video camera, that is to say scale(s) of the objects to track.
  • FIG. 2 c illustrates the image 215 resulting from the comparison of the regions of interest 200 - 1 and 200 - 2 .
  • the black part forming a mask of interest, represents the pixels whose difference is greater than a predetermined threshold whereas the white part represents the pixels whose difference is less than that threshold.
  • the black part comprises in particular the part referenced 220 corresponding to the difference in position of the hand 205 between the regions of interest 200 - 1 and 200 - 2 . It also comprises the part 225 corresponding to the difference in position of the object 210 between those regions of interest.
  • the part 230 corresponds to the part of the hand 205 present in both these regions of interest.
  • a skeletonizing step making it possible in particular to eliminate adjoining movements such as the movement referenced 225 is, preferably, carried out before analyzing the movement of the points of interest belonging to the mask of interest.
  • This skeletonizing step may take the form of a morphological processing operation such as for example operations of opening or closing applied to the mask of interest.
  • the mask of interest obtained is modified in order to eliminate the parts situated around the points of interest identified recursively between the image from which is extracted the region of interest 200 - 1 and the image preceding it.
  • FIG. 2 d thus illustrates the mask of interest represented in FIG. 2 c , here referenced 235 , to which the parts 240 situated around the points of interest identified by 245 have been eliminated.
  • the parts 240 are, for example, circular. They are of predetermined radius here.
  • the mask of interest 235 thus has cropped from it zones in which are situated already detected points of interest and where it is thus not necessary to detect new ones.
  • this modified mask of interest 235 has just excluded a part of the mask of interest 220 in order to avoid the accumulation of points of interest in the same zone of the region of interest.
  • the mask of interest 235 may be used to identify points of interest whose movements may be analyzed in order to trigger, the case arising, a particular action.
  • FIG. 3 is again a diagrammatic illustration of the determination of a movement of an object of which at least one part is represented in a region and a mask of interest of two consecutive (or close) images.
  • the image 300 here corresponds to the mask of interest resulting from the comparison of the regions of interest 200 - 1 and 200 - 2 as described with reference to FIG. 2 d .
  • a skeletonizing step has been carried out to eliminate the disturbances (in particular the disturbance 225 ).
  • the image 300 comprises a mask 305 which may be used for identifying new points of interest whose movements characterize the movement of objects in that region of interest.
  • Reference 310 - 1 designates this point of interest according to its position in the region of interest 200 - 1
  • reference 310 - 2 designates that point of interest according to its position in the region of interest 200 - 2 .
  • FIG. 4 is a diagrammatic illustration of certain steps implemented in accordance with the invention to identify, in continuous operation, variations in arrangement of objects between two consecutive (or close) images of a sequence of images.
  • the images here are acquired via an image sensor such as a video camera, in particular a video camera of webcam type, connected to a computer system implementing the method described here.
  • an image sensor such as a video camera, in particular a video camera of webcam type
  • a first step of initializing is executed.
  • An object of this step is in particular to define features of at least one region of interest, for example a shape, a size and an initial position.
  • a region of interest not to be defined in an initial state, the system being on standby for a triggering event, for example a particular movement of the user facing the video camera (the moving pixels in the image are analyzed in search for a particular movement), the location of a particular color such as the color of skin or the recognition of a particular predetermined object whose position defines that of the region of interest.
  • a triggering event for example a particular movement of the user facing the video camera (the moving pixels in the image are analyzed in search for a particular movement), the location of a particular color such as the color of skin or the recognition of a particular predetermined object whose position defines that of the region of interest.
  • the size and the shape of the region of interest may be predefined or be determined according to features of the detected event.
  • the initializing step 410 may thus take several forms depending on the object to track in the image sequence and depending on the application implemented.
  • the initial position of the region of interest is predetermined (off-line determination) and the tracking algorithm is on standby for a disturbance.
  • step 415 a region of interest whose features have been determined beforehand (on initialization or in the preceding image) is positioned in the current image to extract the corresponding image part. If the current image is the first image of the video stream to be processed, that image becomes the preceding image, a new image current is acquired and step 415 is repeated.
  • step 460 is only carried out if there are validated points of interest. As indicated earlier, this step consists in eliminating zones from the mask created, for example disks of a predetermined diameter, around points of interest validated beforehand.
  • Points of interest are then searched for in the region of the preceding image corresponding to the mask of interest so defined (step 435 ), the mask of interest here being the mask of interest created at step 430 or the mask of interest created at step 430 and modified during step 460 .
  • the search for points of interest is, for example, limited to the detection of twenty points of interest. Naturally, this number may be different and may be estimated according to the size of the mask of interest.
  • This search is advantageously carried out with the algorithm known by the name FAST.
  • FAST the algorithm known by the name FAST.
  • a Bresenham circle for example with a perimeter of 16 pixels, is constructed around each pixel of the image. If k contiguous pixels (k typically having a value of 9, 10, 11 or 12) contained in that circle all have either greater intensity than the central pixel, or all have lower intensity than the central pixel, that central pixel is considered as a point of interest. It is also possible to identify points of interest with an approach based on image gradients as provided in the approach known by the name of Harris points detection.
  • the points of interest detected in the preceding image according to the mask of interest as well as, where applicable, the points of interest detected and validated beforehand are used to identify the corresponding points of interest in the current image.
  • a search for corresponding points of interest in the current image is thus carried out (step 440 ), preferably using a method known under the name of optical flow.
  • This technique gives better robustness when the image is blurred, in particular thanks to the use of pyramids of images smoothed by a Gaussian filter. This is for example the approach implemented by Lucas, Kanade and Tomasi in the algorithm known under the name KLT.
  • movement parameters are estimated for objects tracked in the region of interest of the preceding image relative to the region of interest of the current image (step 445 ).
  • Such parameters also termed degrees of freedom, comprise, for example, a parameter of translation along the x-axis, a parameter of translation along the y-axis, a rotation parameter and/or a scale parameter, the transformation making a set of bi-directional points pass from one plane to another, grouping together these four parameters, being termed the similarity.
  • NLSE Nonlinear Least Squares Error
  • Gauss-Newton method This method is directed to minimizing a re-projection error over the set of the tracked points of interest.
  • NLSE Nonlinear Least Squares Error
  • Gauss-Newton method Gauss-Newton method
  • a threshold typically expressed in pixels and having a predetermined value, is advantageously used to authorize a certain margin of error between the theoretical position of the point in the current image (obtained by applying the parameters of step 445 ) and its real position (obtained by the tracking method of step 440 ).
  • the valid points of interest here referenced 455 , are considered as belonging to an object whose movement is tracked whereas the non-valid points (also termed outliers), are considered as belonging to the image background or to portions of an object which are not visible in the image.
  • the valid points of interest are tracked in the following image and are used to modify the mask of interest created by comparison of a region of interest of the current image with the corresponding region of interest of the following image (step 460 ) in order to exclude from the portions of mask, pixels in movement between the current and following images as described with reference to FIG. 2 d .
  • This modified mask of interest makes it possible to eliminate portions of images in which points of interest are recursively tracked.
  • the valid points of interest are thus kept for several processing operations on successive images and in particular enable stabilization of the tracking of objects.
  • the new region of interest (or modified region of interest) which is used for processing the current image and the following image is then estimated thanks to the previously estimated degrees of freedom (step 445 ). For example, if the degrees of freedom are x and y translations, the new position of the region of interest is estimated according to the previous position of the region of interest, using those two items of information. If a change (or changes) of scale is estimated and considered in this step, it is possible, according to the scenario considered, also to modify the size of the new region of interest which is used in the current and following images of the video stream.
  • the estimation of a change (or changes) of scale is used for detecting the triggering of an action in similar manner to the click of a mouse.
  • changes of orientation particularly those around the viewing axis of the video camera (referred to as roll) in order, for example, to enable the rotation of a virtual element displayed in a scene or to control a button of “potentiometer” type in order, for example, to adjust a volume of sound of an application.
  • the algorithm preferably returns to the initializing step. Furthermore, loss of tracking leading to the initializing step being re-executed may be identified by measuring the movements of a user. Thus, it may be decided to reinitialize the method when those movements are stable or non-existent for a predetermined period or when a tracked object leaves the field of view of the image sensor.
  • FIG. 5 illustrates more precisely certain aspects of the invention when four parameters characterize a movement of an object tracked in consecutive (or close) images of a sequence of images; These four parameters here are a translation denoted (T x , T y ), a rotation denoted ⁇ around the optical axis of the image sensor and a scale factor denoted s. These four parameters represent a similarity which is the transformation enabling a point M to be transformed from a plane to a point M′.
  • X M′ s ⁇ ( X M ⁇ X O ) ⁇ cos( ⁇ ) ⁇ s ⁇ ( Y M ⁇ Y O ) ⁇ sin( ⁇ )+ T x +X O
  • Y M′ s ⁇ ( X M ⁇ X O ) ⁇ sin( ⁇ )+ s ⁇ ( Y M ⁇ Y O ) ⁇ cos( ⁇ )+ T y +Y O
  • the points M s and M s ⁇ respectively represent the transformation of the point M according to the change in scale s and the change of scale s combined with the rotation ⁇ , respectively.
  • the partial derivatives of each point considered that is to say the movements associated with each of those points, are weighted according to the associated movement.
  • the points of interest moving the most have greater importance in the estimation of the parameters, which avoids the points of interest linked to the background disturbing the tracking of objects.
  • Y O′ Y O +W GC ⁇ ( Y GC ⁇ Y O )+ W T ⁇ T y
  • (X GC , Y GC ) represent the center of gravity of the points of interest in the current image and W GC represents the weight on the influence of the current center of gravity and W T the weight on the influence of the translation.
  • the parameter W GC is positively correlated here with the velocity of movement of the tracked object whereas the parameter W T may be fixed depending on the desired influence of the translation.
  • FIG. 6 comprising FIGS. 6 a , 6 b and 6 c , illustrates an example of implementation of the invention in the context of a driving simulation game in which two regions of interest enable the tracking of a user's hands in real time, characterizing a vehicle steering wheel movement, in a sequence of images.
  • FIG. 6 a is a pictorial presentation of the context of the game
  • FIG. 6 b represents the display of the game as perceived by a user
  • FIG. 6 c illustrates the estimation of the movement parameters, or degrees of freedom, of the tracked objects to deduce therefrom a movement of a vehicle steering wheel.
  • FIG. 6 a comprises an image 600 extracted from the sequence of images provided by the image sensor used. The latter is placed facing the user, as if it were fastened to the windshield of the vehicle driven by the user.
  • This image 600 here contains a zone 605 comprising two circular regions of interest 610 and 615 associated with a steering wheel 620 drawn in overlay by computer graphics.
  • the image 600 also comprises elements of the real scene in which the user is situated.
  • the frame of reference Ow here corresponds to an overall frame of reference (“world” frame of reference)
  • the frame of reference Owh is a local frame of reference linked to the steering wheel 620
  • the frames of reference Oa 1 and Oa 2 are two local frames of reference linked to the regions of interest 610 and 615 .
  • the vectors Va 1 (Xva 1 , Yva 1 ) and Va 2 (Xva 2 , Yva 2 ) are the movement vectors resulting from the analysis of the movement of the user's hands in the regions of interest 610 and 615 , expressed in the frames of reference Oa 1 and Oa 2 , respectively.
  • ⁇ 1 and ⁇ 2 represent the rotation of the user's hands.
  • ⁇ 1 may be computed by the following relationship:
  • ⁇ 1 a tan2( Yva 1 wh, D/ 2)
  • ⁇ 2 may be computed in similar manner.
  • the new diameter D′ of the steering wheel is computed on the basis of its previous diameter D and on the basis of the movement of the user's hands (determined via the two regions of interest 610 and 615 ). It may be computed in the following manner:
  • the game scenario may in particular compute a corresponding computer graphics image.
  • FIG. 7 illustrates an example of a device which may be used to identify the movements of objects represented in images provided by a video camera and to trigger particular actions according to identified movements.
  • the device 700 is for example a mobile telephone of smartphone type, a personal digital assistant, a micro-computer or a workstation.
  • the device 700 preferably comprises a communication bus 702 to which are connected:
  • the device 700 may also have the following items:
  • the communication bus allows communication and interoperability between the different elements included in the device 700 or connected to it.
  • the representation of the bus is non-limiting and, in particular, the central processing unit may communicate instructions to any element of the device 700 directly or by means of another element of the device 700 .
  • the executable code of the programs can be received by the intermediary of the communication network 728 , via the interface 726 , in order to be stored in an identical fashion to that described previously.
  • program or programs may be loaded into one of the storage means of the device 700 before being executed.
  • the central processing unit 704 will control and direct the execution of the instructions or portions of software code of the program or programs according to the invention, these instructions being stored on the hard disk 720 or in the read-only memory 706 or in the other aforementioned storage elements.
  • the program or programs which are stored in a non-volatile memory for example the hard disk 720 or the read only memory 706 , are transferred into the random-access memory 708 , which then contains the executable code of the program or programs according to the invention, as well as registers for storing the variables and parameters necessary for implementation of the invention.
  • the communication apparatus comprising the device according to the invention can also be a programmed apparatus.
  • This apparatus then contains the code of the computer program or programs for example fixed in an application specific integrated circuit (ASIC).
  • ASIC application specific integrated circuit

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)
US13/300,509 2010-11-19 2011-11-18 Method and device for detecting and tracking non-rigid objects in movement, in real time, in a video stream, enabling a user to interact with a computer system Abandoned US20120129605A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR1059541 2010-11-19
FR1059541A FR2967804B1 (fr) 2010-11-19 2010-11-19 Procede et dispositif de detection et de suivi d'objets non rigides en mouvement, en temps reel, dans un flux video, permettant a un utilisateur d'interagir avec un systeme informatique

Publications (1)

Publication Number Publication Date
US20120129605A1 true US20120129605A1 (en) 2012-05-24

Family

ID=44168356

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/300,509 Abandoned US20120129605A1 (en) 2010-11-19 2011-11-18 Method and device for detecting and tracking non-rigid objects in movement, in real time, in a video stream, enabling a user to interact with a computer system

Country Status (5)

Country Link
US (1) US20120129605A1 (ko)
EP (1) EP2455916B1 (ko)
JP (1) JP5967904B2 (ko)
KR (1) KR20120054550A (ko)
FR (1) FR2967804B1 (ko)

Cited By (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100172573A1 (en) * 2009-01-07 2010-07-08 Michael Bailey Distinguishing Colors of Illuminated Objects Using Machine Vision
US20130308001A1 (en) * 2012-05-17 2013-11-21 Honeywell International Inc. Image stabilization devices, methods, and systems
US20140051513A1 (en) * 2012-05-14 2014-02-20 Fabrizio Polo Interactive augmented reality using a self-propelled device
US20140254870A1 (en) * 2013-03-11 2014-09-11 Lenovo (Singapore) Pte. Ltd. Method for recognizing motion gesture commands
US8933970B2 (en) 2012-09-11 2015-01-13 Longsand Limited Controlling an augmented reality object
US9011246B1 (en) * 2013-11-18 2015-04-21 Scott Kier Systems and methods for immersive backgrounds
WO2016106052A1 (en) * 2014-12-23 2016-06-30 Matthew Daniel Fuchs Augmented reality system and method of operation thereof
US9569661B2 (en) * 2015-05-21 2017-02-14 Futurewei Technologies, Inc. Apparatus and method for neck and shoulder landmark detection
US20170092009A1 (en) * 2012-05-14 2017-03-30 Sphero, Inc. Augmentation of elements in a data content
US20170206666A1 (en) * 2014-08-26 2017-07-20 Sony Corporation Information processing apparatus, information processing method, and program
US9766620B2 (en) 2011-01-05 2017-09-19 Sphero, Inc. Self-propelled device with actively engaged drive system
US9829882B2 (en) 2013-12-20 2017-11-28 Sphero, Inc. Self-propelled device with center of mass drive system
US9886032B2 (en) 2011-01-05 2018-02-06 Sphero, Inc. Self propelled device with magnetic coupling
US20180075660A1 (en) * 2016-09-15 2018-03-15 Thomson Licensing Method and device for blurring a virtual object in a video
US10022643B2 (en) 2011-01-05 2018-07-17 Sphero, Inc. Magnetically coupled accessory for a self-propelled device
US10056791B2 (en) 2012-07-13 2018-08-21 Sphero, Inc. Self-optimizing power transfer
US20180260619A1 (en) * 2017-03-09 2018-09-13 Fujitsu Limited Method of determining an amount, non-transitory computer-readable storage medium and information processing apparatus
US10168701B2 (en) 2011-01-05 2019-01-01 Sphero, Inc. Multi-purposed self-propelled device
US10192310B2 (en) 2012-05-14 2019-01-29 Sphero, Inc. Operating a computing device by detecting rounded objects in an image
US10248118B2 (en) 2011-01-05 2019-04-02 Sphero, Inc. Remotely controlling a self-propelled device in a virtualized environment
US10474921B2 (en) 2013-06-14 2019-11-12 Qualcomm Incorporated Tracker assisted image capture
US20190370979A1 (en) * 2015-12-30 2019-12-05 Deepak Kumar Poddar Feature point identification in sparse optical flow based tracking in a computer vision system
US10846838B2 (en) * 2016-11-25 2020-11-24 Nec Corporation Image generation device, image generation method, and storage medium storing program
US12001203B2 (en) 2022-02-14 2024-06-04 Sphero, Inc. Self propelled device with magnetic coupling

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
BE1029378B1 (fr) * 2021-05-05 2022-12-05 P³Lab Procede de traitement d'images

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080030459A1 (en) * 2004-06-30 2008-02-07 Tsutomu Kouno Information Processing Device For Controlling Object By Using Player Image And Object Control Method In The Information Processing Device

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR957353A (ko) 1950-02-18
FR752809A (fr) 1932-03-31 1933-09-30 Lorenz C Ag Dispositif pour diriger l'atterrissage des avions
US1966429A (en) 1932-04-21 1934-07-17 Buxton Inc Key case
BE395422A (ko) 1933-03-08
FR854382A (fr) 1938-12-30 1940-04-11 Gaine anti-fuite pour haute pression
FR902764A (fr) 1943-02-09 1945-09-12 Petit cadre pour diapositives
JPH05205052A (ja) * 1992-01-23 1993-08-13 Matsushita Electric Ind Co Ltd 自動追尾装置
EP1851749B1 (en) * 2005-01-21 2012-03-28 Qualcomm Incorporated Motion-based tracking
JP4635663B2 (ja) * 2005-03-15 2011-02-23 オムロン株式会社 画像処理システム、画像処理装置および方法、記録媒体、並びにプログラム
JP2008071172A (ja) * 2006-09-14 2008-03-27 Toshiba Corp 顔認証装置、顔認証方法および入退場管理装置

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080030459A1 (en) * 2004-06-30 2008-02-07 Tsutomu Kouno Information Processing Device For Controlling Object By Using Player Image And Object Control Method In The Information Processing Device

Cited By (50)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100172573A1 (en) * 2009-01-07 2010-07-08 Michael Bailey Distinguishing Colors of Illuminated Objects Using Machine Vision
US8320662B2 (en) * 2009-01-07 2012-11-27 National Instruments Corporation Distinguishing colors of illuminated objects using machine vision
US11630457B2 (en) 2011-01-05 2023-04-18 Sphero, Inc. Multi-purposed self-propelled device
US10022643B2 (en) 2011-01-05 2018-07-17 Sphero, Inc. Magnetically coupled accessory for a self-propelled device
US10248118B2 (en) 2011-01-05 2019-04-02 Sphero, Inc. Remotely controlling a self-propelled device in a virtualized environment
US10423155B2 (en) 2011-01-05 2019-09-24 Sphero, Inc. Self propelled device with magnetic coupling
US10168701B2 (en) 2011-01-05 2019-01-01 Sphero, Inc. Multi-purposed self-propelled device
US10678235B2 (en) 2011-01-05 2020-06-09 Sphero, Inc. Self-propelled device with actively engaged drive system
US9841758B2 (en) 2011-01-05 2017-12-12 Sphero, Inc. Orienting a user interface of a controller for operating a self-propelled device
US10281915B2 (en) 2011-01-05 2019-05-07 Sphero, Inc. Multi-purposed self-propelled device
US10012985B2 (en) 2011-01-05 2018-07-03 Sphero, Inc. Self-propelled device for interpreting input from a controller device
US11460837B2 (en) 2011-01-05 2022-10-04 Sphero, Inc. Self-propelled device with actively engaged drive system
US9766620B2 (en) 2011-01-05 2017-09-19 Sphero, Inc. Self-propelled device with actively engaged drive system
US9952590B2 (en) 2011-01-05 2018-04-24 Sphero, Inc. Self-propelled device implementing three-dimensional control
US9886032B2 (en) 2011-01-05 2018-02-06 Sphero, Inc. Self propelled device with magnetic coupling
US9836046B2 (en) 2011-01-05 2017-12-05 Adam Wilson System and method for controlling a self-propelled device using a dynamically configurable instruction library
US9827487B2 (en) * 2012-05-14 2017-11-28 Sphero, Inc. Interactive augmented reality using a self-propelled device
US10192310B2 (en) 2012-05-14 2019-01-29 Sphero, Inc. Operating a computing device by detecting rounded objects in an image
US20140051513A1 (en) * 2012-05-14 2014-02-20 Fabrizio Polo Interactive augmented reality using a self-propelled device
US20170092009A1 (en) * 2012-05-14 2017-03-30 Sphero, Inc. Augmentation of elements in a data content
US8854481B2 (en) * 2012-05-17 2014-10-07 Honeywell International Inc. Image stabilization devices, methods, and systems
US20130308001A1 (en) * 2012-05-17 2013-11-21 Honeywell International Inc. Image stabilization devices, methods, and systems
US10056791B2 (en) 2012-07-13 2018-08-21 Sphero, Inc. Self-optimizing power transfer
US8933970B2 (en) 2012-09-11 2015-01-13 Longsand Limited Controlling an augmented reality object
US20140254870A1 (en) * 2013-03-11 2014-09-11 Lenovo (Singapore) Pte. Ltd. Method for recognizing motion gesture commands
US11538232B2 (en) 2013-06-14 2022-12-27 Qualcomm Incorporated Tracker assisted image capture
US10474921B2 (en) 2013-06-14 2019-11-12 Qualcomm Incorporated Tracker assisted image capture
US9011246B1 (en) * 2013-11-18 2015-04-21 Scott Kier Systems and methods for immersive backgrounds
US9829882B2 (en) 2013-12-20 2017-11-28 Sphero, Inc. Self-propelled device with center of mass drive system
US11454963B2 (en) 2013-12-20 2022-09-27 Sphero, Inc. Self-propelled device with center of mass drive system
US10620622B2 (en) 2013-12-20 2020-04-14 Sphero, Inc. Self-propelled device with center of mass drive system
US10169880B2 (en) * 2014-08-26 2019-01-01 Sony Corporation Information processing apparatus, information processing method, and program
US20170206666A1 (en) * 2014-08-26 2017-07-20 Sony Corporation Information processing apparatus, information processing method, and program
US10335677B2 (en) 2014-12-23 2019-07-02 Matthew Daniel Fuchs Augmented reality system with agent device for viewing persistent content and method of operation thereof
US11633667B2 (en) 2014-12-23 2023-04-25 Matthew Daniel Fuchs Augmented reality system and method of operation thereof
WO2016106052A1 (en) * 2014-12-23 2016-06-30 Matthew Daniel Fuchs Augmented reality system and method of operation thereof
US11040276B2 (en) 2014-12-23 2021-06-22 Matthew Daniel Fuchs Augmented reality system and method of operation thereof
US11433297B2 (en) 2014-12-23 2022-09-06 Matthew Daniel Fuchs Augmented reality system and method of operation thereof
US9569661B2 (en) * 2015-05-21 2017-02-14 Futurewei Technologies, Inc. Apparatus and method for neck and shoulder landmark detection
US11915431B2 (en) * 2015-12-30 2024-02-27 Texas Instruments Incorporated Feature point identification in sparse optical flow based tracking in a computer vision system
US20190370979A1 (en) * 2015-12-30 2019-12-05 Deepak Kumar Poddar Feature point identification in sparse optical flow based tracking in a computer vision system
US10825249B2 (en) * 2016-09-15 2020-11-03 Interdigital Ce Patent Holdings Method and device for blurring a virtual object in a video
US20180075660A1 (en) * 2016-09-15 2018-03-15 Thomson Licensing Method and device for blurring a virtual object in a video
US10878549B2 (en) * 2016-11-25 2020-12-29 Nec Corporation Image generation device, image generation method, and storage medium storing program
US11620739B2 (en) 2016-11-25 2023-04-04 Nec Corporation Image generation device, image generation method, and storage medium storing program
US10846838B2 (en) * 2016-11-25 2020-11-24 Nec Corporation Image generation device, image generation method, and storage medium storing program
US11989859B2 (en) 2016-11-25 2024-05-21 Nec Corporation Image generation device, image generation method, and storage medium storing program
US10706272B2 (en) * 2017-03-09 2020-07-07 Fujitsu Limited Method of determining an amount, non-transitory computer-readable storage medium and information processing apparatus
US20180260619A1 (en) * 2017-03-09 2018-09-13 Fujitsu Limited Method of determining an amount, non-transitory computer-readable storage medium and information processing apparatus
US12001203B2 (en) 2022-02-14 2024-06-04 Sphero, Inc. Self propelled device with magnetic coupling

Also Published As

Publication number Publication date
KR20120054550A (ko) 2012-05-30
FR2967804A1 (fr) 2012-05-25
JP2012113714A (ja) 2012-06-14
JP5967904B2 (ja) 2016-08-10
EP2455916B1 (fr) 2017-08-23
EP2455916A1 (fr) 2012-05-23
FR2967804B1 (fr) 2013-01-04

Similar Documents

Publication Publication Date Title
US20120129605A1 (en) Method and device for detecting and tracking non-rigid objects in movement, in real time, in a video stream, enabling a user to interact with a computer system
Habermann et al. Livecap: Real-time human performance capture from monocular video
US20100002909A1 (en) Method and device for detecting in real time interactions between a user and an augmented reality scene
US20210001885A1 (en) Method for predicting direction of movement of target object, vehicle control method, and device
US9489739B2 (en) Scene analysis for improved eye tracking
US11158121B1 (en) Systems and methods for generating accurate and realistic clothing models with wrinkles
US6697072B2 (en) Method and system for controlling an avatar using computer vision
US9076257B2 (en) Rendering augmented reality based on foreground object
KR100974900B1 (ko) 동적 임계값을 이용한 마커 인식 장치 및 방법
Davis et al. Real-time motion template gradients using Intel CVLib
WO2015026902A1 (en) Multi-tracker object tracking
AU2013242830A1 (en) A method for improving tracking in crowded situations using rival compensation
Elhayek et al. Fully automatic multi-person human motion capture for vr applications
CN112088348A (zh) 经由头部姿态远程控制显示设备的方法、系统和计算机程序
Matikainen et al. Prop-free pointing detection in dynamic cluttered environments
Amri et al. A robust framework for joint background/foreground segmentation of complex video scenes filmed with freely moving camera
Fan et al. A feature-based object tracking approach for realtime image processing on mobile devices
Chun et al. 3D face pose estimation by a robust real time tracking of facial features
Dornaika et al. Real time 3D face and facial feature tracking
Fiore et al. Towards achieving robust video selfavatars under flexible environment conditions
Habermann et al. Reticam: Real-time human performance capture from monocular video
Kitanovski et al. Augmented reality mirror for virtual facial alterations
Habermann et al. Nrst: Non-rigid surface tracking from monocular video
Huo et al. Multiple people tracking and pose estimation with occlusion estimation
Song et al. Real-time single camera natural user interface engine development

Legal Events

Date Code Title Description
AS Assignment

Owner name: TOTAL IMMERSION, FRANCE

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LIVET, NICOLAS;PASQUIER, THOMAS;SIGNING DATES FROM 20111104 TO 20111106;REEL/FRAME:027311/0572

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION

AS Assignment

Owner name: QUALCOMM CONNECTED EXPERIENCES, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TOTAL IMMERSION, SA;REEL/FRAME:034260/0297

Effective date: 20141120

AS Assignment

Owner name: QUALCOMM INCORPORATED, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:QUALCOMM CONNECTED EXPERIENCES, INC.;REEL/FRAME:038689/0718

Effective date: 20160523