AU2012203097B2 - Segment tracking in motion picture - Google Patents

Abstract

There is provided a method, comprising: applying a plurality of marking materials to different locations on a 5 surface, each of the marking materials including a uniquely known pattern; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of uniquely known patterns covering the surface; deriving position and rotational orientation information for a 10 portion of the different locations on the surface by analyzing position and rotational orientation information of those uniquely known patterns which are identifiable in each image frame of the sequence; wherein the rotational orientation information for a portion of the different 15 locations is derived from the uniquely known pattern generated from each marking material; and generating animation data incorporating the position and orientation information. Motion Capture 130 6 160A 15012 172 160B 160B 172

Description

AUSTRALIA Patents Act 1990 COMPLETE SPECIFICATION Standard Patent Applicant (s): Sony Corporation and SONY PICTURES ENTERTAINMENT INC. Invention Title: Segment tracking in motion picture The following statement is a full description of this invention, including the best method for performing it known to me/us: 2 SEGMENT TRACKING IN MOTION PICTURE RELATED APPLICATION This application is a divisional application of Australian application no. 2007317452 the disclosure of which is incorporated herein by reference. Most of the disclosure of 5 that application is also included herein, however, reference may be made to the specification of application no. 2007317452 as filed to gain further understanding of the invention claimed herein. BACKGROUND 10 The present invention relates generally to motion capture, and more particularly to segment tracking using motion marker data. Motion capture systems are used to capture the movement of an actor or object and map it onto a computer-generated 15 actor/object as a way of animating it. These systems are often used in the production of motion pictures and video games for creating a digital representation of an actor or object for use as source data to create a computer graphics ("CG") animation. In a typical system, an actor wears a 20 suit having markers attached at various locations (e.g., small reflective markers are attached to the body and limbs). Appropriately placed digital cameras then record the actor's body movements in a capture volume from different angles while the markers are illuminated. The 25 system later analyzes the images to determine the locations (e.g., spatial coordinates) of the markers on the actor's suit in each frame. By tracking the locations of the markers, the system creates a spatial representation of the markers over time and builds a digital representation of the 30 actor in motion. The motion is then applied to a digital model in virtual space, which may be textured and rendered to produce a complete CG representation of the actor and/or the performance. This technique has been used by special 33a9123.1 (GHManes) P80909AU.1 25/05/2012 3 effects companies to produce realistic animations in many popular movies. SUMMARY In a first aspect, the present invention provides a 5 method, comprising: applying a plurality of marking materials to different locations on a surface, each of the marking materials including a uniquely known pattern; acquiring a sequence of image frames, each image frame 10 of the sequence including a plurality of images of uniquely known patterns covering the surface; deriving position and rotational orientation information for a portion of the different locations on the surface by analyzing position and rotational orientation 15 information of those uniquely known patterns which are identifiable in each image frame of the sequence; wherein the rotational orientation information for a portion of the different locations is derived from the uniquely known pattern generated from each marking material; 20 and generating animation data incorporating the position and orientation information. In a second aspect, the present invention provides a system, comprising: 25 an image acquisition module configured to generate a sequence of image frames, each image frame including a plurality of synchronized images of a plurality of uniquely known patterns disposed on a surface; and a segment tracking module configured to receive the 30 sequence of image frames and generate animation data based on the uniquely known pattern disposed on the surface wherein the segment tracking module includes: an identification module configured to receive the sequence of image frames and generates identification 3389123_1 (GHMaters) P80909 AU.? 25/O5/2012 4 information regarding those uniquely known patterns that are identifiable in each image frame of the sequence; an orientation module configured to receive the sequence of image frames and the identification information 5 and generates rotational orientation information for different locations on the surface by determining the rotational orientation of those uniquely known patterns identified in each image frame; wherein rotational orientation information for each 10 location on the surface is derived from a single uniquely known pattern; a tracking module configured to receive the identification information and generate marker trajectory information for different locations on the surface by is determining the trajectory information for those uniquely known patterns identified in each image frames; and an animation module configured to receive the orientation information and the marker trajectory information and generate animation data. 20 In a third aspect, the present invention provides a computer program, stored in a computer-readable storage medium, the program comprising executable instructions that cause a computer to: acquire a sequence of image frames, each image frame of 25 the sequence including a plurality of images of a uniquely known pattern covering a surface; derive position and rotational orientation information regarding the known pattern; wherein rotational orientation information for each 30 location on the surface is derived from a single uniquely known pattern; and generate animation data incorporating the position and orientation information. 3389123_1 (GHMatter) P80909.AU.1 25/05/2012 5 Features and advantages of the present invention will become more readily apparent to those of ordinary skill in the art after reviewing the following detailed description and accompanying drawings. 5 BRIEF DESCRIPTION OF THE DRAWINGS The details of the present invention, both as to its structure and operation, may be gleaned in part by study of the accompanying drawings, in which: Figure 1 is a block diagram of a motion capture system 10 in accordance with one implementation; Figure 2 shows a sample collection of markers according to an implementation of the present invention; Figure 3 is an illustration of a human figure with marker placement positions according to one implementation; 15 Figure 4 presents a frontal view of a human body model equipped with markers; Figure 5 depicts a quartering back view of a human body model configured with markers; Figure 6 is a flowchart describing a method of segment 20 tracking; Figure 7A illustrates a marker labeled or patterned to represent a capital letter A; Figure 7B illustrates a known pattern comprising a plurality of markers forming the letter A as a pattern; 25 Figure 7C illustrates a marker comprising a literal expression of the letter A; Figure 8 is a flowchart depicting an example method of utilizing strips of marking material; Figure 9 is a functional block diagram of an 30 implementation of a segment tracking system; Figure 10A illustrates a representation of a computer system and a user; Figure 10B is a functional block diagram illustrating the computer system hosting the segment tracking system; and 3389 23_1 (GHMater) P80909.AU.1 25/05/2012 6 Figure 11 is a functional block diagram illustrating an implementation of a segment tracking module. DETAILED DESCRIPTION 5 Certain implementations as disclosed herein provide segment tracking in motion capture. Implementations include using one or more known patterns of markers applied on actors and/or objects. The marker(s) comprising a pattern are tracked as a group rather than individually. Thus, the 10 pattern can provide information, such as identification, position/translation, and orientation/rotation, which significantly aids marker tracking. After reading this description it will become apparent to one skilled in the art how to practice the invention in 15 various alternative implementations and alternative applications. However, although various implementations of the present invention will be described herein, it is understood that these embodiments are presented by way of example only, and not limitation. As such, this detailed 20 description of various alternative implementations should not be construed to limit the scope or breadth of the present invention as set forth in the appended claims. According to implementations of the present invention, markers encoded with known patterns are applied to actors 25 and/or objects, normally covering various surfaces of the actor and/or object. Identification, position/translation, and orientation/rotation of the actor and/or object are obtained by recording and digitizing images of the patterns. The patterns of markers can be selected to mitigate 30 "missing-marker" effects in motion capture during marker occlusions. In another implementation, identifiable random patterns are used as "known patterns." The known (and random) patterns may be generated, for example, using materials including quantum nanodots, glow 3389123_1 (GHMatten) P90909.AU.1 2505/2012 7 in-the dark (i.e., fluorescent) material, tattoos, and virtually any visible, infra-red, or ultra-violet ink, paint, or material which can be applied in a sufficiently identifiable pattern. The patterns may also include 5 patterns of inherent features (e.g., moles or wrinkles) of the actor and/or objects. In one implementation, a pattern comprises a plurality of markers or features coupled to the actor's body. In another implementation, a pattern comprises a single marker 10 (e.g., a marker strip) or feature. The pattern can be applied or affixed on or around the surfaces of an actor's limbs, hands, and feet. In one example, centroid information relating to the pattern can be derived from the circular disposition of a pattern strip wrapped around a 15 limb (i.e., appendage). By applying known patterns to the actors and/or objects and then recording their movements with cameras, it is possible to obtain not only position but also marker identity and spatial orientation. Figure 1 is a functional block diagram of a motion 20 capture system 100 in accordance with one implementation. The motion capture system 100 includes a motion capture processor 110, motion capture cameras 120, 122, 124, a user workstation 130, and an actor's body 140 and face 150 appropriately equipped with marker material 160 in a 25 predetermined pattern. Although Figure 1 shows only thirteen markers 160A-160F, substantially more markers can be used on the body 140 and face 150. The motion capture processor 110 is connected to the workstation 130 by wire or wirelessly. The motion capture processor 110 is typically 30 configured to receive control data packets from the workstation 130. As shown, three motion capture cameras 120, 122, 124, are connected to the motion capture processor 110. Generally more than three motion capture cameras are 3389123.1 (GHMatters) P80909AU.1 25/05/2012 8 required according to a variety of user- and animation related needs and requirements. The motion capture cameras 120, 122, 124 are focused on the actor's body 140 and face 150, on which markers 160A-160F have been applied. 5 The placement of the markers 160A-160F is configured to capture motions of interest including, for example, the body 140, face 150, hands 170, arms 172, legs 174, 178, and feet 176 of the actor. In the implementation illustrated in Figure 1, markers 160A capture motions of the face 150; 10 markers 160B capture motions of the arms 172; markers 160C capture motions of the body 140; markers 160D, 160E capture motions of the legs 174; and markers 160F capture motions of the feet 176. Further, uniqueness of the patterns on the markers 160A-160F provide information that can be used to 15 obtain identification and orientation of the markers. The marker 160D is configured as a strip of pattern wrapped around a leg of the actor. The motion capture cameras 120, 122, 124 are controlled by the motion capture processor 110 to capture synchronous 20 sequences of two-dimensional ("2-D") images of the markers. The synchronous images are integrated into image frames, each image frame representing one frame of a temporal sequence of image frames. That is, each individual image frame comprises an integrated plurality of simultaneously 25 acquired 2-D images, each 2-D image generated by an individual motion capture camera 120, 122, or 124. The 2-D images thus captured may typically be stored, or viewed in real-time at the user workstation 130, or both. The motion capture processor 110 performs the 30 integration (i.e., performs a "reconstruction") of the 2-D images to generate the frame sequence of three-dimensional ("3-D," or "volumetric") marker data. This sequence of volumetric frames is often referred to as a "beat," which can also be thought of as a "take" in cinematography. 3389123_1 (GHMatters) P80909AU.1 25/05/2012 9 Conventionally, the markers are discrete objects or visual points, and the reconstructed marker data comprise a plurality of discrete marker data points, where each marker data point represents a spatial (i.e., 3-D) position of a 5 marker coupled to a target, such as an actor 140. Thus, each volumetric frame includes a plurality of marker data points representing a spatial model of the target. The motion capture processor 110 retrieves the volumetric frame sequence and performs a tracking function to accurately 10 associate (or, "map") the marker data points of each frame with the marker data points of preceding and subsequent frames in the sequence. For example, each individual marker data point in a first volumetric frame corresponds to a single marker placed 15 on an actor's body 140. A unique label is assigned to each such marker data point of the first volumetric frame. The marker data points are then associated with corresponding marker data points in a second volumetric frame, and the unique labels for the marker data points of the first 20 volumetric frame are assigned to the corresponding marker data points of the second volumetric frame. When the labeling (i.e., tracking) process is completed for the entire volumetric frame sequence, the marker data points of the first volumetric frame are thus traceable through the 25 sequence, resulting in an individual trajectory for each marker data point. Discrete markers are conventionally used to capture the motion of rigid objects or segments of an object or body. For example, rigid markers attached at an elbow and a wrist 30 define the positions of each end of a forearm. When the forearm is moved, the motions of the elbow and wrist markers are tracked and resolved as described above in a sequence of volumetric frames. The motion of the forearm is thus modeled as a rigid body (e.g., a rod) with only the ends 3389123_1 (GHMaters) P80909.AU.1 25/05/2012 10 defined by the elbow and wrist markers. However, while translational movements of the forearm are easily resolved by analyzing the changes in spatial positions of the elbow and wrist markers, a common twisting motion of the forearm 5 is difficult to detect because a twist can be performed without substantially moving the wrist or elbow. In one implementation, contrasting with the use of conventional discrete markers, markers disposed in a pattern are used, allowing the motion capture processor 110 to track 10 the pattern as a group rather than individually tracking markers. Because the pattern provides identification information, movement of the markers of one pattern with respect to another pattern can be computed. In one implementation, a pattern tracked in this way is 15 reconstructed in each volumetric frame as an individual object having spatial position information. The object is tracked through the sequence of volumetric frames, yielding a virtual animation representing the various spatial translations, rotations, and twists, for example, of the 20 part of the actor to which the pattern is applied. In one implementation, one or more known patterns are printed onto strips 160D. The strips 160D are then wrapped around each limb (i.e., appendage) of an actor such that each limb has at least two strips. For example, two strips 25 160D are depicted in Figure 1, wrapped around the actor's left thigh 178. End effectors (e.g., hands, feet, head), however, may be sufficiently marked with only one strip. Once captured, as discussed above, the printed patterns of the wrapped strips 160D enable the motion capture processor 30 110 to track the position and orientation of each "segment" representing an actor's limb from any angle, with as few as only one marker on a segment being visible. Illustrated in Figure 1, the actor's thigh 178 is treated as a segment at the motion capture processor 110. By wrapping a patterned 3389123_1 (GHMatters) P80909AU.1 25/05/2012 11 strip 160D with multiple markers around a limb in substantially a circle, the "centroid" of the limb (i.e., segment) can be determined. Using multiple patterned strips 160D of markers, a centroid may be determined to provide an 5 estimate or model of the bone within the limb. Further, it is possible to determine orientation, translation and rotation information regarding the entire segment from one (or more if visible) markers and/or strips applied on the segment. 10 In one implementation, the motion capture processor 110 performs segment tracking according to techniques disclosed herein from which identification, positioning/translation, and orientation/rotation information is generated for a group of markers (or marked areas). While a conventional 15 optical motion capture system typically records only position for a marker, segment tracking enables the motion capture processor 110 to identify which marker(s) are being captured and to locate the position and orientation of the captured markers of the segment. Once the markers are 20 detected and identified, position and orientation/rotation information regarding the segment can be derived from the identified markers. Confidence in the determinations of position and orientation information for the segment increases as more markers are detected and identified. 25 Markers or marking material applied in known patterns (and identifiable random patterns) essentially encode identification and orientation information facilitating efficient segment tracking. In one implementation, a known pattern of smaller markers constitutes a single marker or 30 marking area. For example, a marker labeled or patterned as representing a capital letter A, as shown in Figure 7A, actually comprises six smaller markers A, B, C, D, E and F. In another implementation, a known pattern comprises a plurality of markers or marking areas forming the letter A 3389123_1 (GHMatters) PB0909.AU.1 2S/05/2012 12 as a pattern, as shown in Figure 7B. The eight dots forming the letter A function as a single, larger marker. Alternatively, a marker can comprise a literal expression of the letter A, as shown in Figure 7C. A common 5 characteristic to the markers illustrated in Figures 7A-C is that they all have an identifiable orientation. In each case, the marker can be rotated, but will remain identifiable as a letter A, which significantly enhances marker tracking efficiency because it will be readily 10 trackable from frame to frame of captured image data. In addition to identification, an orientation may also be determined for each of the marker examples from frame to frame. If a marker 160B (see Figure 1) is coupled to an actor's forearm, for example, it will therefore rotate 15 substantially 180 degrees when the actor moves the forearm from a hanging position to a straight-up, overhead reaching position. Tracking the marker will reveal not only the spatial translation to the overhead position, but that the forearm segment is oriented 180 degrees differently from 20 before. Thus, marker tracking is enhanced and improved using markers or groups of markers encoding identification and orientation information. Figure 2 shows a sample collection of markers according to an implementation of the present invention. Each marker 25 comprises a 6x6 matrix of small white and black squares. Identification and orientation information is encoded in each marker by a unique placement of white squares within the 6x6 matrix. In each case, rotating the marker causes no ambiguity in terms of determining the identity and 30 orientation of the marker, thus demonstrating the effectiveness of this scheme for encoding information. It will be appreciated that encoding schemes using arrangements other than the 6x6 matrix of black and white elements disclosed herein by example may also be implemented. 3389123.1 (GHMattem) P80909.AU.1 25105/2012 13 Figure 3 is an illustration of a human figure with marker placement positions according to one implementation. The markers shown encode identification and orientation information using a scheme similar to that depicted in 5 Figure 2. They are positioned substantially symmetrically, and such that each major extremity (i.e., segment) of the body is defined by at least one marker. Approximately half of the markers depicted are positioned on a surface of the body not visible in the frontal view shown, and instead 10 include arrows pointing to their approximate occluded positions. Referring to Figure 1, motion capture cameras 120, 122, 124 (representing typically a much larger plurality of cameras) encompass a capture space in which the actor's body 140 and face 150 are in motion. Even when the 15 view of any of the markers is occluded to some subset of motion capture cameras 120, 122, 124, another subset will retain a view and capture the motions of the occluded markers. Thus, virtually all movements by an actor so equipped with markers can be captured using the systems 20 described in relation to Figure 1, and tracked using segment tracking methods. Figure 4 presents a frontal view of a human body model equipped with markers as described in Figure 3. As shown, only the markers on the forward-facing surfaces of the model 25 are visible. The rest of the markers are partially or fully occluded. Figure 5 depicts a quartering back view of the same human body model in substantially.the same pose as shown in Figure 4. From this view, the frontally-placed markers are occluded, but many of the markers occluded in 30 Figure 4 are now visible. Thus, at any given time, substantially all of the markers are visible to some subset of the plurality of motion capture cameras 120, 122, 124 placed about the capture space. 339123.1 (GHMatters) P80909AU.1 25/0$,2012 14 Also, the marker placements on the 3-D model depicted in Figures 4 and 5 substantially define the major extremities (segments) and areas on the body that articulate motions (e.g., the head, shoulders, hips, ankles, etc.). 5 When segment tracking is performed on the captured data, the positions of the body on which the markers are placed will be locatable and their orientations determinable. Further, the segments of the body defined by the marker placements, e.g., an upper arm segment between an elbow and a shoulder, 10 will also be locatable because of the markers placed substantially at each end of that segment. The position and orientation of the upper arm segment will also be determinable from the orientations derived from the individual markers defining the upper arm. 15 Figure 6 is a flowchart describing a method of segment tracking 600 according to an implementation. A marking material with a known pattern, or an identifiable random pattern, is applied to a surface, at 610. In one implementation, the surface is that of an actor's body, and 20 a pattern comprises a plurality of markers that is coupled to the actor's body. In another implementation, a pattern comprises a single marker (e.g., a marker strip) that is coupled to the actor's body. The pattern may also be formed as a strip 160D and affixed around the actor's limbs, hands, 25 and feet, as discussed in relation to Figure 1. Markers also include reflective spheres, tattoos glued on an actor's body, material painted on an actor's body, or inherent features (e.g., moles or wrinkles) of an actor. In one implementation, patterns of markers can be applied to the 30 actor using temporary tattoos. A sequence of image frames is acquired next, at 620, according to methods and systems for motion capture described herein, and above in relation to Figure 1. Once captured, the image data are used to reconstruct a 3-D model 3389123_1 (GHMatters) P80909 AU.1 25105/2012 15 of an actor or object equipped with the markers, for example. In one implementation, this includes deriving position and orientation information of the marker pattern for each image frame, at 630. Position information is 5 facilitated by the unique identification information encoded in the markers provided by the pattern using characteristic rotational invariance, as discussed with respect to Figures 4 and 5. Orientation information may be derived by determining the amount of rotation of the markers comprising 10 the pattern. Marker rotations, and orientation in general, may be defined by affine representations in the 3-D space facilitating marker tracking. For example, an orientation can be represented by values representing the six degrees of freedom ("6DOF") of an object in 3-D space. Thus, the 15 orientation can have three values representing any displacement in position (translation) from the origin of the coordinate system (e.g., a Euclidean space), and three values representing angular displacements (rotations) relative to the primary axes of the coordinate system. 20 Animation data based on the movements of the markers are generated, at 640. For example, a virtual digital model is to be activated by the data derived from the movements of an actor captured in the image data. During the performance, the actor swings the leg according to a script. 25 The actor's legs are equipped with markers at each major joint, i.e., the hip, knee, and ankle, for instance. The movements of the markers are determined and used to define the movements of the segments of the actor's leg. The segments of the actor's leg correspond to the segments of 30 the leg of the digital model, and the movements of the segments of the actor's leg are therefore mapped to the leg of the virtual digital model to animate it. As discussed in relation to Figure 6, at 610, a pattern may be formed as a strip 160D (see Figure 1) and affixed 3389123_1 (GHMatters) P80909.AU.1 25/05/2012 16 around an actor's limbs, hands, and feet. A centroid may then be determined from the circular disposition of the strip wrapped around the limb. Two such strips positioned at each end of the limb may then be used to determine a 5 longitudinal centroid relative to the limb (segment), approximating an underlying skeletal element, for example. A flowchart depicting a method of utilizing strips of marking material 800 is provided in Figure 8. Strips of marking material with known patterns are applied to one of 10 an actor's limbs, at 810, typically by wrapping it around the limb. Because the wrapped strip of marking material is disposed in a substantially circular shape, centroids of the strips may be determined and used to express the movements of the body area wrapped with the strip. Further, the is motion of a skeletal element of the limb can thus be approximated and used for skeleton modeling. A sequence of image frames is acquired next, at 820, according to methods and systems for motion capture described above in relation to Figure 1. Once captured, the 20 image data are used to reconstruct a 3-D model of an actor equipped with the markers, for example, in essentially the same manner as described in Figure 6, with respect to 620. In one implementation, this includes deriving position and orientation information of the known pattern for each strip 25 in each image frame, at 830, in essentially the same manner as described in Figure 6, with respect to 630. Centroid information is derived for the actor's limb equipped with one or more strips from the position and orientation information, at 840. As discussed above, the 30 centroids of two or more strips can be used to approximate the skeletal structure (i.e., a bone) within the actor's limb. The movements of the centroids of the marker strips are then used to express the movements of the skeletal structure, and thus the movement of the limb. 3389123_1 (GHMatter) POOO99AU.1 25/05/2012 17 Animation data based on the movements of the marker strips is generated, at 850. For example, during the performance, the actor swings his leg according to a script. The actor's legs are equipped with strips of marker material 5 wrapped around as described above. That is, for instance, a strip may be wrapped around the upper thigh and another wrapped around near the knee. The movements of the centroids of the marker strips are determined and used to define the skeletal structure of the upper leg, and 10 consequently a skeletal element within the upper leg of a virtual digital model corresponding to the actor. The movements of the centroids can therefore used to animate the leg of the virtual digital model. In one implementation, the markers are printed or 15 formed using quantum nanodots (also known as "Quantum Dots," or "QDs"). The system would be similar in configuration to the traditional retro-reflective system, but with the addition of an exciter light source of a specific frequency (e.g., from existing filtered ring lights or from another 20 source) and narrow band gap filters placed behind the lenses of the existing cameras, thus tuning the cameras to the wavelength of light emitted by the QDs. QDs are configured to be excited by light of a particular wavelength, causing them to emit light (i.e., 25 fluoresce) at a different wavelength. Because they can emit light that has been quantum shifted up the spectrum from the excitation wavelength, the excitation light can be filtered from the cameras. This causes any light that falls outside of the particular emission spectrum for the QDs to be 30 substantially blocked at the camera. QDs can be tuned to virtually any wavelength in the visible or invisible spectrum. This means that a group of cameras can be filtered to only see a specific group of markers and nothing else, significantly cutting down the 3389123_1 (GHMatters) P80909.AU.1 2SIOS2012 18 workload required of a given camera, and allow the camera to work "wide open" in the narrow response range of the QDs. In one implementation, the quantum nanodots are added to a medium such as ink, paint, plastic, temporary tattoo 5 blanks, etc. Figure 9 is a functional block diagram of an implementation of a segment tracking system 900. An image acquisition module 910 generates image frames of motion capture image data, and a segment tracking module 920 10 receives image frames and generates animation data. The image acquisition module 910 operates according to methods discussed above relating to the motion capture system 100 described in Figure 1. In one implementation, the image frames comprise volumetric data including is untracked marker data. That is, marker data exist in each frame as unlabeled spatial data, unintegrated with the marker data of the other frames. The segment tracking module 920 operates according to methods and schemes described in relation to Figure 2 20 through Figure 9 above. Figure 11 is a functional block diagram depicting an example configuration of a segment tracking module 920. As shown, the segment tracking module 920 includes an identification module 1100, an orientation module 1110, a 25 tracking module 1120, and a animation module 1130. The identification module 1100 includes capability to identify markers having known patterns, such as the markers 160A-16OF shown in Figure 1. In one implementation, the identification module 1100 performs pattern matching to 30 locate known and random patterns of markers 160 from frame to frame. In another implementation, the identification module 1100 is configured to identify a single larger marker comprising a group of smaller markers, as discussed in relation to Figures 7A-7C. A marker 160 is designed to be 3380123_1 (GHMaiters) P80909.AU.1 2/O5/2012 19 identifiable regardless of its rotational state, and the identification module 1100 is configured accordingly to perform rotationally invariant identification of the markers 160. Once the markers 160 are uniquely identified, they are 5 appropriately associated with the particular part of the actor's body, for example, to which it is coupled. As depicted in Figure 1, a marker 160B is associated with the actor's upper arm 172, markers 160D with the upper and lower thigh 178 (marker strips wrapped around the leg segment), 10 and markers 160C associated with the torso. The identification information is then passed to the orientation module 1110 and the tracking module 1120. In the example configuration illustrated in Figure 11, the orientation module 1110 receives the identification 15 information and generates orientation information. In one example, the orientation information comprises 6DOF data. The marker 160 is then analyzed in each frame to determine its evolving positional and rotational (i.e., affine) state. In one implementation, a 3-D affine transformation is 20 developed describing the changes in these states from frame to frame for each marker 160. In the case where strip markers 160D are wrapped around each end of a segment of an actor's limb, such as the thigh 178 (see Figure 1), a centroid may be determined for that limb segment 25 approximating the underlying skeletal structure (i.e., bone). The orientation information (6DOF, affine transformation) is passed to the animation module 1130. The tracking module 1120 receives identification information and generates marker trajectory information. 30 That is, the markers are tracked through the sequence of image frames, labeled, and a trajectory is determined. In one implementation, labeling and FACS approaches are employed according to U.S. Patent Application 11/467,503, filed August 25, 2006, entitled "Labeling Used in Motion 3389123_1 (GHMatters) P8OOS.9AU.1 25/05/2012 20 Capture", and U.S. Patent Application 11/829,711, filed July 27, 2007, entitled "FACS Cleaning in Motion Capture." The labeling information (i.e., trajectory data) is passed to the animation module 1130. 5 The animation module 1130 receives orientation and labeling information and generates animation data. In one implementation, marker positions and orientations are mapped to positions on a (virtual) digital character model corresponding to the positions on the actor at which the 10 markers were coupled. Similarly, a segment defined by markers on the body of the actor is mapped to a corresponding part on the digital character model. For example, the movement of the centroid approximating the skeletal structure of the actor's limb also models the 15 movement of that part of the actor's body. The transformations describing the movements of the markers 160 and related centroids are then appropriately formatted and generated as animation data for animating a corresponding segment of the digital character. 20 The animation module 1130 receives the animation information and applies it to the digital character, resulting in its animation. The animation is typically examined for fidelity to the actor's movements, and for purposes of determining whether any, an how much, 25 reprocessing may be required to obtain a desired result. Figure 10A illustrates a representation of a computer system 1000 and a user 1002. The user 1002 uses the computer system 1000 to perform segment tracking. The computer system 1000 stores and executes a segment tracking 30 system 1090, which processes image frame data. Figure 10B is a functional block diagram illustrating the computer system 1000 hosting the segment tracking system 1090. The controller 1010 is a programmable processor and controls the operation of the computer system 1000 and its 3389123_1 (GHMatters) P809O9AU.1 25/05/2012 21 components. The controller 1010 loads instructions (e.g., in the form of a computer program) from the memory 1020 or an embedded controller memory (not shown) and executes these instructions to control the system. In its execution, the 5 controller 1010 provides the segment tracking system 1090 as a software system. Alternatively, this service can be implemented as separate components in the controller 1010 or the computer system 1000. Memory 1020 stores data temporarily for use by the 10 other components of the computer system 1000. In one implementation, memory 1020 is implemented as RAM. In one implementation, memory 1020 also includes long-term or permanent memory, such as flash memory and/or ROM. Storage 1030 stores data temporarily or long term for 15 use by other components of the computer system 1000, such as for storing data used by the segment tracking system 1090. In one implementation, storage 1030 is a hard disk drive. The media device 1040 receives removable media and reads and/or writes data to the inserted media. In one 20 implementation, for example, the media device 1040 is an optical disc drive. The user interface 1050 includes components for accepting user input from the user of the computer system 1000 and presenting information to the user. In one 25 implementation, the user interface 1050 includes a keyboard, a mouse, audio speakers, and a display. The controller 1010 uses input from the user to adjust the operation of the computer system 1000. The I/O interface 1060 includes one or more I/O ports 30 to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA). In one implementation, the ports of the I/O interface 660 include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports. In another implementation, 3389123_1 (GHMatters) P80909.AU.1 25/05/2012 22 the I/O interface 1060 includes a wireless interface for communication with external devices wirelessly. The network interface 1070 includes a wired and/or wireless network connection, such as an RJ-45 or "Wi-Fi" 5 interface (including, but not limited to 802.11) supporting an Ethernet connection. The computer system 1000 includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), though these components are not 10 specifically shown in Figure 10B for simplicity. In other implementations, different configurations of the computer system can be used (e.g., different bus or storage configurations or a multi-processor configuration). Various illustrative implementations of the present 15 invention have been described. However, one of ordinary skill in the art will recognize that additional implementations are also possible and within the scope of the present invention. For example, known and identifiable random patterns may be printed, painted, and inked onto a 20 surface of an actor or object. Further, any combination of printing, painting, inking, and tattoos, quantum nanodots, and inherent body features may be used to obtain a desired pattern. It will be further appreciated that grouping 25 functionalities within a module or block is for ease of description. Specific functionalities can be moved from one module or block to another without departing from the invention. Accordingly, the present invention is not limited to 30 only those embodiments described above. In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word "comprise" or variations such as 3389123.1 (GHMatters) P80909.AU. 25/05/2012 23 "comprises" or "comprising" is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention. 5 It is to be understood that references to prior art referred herein do not constitute an admission that the publication forms or formed a part of the common general knowledge in the art, in Australia or any other country. 10 3389t23. (GHMaters) PBOOAU.1 25/05/2012

Claims (20)

1. A method, comprising: applying a plurality of marking materials to different 5 locations on a surface, each of the marking materials including a uniquely known pattern; acquiring a sequence of image frames, each image frame of the sequence including a plurality of images of uniquely known patterns covering the surface; 10 deriving position and rotational orientation information for a portion of the different locations on the surface by analyzing position and rotational orientation information of those uniquely known patterns which are identifiable in each image frame of the sequence; 15 wherein the rotational orientation information for a portion of the different locations is derived from the uniquely known pattern generated from each marking material; and generating animation data incorporating the position 20 and orientation information.

2. The method of claim 1, wherein the marking materials conform to the surface. 25

3. The method of claim 1, wherein the uniquely known patterns are disposed on the marking material with quantum nano dots.

4. The method of claim 3, wherein the quantum nanodots 30 are supported in a medium including an ink, a paint, and a plastic.

5. The method of claim 1, wherein the marking materials include a temporary tattoo. 3389123_1 (GHMatter) POM09.AU.1 25/05i2012 25

6. The method of claim 1, wherein the marking materials include a reflective material. 5

7. The method of claim 1, wherein the marking materials include a fluorescent material.

8. The method of claim 1, wherein the surface includes a surface of a person. 10

9. The method of claim 1, wherein the surface includes a surface of an object.

10. The method of claim 9, wherein the object includes 15 at least one of a stage set and a stage prop.

11. The method of claim 1, wherein the patterns are predetermined patterns. 20

12. The method of claim 1, wherein the uniquely known patterns are randomly generated.

13. The method of claim 1, wherein said applying the plurality of marking materials to the surface includes 25 wrapping a strip onto which a uniquely known pattern is printed around an appendage of an actor.

14. The method of claim 13, wherein said deriving position and rotation orientation information includes 30 determining a centroid of the strip.

15. The method of claim 1, wherein said applying the plurality of marking materials to the surface includes

3389123.1 (GHMatters) P80909AU.1 25/05/2012 26 wrapping at least two strips onto which uniquely known patterns are printed around an appendage of an actor.

16. The method of claim 15, wherein said deriving 5 position and rotational orientation information includes determining a centroid of the appendage of the actor.

17. The method of claim 1, wherein each of the patterns on the plurality of marking materials is unique from every 10 other pattern on the plurality of marking materials.

18. The method of claim 1, wherein at least a portion of the patterns on the plurality of marking materials are unique from another portion of the patterns on the plurality 15 of marking materials.

19. A system, comprising: an image acquisition module configured to generate a sequence of image frames, each image frame including a 20 plurality of synchronized images of a plurality of uniquely known patterns disposed on a surface; and a segment tracking module configured to receive the sequence of image frames and generate animation data based on the uniquely known pattern disposed on the surface 25 wherein the segment tracking module includes: an identification module configured to receive the sequence of image frames and generates identification information regarding those uniquely known patterns that are identifiable in each image frame of the sequence; 30 an orientation module configured to receive the sequence of image frames and the identification information and generates rotational orientation information for different locations on the surface by determining the 3389123.1 (GHMatters) P090O9.AU.1 25105/2012 27 rotational orientation of those uniquely known patterns identified in each image frame; wherein rotational orientation information for each location on the surface is derived from a single uniquely 5 known pattern; a tracking module configured to receive the identification information and generate marker trajectory information for different locations on the surface by determining the trajectory information for those uniquely 10 known patterns identified in each image frames; and an animation module configured to receive the orientation information and the marker trajectory information and generate animation data. 15

20. A computer program, stored in a computer-readable storage medium, the program comprising executable instructions that cause a computer to: acquire a sequence of image frames, each image frame of the sequence including a plurality of images of a uniquely 20 known pattern covering a surface; derive position and rotational orientation information regarding the known pattern; wherein rotational orientation information for each location on the surface is derived from a single uniquely 25 known pattern; and generate animation data incorporating the position and orientation information. 3389123_1 (GHMattes) P80909PAU.1 25/05212