JP2003512802A - System and method for three-dimensional modeling - Google Patents

Abstract

  (57) [Summary] A method and an image processing system for providing a three-dimensional model using information from a pair of input image frames are disclosed. A three-dimensional surface is obtained by first identifying features in the input frame. The feature correspondence matching is performed using the difference information based on the frame. The identified features may be related to each other using the time information.

Description

Detailed Description of the Invention

FIELD OF THE INVENTION The present invention relates generally to the field of three-dimensional modeling, and more particularly to a system for three-dimensional modeling objects contained in digital images using disparity-based information. And method.

BACKGROUND OF THE INVENTION The Internet or Public Switched Telephone Net
Applications for video / image communication over work (PSTN) are increasing in popularity and use. In conventional video / image communication technology, pictures (in JPEG or GIF format) are captured and transmitted over a transport network. However, this approach requires a large bandwidth due to the size of the picture (ie the amount of data).

Block-based hybrid encoders are commonly used when encoding video at 64 K to 2 Mbits / sec. The encoder subdivides each image of the sequence into individual moving blocks. Each block is then encoded by 2D motion estimation to transform the coding. Depending on the transmission rate, the resulting image that is received is not played smoothly and cannot be played in real time.

Video / visual communications and / or methods for reducing the amount of information required to be transmitted are used. One of these methods is used in video phone applications. An image is encoded with three sets of parameters that define its motion, shape, and surface color. Since the subject of visual communication is typically a human, the primary focus can be directed at the subject's head or face.

In conventional video telephone communication systems, a single camera is commonly used to capture an image of the person making the video call. Since only one camera is used, it is difficult to capture the true three-dimensional (3D) face surface (ie, shape parameter). In general, multiple 2D (typically 6) views of an object are required to generate a 3D surface. A distance transform is applied using these views. For example, an ellipse can be used as the generating function to obtain the Z coordinate of the shape in the object as a function of its distance to the object's boundary. These contours can only approximate 3D shapes.

Another method is called model-based coding. Low bit rate communication can be achieved by encoding and transmitting only typical facial parameters of the subject's head. A facial image is composited using the parameters transmitted at the remote site. Model-based coding generally requires at least four tasks: face segmentation, facial feature extraction, feature tracking, and motion estimation.

One known method for face segmentation is to form a data set that describes the parametrically displayed face. This dataset defines a three-dimensional description of facial objects. The parametrically represented face is provided as an anatomy-based structure by modeling actuator and force-based deformations of muscle and skin.

As shown in FIG. 1, each vertex of a polygon, the set of polygons defining a human face model 100, is defined by an X, Y, and Z coordinate. Each vertex is identified by an index. A particular polygon is defined by a set of indices surrounding the polygon. Signs may be added to the set of indices to limit the color for a particular polygon.

Systems and methods for analyzing digital images, recognizing human faces, and extracting facial features are also known. Conventional facial feature detection systems use methods such as facial color tone detection, template matching, or edge detection approaches.

One of the most difficult problems in model-based coding is to provide face feature support quickly, easily, and crudely. In sequential frames,
Features of the same face must be matched exactly. Conventionally, a block matching process is used to compare pixels in the current frame and the next frame to determine feature correspondences. If searched for face correspondences over the entire frame, the process is slow and can result in inaccurate results due to the mismatch of regions with the same gradient value. Processing time may be improved if only a subset of frames is searched. However, in this situation the process may not be able to determine all face correspondences.

Accordingly, there is a need in the art for improved systems and methods for three-dimensional modeling of objects contained in digital images for reduced data rate transmission.

BRIEF SUMMARY OF THE INVENTION The present invention aims to address the limitations of conventional video / image communication systems and model-based coding described above.

Another object of the present invention is to provide a cross-platform method directed to objects that send compressed video information in real time.

It is a further object of the invention to make it possible to code a particular object within an image frame.

It is a further object of the present invention to incorporate synthetic and natural visual objects interactively or in real time.

In one aspect of the present invention, the image processing apparatus includes at least one feature extraction determiner configured to extract feature position information from a pair of input image signals, and feature position information and difference information. And a matching unit for matching corresponding features in the input image signal according to.

One embodiment of the invention relates to a method of determining parameters related to a 3D model. This method comprises the steps of extracting positional information of a feature related to a pair of input images,
Matching corresponding features in the pair of input images according to the extracted position information and the difference information of the features. The method further comprises determining parameters for the 3D model according to corresponding matching of features.

[0018]   The above as well as the embodiments and aspects of the present invention are illustrated in the following detailed description.

The features and advantages of the invention may be understood with reference to the detailed description of the preferred embodiments given above with reference to the drawings.

Description of the Preferred Embodiments Referring to FIG. 2, a 3D modeling system 10 is shown. Generally, system 1
0 is at least one feature extraction determiner 11 and at least one set of time information 1
2 and a matching unit 13 corresponding to the feature. The left frame 14 and the right frame frame 15 are input into the system 10. The frame frames of the left and right frames consist of image data which may be digital or analog. If the image data is analog, analog-to-digital circuitry can be used to convert the data into digital form.

The feature extraction determiner 11 determines the feature position / location in the digital image, such as the feature positions of the nose, eyes, and mouth faces. While two feature extraction determiners 11 are shown in FIG. 2, one determiner may be used to extract position information from the left and right frames 14 and 15, respectively. The time information 12 includes data such as preceding and / or subsequent frames used to impose constraints on the exact feature correspondence. It should be appreciated that the current frame to be processed needs to be the first frame input to system 10. The test frame may be used to establish some hysteresis.

In the preferred embodiment, system 10 is implemented with computer readable code executed by data processing equipment. The code may be stored in memory within the data processing device or may be read / downloaded from a storage medium such as a CD-ROM or floppy disk. In other embodiments, hardware circuitry may be used in place of or in combination with software instructions to implement the invention. For example, the present invention may be implemented on a digital television platform that uses a Trimedia processor for processing and a television monitor for displaying. The present invention can also be implemented on the computer 30 shown in FIG.

As shown in FIG. 3, the computer 30 has a network connection 3 that interfaces to a data network such as a variable bandwidth network or the Internet.
1 and a fax / modem connection 32 for interfacing with other remote sources such as video or digital cameras (not shown). The computer 30
Furthermore, a display 33 for displaying information (including video data) to a user, a keyboard 34 for inputting texts and user commands, a mouse 35 for positioning a cursor on the display 33 and inputting user commands, and are installed therein. A disk drive 36 for reading and writing from a floppy disk, C
A CD-ROM drive 37 for accessing information stored in the D-ROM. The computer 30 may be fitted with a pair of video conferencing cameras for inputting images or the like and one or more peripherals such as a printer 38 for outputting images, text or the like.

FIG. 4 shows a random access memory (RAM) and a read only memory (ROM).
), And the internal structure of a computer 30 having a memory 40 that may include a computer-readable medium such as a hard disk. The items stored in the memory 40 include an operating system 41, data 42, and applications 43. The data stored in memory 40 may include time information 12. In the preferred embodiment of the invention, the operating system is UN
A windowing operating system such as IX® and the present invention may be used with other operating systems such as Microsoft's Windows 95. Among the applications stored in memory 40 are video encoder 44, video decoder 45, and frame grabber 46. The video encoder 44 encodes the video data in a conventional manner, and the video decoder 45 decodes the video data encoded in the conventional manner. The frame grabber 46 is
It allows a single frame from a video signal stream to be captured and processed.

The computer 30 includes a central processing unit (CPU) 50, a communication interface 51, a memory interface 52, and a CD-ROM drive interface 53.
Also included are a video interface 54 and a bus 55. The CPU 50 stores the computer-readable code, that is, the application, in the memory 50.
It has a microprocessor and the like to be executed from. Such applications are
It may be stored in memory 40 (as described above) or, alternatively, a floppy disk in disk drive 36 or a CD-R in CD-ROM drive 37.
It may be stored in the OM. The CPU 50 accesses the application (or other data) stored in the floppy disk via the memory interface 52, and CD-Rs the application (or other data) stored in the CD-ROM.
Access via the OM drive interface 53.

Execution of applications on the computer 30 and other tasks are performed by the keyboard 3
4 or mouse 35 may be used. The output results from the application running on the computer may be displayed to the user on the display 34 or, alternatively, may be output via the network connection 31. For example, input video data may be received through video interface 54 or network connection 31. The input video data may be decoded by the video decoder 45. The output video data may be encoded by video encoder 44 for transmission over video interface 54 or network interface 31. Display 33 preferably comprises a display processor that forms a video image based on encoded video data provided by CPU 50 on bus 55. The output results from various applications are
It may be supplied to the printer 38.

Referring to FIG. 2, the frame 14 of the left frame and the frame 1 of the right frame
5, preferably having a pair of stereo digital images. For example, digital images may be received from two (static or video) cameras 60 and 61 (shown in Figure 5) and stored in memory 40 for subsequent processing. Other frames captured at different corners or views, or a pair of frames may be used. The cameras 60 and 61 may be part of another system such as a video conferencing system or an animation system.

The cameras 60 and 61 are in close proximity to each other and the subject 64 is placed at a short distance from the cameras 62 and 63. As shown in FIG. 5, the cameras 60 and 61 are at a distance b (center-to-center) from each other. The object 62 is the camera 60
And 61, respectively, at a distance f. b equals approximately 12.7 to 15.24 centimeters (5 to 6 inches) and f approximately 91.44 centimeters (3 feet)
Is preferred. However, it should be understood that the invention is not limited to these distances and that these distances are merely examples.

The camera 60 preferably captures a front view and the camera 61 preferably captures an offset or side view of the object 62. This allows to compare the left frame 14 and the right frame frame 15 to determine the difference map. In the preferred embodiment of the invention, the left frame 14 (image A) is compared with the frame frame 15 (image B) of the right frame. However, the reverse comparison may be performed.

A digital frame or image may be conceptually described as having a plurality of horizontal scan lines and a plurality of vertical columns forming an array of pixels. The number of scan lines and columns determines the resolution of the digital image. To determine the difference map, the scan lines are aligned, for example scan line 10 of image A matches scan line 10 of image B. The pixel on scanline 10 of image A is matched to its corresponding pixel on scanline 10 of image B. Thus, for example, if the 15th pixel of scanline 10 of image A is matched with the 10th pixel of scanline 10 of image B, the difference is calculated as 15-10 = 5. When the cameras 60 and 61 of the left and right frame of the frame are brought in close proximity, pixels of the foreground information of the image, eg
A person's face has more differences than pixels of background information.

The difference map based on the difference calculation may be stored in the memory 40. Each scanline (or column) of the image has a profile that contains the difference for each pixel in that scanline (or column). In this example, the grayscale level for each pixel indicates the magnitude of the calculated difference for that pixel. The darker the grayscale level, the lower the difference.

The difference threshold may be selected, for example, 10 and all differences above the difference threshold indicate that the pixel is foreground information (ie object 64), while all below 10 The difference indicates that the pixel is background information. The selection of the difference threshold is based in part on the camera distance described above. For example, a lower difference threshold may be used if the object 62 is positioned at a greater distance from the camera 60 or 61, or a higher difference threshold than if the cameras 50 and 61 are further from each other. May be used.

The difference map is used to extract facial feature positions or coordinates from the left or right frame frames 14 and 15. The system and method described in US patent application Ser. No. 08 / 385,280, filed Aug. 30, 1999, preferably has a feature extraction determiner 11. The facial features preferably include positions for the eyes, nose, and mouth, as well as positions for the outline of the head. With reference to FIG. 1, these positions correlate to various vertices of the face model 100. For example, with respect to the nose, the facial feature extraction determiner may use
It is preferable to provide information directly related to 3 and 58.

However, the feature extraction determiner 11 only provides the X and Y coordinates of the face feature. The feature-matching matching unit 13 provides the Z coordinate. Feature extraction determiner 11
Is a triangulation based on the estimation of the position of the 3D points given to the perspective projection on the stereo image frames 14 and 15 of the left and right frame.
ation) procedure. For example, given the X and Y coordinates of the feature points (F _L and F _R ) in frames 14 and 15 of the left and right frame, the 3D surface (ie, Z or depth information) is given.

[0035]

[Equation 1] The distance f (shown in FIG. 5) can be determined by the cameras 60 and 6
Is the focal length of 1, the distance b is the distance of the baseline between the cameras 60 and 61 (shown in FIG. 5), and | F _L and F _R | are the differences calculated as described above. Is.

In this example, the above equations provide the relationship between the difference and the surface Z under some geometric conditions. In particular, the image plane in front of each camera is at the focal length f and each camera is oriented the same as the X axis of the camera's reference frame, which is oriented in the direction of the line defined by the positions of the cameras 60 and 61. The focal lengths of cameras 60 and 61 are assumed to be the same. It is also estimated that all geometric distortions of the lenses of cameras 60 and 61 are compensated. Other geometries may be used, but the relationship between the difference and the surface Z becomes more complex.

The other vertices of the face model 100 shown in FIG. 1 are interpolated or determined based on the position from the feature extraction / determination unit 11 (ie, the vertex information of the face feature) and the determination from the feature matching unit 13. Can be extrapolated. The interpolation may be linear, non-linear, or based on a scaled model or function. For example, the vertex between two other known vertices may be determined using a predetermined parabolic function that the three vertices satisfy. Other facial models with additional vertices may be used to provide enhanced or improved modeling results.

The face model 100 shown in FIG. 1 is a general expressionless face. The face model 100 is controlled according to a fixed ratio. The 100 templates of the facial model may be stored or loaded at the remote site before all communications are initiated. Using the extracted facial features, the vertices of the polygon can be adjusted to better fit the face of a particular person. In addition, based on the information and processing performed by the feature-matching matching unit 13 and the feature extraction determiner 11, the template of the face model 100 is adapted to allow movement, facial expression, and audio synchronization (ie, voice). To be animated. In essence, the generic face model 100 is dynamically transformed in real time to a particular face. Real-time or non-real-time transmission of model facial parameters / data provides low bit rate animation of synthetic facial models. The data transfer rate is preferably 64 kilobits per second or less, but a data transfer rate of 64 kilobits to 4 megabits per second is acceptable for moving images.

In another embodiment, the time information 12 is used to confirm the result of the feature matching from the feature-matching matching unit 13 and / or to perform an alternative feature matching process. In this embodiment, for example, the matching is only performed by the feature-matching matching unit 13 for the selected frame, preferably the "key" frame (in the MPEG system). Once a keyframe is feature matched, the corresponding matching (ie, depth) in the other non-keyframe frame (or other keyframe) is determined by tracking the corresponding feature point in a time-wise manner. Can be done. 3D motion is scaled in one translational direction from two views (ie the time information 12 may consist of two left or right sequential or consecutive frames) given the first feature correspondence. Can be determined up to. The feature-matching matching unit 13 is used to regularly feature-match other keyframes to eliminate all construction errors from the time feature-match. The feature enabled matching unit 13 may be configured to perform feature enabled matching and time feature matched, respectively, if desired. Temporal feature matching may be performed faster than feature-based matching, which is advantageous for real-time processing.

The invention has numerous applications in fields such as video conferencing and animation / simulation of real objects, or in all applications where modeling of objects is required. For example, typical applications include video games, multimedia creation, and improved navigation on the Internet.

Furthermore, the invention is not limited to 3D face models. The present invention may be used with other models of physical objects and scenes such as 3D models of cars and rooms. In this embodiment, the feature extraction determiner 11 collects information about the location associated with the particular object or scene of interest, eg, the location of the wheel or furniture location. Further processing is based on this information.

While the present invention has been described above with respect to particular embodiments, it is understood that the present invention is not intended to be limited or limited to the embodiments described therein. For example, the invention is not limited to any particular type of filtering or mathematical transformation, or to any particular input image scale or orientation. On the contrary, the present invention
It is intended to cover various structures and modifications of the present invention that fall within the scope of the appended claims and their spirit.

[Brief description of drawings]

FIG. 1 is a schematic front view of a human face model used in three-dimensional model-based coding.

[Fig. 2]   FIG. 3 is a block diagram of a 3D modeling system according to one aspect of the present invention.

3 is a block diagram of an exemplary computer system capable of supporting the system of FIG.

[Figure 4]   3 is a block diagram illustrating the architecture of the computer system of FIG. 2.

[Figure 5]   FIG. 3 is a block diagram illustrating an exemplary arrangement according to a preferred embodiment of the present invention.

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI Theme Coat (reference) H04N 7/24 H04N 7/13 Z (72) Inventor Sharapari, Kiran Netherlands, 5656 Aer Eindouven, Prof. Holstraan 6F Term (Reference) 5B057 BA13 CA13 CA16 CB13 CB16 CG01 CH14 DA07 DB03 DC05 DC32 5C059 MA00 MA05 MB01 MB08 MB12 MB18 PP04 SS07 SS08 SS26 UA02 UA05 UA33 5C061 AA20 AB08 5L096 AA09 CA05 FA69 HA07

Claims (11)

[Claims] 1. At least one feature extraction determiner configured to extract feature position information from a pair of input image signals, and matching with corresponding features in the input image signals according to the feature position information and the difference information. An image processing device having a matching unit coupled to the feature extraction determiner arranged to cause the feature extraction determiner. 2. The image processing apparatus according to claim 1, wherein the matching unit outputs three-dimensional information related to the input image. 3. The image processing apparatus according to claim 2, wherein the 3D surface information is based on a predetermined model. 4. The image processing apparatus according to claim 4, wherein the predetermined model is a human face model. 5. The image processing apparatus according to claim 1, wherein the matching unit performs matching on at least one frame of the input image signal and performs time feature matching on at least the other frame. 6. The image processing device of claim 6, wherein the temporal feature matching is performed using sequential input image frames. 7. The image processing apparatus according to claim 6, wherein one of the frames is a key frame. 8. A method of determining parameters related to a 3D model, comprising: extracting feature position information associated with a pair of input images; and the pair of inputs according to the extracted feature position information and the difference information. A method comprising matching corresponding features in an image, and determining parameters for a 3D model according to a result of corresponding matching of the features. 9. The method of claim 9, wherein the 3D model is a human face model. 10. A method of encoding an object in a digital image for transmission, the method comprising: extracting feature location information associated with at least one pair of digital images; A method comprising matching corresponding features in the pair of digital images, and encoding information for transmission according to the result of the corresponding matching of features. 11. The method of claim 11, wherein the information for transmitting comprises parameters related to a 3D model.