EP1336154A2 - Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features - Google Patents

Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features

Info

Publication number
EP1336154A2
EP1336154A2 EP01992983A EP01992983A EP1336154A2 EP 1336154 A2 EP1336154 A2 EP 1336154A2 EP 01992983 A EP01992983 A EP 01992983A EP 01992983 A EP01992983 A EP 01992983A EP 1336154 A2 EP1336154 A2 EP 1336154A2
Authority
EP
European Patent Office
Prior art keywords
person
image
statistical model
tagged
processing system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP01992983A
Other languages
German (de)
French (fr)
Inventor
Antonio J. Colmenarez
Srinivas Gutta
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips Electronics NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics NV filed Critical Koninklijke Philips Electronics NV
Publication of EP1336154A2 publication Critical patent/EP1336154A2/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • G06V40/173—Classification, e.g. identification face re-identification, e.g. recognising unknown faces across different face tracks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition

Definitions

  • the present invention relates generally to the field of image processing, and more particularly to techniques for implementing a person tagging feature in an image processing system.
  • Person tagging in image processing systems refers generally to the process of characterizing a person observed in an image or sequence of images of a video signal
  • a detected person is "tagged" by association with the characterization, and can thereby be identified as the tagged person in subsequent images.
  • the process of person tagging is thus distinct from a person recognition process in that it does not necessarily involve definitive identification of a given person as being a particular known individual. Instead, it simply generates an indication that a person in a current image is believed to match a person detected in a previous image.
  • the person tagging process is also referred to as person matching.
  • appearance-based algorithms include techniques such as template matching and color histograms.
  • features used in geometry-based algorithms include size, shape, etc.
  • the conventional techniques have been unable to combine appearance and geometric features in a manner which provides more efficient and effective person tagging for an image processing system.
  • the present invention solves the above-noted problem of conventional person tagging techniques by providing a method and apparatus in which appearance features and geometric features are both incorporated into a statistical model of a particular tagged person.
  • the statistical models generated for a given set of persons present in images of a given video segment or other image sequence may be used for detection, location and tracking of the persons in subsequently-processed images.
  • an image processing system processes a sequence of images to generate a statistical model for each of a number of different persons to be tagged so as to be identifiable in subsequent images.
  • the statistical model for a given tagged person incorporates at least one appearance feature, such as color, texture, etc., and at least one geometric feature, such as shape or position of a designated region of similar appearance within one or more images.
  • the models are applied to subsequent images in order to perform a person detection, person location and/or person tracking operation.
  • An action of the image processing system is controlled based on a result of the operation.
  • the statistical model for a given tagged person may be generated by separating one or more images into a number N of
  • the statistical model generated for a given person may be in the form of a likelihood probability function which indicates the likelihood that the person is present in a given image or set of images.
  • a significant advantage of the present invention is that it utilizes statistical models which incorporate both appearance and geometric features.
  • the use of models which combine these different types of features significantly improves the performance of the person tagging process. For example, such an approach ensures that the system will be less likely to confuse persons crossing one another or persons partially occluded by other objects in given image sequence.
  • the present invention can be used in a wide variety of image processing applications, such as video conferencing systems, video surveillance and monitoring systems, and human-machine interfaces.
  • FIG. 1 is a block diagram of an image processing system in which the present invention may be implemented.
  • FIG. 2 illustrates an example person tagging process in accordance with the present invention.
  • FIG. 3 illustrates a translation operation that may be utilized in a person tagging process in accordance with the present invention.
  • FIG. 4 is a flow diagram of an example person tagging process in accordance with the present invention.
  • FIG. 1 shows an image processing system 10 in which person tagging techniques in accordance with the invention may be implemented.
  • the system 10 includes a processor 12, a memory 14, an input/output (I/O) device 15 and a controller 16, all of which are connected to communicate over a set 17 of one or more system buses or other type of interconnections.
  • the system 10 further includes a camera 18 that is coupled to the controller 16 as shown.
  • the camera 18 may be, e.g., a mechanical pan-tilt-zoom (PTZ) camera, a wide- angle electronic zoom camera, or any other suitable type of image capture device.
  • PTZ mechanical pan-tilt-zoom
  • the system 10 may be adapted for use in any of a number of different image processing applications, including, e.g., video conferencing, video surveillance, human- machine interfaces, etc. More generally, the system 10 can be used in any application that can benefit from the improved person tagging capabilities provided by the present invention.
  • the image processing system 10 generates a video signal or other type of sequence of images of a person 20.
  • the camera 18 may be adjusted such that the person 20 comes within a field of view 22 of the camera 18.
  • a video signal corresponding to a sequence of images generated by the camera 18 is then processed in system 10 using the person tagging techniques of the invention, as will be described in greater detail below.
  • An output of the system may then be adjusted based on the detection of a particular tagged person in a given sequence of images.
  • a video conferencing system, human- machine interface or other type of system application may generate a query or other output or take another type of action based on the detection of a tagged person. Any other type of control of an action of the system may be based at least in part on the detection of a tagged person.
  • Elements or groups of elements of the system 10 may represent corresponding elements of an otherwise conventional desktop or portable computer, as well as portions or combinations of these and other processing devices. Moreover, in other embodiments of the invention, some or all of the functions of the processor 12, memory 14, controller 16 and/or other elements of the system 10 may be combined into a single device. For example, one or more of the elements of system 10 may be implemented as an application specific integrated circuit (ASIC) or circuit card to be incorporated into a computer, television, set-top box or other processing device.
  • ASIC application specific integrated circuit
  • processor as used herein is intended to include a microprocessor, central processing unit (CPU), microcontroller, digital signal processor (DSP) or any other data processing element that may be utilized in a given image processing system.
  • memory 14 may represent an electronic memory, an optical or magnetic disk-based memory, a tape-based memory, as well as combinations or portions of these and other types of storage devices.
  • the present invention provides improvements over conventional person tagging techniques through the use of statistical models based on both appearance features and geometric features.
  • tagging refers generally to the generation of a statistical model characterizing a particular person in one or more images of a given image sequence. A person that has been "tagged” in this manner can then be detected, located and/or tracked in one or more subsequent images of the same sequence or of another sequence.
  • FIG. 2 illustrates an example of a person tagging process in accordance with the present invention.
  • An image 25 which includes person 20 is generated and processed in system 10 such that the image is segmented into a number N of different regions of similar appearance.
  • the index r is used to identify a particular one of the regions.
  • ⁇ ) denotes the likelihood probability function of a statistical model generated for a given person ⁇ , and indicates the likelihood that the person ⁇ is present in a given image I.
  • ⁇ ) of the statistical model for person ⁇ may be computed as
  • R r is a function of at least one appearance feature and at least one geometric feature.
  • the appearance features may include color, texture, etc.
  • the geometric features may include region shape as well as relative region position within the image.
  • the general person tagging process illustrated in FIG. 2 involves building statistical models of persons from one or more images and using those models for detection and location of the tagged persons in subsequent images.
  • the process can also be configured to provide tracking of a tagged person, as will now be described in detail in conjunction with FIG. 3.
  • T, ⁇ , ⁇ ) be the likelihood probability function of the statistical model of the person ⁇ .
  • T is a linear transformation used to capture global motion of the person in the image space
  • ⁇ is a discrete variable introduced to capture the state of the local motion of the person at a given point in time, where the term "local motion" is intended to include articulated motion, i.e., the relative motion of different parts of a whole.
  • the position of a person in a room can be obtained from the linear transformation T, while the pose of the person (standing, sitting, etc.) can be determined from the discrete variable ⁇ .
  • FIG. 3 illustrates the operation of the linear transformation T.
  • the linear transformation T is used to obtain a sub-window 30 of the image I that is invariant to rotation and scale. It may be implemented, for example, using a bilinear interpolation technique with a reference point x c in the input image I, a rotation angle ⁇ , and a scaling factor s .
  • the above-noted local motion is modeled using a discrete set of states of the variable ⁇ to capture M different poses of the person ⁇ .
  • the detection and location of the person ⁇ in the image I in the person tagging process of the invention may be implemented using the following maximum likelihood search:
  • Tracking a tagged person in contrast to detection and location, takes advantage of the history of the known positions and poses of the person from previous images, e.g., previous frames of a given video segment.
  • V t ⁇ I 0 , l lt ... , I t ⁇
  • , ⁇ t , T t _ ⁇ r _. , ... , ⁇ ⁇ Qf ⁇ ) is maximized to obtain the optimum trajectory of the person ⁇ i ⁇ , ⁇ Q , T, " , ⁇ 1 . . , T ⁇ * , ⁇ t ) .
  • This maximum likelihood search provides tracking of a tagged person, and can be efficiently implemented using well-known conventional techniques such as the Viterbi algorithm or a forward-backward algorithm.
  • T t _ ⁇ r . . . , T 0 ) characterizes global motion model and could be implemented using, e.g., a Kalman filter
  • ⁇ t _ ⁇ r ... , ⁇ 0 , ⁇ ) characterizes local motion, and could be implemented as a first order Markov model using a transition matrix.
  • different statistical models of the type described above are generated, one for each person present in a given video segment or other type of image sequence.
  • the person tagging process can then provide detection, location and tracking by associating the trajectory of each tagged person with an identifier of the best matching model.
  • a significant advantage of the present invention is that it utilizes statistical models which incorporate both appearance and geometric features.
  • the use of models which combine these different types of features significantly improves the performance of the person tagging process. For example, it ensures that the system will be less likely to confuse persons crossing one another or persons partially occluded by other objects in the sequence of video frames.
  • r is an index to regions of similar appearance and N is the total number of such regions, r - 1, 2, . . . N, so that:
  • r, T, ⁇ , ⁇ ) is the probability of observing the pixel pix assuming that it belongs to the r-th region of the person's model on that pose
  • ⁇ , ⁇ ) is the prior probability of the region at that pose.
  • a dummy region may be added with a constant probability as follows:
  • Every pixel in the image may be characterized by its position x (a two- dimensional vector), and by its appearance features f (color, texture, etc.), so that:
  • r, T, , ⁇ ) may both be approximated as Gaussian distributions over their corresponding feature spaces.
  • the above-noted appearance features vector f can be obtained for a given pixel from the pixel itself or from a designated "neighborhood" of pixels around the given pixel.
  • examples of such appearance features include color and texture. Color features may be determined in accordance with parameters of well-known color spaces such as RGB, HIS, CIE, etc.
  • the texture features may be obtained using well-known conventional techniques such as edge detection, texture gradients, Gabor filters, Tamura feature generation, etc.
  • FIG. 4 is a flow diagram summarizing the above-described person tagging process of the present invention.
  • a video segment or other type of image sequence is processed to generate an appearance and geometry based statistical model P (I
  • the resulting model or set of models is stored in a memory of the image processing system, e.g., in memory 14 of system 10.
  • one or more subsequent images are processed using the stored models to perform at least one of a person detection, person location and person tracking operation.
  • the one or more subsequent images may be subsequent images from the same video segment or other image sequence, or from a different image sequence.
  • the processing operations of steps 40, 42 and 44 may be carried out using software executed by processor 12 of system 10.
  • the techniques of the invention can be implemented using a variety of different person tagging processes, including processes involving any one or more of person detection, person location and person tracking.
  • the invention can be used to provide person tagging capability in a wide variety of applications, including video conferencing systems, video surveillance systems, and other camera-based systems.
  • the invention can be implemented at least in part in the form of one or more software programs which are stored on an electronic, magnetic or optical storage medium and executed by a processing device, e.g., by the processor 12 of system 10.

Abstract

An image processing system (10) processes a sequence of images (25) to generate a statistical model for each of a number of different persons to be tagged so as to be identifiable in subsequent images. The statistical model for a given tagged person incorporates at least one appearance feature, such as color, texture, etc., and at least one geometric feature, such as shape or position of a designated region of similar appearance within one or more images. The models are applied to subsequent images in order to perform a person detection, person location and/or person tracking operation. An action of the image processing system (10) is controlled based on a result of the operation.

Description

Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features
Field of the Invention
The present invention relates generally to the field of image processing, and more particularly to techniques for implementing a person tagging feature in an image processing system.
Background of the Invention
Person tagging in image processing systems refers generally to the process of characterizing a person observed in an image or sequence of images of a video signal, and
» using the characterization to determine if the same person is present in one or more subsequent images. A detected person is "tagged" by association with the characterization, and can thereby be identified as the tagged person in subsequent images. The process of person tagging is thus distinct from a person recognition process in that it does not necessarily involve definitive identification of a given person as being a particular known individual. Instead, it simply generates an indication that a person in a current image is believed to match a person detected in a previous image. The person tagging process is also referred to as person matching.
Conventional person tagging generally involves the use of either appearance- based or geometry-based detection algorithms. The appearance-based algorithms include techniques such as template matching and color histograms. Examples of features used in geometry-based algorithms include size, shape, etc. The conventional techniques, however, have been unable to combine appearance and geometric features in a manner which provides more efficient and effective person tagging for an image processing system.
Summary of the Invention
The present invention solves the above-noted problem of conventional person tagging techniques by providing a method and apparatus in which appearance features and geometric features are both incorporated into a statistical model of a particular tagged person. The statistical models generated for a given set of persons present in images of a given video segment or other image sequence may be used for detection, location and tracking of the persons in subsequently-processed images.
In accordance with one aspect of the invention, an image processing system processes a sequence of images to generate a statistical model for each of a number of different persons to be tagged so as to be identifiable in subsequent images. The statistical model for a given tagged person incorporates at least one appearance feature, such as color, texture, etc., and at least one geometric feature, such as shape or position of a designated region of similar appearance within one or more images. The models are applied to subsequent images in order to perform a person detection, person location and/or person tracking operation. An action of the image processing system is controlled based on a result of the operation.
In accordance with another aspect of the invention, the statistical model for a given tagged person may be generated by separating one or more images into a number N of
* different regions of similar appearance.
In accordance with a further aspect of the invention, the statistical model generated for a given person may be in the form of a likelihood probability function which indicates the likelihood that the person is present in a given image or set of images.
As noted previously, a significant advantage of the present invention is that it utilizes statistical models which incorporate both appearance and geometric features. The use of models which combine these different types of features significantly improves the performance of the person tagging process. For example, such an approach ensures that the system will be less likely to confuse persons crossing one another or persons partially occluded by other objects in given image sequence.
The present invention can be used in a wide variety of image processing applications, such as video conferencing systems, video surveillance and monitoring systems, and human-machine interfaces.
Brief Description of the Drawings
FIG. 1 is a block diagram of an image processing system in which the present invention may be implemented.
FIG. 2 illustrates an example person tagging process in accordance with the present invention.
FIG. 3 illustrates a translation operation that may be utilized in a person tagging process in accordance with the present invention. FIG. 4 is a flow diagram of an example person tagging process in accordance with the present invention.
Detailed Description of the Invention
FIG. 1 shows an image processing system 10 in which person tagging techniques in accordance with the invention may be implemented. The system 10 includes a processor 12, a memory 14, an input/output (I/O) device 15 and a controller 16, all of which are connected to communicate over a set 17 of one or more system buses or other type of interconnections. The system 10 further includes a camera 18 that is coupled to the controller 16 as shown. The camera 18 may be, e.g., a mechanical pan-tilt-zoom (PTZ) camera, a wide- angle electronic zoom camera, or any other suitable type of image capture device. It should therefore be understood that the term "camera" as used herein is intended to include any type of image capture device as well as any configuration of multiple such devices.
The system 10 may be adapted for use in any of a number of different image processing applications, including, e.g., video conferencing, video surveillance, human- machine interfaces, etc. More generally, the system 10 can be used in any application that can benefit from the improved person tagging capabilities provided by the present invention.
In operation, the image processing system 10 generates a video signal or other type of sequence of images of a person 20. The camera 18 may be adjusted such that the person 20 comes within a field of view 22 of the camera 18. A video signal corresponding to a sequence of images generated by the camera 18 is then processed in system 10 using the person tagging techniques of the invention, as will be described in greater detail below. An output of the system may then be adjusted based on the detection of a particular tagged person in a given sequence of images. For example, a video conferencing system, human- machine interface or other type of system application may generate a query or other output or take another type of action based on the detection of a tagged person. Any other type of control of an action of the system may be based at least in part on the detection of a tagged person.
Elements or groups of elements of the system 10 may represent corresponding elements of an otherwise conventional desktop or portable computer, as well as portions or combinations of these and other processing devices. Moreover, in other embodiments of the invention, some or all of the functions of the processor 12, memory 14, controller 16 and/or other elements of the system 10 may be combined into a single device. For example, one or more of the elements of system 10 may be implemented as an application specific integrated circuit (ASIC) or circuit card to be incorporated into a computer, television, set-top box or other processing device.
The term "processor" as used herein is intended to include a microprocessor, central processing unit (CPU), microcontroller, digital signal processor (DSP) or any other data processing element that may be utilized in a given image processing system. In addition, it should be noted that the memory 14 may represent an electronic memory, an optical or magnetic disk-based memory, a tape-based memory, as well as combinations or portions of these and other types of storage devices.
The present invention provides improvements over conventional person tagging techniques through the use of statistical models based on both appearance features and geometric features. The term "tagging" as used herein refers generally to the generation of a statistical model characterizing a particular person in one or more images of a given image sequence. A person that has been "tagged" in this manner can then be detected, located and/or tracked in one or more subsequent images of the same sequence or of another sequence.
FIG. 2 illustrates an example of a person tagging process in accordance with the present invention. An image 25 which includes person 20 is generated and processed in system 10 such that the image is segmented into a number N of different regions of similar appearance. The index r is used to identify a particular one of the regions. In this example, the image 25 is segmented into a total of N= 3 different regions corresponding to portions 26-1, 26-2 and 26-3 of the original image 25. P (I | Ω) denotes the likelihood probability function of a statistical model generated for a given person Ω, and indicates the likelihood that the person Ω is present in a given image I. The likelihood probability function P (I | Ω) of the statistical model for person Ω may be computed as
P(l|Ω) = ∑ P(£r |Ω)P(r|Ω) 5 r = l,2, ... W
where Rr is a function of at least one appearance feature and at least one geometric feature. The appearance features may include color, texture, etc., and the geometric features may include region shape as well as relative region position within the image.
The general person tagging process illustrated in FIG. 2 involves building statistical models of persons from one or more images and using those models for detection and location of the tagged persons in subsequent images. The process can also be configured to provide tracking of a tagged person, as will now be described in detail in conjunction with FIG. 3. Let P (I | T, ξ, Ω) be the likelihood probability function of the statistical model of the person Ω . T is a linear transformation used to capture global motion of the person in the image space, and ξ is a discrete variable introduced to capture the state of the local motion of the person at a given point in time, where the term "local motion" is intended to include articulated motion, i.e., the relative motion of different parts of a whole. For example, the position of a person in a room can be obtained from the linear transformation T, while the pose of the person (standing, sitting, etc.) can be determined from the discrete variable ξ .
FIG. 3 illustrates the operation of the linear transformation T. As shown in the figure, the linear transformation T is used to obtain a sub-window 30 of the image I that is invariant to rotation and scale. It may be implemented, for example, using a bilinear interpolation technique with a reference point xc in the input image I, a rotation angle θ , and a scaling factor s .
The above-noted local motion is modeled using a discrete set of states of the variable ξ to capture M different poses of the person Ω.
The detection and location of the person Ω in the image I in the person tagging process of the invention may be implemented using the following maximum likelihood search:
Tracking a tagged person, in contrast to detection and location, takes advantage of the history of the known positions and poses of the person from previous images, e.g., previous frames of a given video segment. For a video segment Vt = {I0, llt ... , It} , the likelihood probability P (Vt | , ξt, Tt_ι r _. , ... , ^ ξQf Ω) is maximized to obtain the optimum trajectory of the person {iζ, ξQ, T, " , ξ1 . . , T^ *, ξt) .
This maximum likelihood search provides tracking of a tagged person, and can be efficiently implemented using well-known conventional techniques such as the Viterbi algorithm or a forward-backward algorithm.
The likelihood probability of a video sequence can be written in terms of the likelihood probability of individual frames as follows: p (vt i ^, 4, ^, ^, ... , ^, ^) =
P (It I ,, ξtf CDPi , I •ς_1, ... , ι-„ Ω> P(& I ξt_lf ... r ξ0, Ω)
where P (Tt | Tt_ι r . . . , T0) characterizes global motion model and could be implemented using, e.g., a Kalman filter, and P [ξt | ξt_ιr ... , ξ0, Ω) characterizes local motion, and could be implemented as a first order Markov model using a transition matrix.
In accordance with the invention, different statistical models of the type described above are generated, one for each person present in a given video segment or other type of image sequence. The person tagging process can then provide detection, location and tracking by associating the trajectory of each tagged person with an identifier of the best matching model.
As noted previously, a significant advantage of the present invention is that it utilizes statistical models which incorporate both appearance and geometric features. The use of models which combine these different types of features significantly improves the performance of the person tagging process. For example, it ensures that the system will be less likely to confuse persons crossing one another or persons partially occluded by other objects in the sequence of video frames.
The generation of the statistical models based on both appearance and geometric features will now be described in greater detail. For simplicity and clarity of illustration, the pixels in an image I of a person Ω may be considered independent from one another. In other words,
P (I I T, ξ, Ω) = ∑ P (pix I T, ξ, Ω) . pix e l
As previously noted in conjunction with FIG. 2, r is an index to regions of similar appearance and N is the total number of such regions, r - 1, 2, . . . N, so that:
P (pix I T, ξ, Ω) = max [P (pix I r, T, ξ, Ω) P(r | ξ, Ω)J, r = 1, ... , Ν
where P (pix | r, T, ξ, Ω) is the probability of observing the pixel pix assuming that it belongs to the r-th region of the person's model on that pose, and P(r | ξ, Ω) is the prior probability of the region at that pose. In order to handle occlusions and new exposures, a dummy region may be added with a constant probability as follows:
P (piX I -occlusion' "' ζ' ^) P (r 0cclusion ' S' ^) = "occlusion •
Every pixel in the image may be characterized by its position x (a two- dimensional vector), and by its appearance features f (color, texture, etc.), so that:
P (pix I r, T, ξ, Ω) = P (x | r, T, ξ, Ω) P (f | r, T, ξ, Ω),
where P (x | r, T, ξ, Ω) and P (f | r, T, , Ω) may both be approximated as Gaussian distributions over their corresponding feature spaces. The above-noted appearance features vector f can be obtained for a given pixel from the pixel itself or from a designated "neighborhood" of pixels around the given pixel. As previously noted, examples of such appearance features include color and texture. Color features may be determined in accordance with parameters of well-known color spaces such as RGB, HIS, CIE, etc. The texture features may be obtained using well-known conventional techniques such as edge detection, texture gradients, Gabor filters, Tamura feature generation, etc.
FIG. 4 is a flow diagram summarizing the above-described person tagging process of the present invention. In step 40, a video segment or other type of image sequence is processed to generate an appearance and geometry based statistical model P (I | T, ξ, Ω) for each person Ω to be tagged. In step 42, the resulting model or set of models is stored in a memory of the image processing system, e.g., in memory 14 of system 10. Finally, in step 44, one or more subsequent images are processed using the stored models to perform at least one of a person detection, person location and person tracking operation. The one or more subsequent images may be subsequent images from the same video segment or other image sequence, or from a different image sequence. The processing operations of steps 40, 42 and 44 may be carried out using software executed by processor 12 of system 10.
The above-described embodiments of the invention are intended to be illustrative only. For example, the techniques of the invention can be implemented using a variety of different person tagging processes, including processes involving any one or more of person detection, person location and person tracking. In addition, the invention can be used to provide person tagging capability in a wide variety of applications, including video conferencing systems, video surveillance systems, and other camera-based systems. Furthermore, the invention can be implemented at least in part in the form of one or more software programs which are stored on an electronic, magnetic or optical storage medium and executed by a processing device, e.g., by the processor 12 of system 10. These and numerous other embodiments within the scope of the following claims will be apparent to those skilled in the art.

Claims

CLAIMS:
1. A method of person tagging in an image processing system (10), the method comprising the steps of: processing (40) sequence of images (25) to generate a statistical model for each person to be tagged, the statistical model incorporating at least one appearance feature and at least one geometric feature of the tagged person; applying (44) the model to at least one subsequent image in order to perform at least one of a detection operation, a location operation and a tracking operation for the tagged person; and controlling an action of the image processing system (10) based on a result of the at least one operation.
2. The method of claim 1 wherein the sequence of images (25) comprises a video segment.
3. The method of claim 1 wherein the processing step (40) further includes processing the sequence of images(25) to generate a plurality of statistical models, each of the models corresponding to a particular tagged person.
4. The method of claim 1 wherein the appearance feature comprises at least one of a color feature and a texture feature.
5. The method of claim 1 wherein the geometric feature comprises at least one of a region shape and a region position for a given one of a plurality of regions associated with the statistical model.
6. The method of claim 1 wherein the statistical model is generated at least in part by segmenting a given image (25) into a number N of different regions (26-1, 26-2, 26-3) of similar appearance.
7. The method of claim 1 wherein the statistical model generated for a given person Ω comprises a likelihood probability function P (I | Ω) which indicates the likelihood that the person Ω is present in a given image I.
8. The method of claim 7 wherein the likelihood probability function P (I I Ω) for person Ω is computed as
P(l|Ω) = ∑ P(Λr |Ω)P(r|Ω) f r = l,2, ... N
where Rr is a function of the at least one appearance feature and the at least one geometric feature, and r is an index identifying one of N regions of similar appearance within the image I .
9. The method of claim 1 wherein the statistical model generated for a given person Ω comprises a likelihood probability function P (I | T, ξ, Ω) , where T is a linear transformation used to capture global motion of the person in an image I, and ξ is a discrete variable used to capture local motion of the person at a given point in time.
10. The method of claim 9 wherein a location of the person is determined using the linear transformation .
11. The method of claim 9 wherein the linea* transformation T is used to obtain a sub-window (30) of the image I that is invariant to rotation and scale.
12. The method of claim 9 wherein the linear transformation T is implemented using a bilinear interpolation technique with a reference point xc in the image I, a rotation angle θ , and a scaling factor s .
13. The method of claim 9 wherein the local motion is modeled using a discrete set of states {ξλ,ξ2, ... ξM} of the variable ξ to capture M different poses of the person Ω.
14. The method of claim 1 wherein the statistical model generated for a given person Ω and image I comprises a likelihood probability function
P (I I T, ξ, Ω) = J P (pix I T, ξ, Ω) , pixe l
where r is an index to regions of similar appearance and N is a total number of such regions, r = 1, 2, . . . N and
P (pix I T, ξ, Ω) = max [p (pix \ r, T, ξ, Ω) P(r | ξ, Ω)], r = 1, ... , W
where P (pix I r, T, ξ, Ω) is the probability of observing pixel pix assuming that it belongs to an r-th region of the model on a pose ξ, and P(r | ξf Ω) is the prior probability of the region at that pose.
15. The method of claim 14 wherein the regions of similar appearance include a dummy region having a constant probability as follows:
P (P^-X I rocclusion' ^' i ") (rocclusion I ζ' ^) = Poccιusl0n •
16. The method of claim 14 wherein each of at least a subset of the pixels of the image I is characterized by a two-dimensional position vector x and by an appearance feature vector f such that:
P (pix I r, T, ξ, Ω) = P (x | r, T, ξ, Ω) P (f | r, T, ξ, Ω),
where P (x | r, T, ξ, Ω) and P (f | r, T, ξ, Ω) are approximated as Gaussian distributions over corresponding feature spaces.
17. The method of claim 1 wherein the controlling step comprises generating an output of the image processing system based on the result of the at least one operation.
18. The method of claim 1 wherein the controlling step comprises altering an operating parameter of the image processing system (10) based on the result of the at least one operation.
19. An apparatus for use in providing person tagging in an image processing system (10), the apparatus comprising: a processor (12) operative to process a sequence of images to generate a statistical model for each person to be tagged, the statistical model incoφorating at least one appearance feature and at least one geometric feature of the tagged person, the processor (12) being further operative to apply the model to at least one subsequent image in order to perform at least one of a detection operation, a location operation and a tracking operation for the tagged person, and further wherein an action of the image processing system (10) is controlled based on a result of the at least one operation.
20. An article of manufacture comprising a storage medium for storing one or more programs for use in providing person tagging in an image processing system (10), wherein the one or more programs when executed by a processor (12) implement the steps of: processing (40) a sequence of images (25) to generate a statistical model for each person to be tagged, the statistical model incoφorating at least one appearance feature and at least one geometric feature of the tagged person; and applying (44) the model to at least one subsequent image in order to perform at least one of a detection operation, a location operation and a tracking operation for the tagged person; wherein an action of the image processing system (10) is controlled based on a result of the at least one operation.
EP01992983A 2000-11-01 2001-10-17 Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features Withdrawn EP1336154A2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US703423 2000-11-01
US09/703,423 US6973201B1 (en) 2000-11-01 2000-11-01 Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features
PCT/EP2001/012077 WO2002037415A2 (en) 2000-11-01 2001-10-17 Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features

Publications (1)

Publication Number Publication Date
EP1336154A2 true EP1336154A2 (en) 2003-08-20

Family

ID=24825325

Family Applications (1)

Application Number Title Priority Date Filing Date
EP01992983A Withdrawn EP1336154A2 (en) 2000-11-01 2001-10-17 Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features

Country Status (5)

Country Link
US (2) US6973201B1 (en)
EP (1) EP1336154A2 (en)
JP (1) JP2004513442A (en)
CN (1) CN1332349C (en)
WO (1) WO2002037415A2 (en)

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8954432B2 (en) 2000-11-15 2015-02-10 Mark Frigon Users tagging users in photos online
US7912246B1 (en) 2002-10-28 2011-03-22 Videomining Corporation Method and system for determining the age category of people based on facial images
GB2414615A (en) 2004-05-28 2005-11-30 Sony Uk Ltd Object detection, scanning and labelling
DE602006017812D1 (en) * 2005-03-17 2010-12-09 British Telecomm METHOD FOR TRACKING OBJECTS IN A VIDEO SEQUENCE
US7945653B2 (en) * 2006-10-11 2011-05-17 Facebook, Inc. Tagging digital media
US10614294B1 (en) 2006-06-16 2020-04-07 Videomining Corporation Method and system for measuring viewership of people for displayed object
KR100793989B1 (en) * 2006-07-11 2008-01-16 삼성전자주식회사 Method for classifing category of photographic and system thereof
US8131011B2 (en) * 2006-09-25 2012-03-06 University Of Southern California Human detection and tracking system
US20080111814A1 (en) * 2006-11-15 2008-05-15 Yahoo! Inc. Geometric tagging
US7848548B1 (en) 2007-06-11 2010-12-07 Videomining Corporation Method and system for robust demographic classification using pose independent model from sequence of face images
WO2009039288A1 (en) * 2007-09-19 2009-03-26 Panasonic Corporation System and method for identifying objects in an image using positional information
US8254684B2 (en) * 2008-01-02 2012-08-28 Yahoo! Inc. Method and system for managing digital photos
US8325976B1 (en) * 2008-03-14 2012-12-04 Verint Systems Ltd. Systems and methods for adaptive bi-directional people counting
US20090316961A1 (en) * 2008-06-21 2009-12-24 Microsoft Corporation Method for tagging image content
US8284258B1 (en) * 2008-09-18 2012-10-09 Grandeye, Ltd. Unusual event detection in wide-angle video (based on moving object trajectories)
US9110927B2 (en) * 2008-11-25 2015-08-18 Yahoo! Inc. Method and apparatus for organizing digital photographs
US8422781B2 (en) * 2008-12-03 2013-04-16 Industrial Technology Research Institute Methods and systems for creating a hierarchical appearance model
US8965044B1 (en) * 2009-06-18 2015-02-24 The Boeing Company Rotorcraft threat detection system
KR101064952B1 (en) * 2009-11-23 2011-09-16 한국전자통신연구원 Method and apparatus for providing human body parts detection
KR101272611B1 (en) * 2009-12-07 2013-06-10 한국전자통신연구원 Method and Apparatus for Analyzing Nudity of Image Using Body Parts Detection Models and Managing Images Database Using Nudity and Body Parts in Pornography
US8645287B2 (en) * 2010-02-04 2014-02-04 Microsoft Corporation Image tagging based upon cross domain context
US8532390B2 (en) 2010-07-28 2013-09-10 International Business Machines Corporation Semantic parsing of objects in video
US10424342B2 (en) 2010-07-28 2019-09-24 International Business Machines Corporation Facilitating people search in video surveillance
US8649592B2 (en) 2010-08-30 2014-02-11 University Of Illinois At Urbana-Champaign System for background subtraction with 3D camera
US8510166B2 (en) * 2011-05-11 2013-08-13 Google Inc. Gaze tracking system
US8860787B1 (en) 2011-05-11 2014-10-14 Google Inc. Method and apparatus for telepresence sharing
NO333851B1 (en) 2012-01-23 2013-09-30 Ares Turbine As Procedure and system for registration of studded tires on vehicles.
US8862764B1 (en) 2012-03-16 2014-10-14 Google Inc. Method and Apparatus for providing Media Information to Mobile Devices
US9892203B2 (en) * 2012-10-29 2018-02-13 Dropbox, Inc. Organizing network-stored content items into shared groups
CN103886315B (en) * 2012-12-21 2017-05-24 本田技研工业株式会社 3d Human Models Applied To Pedestrian Pose Classification
US9355334B1 (en) * 2013-09-06 2016-05-31 Toyota Jidosha Kabushiki Kaisha Efficient layer-based object recognition
US9542626B2 (en) * 2013-09-06 2017-01-10 Toyota Jidosha Kabushiki Kaisha Augmenting layer-based object detection with deep convolutional neural networks
US9485433B2 (en) 2013-12-31 2016-11-01 Personify, Inc. Systems and methods for iterative adjustment of video-capture settings based on identified persona
US9317785B1 (en) 2014-04-21 2016-04-19 Video Mining Corporation Method and system for determining ethnicity category of facial images based on multi-level primary and auxiliary classifiers
US9916668B2 (en) 2015-05-19 2018-03-13 Personify, Inc. Methods and systems for identifying background in video data using geometric primitives
US9563962B2 (en) 2015-05-19 2017-02-07 Personify, Inc. Methods and systems for assigning pixels distance-cost values using a flood fill technique
US9883155B2 (en) 2016-06-14 2018-01-30 Personify, Inc. Methods and systems for combining foreground video and background video using chromatic matching
US9881207B1 (en) * 2016-10-25 2018-01-30 Personify, Inc. Methods and systems for real-time user extraction using deep learning networks
CN107133646B (en) * 2017-05-03 2019-09-17 山东合天智汇信息技术有限公司 A kind of method and system by human behavior track identification emphasis personnel

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4051458A (en) 1976-05-24 1977-09-27 Bausch & Lomb Incorporated Video amplitude related measurements in image analysis
DE3602299A1 (en) 1986-01-27 1987-07-30 Bernd A Langenfeld Person detection and identification system
US5185815A (en) * 1991-11-04 1993-02-09 Grumman Aerospace Corporation Multiple target correlator system
US5550928A (en) * 1992-12-15 1996-08-27 A.C. Nielsen Company Audience measurement system and method
JPH0863603A (en) * 1994-06-15 1996-03-08 Olympus Optical Co Ltd Image analyzer
EP0727760A3 (en) 1995-02-17 1997-01-29 Ibm Produce size recognition system
DE69636578T2 (en) 1995-06-09 2007-07-05 Interact Medical Technologies Corp. ANATOMICAL DISPLAY SYSTEM
JP3625910B2 (en) * 1995-09-11 2005-03-02 松下電器産業株式会社 Moving object extraction device
US5825908A (en) 1995-12-29 1998-10-20 Medical Media Systems Anatomical visualization and measurement system
US5999651A (en) * 1997-06-06 1999-12-07 Matsushita Electric Industrial Co., Ltd. Apparatus and method for tracking deformable objects
US6185314B1 (en) * 1997-06-19 2001-02-06 Ncr Corporation System and method for matching image information to object model information
US6188777B1 (en) * 1997-08-01 2001-02-13 Interval Research Corporation Method and apparatus for personnel detection and tracking
US6542621B1 (en) * 1998-08-31 2003-04-01 Texas Instruments Incorporated Method of dealing with occlusion when tracking multiple objects and people in video sequences
JP4392886B2 (en) * 1999-01-22 2010-01-06 キヤノン株式会社 Image extraction method and apparatus
US6590999B1 (en) * 2000-02-14 2003-07-08 Siemens Corporate Research, Inc. Real-time tracking of non-rigid objects using mean shift

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO0237415A2 *

Also Published As

Publication number Publication date
US7181046B2 (en) 2007-02-20
CN1423795A (en) 2003-06-11
WO2002037415A3 (en) 2002-10-03
US20060002589A1 (en) 2006-01-05
US6973201B1 (en) 2005-12-06
CN1332349C (en) 2007-08-15
JP2004513442A (en) 2004-04-30
WO2002037415A2 (en) 2002-05-10

Similar Documents

Publication Publication Date Title
US7181046B2 (en) Person tagging in an image processing system utilizing a statistical model based on both appearance and geometric features
US10339386B2 (en) Unusual event detection in wide-angle video (based on moving object trajectories)
Nguyen et al. Fast occluded object tracking by a robust appearance filter
TW202201944A (en) Maintaining fixed sizes for target objects in frames
US8873798B2 (en) Methods for tracking objects using random projections, distance learning and a hybrid template library and apparatuses thereof
WO2022078041A1 (en) Occlusion detection model training method and facial image beautification method
WO2022083504A1 (en) Machine-learning model, methods and systems for removal of unwanted people from photographs
JP2022534666A (en) Image processing method, device, electronic device and storage medium
US20030052971A1 (en) Intelligent quad display through cooperative distributed vision
WO2022040886A1 (en) Photographing method, apparatus and device, and computer-readable storage medium
Dockstader et al. Tracking multiple objects in the presence of articulated and occluded motion
CN115019241B (en) Pedestrian identification and tracking method and device, readable storage medium and equipment
Xin et al. Automatic 3D face modeling from video
Demirdjian Combining geometric-and view-based approaches for articulated pose estimation
CN112184766B (en) Object tracking method and device, computer equipment and storage medium
De la Torre et al. View-based adaptive affine tracking
Lerdsudwichai et al. Algorithm for multiple faces tracking
Javadi et al. Video stabilization and tampering detection for surveillance systems using homography
Zhang et al. Semantic saliency driven camera control for personal remote collaboration
JP3426076B2 (en) Gesture video recognition method
Raskin et al. Using gaussian processes for human tracking and action classification
Ozer et al. Video analysis for smart rooms
TWI819219B (en) Photographing method for dynamic scene compensation and a camera using the method
Tran et al. A robust framework for tracking simultaneously rigid and non-rigid face using synthesized data
Kosmopoulos et al. Human behavior classification using multiple views

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20030602

AK Designated contracting states

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

AX Request for extension of the european patent

Extension state: AL LT LV MK RO SI

17Q First examination report despatched

Effective date: 20071115

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20080326