Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/107—Static hand or arm

Definitions

• the present invention relates to a method of extracting, in a recorded image, an object situated in the foreground of a projected background. It also relates to an extraction module and a computer program for implementing said method.
• the invention finds a particularly advantageous application in the field of the automatic extraction of objects on the background of projected images, shared between an interlocutor and a remote assembly, for viewing by the remote assembly of the gestures of the interlocutor around projected images.
• the fields of application of the invention are therefore multiple. Examples include videoconferencing, distance learning, television presentations, and so on.
• the invention relates more particularly to situations where it is desired to retransmit to remote persons a scene constituted by a speaker who, during a presentation for example, designates with his hands areas of interest, such as a formula, a diagram, a map, located on a digital image projected on a monitor, a video projection screen or rear projection.
• areas of interest such as a formula, a diagram, a map, located on a digital image projected on a monitor, a video projection screen or rear projection.
• the visualization of the gestures of a distant interlocutor is also essential to promote the mutual awareness of users of remote collaborative platforms. It can be seen that starting from three users the coordination of the actions of the different actors becomes problematic. Visualization of the gestures of each remote user makes it possible to better identify the author of an action in progress and also to become aware of the intentionalities of each one.
• the invention therefore also extends to gestural interfaces on screens, monitors or graphic tables.
• the extraction of the arm and His hand from the user is essential for the identification of the gesture and the associated interaction.
• a first way to transmit to distant people a scene of a speaker speaking in front of a projected background is to record it with a video camera and retransmit to remote people through a telecommunication network.
• one solution consists in sharing between the speaker and the remote assembly the same digital images forming the projected background, extracting from the recorded image the gestures of the user, to transmit them to the remote persons and to insert them into the shared images.
• This known method is based on an analysis of local characteristics extracted from the background image, in particular by the discrete cosine transform (DCT) method (Discrete Cosine Transform).
• DCT discrete cosine transform
• the background model is estimated as a block of pixels per block of pixels, by learning on a sequence of background images, according to a hypothesis of Gaussian distributions independent of the local characteristics. These characteristics are then estimated on the current image, and the pixels, or groups of pixels, which do not satisfy the model learned, according to a given thresholding criterion, are considered as belonging to the objects of the foreground.
• a progressive temporal update of the bottom model is carried out by means of a linear weighting of the learning parameter between the local characteristics of the background model and those from the current image.
• the technical problem to be solved by the object of the present invention is to propose a method of extracting, in a recorded image, an object situated in the foreground of a projected background, which would make it possible to obtain a object extracted reliable and insensitive to fluctuations in position, lighting and composition of the background due to changes in projected images that may occur during the recording of the scene.
• said object as the set of elements of the recorded image having a deviation from said correspondence law.
• the invention is based on the fact that the projected background is known a priori as a digital image stored for example in a personal computer.
• the recorded image of the entire scene consisting of the background in the background and the object to be extracted in the foreground is also known in digitized form, so that a very close comparison can be made between the background projected and the recorded background, allowing to establish the law of correspondence sought with a lot of precision, which guarantees a great robustness to the extraction carried out.
• the method according to the invention is insensitive to the bottom of the position variations and the variations in illumination, the latter being automatically taken into account.
• the projected background can be any and modified during the time, such as a videogram, a graphical user interface, etc.
• said correspondence law is described by a geometric transformation function H and a light transfer function T by means of the relation;
• a real-time update is performed of the geometric transformation function H and the light transfer function T.
• the updating of the function H is however not necessary if the mechanical device coupling the projection surface to the recording camera is rigid.
• the light transfer function T can be limited to intensity only in terms of gray levels s with the advantage of a low computing load, or extended to each of the three color channels whatever the representation (RGB, Lab “ Luv, Yuv, IHS “ StC) 8 or any other focal characteristics such as those associated with a Gabor filter bank to account for its texture of images (HG Feissoninger and T. Strohmer, "Gabor Analysis and Afgoritms, "Applied and Numericai Harmony Analysis, Birkhauser Boston Inc., Boston, MA 5 1998).
• said method comprises an initialization step comprising:
• said initialization step further comprises an estimation of the geometric transformation function H and the light transfer function T of minimizing the function F (H, T):
• the quality of the image of the object obtained after the extraction step can be improved because, according to the invention, said method comprises a post-processing step of regularizing the extracted object. Regularization is understood to mean the operations of eliminating background areas still present in the extracted object and eliminating false detections of the extracted object appearing in the background.
• the extraction step includes a preliminary step of adjusting the channel intensity bright f from the bottom.
• the invention also relates to an extraction module, in a recorded image, of an object situated in the foreground of a projected background, remarkable in that said module comprises means for:
• said object as the set of elements of the recorded image having a deviation from said correspondence law.
• said module comprises means for calculating a geometric transformation function H and a light transfer function T minimizing the function F (H, T):
• said module comprises post-processing means able to regulate the extracted object.
• said module comprises channel adjustment means derived from the light intensity I of the bottom.
• the invention further relates to a computer program comprising program code instructions for carrying out the steps of the method according to the invention when said program is executed on a computer.
• Figure 1a is a diagram of a first embodiment of the invention.
• Figure 1b is a diagram of a second embodiment of the invention.
• Figure 2a shows a projected background constituting a background for a recorded image.
• FIG. 2b shows a recorded image of an object in the foreground on the background of Figure 2a.
• FIG. 2c represents the distant image obtained by applying the method according to the invention to the recorded image of FIG. 2b.
• FIG. 3 is a general diagram of the extraction process according to the invention.
• FIG. 4 is a diagram of the initialization step of the method of FIG. 3.
• FIG. 5 is a diagram of the extraction step of the method of FIG.
• FIG. 1a shows a first exemplary embodiment of a method for extracting automatically from an image recorded by a video camera 20 an object 1 situated in the foreground of a background 10 formed from images driven by a personal computer 30 and projected on a surface 10 by a device 11 of video projection or backprojection, as shown in Figures 1a and 1b.
• the projection surface is a digital table 10 '.
• the object to be extracted is the arm 1 of an intervener during a videoconference to which remote persons are present.
• the video camera 20 is connected to a telecommunication network 2 capable of transmitting the digital image supplied at the output of the camera 20 by a module 40 responsible for extracting the object 1, in accordance with the method of the invention. 'invention.
• Figure 2a shows an example of a background 10 projected on surfaces 10 or 10 * .
• FIG. 2b The image of the interference recorded directly by the camera 20 is given in FIG. 2b. It can be seen in this figure that the background is of poor quality to the point of making it unreadable by a remote person receiving this image in the state.
• remote persons have the same background images as the intervener " which does not poses no difficulty with regard to images that can be transmitted as digital files or when these images available on each of the remote stations are viewed synchronously. It then suffices to extract the object 1 from the image recorded by the camera 20 by means of the extraction module 40, to transmit to the remote persons the object 1 thus extracted and to superimpose it locally at the bottom 10. in this way the image of Figure 2c where the extracted object appears on a background of good quality.
• the extraction in the foreground / background made by the module 40 is based on a priori knowledge of the background, which is an image projected on a flat surface, 10 or 10 '. This background image will be noted later.
• the camera 20 records the scene constituted by the projected image and any objects 1 placed in the foreground. We call E the image recorded by the camera.
• the image recorded by the camera 20 of the projected background is known to a geometric transformation function H and a near light transfer function T.
• the geometric transformation function H is modeled in the general case by homography or affinity if the focal axis of the camera is substantially perpendicular to the projection surface.
• the index i will be omitted when it will not be necessary.
• the extraction method implemented by the module 40 is based on the construction of a correspondence faith between the pixels, or pixels, of the projected background and the rear of the image recorded by the camera.
• FIG. 3 indicates the different steps of an extraction method according to the invention, namely an initialization step, an extraction step proper. and a post-processing step.
• the object obtained at the end of all the steps of the method is transmitted for example to all remote participants in a videoconference.
• the actual extraction step will now be described in detail with reference to FIG.
• This step begins with the calculation of the analysis channels from the video signals.
• These channels can be limited to the single intensity, or correspond to the three color channels (RGB, Lab, Luv, etc.) (G. Wyszecki and WS Stiles, Color Science: Concepts and Methods, John Wiley and Sons, 1982) and also local analyzes, such as averages, variances, Gabor filters, etc.
• estimating the transfer function T on each of the channels requires sufficient data. Typically, this requires knowing for each channel the interval [Li n , ImaJ where i m i n and l max are the minimum and maximum values observed for! for this channel. Outside of these values, the transfer function can not be estimated because no data is available. Rather than modifying the starting set of the transfer function according to the original image, it is preferable to impose a transformation of [Q, 255] to [Emin, E max
• the next step is to estimate the H and T functions.
• the geometric transformation function H can be represented in the general case of planar projection surface by a homography defined by 8 parameters denoted a.b ⁇ .d ⁇ .f.g and h according to the usual definition:
• the geometric transformation H can be defined by 6 parameters a, b, c, d, e and f according to the following definition: ax + by + c dx + ey + f
• the transfer function T is modeled by a decomposition on the basis of generating functions that can be indifferently wavelets (S. Mallat, “Wavelet Tour of Signal Processing", Academy Press, 2nd Edition, 1999) or splines (B. Chatmond, “Modeling and Inverse Problem in Image Analysis,” Applied Mathematics, Springer Verlag, Vol.155, Chap.3, pp.53-57, 2003).
• functions can be indifferently wavelets (S. Mallat, "Wavelet Tour of Signal Processing", Academy Press, 2nd Edition, 1999) or splines (B. Chatmond, “Modeling and Inverse Problem in Image Analysis,” Applied Mathematics, Springer Verlag, Vol.155, Chap.3, pp.53-57, 2003).
• IRLS Intelligent Least Square Algorithm
• h ⁇ represents the ith parameter of H and r the number of parameters of H.
• step t we denote by Ht the estimate of H: * we minimize F ° H t in ⁇ , at fixed Ht: linear in ⁇ , the function F ° H t allows minimization according to the IRLS;
• a stopping criterion chosen may be the following: iteration as long as the ratio (F 0 H 4 + 1 - F 0 H) / F ° H t is greater than a threshold.
• Ht is the estimate of H:
• the iteration is continued as long as the stopping criterion is not checked.
• a stopping criterion retained may be the following: iteration as long as the ratio (F 0 H 1 + 1 - F 0 H 4 ) / F 0 H 1 is greater than a threshold.
• the actual extraction implements the M-emitters whose interesting property is to offer a measure, called weight, of the adequacy between the observation and the law of correspondence.
• This weight takes a value close to 0 when the error ⁇ s is large, in this case the pixel does not follow the law of correspondence.
• the pixel is a foreground pixel if the weight ⁇ s estimated for at least one of the channels i used is less than 0.25.
• the number of iterations of the IRLS per image can be increased to reduce this disadvantage, within the limit of 25Hz of refreshment.
• Another possibility is to perform an iteration by image; given the small number of iterations, the update of the estimate will not last more than one second.
• a final step can be performed to regularize the binary segmentation. This is to eliminate from the image provided by the module 40 the background elements present in the extracted object and, conversely, the background elements identified as to be extracted.
• One possible method is to successively use a morphological erosion filter and then a dilation filter (CR Giardina and E. R. Dougherty, Morphological Methods in Image and Signal Processing, Englewood Cliffs, New Jersey: Prentice-Haff, 321, 1988).
• the erosion and expansion range selected is 2,
• a rough estimate of H is made manually by pointing to the image recorded four particular points of the projected image: the four corners for example if the recorded image contains the entire projected image, or other points in the image. opposite case.
• the correspondence between the four points of the projected image and their projection on the recorded image provides eight linear equations making it possible to obtain an identification of the six or eight parameters of the geometric transformation either by inversion of a direct linear system in the case homography, or by least-squares minimization in the case of affinity.
• the transfer function T or the transfer functions Ti are initialized with the identity function.

Landscapes

• Engineering & Computer Science (AREA)
• Human Computer Interaction (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• Theoretical Computer Science (AREA)
• Image Processing (AREA)
• Silver Salt Photography Or Processing Solution Therefor (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=37442055

Family Applications (1)

Country Status (4)

Families Citing this family (7)

Family Cites Families (6)

• 2005
  • 2005-12-02 FR FR0553697A patent/FR2894353A1/fr active Pending
• 2006
  • 2006-12-04 US US12/085,976 patent/US9036911B2/en not_active Expired - Fee Related
  • 2006-12-04 WO PCT/FR2006/051276 patent/WO2007063262A2/fr not_active Ceased
  • 2006-12-04 EP EP06842087A patent/EP1964057A2/de not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events