Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/084—Backpropagation, e.g. using gradient descent
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/048—Activation functions
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/40—Extraction of image or video features
  • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
  • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/168—Feature extraction; Face representation
  • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships

Definitions

• a system and method for locating points of interest in an object image implementing a neural network is disclosed.
• the field of the invention is that of digital processing of still or moving images. More specifically, the invention relates to a technique for locating one or more point (s) of interest in an object represented on a digital image.
• the invention finds in particular, but not exclusively, an application in the field of the detection of physical characteristics in the faces present on a digital or digitized image, such as the pupil of the eye, the corner of the eyes, the tip of the nose , mouth, eyebrows, etc.
• a digital or digitized image such as the pupil of the eye, the corner of the eyes, the tip of the nose , mouth, eyebrows, etc.
• the automatic detection of points of interest in face images is a major issue in the field of facial analysis.
• the detectors used are based on an analysis of the chrominance of the face: the pixels of the face are labeled as belonging to the skin or to the facial elements according to their color. Other detectors use contrast variations. For this, a contour detector is applied, based on the analysis of the light gradient. We then try to identify the facial elements from the shape of the different contours detected.
• Some known techniques implement a second phase of applying a geometric face model to all candidate positions determined in the first phase of independent detection of each element.
• the elements detected in the initial phase form constellations of candidate positions and the geometric model that can be deformable makes it possible to select the best constellation.
• a recent method allows to get rid of the classic two-step schema (independent search of facial elements followed by the application of geometric rules).
• This method is based on the use of Active Models of Appearance (AAMs) and is described in particular by D. Chrisce and T. Cootes, in "A comparison of shape constrained facial feature detectors" (Proceedings of the 6 * International Conference on Automatic Face and Gesture Recognition 2004, Seoul, Korea, pages 375-380, 2004). It consists in predicting the position of the facial elements by trying to match an active face model on the face in the image, by adapting the parameters of a linear model combining form and texture.
• AAMs Active Models of Appearance
• detectors designed specifically for the detection of different facial elements do not withstand the extreme conditions of illumination of images, such as over-lighting or under lighting, side lighting, from below. They are also not very robust vis-à-vis the variations in image quality, especially in the case of low resolution images from video streams (acquired for example by means of a "webcam") or previously compressed.
• Methods based on chrominance analysis are also sensitive to lighting conditions. In addition, they can not be applied to grayscale images.
• the statistical models used being linear, created by ACP, they are not robust to the overall variations of the image, in particular the variations of lighting. They are also not very robust with regard to partial occlusions of the face.
• the invention particularly aims to overcome these disadvantages of the prior art.
• an object of the invention is to provide a technique for locating several points of interest in an image representative of an object that does not require long and tedious development of filters specific to each point of interest that the we want to be able to locate, and to each type of object.
• Another object of the invention is to propose such a localization technique that is particularly robust to all noises that can affect the image, such as illumination conditions, chromatic variations, partial occlusions, etc.
• the invention also aims to provide such a technique that takes into account occultations partially affecting the images, and that allows the inference of the position of the occulted points.
• the invention also aims to propose such a technique that is simple to implement and inexpensive to implement.
• Another object of the invention is to provide such a technique which is particularly well suited to the detection of facial elements in face images.
• a system for locating at least two points of interest in an object image which implements an artificial neural network and has a layered architecture comprising: an input layer receiving said object image; at least one intermediate layer, called the first intermediate layer, comprising a plurality of neurons making it possible to generate at least two saliency maps each associated with a predefined distinct point of interest of said object image; at least one output layer comprising said saliency maps, which themselves comprise a plurality of neurons each connected to all the neurons of said first intermediate layer.
• Said points of interest are localized, in said object image, by the position of a single global maximum on each of said saliency maps.
• the invention is based on a completely new and inventive approach to the detection of several points of interest in a representative image of an object, since it proposes to use a layered neural architecture, which makes it possible to generate at the output several saliency cards allowing a direct detection of the points of interest to locate, by simple search of maximum.
• the invention therefore proposes a global search, on the whole of the object image, of the various points of interest by the neural network, which makes it possible in particular to take account of the relative positions of these points, and also allows to overcome the problems related to their total or partial occultation.
• the output layer comprises at least two saliency maps each associated with a predefined distinct point of interest.
• Such a neural architecture is also more robust than techniques prior to possible problems of lighting images of objects. It is specified that here is meant by "predefined point of interest" a remarkable element of an object, such as for example an eye, a nose, a mouth, etc., in a face image.
• the invention therefore does not consist of searching for any contour in an image, but rather a predefined identified element.
• said object image is a face image.
• the points of interest sought are then permanent physical traits, such as eyes, nose, mouth, eyebrows, etc.
• such a location system also comprises at least a second intermediate convolution layer comprising a plurality of neurons.
• a layer may specialize in the detection of low level elements, such as contrast lines, in the object image.
• such a location system also comprises at least a third subsampling intermediate layer comprising a plurality of neurons.
• a locating system comprises, between said input layer and said first layer intermediate: a second intermediate convolution layer comprising a plurality of neurons and for detecting at least one line-like elementary form in said object image, said second intermediate layer delivering a convolved object image; a third subsampling intermediate layer comprising a plurality of neurons and for reducing the size of said convoluted object image, said third intermediate layer providing a reduced convolved object image; a fourth convolutional intermediate layer comprising a plurality of neurons and for detecting at least one wedge type complex shape in said reduced convolved object image.
• the invention also relates to a method of learning a neural network of a system for locating at least two points of interest in an object image as described above.
• Each of said neurons has at least one weighted input by a synaptic weight, and a bias.
• Such a learning method comprises steps of: constructing a learning base comprising a plurality of object images annotated according to said points of interest to be located; initialization of said synaptic weights and / or said biases; for each of said annotated images of said learning base: preparing said at least two desired saliency maps output from said at least two predefined points of interest annotated on said image; presenting said image at the input of said location system and determining said at least two saliency maps delivered at the output; minimizing a difference between said desired saliency maps and outputted to all of said annotated images of said learning base, so as to determine said synaptic weights and / or said optimal biases.
• the neural network thus learns, based on examples manually annotated by a user, to recognize certain points of interest on the object images. He will then be able to locate them in any image provided at the input of the network.
• said minimization is a minimization of a mean squared error between said desired saliency maps and output and implements an iterative gradient retropropagation algorithm. This algorithm is described in detail in Appendix 2 of this document, and allows a fast convergence towards the optimal values of the different synaptic biases and weights of the network.
• the invention also relates to a method for locating at least two points of interest in an object image, which comprises steps of: presenting said object image at the input of a layered architecture implementing a artificial neural network; sequentially activating at least one intermediate layer, called the first intermediate layer, comprising a plurality of neurons and making it possible to generate at least two saliency maps each associated with a predefined distinct point of interest of said object image, and at least one output layer comprising said saliency maps, said saliency maps comprising a plurality of neurons each connected to all the neurons of said first intermediate layer; locating said points of interest in said object image by searching, in said saliency maps, for a position of a single global maximum on each of said maps.
• such a locating method comprises preliminary steps of: detecting, in any image, a zone encompassing said object, and constituting said object image; resizing said object image.
• This detection can be done from a conventional detector well known to those skilled in the art, for example a face detector which makes it possible to determine a box encompassing a face in a complex image. Resizing can be carried out automatically by the detector, or independently by dedicated means: it provides input to the neural network images all having the same size.
• the invention further relates to a computer program comprising program code instructions for executing the method of learning a neural network described above when said program is executed by a processor, as well as a program of computer comprising program code instructions for executing the method of locating at least two points of interest in an object image previously described when said program is executed by a processor.
• Such programs can be downloaded from a communication network (eg the global Internet network) and / or stored in a computer readable data medium.
• a communication network eg the global Internet network
• FIG. 1 presents a synoptic of the neural architecture of the location system of points of interest in an object image of the invention
• Figure 2 more specifically illustrates a convolution map, followed by a subsampling map, in the neural architecture of Figure 1
• Figures 3a and 3b show some examples of face images of the learning base
• FIG. 4 describes the main steps of the method for locating facial elements in a face image according to the invention
• Figure 5 shows a simplified block diagram of the locating system of the invention
• FIG. 6 shows an example of a network of artificial neurons of the perceptron multi-layer type
• Figure 7 illustrates more precisely the structure of an artificial neuron
• Figure 8 shows the characteristic of the hyperbolic tangent function used as a transfer function for sigmoidal neurons. 7. Description of an embodiment of the invention
• the general principle of the invention is based on the use of a neural architecture to be able to automatically detect several points of interest in object images (more particularly of semi-rigid objects), and in particular in images faces (detection of permanent features such as eyes, nose or mouth). More precisely, the principle of the invention consists in constructing a neural network making it possible to learn how to transform, in one pass, an object image into several saliency maps whose positions of the maxima correspond to the positions of points of interest. selected by the user in the input object image.
• This neural architecture is composed of several heterogeneous layers, which make it possible to automatically develop robust low-level detectors, while learning rules allowing to regulate the plausible relative dispositions of the detected elements and to take into account naturally all information available to locate possible hidden elements.
• All neuron connection weights are set during a learning phase, from a set of images of pre-segmented objects, and positions of points of interest in these images.
• the neural architecture then acts as a cascade of filters making it possible to transform an image zone containing an object, previously detected in a larger image or in a video sequence, into a set of digital maps, of the size of the input image, whose elements are between -1 and 1.
• Each map corresponds to a particular point of interest whose position is identified by a simple search of the position of the element whose value is maximum.
• the method of the invention allows a robust detection of facial elements in faces, in various poses (orientations, semi-front), with various facial expressions, which can contain occulting elements, and appearing in images that exhibit significant variability in terms of resolution, contrast and illumination. 7.1 Neural architecture
• the architecture of the artificial neural network of the point of interest localization system of the invention is presented.
• the operating principle of such artificial neurons, as well as their structure, is recalled in Appendix 1, which forms an integral part of the present description.
• Such a neural network is for example a multi-layer perceptron network, also described in appendix 1.
• Such a neural network is composed of six interconnected heterogeneous layers referenced E, C 1 , S 2 , C 3 , N 4 and R 5 , which contain a series of cards resulting from a succession of convolution and subtraction operations. sampling.
• E interconnected heterogeneous layers referenced E, C 1 , S 2 , C 3 , N 4 and R 5 , which contain a series of cards resulting from a succession of convolution and subtraction operations. sampling.
• the proposed architecture includes: - an input layer E: it is a retina, which is a large image matrix
• H x L where H is the number of lines and L is the number of columns.
• the input layer E receives the elements of an image zone of the same size H x L.
• E 1J (P 1J - 128) / 128, of value between -1 and 1.
• HxL is also the size of the faces images of the learning base used for the parameterization of the neural network, and face images in which it is desired to detect one or more facial elements. This size can be that obtained directly at the output of the face detector that extracts face images, larger images or video sequences.
• a first convolution layer C 1 consisting of NC 1 cards referenced C 11 .
• Each card C 11 is connected 1O 1 to the input card E, and comprises a plurality of linear neurons (as presented in Appendix 1).
• Each of these neurons is connected by synapses to a set of M 1 x M 1 neighboring elements in the E-card (receptive fields) as described in more detail in Figure 2.
• Each of these neurons further receives bias.
• These M 1 x M 1 synapses, plus the bias, are shared by all the C 11 neurons.
• Each card C 11 thus corresponds to the result of a convolution by a core M 1 x M 1 11 augmented by a bias, in the input card E.
• This convolution specializes in a detector of certain low-level shapes in the card input such as oriented contrast lines of the image.
• Each card S 2j is connected 12 to a corresponding card C 11 .
• Each neuron of a map S 2j receives the average of M 2 ⁇ M 2 neighboring elements 13 in the map C 11 (receptive fields), as illustrated in more detail in FIG. 2.
• Each neuron multiplies this average by a synaptic weight, and adds a bias.
• Synaptic weight and bias whose optimal values are determined during a learning phase, are shared by the set of neurons of each card S 2j .
• the output of each neuron is obtained after passing through a sigmoid function.
• Each card C 3k is connected 14 k to each of the cards S 2j of the subsampling layer
• the neurons of a C 3k map are linear, and each of these neurons is connected by synapses to a set of M 3 ⁇ M 3 neighboring elements 15 in each of the maps S 2j . He also receives a bias.
• the M 3 x M 3 synapses per card, plus the bias, are shared by all the neurons of the C 3k cards.
• the cards C 3k correspond to the result of the sum of NC 3 convolutions by M 3 x M 3 15 nuclei, increased by a bias.
• Each neuron of the N layer 4 is connected 1O 1 to all the neurons of the layer C 3 , and receives a bias.
• N 41 neurons make it possible to learn how to generate the R 5m output cards by maximizing the responses on the positions of the points of interest in each of these cards, while taking into account the overallity of the C 3 cards, which makes it possible to detect a particular point of interest taking into account the detection of others.
• NN 4 100 neurons are chosen, and the hyperbolic tangent function (denoted th or tanh) for the transfer function of sigmoidal neurons; a layer R 5 of cards, consisting of NR 5 R 5m cards, one for each point of interest chosen by the user (right eye, left eye, nose, mouth, etc.).
• Each R 5m card is connected to all the neurons of the N layer 4 .
• the neurons of an R 5m map are sigmoidal, and each is connected to all neurons in the N 4 layer.
• Each card R 5m is of size Hx L, which is the size of the input layer E.
• NR 5 4 cards of size 56 x 46 are chosen.
• each R 5m card having a maximum output in each R 5m card corresponds to the position of the corresponding facial element in the image presented at the input of the network.
• the layer R 5 has only one saliency map on which all the points of interest that one wishes to locate in the image.
• FIG. 2 illustrates an example of a 5 ⁇ 5 11 convolution card C 11 followed by a 2 ⁇ 2 subsampling card S 2j .
• the convolution performed does not take into account the pixels located on the edges of the card C 11 , to avoid edge effects.
• a set T of face images is first manually extracted from a corpus of large images.
• Each face image is resized to the size H x L of the input layer E of the neural architecture, preferably respecting the natural proportions of the faces.
• the positions of the eyes, nose and center of the mouth are manually marked, as shown in Figure 3a: thus obtaining a set of annotated images based on the points of interest that the neural network will have to learn to locate.
• These points of interest to locate in the images can be chosen freely by the user.
• a training database of approximately 2500 manually annotated face images can be used depending on the position of the center of the left eye, the right eye, the nose and the mouth. After applying geometric modifications to these annotated images (translations, rotations, zooms, etc.), we obtain about 32,000 examples of annotated faces with significant variability.
• each point corresponds to the point of value +1, whose position corresponds to that of a facial element to be located (right eye, left eye, nose or center of the mouth).
• N 4 and R 5 of the neural network are activated one after the other.
• the response of the neural network to image I is then obtained.
• the goal is to obtain cards R 5m identical to the desired cards D 5m .
• the following parameters can be used in the gradient retropropagation algorithm: a learning step of 0.005 for the neurons of the C 1 , S 2 , C 3 layers; a learning step of 0.001 for neurons of layer N 4 ; a learning step of 0.0005 for the neurons of the layer R 5 ; a momentum of 0.2 for all the neurons of the architecture.
• the gradient retropropagation algorithm then converges to a stable solution after 25 iterations, if we consider that an iteration of the algorithm corresponds to the presentation of all the images of the training set T.
• the neural network of FIG. 1 is ready to process any digital face image, in order to extract annotated points of interest from the images of the set of images. learning T.
• FIG. 4 It is now possible to use the neural network of FIG. 1, set during the learning phase, for searching the facial elements in a face image.
• the method implemented to achieve such a location is shown in FIG. 4.
• the faces 44 and 45 present in the image 46 are detected using a face detector.
• the latter locates the box enclosing the inside of each face 44, 45.
• the image zones contained in each bounding box are extracted 41 and constitute the images of faces 47, 48 in which the search of the facial elements must be carried out.
• Each face image extracted I 47, 48 is resized 41 at the waist
• H x L is placed at the input E of the neural architecture of FIG. 1.
• the input layer E, the intermediate layers C 1 , S 2 , C 3 , N 4 , and the output layer R 5 are activated one after the other, so as to perform a filter 42 of the image I 47, 48 by the neural architecture.
• the response of the neural network to image I 47, 48 is obtained in the form of four saliency maps R 5m for each of images 1 47, 48.
• the faces are detected in the images 46 by the CFF face detector presented by C. Garcia and M. Delakis, in “Convolutional Face Finder: a Neural Architecture for Fast and Robust Face”. Detection, "IEEE Transactions on Pattern Analysis and Machine
• Such a face detector can in fact robustly detect faces of minimum size 20x20, tilted up to ⁇ 25 degrees and rotated up to ⁇ 60 degrees, in scenes with complex background, and under variable lighting.
• FIG. 4 a simplified block diagram of a system or device for locating points of interest in an object image.
• a system or device for locating points of interest in an object image comprises a memory M 51, and a processing unit 50 equipped with a ⁇ P processor, which is controlled by the computer program Pg 52.
• the processing unit 50 receives as input a set T of learning face images, annotated according to the points of interest that the system must be able to locate in an image, from which the microprocessor ⁇ P performs, according to the instructions of the program Pg 52, the implementation of a gradient retropropagation algorithm for optimizing the synaptic bias and weight values of the neural network. These optimum values 54 are then stored in the memory M 51.
• the optimal values of the synaptic bias and weight are loaded from the memory M 51.
• the processing unit 50 receives as input an object image I, from which the microprocessor ⁇ P performs, according to the instructions of the program Pg 52, a filtering by the neural network and a maxima search in the saliency cards obtained at the output. At the output of the processing unit 50, the coordinates 53 of each of the points of interest sought in the image I are obtained.
• APPENDIX 1 Artificial neurons and multi-layer perceptron neuron networks 1.
• the multi-layered perceptron is a structured network of artificial neurons organized in layers, in which the information travels in one direction, from the input layer to the output layer.
• FIG. 6 shows the example of a network containing an input layer 60, two hidden layers 61 and 62 and an output layer 63.
• the input layer 60 always represents a virtual layer associated with the inputs of the system. It does not contain any neurons.
• the following layers 61 to 63 are layers of neurons.
• a multi-layer perceptron may have any number of layers and a number of neurons (or inputs) per layer of any kind.
• the neural network has 3 inputs, 4 neurons on the first hidden layer 61, three neurons on the second 62 and four neurons on the output layer 63.
• the outputs of the neurons of the last layer 63 correspond to the outputs of the system.
• An artificial neuron is a computing unit that receives an input signal (X, vector of real values), through synaptic connections carrying weights
• FIG. 7 shows the structure of such an artificial neuron, the operation of which is described in paragraph ⁇ 2 below.
• the neurons of the network of FIG. 6 are connected together, from layer to layer, by the weighted synaptic connections. It is the weights of these connections that govern the operation of the network and "program" an application of the space of the inputs to the space of the outputs by means of a nonlinear transformation.
• the creation of a multi-layer perceptron to solve a given problem thus passes through the inference of the best possible application, as defined by a set of training data consisting of desired input and output vector pairs. 2.
• Each of the entries x is excites a weighted synapse W 1 .
• a summing function 70 calculates a potential V, which, after passing through an activation function ⁇ , delivers a real value output y.
• the quantity w o x o is called bias and corresponds to a threshold value for the neuron.
• ⁇ can take different forms depending on the intended applications.
• APPENDIX 2 Gradient Retropropagation Algorithm
• neural network learning consists in determining all the weights of the synaptic connections, so as to obtain a vector of desired outputs D as a function of a vector of neurons.
• input X For this, a learning base is constituted, which consists of a list of K input / output pairs (X k , D k ) corresponding.
• Sj c is the set of neurons of the index layer c-1 connected to the inputs of neuron i of the layer of index c;
• W j j is the weight of the synaptic connection extending from neuron j to neuron i.
• the gradient backpropagation algorithm operates in two successive passes, which are direct propagation and backpropagation passes: during the propagation pass, the input signal X k crosses the neural network and activates a response Y k in exit ; during the backpropagation, the error signal E k is backpropagated in the network, which makes it possible to modify the synaptic weights to minimize the error E k .
• such an algorithm comprises the following steps: Set the learning step p to a sufficiently small positive value (of the order of 0.001) Set the momentum ⁇ to a positive value between 0 and 1 (of the order of 0.2) Randomly initialize the synaptic weights of the network at small values Repeat choose an example pair (X k , D k ): propagation: calculate in the order of the layers the outputs of the neurons:
• V 1 c " V w ] r y ⁇ c _ ⁇ and the output
• Aw Z P ⁇ , c y j, cx + ⁇ Aw Z> Y / e s ,, c
• p is the learning step and ⁇ the momentum

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• General Health & Medical Sciences (AREA)
• Oral & Maxillofacial Surgery (AREA)
• Biomedical Technology (AREA)
• Evolutionary Computation (AREA)
• Software Systems (AREA)
• Life Sciences & Earth Sciences (AREA)
• Artificial Intelligence (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Biophysics (AREA)
• Computational Linguistics (AREA)
• Data Mining & Analysis (AREA)
• Multimedia (AREA)
• Molecular Biology (AREA)
• Computing Systems (AREA)
• General Engineering & Computer Science (AREA)
• Mathematical Physics (AREA)
• Human Computer Interaction (AREA)
• Image Processing (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=35748862

Family Applications (1)

Country Status (6)

Families Citing this family (74)

• 2005
  • 2005-03-31 FR FR0503177A patent/FR2884008A1/fr active Pending
• 2006
  • 2006-03-28 EP EP06725370A patent/EP1866834A2/fr not_active Withdrawn
  • 2006-03-28 WO PCT/EP2006/061110 patent/WO2006103241A2/fr active Application Filing
  • 2006-03-28 CN CNA2006800149363A patent/CN101171598A/zh active Pending
  • 2006-03-28 JP JP2008503506A patent/JP2008536211A/ja active Pending
  • 2006-03-28 US US11/910,159 patent/US20080201282A1/en not_active Abandoned

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events