EP2370932B1 - Procédé, appareil et produit de programme d'ordinateur pour fournir une estimation de pose de visage - Google Patents

Procédé, appareil et produit de programme d'ordinateur pour fournir une estimation de pose de visage Download PDF

Info

Publication number
EP2370932B1
EP2370932B1 EP09836139.7A EP09836139A EP2370932B1 EP 2370932 B1 EP2370932 B1 EP 2370932B1 EP 09836139 A EP09836139 A EP 09836139A EP 2370932 B1 EP2370932 B1 EP 2370932B1
Authority
EP
European Patent Office
Prior art keywords
face
pose
classifiers
detectors
pose information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09836139.7A
Other languages
German (de)
English (en)
Other versions
EP2370932A1 (fr
EP2370932A4 (fr
Inventor
Jiangwei Li
Kong Qiao Wang
Lei Xu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nokia Technologies Oy
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Publication of EP2370932A1 publication Critical patent/EP2370932A1/fr
Publication of EP2370932A4 publication Critical patent/EP2370932A4/fr
Application granted granted Critical
Publication of EP2370932B1 publication Critical patent/EP2370932B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/77—Determining position or orientation of objects or cameras using statistical methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06V10/7747—Organisation of the process, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face

Definitions

  • Embodiments of the present invention relate generally to image processing technology and, more particularly, relate to a method, apparatus and computer program product for providing face pose estimation.
  • Face detection and recognition is becoming an increasingly more important technology.
  • face detection may be useful in biometrics, user interface, gaming and other areas such as creating context for accessing communities in the mobile domain. Face detection may also be important going forward in relation to initiatives such as metadata standardization.
  • face detection techniques continue to improve, many current methods require either a high computation capability (e.g., statistical methods of detecting faces by scanning images in a traversing way on multiple scales) or suffer from limited face detection performance (e.g., structure-based methods with relatively high false alarms of face detection).
  • some statistical face detection mechanisms have degraded performance for multi-view face detection in relation to front face detection.
  • faces encountered in a particular image may not always be oriented in the same direction relative to the camera, which could negatively impact face detection. For example, in-plane rotation (e.g., faces that are rotated along the axis from the face to the observer) may complicate face detection in some cases.
  • the whole 360-degree range is divided into 12 sub-ranges and construct their corresponding view based detectors separately.
  • a pose estimation method is introduced and results in a processing speed of four frames per second on 320/spl times/240 sized image.
  • a conference article by Heng Liu et al. titled "Rotated face detection in color images using radial template” proposes a face detection algorithm to locate faces rotated in any orientation. Detecting rotated faces is important for a face detection system.
  • a model named radial template (RT) is provided to detect rotated faces. This template is designed to find stable features of center-rotated objects in edge maps. Based on skin detection and edge extraction, the method searches for face-like areas and gets their orientations by RT searching. Then the candidates are rotated upright and a frontal face detector is used to determine the existence of faces.
  • a method, apparatus and computer program product are therefore provided to enable an improved face detection mechanism.
  • a mechanism is provided that is capable of incorporating a fusion of certain aspects of statistical and structural face detection methods in order to realize advantages associated with both, while reducing disadvantages associated therewith.
  • embodiments of the present invention may provide a relatively robust ability for detecting faces even under different conditions (e.g., having different head poses or facial views).
  • a method of providing face pose estimation for use in face detection may include utilizing a selected portion of classifiers in detectors of a boosting based statistical analysis scheme to determine coarse pose information for a candidate face in an image, determining fine pose information for the candidate face based at least in part on the determined coarse pose information by utilizing a structure based analysis scheme by employing a series of differently oriented facial templates, rotating the image to verticality prior to performing face detection, and employing another portion of the classifiers in the detectors to perform face detection on the rotated image to determine whether the candidate face corresponds to a face.
  • a computer program product for providing face pose estimation for use in face detection.
  • the computer program product includes computer-executable program code instructions, which program code instructions cause an apparatus to perform the method outlined above.
  • an apparatus for providing face pose estimation for use in face detection may include a processor.
  • the processor may be configured to utilize a selected portion of classifiers in detectors of a boosting based statistical analysis scheme to determine coarse pose information for a candidate face in an image, determine fine pose information for the candidate face based at least in part on the determined coarse pose information by utilizing a structure based analysis scheme by employing a series of differently oriented facial templates, rotating the image to verticality prior to performing face detection, and employ another portion of the classifiers in the detectors to perform face detection on the rotated image to determine whether the candidate face corresponds to a face.
  • Embodiments of the invention may provide a method, apparatus and computer program product for employment, for example, in mobile or fixed environments. As a result, for example, computing device users may enjoy an improved capability for face detection.
  • Some embodiments of the present invention may provide a mechanism by which improvements may be experienced in relation to face detection and/or recognition.
  • some embodiments may provide for a real-time or near real-time face detection scheme, which may be suitable for metadata assignment, biometrics, gaming, and numerous other activities on hand-held or other computing devices.
  • a first category includes pose evaluation using multiple trained detectors.
  • multiple face detectors are trained given corresponding training sets with various poses.
  • the training sets include a plurality of faces oriented with various different amounts of in-plane and/or out of plane rotations (e.g., various poses).
  • Image patches may then be attempted to be passed through the trained detectors, and the one through which the image patch passes and is recognized as a face, may be indicative of the corresponding pose of the face therein.
  • FIG. 1 shows an example of a division of the rotation plane into 20 sectors.
  • a face detector may be trained based on a face with a corresponding pose.
  • each sector defines a particular in-plane rotation and a particular out of plane rotation.
  • the 20 trained detectors may be able to detect faces having various different poses within an image to provide an estimate of face pose information in a multi-view face detection application or system.
  • one disadvantage of this approach is that the approach only provides a pose scope, and not necessarily precise pose information.
  • sector A of FIG.1 covers pose variations from -45°to 45° of in plane rotation.
  • a corresponding training database for providing training examples for training the detector has to include many examples to cover incremental changes within the range, which may lead to difficult convergence in training because of relatively large pattern variations.
  • This disadvantage could be partially solved by increasing the number of sectors in order to make convergence easier.
  • increasing the number of sectors e.g., by doubling
  • Another category of approaches to multi-view face detection may include pose evaluation by training a pose classifier.
  • the pose may initially be estimated by a pose classifier. Then, the image patch is sent to a corresponding face detector of the classified pose to verify whether the image patch includes a face or not.
  • the pose classifier is trained to distinguish the corresponding 20 poses.
  • This approach also only provides a pose scope instead of precise pose.
  • training a unique pose classifier to distinguish as many as 20 poses can be a difficult task to realize as it relates to a multiple category classification problem, which is a classical difficulty in pattern analysis. In some cases, it may be possible to train 20 pose classifiers and use each one to classify one pose and remaining poses. However, this solution is also potentially time consuming and therefore may not be effectively useful in real-time scenarios.
  • embodiments of the present invention may be aimed at addressing problems related to difficulty in training convergence, coarse precision or lack of accuracy in pose evaluation, and low time efficiency in pose evaluation.
  • embodiments of the present invention may provide a unique fusion of structure based and statistical based face detection methods.
  • embodiments of the present invention may initially employ coarse pose determination using a statistical method, followed by fine pose determination using a structural method to analyze an image patch for a face. Normalized data (e.g., rotated to verticality) may then be used to analyze for the existence of a face in the image patch (e.g., using another statistical method). Accordingly, some embodiments provide for a fine pose determination prior to performing a pose-based face detection application.
  • embodiments of the present invention may provide fine pose information with relatively low increase in workload for a face-based application.
  • embodiments of the present invention may be referred to as relating to a push-switch based face pose evaluator because information is pushed between two different methods and then a method switch is made to push information back to the original method.
  • Face detection may be accomplished in some cases using cascaded classifiers forming a detector.
  • cascaded classifiers forming a detector.
  • twenty different detectors could be formed by cascading weak classifiers.
  • a weak classifier may be defined as a classifier which is only slightly correlated with the true classification.
  • a strong classifier may be a classifier that is arbitrarily well-correlated with the true classification.
  • Weak classifiers may be easier to create than strong classifiers.
  • a plurality of weak classifiers may, in some cases, provide results that may be on par with results provided by a strong classifier with less cost.
  • FIG. 2 illustrates a potential mechanism for pose evaluation that could be used in conjunction with the example of FIG. 1 .
  • n detectors e.g., 20 detectors in the example of FIG. 1
  • n detectors may be comprised of cascaded sets of weak classifiers trained on faces with a particular pose or range of poses to detect faces having characteristics similar to the corresponding training poses.
  • each detector may be trained for a corresponding particular pose.
  • the detected face can be said to have a corresponding pose similar to that associated with the training faces used to train the detector (e.g., pose 1).
  • the image patch may still generate a face detection with a corresponding pose for any one or more of the other detectors (e.g., detector 2 to detector n). Therefore, in operation, the image patch may be input into all of the detectors in order to realize a multi-view face detector configured to detect possible faces with various corresponding poses.
  • the pose information for each possible face may be simultaneously determinable based on the detector that detected the face.
  • FIG. 3 illustrates one example of a fusion of structural and statistical methods for face detection according to an exemplary embodiment.
  • a statistical method 20 of pose evaluation e.g., similar to the initial stages shown in FIG.
  • a statistical method 28 may be employed on the normalized patches 30 to provide a face detection output 32 that may be relatively accurate and robust without greatly increasing overhead costs.
  • a characteristic aspect of the embodiment shown in FIG. 3 relates to the combination of detectors comprising several weak classifiers with a structure-based method to form a pose evaluator.
  • an embodiment of the present invention could be realized by employing the initial stage or stages of the statistical method of FIG. 2 for coarse pose determination and then employing a structural method to perform fine pose determination. Some or all of the remaining weak classifiers of the statistical method for face detection may then be used for corresponding face detection in each respective detector.
  • separate generic statistical methods could be used for the respective operations associated with the statistical method 20 and the statistical method 28 of FIG. 3 instead of using portions of the statistical method shown in FIG. 2 as shown in FIG. 4 .
  • FIG. 4 shows a block diagram of the example described in reference to FIG. 3 with an example image patch.
  • the image patch 40 may be analyzed for coarse pose determination using statistical method 42.
  • a coarse pose determination may be achieved indicating a pose with between about 90 and 135 degrees of in-plane rotation and approximately no out of plane rotation.
  • a structural method 44 may then be employed to obtain a fine pose determination of 135 degrees of in-plane rotation and no out of plane rotation.
  • the image patch 46 may then be rotated to verticality for normalization and another statistical method 48 (or a portion of the same statistical method) may be employed for face detection.
  • FIGS. 2-4 embodiments of the present invention may perform face detection for face-based applications without requiring a separate pose classifier. Accordingly, time efficiency of exemplary embodiments of the present invention may be increased relative to existing methods.
  • FIG. 5 illustrates an example of a push-switch based pose estimator in accordance with an exemplary embodiment of the present invention.
  • a statistical-based method e.g., Adaboost (adaptive boosting) based statistical method 50 as described in reference to FIG. 3 above
  • the coarse pose estimation 52 may be provided to a structure based detection method 54 to result in a fine pose evaluation 56.
  • the patch may be rotated to verticality by the structure based scheme and pushed back to the Adaboost based statistical method 50 for face detection, recognition, etc.
  • Adaboost may be an example of a statistical method that may be employed in connection with embodiments of the present invention.
  • Other examples of statistical methods may include Gentle-Boost, RealBoost, FloatBoost, and/or the like.
  • exemplary structural methods may include employment of detectors in various different orientations that provide for coverage of respective different in-plane rotation angles, and employment of a series of differently oriented facial templates against which features of an image may be compared.
  • a result of employment of structure based scheme may be that the image patch analyzed is rotated to verticality.
  • FIG. 6 illustrates one example of a schematic block diagram of an apparatus for enabling face pose estimation for face detection according to an exemplary embodiment of the present invention.
  • An exemplary embodiment of the invention will now be described with reference to FIG. 6 , in which certain elements of an apparatus for enabling face pose estimation for face detection are displayed.
  • the apparatus of FIG. 6 may be employed, for example, on a mobile terminal (e.g., a mobile phone, gaming device, personal digital assistant (PDA) and/or the like) or a variety of other devices, both mobile and fixed (such as, for example, a network device, personal computer, laptop computer, digital camera, or the like).
  • PDA personal digital assistant
  • embodiments may be employed on a combination of devices.
  • some embodiments of the present invention may be embodied wholly at a single device or by devices in a client/server relationship.
  • the devices or elements described below may not be mandatory and thus some may be omitted in certain embodiments.
  • the apparatus may include or otherwise be in communication with a processor 70, a user interface 72, a communication interface 74 and a memory device 76.
  • the memory device 76 may include, for example, volatile and/or non-volatile memory.
  • the memory device 76 may be configured to store information, data, applications, instructions or the like for enabling the apparatus to carry out various functions in accordance with exemplary embodiments of the present invention.
  • the memory device 76 could be configured to buffer input data for processing by the processor 70.
  • the memory device 76 could be configured to store instructions for execution by the processor 70.
  • the memory device 76 may be one of a plurality of databases that store information and/or media content.
  • the processor 70 may be embodied in a number of different ways.
  • the processor 70 may be embodied as various processing means such as a processing element, a coprocessor, a controller or various other processing devices including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), a hardware accelerator, or the like.
  • the processor 70 may be configured to execute instructions stored in the memory device 76 or otherwise accessible to the processor 70.
  • the processor 70 may represent an entity capable of performing operations according to embodiments of the present invention while configured accordingly.
  • the processor 70 when the processor 70 is embodied as an ASIC, FPGA or the like, the processor 70 may be specifically configured hardware for conducting the operations described herein.
  • the processor 70 when the processor 70 is embodied as an executor of software instructions, the instructions may specifically configure the processor 70, which may otherwise be a general purpose processing element, to perform the algorithms and operations described herein.
  • the communication interface 74 may be any means such as a device or circuitry embodied in either hardware, software, or a combination of hardware and software that is configured to receive and/or transmit data from/to a network and/or any other device or module in communication with the apparatus.
  • the communication interface 74 may include, for example, an antenna (or multiple antennas) and supporting hardware and/or software for enabling communications with a wireless communication network (e.g., network 78).
  • the communication interface 74 may alternatively or also support wired communication.
  • the communication interface 74 may include a communication modem and/or other hardware/software for supporting communication via cable, digital subscriber line (DSL), universal serial bus (USB), Ethernet or other mechanisms.
  • the user interface 72 may be in communication with the processor 70 to receive an indication of a user input at the user interface 72 and/or to provide an audible, visual, mechanical or other output to the user.
  • the user interface 72 may include, for example, a keyboard, a mouse, a joystick, a display, a touch screen, a microphone, a speaker, or other input/output mechanisms.
  • the user interface 72 may be limited, or eliminated.
  • the user interface 72 may include, among other devices or elements, any or all of a speaker, a microphone, a display, and a keyboard or the like.
  • the processor 70 may be embodied as, include or otherwise control a coarse pose evaluator 80, a fine pose evaluator 82, a face detector 84, and in some cases also a candidate rotator 86.
  • the candidate rotator 86 may be a portion of the fine pose evaluator 82 in some instances.
  • the coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, and the candidate rotator 86 may each be any means such as a device or circuitry embodied in hardware, software or a combination of hardware and software (e.g., processor 70 operating under software control, the processor 70 embodied as an ASIC or FPGA specifically configured to perform the operations described herein, or a combination thereof) that is configured to perform the corresponding functions of the coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, and the candidate rotator 86, respectively, as described below.
  • processor 70 operating under software control, the processor 70 embodied as an ASIC or FPGA specifically configured to perform the operations described herein, or a combination thereof
  • any or all of the coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, and the candidate rotator 86 may include instructions, code, modules, applications and/or circuitry for providing respective portions of face detection services.
  • code, circuitry and/or instructions associated with the coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, and the candidate rotator 86 need not necessarily be modular.
  • the coarse pose evaluator 80 and the face detector 84 may each implement portions of the same statistical based method or scheme (e.g., including a series of cascaded detectors) and in some cases the implemented portions may overlap.
  • communication between the coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, and the candidate rotator 86 is conducted via the processor 70.
  • the coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, and the candidate rotator 86 are alternatively in direct communication With each other or may have no communication with each other in other embodiments.
  • the coarse pose evaluator 80 employs a series of trained detectors of a confidence based or statistical based method of face detection for determining an estimate of the pose of any identified face within a particular image patch.
  • the coarse pose evaluator 80 does not implement all of the classifiers of the detectors of the statistical based method. Instead, only a selected number of the classifiers may be employed to get a rough estimate of pose in a relatively short time due to the avoidance of using all of the classifiers of the detectors.
  • a typical trained detector could be used to determine whether the image patch included a face.
  • time efficiency may be low if the image patch is analyzed using all of the classifiers.
  • embodiments of the present invention may utilize less than the full complement of trained classifiers for pose determination in combination with a structural method to refine the pose information determined by the trained classifiers of the statistical method that are employed.
  • the coarse pose evaluator 80 may be configured to quickly perform coarse pose evaluation to enable improved real-time performance by improving detection speed.
  • the coarse pose evaluator 80 may be configured to employ data used for training that includes relatively small pose variations. Even though relatively small pose variations are shown in the training data, given a face beyond the training scope, the probability of correctly classifying the given face with respect to its corresponding pose is relatively high since similarities between the training data and evaluated face are highest in both appearance and structure.
  • the number of weak classifiers in each detector is more than 1,000.
  • Each weak classifier would typically contribute a confidence score of indicating the degree to which the testing image patch is likely to be a face with the pose associated with the corresponding classifier.
  • W i is actually a normalized score to measure, in a particular partition, which probability is more dominant. If the confidence score is large, it is more likely that the image patch being analyzed includes a face with the pose associated with the corresponding classifier.
  • a pose may be determined by utilizing a fraction of the weak classifiers of each trained detector (e.g., only the first 6 weak classifiers in one example) for pose estimation.
  • the coarse pose evaluator 80 is configured to implement a selected portion of the classifiers for each detector of a statistical based analysis scheme in order to obtain confidence scores used to achieve relatively good (albeit coarse) pose evaluation performance.
  • FIG. 7 shows an example of coarse pose evaluation that may be accomplished using the coarse pose evaluator 80.
  • convergence performance may be increased at least in part by enabling a reduction in the coverage range of the detectors.
  • sector A in FIG. 1 covered from -45 to 45 degrees of in-plane rotation, which may require a relatively large amount of training data to enable reliable pose detection and face detection over such a range.
  • the coverage range may be reduced to -15 to 15 degrees of in-plane rotation because rotated faces can be calibrated by a structural method by the fine pose evaluator 82 (thus the rotation of the face indicated outside the -15 to 15 degrees coverage area to within the -15 to 15 degrees coverage area may be accomplished by the structural method).
  • the coarse pose evaluator 80 may therefore implement less classifiers in each detector (e.g., N may be smaller).
  • the fine pose evaluator 82 may be configured to implement a structure based analysis scheme to provide fine pose information including a relatively accurate indication of the pose of any face detected in a particular image patch.
  • the structure based analysis scheme may be one of the examples described above or any other structure based analysis scheme. Though some embodiments may estimate coarse pose information using the statistical scheme, utilizing the structure based scheme may enable refinement of the coarse pose information. In this regard, for example, if the face has a relatively large rotation, the face may not have a high likelihood of being correctly classified by using only the statistical based method employed by the coarse pose evaluator 80 (e.g., by classifiers of the detector associated with a corresponding pose), because for ensuring good convergence, only small pose variations are generally permitted in training data.
  • embodiments of the present invention utilize the structure based analysis scheme to get fine pose information.
  • the face may be rotated to verticality to enable face detection.
  • the rotation to verticality may be accomplished by the candidate rotator 86 either before or after operation of the fine pose evaluator 82 in determining a relatively accurate pose to define pose information for the corresponding face candidate.
  • the structural method may be better enabled to analyze structural features of the image patch to determine accurate pose information.
  • the face detector 84 may be configured to detect face candidates in a vertically aligned image patch with a fine pose determination already made using the statistical based analysis scheme used by the coarse pose evaluator 80. In some cases, the face detector 84 may use all or a portion of the remaining cascaded weak classifiers not initially used by the coarse pose evaluator 80. However, in other cases, an unrelated set of detectors employing a statistical method may be employed. In operation, after candidate faces have been detected, the candidate rotator 86 may rotate the detected candidates to vertical after which the face detector 84 may more easily and quickly determine whether the candidates correspond to faces. In an exemplary embodiment, the apparatus of FIG.
  • the apparatus of FIG. 6 may be in communication with a media capturing module (e.g., a camera) to receive image data for use in analysis as described herein.
  • a media capturing module e.g., a camera
  • the apparatus of FIG. 6 may receive images from a storage location (e.g., the memory device 76) or another device.
  • the coarse pose evaluator 80 and the face detector 84 may utilize Adaboost or another statistical based analysis scheme.
  • Adaboost is merely one example of a specific implementation of the coarse pose evaluator 80 and the face detector 84 and other mechanisms may alternatively be employed.
  • Adaboost adaptive boost
  • Adaboost is a machine learning algorithm that may be utilized in conjunction with other learning mechanisms to improve performance.
  • Adaboost may be used in conjunction with a structure based analysis scheme for pose evaluation.
  • the structure based analysis scheme may refine the pose information to provide fine pose information and rotate the corresponding image patch to verticality.
  • Other Adaboost classifiers may then be employed to complete face detection on the vertically rotated image.
  • the vertical rotation of the image by the structure based analysis scheme may make it unnecessary for training data with large rotation angles to be utilized by the statistical based analysis scheme.
  • less weak classifiers may need to be trained in each detector and detectors may achieve convergence more rapidly.
  • search scope of the structure based analysis scheme is refined for finding a precise rotation angle.
  • appropriate operators may be constructed to better estimate structure information to improve efficiency of the structure based analysis scheme.
  • some embodiments of the present invention fuse statistical and structure based analysis schemes for the determination of pose information to augment face detection.
  • Embodiments of the present invention may, in some cases, decrease time needed to accomplish pose estimation relatively accurately without appending extra computational load in face based applications.
  • a push-switch based face pose evaluator as described herein fuses both statistical and structure based analysis schemes for reliable pose estimation.
  • convergence performance in detector training may be improved, pose estimates may be quickly and accurately determined, and does so without requiring an extra pose classifier to train.
  • the structure based analysis scheme employed by the fine pose evaluator 82 may utilize eye positioning for fine pose information determination. As such, fine pose information may not be determinable for certain face rotations where the eyes are not visible. In such situations, only coarse pose information may be provided.
  • Embodiments of the present invention may be utilized in connection with multi-view face detection and recognition and also in connection with human-computer interface (HCI) applications in order to provide a user interface for mobile phone operation, gaming, and carrying out tasks using computers.
  • HCI human-computer interface
  • FIG. 8 is a flowchart of a system, method and program product according to some exemplary embodiments of the invention. It will be understood that each block or step of the flowchart, and combinations of blocks in the flowchart, can be implemented by various means, such as hardware, firmware, and/or software including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, the computer program instructions which embody the procedures described above may be stored by a memory device of a mobile terminal or other apparatus employing embodiments of the present invention and executed by a processor in the mobile terminal or other apparatus.
  • any such computer program instructions may be loaded onto a computer or other programmable apparatus (i.e., hardware) to produce a machine, such that the instructions which execute on the computer (e.g., via a processor) or other programmable apparatus create means for implementing the functions specified in the flowchart block(s) or step(s).
  • These computer program instructions may also be stored in a computer-readable memory that can direct a computer (e.g., the processor or another computing device) or other programmable apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart block(s) or step(s).
  • the computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block(s) or step(s).
  • blocks or steps of the flowchart support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that one or more blocks or steps of the flowchart, and combinations of blocks or steps in the flowchart, can be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.
  • one embodiment of a method for providing pose evaluation for face detection as illustrated, for example, in FIG. 8 may include utilizing a selected portion of classifiers in detectors (e.g., associated with a statistical based analysis scheme) to determine coarse pose information for a candidate face in an image at operation 100.
  • the method may further include determining fine pose information for the candidate face (e.g., by employing a structure based analysis scheme) based at least in part on the determined coarse pose information at operation 110 and employing another portion of the classifiers in the detectors (e.g., associated with the statistical based analysis scheme) to perform face detection based at least in part on the fine pose information to determine whether the candidate face corresponds to a face at operation 120.
  • the method may further include an operation of rotating the image to verticality prior to performing the face detection at operation 115.
  • determining fine pose information for the candidate face by employing the structure based analysis scheme comprises utilizing a face template.
  • the statistical based analysis scheme may be Adaboost.
  • utilizing the selected portion of classifiers may include utilizing N number of initial classifiers in the detectors of the statistical based analysis scheme.
  • employing another portion of the statistical based analysis scheme may include utilizing at least a portion of remaining classifiers not used for determining the coarse pose information.
  • an apparatus for performing the method of FIG. 8 above may comprise a processor (e.g., the processor 70) configured to perform some or each of the operations (100-120) described above.
  • the processor may, for example, be configured to perform the operations (100-120) by performing hardware implemented logical functions, executing stored instructions, or executing algorithms for performing each of the operations.
  • the apparatus may comprise means for performing each of the operations described above.
  • examples of means for performing operations 100-120 may comprise, for example, the processor 70, coarse pose evaluator 80, the fine pose evaluator 82, the face detector 84, the candidate rotator 86, and/or an algorithm executed by the processor 70 for processing information as described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Probability & Statistics with Applications (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Claims (7)

  1. Procédé mis en oeuvre par ordinateur comprenant les étapes suivantes :
    utiliser (100) une partie sélectionnée des classificateurs dans des détecteurs d'un schéma d'analyse statistique basé sur l'amplification (42) pour déterminer des informations de pose grossières pour un visage candidat dans une image ;
    caractérisé par les étapes suivantes :
    déterminer (110) des informations de pose fines pour le visage candidat sur la base, au moins en partie, des informations de pose grossières déterminées en utilisant un schéma d'analyse basé sur la structure (44) en employant une série de modèles faciaux orientés différemment ;
    faire tourner (115) l'image à la verticale avant d'effectuer la détection de visage ; et
    employer (120) une autre partie des classificateurs dans les détecteurs pour effectuer la détection de visage sur l'image tournée pour déterminer si le visage candidat correspond à un visage.
  2. Procédé selon la revendication 1, dans lequel l'utilisation de la partie sélectionnée des classificateurs comprend d'utiliser un nombre N de classificateurs initiaux dans les détecteurs dudit schéma d'analyse statistique (42).
  3. Procédé selon la revendication 1, dans lequel l'emploi d'une autre partie des classificateurs comprend d'utiliser au moins une partie des classificateurs restants non utilisés pour déterminer les informations de pose grossières.
  4. Appareil comprenant un processeur (70) configuré pour :
    utiliser une partie sélectionnée de classificateurs dans des détecteurs d'un schéma (42) d'analyse statistique basé sur l'amplification pour déterminer des informations de pose grossières pour un visage candidat dans une image ;
    caractérisé en ce que le processeur (70) est en outre configuré pour :
    déterminer des informations de pose fines pour le visage candidat sur la base, au moins en partie, des informations de pose grossières déterminées en utilisant un schéma d'analyse basé sur la structure (44) en employant une série de modèles faciaux orientés différemment ;
    faire tourner l'image à la verticale avant d'effectuer la détection de visage ; et
    employer une autre partie des classificateurs dans les détecteurs (48) pour effectuer la détection de visage sur l'image tournée pour déterminer si le visage candidat correspond à un visage.
  5. Appareil selon la revendication 4, dans lequel l'appareil comprend un terminal mobile.
  6. Appareil selon la revendication 4, dans lequel le processeur est configuré pour effectuer une reconnaissance de visage par rapport au visage.
  7. Produit programme informatique comprenant des instructions de code de programme exécutables par ordinateur qui, lorsque le programme est exécuté par un ordinateur, amènent l'ordinateur à exécuter les étapes suivantes :
    utiliser une partie sélectionnée des classificateurs dans des détecteurs d'un schéma d'analyse statistique basé sur l'amplification (42) pour déterminer des informations de pose grossières pour un visage candidat dans une image ;
    caractérisé par les étapes suivantes :
    déterminer des informations de pose fines pour le visage candidat sur la base, au moins en partie, des informations de pose grossières déterminées en utilisant un schéma d'analyse basé sur la structure (44) en employant une série de modèles faciaux orientés différemment ;
    faire tourner l'image à la verticale avant d'effectuer la détection de visage ; et
    employer une autre partie des classificateurs dans les détecteurs pour effectuer la détection de visage sur l'image tournée pour déterminer si le visage candidat correspond à un visage.
EP09836139.7A 2008-12-30 2009-12-14 Procédé, appareil et produit de programme d'ordinateur pour fournir une estimation de pose de visage Active EP2370932B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US12/346,158 US8396263B2 (en) 2008-12-30 2008-12-30 Method, apparatus and computer program product for providing face pose estimation
PCT/IB2009/007744 WO2010076621A1 (fr) 2008-12-30 2009-12-14 Procédé, appareil et produit de programme d'ordinateur pour fournir une estimation de pose de visage

Publications (3)

Publication Number Publication Date
EP2370932A1 EP2370932A1 (fr) 2011-10-05
EP2370932A4 EP2370932A4 (fr) 2012-09-26
EP2370932B1 true EP2370932B1 (fr) 2018-04-04

Family

ID=42285068

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09836139.7A Active EP2370932B1 (fr) 2008-12-30 2009-12-14 Procédé, appareil et produit de programme d'ordinateur pour fournir une estimation de pose de visage

Country Status (7)

Country Link
US (1) US8396263B2 (fr)
EP (1) EP2370932B1 (fr)
JP (1) JP2012512478A (fr)
KR (1) KR20110100666A (fr)
CN (1) CN102272774B (fr)
RU (1) RU2011130626A (fr)
WO (1) WO2010076621A1 (fr)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102147851B (zh) * 2010-02-08 2014-06-04 株式会社理光 多角度特定物体判断设备及多角度特定物体判断方法
US9251588B2 (en) 2011-06-20 2016-02-02 Nokia Technologies Oy Methods, apparatuses and computer program products for performing accurate pose estimation of objects
CN103177269B (zh) 2011-12-23 2017-12-15 北京三星通信技术研究有限公司 用于估计对象姿态的设备和方法
JP6222900B2 (ja) * 2012-07-09 2017-11-01 キヤノン株式会社 画像処理装置、画像処理方法およびプログラム
US9727776B2 (en) * 2014-05-27 2017-08-08 Microsoft Technology Licensing, Llc Object orientation estimation
CN109325393B (zh) * 2017-08-01 2022-12-09 苹果公司 使用单一网络的面部检测、姿态估计和距相机距离的估计
EP3629290B1 (fr) * 2018-09-26 2023-01-04 Apple Inc. Localisation pour dispositifs mobiles
CN110796029B (zh) * 2019-10-11 2022-11-11 北京达佳互联信息技术有限公司 人脸校正及模型训练方法、装置、电子设备及存储介质
CN111339345B (zh) * 2020-02-26 2023-09-19 北京国网信通埃森哲信息技术有限公司 多平台人脸识别服务接口差异化屏蔽方法、系统及存储介质

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5497430A (en) * 1994-11-07 1996-03-05 Physical Optics Corporation Method and apparatus for image recognition using invariant feature signals
US6940545B1 (en) * 2000-02-28 2005-09-06 Eastman Kodak Company Face detecting camera and method
US7050607B2 (en) * 2001-12-08 2006-05-23 Microsoft Corp. System and method for multi-view face detection
US7024033B2 (en) * 2001-12-08 2006-04-04 Microsoft Corp. Method for boosting the performance of machine-learning classifiers
US7689033B2 (en) * 2003-07-16 2010-03-30 Microsoft Corporation Robust multi-view face detection methods and apparatuses
KR100643303B1 (ko) * 2004-12-07 2006-11-10 삼성전자주식회사 다면 얼굴을 검출하는 방법 및 장치
EP1772816A1 (fr) 2005-10-09 2007-04-11 Omron Corporation Appareil en procédé de détection d'une personne
CN100472556C (zh) 2005-10-09 2009-03-25 欧姆龙株式会社 特定被摄体检测装置及方法
JP4540661B2 (ja) * 2006-02-28 2010-09-08 三洋電機株式会社 物体検出装置
JP2008021266A (ja) 2006-07-14 2008-01-31 Matsushita Electric Ind Co Ltd 顔向き検出装置、顔向き検出方法、および顔向き検出プログラム
JP2008191816A (ja) 2007-02-02 2008-08-21 Sony Corp 画像処理装置、および画像処理方法、並びにコンピュータ・プログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
CN102272774B (zh) 2014-12-10
WO2010076621A1 (fr) 2010-07-08
KR20110100666A (ko) 2011-09-14
RU2011130626A (ru) 2013-03-27
EP2370932A1 (fr) 2011-10-05
US20100166317A1 (en) 2010-07-01
JP2012512478A (ja) 2012-05-31
CN102272774A (zh) 2011-12-07
US8396263B2 (en) 2013-03-12
EP2370932A4 (fr) 2012-09-26

Similar Documents

Publication Publication Date Title
EP2370932B1 (fr) Procédé, appareil et produit de programme d'ordinateur pour fournir une estimation de pose de visage
US11610384B2 (en) Zero-shot object detection
US10235623B2 (en) Accurate tag relevance prediction for image search
US20200250465A1 (en) Accurate tag relevance prediction for image search
US10572072B2 (en) Depth-based touch detection
Prasad et al. Edge curvature and convexity based ellipse detection method
US9448651B2 (en) Intelligent touchscreen keyboard with finger differentiation
WO2017088727A1 (fr) Procédé et appareil de traitement d'image
CN107077609B (zh) 用于检测在空间上不同的时间模式的非参数化模型
US9207757B2 (en) Gesture recognition apparatus, method thereof and program therefor
EP2370931B1 (fr) Procédé, appareil et programme d ordinateur pour fournir un détecteur de visages indépendant de l'orientation
EP3329399A1 (fr) Fusion et classification de données avec contexte d'ensembles de données déséquilibrés
CN111414888A (zh) 低分辨率人脸识别方法、系统、装置及存储介质
US11915500B2 (en) Neural network based scene text recognition
CN110852311A (zh) 一种三维人手关键点定位方法及装置
CN103793926A (zh) 基于样本重选择的目标跟踪方法
JP2021534480A (ja) 顔認識方法、装置、電子機器及びコンピュータ不揮発性読み取り可能な記憶媒体
Nanni et al. Combining face and eye detectors in a high-performance face-detection system
US20210406568A1 (en) Utilizing multiple stacked machine learning models to detect deepfake content
US11809990B2 (en) Method apparatus and system for generating a neural network and storage medium storing instructions
Khurana et al. Static hand gestures recognition system using shape based features
Lahiani et al. Hand pose estimation system based on Viola-Jones algorithm for android devices
CN110020593B (zh) 信息处理方法及装置、介质及计算设备
Karappa et al. Detection of sign-language content in video through polar motion profiles
Khoh et al. Spatiotemporal spectral histogramming analysis in hand gesture signature recognition

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20110510

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20120823

RIC1 Information provided on ipc code assigned before grant

Ipc: G06K 9/00 20060101AFI20120817BHEP

Ipc: G06K 9/62 20060101ALI20120817BHEP

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NOKIA CORPORATION

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: NOKIA TECHNOLOGIES OY

17Q First examination report despatched

Effective date: 20170406

RIC1 Information provided on ipc code assigned before grant

Ipc: G06K 9/00 20060101AFI20170921BHEP

Ipc: G06K 9/62 20060101ALI20170921BHEP

Ipc: G06T 7/77 20170101ALN20170921BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20171117

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 986343

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180415

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602009051658

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20180404

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 986343

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180404

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180806

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602009051658

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

26N No opposition filed

Effective date: 20190107

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20181214

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181214

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20181231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181214

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181231

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181231

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181214

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181214

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20091214

Ref country code: MK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180404

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180404

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180804

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602009051658

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06K0009000000

Ipc: G06V0010000000

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231031

Year of fee payment: 15