KR20130137999A - Method and system for measuring viewership of audience - Google Patents

Abstract

The present invention relates to a method for measuring viewer ratings of viewers and a system thereof. A method for generating a set of viewer face tracking information capable of measuring the viewer ratings of display devices corresponding to a set of viewer information comprises a step for detecting a facial area of a viewer from images extracted from a video inputted through a video input means positioned in one position of a display device; a step for detecting a facial feature from the detected facial area; a step for estimating an optimal conversion matrix which generates a 3D viewer facial model corresponding to the facial feature by converting a model feature of a 3D standard facial model; and a step for generating the viewer facial tracking information by estimating the viewer information based on the optimal conversion matrix. [Reference numerals] (100) Video input means;(200) Face detection module;(300) 3D modeling;(400) 3D face tracking;(500) Track eyes, eyebrows, and lips;(600) 3D data output

Description

How to measure viewership and its system {Method and system for measuring viewership of audience}

The present invention relates to a method and system for measuring viewer ratings, and more particularly, to detect facial feature points in a viewer's face from an image extracted from an image input through an image input means, and to detect such facial feature points. The present invention relates to a method and a system for measuring viewer ratings that generate information on viewer ratings such as gender, age, viewer's gaze direction, and gaze distance using facial feature points and optimal transformation matrix.

In general, a people meter is a dedicated terminal for measuring viewer ratings, and refers to a device installed in each panel generation to collect, store, and transmit data on viewer ratings at the time of viewing.

The people meter terminal is a device used to obtain viewer ratings by individually inputting the viewing of each household member by using a remote controller while mechanically examining on / off for each channel using a mechanical device.

The people meter as described above is composed of a device which is connected to the left and right of the TV and a telephone line and stores and judges and transmits the ratings in addition to the display, and a handset. It allows you to.

The People Meter automatically records viewer ratings on the People Meter, allowing for more accurate surveys and faster, more consistent measurements than those that rely on memory or questionnaires.

Based on the measured data, advertisers can get detailed information such as sales trends and time periods of new products and promotional products, sluggish sales, sales trends of similar products or competitors, and so on. Marketing strategies such as advertising plans can be effectively established.

However, the conventional people meter terminal absolutely requires the cooperation of the panel family. Pressing the handset correctly and pressing it can be tedious, especially for children, which can be mischievous, which can affect audience ratings. And with just the current people meter, you can't tell whether you're watching CM correctly.

That is, the conventional people meter terminal could not utilize for generating information based on the personal information of the viewer, such as information on the age-specific preferences of specific viewers, gender preference of specific products, and the like. In addition to the various types of information as a reference, a terminal related technology for measuring viewer ratings for generating various statistical information by generating information based on human information based on a viewer is required.

An object of the present invention for solving the problems according to the prior art is to detect the facial features in the viewer's face from the image extracted from the image input through the image input means provided at one position on the TV viewer side, and the facial features As the system generates personal information such as gender and age of the viewer, a method and system for measuring viewer ratings for generating various statistics based on the viewer's personal information such as gender and age-specific viewing information In providing.

One embodiment of the present invention for achieving the above object, a viewer face tracking information generation method for measuring the viewer ratings corresponding to at least one information of the viewer's gaze direction and gaze distance, (a) the display device Detecting a face region of the viewer from an image extracted from an image input through an image input means provided at one side of the side; (b) detecting a facial feature point in the detected face region; (c) estimating an optimal transformation matrix for generating a 3D viewer face model corresponding to the face feature by converting the model feature points of the 3D standard face model; And (d) estimating at least one of the gaze direction and gaze distance of the viewer based on the optimal transformation matrix to generate viewer face tracking information.

According to another embodiment of the present invention, a viewer face tracking information generation method for measuring an audience rating in response to at least one piece of information of a viewer's gaze direction and gaze distance, provided at a position on the display device side A face region detection step of detecting a face region of the viewer from an image extracted from an image input through the input image input means; A gaze information generation step of generating gaze information by estimating at least one information of gaze direction and gaze distance of the viewer based on the detected face region; And generating viewer information by estimating at least one piece of information of the gender and the age of the viewer based on the detected face region.

According to another aspect of the present invention, there is provided a computer-readable recording medium recording a program for executing each step of the viewer face tracking information generation method.

According to another aspect of the invention, there is provided a display device for measuring the viewer rating using the viewer face tracking information generation method.

In accordance with another aspect of the present invention, there is provided a viewer face tracking information generation device for measuring an audience rating of a display device in response to at least one piece of information of a viewer's gaze direction and gaze distance. A face region detection module for detecting a face region of the viewer from an image extracted from an image input through an image input means provided at a position; A facial feature point detection module for detecting a facial feature point in the detected face area; A matrix estimation module for transforming a model feature point of a 3D standard face model to estimate an optimal transformation matrix for generating a 3D viewer face model corresponding to the face feature point; And a tracking information generation module for estimating at least one of a gaze direction and a gaze distance of the viewer based on the estimated optimal transformation matrix to generate viewer face tracking information.

In accordance with another aspect of the present invention, there is provided a viewer face tracking information generation device for controlling a stereoscopic feeling of a display device in response to at least one of a gaze direction and a gaze distance of a viewer, Means for detecting a face region of the viewer from an image extracted from an image input through an image input means provided at a position; Means for generating gaze information by estimating at least one of gaze direction and gaze distance of the viewer based on the detected face region; And means for estimating at least one of gender and age of the viewer based on the detected face region to generate viewer information.

As described above, the present invention estimates the gaze direction and gaze distance of a viewer by using an optimal transformation matrix for converting the model feature points of the 3D standard face model to generate a 3D viewer face model corresponding to the face feature points of the face region. Therefore, the fast tracking speed is suitable for real-time tracking, and there is an advantage in that the accurate face area can be tracked even with some distortion of the face area.

In addition, since it is determined whether the detected face area is valid and face feature points are detected for the face area determined to be valid, there is an advantage that the detection reliability of the face feature point is high and the tracking performance of the face area is increased.

In addition, since an asymmetric similar feature (harr-like feature) is used to detect the non-frontal face region, the detection reliability of the face region with respect to the non-frontal face is high, thereby increasing the tracking performance of the face region.

In addition, basically, the gaze direction and gaze distance of the viewer are estimated to generate gaze direction information and gaze distance information, and additionally, at least one of the gender or age of the viewer is estimated to generate viewer information. And not only gaze distance information, but also by additionally utilizing the viewer information to measure the viewer rating, there is an advantage that can be measured more accurate viewer rating.

In addition, by estimating whether or not the viewer's eyes are closed, when the viewer's eyes viewing the display device are estimated to be closed, the screen output of the display device may be used as information for turning off or stopping playback.

In addition, there is an advantage that it is possible to accurately track the gaze direction, gaze distance of the viewer with only one image input means (for example, a camera).

1 is a block diagram showing a schematic configuration of an apparatus for generating face tracking information for measuring viewer ratings according to an embodiment of the present invention;
Figure 2 is a photo showing the 28 facial features that can be defined for each position of the eyebrow, eye, nose, mouth
3 Model of 3D mesh form consisting of 331 points and 630 triangles
4 is a flowchart illustrating a process of creating a 3D face model by detecting a face
5 is a structural diagram showing the overall structure of face detection using the Adaboost algorithm
6 is a flowchart illustrating a 3D face tracking process.
Fig. 7 Extracting 4 feature vectors using 12 feature points
8 is a plan view illustrating a measurement environment related to generation of viewer face tracking information for viewer rating measurement
9 3D face model tracking result screen generated through face recognition
10 A viewer human recognition implementation screen through face recognition

The present invention can be embodied in many other forms without departing from the spirit or main features thereof. Accordingly, the embodiments of the present invention are to be considered in all respects as merely illustrative and not restrictive.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, the terms "comprises", "having", "having", and the like are intended to specify the presence of stated features, integers, steps, operations, components, Steps, operations, elements, components, or combinations of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or corresponding elements are denoted by the same reference numerals, and a duplicate description thereof will be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a schematic configuration of an apparatus for generating face tracking information for measuring viewer ratings according to an embodiment of the present invention.

Disclosed is a viewer face tracking information generation apparatus for measuring an audience rating in response to at least one piece of information of a viewer's gaze direction and gaze distance.

The viewer face tracking information generating device may be a conventional computer system having a computing element such as a central processing unit, a system DB, a system memory, an interface, and the like, connected to a display device such as a TV so as to transmit and receive control signals. It can be regarded as functioning as a viewer's face tracking information generating device by installing and driving the viewer's face tracking information generating program in the computer system of the present invention.

In another aspect, the viewer face tracking information generating device of the present embodiment may be configured in the form of an embedded device in a display device such as a TV.

A description of the general configuration of such a computer system is omitted, and the following description will focus on the configuration of functional aspects required for the description of the embodiments of the present invention.

The viewer face tracking information generation device includes a face area detection module 200.

The face region detection module 200 may include a face region of the viewer from an image captured by an image input means 100 provided at one position of the display apparatus, for example, an image input through a camera. Is detected. In this case, the detection viewing angle may be all faces in the range of -90 to +90.

The image input means 100 is, for example, installed at the top or bottom side of the central portion of the TV, a camera that can capture the viewer's face in front of the TV screen in real time, more preferably, an image sensor Can be a digital camera attached.

Even if only one image input means 100 is provided, the viewer face tracking information described later may be generated.

As shown in FIG. 1, the facial feature point 200 is first detected from an image stream input from the image input means 100. The 3D model 300 of the face is generated by extracting feature points of the detected face region. The 3D face position 400 is tracked from the camera stream based on the generated 3D face model. In addition to tracking progress for the entire face, it also tracks the 3D position of the detailed areas for the eyes, eyebrows, and lips 500. The tracked 3D face data is displayed on the screen or stored in metadata format (600).

The facial feature point detection module 200 performs facial feature point detection on face areas determined to be valid, and includes, for example, an angle of eyebrows, eyes, nose, and mouth as shown in FIG. 2. It is possible to detect 28 facial feature points that can be defined for location.

In the present embodiment, preferably, a total of eight feature points of four eyes, two noses, and two mouths, which are basic facial feature points, may be detected as facial feature points.

Here, the 3D standard face model may be a 3D mesh model composed of 331 points and 630 triangles, as shown in FIG. 3.

4 is a flowchart illustrating a process of creating a 3D face model by detecting a face. In general, a process of creating a 3D face model is generated by calculating depth information using a phase difference using two cameras. In the present patent, a face region obtained through a single camera input is a 3D transformation matrix using a front face model. We construct a 3D face model and map it to a 3D face model to obtain 3D coordinates.

5 is a structural diagram showing the overall structure of face detection using the Adaboost algorithm. When the input stream S310 comes in, the face candidate region S320 is obtained from the skin color and the elliptical face model in the image of the stream. A partial image having a size of 24x24 is extracted from the face candidate region (S330), and the medicine classifier S340 obtained through learning is substituted into the image to determine the face. In the present patent, the input image is determined to be a face S360 when the value of the feature points is calculated using 28 feature points and the value exceeds a threshold for determining that the face is a face. In order to cope with various face sizes, a face is detected while repeating the previous process through a stream image pyramid process.

The detected face region is matched with the front face of the pre-made standard 3D face model (Wire Frame Model) as shown in FIG. 3 to determine the position by adaboost for each feature point (eye, nose, mouth, etc.). After the validation test based on the matching rate with the standard 3D model, the feature point of the lip region using the mouth model is detected again, and the feature of the eyebrow using the eyebrow model is extracted. In order to reduce the error, a 3D face map is constructed by best matching using a nonlinear optimiser by NelderMead.

In order to detect a feature point, a method of thresholding the distribution of luminance values of an image is used. The threshold value of the luminance value distribution corresponding to the face is determined by projecting the multi-level thresholded section vertically by probabilistic statistical analysis obtained from arbitrary target images. The face is divided from components other than the face according to the determined threshold value, and facial feature points (eye, nose, mouth, etc.) are extracted based on the divided face parts.

The generated 3D face model will contain eight basic points: two at the ends of the eyes, two at the nose and two at the end of adoption. If the eyebrows and lips are included, the 3D model including 28 points is included.

6 is a flowchart illustrating a 3D face tracking process.

As shown, the generated 3D face model 210 according to the present embodiment establishes a hypothesis 240 that can move up, down, left, and right based on the current position to track the face movement, and based on the hypothesis, The position evaluator sets the position. The background model is updated 250 by the newly input stream image, and the location of the new feature point is calculated 270 based on the information of the model 260 in the history to track the 3D face model.

For eyebrow tracking, after generating Eyebrow PCA Shape Model, calculate Eyebrow Correspondences and optimize using Eyebrow PCA Minimizer. Unlike eyebrows, lip tracking consists of complex movements such as lip movements, mouth opening, jaw movements, lip protrusions and cheek movements. Therefore, these complex movements should be considered, but the protrusion of the lips and the movement of the cheeks are difficult with a two-dimensional approach. Therefore, the 3D feature vector can be obtained from the 3D model matched with the input video.

7 extracts four feature vectors using twelve feature points. The four feature vectors are as follows.

Upper lip movement vector (UP): UL = P2-P0

Lower lip movement vector (Down): DL = P3-P0

Mouth openness: ML = P5-P6

-Up and down movement of jaw: JM = P7 P10

In order to deal with such a motion vector quantitatively, a basic amount capable of absorbing individual differences is required. In this case, the width and width of the mouth are used as the standard quantities. The per-frame movement displacement of the motion vector tracks the lips by normalizing the movement amount of the feature point to a reference amount.

8 is a plan view illustrating a measurement environment related to generation of viewer face tracking information for viewer rating measurement.

The estimated viewer eye information is output to the screen power control means as information for controlling ON / OFF of the 3 display apparatus screen output ON / OFF.

That is, when it is estimated that the viewer's eye-closing state continues, the screen power control means may turn off the image output to the display device screen so that no further image output is performed.

9 is a 3D face model tracking result screen generated through face recognition.

It is a human recognition test result screen to check whether a viewer's face is recognized as 3D and overlayed it with a 3D wire frame on the original face.

10 is a viewer human recognition implementation screen through face recognition.

The right side of FIG. 10 is a standard 3D face model, and the left side is a screen in which 3D face model information is overlaid on the recognized face.

Embodiments of the present invention include a computer readable recording medium including program instructions for performing various computer-implemented operations. The computer-readable recording medium may include a program command, a data file, a data structure, and the like, alone or in combination. The recording medium may be one specially designed and configured for the present invention, or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs, DVDs, magnetic-optical media such as floppy disks, and ROM, RAM, flash memory, and the like. Hardware devices specifically configured to store and execute the same program instructions are included. The recording medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

Although the present invention has been described with reference to the preferred embodiments thereof with reference to the accompanying drawings, it will be apparent to those skilled in the art that many other obvious modifications can be made therein without departing from the scope of the invention. Accordingly, the scope of the present invention should be interpreted by the appended claims to cover many such variations.

100: video input means
200: facial feature detection
300: 3D face modeling
400: 3D Face Tracking
500: eyebrow and lip tracking
600: viewership data output

Claims (3)

A method for generating viewer face tracking information for measuring an audience rating of a display device in response to at least one of gazing direction and gaze distance for each gender age of a viewer,
(a) detecting a face region of the viewer from an image extracted from an image input through an image input means provided at one position of the display apparatus;
(b) detecting a facial feature point in the detected face region;
(c) estimating an optimal transformation matrix for generating a 3D viewer face model corresponding to the face feature by converting the model feature points of the 3D standard face model; And
(d) generating viewer face tracking information by estimating at least one of the gaze direction and gaze distance for each gender of the viewer based on the optimal transformation matrix; and generating a viewer face tracking information. . The method of claim 1,
After the step (d)
and (g) estimating eyelids of the viewer using the detected face region. A method for generating viewer face tracking information for measuring an audience rating of a display device in response to at least one of gazing direction and gaze distance for each gender age of a viewer,
A face region detecting step of detecting a face region of the viewer from an image extracted from an image input through an image input means provided at one position of the display apparatus;
A gaze information generation step of generating gaze information by estimating at least one information of gaze direction and gaze distance of the viewer based on the detected face region; And
And generating viewer information by estimating at least one piece of information of a gender and an age of the viewer based on the detected face region.

Images