KR101802062B1

KR101802062B1 - Method and system to measure stereoscopic 3d content-induced visual discomfort based facial expression recognition for sensitivity tv

Info

Publication number: KR101802062B1
Application number: KR1020160030246A
Authority: KR
Inventors: 노용만; 이성일
Original assignee: 한국과학기술원
Priority date: 2016-03-14
Filing date: 2016-03-14
Publication date: 2017-11-27
Also published as: KR20170106755A

Abstract

A method of measuring visual fatigue due to stereoscopic 3D content based on facial expression recognition for emotional TV comprises the steps of: acquiring a face image of the viewer when the 3D content is provided to a viewer; Performing face area detection / normalization on the face image; Extracting a facial feature from the normalized facial image based on at least one of a facial geometry or an appearance; And applying a machine learning algorithm to the facial feature to determine whether the viewer is uncomfortable.

Description

TECHNICAL FIELD The present invention relates to a method and system for measuring visual fatigue due to stereoscopic 3D contents based on facial expression recognition for an emotional TV. BACKGROUND OF THE INVENTION 1. Field of the Invention [0002]

The following embodiments relate to a system and method for measuring visual fatigue due to stereoscopic 3D content based on facial expression recognition for emotional TV, and more particularly, to a system and method for measuring visual fatigue caused by a viewer's discomfort Quot ;. < / RTI >

Recently, there is an increasing demand for emotional TV technology for analyzing emotional information (for example, gestures, facial expressions, or melancholy) of viewers to search for / recommend suitable media or automatically change content episodes.

Accordingly, the conventional emotional TV technology mainly aims at classifying the basic emotions (for example, happiness, surprise, nausea, anger, fear, or sadness) caused by the experience of viewing 2D contents.

However, as content provided on TV changes from 2D to 3D, basic emotion classification for 3D contents is required. Particularly, stereoscopic 3D (S3D) TV can improve the perceived viewing quality by providing better presence than 2D TV, but it can also cause negative feeling of visual inconvenience. Therefore, A technique for measuring the visual discomfort due to watching the S3D contents is required.

Conventional discomfort measurement technology uses various sensing technologies such as eye movement, brain wave, and fMRI to measure the viewer's visual discomfort about 3D contents. However, considering the convenience of the viewer and the cost of the equipment, there is a limit to apply to the TV viewing environment.

Therefore, the following embodiments propose a camera image-based visual fatigue measurement technique which is relatively unrestricted.

In particular, since the correlation between the change in the facial expression of the viewer and the inconvenience is not revealed by the existing literature and the invention, it is difficult to immediately apply the technique for recognizing the change in the facial expression in the past. Accordingly, since the conventional techniques can not achieve the objects to be achieved by the following embodiments, the following embodiments provide a new discomfort analysis technique applying a technique of recognizing a facial expression change.

The following embodiments provide a method and system for measuring visual fatigue attributable to 3D content by analyzing a change in facial expression of a viewer based on a camera image.

Specifically, in the following embodiments, a facial feature of a viewer is extracted based on a face image of a viewer obtained through a camera, and a visual fatigue Measuring method and system.

According to an embodiment, there is provided a method of measuring visual fatigue due to stereoscopic 3D content based on facial expression recognition for emotional TV, comprising: obtaining a face image of the viewer when the 3D content is provided to a viewer; Performing face area detection / normalization on the face image; Extracting a facial feature from the normalized facial image based on at least one of a facial geometry or an appearance; And applying a machine learning algorithm to the facial feature to determine whether the viewer is uncomfortable.

Wherein extracting the facial feature from the normalized facial image comprises: extracting a face landmark from the normalized facial image; Calculating a displacement between the extracted face landmark and a reference face landmark previously extracted from a face image of the viewer in a normal time; And obtaining the displacement with the facial feature.

The step of determining whether the viewer is uncomfortable by applying the machine learning algorithm to the facial feature may include determining whether the viewer is uncomfortable by performing the machine learning algorithm with the displacement input.

Wherein the step of determining whether the viewer is uncomfortable by performing the machine learning algorithm based on the displacement is characterized in that when the deviation is greater than or equal to a preset reference value, And judging that there is a presence.

The step of performing face area detection / normalization on the face image includes: detecting a face area from the face image using an object detection algorithm; And performing normalization by performing scale / rotation compensation on the detected face area.

The step of applying the machine learning algorithm to the facial feature to determine whether the viewer is uncomfortable may further include adjusting a depth sense of the 3D content based on the determination result.

The adjusting the depth of the 3D content based on the determination result may include performing either disparity scaling or disparity shifting on the 3D content.

Wherein the step of determining whether the viewer feels uncomfortable by applying the machine learning algorithm to the facial feature comprises: collecting features of the 3D content when it is determined that the viewer's discomfort exists; Generating a discomfort pattern of the viewer by using the feature of the 3D content; And recommending at least one 3D content of the plurality of 3D contents to the viewer based on the discomfort pattern of the viewer.

According to one embodiment, a system for measuring visual fatigue based on facial expression recognition based stereoscopic 3D content for emotional TV includes a face image obtaining unit for obtaining a face image of the viewer when the 3D content is provided to a viewer; A face detection / normalization unit for performing face detection / normalization on the face image; A facial feature extraction unit for extracting a facial feature from the normalized facial image based on at least one of facial geometry and appearance; And a discomfort determination unit for determining whether the viewer feels uncomfortable by applying a machine learning algorithm to the facial feature.

The facial feature extraction unit extracts a face landmark from the normalized face image and calculates a displacement between the extracted face landmark and a reference face landmark extracted in advance from the normal facial image of the viewer , The displacement can be obtained with the facial feature.

The discomfort judgment unit may determine whether the viewer is uncomfortable by performing the machine learning algorithm with the displacement input.

The discomfort judgment unit may determine that the discomfort of the viewer is present when the defect value on which the machine learning algorithm is performed is equal to or greater than a preset reference value.

The face detection / normalization unit may detect a face region from the face image using an object detection algorithm, and perform normalization by performing scale / rotation compensation on the detected face region.

The discomfort determination unit may adjust the depth sense of the 3D content based on the determination result.

Wherein the discomfort determination unit collects the 3D feature information and generates a discomfort pattern of the viewer using the 3D feature if the discomfort of the viewer is determined as a result of the determination, It is possible to recommend at least one of the plurality of 3D contents to the viewer based on the discomfort pattern of the viewer.

The embodiments described below can provide a method and system for measuring visual fatigue attributable to 3D contents by analyzing changes in facial expression of a viewer based on a camera image.

Specifically, in the following embodiments, a facial feature of a viewer is extracted based on a face image of a viewer obtained through a camera, and a visual fatigue A measurement method and a system can be provided.

Therefore, the following embodiments are different from the discomfort measuring technique using various sensing technologies such as eye movements, brain waves, and fMRI, so that the embodiments do not use additional equipment, so that they are easy to implement, costly, A method and system for measuring visual fatigue can be provided.

Such a visual fatigue measurement method and system can be utilized as an emotional TV technology that enables an auto-interaction between 3D content and viewer's discomfort.

1 is a view for explaining the principle of a method of measuring visual fatigue according to an embodiment.
2 is a view for explaining a method of measuring visual fatigue according to an embodiment.
FIGS. 3A and 3B are views for explaining a process of extracting the facial feature shown in FIG. 2 and a process of determining whether the facial feature is uncomfortable.
4 is a view illustrating a result of performing a visual fatigue measurement method according to an embodiment.
5 is a diagram illustrating an emotional TV technique based on a visual fatigue measurement method according to an embodiment.
FIG. 6 is a diagram illustrating an emotional TV technique based on a visual fatigue measurement method according to another embodiment.
FIG. 7 is a flowchart illustrating a method of measuring visual fatigue according to an embodiment.
8 is a block diagram illustrating a visual fatigue measurement system according to an embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Also, terminologies used herein are terms used to properly represent preferred embodiments of the present invention, which may vary depending on the viewer, the intention of the operator, or the custom in the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

1 is a view for explaining the principle of a method of measuring visual fatigue according to an embodiment.

Referring to FIG. 1, a method for measuring visual fatigue according to an embodiment is performed based on a face image of a viewer when 3D content is provided to a viewer.

For example, in the face image 120 during the viewing of the 3D content 110 having a low depth of feeling (hereinafter, the 3D content 110 and 130 are normally not provided), the viewer does not have a special feeling But the facial expression of a viewer who feels discomfort such as frowning in the face image 140 while watching the 3D content 130 having a high depth of view can be observed.

Accordingly, the visual fatigue measurement method according to an embodiment analyzes the facial expression change of the viewer based on the face images 120 and 140 of the viewer when the 3D contents 110 and 130 are provided, 130 can be judged by the viewer. A detailed description thereof will be described with reference to Fig.

2 is a view for explaining a method of measuring visual fatigue according to an embodiment.

Referring to FIG. 2, a method for measuring visual fatigue according to an embodiment is performed by the visual fatigue measurement system 200. Here, the visual fatigue measurement system 200 may be realized in the form of a server linked to a terminal of a viewer, or in the form of a program code of an application installed in a viewer's terminal.

The visual fatigue measurement system 200 acquires the face image of the viewer when the 3D content is provided to the viewer through the face image acquisition unit (210). For example, the face image acquisition unit may be a camera that captures a face image of a viewer when a 3D content is provided to a viewer.

Next, the visual fatigue measurement system 200 performs face area detection / normalization on the face image through the face detection / normalization unit (220). For example, the face detection / normalization unit may perform normalization by performing scale / rotation compensation on the detected region after detecting the face region from the face image using an object detection algorithm (220). More specifically, the face detection / normalization unit can detect the face region from the face image using the Viola & Jones algorithm, and compensate the scale and rotation for the face region based on the position information of the eye in the detected region (220). However, the algorithm and normalization method used by the face detection / normalization unit may be various algorithms and normalization methods for preprocessing the face image so as to analyze changes in the facial expression of the viewer.

The visual fatigue measurement system 200 then extracts the facial feature from the normalized facial image based on at least one of the facial geometry or appearance through the facial feature extraction unit 230, . At this time, the facial feature may be a parameter indicating the change of facial expression of the viewer, and may be a parameter related to a face landmark (e.g. eyebrow, eye, nose, mouth, etc.) in the face region in the normalized face image.

That is, the facial expression feature extraction unit 230 can extract the facial feature, which is a parameter related to the face landmark in the face region in the normalized face image, based on at least one of the face geometry and the shape. A detailed description thereof will be described with reference to Figs. 3A to 3B.

Thereafter, the visual fatigue measurement system 200 applies a machine learning algorithm to the facial feature through the inconvenience determination unit 240 to determine whether the viewer feels uncomfortable (240). For example, the inconvenience determination unit 240 may determine that the inconvenience of the viewer exists (240) if the result of applying the machine learning algorithm to the facial feature is equal to or greater than a preset reference value. A detailed description thereof will be described with reference to Figs. 3A and 4.

In this way, the visual fatigue measurement system 200 extracts the facial feature of the viewer based on the facial image of the viewer (230), determines whether the viewer feels uncomfortable with respect to the 3D content using the extracted facial feature, It is possible to provide emotional TV technology that automatically changes the 3D content according to whether or not it is uncomfortable. A detailed description thereof will be described with reference to Figs. 5 to 6. Fig.

FIGS. 3A and 3B are views for explaining a process of extracting the facial feature shown in FIG. 2 and a process of determining whether the facial feature is uncomfortable.

3A, the facial feature extracting unit included in the visual fatigue measuring system 300 according to an exemplary embodiment of the present invention includes a face image acquiring unit and a face detecting / normalizing unit as described with reference to FIG. 2, (Step 310).

Specifically, the facial feature extraction unit extracts a face landmark from the face image on which the normalization has been performed (311), and calculates a displacement between the extracted face landmark and the reference face landmark extracted in advance from the normal face image of the viewer By calculating (312), the calculated displacement can be obtained as a facial feature.

3B, in general, a change in the facial expression of a viewer (particularly, a change in facial expression when there is an uncomfortable feeling) is detected by a face landmark 320 (see FIG. 3B), which is an eyeball, eye, nose, As shown in FIG. Thus, the change between the viewer's normal facial expression and the viewer's particular facial expression when the 3D content is presented is usually expressed as the displacement between the viewer's facial landmark and the viewer's facial landmark when the 3D content is presented .

Accordingly, the facial expression feature extraction unit uses the displacement between the reference face landmark of the viewer and the face landmark of the viewer when the 3D content is provided, as a facial feature to determine whether or not the viewer is uncomfortable.

For example, the facial feature extraction unit may calculate the displacement d by calculating the displacement d between the face landmark extracted from the normalized facial image and the reference face landmark as shown in Formula 1 (312) It can be used as a facial feature in judging process.

In Equation 1,

Represents the kth face landmark extracted from the face image subjected to the normalization,

Represents the kth reference face landmark. Here, k represents each of the points of the viewer's face landmark 320, and the total number of the face landmarks 320 is shown as 49 in the drawing, but the present invention is not limited to this and is not limited thereto, and an appropriate number Lt; / RTI >

At this time, in the process of extracting the face landmark from the normalized face image, when the viewer wears the 3D glasses, the occlusion effect can not be ignored. Accordingly, the facial expression feature extraction unit may extract the face landmark from the normalized facial image using at least one of face geometry and shape so that the occlusion effect is minimized (311). In addition, the facial expression feature extraction unit may perform additional image processing operations such as histogram adjustment on the normalized facial image so that the occlusion effect is minimized.

In other words, if the machine learning algorithm is performed by inputting the displacement between the face landmark extracted by the facial feature extraction unit and the reference face landmark, it can be determined whether or not the viewer's discomfort exists (330).

Accordingly, the inconvenience determination unit included in the visual fatigue measurement system 300 can determine whether the viewer is uncomfortable by performing the machine learning algorithm with the displacement obtained in the facial feature extraction unit as an input (330). For example, the inconvenience determination unit may determine whether the viewer is uncomfortable by performing a machine learning algorithm using a SVM (support vector machine) classifier as input of the displacement (331). However, the present invention is not limited to this, and the inconvenience determination unit may include various machine learning methods that can classify / judge the inconvenience of the viewer by inputting the displacement between the face landmark extracted from the normalized face image and the reference face landmark Algorithm can be used.

In addition, the discomfort judgment unit not only determines whether the viewer is uncomfortable according to the defect value in which the machine learning algorithm is performed, by inputting the displacement between the face landmark extracted from the normalized face image and the reference face landmark, It is also possible to measure the degree of discomfort of the viewer (the parameter in which the viewer feels the discomfort scale).

4 is a view illustrating a result of performing a visual fatigue measurement method according to an embodiment.

Referring to FIG. 4, the visual fatigue system according to an exemplary embodiment of the present invention may be realized by inputting a displacement between a face landmark extracted from a normalized face image and a reference face landmark as input, as described with reference to FIGS. 3A through 3B. If the defect value 410 on which the learning algorithm is performed is equal to or greater than a preset reference value, it can be determined that the viewer's discomfort is present.

In addition, the visual fatigue system inputs the displacement between the face landmark extracted from the face image subjected to the normalization and the reference face landmark, and then, according to the variation of the defect value 410 in which the machine learning algorithm is performed, Can be judged to exist. For example, the visual fatigue system determines that the viewer feels uncomfortable in a period in which the variation of the defect value 410 rapidly increases (when the variation of the defect value 410 is sharper than the preset reference variation trend) .

Here, the visual fatigue system can use the defect value 410 in which the machine learning algorithm is performed, as the input of the displacement between the face landmark extracted from the normalized face image and the reference face landmark, as the degree of inconvenience of the viewer.

5 is a diagram illustrating an emotional TV technique based on a visual fatigue measurement method according to an embodiment.

Referring to FIG. 5, the visual fatigue measurement system 500 according to an embodiment determines whether or not a viewer feels uncomfortable, as described with reference to FIGS. 2 to 4, Can be adjusted.

For example, the visual fatigue measurement system 500 may be configured to perform disparity scaling or disparity shifting on the 3D content if it is determined 510 that the viewer's discomfort is present through the inconvenience determination unit 510, (520), the 3D content having the controlled depth can be provided to the viewer (530).

At this time, the inconvenience determination unit may use various rendering techniques such as depth image-based rendering (DIBR) or warping-based image rendering for depth control 520 such as parallax scaling or parallax movement.

In operation 510, the visual fatigue measurement system 500 determines whether a discomfort is sensed through the inconvenience determination unit 510. If the displacement is between the face landmark extracted from the normalized face image and the reference face landmark, The degree of the discomfort of the viewer can be determined according to the defect value on which the algorithm is performed, thereby determining the extent to which the depth sense of the 3D content is adjusted 520 based on the degree of discomfort. For example, the inconvenience judging unit adjusts the depth feeling of the 3D content so that the depth feeling of the 3D content differs more than the reference depth feeling of the 3D content when the degree of the inconvenience of the viewer is severe, The difference between the reference depth of the 3D content and the 3D depth may be adjusted to be small.

As described above, the visual fatigue measurement system 500 according to the embodiment maximizes the viewer's satisfaction with the 3D content by providing the viewer with the emotional TV technique of adjusting the depth feeling of the 3D content (520) by reflecting the inconvenience of the viewer can do.

FIG. 6 is a diagram illustrating an emotional TV technique based on a visual fatigue measurement method according to another embodiment.

Referring to FIG. 6, the visual fatigue measurement system 600 according to another embodiment determines whether or not a viewer is uncomfortable as described with reference to FIGS. 2 to 4 (610), and collects features of 3D contents 620), generates a discomfort pattern of the viewer (630), and recommends at least one of the plurality of 3D contents to the viewer based on the generated discomfort pattern (640).

In general, susceptibility to visual discomfort per person for the same 3D content may be different. Accordingly, the visual fatigue measurement system 600 determines whether or not the viewer is uncomfortable through the inconvenience determination unit (610). If it is determined that there is an uncomfortable feeling of the viewer, (620) and generate (630) a discomfort pattern of the viewer (e.g., the main features of the 3D content where the viewer feels discomfort).

Accordingly, the inconvenience determination unit stores and maintains the generated discomfort pattern of the viewer in the personal inconvenience database, thereby extracting at least one 3D content suitable for the discomfort pattern of the viewer among the plurality of 3D contents from the 3D content database , And may be recommended to the viewer (640).

As described above, the visual fatigue measurement system 600 according to another embodiment provides the viewer with the emotional TV technique for recommending the customized 3D content to the viewer by reflecting the inconvenience of the viewer, thereby maximizing the satisfaction of the viewer's 3D content .

FIG. 7 is a flowchart illustrating a method of measuring visual fatigue according to an embodiment.

Referring to FIG. 7, a visual fatigue measurement system according to an embodiment obtains a face image of a viewer when 3D content is provided to a viewer (710).

Next, the visual fatigue measurement system performs face area detection / normalization on the face image (720).

For example, in step 720, the visual fatigue measurement system may perform normalization by detecting a face area from a face image using an object detection algorithm, and then performing scale / rotation compensation on the detected face area.

Next, the visual fatigue measurement system extracts the facial feature from the normalized facial image based on at least one of the facial geometry and the appearance (730).

Specifically, in step 730, the visual fatigue measurement system extracts a face landmark from the normalized face image, and calculates a displacement between the extracted face landmark and the reference face landmark extracted in advance from the normal face image of the viewer ), The displacement can be obtained as a facial feature.

Thereafter, the visual fatigue measurement system applies a machine learning algorithm to the facial feature to determine whether the viewer is uncomfortable (740).

For example, in step 740, the visual fatigue measurement system performs a machine learning algorithm by inputting the displacement between the face landmark extracted from the normalized face image and the reference face landmark to determine whether the viewer is uncomfortable .

At this time, the visual fatigue measurement system can determine that there is a feeling of discomfort of the viewer when the defect value in which the machine learning algorithm is performed by inputting the displacement is equal to or greater than a preset reference value. In addition, the visual fatigue measurement system can use the displacement value as input, and the result of the machine learning algorithm is used as the degree of inconvenience of the viewer.

Also, although not shown in the figure, in step 740, the visual fatigue measurement system may adjust the depth sense of the 3D content based on the determination result. For example, the visual fatigue measurement system can adjust the depth of 3D content by performing either disparity scaling or disparity shifting on the 3D content.

Although it is not shown in the drawing, if the visual fatigue measurement system determines in step 740 that the viewer's discomfort is present, the system collects the characteristics of the 3D content, And may recommend at least one of the plurality of 3D contents to the viewer based on the discomfort pattern of the viewer.

8 is a block diagram illustrating a visual fatigue measurement system according to an embodiment.

8, the system for measuring visual fatigue includes a face image acquisition unit 810, a face detection / normalization unit 820, a facial expression feature extraction unit 830, and a discomfort sense determination unit 840 do.

The face image obtaining unit 810 obtains the face image of the viewer when the 3D content is provided to the viewer.

The face detection / normalization unit 820 performs face region detection / normalization on the face image. For example, the face detection / normalization unit 820 may perform normalization by performing scale / rotation compensation on the detected face region after detecting the face region from the face image using the object detection algorithm.

The facial expression feature extraction unit 830 extracts the facial feature from the normalized facial image based on at least one of the facial geometry and the appearance.

Specifically, the facial feature extracting unit 830 extracts a face landmark from the normalized facial image, and calculates a displacement between the extracted facial landmark and the reference face landmark extracted in advance from the normal facial image of the viewer, , The displacement can be obtained as a facial feature.

The inconvenience determination unit 840 determines the inconvenience of the viewer by applying a machine learning algorithm to the facial feature. For example, the discomfort determination unit 840 can determine whether the viewer is uncomfortable by performing a machine learning algorithm by inputting the displacement between the face landmark extracted from the normalized face image and the reference face landmark have.

At this time, the inconvenience judging unit 840 can judge that the inconvenience of the viewer is present when the defect value in which the machine learning algorithm is performed is equal to or larger than a preset reference value. In addition, the inconvenience determination unit 840 may use the displacement value as a degree of inconvenience of the viewer by inputting the machine learning algorithm.

The discomfort sense determiner 840 may adjust the depth sense of the 3D content based on the determination result. For example, the discomfort determination unit 840 may adjust the depth of the 3D content by performing either disparity scaling or disparity shifting on the 3D content.

If it is determined that the viewer's discomfort is present as a result of the determination, the discomfort sense determiner 840 generates the discomfort pattern of the viewer using the feature of the content by collecting the 3D content feature, The 3D content of at least one of the plurality of 3D contents may be recommended to the viewer based on the discomfort pattern.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magnetic recording media such as floptical disks, Includes hardware devices that are specially configured to store and execute program instructions such as magneto-OTPical media and ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims

A method of measuring visual fatigue due to stereoscopic 3D content based on facial expression recognition for emotional TV,
Obtaining a face image of the viewer when the 3D content is provided to a viewer;
Performing face region detection and normalization on the face image;
Extracting a face landmark from the normalized face image based on at least one of face geometry or appearance;
Extracting a facial feature by calculating a displacement between the extracted face landmark and a reference face landmark previously extracted from the facial image of the viewer; And
Determining whether the viewer feels uncomfortable by applying a machine learning algorithm that inputs the displacement to the facial feature;
Lt; / RTI >
The step of determining whether the viewer is uncomfortable by applying the machine learning algorithm
Determining that there is an uncomfortable feeling of the viewer if the result value to which the machine learning algorithm is applied is greater than or equal to a preset reference value,
When the result of applying the machine learning algorithm is equal to or greater than a preset reference value, the step of determining that the viewer's discomfort exists
Measuring a degree of discomfort of the viewer according to a result value to which the machine learning algorithm is applied; measuring a degree of discomfort of the viewer based on a measure of discomfort when the viewer views the 3D content;
Determining a degree of controlling a depth sense of the 3D content provided to the viewer based on the degree of discomfort; And
Adjusting a depth sense of the 3D content according to a degree of controlling the depth feeling of the 3D content
Lt; / RTI >
Collecting characteristics of the 3D content when it is determined that the viewer's discomfort exists;
Generating a discomfort pattern of the viewer by using the feature of the 3D content; And
And recommending at least one of the plurality of 3D contents to the viewer based on the discomfort pattern of the viewer
And measuring the visual fatigue.

delete

The method according to claim 1,
Wherein the step of performing face region detection and normalization on the face image comprises:
Detecting a face region from the face image using an object detection algorithm; And
Performing a scale on the detected face area, performing normalization by performing rotation compensation
Wherein the visual fatigue measuring method comprises the steps of:

delete

The method according to claim 1,
The step of adjusting the depth of the 3D content
Performing either disparity scaling or disparity shifting on the 3D content;
Wherein the visual fatigue measuring method comprises the steps of:

delete

A visual fatigue measurement system based on face recognition recognition based stereoscopic 3D contents for emotional TV,
A face image obtaining unit that obtains a face image of the viewer when the 3D content is provided to a viewer;
A face detection / normalization unit for performing face detection and normalization on the face image;
Extracting a face landmark from the normalized face image based on at least one of face geometry and appearance and extracting a face landmark extracted from the extracted face landmark and a face image A facial feature extracting unit for extracting a facial feature by calculating a displacement between reference face landmarks; And
A discomfort judgment unit for determining whether the viewer feels uncomfortable by applying a machine learning algorithm that inputs the displacement to the facial feature,
Lt; / RTI >
The discomfort judgment unit
When the result of applying the machine learning algorithm is equal to or greater than a preset reference value, it is determined that the viewer's discomfort is present,
A degree of discomfort of the viewer according to a result value to which the machine learning algorithm is applied, and a degree of the discomfort is a parameter that is a numerical value of a measure in which the viewer feels discomfort when watching the 3D content, And adjusting a depth sense of the 3D content according to a degree of adjusting a depth sense of the 3D content by determining a degree of adjusting a depth sense of the 3D content provided to the viewer,
Collecting a feature of the 3D content, generating a discomfort pattern of the viewer by using a feature of the 3D content when the discomfort of the viewer is judged to exist, The 3D content of at least one of the 3D contents of the 3D content is recommended to the viewer.

delete

10. The method of claim 9,
The face detection / normalization unit
A visual fatigue measurement system for detecting a face region from a face image using an object detection algorithm, performing scale on the detected face region, or performing normalization by performing rotation compensation.

delete