KR20130117624A - Method and apparatus for detecting talking segments in a video sequence using visual cues - Google Patents

Abstract

PURPOSE: A method and an apparatus thereof capable of detecting a talking segment in a video sequence by using a visual cue are provided to accurately classify a human emotion by dividing a movement of lips into talking and non-talking states in a temporal way. CONSTITUTION: A histogram of structure descriptive features of a face is obtained from a visual cue about a frame (205). An integrated gradient histogram (IGH) is drawn out from the structure descriptive features about the frame (206). The entropy of the IGH is calculated about the frame in the visual cue (207). The IGH is divided in order to detect a talking segment of the face in the visual cue (208). The talking segment is analyzed about the frame in the visual cue in order to reason an emotion (210,211). [Reference numerals] (201) Achieve a video frame; (202) Detect the face, the pupils, and the nose; (203) Normalize the face by using the pupils; (204) Estimate the mouth by using the nose; (205) Obtain an LBP histogram; (206) Encode IGH; (207) Obtain an IGH entropy; (208) Execute online time division of the IGH entropy; (209) Is a character speaking?; (210) Analyze a behavior unit of the upper and lower face; (211) Analyze a behavior unit of the upper face; (212) Infer feeling; (AA) No; (BB) Yes

Description

METHOD AND APPARATUS FOR DETECTING TALKING SEGMENTS IN A VIDEO SEQUENCE USING VISUAL CUES}

The description below relates to image processing, computer vision and machine learning, and more particularly to emotion recognition in video sequences.

With recent advances in technology, considerable interest continues to enhance human computer interaction (HCI). In particular, engineers and scientists are attempting to exploit basic human attributes such as voice, gaze, gesture and emotional state to improve HCI. The ability of a device to detect human emotions and respond to human emotions is known as 'affective computing'.

Automatic facial expression recognition is an important element in HCI's research field. Automatic facial expression recognition also plays an important role in human behavior modeling, which is likely to develop in applications such as video conferencing, video games, video surveillance, and the like. In automatic facial expression recognition, most research aims to identify six basic emotions (sadness, fear, anger, happiness, disgust, surprise) on facial expression datasets that are posing in a controlled environment. Engineers and scientists have used static as well as dynamic methods to infer different emotions from facial expression datasets. Dynamic methods consider contiguous groups of frames to infer specific emotions, whereas static methods are methods that independently analyze frames in a video sequence.

The mouth area of the face contains highly identifiable information about human emotions and plays an important role in facial expression recognition. However, in a typical situation such as video conferencing, there may be important temporal segments associated with a person's talking, and a facial expression recognition system that uses the mouth area to infer a person's feelings may be around the mouth area. The arbitrary and complex shape of can potentially misinterpret the mouth area. Time segment information about talking segments in the video sequence is very important because it can be used to improve the emotion recognition system.

In recognition of a person's emotions, not many methods that have dealt with the condition of a 'talking face' in accordance with Action Units (AU) inferred for the mouth area can potentially misrecognize a person's feelings. can do. Currently, known methods aim to determine who is speaking in a multi-person environment, but the movement of the lips in a time of talking and non-talking (a variety of emotional segments as well as neutral states). It is not intended to divide into a state). As a result, systems currently associated with emotion recognition do not capture accurate emotions.

For the reasons mentioned above, there is a need for a method for accurately classifying a person's emotions by dividing the movement of the lips into talking and non-talking states in time.

According to an embodiment, a method of detecting and classifying a talking segment of a face in a visual cue may include tracking and normalizing a face region for each frame of the visual cue, and a structural description feature of the facial region for each frame in the visual cue. obtaining a histogram of the structure descriptive features).

According to an embodiment, a method of detecting and classifying a talking segment of a face in a visual cue may derive an Integrated Gradient Histogram (IGH) from a structural descriptive feature for the frame in the visual cue. Compute the entropy of IGH for the frame in the visual cue. The method of detecting and classifying talking segments of a face in the visual cue may perform segmentation of an IGH to detect a talking segment for the face region in the visual cue, and infer the frame from the visual cue to infer emotions. Analyze the talking segment for.

A computer program product for detecting and classifying talking segments of a face in a visual cue according to one embodiment includes an integrated circuit.

According to an embodiment, an integrated circuit of the computer program product may include at least one processor, at least one memory having computer program code in the integrated circuit, at least one memory, and at least one processor causing the computer program product to display the visual. Computer program code configured to track and normalize the face area for each frame of a queue.

According to an embodiment, the computer program product obtains a histogram of a structural depiction feature for the frame in the visual queue, derives an IGH from the structural depiction feature for the frame in the visual queue, and in the visual queue. Computing the entropy of IGH for the frame, the computer program product also performs segmentation of IGH to detect talking segments for the face in the visual cue, and frames in the visual cue to infer emotions. Analyze the talking segment for.

1 is a flowchart illustrating an example of a method of recognizing a character's emotion in a video sequence, according to an exemplary embodiment.
2 is a detailed flowchart illustrating an example of a method of detecting a talking segment in a video sequence using a visual cue according to an embodiment.
3 illustrates a computing environment for executing an application according to an embodiment.

It is a primary object of the following embodiments to provide a system and method for detecting talking segments in a visual cue. Another object of the embodiments is also to provide unsupervised temporal segmentation to detect the talking phase.

The embodiments described herein can be better appreciated and understood by the accompanying drawings and the detailed description below. However, preferred embodiments and details of the invention described in the following description are described for the examples, and the content of the invention is not limited by the contents described. Many changes and modifications of the embodiments can be made within the scope of the spirit of the invention, and the embodiments described below may include such changes and modifications.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Reference numerals shown in each drawing denote the same members. The embodiments described herein may be better understood by the accompanying drawings and the detailed description.

Various features of the embodiments and details thereof will be described in detail in the following detailed description, which can be better explained by the embodiments shown in the accompanying drawings, but the embodiments are not limited to the described contents. In order not to unnecessarily obscure the description of the embodiments, descriptions of well-known configurations or processing techniques have been omitted. The embodiments in the following description have been described only for the purpose of easily understanding the method of implementing the embodiments and for those skilled in the art to easily implement the embodiments. Accordingly, the following description of the embodiments should not be construed as limiting the scope of the embodiments.

Embodiments in the description below relate to a system and method for detecting talking and non-talking segments in a sequence of image frames using a visual cue.

The method of detecting talking segments in a visual cue can be caused by an audio cue from someone other than the speaker to be detected, and the audio cue can be detected incorrectly because the talking and non-talking segments can be detected incorrectly. Use visual cues instead. In addition, a method of detecting talking segments in a visual cue aims to classify non-talking segments and talking segments that may have other expressions, including audio such as laughter, interjection, and the like. For the same reason as above, a visual cue, not an audio cue, should be used to distinguish between talking and non-talking segments.

The method of detecting talking segments in the visual cue may identify the temporal segments of the talking face in the video sequence by estimating uncertainties associated with the depiction of mouth or lip movements. According to one embodiment, after the step of tracking the mouth area, the movement of the mouth is encoded into an Integrated Gradient Histogram (IGH) of Local Binary Pattern (LBP) values. Uncertainty in mouth movement is quantified by estimating the entropy of IGH. As the online K-Means algorithm is used to distinguish the pattern of the mouth speaking among other movements of the mouth, time series data of entropy values in each frame is further clustered.

Here, the visual cue may be a video including a photo or a sequence of frames.

Referring to the drawings, preferred embodiments are shown in the drawings, wherein reference numerals given in the drawings represent the same members.

1 is a flowchart illustrating an example of a method of recognizing a character's emotion in a video sequence, according to an exemplary embodiment.

In the method of recognizing the emotion of the character, in step 101, a video frame is acquired in the video, and in step 102, the face is detected by fixing the pupil position of the character. Here, the character may be an object to infer emotions from the video frame. In step 103, the method of recognizing the emotion of the character checks whether the character is speaking. In step 104, the method of recognizing the emotion of the character acquires a feature for the entire face if it is checked that the character is not speaking. In step 105, the method of recognizing the emotion of the character predicts action units (AUs). Behavioral units are defined in the Facial Action Coding System (FACS) and represent the movement of muscles that cause changes in facial appearance. In step 106, the method of recognizing the emotion of the character infers the emotion of the character based on the action unit. In step 107, the method of recognizing the emotion of the character according to an embodiment acquires the feature only for the top of the face when it is identified that the character is speaking. The method of recognizing a character's emotions includes, at step 108, predicting a behavioral unit, and at step 109, inferring the character's emotions.

According to an embodiment, the talking face refers to a face that speaks in a state with no emotion or no emotion. A non-talking face is a face that doesn't speak but reveals emotions. The steps of the method 100 of recognizing the emotions of a character may be performed in the order presented or in another order, or simultaneously. In addition, in some embodiments, some steps shown in FIG. 1 may be omitted.

2 is a detailed flowchart illustrating an example of a method of detecting a talking segment in a video sequence using a visual cue according to an embodiment.

As shown in FIG. 2, the method for detecting the talking segment may use an algorithm to perform the corresponding step.

The algorithm obtains a sequence of video frames in step 201, further detects the first face in step 202 and tracks the positions of the pupil and nose. According to one embodiment, a method based on a version of a standard face detector and an Active Appearance Model (AAM) may be used to identify the positions of the face, eyes, and nose in every frame of the video. Can be. The active appearance model is a generalized method for the widely used Active Shape Model (ASM) approach, but uses all the information of the image area occupied by the target object rather than just information near the modeled contour. In step 203, the method of detecting the talking segment normalizes the face using the pupil. The position of the pupil is used to normalize all face images to MxN size. In step 204, the method of detecting the talking segment tracks the position of the nose to expose the mouth area in each frame in further processing.

According to one embodiment, the distance between the pupils is maintained at 48 pixels to normalize the face and crop the mouth area to 56x46 pixel size.

Since the truncated sequence of mouth region images may have an alignment variable across the illumination parameter and frames, the method of detecting the talking segment selects a feature descriptor that can handle these conditions. . According to one embodiment, the method of detecting the talking segment in step 205 derives a histogram of the Local Binary Pattern (LBP) values to encode the appearance of the mouth region. LBP is a powerful feature used for texture classification, which later proved to be very effective for face recognition and related applications. According to one embodiment, the LBP pattern is calculated for every pixel in the image of the cropped mouth region. In addition, uniform LBP patterns (at most, patterns with 2-bit shifting) are all similarly used and classified. A histogram of the estimated LBP values for the image of the truncated mouth region is used to depict the appearance of the mouth region in the relevant frame.

(타우)의 시간 주기에 걸쳐 입의 외형 변화를 캡쳐할 목적으로 프레임 i라 불리는 특정 프레임으로부터 그래디언트 히스토그램이 계산된다.Systems and methods for detecting talking segments distinguish between smooth appearance changes in mouth movement at the beginning and offset of emotions such as laughter, surprise, disgust, and the like, and complex appearance changes in the mouth when speaking. Also, for a neutral face that is not spoken, there will not be much change in the appearance of the mouth. According to one embodiment, to distinguish the complex appearance changes of the mouth, 2

The gradient histogram is computed from a particular frame called frame i for the purpose of capturing changes in the appearance of the mouth over the time period of (tau).

Gradient LBP histograms are calculated as follows.

는 i번째 프레임과 (i+n)번째 프레임의 히스토그램 간 차이를 이용하여 계산되는 그래디언트 히스토그램이고

는 i번째 프레임과 (i-n)번째 프레임의 히스토그램 간 차이를 이용하여 계산되는 그래디언트 히스토그램이다.

Is a gradient histogram computed using the difference between the histograms of the i th frame and (i + n) th frame.

Is a gradient histogram computed using the difference between the histogram of the i th frame and the (in) th frame.

+1)의 시간 세그먼트에 걸쳐 입의 외형 변화와 관련된 완전한 정보를 얻고, 다음과 같이 하나의 IGH로 인코딩한다.The gradient histogram encodes the change in appearance in the mouth pattern over the time dimension. At step 206, the method of detecting the talking segment is (2).

Obtain complete information related to the appearance change of the mouth over the time segment of +1) and encode it into one IGH as follows.

The series of talking frames will have a more evenly distributed IGH value compared to the frame representing the particular emotion. In other words, the uncertainty associated with the depiction of the integrated gradient histogram may be closer to the talking segments as compared to the emotion segments. In step 207, the method of detecting the talking segment derives the entropy of the IGH. In step 208, the method of detecting the talking segment performs online temporal segmentation on the entropy of the IGH and uses the entropy of the IGH to quantify the uncertainty in the video segment.

The entropy of the IGH of the i th frame is calculated as follows.

Ep _i is the entropy value of the IGH of the i th frame, and p _k is the histogram value for the k th bin.

In addition, the IGH is normalized before estimating the entropy value of the IGH of the i th frame. This is because of the need to compare entropy values over different time segments. The energy value of IGH over different time segments may vary depending on the result of the gradient process. The entropy value of IGH is normalized by adding a common energy value between the first LBP histograms, such as a separate bin in IGH. In static segments, common energy values are very large in IGH, and can result in very small entropy values. In the emotional segment, the common energy value may correspond to a slow talking process. However, the gradient energy part of the IGH is more widespread in the talking segment and therefore may have a higher entropy value compared to the emotional segment. Time series data of entropy values estimated from the IGH of every frame are used for unsupervised online segmentation of the talking face and the non-talking face.

According to one embodiment, the entropy values obtained for every frame in the video sequence form time series data. Next, the time series data is divided into an unsupervised online format to provide the input required for the emotion recognition system in relation to the presence of the talking face in the video sequence. According to one embodiment, the emotion recognition system uses an online K-average algorithm with k equal to two to partition the time series data. There are no additional assumptions regarding the initial value of the data or the range of data.

Problems that arise in emotional inference with occlusions around the mouth area have been raised to improve the accuracy of emotion detection. In step 209, the method of detecting the talking segment checks whether the character is speaking. According to one embodiment, the method of detecting the talking segment takes into account whether the mouth area is occluded whenever it is detected that the character is speaking. In step 210, the method of detecting the talking segment analyzes the action units at the top of the face and the bottom of the face when the character is not speaking. According to an embodiment, the method of detecting the talking segment may identify whether the character is speaking, or, in a simple method, may be a method of removing the visual cue without the mouth area in a specific time segment. According to one embodiment, in step 211, the method of detecting the talking segment only analyzes the behavioral unit from the top of the face. In step 212, the method of detecting the talking segment infers emotion based on a talking visual cue or a non-talking visual cue. In a normal situation, the method of analyzing the action unit from the top of the face would be inferior to that of using all the action units, but in the situation that the character is speaking, the method of using all the action units has a lot of misinterpretation. It can be better than how to use all the action units.

According to another embodiment, improving the performance of emotion recognition uses the mouth area for emotion recognition, but once it is detected that the character is speaking, it changes the manner of emotion recognition. Although the features of the image from the talking face cannot be easily interpreted, the mouth area still has clues about the current emotion. For example, a talking face in a happy state and a talking face in a sad state can be distinguished. The approach of inferring emotion at the talking face using the mouth area can be different from the usual emotion recognition system. Those skilled in the art will recognize that the movement of the mouth tail can help to distinguish certain emotions even when they are talking. The various steps in the method 200 of detecting a talking segment in a video sequence may be performed in the order presented or in another order, or simultaneously. In addition, in some embodiments, some steps shown in FIG. 2 may be omitted.

According to one embodiment, a method of detecting a talking segment may be used in a video conference, video meeting, or interview situation in which the camera focuses on a person, and detects a talking face and a non-talking face of the person associated with it. Determines people's feelings In addition, the method of detecting the talking segment may be performed in the emotion recognition system for better emotion classification.

3 illustrates a computing environment for executing an application according to an embodiment.

As shown in Figure 3, the computing environment includes a control unit, an Arithmetic Logic Unit (ALU), memory, storage, a plurality of networking devices, and a plurality of input / output (I / O). at least one processing unit having an input output device. The processing unit processes the instructions of the algorithm. The processing unit receives the instructions from the control unit to process the instructions. In addition, logical and arithmetic tasks related to the execution of instructions are handled with the aid of arithmetic logic units.

The overall computing environment may consist of a plurality of homogeneous and / or heterogeneous cores, a plurality of different types of CPUs, special media, and other accelerators. In addition, the plurality of processing units may be located on one chip or a plurality of chips.

Algorithms consisting of code and instructions for execution are stored in memory units, storage, or both. When the instructions are executed, the instructions may be loaded from the corresponding memory unit and / or storage and executed by the processing unit.

In the implementation of hardware, various networking devices or external input / output devices may be coupled to the computing environment through networking units and input / output devices to support hardware execution.

The embodiments described herein may be performed through at least one software program, the software program being operated on at least one hardware device and performing network management functions to control the components. The components shown in FIG. 3 include blocks that can be at least one of a hardware device or a combination of hardware device and software module.

The foregoing descriptions of the specific embodiments are sufficient to reveal the essence of the embodiments, so that anyone can readily modify various applications to the specific embodiments above without departing from the spirit of the invention by applying current knowledge. Can be modified. Such variations and modifications are to be understood within the meaning and equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology used herein is for the purpose of description and should not be regarded as limiting. Therefore, while the embodiments described herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments described herein may be modified and practiced without departing from the scope and spirit of the embodiments as described.

Claims (16)

In a method of detecting and classifying talking segments of a face in a visual cue,
Tracking and normalizing the area of the face for each frame of the visual cue;
Obtaining a histogram of the structure descriptive features of the face with respect to the frame in the visual cue;
Deriving an integrated gradient histogram (IGH) in the structural depiction feature for the frame of the visual cue;
Calculating an entropy of the integrated gradient histogram for the frame in the visual cue;
Dividing the integrated gradient histogram to detect the talking segment of the face in the visual cue; And
Analyzing the talking segment for the frame in the visual cue to infer emotions
Method for detecting and classifying the talking segment comprising a. The method of claim 1,
Tracking and normalizing the area of the face,
Using the position of the pupil to normalize the image of the face with respect to the frame of the visual cue
Method for detecting and classifying the talking segment comprising a. The method of claim 1,
Tracking and normalizing the area of the face,
Using the position of the nose to crop the mouth area in an accurate manner relative to the frame of the visual cue
Method for detecting and classifying the talking segment comprising a. The method of claim 1,
Deriving the integrated gradient histogram,
Obtaining uncertainty associated with the depiction of the integrated gradient histogram for the talking segment in comparison to non-talking segments
Method for detecting and classifying the talking segment comprising a. The method of claim 1,
The entropy of the integrated gradient histogram is
And calculate the amount of uncertainty associated with the talking segment of the visual cue. The method of claim 1,
Analyzing the talking segment,
Using an action unit on top of the face to infer feelings about the talking face
Method for detecting and classifying the talking segment comprising a. The method of claim 1,
Analyzing the talking segment,
Using the behavior unit of the entire face to infer emotions against the non talking face
Method for detecting and classifying the talking segment comprising a. The method of claim 1,
The visual cue,
A method of detecting and classifying talking segments that are at least one of an image, a frame, and a video. The system of claim 1, wherein the system detects and classifies talking segments of a face in a visual cue that performs at least one of the claimed steps. A computer program product for detecting and classifying talking segments of a face in a visual cue,
An integrated circuit further comprising at least one processor;
At least one memory having computer program code in the integrated circuit;
With at least one processor, the computer program product,
Track and normalize the area of the face for each frame of the visual cue,
Obtain a histogram of the structural depiction features of the face with respect to the frame in the visual cue,
Derive an integrated gradient histogram in the structural depiction feature for the frame of the visual cue,
Calculate an entropy of the integrated gradient histogram for the frame in the visual cue,
Dividing the integrated gradient histogram to detect talking segments of the face in the visual cue,
Analyze the talking segment for the frame in the visual cue to infer emotions
The computer program code configured and the at least one memory
Computer program product comprising a. The method of claim 10,
Tracking and normalizing the area of the face,
And use the position of the pupil to normalize the image of the face with respect to the frame of the visual cue. The method of claim 10,
Tracking the area of the face,
And use the position of the nose to crop the mouth area in an accurate manner relative to the frame of the visual cue. The method of claim 10,
Deriving the integrated gradient histogram,
Obtaining uncertainty associated with the depiction of the integrated gradient histogram for the talking segment as compared to the non-talking segment. The method of claim 10,
The entropy of the integrated gradient histogram is
And calculate to determine the amount of uncertainty associated with the talking segment of the visual cue. The method of claim 10,
Analyzing the talking segment,
A computer program product that uses an action unit on top of a face to infer emotions about a talking face. The method of claim 10,
Analyzing the talking segment,
A computer program product that uses an entire face action unit to infer emotions against a non-talking face.

Images