CN113570211A

CN113570211A - Method for quantitatively evaluating food consumer acceptance by applying facial expression emotion recognition and electroencephalogram analysis

Info

Publication number: CN113570211A
Application number: CN202110788842.8A
Authority: CN
Inventors: 冯婧; 皇甫洁; 王成; 吕高冲; 董建辉; 王德良; 韩兴林
Original assignee: China National Research Institute of Food and Fermentation Industries
Current assignee: China National Research Institute of Food and Fermentation Industries
Priority date: 2021-07-13
Filing date: 2021-07-13
Publication date: 2021-10-29

Abstract

The patent develops a consumer acceptance evaluation method comprehensively applying facial expressions, electroencephalogram analysis and questionnaires aiming at the evaluation of the acceptance of consumers to foods. Currently, direct and indirect consumer evaluation methods are rarely combined and applied to food evaluation. The indirect evaluation method takes facial expression emotion analysis and electroencephalogram analysis as examples, captures subconscious emotional reactions of the consumers in the evaluation process, and can be combined with direct questionnaires to more comprehensively and accurately know the product acceptance of the consumers. The product evaluation process is divided into four stages of appearance, aroma, taste, mouthfeel and aftertaste, and the multivariate factor analysis and the multivariate linear regression model are introduced into an evaluation system, so that the evaluation method of the consumer acceptance by applying facial expression, electroencephalogram analysis and questionnaires is optimized. The evaluation method provided by the invention is suitable for various foods, such as beverages, wines, fermented foods, seasonings and the like.

Description

Method for quantitatively evaluating food consumer acceptance by applying facial expression emotion recognition and electroencephalogram analysis

Technical Field

Aiming at the evaluation of the acceptance of consumers to food, the patent develops a consumer acceptance evaluation method applying facial expression, electroencephalogram analysis and sensory evaluation.

Background

Consumer research plays an increasingly important role in the development and promotion of food products. Consumer choice is influenced by factors such as product quality, brand, price, purchasing environment, etc. Food shopping driving factor research finds that the influence of brand effect on current consumers is gradually reduced, and meanwhile, a good product with high cost performance and suitability is more favored by the consumers. This reflects the consumer concept starting to return product quality and the actual value of the product. The current consumer market is constantly changing and it is becoming increasingly important for food research to accurately capture consumer needs and thoughts.

Consumer research methods can be divided into direct and indirect. Direct research methods such as interviews, questionnaires, etc. have the advantages of ease of operation, low cost, etc., but the results of direct research methods may be affected by the cognitive bias of the consumer, meaning that these methods only measure the subjective perception and emotional response of the consumer, but not the emotional response itself (whether intentional or unintentional, or some consumers may not be perfectly honest when filling in the questionnaire). Direct research methods are based on subjective self-perception and lack the ability to capture subconscious emotions. The indirect research method is not influenced by the cognitive prejudice of the consumers, and the subconscious autonomous emotional response of the consumers can be evaluated to obtain more objective emotional response. Currently, methods such as facial expressions, electroencephalogram, electrodermal, heart rate, etc. can measure typical physiological or physical responses that represent different emotions.

Facial expression communication enables more efficient transfer of emotional information than text language. Previous studies have shown that speech accounts for only 7% of the total information in the communication, the voice (intonation) 38%, and facial expressions 55% of the information. Thus, "reading" facial expressions is a useful tool to assess emotional response. An important advantage of the facial expression analysis method is that it can capture subconscious emotions. Facial expression analysis can measure and analyze emotions from dimensions such as valence, arousal degree and the like of emotional response.

Research of neuroscience, psychology and cognitive science shows that electroencephalogram data can reflect human psychology activities and cognitive behaviors, and are more and more widely applied to the emotional process of checking cognition of consumers or responding to prefabricated marketing stimulation. The delta frequency band (1-4 Hz), the theta frequency band (4-8 Hz), the alpha frequency band (8-13 Hz), the beta frequency band (13-30 Hz) and other frequency bands of the brain electricity have close relation with various physiological and psychological activities of people, and can reflect the emotional changes of consumers. When extracting the electroencephalogram frequency domain characteristics, many scholars map electroencephalogram signals to the frequency bands and then extract the frequency domain characteristics corresponding to the frequency bands respectively, so that the exciting value, the relaxation value, the emotional valence and the like of a consumer in the food evaluation process are obtained.

The method combines facial expression emotion recognition Analysis, electroencephalogram Analysis and consumer questionnaires, and analyzes and predicts the evaluation and acceptance of consumers on food by means of multi-Factor Analysis (MFA) and a method for establishing a multi-element linear regression model.

Disclosure of Invention

The invention aims to improve the evaluation method of the food consumer acceptance by combining the facial expression recognition and emotion analysis technology, the electroencephalogram analysis technology and the consumer acceptance evaluation questionnaire.

The invention is mainly realized by the following research methods and data analysis means: facial expression recognition, electroencephalogram analysis, a nine-point preference scale questionnaire, variance analysis, a multivariate factor analysis Model (MFA) and a multivariate linear regression model;

1. the experimental participants: consumers who have not undergone professional sensory evaluation, average age 18-50 years, no smoking, no disease history, are regular consumers who evaluate the product (consumption frequency greater than or equal to once a month), and have no product dependence or allergic behavior. After the experiment participants know the experiment contents and possible risks and sign the informed consent, the prescription can participate in the experiment;

2. the experimental method comprises the following steps: and (4) prompting the consumers to comprise the contents of a test process, timing and the like through a test standard program edited by MATLAB. In the experimental process, the high-definition camera captures facial expression information of a consumer in the whole evaluation process, and emotion recognition and analysis are carried out through FaceReader software. Meanwhile, a consumer captures the electroencephalogram signals in the evaluation process by wearing the electroencephalogram cap, and captures and analyzes the electroencephalogram signals through Emotiv software and hardware. The consumer acceptance questionnaire adopts nine-point preference marks;

3. selecting and preparing experimental samples: blind test of blind sample is adopted in the experiment, a group of 5-6 samples are selected and respectively put into transparent containers marked with 3-bit random number codes. The sample evaluation adopts disorder to eliminate the interference of the evaluation sequence on the sample evaluation;

4. sensory evaluation and facial expression testing process: and (4) carrying out experimental operation by the consumer according to the screen prompt, and evaluating the samples one by one. The rating for each sample followed: appearance, aroma, taste and mouthfeel, aftertaste (see figure 1); a consumer evaluates samples one by one, each sample firstly observes appearance, smells smell and tastes taste of the samples, then swallows or spits the samples to feel aftertaste, meanwhile, a camera records facial expression emotion data of the consumer in a sensory evaluation process, and electroencephalogram equipment captures and records electroencephalogram data. The consumer's acceptance of the samples was scored (9 points) and from 1 to 9 represented 1-extreme dislike, 2-very dislike, 3 general dislike, 4-slight dislike, 5-neither like nor dislike, 6-slight like, 7-general like, 8-very like, 9-extreme like, in that order. The specific evaluation steps and screen prompts are as follows:

1) appearance (30 seconds): holding the sample, holding the cup for light, and observing the color tone, transparency and the like of the sample in the cup for 30 seconds;

2) fragrance (30 seconds): holding the sample under the nose, rotating slightly, smelling the fragrance for 30 seconds, and sensing the fragrance of the sample;

3) taste and mouthfeel (5 seconds): the sample was held up, tasted a bite (not swallowed), and the texture and taste of the sample were experienced for 5 seconds;

4) aftertaste (20 seconds): swallowing the sample, and then, within the next 20 seconds, asking for a quiet taste for the overall feeling brought by the sample;

5) the taste, texture, aftertaste, overall perception and acceptability of the sample was scored by questionnaire;

5. data analysis

1) Facial expression data processing: firstly, classifying facial expression emotion data according to emotion (7 basic emotions: neutrality, joy, sadness, anger, surprise, fear, disgust, comprehensive dimensionality emotion valence and emotion arousal degree), respectively calculating average values and standard deviations, and carrying out 2-factor (sample and evaluation stage) variance analysis to find out emotion capable of remarkably distinguishing different samples and different evaluation stages;

2) and (3) processing acceptance data: calculating the acceptance degree average score of each product to obtain the product most popular with consumers, and making bar graphs for acceptance degree score comparison;

3) brain electric data processing: after the electroencephalogram data are processed by extraction, artifact removal and the like, 4 emotion dimensions of an excitation value, an interest value, a pressure value and a relaxation value are obtained through analysis. And respectively calculating the emotion mean values of different samples and different evaluation stages. Establishing a 2-factor (sample and evaluation stage) variance analysis model, and analyzing the electroencephalogram emotion capable of remarkably distinguishing different products and different evaluation stages;

4) MFA analysis of consumer facial expression emotion, product sensory and consumer acceptability: facial expression emotion, electroencephalogram emotion and product sensory characteristics are divided into one group, and consumer acceptance is used as a supplement group to perform multivariate factor analysis so as to explore the correlation among several data;

5) multivariate linear regression model: the method comprises the steps of establishing a multiple linear regression model by taking electroencephalogram emotion and facial expression emotion which can obviously distinguish different samples and different evaluation stages as independent variables and taking consumer acceptance score as dependent variables, and obtaining a correlation model of consumer acceptance, electroencephalogram emotion and facial expression emotion.

Description of the drawings:

FIG. 1: schematic diagram of experimental design

FIG. 2 is a drawing: data processing schematic

FIG. 3: example 1 sensory evaluation data for evaluation of samples of white spirit

FIG. 4 is a drawing: example 1 consumer acceptance data for liquor sample evaluation

FIG. 5 a: example 1-evaluation of samples of white spirit facial expressions neutral, happy, sad, aversive

FIG. 5 b: example 1-evaluation of samples of white spirit with facial expressions of anger, surprise, fear

FIG. 6: example 2 sensory evaluation data for evaluation of wine samples

FIG. 7: example 2 consumer preference data for wine sample ratings

FIG. 8 a: example 2-Mild facial expressions, sadness, anger, disgust of the wine sample rating

FIG. 8 b: example 2 facial expressions evaluated on a wine sample pleasure, surprise, fear

FIG. 9: example 2-liqueur sample consumer correlation of facial expressions, product sensory characteristics and receptivity MFA analysis

FIG. 10: example 2 liqueur sample consumer product and evaluation stage correlation MFA analysis

Detailed Description

The following detailed description of the present invention is provided to facilitate a full understanding of the present invention, and it will be readily apparent to those skilled in the art that the present invention may be embodied without departing from the spirit or scope of the invention.

Example 1: sensory evaluation and consumer evaluation of liquor product by applying facial expression emotion analysis and electroencephalogram analysis

The experiment evaluates 4 types of white spirit products, 10 volunteers are recruited, wherein 3 women and 7 men are in the experiment, and the age of consumers is between 27 and 46 years (the average age is 33.6 years);

1. the experimental process comprises the following steps: each testee is provided with a high-definition camera for recording facial expression data; the testee wears an Emotiv commercial 14 electroencephalogram conducting cap to capture data of transmitted electroencephalogram; the receptivity questionnaire takes the form of an online questionnaire. Each testee is provided with a computer for prompting and answering questionnaires in the experimental steps; blind sample test (code: sample 1, sample 2, sample 3, sample 4) was performed, and the participants rated the wine samples one by one according to 4 steps of looking at the wine body, smelling the fragrance, tasting and swallowing. Recording facial expression information and electroencephalogram information of a consumer in the experimental process; there was a 3 minute (or more) rest time between each sample evaluation during which the participants were asked to drink water, eat a salt-free soda cracker to clear their mouths, and fill out an acceptance questionnaire;

2. data processing and analysis

1) Sensory evaluation data: the consumer evaluates the sensory characteristics of the 4 samples from aspects of appearance, aroma, taste and the like (see figure 3 in detail); in addition, consumer acceptance of the product was evaluated (fig. 4); it can be seen that sample 1 has high fragrance concentration, high pleasure degree, high refreshing degree, highest overall preference degree, and high fragrance and taste preference degrees. The sample is totally sweet, the taste is coordinated and the aftertaste is slightly different;

2) facial expression data: the scores of different emotion dimensions of consumers in the process of evaluating the white spirit samples (fig. 5a and 5 b) can be seen, the difference of different emotion intensities is obvious, wherein the neutral numerical value is obviously higher than other emotions, and the maximum value is obtained in each evaluation stage of each sample. In comparison with other emotions, four emotion values of sadness, joy, disgust and anger are relatively high, while surprise and fear emotion data are low;

3) electroencephalogram data: through the electroencephalogram processing of capturing different frequency bands and preprocessing, emotion values of excitation, interest, stress and relaxation of 4 dimensions in the evaluation process are obtained (table 1). In general, relaxation and excitation values are generally higher than interest and stress values. It can be seen that, in terms of the excitation values, sample 1 is the highest at each evaluation stage; in terms of value of interest, sample 3 was higher during the stages of smelling, tasting and swallowing;

TABLE 1 mean values of electroencephalogram emotion in consumer evaluation process

Sample (I)	Evaluation stage	Excitement	Interests in	Pressure of	Relax the body
						Sample
1	Color observation	0.70	0.39	0.30	0.58
						Sample 2	Color observation	0.44	0.43	0.33	0.59
Sample 3	Color observation	0.47	0.39	0.27	0.54
						Sample No. 4	Color observation	0.41	0.56	0.31	0.55
Sample 1	Smelling fragrance	0.52	0.47	0.37	0.58
						Sample 2	Smelling fragrance	0.45	0.52	0.36	0.62
Sample 3	Smelling fragrance	0.43	0.59	0.49	0.63
						Sample No. 4	Smelling fragrance	0.38	0.49	0.41	0.67
Sample 1	Tasting at the mouth	0.63	0.44	0.50	0.67
						Sample 2	Tasting at the mouth	0.45	0.40	0.35	0.56
Sample 3	Tasting at the mouth	0.45	0.58	0.41	0.63
						Sample No. 4	Tasting at the mouth	0.48	0.30	0.47	0.62
Sample 1	After swallowing	0.63	0.48	0.49	0.67
						Sample 2	After swallowing	0.48	0.56	0.38	0.57
Sample 3	After swallowing	0.43	0.69	0.44	0.63
						Sample No. 4	After swallowing	0.44	0.51	0.50	0.58

Example 2: facial expression emotion analysis liqueur product sensory evaluation and consumer evaluation

The experiment evaluates 3 types of liqueur products, and 10 volunteers are recruited, wherein 4 women and 6 men are selected, and the age of consumers is between 22 and 26 years (the average age is 24.3 years);

1. the experimental process comprises the following steps: the experiments were performed in a standard sensory laboratory. Each testee is placed with a high-definition camera in front of the testee for recording facial expression data. Adopting an online questionnaire form, and equipping each testee with a computer for prompting and answering questionnaires in the experimental steps; blind tests (sample codes 192, 738, 645) were used for the experiments. The participants evaluate 3 wine samples one by one according to 4 steps of color observation, fragrance smelling, tasting and aftertaste feeling. There was a 3 minute rest time between each sample evaluation and participants were asked to drink water and eat salt-free soda biscuits for a fresh taste. Filling in a sense and acceptance questionnaire after evaluation;

2. data processing and analysis

1) Sensory evaluation data:

the consumer evaluated the sensory characteristics of the 3-style liqueur sample in terms of appearance, aroma, flavor, etc. (see fig. 6 for details). FIG. 7 illustrates consumer acceptance of samples;

2) facial expression data: recording facial data (5 Hz) of the evaluation process of the consumer by using a FaceReader and analyzing the facial data to obtain 7 emotions (data range 0 to 1) of neutrality, pleasure, sadness, anger, surprise, fear and disgust, and analyzing and calculating the data (data range-1 to 1) of the positive direction and the negative direction of the emotion and the data (data range 0 to 1) of the arousal degree; fig. 8a and 8b show the scores of different emotional dimensions of the consumer during the scoring of the wine sample. It can be seen that the different emotions differed significantly, with the neutral value being significantly higher than the other emotions, with a maximum value at each evaluation stage for each sample. In comparison with other emotions, four emotion values of sadness, disgust, anger and joy are relatively high, while two emotion data of surprise and anger are low; and (3) establishing a variance model by taking the product, the evaluation stage and the mutual influence thereof as independent variables and facial expression emotion as dependent variables, and analyzing the difference between emotions of different samples of the liqueur and different tasting stages of the liqueur. Of the 7 basic emotions, the anger emotion can significantly distinguish different samples (P <0.05), i.e. the consumer feels a significantly different value of emotion when evaluating different samples; in different evaluation stages, 3 expressions which are neutral, happy and surprised are significantly different (P <0.05), and 3 emotions which are neutral, happy and surprised are obviously different for consumers in different evaluation stages; no obvious mutual influence exists between the sample and the evaluation stage (P is more than or equal to 0.05);

3) consumer facial expression, product sensory and consumer acceptance Multivariate Factor Analysis (MFA): MFA analysis models were established with sensory characteristics, emotional dimensions, and consumer acceptance, and their correlations were studied. Fig. 9 and 10 show the similarity and difference in the facial expression mood and sensory characteristics at different evaluation stages between the liqueur samples. It can be seen that the first quadrant is able to distinguish between different products; the second quadrant distinguishes different evaluation stages, and the numerical value of the color observation (viewing) stage is highest in the second quadrant, and the numerical value of the color observation (viewing) stage is the lowest in swallowing (after) and evaluation (tasting) and smelling (smelling) in the different evaluation stages of each sample; the sweetness, harmonious and soft mouthfeel and fruity flavor of the product are closely related to the acceptance of consumers. The neutral, fear and surprise emotions of consumers are closely related to the color, drug aroma and personality of the product, and the happy, disgusting and sad emotions are closely related to the harmony and pleasure of the aroma of the product.

Claims

1. A method for quantitatively evaluating food consumer acceptance by applying facial expression emotion recognition and electroencephalogram analysis comprises the following steps:

(1) evaluating the recruitment of the tested personnel and the design of test experiments;

(2) the evaluation method comprises the following steps:

the experiment adopts facial expression recognition analysis, electroencephalogram recording analysis and a method of combining with the evaluation of a consumer preference questionnaire to analyze and evaluate the food acceptance of consumers; prompting the consumers to include the contents of a test process, timing and the like through a test standard program edited by MATLAB;

(3) the evaluation process comprises the following steps: and (4) carrying out experimental operation by the consumer according to the screen prompt, and evaluating the samples one by one.

2. The rating for each sample followed: appearance, aroma, taste and mouthfeel and aftertaste;

the method comprises the following specific steps:

appearance: holding the sample, and observing the color tone, transparency and the like of the sample in the cup by light for 30 seconds;

fragrance: holding the sample under the nose, rotating slightly, and continuously smelling fragrance for 30 seconds to feel the fragrance of the sample;

taste and texture: the sample was held up, tasted a bite (not swallowed), and the texture and taste of the sample were experienced for 5 seconds;

aftertaste: swallow the sample, please taste the overall sensation that the sample brought in the next 20 seconds;

fifthly, scoring the taste, texture, aftertaste, overall feeling and acceptance of the sample;

(4) data processing and analytical modeling:

processing facial expression data: firstly, classifying facial expression emotion data according to emotion (7 basic emotion dimensions: neutrality, joy, sadness, anger, surprise, fear and disgust), calculating average values and standard deviations respectively, performing 2-factor (sample and evaluation stage) variance analysis, and analyzing facial expression emotion capable of remarkably distinguishing different samples and different evaluation stages;

receiving data processing: calculating the acceptance degree average of each product to obtain the product most popular with consumers, and making bar graphs for comparing the acceptance degrees of the consumers of the products;

processing the electroencephalogram data: after the electroencephalogram data are subjected to extraction, artifact removal and other processing, 4 emotion dimensions of an excitation value, an interest value, a pressure value and a relaxation value are obtained through analysis; respectively calculating the emotion mean values of different samples and different evaluation stages; establishing a 2-factor (sample and evaluation stage) variance analysis model, and analyzing the electroencephalogram emotion capable of remarkably distinguishing different products and different evaluation stages;

fourthly, analyzing the facial expression emotion, product sense and consumer acceptance multi-Factor (MFA): facial expression emotion, electroencephalogram emotion and product sensory characteristics are divided into a group, consumer acceptance is used as a supplement group to carry out MFA analysis, and correlation among several kinds of data is analyzed;

a multiple linear regression model: establishing a multivariate linear regression model by taking electroencephalogram emotion and facial expression emotion which can obviously distinguish different samples and different evaluation stages as independent variables and taking the consumer acceptance score as a dependent variable to obtain a correlation model of the consumer acceptance, the electroencephalogram emotion and the facial expression emotion;

use of the method of claim 1 for assessing food consumer acceptance using facial expression emotion recognition, electroencephalographic analysis, and assessing quality differences between different products; the evaluation method is suitable for various foods, such as beverages, wines, dairy products and the like.

3. The evaluation subject recruitment according to the step (1) of claim 1 is characterized in that: the evaluation subject is composed of consumers without professional sensory evaluation, is 18-50 years old, has no sense of taste, smell and vision disorder, is healthy during the experiment, has no smoking, and has no product dependence or allergic behavior for evaluating regular consumers of the product (consumption frequency is more than or equal to once a month).

4. The experimental method of step (2) as set forth in claim 1, wherein: establishing an evaluation method by comprehensively using methods of facial expression recognition analysis, electroencephalogram analysis and consumer acceptance questionnaires; facial expression (faceReader8.1, Noldus) analysis included 7 basic emotions (neutral, happy, sad, angry, surprised, fear, and disgust), and two composite dimensions of emotional valence, emotional arousal; electroencephalogram analysis (Emotiv Epoc +) finds out that the related emotions comprise an excitation value, an interest value, a pressure value and a relaxation value; the consumer questionnaire uses a 9-point preference scale, with 1 to 9 being in order: 1-extreme aversion, 2-very aversion, 3 general aversion, 4-slight aversion, 5-neither likes nor aversion, 6-slight likes, 7-general likes, 8-very likes, 9-extreme likes.

5. The experimental design of step (3) as set forth in claim 1, wherein: the experimental design follows three parts of facial expression emotion recognition, electroencephalogram capture analysis and consumer acceptance questionnaire evaluation, facial expression emotion data and electroencephalogram data of a consumer are recorded in the evaluation process of the consumer, and the acceptance of a sample is scored after evaluation.

6. The MATLAB editing experiment prompting program is adopted, and the grade of each sample is as follows: four evaluation stages of appearance (30 seconds), aroma (30 seconds), taste and mouthfeel (5 seconds) and aftertaste (20 seconds) were performed.

7. In the middle of each sample evaluation, the testee needs to drink water and eat salt-free soda biscuits for mouth cleaning.

8. The step (4) of data analysis as set forth in claim 1, wherein: respectively establishing 2-factor (sample and evaluation stage) variance analysis models according to neutral facial expressions, joys, anger, sadness, surprise, disgust, fear, valence and arousal degree, and finding out emotions which can obviously influence and distinguish different samples and different evaluation stages; for electroencephalogram data, after pretreatment such as artifact removal, calculation mean values of exciting, interesting, relaxing and pressure values are obtained, variance analysis of 2 factors (samples and evaluation stages) is carried out respectively, and emotion which can obviously influence and distinguish different samples and different evaluation stages is found out.

9. Establishing an MFA model: the method comprises the following steps of establishing an MFA model by taking facial expression emotion, electroencephalogram emotion and product sensory characteristic scores as a group and taking product acceptance as a supplement group, and finding out the correlation between the facial expression emotion and electroencephalogram analysis emotion and consumer acceptance; and establishing a multivariate linear regression model by taking facial expression emotion and electroencephalogram emotion which can obviously distinguish different samples and different evaluation stages as independent variables and taking consumer acceptance scores as dependent variables.