CN110148468B - Method and device for reconstructing dynamic face image - Google Patents

Method and device for reconstructing dynamic face image Download PDF

Info

Publication number
CN110148468B
CN110148468B CN201910382834.6A CN201910382834A CN110148468B CN 110148468 B CN110148468 B CN 110148468B CN 201910382834 A CN201910382834 A CN 201910382834A CN 110148468 B CN110148468 B CN 110148468B
Authority
CN
China
Prior art keywords
image
face
response data
dynamic
reconstruction model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
CN201910382834.6A
Other languages
Chinese (zh)
Other versions
CN110148468A (en
Inventor
张慧
王蕴红
魏子翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN201910382834.6A priority Critical patent/CN110148468B/en
Publication of CN110148468A publication Critical patent/CN110148468A/en
Application granted granted Critical
Publication of CN110148468B publication Critical patent/CN110148468B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/50—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for simulation or modelling of medical disorders

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Public Health (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Pathology (AREA)
  • General Engineering & Computer Science (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a method and a device for reconstructing a dynamic face image, which aim at the characteristic that the dynamic face image mainly presents high-level visual feature information and the characteristic that facial features with different attributes are processed by different high-level cognitive brain regions, adopt three kinds of high-level feature information with different attributes, utilize the three different high-level cognitive brain regions to acquire first class neural response data, second class neural response data and third class neural response data corresponding to the three kinds of high-level feature information of the face, simultaneously construct models of the different high-level cognitive brain regions and the dynamic face from a visual image space to a brain perception space and a multi-dimensional mapping relation among the models, acquire a face basic image, a face expression image and a face identity image to realize the reconstruction of multi-dimensional facial features, acquire the dynamic face image and reconstruct the dynamic face image perceived by some patients, leading us to deeper understanding and cognition of the cognitive disorder mechanism of mental diseases.

Description

Method and device for reconstructing dynamic face image
Technical Field
The present invention relates to image processing technologies, and in particular, to a method and an apparatus for reconstructing a dynamic face image.
Background
The visual object reproduced and sensed from the brain nerve signal is a leading technical field which is widely concerned at present, and it means that by acquiring the functional Magnetic Resonance signal (fMRI for short) of the human brain and by means of image processing and machine learning algorithm, the visual image seen by vision is restored, the human face is used as the most important visual perception object in understanding nature and social interaction, and some diseases with cognitive and mental disorders such as facial agnosia, autism, senile dementia and parkinson disease have defects when recognizing the high-level characteristic attribute of dynamic facial foramen, therefore, the image reconstruction of the human face imagined in the brain of the user to be tested needs to be performed by the human face reconstruction technology.
In the prior art, Principal Component Analysis (PCA) is used to establish a single linear mapping relationship between characteristic human faces and neural response signals, so as to reconstruct a human face image.
However, the prior art can only reconstruct a static face picture, and is difficult to meet the requirements for face multi-dimensional information reconstruction in the field of image reconstruction.
Disclosure of Invention
The embodiment of the invention provides a method and a device for reconstructing a dynamic face image, which realize dynamic face reconstruction, reconstruct expression characteristics and identity characteristics in the reconstructed dynamic face image, enrich reconstructed information and improve the accuracy of face reconstruction.
In a first aspect of the embodiments of the present invention, a method for reconstructing a dynamic face image is provided, including:
and extracting first-class nerve response data, and reconstructing a model according to the first-class nerve response data and a preset human face image to obtain a human face basic image.
And extracting second nerve response data, and acquiring a facial expression image according to the second nerve response data and a preset facial expression reconstruction model.
And extracting third-class nerve response data, and acquiring a face identity image according to the third-class nerve response data and a preset face identity reconstruction model.
And acquiring a dynamic face image according to the face basic image, the face expression image and the face identity image.
Optionally, in a possible implementation manner of the first aspect, the obtaining, according to the first type of neural response data and a preset human face image reconstruction model, a human face base image includes:
and acquiring a human face basic image according to the following formula I and the first type of nerve response data.
Figure GDA0003053384790000021
Wherein, XG_RECONIs the base image of the face of the person,
Figure GDA0003053384790000022
is an average image, Y, of dynamic human face basic image samples preset in a human face image reconstruction modeltestIs the first type of neural response data,
Figure GDA0003053384790000023
is the average data, s, of the first type of neural response data samples caused by the dynamic human face basic image samples preset in the human face image reconstruction modeltestIs in YtestProjected coordinates of (a), ttestIs XG_RECONProjected coordinates of (1), WtrainIs s in the face image reconstruction modeltest-ttestTransformation matrix, UtrainIs Y in the human face image reconstruction modeltestCharacteristic vector of (V)trainThe feature vector of the dynamic human face basic image sample is preset in the human face image reconstruction model.
Optionally, in a possible implementation manner of the first aspect, before the obtaining, by the first type of neural response data and a preset human face image reconstruction model, a human face base image, the method further includes:
the method comprises the steps of obtaining dynamic human face basic image training samples and first type neural response data training samples caused by the dynamic human face basic image samples.
And performing parameter learning by using the dynamic human face basic image sample as an output quantity and the first type of neural response data sample as an input quantity through a two-pair s-t transformation matrix, the characteristic vector of the first type of neural response data sample and the characteristic vector of the dynamic human face basic image training sample according to the following formula to obtain s in a human face image reconstruction modeltest-ttestA transformation matrix, a characteristic vector of a first class of neural response data sample in a face image reconstruction model, a characteristic vector of a face basic image in the face image reconstruction model,
Figure GDA0003053384790000024
wherein X is the dynamic face base image sample,
Figure GDA0003053384790000025
is an average image of X, Y is the first type of neural response data sample,
Figure GDA0003053384790000026
is the average data of Y, s is the projection coordinates of Y, t is the projection coordinates of X, W is the s-t transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X.
Reconstructing the model according to the human face imagetest-ttestAnd transforming a matrix, the characteristic vector of the first type of neural response data sample in the face image reconstruction model, and the characteristic vector of the face basic image in the face image reconstruction model to obtain the face image reconstruction model.
Optionally, in a possible implementation manner of the first aspect, the obtaining, according to the second type of neural response data and a preset facial expression reconstruction model, a facial expression image includes:
and acquiring a facial expression image according to the following formula III and the second type of nerve response data.
Figure GDA0003053384790000031
Wherein, XE_RECONIs the image of the facial expression of the person,
Figure GDA0003053384790000032
is an average image, Y, of dynamic facial expression image samples preset in the facial expression reconstruction modelE_testIs a second type of neural response data,
Figure GDA0003053384790000033
is the average data, s, of second type neural response data samples caused by preset dynamic facial expression image samples in the facial expression reconstruction modelE_testIs YE_testProjected coordinates of (a), tE_testIs XE_RECONProjected coordinates of (1), WE_trainIs s in the facial expression reconstruction modelE_test-tE_testTransformation matrix, UE_trainY being a reconstruction model of facial expressionsE_testCharacteristic vector of (V)E_trainThe feature vectors of dynamic facial expression image samples preset by the facial expression reconstruction model.
Optionally, in a possible implementation manner of the first aspect, before the obtaining, by the second type of neural response data and a preset facial expression reconstruction model, a facial expression image, the method further includes:
and acquiring a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample.
Taking the dynamic facial expression image sample as an output quantity and the second type neural response data sample as an input quantity, and performing four pairs of s according to the following formulaE-tEPerforming parameter learning by using the transformation matrix, the feature vector of the second class neural response data sample and the feature vector of the dynamic facial expression image training sample to obtain s in the facial expression reconstruction modelE_test-tE_testA transformation matrix, a characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, a characteristic vector of a facial expression image in the facial expression reconstruction model,
Figure GDA0003053384790000034
wherein, XEIs the dynamic facial expression image sample,
Figure GDA0003053384790000035
is XEAverage image of (2), YEIs the second type of neural response data sample,
Figure GDA0003053384790000036
is YEAverage data of sEIs YEProjected coordinates of (a), tEIs XEProjected coordinates of (1), WEIs the above-mentioned sE-tETransformation matrix, UEIs YECharacteristic vector of (V)EIs XEId is a label of each individual face identity.
Reconstructing the model according to the facial expressionE_test-tE_testAnd transforming a matrix, the characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, and the characteristic vector of a facial expression image in the facial expression reconstruction model to obtain the facial expression reconstruction model.
Optionally, in a possible implementation manner of the first aspect, the obtaining, according to the third type of neural response data and a preset human face identity reconstruction model, a human face identity image includes:
and acquiring a face identity image according to the following formula five and the third type of nerve response data.
Figure GDA0003053384790000041
Wherein, XI_RECONIs the image of the identity of the human face,
Figure GDA0003053384790000042
average image, Y, of dynamic face identity image samples preset in the face identity reconstruction modelI_testThe neural response data of the third type is,
Figure GDA0003053384790000043
is the average data, s, of the third type of neural response data samples caused by the preset dynamic human face identity image samples in the human face identity reconstruction modelI_testIs YI_testProjected coordinates of (a), tI_testIs XI_RECONProjected coordinates of (1), WI-trainIs in the face identity reconstruction modelI_test-tI_testTransformation matrix, UI_trainIs a human face identity reconstruction model YI_testCharacteristic vector of (V)I_trainThe feature vector of the dynamic face identity image sample is preset in the face identity reconstruction model.
Optionally, in a possible implementation manner of the first aspect, before the obtaining, by the third type of neural response data and a preset human face identity reconstruction model, a human face identity image, the method further includes:
and acquiring a dynamic face identity image training sample and a third type of neural response data training sample caused by the dynamic face identity image training sample.
Taking the dynamic face identity image sample as an output quantity and the third type nerve response data sample as an input quantity, and carrying out six pairs of transformation matrixes s through the following formulaI-tIAnd performing parameter learning on the characteristic vector of the third type of neural response data sample and the characteristic vector of the dynamic face identity image training sample to obtain s in the face identity reconstruction modelI_test-tI_testThe transformation matrix, the characteristic vector of a third class of neural response data sample in the face identity reconstruction model and the characteristic vector of a face identity image training sample,
Figure GDA0003053384790000044
wherein, XIIs the dynamic face identity image sample,
Figure GDA0003053384790000045
is XIAverage image of (2), YIIs the third type of neural response data sample,
Figure GDA0003053384790000046
is YIAverage data of sIIs YIProjected coordinates of (a), tIIs XIProjected coordinates of (1), WIIs the above-mentioned sI-tITransformation matrix, UIIs YICharacteristic vector of (V)IIs a human face identity reconstruction model XIEx is a label for each facial expression.
Reconstructing model s according to the face identityI_test-tI_testThe face identity reconstruction model is obtained by the transformation matrix, the feature vector of the third type of neural response data sample in the face identity reconstruction model and the feature vector of the face identity image in the face identity reconstruction model.
Optionally, in a possible implementation manner of the first aspect, the first type of neural response data is neural response data obtained from a brain primary visual cortex brain region of a user to be tested.
The second type of nerve response data is obtained from the posterior temporal superior sulcus and the amygdala brain area of the user to be tested.
The third type of nerve response data is obtained from a brain area of a self-fusiform back face hole processing area and a front temporal lobe brain area of the user to be detected.
Optionally, in a possible implementation manner of the first aspect, the obtaining a dynamic face image according to the face basic image, the facial expression image, and the face identity image includes:
and determining the human face basic image, the human face expression image and the average image of the human face identity image as the dynamic human face image.
In a second aspect of the embodiments of the present invention, there is provided a device for reconstructing a dynamic face image, including:
the first acquisition module is used for extracting first-class nerve response data and acquiring a human face basic image according to the first-class nerve response data and a preset human face image reconstruction model.
And the second acquisition module is used for extracting second-class neural response data and acquiring a facial expression image according to the second-class neural response data and a preset facial expression reconstruction model.
And the third acquisition module is used for extracting third-class nerve response data and acquiring a face identity image according to the third-class nerve response data and a preset face identity reconstruction model.
And the dynamic face image acquisition module is used for acquiring a dynamic face image according to the face basic image, the face expression image and the face identity image.
Optionally, in a possible implementation manner of the second aspect, the first obtaining module is configured to obtain a facial basis image according to the following formula one and the first type of neural response data.
Figure GDA0003053384790000051
Wherein, XG_RECONIs the base image of the face of the person,
Figure GDA0003053384790000052
is an average image, Y, of dynamic human face basic image samples preset in a human face image reconstruction modeltestIs a sample of the first type of neural response data,
Figure GDA0003053384790000053
is the average data, s, of the first type of neural response data samples caused by the dynamic human face basic image samples preset in the human face image reconstruction modeltestIs YtestProjected coordinates of (a), ttestIs XG_RECONProjected coordinates of (1), WtrainIs s in the face image reconstruction modeltest-ttestTransformation matrix, UtrainIs Y in the human face image reconstruction modeltestCharacteristic vector of (V)trainThe feature vector of the dynamic human face basic image sample is preset in the human face image reconstruction model.
Optionally, in a possible implementation manner of the second aspect, the first obtaining module 401 further includes a module for obtaining a dynamic face basic image training sample and a first type of neural response data training sample caused by the dynamic face basic image sample.
In the said dynamic stateUsing the human face basic image sample as an output quantity and the first type of neural response data sample as an input quantity, and performing parameter learning by using a two-pair s-t transformation matrix, the feature vector of the first type of neural response data sample and the feature vector of the dynamic human face basic image training sample according to the following formula to obtain s in the human face image reconstruction modeltest-ttestA transformation matrix, a characteristic vector of a first class of neural response data sample in a face image reconstruction model, a characteristic vector of a face basic image in the face image reconstruction model,
Figure GDA0003053384790000061
wherein X is the dynamic face base image sample,
Figure GDA0003053384790000062
is an average image of X, Y is the first type of neural response data sample,
Figure GDA0003053384790000063
is the average data of Y, s is the projection coordinates of Y, t is the projection coordinates of X, W is the s-t transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X.
Reconstructing the model according to the human face imagetest-ttestAnd transforming a matrix, the characteristic vector of the first type of neural response data sample in the face image reconstruction model, and the characteristic vector of the face basic image in the face image reconstruction model to obtain the face image reconstruction model.
Optionally, in a possible implementation manner of the second aspect, the second obtaining module is configured to obtain a facial expression image according to the following formula three and the second type of neural response data;
Figure GDA0003053384790000064
wherein, XE_RECONIs the image of the facial expression of the person,
Figure GDA0003053384790000065
is an average image, Y, of dynamic facial expression image samples preset in the facial expression reconstruction modelE_testIs a second type of neural response data sample,
Figure GDA0003053384790000066
is the average data, s, of second type neural response data samples caused by preset dynamic facial expression image samples in the facial expression reconstruction modelE_testIs YE_testProjected coordinates of (a), tE_testIs XE_RECONProjected coordinates of (1), WE_trainIs s in the facial expression reconstruction modelE_test-tE_testTransformation matrix, UE_trainY being a reconstruction model of facial expressionsE_testCharacteristic vector of (V)E_trainThe feature vectors of dynamic facial expression image samples preset by the facial expression reconstruction model.
Optionally, in a possible implementation manner of the second aspect, the second obtaining module further includes a module for obtaining a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample.
Taking the dynamic facial expression image sample as an output quantity and the second type neural response data sample as an input quantity, and performing four pairs of s according to the following formulaE-tEPerforming parameter learning by using the transformation matrix, the feature vector of the second class neural response data sample and the feature vector of the dynamic facial expression image training sample to obtain s in the facial expression reconstruction modelE_test-tE_testA transformation matrix, a characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, a characteristic vector of a facial expression image in the facial expression reconstruction model,
Figure GDA0003053384790000071
wherein, XEIs the dynamic facial expression graphLike the sample, the position of the sample,
Figure GDA0003053384790000072
is XEAverage image of (2), YEIs the second type of neural response data sample,
Figure GDA0003053384790000073
is YEAverage data of sEIs YEProjected coordinates of (a), tEIs XEProjected coordinates of (1), WEIs the above-mentioned sE-tETransformation matrix, UEIs YECharacteristic vector of (V)EIs XEId is a label of each individual face identity.
Reconstructing the model according to the facial expressionE_test-tE_testAnd transforming a matrix, the characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, and the characteristic vector of a facial expression image in the facial expression reconstruction model to obtain the facial expression reconstruction model.
Optionally, the third obtaining module is configured to obtain a face identity image according to the following formula five and the third type of neural response data.
Figure GDA0003053384790000074
Wherein, XI_RECONIs the image of the identity of the human face,
Figure GDA0003053384790000075
average image, Y, of dynamic face identity image samples preset in the face identity reconstruction modelI_testThe third type of neural response data sample,
Figure GDA0003053384790000076
is the average data, s, of the third type of neural response data samples caused by the preset dynamic human face identity image samples in the human face identity reconstruction modelI_testIs YI_testProjection seatMark, tI_testIs XI_RECONProjected coordinates of (1), WI-trainIs in the face identity reconstruction modelI_test-tI_testTransformation matrix, UI_trainIs a human face identity reconstruction model YI_testCharacteristic vector of (V)I_trainThe feature vector of the dynamic face identity image sample is preset in the face identity reconstruction model.
Optionally, in a possible implementation manner of the second aspect, the third obtaining module further includes a third module for obtaining a dynamic face identity image training sample and a third type of neural response data training sample caused by the dynamic face identity image training sample; taking the dynamic face identity image sample as an output quantity and the third type nerve response data sample as an input quantity, and carrying out six pairs of transformation matrixes s through the following formulaI-tIAnd performing parameter learning on the characteristic vector of the third type of neural response data sample and the characteristic vector of the dynamic face identity image training sample to obtain s in the face identity reconstruction modelI_test-tI_testThe transformation matrix, the characteristic vector of a third class of neural response data sample in the face identity reconstruction model and the characteristic vector of a face identity image training sample,
Figure GDA0003053384790000081
wherein, XIIs the dynamic face identity image sample,
Figure GDA0003053384790000082
is XIAverage image of (2), YIIs the third type of neural response data sample,
Figure GDA0003053384790000083
is YIAverage data of sIIs YIProjected coordinates of (a), tIIs XIProjected coordinates of (1), WIIs the above-mentioned sI-tITransformation matrix, UIIs YICharacteristic vector of (V)IIs the identity of a human faceReconstruction model XIEx is a label for each facial expression.
Reconstructing model s according to the face identityI_test-tI_testThe face identity reconstruction model is obtained by the transformation matrix, the feature vector of the third type of neural response data sample in the face identity reconstruction model and the feature vector of the face identity image in the face identity reconstruction model.
Optionally, in a possible implementation manner of the second aspect, the first type of neural response data is neural response data obtained from a brain primary visual cortex brain region of a user to be tested.
The second type of nerve response data is obtained from the posterior temporal superior sulcus and the amygdala brain area of the user to be tested.
The third type of nerve response data is obtained from a brain area of a self-fusiform back face hole processing area and a front temporal lobe brain area of the user to be detected.
Optionally, in a possible implementation manner of the second aspect, the dynamic face image obtaining module is configured to determine the face basic image, the facial expression image, and the average image of the face identity image as the dynamic face image.
The invention provides a method for reconstructing a dynamic face image, which aims at the characteristic that the dynamic face image mainly presents high-level visual characteristic information, and the facial characteristics with different attributes are the cognitive characteristics processed by different high-level cognitive brain regions, the scheme adopts three kinds of high-level characteristic information with different attributes, utilizes three different high-level cognitive brain regions to acquire first class neural response data, second class neural response data and third class neural response data corresponding to the three kinds of high-level characteristic information with different attributes of the face, simultaneously constructs different models of the high-level cognitive brain regions and the dynamic face from a visual image space to a brain perception space and multi-dimensional mapping relations among the models, acquires a face basic image, a face expression image and a face identity image to realize the reconstruction of multi-dimensional facial characteristics, acquires the dynamic face image and can reconstruct the dynamic face image perceived by some patients, leading us to deeper understanding and cognition of the cognitive disorder mechanism of mental diseases.
Drawings
FIG. 1 is a schematic flow chart of a method for reconstructing a dynamic human face image according to the present invention;
FIG. 2 is a schematic signal transmission diagram of a dynamic face image reconstruction method according to the present invention;
fig. 3 is a schematic structural diagram of an apparatus for reconstructing a dynamic human face image according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a hardware structure of a dynamic face image reconstruction apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein.
It should be understood that, in various embodiments of the present invention, the sequence numbers of the processes do not mean the execution sequence, and the execution sequence of the processes should be determined by the functions and the internal logic of the processes, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
It should be understood that in the present application, "comprising" and "having" and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The terms to which the present invention relates will be explained first:
functional Magnetic Resonance Imaging (functional Magnetic Resonance Imaging, fMRI for short): it is an emerging neuroimaging method, whose principle is to measure the change of blood power caused by neuron activity by magnetic resonance imaging.
The specific application scene of the invention can be suitable for reconstructing the dynamic face image perceived by the patient with the defects when the patient with the diseases with cognitive and mental disorders such as face agnosia, autism, senile dementia and Parkinson disease identifies the high-level characteristic attribute of the dynamic face, so that the cognitive disorder mechanism of the mental disorders can be deeply understood and recognized.
The invention provides a method for reconstructing a dynamic face image, which aims to solve the technical problems in the prior art, realize dynamic face reconstruction, reconstruct expression characteristics and identity characteristics in the reconstructed dynamic face image, enrich reconstructed information and improve the accuracy of face reconstruction.
The following describes the technical solutions of the present invention and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of a method for reconstructing a dynamic human face image according to the present invention, and an execution subject of the method shown in fig. 1 may be a software and/or hardware device. The method shown in fig. 1 includes steps S101 to S104, which are specifically as follows:
s101, extracting first-class nerve response data, and obtaining a human face basic image according to the first-class nerve response data and a preset human face image reconstruction model.
Specifically, the first type of neural response data is neural response data acquired from a primary visual cortex brain area of a brain of a user to be detected, different attribute characteristics of the face are cognitive processed by the different brain areas of the brain, the primary visual cortex brain area of the brain can sense pixel-level low-level visual characteristics of the face, the first type of neural response data adopts a functional magnetic resonance imaging technology to acquire a functional magnetic resonance signal of the primary visual cortex brain area of the brain of the user to be detected to obtain first type of neural response data, and a preset face image reconstruction model responds to the first type of neural response data to acquire a corresponding face basic image.
S102, second nerve response data are extracted, and a facial expression image is obtained according to the second nerve response data and a preset facial expression reconstruction model.
Specifically, the second type of nerve response data is nerve response data acquired from a back temporal sulcus and an amygdala brain area of the user to be detected, facial expression features of the face are processed by the back temporal sulcus, the amygdala brain area and other brain areas, the second type of nerve response data adopts a functional magnetic resonance imaging technology, functional magnetic resonance signals of the back temporal sulcus and the amygdala brain area of the user to be detected are acquired, the second type of nerve response data is acquired, and the preset face expression reconstruction model responds to the second type of nerve response data to acquire a corresponding face expression image.
S103, extracting third-class nerve response data, and acquiring a face identity image according to the third-class nerve response data and a preset face identity reconstruction model.
Specifically, the third type of nerve response data is nerve response data acquired from a self-fusiform return-to-the-face hole processing area brain area and a front temporal lobe brain area of a user to be detected, facial identity characteristics of a face are processed by the brain areas such as the self-fusiform return-to-the-face hole processing area brain area and the front temporal lobe brain area, the third type of nerve response data adopts a functional magnetic resonance imaging technology to acquire a functional magnetic resonance signal of the self-fusiform return-to-the-face hole processing area brain area or the front temporal lobe brain area of the user to be detected, so that the third type of nerve response data is acquired, and a preset face identity reconstruction model responds to the third type of nerve response data to acquire a corresponding face identity image.
And S104, acquiring a dynamic face image according to the face basic image, the face expression image and the face identity image.
Specifically, the face basic image, the face expression image, and the average image of the face identity image are determined as the dynamic face image.
In this embodiment, the above steps S101 to S103 are not limited by the described operation sequence, and the steps S101 to S103 may be performed in other sequences or simultaneously.
The dynamic face image reconstruction method provided in the above embodiment adopts three different attribute high-level feature information, and obtains a first type of neural response data, a second type of neural response data, and a third type of neural response data corresponding to the three different attribute high-level feature information of the face by using three different high-level cognitive brain regions, and simultaneously constructs models of the different high-level cognitive brain regions and the dynamic face from a visual image space to a brain perception space and a multi-dimensional mapping relationship between the models, and obtains a face basic image, a face expression image, and a face identity image, so as to realize reconstruction of multi-dimensional face features, obtain a dynamic face image, and reconstruct a dynamic face image perceived by some detected users.
On the basis of the above embodiment, the specific implementation manner of step S101 (obtaining a face basic image according to the first-type neural response data and a preset face image reconstruction model) may be:
referring to fig. 2, fig. 2 is a signal transmission schematic diagram of a dynamic face image reconstruction method provided by the present invention, where the expression of a dynamic face in a brain sensing space includes a basic dynamic face sensing space, a face expression sensing space, and a face identity sensing space; the expression of the dynamic human face in the image space comprises a basic image pixel space, a facial image expression space and a facial image identity space.
Acquiring a human face basic image in the step S101 according to the following formula I (namely a human face image reconstruction model) and the first type of nerve response data,
Figure GDA0003053384790000111
wherein, XG_RECONIs the base image of the face of the person,
Figure GDA0003053384790000112
is an average image, Y, of dynamic human face basic image samples preset in a human face image reconstruction modeltestIs a sample of the first type of neural response data,
Figure GDA0003053384790000113
is the average data, s, of the first type of neural response data samples caused by the dynamic human face basic image samples preset in the human face image reconstruction modeltestIs in YtestProjected coordinates of (a), ttestIs XG_RECONProjected coordinates of (1), WtrainIs s in the face image reconstruction modeltest-ttestTransformation matrix, UtrainIs Y in the human face image reconstruction modeltestCharacteristic vector of (V)trainThe feature vector of the dynamic human face basic image sample is preset in the human face image reconstruction model.
On the basis of the above embodiment, before obtaining the face basic image according to the first-class neural response data and the preset face image reconstruction model, a process of learning each parameter in the face image reconstruction model may also be included, specifically as follows:
s201, taking the dynamic human face basic image sample as an output quantity,The first type of neural response data samples are used as input quantity, parameter learning is carried out through the two pairs of s-t transformation matrixes, the feature vectors of the first type of neural response data samples and the feature vectors of the dynamic human face basic image training samples according to the formula, and s in the human face image reconstruction model is obtainedtest-ttestThe method comprises the steps of converting a matrix, the characteristic vector of a first type of neural response data sample in a face image reconstruction model, and the characteristic vector of a face basic image in the face image reconstruction model.
Wherein, the dynamic human face is under the pixel space of the basic image, and X is assumedjFor a dynamic face visual image, where j is 1,2, and N is the number of dynamic face images, each dynamic face is represented in a form of a single-dimensional vector, and then a dynamic face base image sample X is represented as follows: x ═ X1 X2 ... Xj ... XN]。
Performing PCA singular value decomposition on the X to generate a basic image pixel space by using the samples, wherein the projection coordinate of each dynamic face image sample in the basic image pixel space is as follows:
Figure GDA0003053384790000114
wherein
Figure GDA0003053384790000115
Is the average image of X, V is the eigenvector of X, and is sorted from high to low according to the size of the corresponding eigenvalue, and V can be more specifically expressed as V ═ V1,V2,...,VN]V is
Figure GDA0003053384790000116
All (linearly independent) feature vectors of (a) are a set of feature vectors per column.
Under the sample X of the dynamic human face basic image, each dynamic image (not limited to the image in the sample) can be linearly represented by its projection coordinates in the space, and this PCA-based decomposition process is reversible, so that any visual image can be reconstructed according to its projection coordinates in the feature space, and is expressed by the formula:
Figure GDA0003053384790000121
wherein, the dynamic human face is under the basic dynamic human face perception space, and Y is assumedjIs the neural response distribution of a dynamic human face image in a group of brain primary visual cortex brain areas, wherein j is 1,2jExpressed in the form of one-dimensional vector, the neural response of the dynamic human face image sample set in the primary visual cortex brain area of the brain, i.e. the first type of neural response data sample Y, is expressed as Y ═ Y1 Y2 ... Yj ... YN]And performing singular value decomposition on Y by using PCA, wherein the projection coordinate of the neural response of each dynamic human face image in the neural response space can be expressed as:
Figure GDA0003053384790000122
Figure GDA0003053384790000123
is the average data of Y, U is the eigenvector of Y, and is specifically expressed as U ═ U according to the corresponding eigenvalue from large to small1,U2,...,UN]。
Wherein, the multidimensional mapping relation is linear transformation that the projection coordinate t of the dynamic human face image sample under X is expressed as the projection coordinate s of the dynamic human face image sample under Y, namely
Formula 2.4 where t is sW
Where W is the s-t transform matrix, in the case where t and s are known and are full rank matrices, one solution for the transform matrix W is:
W＝(sTs+I)-1sTt equation 2.5
In summary, formula 2.1, formula 2.3, formula 2.4 and formula 2.5 form formula two,
Figure GDA0003053384790000124
wherein X is the dynamic face base image sample,
Figure GDA0003053384790000125
is an average image of X, Y is the first type of neural response data sample,
Figure GDA0003053384790000126
is the average data of Y, s is the projection coordinates of Y, t is the projection coordinates of X, W is the s-t transformation matrix, U is the eigenvector of Y, and V is the eigenvector of X.
S202, reconstructing a model according to the human face imagetest-ttestAnd transforming a matrix, the characteristic vector of a first class of neural response data sample in the face image reconstruction model, and the characteristic vector of a face basic image in the face image reconstruction model to obtain the face image reconstruction model with the formula I.
On the basis of the above embodiment, the specific implementation manner of step S102 (extracting the second type of neural response data, and obtaining the facial expression image according to the second type of neural response data and the preset facial expression reconstruction model) may be:
acquiring a facial expression image according to a formula III (namely a facial expression reconstruction model) and the second type of neural response data;
Figure GDA0003053384790000131
wherein, XE_RECONIs the image of the facial expression of the person,
Figure GDA0003053384790000132
is an average image, Y, of dynamic facial expression image samples preset in the facial expression reconstruction modelE_testIs a second type of neural response data sample,
Figure GDA0003053384790000133
is the average data, s, of second type neural response data samples caused by preset dynamic facial expression image samples in the facial expression reconstruction modelE_testIs YE_testProjected coordinates of (a), tE_testIs XE_RECONProjected coordinates of (1), WE_trainIs s in the facial expression reconstruction modelE_test-tE_testTransformation matrix, UE_trainY being a reconstruction model of facial expressionsE_testCharacteristic vector of (V)E_trainThe feature vectors of dynamic facial expression image samples preset by the facial expression reconstruction model.
On the basis of the above embodiment, before obtaining the facial expression image according to the second type of neural response data and the preset facial expression reconstruction model, a process of learning each parameter in the facial expression reconstruction model may also be included, which is specifically as follows:
s301, taking the dynamic facial expression image sample as an output quantity, taking the second type neural response data sample as an input quantity, and carrying out four pairs of S through the formulaE-tEPerforming parameter learning by using the transformation matrix, the feature vector of the second class neural response data sample and the feature vector of the dynamic facial expression image training sample to obtain s in the facial expression reconstruction modelE_test-tE_testThe transformation matrix, the characteristic vector of a second type of neural response data sample in the facial expression reconstruction model and the characteristic vector of a facial expression image in the facial expression reconstruction model.
Acquiring a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample; the dynamic human face is marked with N sample images in the image sample set again in the facial image expression space, and the dynamic human face sample set is integrated again, and the sample X is trained by the human face expression imageEIs expressed as follows:
Figure GDA0003053384790000134
wherein XEEach column of (A) is formed by splicing P different facial identity single-dimensional vectors of the same facial expression to represent a facial expression, XEEach row of (a) has Q values representing Q expression changes of the same facial identity at local positions of an image.
To XEPerforming PCA-based singular value decomposition with each facial expression at XEThe following projection coordinates may be expressed as:
Figure GDA0003053384790000135
Figure GDA0003053384790000141
is XEAverage image of VEIs XEThe eigenvalues are expressed as a big-to-small order
Figure GDA0003053384790000142
At XEEach facial expression (not limited to the expression type in the sample) can be represented by its projection coordinates in this space, and since the decomposition process of PCA is reversible, any type of facial expression can be reconstructed from its projection coordinates in the expression feature space:
Figure GDA0003053384790000143
wherein, the dynamic human face is in the human face expression perception space, and Y is assumedi,eFor the nerve response distribution of a dynamic human face image in the posterior temporal superior sulcus and amygdala brain region, the Y-shaped face image is obtainedi,eRearranging the neural responses of each column with one kind of expressions to obtain a second kind of neural response data sample YETo YEPerforming singular value decomposition, and determining the neural response of each facial expression in the YEThe following projection coordinates may be expressed as:
Figure GDA0003053384790000144
wherein the content of the first and second substances,
Figure GDA0003053384790000145
is YEAverage data of (U)EIs a feature vector which is specifically expressed as the corresponding feature value from large to small
Figure GDA0003053384790000146
Dynamic human face image sample is in XELower projection coordinate tEExpressed as it is in YELower projection coordinate sELinear transformation of (3), defining tEAnd sEComprises the following steps:
Figure GDA0003053384790000147
Figure GDA0003053384790000148
here id is the label of each individual face identity, and the established mapping relationship is as follows:
tE(id)＝sE(id)WE(id)equation 4.4
WE(id)One parsing of (a) can be represented as:
WE(id)＝(sT E(id)sE(id)+I)-1sT E(id)tE(id)equation 4.5
In summary, equation 4.1, equation 4.3, equation 4.4, and equation 4.5 form equation four,
Figure GDA0003053384790000149
wherein, XEIs the dynamic faceThe sample of the expression image is displayed,
Figure GDA00030533847900001410
is XEAverage image of (2), YEIs the second type of neural response data sample,
Figure GDA00030533847900001411
is YEAverage data of sEIs YEProjected coordinates of (a), tEIs XEProjected coordinates of (1), WEIs the above-mentioned sE-tETransformation matrix, UEIs YECharacteristic vector of (V)EIs XEThe feature vector of (2).
S302, reconstructing the model according to the facial expressionE_test-tE_testAnd transforming a matrix, the characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, and the characteristic vector of a facial expression image in the facial expression reconstruction model to obtain the facial expression reconstruction model.
On the basis of the foregoing embodiment, the specific implementation manner of step S103 (extracting the third type of neural response data, and obtaining the face identity image according to the third type of neural response data and the preset face identity reconstruction model) may be:
acquiring a face identity image according to a formula five (namely a face identity reconstruction model) and the third type of nerve response data;
Figure GDA0003053384790000151
wherein, XI_RECONIs the image of the identity of the human face,
Figure GDA0003053384790000152
average image, Y, of dynamic face identity image samples preset in the face identity reconstruction modelI_testThe third type of neural response data sample,
Figure GDA0003053384790000153
is the average data, s, of the third type of neural response data samples caused by the preset dynamic human face identity image samples in the human face identity reconstruction modelI_testIs YI_testProjected coordinates of (a), tI_testIs XI_RECONProjected coordinates of (1), WI-trainIs in the face identity reconstruction modelI_test-tI_testTransformation matrix, UI_trainIs a human face identity reconstruction model YI_testCharacteristic vector of (V)I_trainThe feature vector of the dynamic face identity image sample is preset in the face identity reconstruction model.
On the basis of the above embodiment, before the face identity image is acquired according to the third type of neural response data and the preset face identity reconstruction model, a process of learning each parameter in the face identity image reconstruction model may also be included, which is specifically as follows:
s401, taking the dynamic face identity image sample as an output quantity, taking the third type nerve response data sample as an input quantity, and transforming a matrix S by six pairs of the formulaI-tIAnd performing parameter learning on the characteristic vector of the third type of neural response data sample and the characteristic vector of the dynamic face identity image training sample to obtain s in the face identity reconstruction modelI_test-tI_testThe transformation matrix, the feature vector of a third type of neural response data sample in the face identity reconstruction model and the feature vector of a face identity image training sample.
And acquiring a dynamic face identity image training sample and a third type of neural response data training sample caused by the dynamic face identity image training sample.
The dynamic human face integrates the marked N sample images again in the face image identity space, and the dynamic human face identity image sample X is usedIIs represented by XIEach column of (A) is formed by splicing Q different facial expression single-dimensional vectors of the same facial identity, representing a facial identity individual, XIEach row of (a) has P values representing P individual identity changes of the same facial expression at local positions in the image.
To XIPerforming PCA-based singular value decomposition, the projection coordinates of each individual face identity in this new identity feature space can be expressed as:
Figure GDA0003053384790000154
Figure GDA0003053384790000155
is XIAverage image of VIIs XIThe eigenvalues in descending order can be represented as
Figure GDA0003053384790000161
At XIThen, each individual face (not limited to the individual in the sample) can be represented by its projection coordinates in this space, and since the decomposition process of PCA is reversible, any individual face can be reconstructed from its projection coordinates in the identity feature space:
Figure GDA0003053384790000162
wherein, the dynamic human face is in the human face identity perception space, and the hypothesis Y isi,eFor the nerve response distribution of a dynamic human face image in a self-fusiform back face hole processing area brain area and an anterior temporal lobe brain area, the Y direction of the nerve response distribution is adjustedi,eRearranging the nerve response data of each row of individuals with the face identity to obtain YIAnd to YIPerforming PCA singular value decomposition, and performing neural response of each facial expression at YIThe following projection coordinates may be expressed as:
Figure GDA0003053384790000163
here, the
Figure GDA0003053384790000164
Is YIAverage data of (U)IIs YIThe feature vector of (1) is expressed as the corresponding feature value from large to small
Figure GDA0003053384790000165
Projecting coordinate t of dynamic human face image sample in face image identity spaceIExpressed as its projection coordinates s in neural response spaceILinear transformation of (2). Redefining tIAnd sIComprises the following steps:
Figure GDA0003053384790000166
Figure GDA0003053384790000167
here ex is a label for each facial expression. The mapping relationship is as follows:
tI(ex)＝sI(ex)WI(ex)equation 6.4
WI(ex)The analytic solution of (d) can be expressed as:
WI(ex)＝(sT I(ex)sI(ex)+I)-1sT I(ex)tI(ex)equation 6.5
In summary, equation six is formed from equations 6.1, 6.3, 6.4, and 6.5,
Figure GDA0003053384790000168
wherein, XIIs the dynamic face identity image sample,
Figure GDA0003053384790000169
is XIAverage image of (2), YIIs the third type of neural response data sample,
Figure GDA00030533847900001610
is YIAverage data of sIIs YIProjected coordinates of (a), tIIs XIProjected coordinates of (1), WIIs the above-mentioned sI-tITransformation matrix, UIIs YICharacteristic vector of (V)IIs a human face identity reconstruction model XIThe feature vector of (2).
S402, reconstructing the model according to the face identityI_test-tI_testThe face identity reconstruction model is obtained by the transformation matrix, the feature vector of the third type of neural response data sample in the face identity reconstruction model and the feature vector of the face identity image in the face identity reconstruction model.
Referring to fig. 3, which is a schematic structural diagram of an apparatus for reconstructing a dynamic face image according to an embodiment of the present invention, the apparatus 40 for reconstructing a dynamic face image shown in fig. 3 includes:
the first obtaining module 401 is configured to extract first-class neural response data, and obtain a basic face image according to the first-class neural response data and a preset face image reconstruction model.
The second obtaining module 402 is configured to extract second-class neural response data, and obtain a facial expression image according to the second-class neural response data and a preset facial expression reconstruction model.
The third obtaining module 403 is configured to extract third type of neural response data, and obtain a face identity image according to the third type of neural response data and a preset face identity reconstruction model.
A dynamic face image obtaining module 404, configured to obtain a dynamic face image according to the face basic image, the face expression image, and the face identity image.
The apparatus for reconstructing a dynamic face image in the embodiment shown in fig. 3 can be correspondingly used to perform the steps in the embodiment of the method shown in fig. 1, and the implementation principle and the technical effect are similar, which are not described herein again.
Optionally, the first obtaining module 401 is configured to obtain a face basic image according to the following formula one and the first type of neural response data;
Figure GDA0003053384790000171
wherein, XG_RECONIs the base image of the face of the person,
Figure GDA0003053384790000172
is an average image, Y, of dynamic human face basic image samples preset in a human face image reconstruction modeltestIs a sample of the first type of neural response data,
Figure GDA0003053384790000173
is the average data, s, of the first type of neural response data samples caused by the dynamic human face basic image samples preset in the human face image reconstruction modeltestIs in YtestProjected coordinates of (a), ttestIs XG_RECONProjected coordinates of (1), WtrainIs s in the face image reconstruction modeltest-ttestTransformation matrix, UtrainIs Y in the human face image reconstruction modeltestCharacteristic vector of (V)trainThe feature vector of the dynamic human face basic image sample is preset in the human face image reconstruction model.
Optionally, the first obtaining module 401 further includes a module for obtaining a dynamic face basic image training sample and a first type of neural response data training sample caused by the dynamic face basic image sample.
And performing parameter learning by using the dynamic human face basic image sample as an output quantity and the first type of neural response data sample as an input quantity through a two-pair s-t transformation matrix, the characteristic vector of the first type of neural response data sample and the characteristic vector of the dynamic human face basic image training sample according to the following formula to obtain s in a human face image reconstruction modeltest-ttestTransformation matrix, feature vector of first-class neural response data sample in face image reconstruction model, and face imageLike the feature vectors of the base image of the face in the reconstruction model,
Figure GDA0003053384790000181
wherein X is the dynamic face base image sample,
Figure GDA0003053384790000182
is an average image of X, Y is the first type of neural response data sample,
Figure GDA0003053384790000183
is the average data of Y, s is the projection coordinate of Y, t is the projection coordinate of X, W is the s-t transformation matrix, U is the eigenvector of Y, V is the eigenvector of X; reconstructing the model according to the human face imagetest-ttestAnd transforming a matrix, the characteristic vector of the first type of neural response data sample in the face image reconstruction model, and the characteristic vector of the face basic image in the face image reconstruction model to obtain the face image reconstruction model.
Optionally, the second obtaining module 402 is configured to obtain a facial expression image according to the following formula three and the second type of neural response data;
Figure GDA0003053384790000184
wherein, XE_RECONIs the image of the facial expression of the person,
Figure GDA0003053384790000185
is an average image, Y, of dynamic facial expression image samples preset in the facial expression reconstruction modelE_testIs a second type of neural response data sample,
Figure GDA0003053384790000186
Figure GDA0003053384790000187
is the average data, s, of second type neural response data samples caused by preset dynamic facial expression image samples in the facial expression reconstruction modelE_testIs YE_testProjected coordinates of (a), tE_testIs XE_RECONProjected coordinates of (1), WE_trainIs s in the facial expression reconstruction modelE_test-tE_testTransformation matrix, UE_trainY being a reconstruction model of facial expressionsE_testCharacteristic vector of (V)E_trainThe feature vectors of dynamic facial expression image samples preset by the facial expression reconstruction model.
Optionally, the second obtaining module 402 further includes a module for obtaining a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample.
Taking the dynamic facial expression image sample as an output quantity and the second type neural response data sample as an input quantity, and performing four pairs of s according to the following formulaE-tEPerforming parameter learning by using the transformation matrix, the feature vector of the second class neural response data sample and the feature vector of the dynamic facial expression image training sample to obtain s in the facial expression reconstruction modelE_test-tE_testA transformation matrix, a characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, a characteristic vector of a facial expression image in the facial expression reconstruction model,
Figure GDA0003053384790000188
wherein, XEIs the dynamic facial expression image sample,
Figure GDA0003053384790000191
is XEAverage image of (2), YEIs the second type of neural response data sample,
Figure GDA0003053384790000192
is YEAverage data of sEIs YEProjected coordinates of (a), tEIs XEProjected coordinates of (1), WEIs the above-mentioned sE-tETransformation matrix, UEIs YECharacteristic vector of (V)EIs XEThe feature vector of (2); reconstructing the model according to the facial expressionE_test-tE_testAnd transforming a matrix, the characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, and the characteristic vector of a facial expression image in the facial expression reconstruction model to obtain the facial expression reconstruction model.
Optionally, the third obtaining module 403 is configured to obtain a face identity image according to the following formula five and the third type of neural response data;
Figure GDA0003053384790000193
wherein, XI_RECONIs the image of the identity of the human face,
Figure GDA0003053384790000194
average image, Y, of dynamic face identity image samples preset in the face identity reconstruction modelI_testThe third type of neural response data sample,
Figure GDA0003053384790000195
is the average data, s, of the third type of neural response data samples caused by the preset dynamic human face identity image samples in the human face identity reconstruction modelI_testIs YI_testProjected coordinates of (a), tI_testIs XI_RECONProjected coordinates of (1), WI-trainIs in the face identity reconstruction modelI_test-tI_testTransformation matrix, UI_trainIs a human face identity reconstruction model YI_testCharacteristic vector of (V)I_trainThe feature vector of the dynamic face identity image sample is preset in the face identity reconstruction model.
Optionally, the third obtaining module 403 further includes a module for obtaining dynamic face identity image training samples and for dynamically obtaining face identity image training samplesAnd training samples of third-class neural response data caused by the face identity image training samples. Taking the dynamic face identity image sample as an output quantity and the third type nerve response data sample as an input quantity, and carrying out six pairs of transformation matrixes s through the following formulaI-tIAnd performing parameter learning on the characteristic vector of the third type of neural response data sample and the characteristic vector of the dynamic face identity image training sample to obtain s in the face identity reconstruction modelI_test-tI_testThe transformation matrix, the characteristic vector of a third class of neural response data sample in the face identity reconstruction model and the characteristic vector of a face identity image training sample,
Figure GDA0003053384790000196
wherein, XIIs the dynamic face identity image sample,
Figure GDA0003053384790000197
is XIAverage image of (2), YIIs the third type of neural response data sample,
Figure GDA0003053384790000198
is YIAverage data of sIIs YIProjected coordinates of (a), tIIs XIProjected coordinates of (1), WIIs the above-mentioned sI-tITransformation matrix, UIIs YICharacteristic vector of (V)IIs a human face identity reconstruction model XIThe feature vector of (2);
reconstructing model s according to the face identityI_test-tI_testThe face identity reconstruction model is obtained by the transformation matrix, the feature vector of the third type of neural response data sample in the face identity reconstruction model and the feature vector of the face identity image in the face identity reconstruction model.
Optionally, the first type of neural response data is neural response data obtained from a primary visual cortex brain region of a brain of the user to be tested.
The second type of nerve response data is obtained from the posterior temporal superior sulcus and the amygdala brain area of the user to be tested.
The third type of nerve response data is obtained from a brain area of a self-fusiform back face hole processing area and a front temporal lobe brain area of the user to be detected.
Optionally, the dynamic face image obtaining module 404 is configured to determine the basic face image, the facial expression image, and the average image of the facial identity image as the dynamic face image.
Referring to fig. 4, which is a schematic diagram of a hardware structure of an apparatus according to an embodiment of the present invention, the apparatus 50 includes: a processor 51, a memory 52 and computer programs; wherein
A memory 52 for storing the computer program, which may also be a flash memory (flash). The computer program is, for example, an application program, a functional module, or the like that implements the above method.
A processor 51 for executing the computer program stored in the memory to implement the steps performed by the terminal in the above method. Reference may be made in particular to the description relating to the preceding method embodiment.
Alternatively, the memory 52 may be separate or integrated with the processor 51.
When the memory 52 is a device independent of the processor 51, the apparatus may further include:
a bus 53 for connecting the memory 52 and the processor 51.
The present invention also provides a readable storage medium, in which a computer program is stored, which, when being executed by a processor, is adapted to implement the methods provided by the various embodiments described above.
The readable storage medium may be a computer storage medium or a communication medium. Communication media includes any medium that facilitates transfer of a computer program from one place to another. Computer storage media may be any available media that can be accessed by a general purpose or special purpose computer. For example, a readable storage medium is coupled to the processor such that the processor can read information from, and write information to, the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may reside in an Application Specific Integrated Circuits (ASIC). Additionally, the ASIC may reside in user equipment. Of course, the processor and the readable storage medium may also reside as discrete components in a communication device. The readable storage medium may be a read-only memory (ROM), a random-access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
The present invention also provides a program product comprising execution instructions stored in a readable storage medium. The at least one processor of the device may read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the dynamic human face image reconstruction method provided by the various embodiments described above.
In the above embodiments of the apparatus, it should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (8)

1. A method for reconstructing a dynamic face image, comprising:
extracting first-class nerve response data, and reconstructing a model according to the first-class nerve response data and a preset human face image to obtain a human face basic image; the first type of neural response data is neural response data acquired from a brain primary visual cortex brain area of a user to be detected;
extracting second nerve response data, and acquiring a facial expression image according to the second nerve response data and a preset facial expression reconstruction model; the second type of nerve response data is obtained from the posterior temporal sulcus and the amygdala brain area of the user to be detected;
extracting third-class nerve response data, and acquiring a face identity image according to the third-class nerve response data and a preset face identity reconstruction model; the third type of nerve response data is obtained from a self-fusiform back-face hole processing area brain area and a front temporal lobe brain area of the user to be detected;
acquiring a dynamic face image according to the face basic image, the face expression image and the face identity image;
the acquiring a dynamic face image according to the face basic image, the face expression image and the face identity image comprises:
and determining the human face basic image, the human face expression image and the average image of the human face identity image as the dynamic human face image.
2. The method according to claim 1, wherein the obtaining a facial basis image according to the first neural response data and a preset facial image reconstruction model comprises:
acquiring a human face basic image according to the following formula I and the first type of nerve response data;
Figure FDA0003053384780000011
wherein, XG_RECONIs the base image of the face of the person,
Figure FDA0003053384780000012
is an average image, Y, of dynamic human face basic image samples preset in a human face image reconstruction modeltestIs the first type of neural response data,
Figure FDA0003053384780000013
is the average data, s, of the first type of neural response data samples caused by the dynamic human face basic image samples preset in the human face image reconstruction modeltestIs in YtestProjected coordinates of (a), ttestIs XG_RECONProjected coordinates of (1), WtrainIs s in the face image reconstruction modeltest-ttestTransformation matrix, UtrainIs Y in the human face image reconstruction modeltestCharacteristic vector of (V)trainThe feature vector of the dynamic human face basic image sample is preset in the human face image reconstruction model.
3. The method of claim 2, wherein before the first neural response data and the predetermined facial image reconstruction model obtain the facial basic image, the method further comprises:
acquiring a dynamic human face basic image training sample and a first type of neural response data training sample caused by the dynamic human face basic image sample;
and performing parameter learning by using the dynamic human face basic image sample as an output quantity and the first type of neural response data sample as an input quantity through a two-pair s-t transformation matrix, the characteristic vector of the first type of neural response data sample and the characteristic vector of the dynamic human face basic image training sample according to the following formula to obtain s in a human face image reconstruction modeltest-ttestTransformation matrix and face image reconstructionThe characteristic vector of the first type of nerve response data sample in the model, the characteristic vector of the face basic image in the face image reconstruction model,
Figure FDA0003053384780000021
wherein X is the dynamic face base image sample,
Figure FDA0003053384780000022
is an average image of X, Y is the first type of neural response data sample,
Figure FDA0003053384780000023
is the average data of Y, s is the projection coordinate of Y, t is the projection coordinate of X, W is the s-t transformation matrix, U is the eigenvector of Y, V is the eigenvector of X;
reconstructing the model according to the human face imagetest-ttestAnd transforming a matrix, the characteristic vector of the first type of neural response data sample in the face image reconstruction model, and the characteristic vector of the face basic image in the face image reconstruction model to obtain the face image reconstruction model.
4. The method of claim 1, wherein the obtaining a facial expression image according to the second neural response data and a preset facial expression reconstruction model comprises:
acquiring a facial expression image according to the following formula III and the second type of nerve response data;
Figure FDA0003053384780000024
wherein, XE_RECONIs the image of the facial expression of the person,
Figure FDA0003053384780000025
is an average image, Y, of dynamic facial expression image samples preset in the facial expression reconstruction modelE_testIs a second type of neural response data,
Figure FDA0003053384780000026
is the average data, s, of second type neural response data samples caused by preset dynamic facial expression image samples in the facial expression reconstruction modelE_testIs YE_testProjected coordinates of (a), tE_testIs XE_RECONProjected coordinates of (1), WE_trainIs s in the facial expression reconstruction modelE_test-tE_testTransformation matrix, UE_trainY being a reconstruction model of facial expressionsE_testCharacteristic vector of (V)E_trainThe feature vectors of dynamic facial expression image samples preset by the facial expression reconstruction model.
5. The method of claim 4, before the obtaining the facial expression image by the second neural response data and the preset facial expression reconstruction model, further comprising:
acquiring a dynamic facial expression image training sample and a second type of neural response data training sample caused by the dynamic facial expression image sample;
taking the dynamic facial expression image sample as an output quantity and the second type neural response data sample as an input quantity, and performing four pairs of s according to the following formulaE-tEPerforming parameter learning by using the transformation matrix, the feature vector of the second class neural response data sample and the feature vector of the dynamic facial expression image training sample to obtain s in the facial expression reconstruction modelE_test-tE_testA transformation matrix, a characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, a characteristic vector of a facial expression image in the facial expression reconstruction model,
Figure FDA0003053384780000031
wherein, XEIs the dynamic facial expression image sample,
Figure FDA0003053384780000032
is XEAverage image of (2), YEIs the second type of neural response data sample,
Figure FDA0003053384780000033
is YEAverage data of sEIs YEProjected coordinates of (a), tEIs XEProjected coordinates of (1), WEIs the above-mentioned sE-tETransformation matrix, UEIs YECharacteristic vector of (V)EIs XEThe feature vector of (2); id is the label of each individual facial identity;
reconstructing the model according to the facial expressionE_test-tE_testAnd transforming a matrix, the characteristic vector of a second type of neural response data sample in the facial expression reconstruction model, and the characteristic vector of a facial expression image in the facial expression reconstruction model to obtain the facial expression reconstruction model.
6. The method according to claim 1, wherein the obtaining a face identity image according to the third type of neural response data and a preset face identity reconstruction model comprises:
acquiring a face identity image according to the following formula five and the third type of nerve response data;
Figure FDA0003053384780000034
wherein, XI_RECONIs the image of the identity of the human face,
Figure FDA0003053384780000035
average image, Y, of dynamic face identity image samples preset in the face identity reconstruction modelI_testThe third type of nerveIn response to the data, the data is transmitted,
Figure FDA0003053384780000036
is the average data, s, of the third type of neural response data samples caused by the preset dynamic human face identity image samples in the human face identity reconstruction modelI_testIs YI_testProjected coordinates of (a), tI_testIs XI_RECONProjected coordinates of (1), WI-trainIs in the face identity reconstruction modelI_test-tI_testTransformation matrix, UI_trainIs a human face identity reconstruction model YI_testCharacteristic vector of (V)I_trainThe feature vector of the dynamic face identity image sample is preset in the face identity reconstruction model.
7. The method of claim 6, before the obtaining the face identity image by the third type of neural response data and the preset face identity reconstruction model, further comprising:
acquiring a dynamic face identity image training sample and a third type neural response data training sample caused by the dynamic face identity image training sample;
taking the dynamic face identity image sample as an output quantity and the third type nerve response data sample as an input quantity, and carrying out six pairs of transformation matrixes s through the following formulaI-tIAnd performing parameter learning on the characteristic vector of the third type of neural response data sample and the characteristic vector of the dynamic face identity image training sample to obtain s in the face identity reconstruction modelI_test-tI_testThe transformation matrix, the characteristic vector of a third class of neural response data sample in the face identity reconstruction model and the characteristic vector of a face identity image training sample,
Figure FDA0003053384780000041
wherein, XIIs the dynamic face identity image sample,
Figure FDA0003053384780000042
is XIAverage image of (2), YIIs the third type of neural response data sample,
Figure FDA0003053384780000043
is YIAverage data of sIIs YIProjected coordinates of (a), tIIs XIProjected coordinates of (1), WIIs the above-mentioned sI-tITransformation matrix, UIIs YICharacteristic vector of (V)IIs a human face identity reconstruction model XIThe feature vector of (2); ex is a label for each facial expression;
reconstructing model s according to the face identityI_test-tI_testThe face identity reconstruction model is obtained by the transformation matrix, the feature vector of the third type of neural response data sample in the face identity reconstruction model and the feature vector of the face identity image in the face identity reconstruction model.
8. An apparatus for dynamic face image reconstruction, comprising:
the first acquisition module is used for acquiring first-class nerve response data and acquiring a human face basic image according to the first-class nerve response data and a preset human face image reconstruction model; the first type of neural response data is neural response data acquired from a brain primary visual cortex brain area of a user to be detected;
the second acquisition module is used for acquiring second-class neural response data and acquiring a facial expression image according to the second-class neural response data and a preset facial expression reconstruction model; the second type of nerve response data is obtained from the posterior temporal sulcus and the amygdala brain area of the user to be detected;
the third acquisition module is used for acquiring third-class nerve response data and acquiring a face identity image according to the third-class nerve response data and a preset face identity reconstruction model; the third type of nerve response data is obtained from a self-fusiform back-face hole processing area brain area and a front temporal lobe brain area of the user to be detected;
the dynamic face image acquisition module is used for acquiring a dynamic face image according to the face basic image, the face expression image and the face identity image;
the dynamic face image obtaining module is specifically configured to determine the face basic image, the face expression image, and the average image of the face identity image as the dynamic face image.
CN201910382834.6A 2019-05-09 2019-05-09 Method and device for reconstructing dynamic face image Expired - Fee Related CN110148468B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910382834.6A CN110148468B (en) 2019-05-09 2019-05-09 Method and device for reconstructing dynamic face image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910382834.6A CN110148468B (en) 2019-05-09 2019-05-09 Method and device for reconstructing dynamic face image

Publications (2)

Publication Number Publication Date
CN110148468A CN110148468A (en) 2019-08-20
CN110148468B true CN110148468B (en) 2021-06-29

Family

ID=67594881

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910382834.6A Expired - Fee Related CN110148468B (en) 2019-05-09 2019-05-09 Method and device for reconstructing dynamic face image

Country Status (1)

Country Link
CN (1) CN110148468B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI719696B (en) * 2019-11-01 2021-02-21 財團法人工業技術研究院 Face image reconstruction method and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008013575A2 (en) * 2006-01-31 2008-01-31 University Of Southern California 3d face reconstruction from 2d images
CN101159015A (en) * 2007-11-08 2008-04-09 清华大学 Two-dimension human face image recognizing method
CN101320484A (en) * 2008-07-17 2008-12-10 清华大学 Three-dimensional human face recognition method based on human face full-automatic positioning
CN102254154A (en) * 2011-07-05 2011-11-23 南京大学 Method for authenticating human-face identity based on three-dimensional model reconstruction
CN109255830A (en) * 2018-08-31 2019-01-22 百度在线网络技术（北京）有限公司 Three-dimensional facial reconstruction method and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108109198A (en) * 2017-12-18 2018-06-01 深圳市唯特视科技有限公司 A kind of three-dimensional expression method for reconstructing returned based on cascade

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008013575A2 (en) * 2006-01-31 2008-01-31 University Of Southern California 3d face reconstruction from 2d images
CN101159015A (en) * 2007-11-08 2008-04-09 清华大学 Two-dimension human face image recognizing method
CN101320484A (en) * 2008-07-17 2008-12-10 清华大学 Three-dimensional human face recognition method based on human face full-automatic positioning
CN102254154A (en) * 2011-07-05 2011-11-23 南京大学 Method for authenticating human-face identity based on three-dimensional model reconstruction
CN109255830A (en) * 2018-08-31 2019-01-22 百度在线网络技术（北京）有限公司 Three-dimensional facial reconstruction method and device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"读心术：扫描大脑活动可重建你想象中的人脸图像";TechPunk;《https://www.sohu.com/a/223687701_102883》;20180223;正文第1-5页 *

Also Published As

Publication number Publication date
CN110148468A (en) 2019-08-20

Similar Documents

Publication Publication Date Title
McIntosh et al. Partial least squares analysis of neuroimaging data: applications and advances
Kay et al. Bottom-up and top-down computations in word-and face-selective cortex
EP4101371A1 (en) Electroencephalogram signal classifying method and apparatus, electroencephalogram signal classifying model training method and apparatus, and medium
Delorme et al. Independent EEG sources are dipolar
Wang et al. Use of neural signals to evaluate the quality of generative adversarial network performance in facial image generation
Calhoun et al. Extracting intrinsic functional networks with feature-based group independent component analysis
Guo et al. Longer fixation duration while viewing face images
CN109875509B (en) System and method for testing rehabilitation training effect of Alzheimer disease patient
CN108363487B (en) Construction method of dream reproduction model, and dream reproduction method and device
Zhao et al. Connectome-scale group-wise consistent resting-state network analysis in autism spectrum disorder
CN114424945B (en) Brain wave biological feature recognition system and method based on random graphic image flash
Nieto-Castanon Brain-wide connectome inferences using functional connectivity MultiVariate Pattern Analyses (fc-MVPA)
Wirsich et al. Multi-timescale hybrid components of the functional brain connectome: A bimodal EEG-fMRI decomposition
Svantesson et al. Virtual EEG-electrodes: Convolutional neural networks as a method for upsampling or restoring channels
CN110148468B (en) Method and device for reconstructing dynamic face image
CN113052800B (en) Alzheimer disease image analysis method and device
Wagner et al. Statistical non-parametric mapping in sensor space
RU2708197C1 (en) Method of measuring memorability of a multimedia message
Hatamimajoumerd et al. Enhancing multivariate pattern analysis for magnetoencephalography through relevant sensor selection
CN113842152B (en) Electroencephalogram signal classification network training method, classification method, equipment and storage medium
Rosero-Rodríguez et al. Automated preprocessing pipeline in visual imagery tasks
He et al. Investigating effective brain networks of action observation tasks from different visual perspectives based on generalized partial directed coherence: An fMRI study
Caplette et al. Rhythmic information sampling in the brain during visual recognition
Wang et al. Dynamic predictions: Oscillatory mechanisms underlying multisensory sequence processing
Brown et al. Reproducible paired sources from concurrent EEG-fMRI data using BICAR

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20210629