US20210183021A1 - Method to provide a video with a computer-modified visual of a desired face of a person - Google Patents
Method to provide a video with a computer-modified visual of a desired face of a person Download PDFInfo
- Publication number
- US20210183021A1 US20210183021A1 US17/117,748 US202017117748A US2021183021A1 US 20210183021 A1 US20210183021 A1 US 20210183021A1 US 202017117748 A US202017117748 A US 202017117748A US 2021183021 A1 US2021183021 A1 US 2021183021A1
- Authority
- US
- United States
- Prior art keywords
- person
- face
- visual
- video
- computer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 121
- 238000000034 method Methods 0.000 title claims description 55
- 238000012986 modification Methods 0.000 claims abstract description 62
- 230000004048 modification Effects 0.000 claims abstract description 61
- 238000013473 artificial intelligence Methods 0.000 claims abstract description 23
- 238000004458 analytical method Methods 0.000 claims abstract description 17
- 238000013135 deep learning Methods 0.000 claims description 20
- 238000011282 treatment Methods 0.000 claims description 14
- 239000002537 cosmetic Substances 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 9
- 239000011521 glass Substances 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 230000008921 facial expression Effects 0.000 claims description 4
- 210000005069 ears Anatomy 0.000 claims description 3
- 210000004209 hair Anatomy 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 3
- 210000000887 face Anatomy 0.000 description 12
- 230000006872 improvement Effects 0.000 description 12
- 241000282414 Homo sapiens Species 0.000 description 4
- 230000037303 wrinkles Effects 0.000 description 4
- 241000282412 Homo Species 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000036449 good health Effects 0.000 description 3
- 230000003867 tiredness Effects 0.000 description 3
- 208000016255 tiredness Diseases 0.000 description 3
- 210000003484 anatomy Anatomy 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 210000004709 eyebrow Anatomy 0.000 description 2
- 230000001815 facial effect Effects 0.000 description 2
- 210000003128 head Anatomy 0.000 description 2
- 230000007935 neutral effect Effects 0.000 description 2
- 206010001488 Aggression Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000036548 skin texture Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/167—Personality evaluation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/001—Image restoration
- G06T5/005—Retouching; Inpainting; Scratch removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
- A61B5/004—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room adapted for image acquisition of a particular organ or body part
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G06K9/00228—
-
- G06K9/00302—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
- G06Q50/01—Social networking
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B27/036—Insert-editing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
- A61B5/0037—Performing a preliminary scan, e.g. a prescan for identifying a region of interest
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/24—Indexing scheme for image data processing or generation, in general involving graphical user interfaces [GUIs]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- the present invention relates to a method and device to provide a video with a computer-modified visual of a desired face of a person.
- the face is one of the main areas of the body relevant for this appearance.
- the appearance is then categorized by others in the course of forming a first impression in a complex way.
- Characteristics attributed to a person when making a first impression are, for example, attractiveness, healthiness, youthfulness, tiredness, sadness, friendliness, dominance, competence, likability or trustworthiness just to name some of these.
- a person may want to improve his/her first impression as being perceived as dominant or competent in a video conference with customers, but may want to improve his/her first impression as being perceived as likeable and trustworthy in an evening video call with his/her family when being on a business trip.
- Another person using a partnership online platform may want to improve his/her first impression as being perceived as attractive or sexy by potential partners.
- WO 2015/017687A2 discloses a method and system with a server that determines the level of “severity” of a defect in a selected anatomical area.
- the server for instance analyses a photo of a person to rate wrinkles at the eyes in line with a standardized wrinkle rating.
- the server uses a database with the likely outcome of a selected medical or cosmetic treatment for the selected anatomical area to support a person in her/his decision about a medical or cosmetic treatment.
- US 2019/0005359 A1 discloses a method and system that processes a computer program to train an artificial intelligence that provides an automated rating of the characteristic of a visual of a face of a person.
- a data set of visuals of faces and extracted face property data thereof linked to face characteristics data provided by a representative set of humans that rate the visuals of these faces about their face characteristics are generated and stored in a database.
- Face property data of further visuals of faces are extracted and used together with the generated data set in the database for training of the artificial intelligence to enable the artificial intelligence to provide an automated rating of the characteristics of the visuals of faces.
- US 2019/0005359 A1 furthermore discloses to use the artificial intelligence to rate photos of different persons about their first impression for the characteristic “dominance” to support an HR manager in his decision which candidate to choose for a new job.
- a face-based personality analyse is processed based on a video that shows one or more persons like in a video conference.
- the invention is based on the finding that when aiming to understand a person's appearance it is only a secondary goal to e.g. look how much wrinkles a person has since persons categorize the appearance of others in the course of forming a first impression in a more complex way and as a whole.
- Characteristics attributed to a person when making a first impression are, for example, attractiveness, healthiness, youthfulness, tiredness, sadness, friendliness, dominance, competence, likability or trustworthiness just to name some of these.
- This inventive method, device and/or computer program uses a completely new concept and technique to enable a person to improve his/her first impression in a live video conference or stored video message as selected by the person.
- An improvement in a characteristic attributed by others during forming the first impression can be achieved in both ways, i.e. increasing a characteristic which is perceived as positive or decreasing a characteristic which is perceived as negative.
- the person selects what characteristic to improve for the upcoming video conference and sends a visual (photo or video) of his/her face to see what kind of different improvements for this characteristic are possible.
- the artificial intelligence proposes at least two sets of modifications how to modify the visual to improve the selected characteristic based on information stored in the database.
- the eyebrow position needs to be lifted and the volume of the jawline needs to be increased.
- This first set of modifications of the face of the person would in principle be possible in real life by a cosmetic and/or medical treatment of the person. Based on the inventive concept there is no limit to only provide sets of modifications and display only improvements that are within the borders of cosmetic and/or medical treatments. Therefore, the second set of modifications could for instance include modifications that reduce or increase the distance of the eyes or the distance between the eyes and the mouth or drastically change the form or size of the eyes, mouth or ears. Further sets of modifications may include objects a person may wear like glasses or earrings to change his/her first impression for other persons.
- All these different kind of sets of modifications of the visual of the face provided by the person are generated and provided by the artificial intelligence that in a next step modifies the visual and displays all possible computer-modified visuals of the face of the person.
- the person selects the one visual with the improved selected characteristic and choses his or her desired face for the upcoming video conference.
- the artificial intelligence identifies the face of the person in the live video and replaces it continuously with the desired face of the person. This provides the major advantage, that the person may select for each video conference or even during a video conference how he/she would like to support his/her message with the appropriate visual appearance.
- a person is of course fee to select one or more characteristics to improve his/her visual appearance.
- FIG. 3 shows a description and rating of the characteristics based on the visual data of the person's face.
- FIG. 4 shows how characteristics may be selected by the user in an App.
- FIG. 5 shows a table with examples of invasive and/or minimally invasive cosmetic and medical treatments to achieve changes of a desired characteristic of a person's face.
- FIG. 6 shows a line drawing of the face of a person with a data set of modifications how to improve the characteristic “competent”.
- FIG. 7 shows a photo of a face of a person with regions of the face marked to be treated to increase the characteristic “dominant” attributed to the person when making a first impression.
- FIG. 1 shows a system 1 to display a video with a computer-modified visual of a desired face of a person 2 with a device 3 .
- the device 3 could be a mobile device or a computer to processes a software and in particular an App for person 2 , who wants to modify her visual first impression as seen by other persons.
- person 2 wants to join a live video conference 4 with device 3 as shown in FIG. 2 , but she would like improve her characteristic “competence” to look more competent than she does in real life to increase her chances to be successful in this video conference 4 .
- Device 3 processes a deep learning based application 7 or artificial intelligence that analyses visual data representing photo 5 or a video of the face of person 2 to rate one or more characteristics attributed to person 2 when making a first impression.
- face characteristics may for example be attractiveness, healthiness, youthfulness, tiredness, sadness, friendliness, dominance, competence, likability or trustworthiness.
- the deep learning based application 7 is a computer program comprising instructions which, when the program is executed by device 3 , causes device 3 to carry out the following steps to display a video with a computer-modified visual of the desired face of person 2 .
- a data set of visual data of visuals of faces and extracted face property data thereof linked to face characteristics data is generated.
- conventional computer vision algorithms like a landmark detection divide the face of persons in regions like the chin and the jawline and automatically extract characteristics and their location in the face.
- face property data may for instance include the distance between the eyes or the distance between the eyes and the mouth and other distances to be measure to describe a face.
- These face property data are stored together with the visual data (photo or video) of these faces by the deep learning based application 7 in a database 8 of the device 3 .
- a representative number of such visuals of faces stored as visual data in database 8 are shown on a display to a representative number of humans to manually rate these visuals of faces about their characteristics.
- the humans may rate them with scores (e.g. from 0 to 7) for different characteristics. These human ratings are stored in database 8 linked to the visual data of the faces and provide a basis information for the deep learning based application 7 to automatically rate characteristics attributed to a person 2 when making a first impression.
- scores e.g. from 0 to 7
- face property data of these visuals of faces are extracted by the conventional computer vision algorithms for example landmark detection, wrinkle detection, skin texture analysis or analysis of facial proportions.
- These face property data of visuals of faces are used together with the data set generated and stored in database 8 in the first step for training of the deep learning based application 7 to enable the artificial intelligence to provide an improved automated rating of the characteristics of the visuals of faces.
- any visual of a face may be provided to the deep learning based application 7 , which will be based on the data set stored in database 8 provide an automated rating of the characteristics of the visuals of the face.
- FIG. 3 shows such a description and result of the automated rating of the characteristics of a person based on the visuals of the person's face displayed on mobile phone 6 .
- Such kind of an artificial intelligence for automated rating of the characteristics of a person based on visuals of the person is known to a man skilled in the art from US 2019/0005359 A1 and therefore not discussed in more detail here.
- system 1 is ready to be used to display a video with a computer-modified visual of a desired face of a person 2 with a device 3 as described in the following steps of the method.
- a third step at least one visual (photo or video) of person 2 has to be provided to device 3 .
- person 2 selects the one photo 5 that shows her normal face and sends this photo 5 from her mobile phone 6 to device 3 for further processing.
- photos may be taken in a standardized or a non-standardized way.
- Instructions for a person and/or a photographer to take a standardized photo may include one or more of the following steps: ask the person to take off e.g. earrings or a nose ring; ask the person not to smile, ask the person to make a neutral facial expression; ask the person to keep head hair out of his/her face; ask the person to look straight into the camera; good general condition of lightning; neutral background.
- device 3 comprises visual data modification means 9 to pre-process the provided photo 5 or video of the face of person 2 in an attempt to standardize their content prior to the analysis of deep learning based application 7 as much as possible.
- the pre-processing of photo 5 may include one or more of the following steps: cut-out the background behind the face from the visual data; cut-out the ears of the person's face to reduce the visual influence of e.g. earrings; cut-out clothes and other wardrobe that might influence with the face; cut-out the head hair of the person 2 .
- the visual data of photo 5 are sent from mobile phone 6 to device 3 .
- person 2 uses input means 10 to input at least one characteristic of her face she would like to improve, which selected characteristic is transferred as face characteristics data to deep learning based application 7 of device 3 .
- person 2 uses input means of the App of mobile phone 6 as shown in FIG. 4 where a button is realized for each characteristic a person may select.
- person 2 selects “competent” as characteristic, which is transmitted via broadband network to deep learning based application 7 of device 3 .
- person 2 may use another way to input the at least one characteristic with mobile phone 6 . This may be done by the selection of the App used by person 2 , as there may be an App to select the characteristic “attractive” and another App to select the characteristic “competent”.
- the deep learning based application 7 of device 3 analyses visual data of photo 5 of person 2 .
- learning based application 7 rates the characteristics of person 2 based on photo 5 and in particular evaluates and rates the characteristic “competent” of person 2 .
- the deep learning based application 7 extracts face properties of photo 5 by dividing the face of person 2 in photo 5 in regions like the chin and the jawline. If photo 5 sent from mobile phone 6 is a non-standardized photo 5 , then deep learning based application 7 is used to fill missing visual information or to estimate how the face on this photo 5 would look like if taken in a standardized way to enable a more or less standardized rating of the characteristics. As a result of this analysis device 3 scores 5 points for the characteristic “competent” with a scale from 0 to 7 points as can be seen in FIG. 3 .
- FIG. 5 shows a table with examples of invasive and/or minimally invasive cosmetic and medical treatments to achieve changes of a desired characteristic of a person's face, which table is based on clinical studies.
- Such data are stored in a modification data base 11 of device 3 .
- the desired characteristic may be improved by 1 or more points in the rating when processing a real treatment of a real person.
- the inventive concept is not limited by achievements of real treatments and furthermore stores data in modification data base 11 that provide an improvement of a characteristic, but in a way a real person would not be treated. So in one example the distance between the eyes would need to be increased significantly to improve one characteristic, which would not be an option in real life.
- Deep learning based application 7 based on data stored in modification data base 11 is built to provide not only one, but several data sets of modifications 12 which indicate areas in the face of the visual of person 2 which need to be modified by visual data modification means 9 to improve the desired selected characteristic in different ways and with different scores of improvement.
- FIG. 6 shows a line drawing of the face of person 2 with a first data set of modifications 12 how to improve the characteristic “competent” based on invasive and/or minimally invasive cosmetic and medical treatments listed in FIG. 5 .
- Deep learning based application 7 based on further data in modification data base 11 provides a second data set of modifications 12 how to improve the characteristic “competent” based on modifications that would not be possible with real persons, but provide a very good improvement of the desired selected characteristic and still look realistic in a way that person 2 still is recognized as the person 2 she is.
- deep learning based application 7 may provide several sets of modifications 12 , some with subtle improvements, which will be only recognized subconsciously, and others with recognizable improvements and some with dramatic improvements of the score of the selected characteristic.
- visual data modification means 9 modify the photo 5 of person 2 with each of the two or more data sets of modifications 12 and generate two or more computer-modified photos 13 of the face of person 2 .
- Data sets of modifications 12 technically are code instructions that are needed to modify the photo like in a software program PhotoShop®.
- FIG. 7 as another example shows the face of person 2 with another data set of modifications 12 how to improve the characteristic “dominant” based on invasive and/or minimally invasive cosmetic and medical treatments.
- a display 14 of device 3 is used to show person 2 all computer-modified photos 13 to enable her to select her desired modified facial appearance in the video conference 4 to come. She therefore may select between stronger improvements of the selected characteristic “competent” and less strong improvements of this characteristic and in an eighth step of the method decides which of the displayed computer-modified photos 13 is the desired face and the desired selected set of modifications 12 to be used.
- a ninth step of the method the live video conference is processed and person 2 uses camera 15 to generate a video 17 that shows her face during the video conference 4 as shown in FIG. 2 .
- Visual data modification means 9 receive the live video 17 and identify an area 16 in the video 17 that shows the face of person 2 .
- deep learning based application 7 uses visual data modification means 9 to continuously generate the computer-modified visual of the desired selected face of person 2 . This means that even during all the movements of the face of person 2 and all facial expressions person 2 does during the video conference 4 , her visual is continuously computer-modified to make her look more competent. This means that continuously her chin is modified to appear less wide and the cheeks less full and her eyebrows are lowered as shown in FIG. 6 .
- Visual data modification means 9 then replace the identified area 16 in the video 17 from the camera 15 with the computer-modified visual of person 2 and send this modified video with the computer-modified visual of person 2 as video stream for the video conference 4 .
- person 2 in video conference 4 appears with a visual that provides a first impression to the other participants of the video conference 4 of a very competent person. This gives person 2 a strong advantage in negotiations during the video conference 4 .
- modification data base 11 may store data about objects a person may wear like glasses or earrings to change his/her first impression.
- person 2 upfront uploads visuals of his/her personal objects, so for instance a photo of her five different glasses and her ten different earrings.
- Deep learning based application 7 based on these data stored in modification data base 11 may generate further sets of modifications 12 where for instance glasses are overlaid over the visual of person 2 as shown in FIG. 2 . This further extends the possibilities to improve the desired characteristic in the computer-modified visual of person 2 .
- person 2 may select to be shown only computer-modified visuals of her face in method step seven that are based on the table in FIG. 5 . So these computer-modifications in principle all would be possible with invasive and/or minimally invasive cosmetic and medical treatments on a real person and therefore the computer-visuals shown on display 14 may look more realistic than others.
- a dating App may use the method of the invention.
- a man/ woman may select a live chat with a person he/she would like to meet.
- the person may choose to improve his characteristic “trustworthy” or “dominant” and select the extent of improvement by selecting his/her computer-modified visual. This enables a complete new and inventive way to connect potential future partners.
- steps 2 to 8 of the method claimed in claim 1 are processed for two or more characteristics or combinations of characteristics and sets of modifications selected for these different characteristics by the person are stored in the visual data modification means 9 .
- the artificial intelligence is built to analyse the actual facial expressions of the face of the person in the video and to recognize content and/or tone of the person.
- the artificial intelligence for instance recognizes when person 2 gets angry or happy during the video conference. This recognition is based on visual and acoustical information provided in the video.
- a speech-to-text recognizer could in addition be used to enable a content analysis of the spoken words.
- video has to be interpreted in its broadest sense and covers any sequence of visual data frames, which may display objects like real persons or animated characters.
- animated character an avatar for a person could for instance be displayed, which could be an animal, phantasy creature or animated human being.
- video furthermore covers any two and three-dimensional visual representation of a live or stored data content. Video conferences like webinars, live presentations or distance teaching are covered as well.
Abstract
At least one characteristic of a face of the person to be improved is inputted. Artificial intelligence is used to analyse a visual of the person's face and generate data sets of modifications to improve the visual appearance of the person in different ways towards a selected characteristic. The visual of the face of the person is modified based on the data sets of modifications and computer-modified visuals of the face of the person are generated and displayed. One of the computer-modified visuals is selected as the desired face of the person. A video is provided that shows a computer-modified visual of the desired face of the person.
Description
- This application claims benefit of European Patent Application Serial No. 19215139.7, filed 11 Dec. 2019 and which application is incorporated herein by reference. To the extent appropriate, a claim of priority is made to the above disclosed application.
- The present invention relates to a method and device to provide a video with a computer-modified visual of a desired face of a person. There is a general wish to improve the own appearance in personal interactions in a private or business environment. The face is one of the main areas of the body relevant for this appearance. The appearance is then categorized by others in the course of forming a first impression in a complex way. Characteristics attributed to a person when making a first impression are, for example, attractiveness, healthiness, youthfulness, tiredness, sadness, friendliness, dominance, competence, likability or trustworthiness just to name some of these. A person may want to improve his/her first impression as being perceived as dominant or competent in a video conference with customers, but may want to improve his/her first impression as being perceived as likeable and trustworthy in an evening video call with his/her family when being on a business trip. Another person using a partnership online platform may want to improve his/her first impression as being perceived as attractive or sexy by potential partners.
- WO 2015/017687A2 discloses a method and system with a server that determines the level of “severity” of a defect in a selected anatomical area. The server for instance analyses a photo of a person to rate wrinkles at the eyes in line with a standardized wrinkle rating. The server uses a database with the likely outcome of a selected medical or cosmetic treatment for the selected anatomical area to support a person in her/his decision about a medical or cosmetic treatment.
- US 2019/0005359 A1 discloses a method and system that processes a computer program to train an artificial intelligence that provides an automated rating of the characteristic of a visual of a face of a person. To achieve that a data set of visuals of faces and extracted face property data thereof linked to face characteristics data provided by a representative set of humans that rate the visuals of these faces about their face characteristics are generated and stored in a database. Face property data of further visuals of faces are extracted and used together with the generated data set in the database for training of the artificial intelligence to enable the artificial intelligence to provide an automated rating of the characteristics of the visuals of faces. US 2019/0005359 A1 furthermore discloses to use the artificial intelligence to rate photos of different persons about their first impression for the characteristic “dominance” to support an HR manager in his decision which candidate to choose for a new job. In another embodiment a face-based personality analyse is processed based on a video that shows one or more persons like in a video conference.
- None of these prior art methods and systems provide technical means to support a person in his/her general wish to improve the own first impression as seen and categorized by other persons. These problems are solved with a method that comprises the following steps:
-
- Provide at least one visual of the face of the person;
- Input at least one characteristic of the face of the person to be improved;
- Use the artificial intelligence to analyse the visual of the person's face and to generate at least two data sets of modifications to improve the visual appearance of the person in at least two different ways towards the selected characteristic;
- Modify the visual of the face of the person based on both data sets of modifications and generate two computer-modified visuals of the face of the person;
- Display both computer-modified visuals of the face of the person;
- Select one of the two computer-modified visuals as the desired face of the person;
- Provide a video that shows the face of the person and use the selected data set of modifications of the selected desired face of the person to modify the video to provide the video with the computer-modified visual of the desired face of the person.
- The invention is based on the finding that when aiming to understand a person's appearance it is only a secondary goal to e.g. look how much wrinkles a person has since persons categorize the appearance of others in the course of forming a first impression in a more complex way and as a whole. Characteristics attributed to a person when making a first impression are, for example, attractiveness, healthiness, youthfulness, tiredness, sadness, friendliness, dominance, competence, likability or trustworthiness just to name some of these.
- This inventive method, device and/or computer program uses a completely new concept and technique to enable a person to improve his/her first impression in a live video conference or stored video message as selected by the person. An improvement in a characteristic attributed by others during forming the first impression can be achieved in both ways, i.e. increasing a characteristic which is perceived as positive or decreasing a characteristic which is perceived as negative. The person selects what characteristic to improve for the upcoming video conference and sends a visual (photo or video) of his/her face to see what kind of different improvements for this characteristic are possible. Based on the classification of the artificial intelligence about the characteristics of the person in his/her photo or video the artificial intelligence proposes at least two sets of modifications how to modify the visual to improve the selected characteristic based on information stored in the database. To e.g. improve the characteristic “competence” the eyebrow position needs to be lifted and the volume of the jawline needs to be increased. This first set of modifications of the face of the person would in principle be possible in real life by a cosmetic and/or medical treatment of the person. Based on the inventive concept there is no limit to only provide sets of modifications and display only improvements that are within the borders of cosmetic and/or medical treatments. Therefore, the second set of modifications could for instance include modifications that reduce or increase the distance of the eyes or the distance between the eyes and the mouth or drastically change the form or size of the eyes, mouth or ears. Further sets of modifications may include objects a person may wear like glasses or earrings to change his/her first impression for other persons. All these different kind of sets of modifications of the visual of the face provided by the person are generated and provided by the artificial intelligence that in a next step modifies the visual and displays all possible computer-modified visuals of the face of the person. The person selects the one visual with the improved selected characteristic and choses his or her desired face for the upcoming video conference. In a final step of the inventive method the artificial intelligence identifies the face of the person in the live video and replaces it continuously with the desired face of the person. This provides the major advantage, that the person may select for each video conference or even during a video conference how he/she would like to support his/her message with the appropriate visual appearance. A person is of course fee to select one or more characteristics to improve his/her visual appearance.
- In a preferred embodiment the artificial intelligence analyses the content and tone of a conversation in a video conference and in case the person changes from a silent and truthful behaviour into a loud shouting and aggressive behaviour, the artificial intelligence at the same time changes the visual appearance of the face to support his/her actual message. These changes of the visual appearance may be in small steps and subtle, to be only recognized subconsciously by the other participants of the conference, or dramatic, as selected by the person.
- These and further advantageous embodiments of the invention will be explained based on the following description and the accompanying drawings.
-
FIG. 1 shows a system to provide a video with a computer-modified visual of a desired face of a person. -
FIG. 2 shows a device of the system with an ongoing video conference and the computer-modified face of the person. -
FIG. 3 shows a description and rating of the characteristics based on the visual data of the person's face. -
FIG. 4 shows how characteristics may be selected by the user in an App. -
FIG. 5 shows a table with examples of invasive and/or minimally invasive cosmetic and medical treatments to achieve changes of a desired characteristic of a person's face. -
FIG. 6 shows a line drawing of the face of a person with a data set of modifications how to improve the characteristic “competent”. -
FIG. 7 shows a photo of a face of a person with regions of the face marked to be treated to increase the characteristic “dominant” attributed to the person when making a first impression. -
FIG. 1 shows asystem 1 to display a video with a computer-modified visual of a desired face of aperson 2 with adevice 3. Thedevice 3 could be a mobile device or a computer to processes a software and in particular an App forperson 2, who wants to modify her visual first impression as seen by other persons. As anexample person 2 wants to join alive video conference 4 withdevice 3 as shown inFIG. 2 , but she would like improve her characteristic “competence” to look more competent than she does in real life to increase her chances to be successful in thisvideo conference 4. To achieve that she only needs to select onephoto 5 or a video that shows her normal face and in this example sends thisphoto 5 from hermobile phone 6 todevice 3 for further processing. -
Device 3 processes a deep learning basedapplication 7 or artificial intelligence that analyses visualdata representing photo 5 or a video of the face ofperson 2 to rate one or more characteristics attributed toperson 2 when making a first impression. Such face characteristics may for example be attractiveness, healthiness, youthfulness, tiredness, sadness, friendliness, dominance, competence, likability or trustworthiness. The deep learning basedapplication 7 is a computer program comprising instructions which, when the program is executed bydevice 3, causesdevice 3 to carry out the following steps to display a video with a computer-modified visual of the desired face ofperson 2. - In a first step of the method, a data set of visual data of visuals of faces and extracted face property data thereof linked to face characteristics data is generated. To extract face properties conventional computer vision algorithms like a landmark detection divide the face of persons in regions like the chin and the jawline and automatically extract characteristics and their location in the face. Such face property data may for instance include the distance between the eyes or the distance between the eyes and the mouth and other distances to be measure to describe a face. These face property data are stored together with the visual data (photo or video) of these faces by the deep learning based
application 7 in adatabase 8 of thedevice 3. A representative number of such visuals of faces stored as visual data indatabase 8 are shown on a display to a representative number of humans to manually rate these visuals of faces about their characteristics. The humans may rate them with scores (e.g. from 0 to 7) for different characteristics. These human ratings are stored indatabase 8 linked to the visual data of the faces and provide a basis information for the deep learning basedapplication 7 to automatically rate characteristics attributed to aperson 2 when making a first impression. - In a second step face property data of these visuals of faces are extracted by the conventional computer vision algorithms for example landmark detection, wrinkle detection, skin texture analysis or analysis of facial proportions. These face property data of visuals of faces are used together with the data set generated and stored in
database 8 in the first step for training of the deep learning basedapplication 7 to enable the artificial intelligence to provide an improved automated rating of the characteristics of the visuals of faces. As a result, any visual of a face may be provided to the deep learning basedapplication 7, which will be based on the data set stored indatabase 8 provide an automated rating of the characteristics of the visuals of the face.FIG. 3 shows such a description and result of the automated rating of the characteristics of a person based on the visuals of the person's face displayed onmobile phone 6. Such kind of an artificial intelligence for automated rating of the characteristics of a person based on visuals of the person is known to a man skilled in the art from US 2019/0005359 A1 and therefore not discussed in more detail here. - After
database 8 has been setup with above described steps,system 1 is ready to be used to display a video with a computer-modified visual of a desired face of aperson 2 with adevice 3 as described in the following steps of the method. - In a third step at least one visual (photo or video) of
person 2 has to be provided todevice 3. As in thisexample person 2 selects the onephoto 5 that shows her normal face and sends thisphoto 5 from hermobile phone 6 todevice 3 for further processing. In principle, photos may be taken in a standardized or a non-standardized way. Instructions for a person and/or a photographer to take a standardized photo may include one or more of the following steps: ask the person to take off e.g. earrings or a nose ring; ask the person not to smile, ask the person to make a neutral facial expression; ask the person to keep head hair out of his/her face; ask the person to look straight into the camera; good general condition of lightning; neutral background.Photo 5 shown inFIG. 1 as an example has been taken as a standardized photo. Ifphoto 5 had been taken in a non-standardized way, the automated rating by deep learning basedapplication 7 would be more difficult, but possible as well. Therefore,device 3 comprises visual data modification means 9 to pre-process the providedphoto 5 or video of the face ofperson 2 in an attempt to standardize their content prior to the analysis of deep learning basedapplication 7 as much as possible. The pre-processing ofphoto 5 may include one or more of the following steps: cut-out the background behind the face from the visual data; cut-out the ears of the person's face to reduce the visual influence of e.g. earrings; cut-out clothes and other wardrobe that might influence with the face; cut-out the head hair of theperson 2. Of course ifperson 2 onphoto 5 does look to one side, then it will not be possible to pre-process thisphoto 5 into astandardized photo 5 like the one shown inFIG. 1 . The visual data ofphoto 5 are sent frommobile phone 6 todevice 3. - In a
fourth step person 2 uses input means 10 to input at least one characteristic of her face she would like to improve, which selected characteristic is transferred as face characteristics data to deep learning basedapplication 7 ofdevice 3. In another embodiment,person 2 uses input means of the App ofmobile phone 6 as shown inFIG. 4 where a button is realized for each characteristic a person may select. According to above explainedexample person 2 selects “competent” as characteristic, which is transmitted via broadband network to deep learning basedapplication 7 ofdevice 3. In another embodiment of theinvention person 2 may use another way to input the at least one characteristic withmobile phone 6. This may be done by the selection of the App used byperson 2, as there may be an App to select the characteristic “attractive” and another App to select the characteristic “competent”. - In a fifth step of the method the deep learning based
application 7 ofdevice 3 analyses visual data ofphoto 5 ofperson 2. As a result, learning basedapplication 7 rates the characteristics ofperson 2 based onphoto 5 and in particular evaluates and rates the characteristic “competent” ofperson 2. - To achieve that technical step, the deep learning based
application 7, based ondatabase 8, extracts face properties ofphoto 5 by dividing the face ofperson 2 inphoto 5 in regions like the chin and the jawline. Ifphoto 5 sent frommobile phone 6 is anon-standardized photo 5, then deep learning basedapplication 7 is used to fill missing visual information or to estimate how the face on thisphoto 5 would look like if taken in a standardized way to enable a more or less standardized rating of the characteristics. As a result of thisanalysis device 3scores 5 points for the characteristic “competent” with a scale from 0 to 7 points as can be seen inFIG. 3 . -
FIG. 5 shows a table with examples of invasive and/or minimally invasive cosmetic and medical treatments to achieve changes of a desired characteristic of a person's face, which table is based on clinical studies. Such data are stored in amodification data base 11 ofdevice 3. With some or a combination of these treatments the desired characteristic may be improved by 1 or more points in the rating when processing a real treatment of a real person. But the inventive concept is not limited by achievements of real treatments and furthermore stores data inmodification data base 11 that provide an improvement of a characteristic, but in a way a real person would not be treated. So in one example the distance between the eyes would need to be increased significantly to improve one characteristic, which would not be an option in real life. Deep learning basedapplication 7 based on data stored inmodification data base 11 is built to provide not only one, but several data sets ofmodifications 12 which indicate areas in the face of the visual ofperson 2 which need to be modified by visual data modification means 9 to improve the desired selected characteristic in different ways and with different scores of improvement.FIG. 6 shows a line drawing of the face ofperson 2 with a first data set ofmodifications 12 how to improve the characteristic “competent” based on invasive and/or minimally invasive cosmetic and medical treatments listed inFIG. 5 . Deep learning basedapplication 7 based on further data inmodification data base 11 provides a second data set ofmodifications 12 how to improve the characteristic “competent” based on modifications that would not be possible with real persons, but provide a very good improvement of the desired selected characteristic and still look realistic in a way thatperson 2 still is recognized as theperson 2 she is. In this way, deep learning basedapplication 7 may provide several sets ofmodifications 12, some with subtle improvements, which will be only recognized subconsciously, and others with recognizable improvements and some with dramatic improvements of the score of the selected characteristic. - In a sixth step of the method visual data modification means 9 modify the
photo 5 ofperson 2 with each of the two or more data sets ofmodifications 12 and generate two or more computer-modifiedphotos 13 of the face ofperson 2. Data sets ofmodifications 12 technically are code instructions that are needed to modify the photo like in a software program PhotoShop®.FIG. 7 as another example shows the face ofperson 2 with another data set ofmodifications 12 how to improve the characteristic “dominant” based on invasive and/or minimally invasive cosmetic and medical treatments. - In a seventh step of the method a
display 14 ofdevice 3 is used to showperson 2 all computer-modifiedphotos 13 to enable her to select her desired modified facial appearance in thevideo conference 4 to come. She therefore may select between stronger improvements of the selected characteristic “competent” and less strong improvements of this characteristic and in an eighth step of the method decides which of the displayed computer-modifiedphotos 13 is the desired face and the desired selected set ofmodifications 12 to be used. - In a ninth step of the method the live video conference is processed and
person 2 usescamera 15 to generate avideo 17 that shows her face during thevideo conference 4 as shown inFIG. 2 . Visual data modification means 9 receive thelive video 17 and identify anarea 16 in thevideo 17 that shows the face ofperson 2. Based on the desired selected data set ofmodifications 12 deep learning basedapplication 7 uses visual data modification means 9 to continuously generate the computer-modified visual of the desired selected face ofperson 2. This means that even during all the movements of the face ofperson 2 and allfacial expressions person 2 does during thevideo conference 4, her visual is continuously computer-modified to make her look more competent. This means that continuously her chin is modified to appear less wide and the cheeks less full and her eyebrows are lowered as shown inFIG. 6 . Visual data modification means 9 then replace the identifiedarea 16 in thevideo 17 from thecamera 15 with the computer-modified visual ofperson 2 and send this modified video with the computer-modified visual ofperson 2 as video stream for thevideo conference 4. As a result,person 2 invideo conference 4 appears with a visual that provides a first impression to the other participants of thevideo conference 4 of a very competent person. This gives person 2 a strong advantage in negotiations during thevideo conference 4. - In a preferred embodiment
modification data base 11 may store data about objects a person may wear like glasses or earrings to change his/her first impression. In an even preferredembodiment person 2 upfront uploads visuals of his/her personal objects, so for instance a photo of her five different glasses and her ten different earrings. Deep learning basedapplication 7 based on these data stored inmodification data base 11 may generate further sets ofmodifications 12 where for instance glasses are overlaid over the visual ofperson 2 as shown inFIG. 2 . This further extends the possibilities to improve the desired characteristic in the computer-modified visual ofperson 2. - In another
embodiment person 2 may select to be shown only computer-modified visuals of her face in method step seven that are based on the table inFIG. 5 . So these computer-modifications in principle all would be possible with invasive and/or minimally invasive cosmetic and medical treatments on a real person and therefore the computer-visuals shown ondisplay 14 may look more realistic than others. - In a further embodiment, a dating App may use the method of the invention. In the profile of the dating App a man/woman may select a live chat with a person he/she would like to meet. To improve his/her chance in this live chat the person may choose to improve his characteristic “trustworthy” or “dominant” and select the extent of improvement by selecting his/her computer-modified visual. This enables a complete new and inventive way to connect potential future partners.
- In another preferred embodiment of the invention steps 2 to 8 of the method claimed in
claim 1 are processed for two or more characteristics or combinations of characteristics and sets of modifications selected for these different characteristics by the person are stored in the visual data modification means 9. The artificial intelligence is built to analyse the actual facial expressions of the face of the person in the video and to recognize content and/or tone of the person. The artificial intelligence for instance recognizes whenperson 2 gets angry or happy during the video conference. This recognition is based on visual and acoustical information provided in the video. A speech-to-text recognizer could in addition be used to enable a content analysis of the spoken words. So ifperson 2 talks about her mother being in hospital all these indications can be used to select one of the stored sets of modifications that fits this message, while another set of modification is selected a few seconds later, when person starts to negotiate the price of a product in the video conference. This means that the appropriate stored set of modifications that supports the content and/or tone of the person is dynamically selected by the artificial intelligence and used by visual data modification means to dynamically modify the visual of the face of the person in the video with the selected set of modifications. - The term video has to be interpreted in its broadest sense and covers any sequence of visual data frames, which may display objects like real persons or animated characters. As animated character an avatar for a person could for instance be displayed, which could be an animal, phantasy creature or animated human being. The term video furthermore covers any two and three-dimensional visual representation of a live or stored data content. Video conferences like webinars, live presentations or distance teaching are covered as well.
Claims (11)
1. A method to provide a video with a computer-modified visual of a desired face of a person, wherein the method comprises the following steps:
A. generating a data set of visuals of faces and extracted face property data thereof linked to face characteristics data provided by a representative set of humans that rate the visuals of these faces about their face characteristics and store the data set in a database;
B. extracting further face property data of these visuals of faces and use these extracted face property data together with the generated data set for training of an artificial intelligence to enable the artificial intelligence to provide an automated rating of the characteristics of the visuals of faces;
C. providing at least one visual of the face of the person;
D. inputting at least one characteristic of the face of the person to be improved;
E. using the artificial intelligence to analyse the visual of the person's face and to generate at least two data sets of modifications to improve the visual appearance of the person in at least two different ways towards the selected characteristic;
F. modifying the visual of the face of the person based on both data sets of modifications and generate two computer-modified visuals of the face of the person;
G. displaying both computer-modified visuals of the face of the person;
H. selecting one of the two computer-modified visuals as the desired face of the person; and
I. providing a video that shows the face of the person and use the selected data set of modifications of the selected desired face of the person to modify the video to provide the video with the computer-modified visual of the desired face of the person.
2. A method according to claim 1 , wherein the method comprises the following detailed steps for above method step I:
continuously:
identifying the visual of the face of the person in the video;
modifying the visual of the identified face of the person based on the selected data set of modifications and generate the computer-modified visual of the desired face of the person;
replacing the identified visual of the face of the person with the computer-modified visual of the desired face of the person in the video; and
displaying the video with the replaced computer-modified visual of the desired face of the person.
3. Method A method according to claim 1 , wherein above method steps A and B are processed to prepare a computer to process above method step E and that above method steps C to H are processed to select the desired face of the person and to process step I for the modification of a live video in particular of a video conference.
4. A method according to claim 1 , wherein above method step E generates only data sets of modifications to improve the visual appearance of the person within the borders of visual modifications of a face achievable by cosmetic and/or medical treatments and/or manual changes of the visual appearance of the person.
5. A method according to claim 4 , wherein above method step B furthermore comprises the following step:
generating a data set of modifications of a face achievable by cosmetic and/or medical treatments and/or manual changes of the visual appearance of the person and store the data set in the database.
6. A method according to claim 5 , wherein the method comprises the following further step:
using predefined samples of tools, like in particular glasses or earrings, to generate the data set of modifications of a face achievable by manual changes of the visual appearance of the person.
7. A method according to claim 5 , wherein the method comprises the following further step:
providing at least one visual of a personal object of the person, like in particular glasses or earrings, to generate the data set of modifications of a face achievable by manual changes of the visual appearance of the person.
8. A method according to above claim 1 , wherein method steps C to H are processed for two or more characteristics or combinations of characteristics and data sets of modifications selected for these different characteristics are stored and that the artificial intelligence is built to analyse the actual facial expressions of the face of the person in the video and to recognize content and/or tone of the person and to select the appropriate stored data set of modifications that supports the content and/or tone of the person and to dynamically modify the visual of the face of the person in the video with the selected data set of modifications.
9. A method according to claim 1 , wherein the method comprises the following further step:
pre-processing the provided visual of the face of the person to standardize their content prior to the analyse of the artificial intelligence, wherein pre-processing may include one or more of the following steps: cut-out the background behind the face from the visual; cut-out the ears of the person's face to reduce their visual influence; cut-out clothes and other wardrobe that might influence with the face; cut-out the head hair of the person.
10. A device to provide a video with a computer-modified visual of a desired face of a person, which device comprises a display and input means to input at least one characteristic of the face of a person to be changed and to provide at least one visual of the face of the person, wherein
the device comprises a deep learning based application to process the steps A and B of the method according to claim 1 to generate a database with data sets of visuals of faces and extracted face property data thereof linked to face characteristics and,
that the deep learning based application is built to process the step E of the method according to claim 1 to analyse the provided visual of the person's face and to generate at least two data sets of modifications to improve the visual appearance of the person in at least two different ways towards the selected characteristic and,
that modification means are built to process the step F of the method according to claim 1 to modify the visual of the face of the person based on both data sets of modifications and to generate two computer-modified visuals of the face of the person and,
that the device is built to display both computer-modified visuals of the face of the person and based on the persons' selection of the desired face with the input means the modification means are built to modify a video that shows the face of the person by the use of the selected data set of modifications of the selected desired face of the person to modify the video to provide a video with a computer-modified visual of a desired face of a person.
11. A computer-program comprising instructions which, when the program is executed by a device, causes the device to carry out the steps of a method claimed in claim 1 to provide a video with a computer-modified visual of a desired face of a person.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/331,195 US11341619B2 (en) | 2019-12-11 | 2021-05-26 | Method to provide a video with a computer-modified visual of a desired face of a person |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP19215139.7 | 2019-12-11 | ||
EP19215139 | 2019-12-11 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/331,195 Continuation US11341619B2 (en) | 2019-12-11 | 2021-05-26 | Method to provide a video with a computer-modified visual of a desired face of a person |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210183021A1 true US20210183021A1 (en) | 2021-06-17 |
Family
ID=68886756
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/117,748 Abandoned US20210183021A1 (en) | 2019-12-11 | 2020-12-10 | Method to provide a video with a computer-modified visual of a desired face of a person |
US17/331,195 Active US11341619B2 (en) | 2019-12-11 | 2021-05-26 | Method to provide a video with a computer-modified visual of a desired face of a person |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/331,195 Active US11341619B2 (en) | 2019-12-11 | 2021-05-26 | Method to provide a video with a computer-modified visual of a desired face of a person |
Country Status (5)
Country | Link |
---|---|
US (2) | US20210183021A1 (en) |
EP (1) | EP4073682B1 (en) |
CN (1) | CN114830193A (en) |
DE (1) | DE212020000467U1 (en) |
WO (1) | WO2021115797A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11341619B2 (en) | 2019-12-11 | 2022-05-24 | QuantiFace GmbH | Method to provide a video with a computer-modified visual of a desired face of a person |
USD953374S1 (en) * | 2020-05-15 | 2022-05-31 | Lg Electronics Inc. | Display panel with animated graphical user interface |
US11445148B1 (en) * | 2021-05-06 | 2022-09-13 | Microsoft Technology Licensing, Llc | Video teleconference curated user profile picture |
US20220295012A1 (en) * | 2021-01-26 | 2022-09-15 | Dell Products, Lp | System and method for intelligent appearance monitoring management system for videoconferencing applications |
Family Cites Families (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006254229A (en) | 2005-03-11 | 2006-09-21 | Fuji Photo Film Co Ltd | Imaging apparatus, imaging method and imaging program |
JP2010182150A (en) | 2009-02-06 | 2010-08-19 | Seiko Epson Corp | Image processing apparatus for detecting coordinate position of characteristic part of face |
US10482333B1 (en) * | 2017-01-04 | 2019-11-19 | Affectiva, Inc. | Mental state analysis using blink rate within vehicles |
US8917914B2 (en) | 2011-04-05 | 2014-12-23 | Alcorn State University | Face recognition system and method using face pattern words and face pattern bytes |
US9520072B2 (en) | 2011-09-21 | 2016-12-13 | University Of South Florida | Systems and methods for projecting images onto an object |
JP2014085796A (en) | 2012-10-23 | 2014-05-12 | Sony Corp | Information processing device and program |
WO2014068567A1 (en) | 2012-11-02 | 2014-05-08 | Itzhak Wilf | Method and system for predicting personality traits, capabilities and suggested interactions from images of a person |
WO2015017687A2 (en) | 2013-07-31 | 2015-02-05 | Cosmesys Inc. | Systems and methods for producing predictive images |
EP3123449B1 (en) | 2014-03-25 | 2020-01-01 | Apple Inc. | Method and system for representing a virtual object in a view of a real environment |
WO2015192263A1 (en) | 2014-06-16 | 2015-12-23 | Xiaoou Tang | A method and a system for face verification |
CN105844202A (en) | 2015-01-12 | 2016-08-10 | 芋头科技(杭州)有限公司 | Image recognition system and method |
US10796480B2 (en) * | 2015-08-14 | 2020-10-06 | Metail Limited | Methods of generating personalized 3D head models or 3D body models |
WO2017177259A1 (en) | 2016-04-12 | 2017-10-19 | Phi Technologies Pty Ltd | System and method for processing photographic images |
US10446189B2 (en) | 2016-12-29 | 2019-10-15 | Google Llc | Video manipulation with face replacement |
CN108664782B (en) | 2017-03-28 | 2023-09-12 | 三星电子株式会社 | Face verification method and device |
US10438415B2 (en) | 2017-04-07 | 2019-10-08 | Unveil, LLC | Systems and methods for mixed reality medical training |
US10977674B2 (en) | 2017-04-28 | 2021-04-13 | Qualtrics, Llc | Conducting digital surveys that collect and convert biometric data into survey respondent characteristics |
CN108960020A (en) | 2017-05-27 | 2018-12-07 | 富士通株式会社 | Information processing method and information processing equipment |
TW201907334A (en) | 2017-07-03 | 2019-02-16 | 華碩電腦股份有限公司 | Electronic apparatus, image processing method and non-transitory computer-readable recording medium |
US10825564B1 (en) | 2017-12-11 | 2020-11-03 | State Farm Mutual Automobile Insurance Company | Biometric characteristic application using audio/video analysis |
US10997703B1 (en) | 2018-04-24 | 2021-05-04 | Igor Khalatian | Methods and systems for automated attractiveness prediction |
US11151362B2 (en) | 2018-08-30 | 2021-10-19 | FaceValue B.V. | System and method for first impression analysis and face morphing by adjusting facial landmarks using faces scored for plural perceptive traits |
US11182963B2 (en) | 2019-04-03 | 2021-11-23 | Posnap, Inc. | Computerized system and method for providing a mobile augmented reality item display and selection experience |
US11501495B2 (en) | 2019-05-10 | 2022-11-15 | Qualcomm Incorporated | Virtual models for communications between autonomous vehicles and external observers |
US11250245B2 (en) | 2019-09-20 | 2022-02-15 | The Trustees Of Princeont University | Data-driven, photorealistic social face-trait encoding, prediction, and manipulation using deep neural networks |
US10764535B1 (en) * | 2019-10-14 | 2020-09-01 | Facebook, Inc. | Facial tracking during video calls using remote control input |
WO2021115798A1 (en) | 2019-12-11 | 2021-06-17 | QuantiFace GmbH | Method and system to provide a computer-modified visualization of the desired face of a person |
CN114830193A (en) | 2019-12-11 | 2022-07-29 | 定量面部股份有限公司 | Generating video including modified face images |
-
2020
- 2020-11-26 CN CN202080087661.6A patent/CN114830193A/en active Pending
- 2020-11-26 DE DE212020000467.6U patent/DE212020000467U1/en active Active
- 2020-11-26 EP EP20811626.9A patent/EP4073682B1/en active Active
- 2020-11-26 WO PCT/EP2020/083540 patent/WO2021115797A1/en active Search and Examination
- 2020-12-10 US US17/117,748 patent/US20210183021A1/en not_active Abandoned
-
2021
- 2021-05-26 US US17/331,195 patent/US11341619B2/en active Active
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11341619B2 (en) | 2019-12-11 | 2022-05-24 | QuantiFace GmbH | Method to provide a video with a computer-modified visual of a desired face of a person |
USD953374S1 (en) * | 2020-05-15 | 2022-05-31 | Lg Electronics Inc. | Display panel with animated graphical user interface |
US20220295012A1 (en) * | 2021-01-26 | 2022-09-15 | Dell Products, Lp | System and method for intelligent appearance monitoring management system for videoconferencing applications |
US11778142B2 (en) * | 2021-01-26 | 2023-10-03 | Dell Products, Lp | System and method for intelligent appearance monitoring management system for videoconferencing applications |
US11445148B1 (en) * | 2021-05-06 | 2022-09-13 | Microsoft Technology Licensing, Llc | Video teleconference curated user profile picture |
Also Published As
Publication number | Publication date |
---|---|
DE212020000467U1 (en) | 2022-03-09 |
US20210279844A1 (en) | 2021-09-09 |
EP4073682B1 (en) | 2023-11-08 |
EP4073682A1 (en) | 2022-10-19 |
US11341619B2 (en) | 2022-05-24 |
CN114830193A (en) | 2022-07-29 |
EP4073682C0 (en) | 2023-11-08 |
WO2021115797A1 (en) | 2021-06-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11341619B2 (en) | Method to provide a video with a computer-modified visual of a desired face of a person | |
US10528801B2 (en) | Method and system for incorporating contextual and emotional visualization into electronic communications | |
CN111145282B (en) | Avatar composition method, apparatus, electronic device, and storage medium | |
US20160134840A1 (en) | Avatar-Mediated Telepresence Systems with Enhanced Filtering | |
JP6259808B2 (en) | Modifying the appearance of participants during a video conference | |
KR101190686B1 (en) | Image processing apparatus, image processing method, and computer readable recording medium | |
US20190197755A1 (en) | Producing realistic talking Face with Expression using Images text and voice | |
Le et al. | Live speech driven head-and-eye motion generators | |
CN110418095B (en) | Virtual scene processing method and device, electronic equipment and storage medium | |
CN113508369A (en) | Communication support system, communication support method, communication support program, and image control program | |
CN110210449B (en) | Face recognition system and method for making friends in virtual reality | |
KR102351008B1 (en) | Apparatus and method for recognizing emotions | |
KR20220127471A (en) | Device, Method and program for providing psychological counseling using VR | |
CN111523981A (en) | Virtual trial method and device, electronic equipment and storage medium | |
JP6796762B1 (en) | Virtual person dialogue system, video generation method, video generation program | |
US11227424B2 (en) | Method and system to provide a computer-modified visualization of the desired face of a person | |
WO2022024956A1 (en) | Emotion analysis system and emotion analysis device | |
WO2022025024A1 (en) | Reaction analysis system and reaction analysis device | |
CN115499613A (en) | Video call method and device, electronic equipment and storage medium | |
CN108399358A (en) | A kind of expression display methods and system in Video chat | |
Seib et al. | Enhancing human-robot interaction by a robot face with facial expressions and synchronized lip movements | |
EP4170609A1 (en) | Automated filter selection for altering a stream | |
JP2019212325A (en) | Information processing device, mirror device, and program | |
WO2022025025A1 (en) | Emotion analysis system and emotion analysis device | |
JP2021086618A (en) | Virtual person interaction system, video generation method, and video generation program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
AS | Assignment |
Owner name: QUANTIFACE GMBH, AUSTRIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BENDITTE-KLEPETKO, HEIKE;REEL/FRAME:057280/0975 Effective date: 20201221 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |