CN110837294B - Facial expression control method and system based on eyeball tracking - Google Patents
Facial expression control method and system based on eyeball tracking Download PDFInfo
- Publication number
- CN110837294B CN110837294B CN201910970941.0A CN201910970941A CN110837294B CN 110837294 B CN110837294 B CN 110837294B CN 201910970941 A CN201910970941 A CN 201910970941A CN 110837294 B CN110837294 B CN 110837294B
- Authority
- CN
- China
- Prior art keywords
- user
- eyeball
- state information
- emotion
- game
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 210000005252 bulbus oculi Anatomy 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000008921 facial expression Effects 0.000 title claims abstract description 12
- 230000008451 emotion Effects 0.000 claims abstract description 45
- 230000014509 gene expression Effects 0.000 claims abstract description 40
- 210000001508 eye Anatomy 0.000 claims abstract description 29
- 230000008859 change Effects 0.000 claims abstract description 20
- 238000010801 machine learning Methods 0.000 claims abstract description 15
- 210000001747 pupil Anatomy 0.000 claims abstract description 14
- 230000009471 action Effects 0.000 claims abstract description 8
- 230000004397 blinking Effects 0.000 claims abstract description 5
- 230000003068 static effect Effects 0.000 claims description 18
- 238000012549 training Methods 0.000 claims description 15
- 230000004044 response Effects 0.000 claims description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000004590 computer program Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 239000011521 glass Substances 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 230000004424 eye movement Effects 0.000 description 2
- 230000004418 eye rotation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/20—Input arrangements for video game devices
- A63F13/21—Input arrangements for video game devices characterised by their sensors, purposes or types
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/40—Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/50—Controlling the output signals based on the game progress
- A63F13/52—Controlling the output signals based on the game progress involving aspects of the displayed game scene
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F13/00—Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F13/80—Special adaptations for executing a specific game genre or game mode
- A63F13/822—Strategy games; Role-playing games
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- A—HUMAN NECESSITIES
- A63—SPORTS; GAMES; AMUSEMENTS
- A63F—CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- A63F2300/00—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
- A63F2300/80—Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
- A63F2300/807—Role playing or strategy games
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The technical scheme of the invention comprises a facial expression control method and a system based on eyeball tracking, which are used for realizing the following steps: the method comprises the steps that a user uses terminal equipment to run a game program; the game program calls an image acquisition device of the terminal equipment to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the equipment, a gazing direction, a blinking action, eye pupil size change and iris stretching or shrinking state; according to the obtained eyeball state information, invoking a machine learning algorithm to calculate the current emotion attribute of the user; and calling the corresponding expression from the expression library according to the current emotion attribute of the client and displaying the corresponding expression by the virtual character appointed in the game according to a preset rule. The beneficial effects of the invention are as follows: the game experience is improved, the game reality is enhanced, the user viscosity and the game liveness are improved, more substituted feelings are provided for players, and the dependence and the loyalty of the players on the game can be improved.
Description
Technical Field
The invention relates to a facial expression control method and system based on eyeball tracking, and belongs to the technical field of computers.
Background
Character models of RPG games (role playing games) now mostly have realistic anthropomorphic representations. Unfortunately, the eye of the character in the game is relatively dull. When the player looks at the game character, the player does not watch the view angle of the player, and the player cannot generate sight interaction, so that the game experience is seriously influenced, and the game experience is not true enough for the user, so that the viscosity of the user is reduced, and the game liveness is reduced.
Eye rotation analysis personality, eye rotation often shows that mental activities of people have been summarized earlier, such as light emission from both eyes during excitation, no light from both eyes during depression, no spirit from pupils during sadness, open eyes during anger, and the like. It is considered that the eyeball lies on the left upper side and thinks on the right upper side. The eyeball is proved to have a certain effect on judging the personality and emotion of the person.
Disclosure of Invention
In order to solve the above problems, an object of the present invention is to provide a facial expression control method and system based on eye tracking, including a user running a game program using a terminal device; the game program calls an image acquisition device of the terminal equipment to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the equipment, a gazing direction, a blinking action, eye pupil size change and iris stretching or shrinking state; according to the obtained eyeball state information, invoking a machine learning algorithm to calculate the current emotion attribute of the user; and calling the corresponding expression from the expression library according to the current emotion attribute of the client and displaying the corresponding expression by the virtual character appointed in the game according to a preset rule.
The invention solves the problems by adopting the technical scheme that: a facial expression control method based on eye tracking, comprising the steps of: s100, a user uses terminal equipment to run a game program; s200, the game program calls an image acquisition device of the terminal equipment to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the equipment, a gazing direction, blink actions, eye pupil size change and iris stretching or shrinking states; s300, according to the acquired eyeball state information, invoking a machine learning algorithm to calculate the current emotion attribute of the user; s400, according to the current emotion attributes of the clients, calling corresponding expressions from the expression library according to preset rules and displaying the expressions by the virtual roles appointed in the game.
Further, the S200 further includes: s210, collecting a certain number of user eyeball static images in a specified time period, wherein the specified time period and the certain number can be customized; s220, selecting a certain number of user eyeball static images for comparison, and acquiring a change state and a change track of the user eyeballs in a specified time according to a comparison result.
Further, the step S220 further includes: s221, selecting two user eyeball static images adjacent in time, and comparing to obtain eyeball difference; s222, calculating the distance between the user glasses and the equipment according to the difference and the fixed position of the corresponding time node image acquisition device on the terminal equipment.
Further, the S300 further includes: s310, acquiring big data information of eyeball state information and corresponding emotion connection; s320, training a machine learning algorithm according to the big data information to obtain a trained machine emotion algorithm; s330, taking the acquired eyeball state information of the user as input, and outputting user emotion information in a certain period of time after the machine emotion algorithm performs corresponding training.
Further, the S400 further includes: s410, based on a response algorithm model, taking the emotion of a user as input, and outputting the corresponding changed back view angle, pupil size and eye contour form data of the eyes of the virtual character; s420, calling the corresponding expression from the expression library and displaying the corresponding expression by the virtual character appointed in the game.
The invention solves the problems by adopting the technical scheme that: a facial expression control system based on eye tracking, comprising: the execution module is used for running the game program at the terminal equipment; the device calling module is used for calling an image acquisition device of the terminal device to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the device, a gazing direction, a blinking action, eye pupil size change and iris stretching or shrinking state; the machine algorithm execution module is used for calling a machine learning algorithm to calculate the current emotion attribute of the user according to the acquired eyeball state information; and the expression calling module is used for calling the corresponding expression from the expression library according to the current emotion attribute of the client and displaying the corresponding expression by the virtual character appointed in the game according to the preset rule.
Further, the image acquisition device includes, but is not limited to, a camera and an infrared projection acquisition device.
Further, the device calling module further includes: the setting unit is used for setting the acquisition time period and the acquisition frequency of the image acquisition device, and acquiring a certain number of user eyeball static images in a specified time period, wherein the specified time period and the certain number can be customized; the track generation unit is used for selecting a certain number of user eyeball static images to compare, and acquiring the change state and the change track of the user eyeballs in the appointed time according to the comparison result.
Further, the track generating unit further includes: the comparison subunit is used for selecting two user eyeball static images adjacent in time and comparing to obtain eyeball differences; and the distance calculating subunit is used for calculating the distance between the user glasses and the equipment according to the difference and the fixed position of the corresponding time node image acquisition device on the terminal equipment.
Further, the machine algorithm execution module further includes: the big data unit is used for acquiring eyeball state information and big data information related to the corresponding emotion; the training unit is used for training the machine learning algorithm according to the big data information to obtain a trained machine emotion algorithm; the computing unit is used for taking the acquired eyeball state information of the user as input, and outputting the emotion information of the user within a certain period of time after the machine emotion algorithm performs corresponding training.
The beneficial effects of the invention are as follows: the game experience is improved, the game reality is enhanced, the user viscosity and the game liveness are improved, more substituted feelings are provided for players, and the dependence and the loyalty of the players on the game can be improved.
Drawings
FIG. 1 is a schematic flow diagram of a method according to a preferred embodiment of the present invention;
fig. 2 is a schematic diagram of a system configuration according to a preferred embodiment of the present invention.
Detailed Description
The conception, specific structure, and technical effects produced by the present invention will be clearly and completely described below with reference to the embodiments and the drawings to fully understand the objects, aspects, and effects of the present invention.
It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly or indirectly fixed or connected to the other feature. Further, the descriptions of the upper, lower, left, right, etc. used in this disclosure are merely with respect to the mutual positional relationship of the various components of this disclosure in the drawings. As used in this disclosure, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the description presented herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and/or" as used herein includes any combination of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element of the same type from another. For example, a first element could also be termed a second element, and, similarly, a second element could also be termed a first element, without departing from the scope of the present disclosure. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed.
According to the report of the united states well-known scientific website, mashable 2018, 7 months and 30 months, an international research team consisting of researchers at universities of stuttgart, friendss, australia and the like, in germany, used the most advanced machine learning algorithm to find the relationship between character and eye movement of people.
The following are some personality characteristics related to eye movement developed by researchers through artificial intelligence techniques:
(1) Curiosity: the eyes are more surrounding;
(2) Open heart state: people with open ideas spend longer time gazing at abstract patterns;
(3) The nerve mass: the blink speed is faster;
(4) Admission of new experience: the eyeball moves more from side to side;
(5) Responsibility center: the pupil size of a person with a high responsibility will fluctuate more;
(6) Optimism: the time to focus on passive emotional content is shorter than pessimistic people.
Referring to fig. 1, there is a schematic flow diagram of a method according to a preferred embodiment of the invention,
s100, a user uses terminal equipment to run a game program;
s200, the game program calls an image acquisition device of the terminal equipment to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the equipment, a gazing direction, blink actions, eye pupil size change and iris stretching or shrinking states;
s300, according to the acquired eyeball state information, invoking a machine learning algorithm to calculate the current emotion attribute of the user;
s400, according to the current emotion attributes of the clients, calling corresponding expressions from the expression library according to preset rules and displaying the expressions by the virtual roles appointed in the game.
S200 further includes: s210, collecting a certain number of user eyeball static images in a specified time period, wherein the specified time period and the certain number can be customized; s220, selecting a certain number of user eyeball static images for comparison, and acquiring a change state and a change track of the user eyeballs in a specified time according to a comparison result.
S220 further includes: s221, selecting two user eyeball static images adjacent in time, and comparing to obtain eyeball difference; s222, calculating the distance between the user glasses and the equipment according to the difference and the fixed position of the corresponding time node image acquisition device on the terminal equipment.
S300 further includes: s310, acquiring big data information of eyeball state information and corresponding emotion connection; s320, training a machine learning algorithm according to the big data information to obtain a trained machine emotion algorithm; s330, taking the acquired eyeball state information of the user as input, and outputting user emotion information in a certain period of time after the machine emotion algorithm performs corresponding training.
S400 further includes: s410, based on a response algorithm model, taking the emotion of a user as input, and outputting the corresponding changed back view angle, pupil size and eye contour form data of the eyes of the virtual character; s420, calling the corresponding expression from the expression library and displaying the corresponding expression by the virtual character appointed in the game.
Referring to fig. 2, there is a schematic diagram of a system architecture according to a preferred embodiment of the present invention,
comprising the following steps: the execution module is used for running the game program at the terminal equipment; the device calling module is used for calling an image acquisition device of the terminal device to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the device, a gazing direction, a blinking action, eye pupil size change and iris stretching or shrinking state; the machine algorithm execution module is used for calling a machine learning algorithm to calculate the current emotion attribute of the user according to the acquired eyeball state information; and the expression calling module is used for calling the corresponding expression from the expression library according to the current emotion attribute of the client and displaying the corresponding expression by the virtual character appointed in the game according to the preset rule.
Image capture devices include, but are not limited to, cameras and infrared projection capture devices.
The device call module further includes: the setting unit is used for setting the acquisition time period and the acquisition frequency of the image acquisition device, and acquiring a certain number of user eyeball static images in a specified time period, wherein the specified time period and the certain number can be customized; the track generation unit is used for selecting a certain number of user eyeball static images to compare, and acquiring the change state and the change track of the user eyeballs in the appointed time according to the comparison result.
The track generation unit further includes: the comparison subunit is used for selecting two user eyeball static images adjacent in time and comparing to obtain eyeball differences; and the distance calculating subunit is used for calculating the distance between the user glasses and the equipment according to the difference and the fixed position of the corresponding time node image acquisition device on the terminal equipment.
The machine algorithm execution module further includes: the big data unit is used for acquiring eyeball state information and big data information related to the corresponding emotion; the training unit is used for training the machine learning algorithm according to the big data information to obtain a trained machine emotion algorithm; the computing unit is used for taking the acquired eyeball state information of the user as input, and outputting the emotion information of the user within a certain period of time after the machine emotion algorithm performs corresponding training.
It should be appreciated that embodiments of the invention may be implemented or realized by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer readable storage medium configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, in accordance with the methods and drawings described in the specific embodiments. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes (or variations and/or combinations thereof) described herein may be performed under control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications), by hardware, or combinations thereof, collectively executing on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable computing platform, including, but not limited to, a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and so forth. Aspects of the invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and/or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, which when read by a computer, is operable to configure and operate the computer to perform the processes described herein. Further, the machine readable code, or portions thereof, may be transmitted over a wired or wireless network. When such media includes instructions or programs that, in conjunction with a microprocessor or other data processor, implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media. The invention also includes the computer itself when programmed according to the methods and techniques of the present invention.
The computer program can be applied to the input data to perform the functions described herein, thereby converting the input data to generate output data that is stored to the non-volatile memory. The output information may also be applied to one or more output devices such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on a display.
The present invention is not limited to the above embodiments, but can be modified, equivalent, improved, etc. by the same means to achieve the technical effects of the present invention, which are included in the spirit and principle of the present invention. Various modifications and variations are possible in the technical solution and/or in the embodiments within the scope of the invention.
Claims (5)
1. A facial expression control method based on eye tracking, comprising the steps of:
s100, a user uses terminal equipment to run a game program;
s200, the game program calls an image acquisition device of the terminal equipment to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the equipment, a gazing direction, blink actions, eye pupil size change and iris stretching or shrinking states;
s300, according to the acquired eyeball state information, invoking a machine learning algorithm to calculate the current emotion attribute of the user;
s400, according to the current emotion attribute of the user, calling the corresponding expression from the expression library according to a preset rule and displaying the corresponding expression by the virtual character appointed in the game;
the S200 further includes:
s210, collecting a certain number of user eyeball static images in a specified time period, wherein the specified time period and the certain number can be customized;
s220, selecting a certain number of user eyeball static images for comparison, and acquiring a change state and a change track of the user eyeballs in a specified time according to a comparison result;
the S220 further includes:
s221, selecting two user eyeball static images adjacent in time, and comparing to obtain eyeball difference;
s222, calculating the distance between eyes of a user and equipment according to the difference and the fixed position of the corresponding time node image acquisition device on the terminal equipment;
the S400 further includes:
s410, based on a response algorithm model, taking the emotion of a user as input, and outputting the corresponding changed back view angle, pupil size and eye contour form data of the eyes of the virtual character;
s420, calling the corresponding expression from the expression library and displaying the corresponding expression by the virtual character appointed in the game.
2. The eye-tracking-based facial expression control method according to claim 1, wherein S300 further comprises:
s310, acquiring big data information of eyeball state information and corresponding emotion connection;
s320, training a machine learning algorithm according to the big data information to obtain a trained machine emotion algorithm;
s330, taking the acquired eyeball state information of the user as input, and outputting user emotion information in a certain period of time after the machine emotion algorithm performs corresponding training.
3. A facial expression control system based on eye tracking, comprising:
the execution module is used for running the game program at the terminal equipment;
the device calling module is used for calling an image acquisition device of the terminal device to acquire eyeball state information of a user, wherein the eyeball state information comprises an angle between an eyeball and the device, a gazing direction, a blinking action, eye pupil size change and iris stretching or shrinking state;
the machine algorithm execution module is used for calling a machine learning algorithm to calculate the current emotion attribute of the user according to the acquired eyeball state information;
the expression calling module is used for calling the corresponding expression from the expression library according to the current emotion attribute of the client and displaying the corresponding expression by the virtual character appointed in the game according to the preset rule;
the device call module further includes:
the setting unit is used for setting the acquisition time period and the acquisition frequency of the image acquisition device, and acquiring a certain number of user eyeball static images in a specified time period, wherein the specified time period and the certain number can be customized;
the track generation unit is used for selecting a certain number of user eyeball static images for comparison, and acquiring the change state and change track of the user eyeballs in the appointed time according to the comparison result;
the trajectory generation unit further includes:
the comparison subunit is used for selecting two user eyeball static images adjacent in time and comparing to obtain eyeball differences;
a distance calculating subunit, configured to calculate a distance between an eye of a user and the device according to the difference in combination with a fixed position of the corresponding time node image acquisition device on the terminal device;
the expression invoking module is also used for outputting the corresponding changed back view angle, pupil size and eye contour form data of the eyes of the virtual character based on the response algorithm model by taking the emotion of the user as input; and calling the corresponding expression from the expression library and displaying the corresponding expression by the virtual character appointed in the game.
4. A facial expression control system based on eye tracking according to claim 3, wherein said image capture device includes, but is not limited to, a camera and an infrared projection capture device.
5. The eye-tracking based facial expression control system of claim 3, wherein the machine algorithm execution module further comprises:
the big data unit is used for acquiring eyeball state information and big data information related to the corresponding emotion;
the training unit is used for training the machine learning algorithm according to the big data information to obtain a trained machine emotion algorithm;
the computing unit is used for taking the acquired eyeball state information of the user as input, and outputting the emotion information of the user within a certain period of time after the machine emotion algorithm performs corresponding training.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910970941.0A CN110837294B (en) | 2019-10-14 | 2019-10-14 | Facial expression control method and system based on eyeball tracking |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910970941.0A CN110837294B (en) | 2019-10-14 | 2019-10-14 | Facial expression control method and system based on eyeball tracking |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110837294A CN110837294A (en) | 2020-02-25 |
| CN110837294B true CN110837294B (en) | 2023-12-12 |
Family
ID=69575401
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910970941.0A Active CN110837294B (en) | 2019-10-14 | 2019-10-14 | Facial expression control method and system based on eyeball tracking |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110837294B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352507A (en) * | 2020-02-27 | 2020-06-30 | 维沃移动通信有限公司 | Information prompting method and electronic equipment |
| CN111632367A (en) * | 2020-05-18 | 2020-09-08 | 歌尔科技有限公司 | Hand-trip system based on visual guidance and hand-trip response method |
| WO2022183424A1 (en) * | 2021-03-04 | 2022-09-09 | 深圳技术大学 | Emotion recognition-based online social method and apparatus, and storage medium |
| CN114187647A (en) * | 2021-12-10 | 2022-03-15 | 深圳爱酷智能科技有限公司 | Drug-taking detection method, device, equipment and storage medium |
| CN114296548B (en) * | 2021-12-14 | 2023-03-24 | 杭州朱道实业有限公司 | Intelligent movement identification information system for exhibition |
| CN116912748A (en) * | 2023-09-13 | 2023-10-20 | 江西工业贸易职业技术学院 | Event view tracking method, system, readable storage medium and computer |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5169319A (en) * | 1990-07-30 | 1992-12-08 | John Potocki | Method for improving a person's skill for playing an interactive video game requiring eye-hand coordination and operation of manual activation means |
| CN101393599A (en) * | 2007-09-19 | 2009-03-25 | 中国科学院自动化研究所 | Game role control method based on human face expression |
| CN105573613A (en) * | 2015-06-24 | 2016-05-11 | 宇龙计算机通信科技(深圳)有限公司 | Program icon sorting method and apparatus |
| CN105797375A (en) * | 2014-12-31 | 2016-07-27 | 深圳市亿思达科技集团有限公司 | Method and terminal for changing role model expressions along with user facial expressions |
| CN106537290A (en) * | 2014-05-09 | 2017-03-22 | 谷歌公司 | Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects |
| CN107589829A (en) * | 2016-07-07 | 2018-01-16 | 迪斯尼实业公司 | Location-based experience to interactive commodity |
| CN108742657A (en) * | 2018-03-30 | 2018-11-06 | 百度在线网络技术（北京）有限公司 | Game assessment method, device, games system and storage medium |
| CN108905204A (en) * | 2018-07-24 | 2018-11-30 | 合肥爱玩动漫有限公司 | A kind of exchange method for immersion virtual game |
| CN109376621A (en) * | 2018-09-30 | 2019-02-22 | 北京七鑫易维信息技术有限公司 | A kind of sample data generation method, device and robot |
| CN109431761A (en) * | 2018-08-31 | 2019-03-08 | 深圳众赢维融科技有限公司 | Method, apparatus, electronic equipment and the storage medium of asthenopia relief |
| CN109621418A (en) * | 2018-12-03 | 2019-04-16 | 网易（杭州）网络有限公司 | The expression adjustment and production method, device of virtual role in a kind of game |
| CN109670385A (en) * | 2017-10-16 | 2019-04-23 | 腾讯科技（深圳）有限公司 | The method and device that expression updates in a kind of application program |
| CN109683709A (en) * | 2018-12-17 | 2019-04-26 | 苏州思必驰信息科技有限公司 | Man-machine interaction method and system based on Emotion identification |
| CN109725418A (en) * | 2017-10-30 | 2019-05-07 | 华为技术有限公司 | Display equipment, the method and device presented for adjusting the image of display equipment |
| CN109844735A (en) * | 2016-07-21 | 2019-06-04 | 奇跃公司 | Affective state for using user controls the technology that virtual image generates system |
| CN109857650A (en) * | 2019-01-14 | 2019-06-07 | 珠海金山网络游戏科技有限公司 | A kind of game performance monitor method and system |
| CN110249337A (en) * | 2017-05-01 | 2019-09-17 | 谷歌有限责任公司 | Using eye tracks camera to facial expression classification |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120190446A1 (en) * | 2011-01-21 | 2012-07-26 | Rogers Henk B | Systems and methods for providing a game or story development platform |
| CN110945405B (en) * | 2017-05-31 | 2024-01-02 | 奇跃公司 | Eye tracking calibration technique |
-
2019
- 2019-10-14 CN CN201910970941.0A patent/CN110837294B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5169319A (en) * | 1990-07-30 | 1992-12-08 | John Potocki | Method for improving a person's skill for playing an interactive video game requiring eye-hand coordination and operation of manual activation means |
| CN101393599A (en) * | 2007-09-19 | 2009-03-25 | 中国科学院自动化研究所 | Game role control method based on human face expression |
| CN106537290A (en) * | 2014-05-09 | 2017-03-22 | 谷歌公司 | Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects |
| CN105797375A (en) * | 2014-12-31 | 2016-07-27 | 深圳市亿思达科技集团有限公司 | Method and terminal for changing role model expressions along with user facial expressions |
| CN105573613A (en) * | 2015-06-24 | 2016-05-11 | 宇龙计算机通信科技(深圳)有限公司 | Program icon sorting method and apparatus |
| CN107589829A (en) * | 2016-07-07 | 2018-01-16 | 迪斯尼实业公司 | Location-based experience to interactive commodity |
| CN109844735A (en) * | 2016-07-21 | 2019-06-04 | 奇跃公司 | Affective state for using user controls the technology that virtual image generates system |
| CN110249337A (en) * | 2017-05-01 | 2019-09-17 | 谷歌有限责任公司 | Using eye tracks camera to facial expression classification |
| CN109670385A (en) * | 2017-10-16 | 2019-04-23 | 腾讯科技（深圳）有限公司 | The method and device that expression updates in a kind of application program |
| CN109725418A (en) * | 2017-10-30 | 2019-05-07 | 华为技术有限公司 | Display equipment, the method and device presented for adjusting the image of display equipment |
| CN108742657A (en) * | 2018-03-30 | 2018-11-06 | 百度在线网络技术（北京）有限公司 | Game assessment method, device, games system and storage medium |
| CN108905204A (en) * | 2018-07-24 | 2018-11-30 | 合肥爱玩动漫有限公司 | A kind of exchange method for immersion virtual game |
| CN109431761A (en) * | 2018-08-31 | 2019-03-08 | 深圳众赢维融科技有限公司 | Method, apparatus, electronic equipment and the storage medium of asthenopia relief |
| CN109376621A (en) * | 2018-09-30 | 2019-02-22 | 北京七鑫易维信息技术有限公司 | A kind of sample data generation method, device and robot |
| CN109621418A (en) * | 2018-12-03 | 2019-04-16 | 网易（杭州）网络有限公司 | The expression adjustment and production method, device of virtual role in a kind of game |
| CN109683709A (en) * | 2018-12-17 | 2019-04-26 | 苏州思必驰信息科技有限公司 | Man-machine interaction method and system based on Emotion identification |
| CN109857650A (en) * | 2019-01-14 | 2019-06-07 | 珠海金山网络游戏科技有限公司 | A kind of game performance monitor method and system |
Non-Patent Citations (2)
| Title |
|---|
| 基于玩家体验的电脑游戏界面情感化设计研究;郭淳;居全伟;;黑龙江科学(第06期);全文 * |
| 视觉交互游戏所隐含的艺术语境变革探析;刘春雷;林熠轩;;现代商贸工业(第19期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110837294A (en) | 2020-02-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110837294B (en) | Facial expression control method and system based on eyeball tracking | |
| US12054227B2 (en) | Matching meshes for virtual avatars | |
| JP7253017B2 (en) | AUGMENTED REALITY SYSTEM AND METHOD USING REFLECTION | |
| US11348300B2 (en) | Avatar customization for optimal gaze discrimination | |
| US11763510B2 (en) | Avatar animation using markov decision process policies | |
| US11107262B2 (en) | Avatar facial expression representation in multidimensional space | |
| EP3381175B1 (en) | Apparatus and method for operating personal agent | |
| US11669726B2 (en) | Methods and systems for interpolation of disparate inputs | |
| US11328467B2 (en) | Pose space dimensionality reduction for pose space deformation of a virtual character | |
| Sundstedt | Gazing at games: An introduction to eye tracking control | |
| AU2007342471B2 (en) | Situated simulation for training, education, and therapy | |
| US20220076474A1 (en) | Computer generated hair groom transfer tool | |
| Korn et al. | Assistive systems for the workplace: Towards context-aware assistance | |
| US11276219B2 (en) | Systems and methods for cross-application authoring, transfer, and evaluation of rigging control systems for virtual characters | |
| Dobre et al. | Immersive machine learning for social attitude detection in virtual reality narrative games | |
| Burelli et al. | Adapting virtual camera behaviour through player modelling | |
| Cláudio et al. | Virtual environment to treat social anxiety | |
| US20240351666A1 (en) | Matching meshes for virtual avatars | |
| Ventrella | Virtual gaze: the communicative energy between avatar faces | |
| Negri et al. | Proposal of Serious Games for a Socially Assistive Robot Based on Eye Contact and Visual Attention of Children with Autism Spectrum Disorder | |
| Guerrini | Sviluppo di uno scenario di realtà virtuale adattivo per il trattamento di fobie | |
| Adamo-Villani et al. | Emoes: Eye motion and ocular expression simulator | |
| Lohman | Reducing Cybersickness in VR on an Omnidirectional Treadmill | |
| CN118160008A (en) | Inferred skeletal structure of a utility 3D asset | |
| Konstenius | Enabling non-intrusive gaze interactions in existing computer games |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |