WO2023116145A1 - 表情模型确定方法、装置、设备及计算机可读存储介质 - Google Patents

表情模型确定方法、装置、设备及计算机可读存储介质 Download PDF

Info

Publication number
WO2023116145A1
WO2023116145A1 PCT/CN2022/125570 CN2022125570W WO2023116145A1 WO 2023116145 A1 WO2023116145 A1 WO 2023116145A1 CN 2022125570 W CN2022125570 W CN 2022125570W WO 2023116145 A1 WO2023116145 A1 WO 2023116145A1
Authority
WO
WIPO (PCT)
Prior art keywords
expression
model
feature information
adjusted
expression model
Prior art date
Application number
PCT/CN2022/125570
Other languages
English (en)
French (fr)
Inventor
韩晨
Original Assignee
北京字跳网络技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京字跳网络技术有限公司 filed Critical 北京字跳网络技术有限公司
Publication of WO2023116145A1 publication Critical patent/WO2023116145A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment

Definitions

  • the present application belongs to the technical field of virtual reality, and in particular relates to a method, device, equipment and computer-readable storage medium for determining an expression model.
  • VR devices can simulate the virtual world based on the calculation of data, and provide users with simulations of senses such as vision, hearing, and touch.
  • the display of characters is generally based on static models, although it can support users.
  • Self-image setting function but the image after the general setting is completed is a fixed image, which cannot be changed in the scene. Even if it involves the display of dynamic images in the scene, it still needs to rely on preset actions and expressions, and can only be displayed in certain plots in the scene, or rely on the user to use buttons to trigger the display, and the user experience is poor.
  • the embodiment of the present application provides an implementation solution different from the prior art, so as to solve the technical problem of poor user experience caused by the image determination method in the prior art.
  • the embodiment of the present application provides a method for determining an expression model, including:
  • the embodiment of the present application provides a data processing method, including:
  • an expression model determining device including:
  • An acquisition module configured to acquire the user's facial image and eyeball feature information
  • the first determination module is used to determine the corresponding expression classification according to the facial image
  • the second determination module is used to determine the corresponding expression model to be adjusted based on the expression classification
  • An adjustment module configured to use the facial image and the eyeball feature information to adjust corresponding parameters of the expression model to be adjusted to obtain a target expression model.
  • the embodiment of the present application provides an electronic device, including:
  • a memory for storing executable instructions of the processor
  • the processor is configured to execute any method in the first aspect, the second aspect, each possible implementation manner of the first aspect, or each possible implementation manner of the second aspect by executing the executable instructions.
  • the embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the first aspect, the second aspect, and possible implementation modes of the first aspect are realized , or any method in each possible implementation manner of the second aspect.
  • the embodiments of the present application provide a computer program product, including a computer program.
  • the computer program When the computer program is executed by a processor, the first aspect, the second aspect, each possible implementation mode of the first aspect, or each possible implementation mode of the second aspect can be realized. Any of the possible implementations.
  • the embodiment of the present application obtains the user's facial image and eyeball characteristic information; determines the corresponding expression classification according to the facial image; determines the corresponding expression model to be adjusted based on the expression classification; utilizes the facial image and the eyeball characteristic information Adjust the corresponding parameters of the expression model to be adjusted to obtain the target expression model.
  • a new three-dimensional expression image can be synthesized in real time according to the acquired user's facial image and eyeball feature information, and the three-dimensional expression model can be locally updated dynamically to generate The new image improves the realism of the image and enhances the user experience.
  • FIG. 1 is a schematic structural diagram of an expression model determination system provided by an embodiment of the present application.
  • Fig. 2a is a schematic flowchart of a method for determining an expression model provided by an embodiment of the present application
  • Fig. 2b is a schematic diagram of a geometric model provided by an embodiment of the present application.
  • Fig. 2c is a schematic diagram of multiple optional expression models provided by an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of an expression model determining device provided in an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • VR Virtual Reality, virtual reality technology
  • Virtual reality technology includes computer, electronic information, and simulation technology. Its basic implementation method is to simulate a virtual environment by computer to give people a sense of environmental immersion.
  • Fig. 1 is a schematic structural diagram of an expression model determination system provided by an exemplary embodiment of the present application, the structure includes: a data processing device 11 and a head-mounted device 12, wherein: the head-mounted device 12 is used to collect a user's facial image, And determine the eyeball feature information of the user; send the facial image and the eyeball feature information to the data processing device 11;
  • the data processing device 11 receives the user's facial image and the user's eyeball feature information; determines the corresponding expression classification according to the facial image; determines the corresponding expression model to be adjusted based on the expression classification; utilizes the facial image and the eyeball feature The information adjusts the corresponding parameters of the expression model to be adjusted to obtain the target expression model.
  • the foregoing data processing device 11 may be a PC, a mobile terminal device, and the like.
  • the aforementioned data processing device 11 can also be a server device.
  • the head-mounted device 12 can transmit the facial image and eyeball feature information to the server through a PC or a mobile terminal device. end device.
  • the data processing device 11 determines the target expression model, it will synthesize the scene picture data to be displayed according to the scene data of the scene corresponding to the target expression model, and send the scene picture data to be displayed to the head-mounted device 13 for display, wherein the head-mounted device 13 and the head-mounted device 12 may be the same device, and the head-mounted device 13 may also be other devices.
  • the head-mounted device 12 after the head-mounted device 12 collects the user's facial image and determines the user's eyeball feature information, it can determine the corresponding expression classification according to the facial image; determine the corresponding expression classification according to the expression classification.
  • the terminal device synthesizes the scene picture data to be displayed according to the target expression model and the scene data where the target expression model is located, and sends the scene picture data to be displayed to the head-mounted device 13 for display, wherein the head-mounted device 13 and the head-mounted device
  • the device 12 can be the same device, and the head-mounted device 13 can also be other devices.
  • the image of the user displayed in the scene can be updated in real time, which improves the authenticity of the VR scene and user experience.
  • Fig. 2a is a schematic flowchart of a method for determining an expression model provided by an exemplary embodiment of the present application.
  • the method is applicable to the aforementioned data processing device, server device, or head-mounted device.
  • the method includes at least the following steps:
  • the aforementioned facial image may be captured by a camera installed on the head-mounted device.
  • the installation position of the camera and the number of the camera are not limited in this application.
  • the aforementioned eyeball feature information includes eyeball position information and/or line of sight direction, etc., wherein the eyeball position information can be the distance information of the eyeball center relative to the center coordinates of the eye where the eyeball is located, or the center coordinates of the eyeball center relative to the eye where the eyeball is located.
  • the ratio information of distance information and eye width that is, the distance between the inner corner of the eye and the outer corner of the same eye.
  • the determination method of the eyeball position information can be determined based on the geometric model shown in Figure 2b, specifically, it can be based on the eyeball and the VR headset.
  • the real-time position of the device builds a geometric model, and calculates the intersection point of the line of sight and the screen in the VR headset, that is, the gaze position of the eyeball on the screen; and further determines the eyeball position information based on the gaze position.
  • the head-mounted device may be provided with a display screen, specifically, it may only include one display screen for displaying pictures for both eyes to watch, or it may include two display screens, and the two display screens respectively show images viewed by the left and right eyes. screen.
  • the display screen includes two display areas, and the two display areas can display pictures for left and right eyes respectively.
  • the display area corresponding to any eye such as the display area corresponding to the left eye as an example
  • the upper left corner of the screen can be used as the origin o
  • the plane where the screen is located is xoy
  • the known distance between the eyeball and the screen is d
  • the interpupillary distance is L.
  • the height of the screen is h
  • the width of the screen is w
  • the coordinate R1 of the left eye is (h/2, w/2-L/2, d).
  • the intersection of the vector RA passing through R1 and the xoy plane can be calculated, that is, the coordinates of the gaze point on the screen.
  • the method of determining the coordinates of the intersection point of the vector RA passing through R1 and the xoy plane can be determined according to the following formula:
  • R1 is the coordinate of the left eye
  • RA is the direction vector of the line of sight
  • RN is the normal vector of the xoy plane
  • R0 is a point on the xoy plane
  • S is the calculated coordinate of the intersection point.
  • R1, RA, RN, and R0 are known information, and the known information can be obtained directly from the eye tracking module, or determined according to the existing technology, and will not be repeated here.
  • the first distance information between the center coordinates and the intersection coordinates can be further determined according to the center coordinates of the display area corresponding to the left eye and the intersection coordinates; obtain the left eye The eye width information; according to the first distance information, the area width information of the display area corresponding to the left eye, and the eye width information, determine the second distance information between the center of the eyeball and the center of the left eye.
  • the second distance information for determining the distance between the eyeball center and the left eye center from the eye width information includes:
  • a ratio of the first distance information to the proportional value is used as the second distance information.
  • the eyeball position information corresponding to the left eye is the second distance information.
  • the determination method of the eyeball position information corresponding to the right eye is similar to the determination method of the eyeball position information corresponding to the left eye, and will not be repeated here.
  • the aforementioned area width information of the display area may be determined according to known screen parameter information of the display screen.
  • determining the corresponding expression classification according to the facial image includes:
  • the aforementioned facial expression recognition model can recognize multiple facial expressions, such as: anger, happiness, surprise, sadness, etc.; any of the expressions.
  • determining the corresponding expression model to be adjusted based on the expression classification may specifically include: determining the expression model to be adjusted according to the correspondence between the expression classification and a preset.
  • the aforementioned preset correspondence is a correspondence between expression classifications and their corresponding expression models.
  • the correspondence between expression classifications and their corresponding expression models may be one-to-one or one-to-many.
  • the expression model corresponding to the expression classification determined according to the preset correspondence can be directly used as the expression model to be adjusted;
  • determining the corresponding expression model to be adjusted based on the expression classification that is, determining the expression model to be adjusted according to the expression classification and the preset correspondence relationship may specifically include:
  • the expression model to be adjusted is determined based on the selection instruction.
  • the aforementioned plurality of optional expression models may be determined according to a preset relationship library containing expression classifications and multiple optional expression models corresponding thereto.
  • different optional expression models may correspond to different model features, and the model features may include any one or more of the following: gender, hairstyle, clothing, and accessories, as shown in Figure 2c for example.
  • the model features may also include other information to further improve the user's personalized experience.
  • using the expression classification to determine the corresponding multiple optional expression models further includes:
  • the plurality of optional expression models are determined according to the expression model library corresponding to the facial proportion information and the expression classification.
  • the aforementioned facial proportion information includes facial width proportion information and facial length proportion information; in some embodiments, the facial length proportion information includes any one or more of the following: the proportion of the upper part of the face to the length of the face, the proportion of the middle part of the face to the length of the face Proportion, and the ratio of the lower part of the face to the face length; the face width ratio information includes one or more of the following: the ratio of the distance from the outer side of the left eye to the left hairline to the face width, the ratio of the length of the left eye to the face width, the left The ratio of the distance between the inner corner of the eye and the inner corner of the right eye to the width of the face, the ratio of the length of the right eye to the width of the face, and the ratio of the distance from the outer side of the right eye to the right hairline to the width of the face.
  • the model face ratio information includes corresponding model face width ratio information and model face length ratio information; in some embodiments, the model face length ratio information includes any one or more of the following: The ratio of the atrium in the model to the length of the model's face, and the ratio of the lower atrium to the length of the model's face in the model; the ratio information of the model's face width includes one or more of the following: the distance from the outside of the left eye to the left hairline in the model The ratio of the distance to the width of the model's face, the ratio of the length of the left eye in the model to the width of the model's face, the ratio of the distance between the inner corner of the left eye and the inner corner of the right eye in the model to the width of the model's face, the length of the right
  • the facial expression model involved in this application may be a two-dimensional model or a three-dimensional model.
  • the corresponding parameters of the expression model to be adjusted are adjusted by using the facial image and the eyeball characteristic information, and the target expression model obtained includes:
  • the facial feature information includes any one or more of the following: distance information between the corners of the mouth on both sides, vertical distance information between the corners of the mouth and the lip peak on one or both sides ;
  • the method of determining the corresponding facial feature information based on the facial image can be realized based on image recognition technology, or the facial image can be input into the trained feature recognition model, so as to determine the facial features information, this application does not limit it.
  • the facial feature information may further include feature point information of the lips, where the feature points of the lips
  • the information may be the feature point coordinates of the lips.
  • the feature points of the lips may include the corner points on both sides of the mouth, the peaks of the two lips, and the lowest point of the lips. The coordinates of the feature points of the lips can be determined according to the face image through the relevant neural network model.
  • the facial image of the user may be a two-dimensional image or a three-dimensional image.
  • the corresponding parameters of the expression model to be adjusted are adjusted by using the facial feature information and the eyeball feature information to obtain the target expression model including:
  • the parameter type of the facial feature parameter to be adjusted is the same as the parameter type of the facial feature information, such as: when the facial feature information is the distance information between the corners of the mouth on both sides, the facial feature parameter to be adjusted is also the distance between the corners of the mouth on both sides. distance information.
  • the parameter type of the eyeball characteristic parameter of the expression model to be adjusted is also the same as the parameter type of the eyeball characteristic information, such as: when the eyeball characteristic information is eyeball position information, the eyeball characteristic parameter of the expression model to be adjusted is also the eyeball position information.
  • Adjusting the to-be-adjusted facial feature parameters of the to-be-adjusted expression model to match the facial feature information refers to adjusting the to-be-adjusted facial feature parameters of the to-be-adjusted facial feature information to be the same as the facial feature information
  • Adjusting the eyeball feature parameters of the expression model to be adjusted to match the eyeball feature information refers to adjusting the eyeball feature parameters of the expression model to be adjusted to be the same as the eyeball feature information.
  • the present application adjusts the corresponding parameters of the expression model to be adjusted according to the facial feature information and the eyeball feature information, other facial information, facial feature information and the eyeball feature information in the expression model to be adjusted
  • the corresponding parameters are adjusted jointly to make the model smoother.
  • the continuous execution of the aforementioned steps S201 to S204 can realize the continuous update of the target expression model, and then reflect the user's eye movement information and expression changes to the 3D image model in the VR scene in real time, making the 3D image in the virtual reality more accurate. Vivid and realistic, it allows VR users to feel the real movements and expressions of each other in the virtual world.
  • the embodiment of the present application obtains the user's facial image and eyeball characteristic information; determines the corresponding expression classification according to the facial image; determines the corresponding expression model to be adjusted based on the expression classification; utilizes the facial image and the eyeball characteristic information Adjust the corresponding parameters of the expression model to be adjusted to obtain the target expression model.
  • a new three-dimensional expression image can be synthesized in real time according to the acquired user's facial image and eyeball feature information, and the three-dimensional expression model can be locally updated dynamically to generate a new expression model. The image improves the realism of the image and improves the user experience.
  • the present application also provides a data processing method, which can be applied to a head-mounted device, and specifically the method may include the following steps:
  • the present application also provides a data processing device, which may specifically include: an acquisition module, a call module, and a synthesis and generation module; wherein:
  • the acquisition module is used for eye tracking, expression recognition, and facial feature information acquisition
  • the synthesizing module is used for synthesizing a new image according to eyeball positions, facial feature information, and two-dimensional or three-dimensional models corresponding to the expression classification.
  • Fig. 3 is a schematic structural diagram of an expression model determination device provided by an exemplary embodiment of the present application; the device includes: an acquisition module 31, a first determination module 32, a second determination module 33, and an adjustment module 34; wherein:
  • Obtaining module 31 for obtaining user's facial image and eyeball feature information
  • the first determination module 32 is used to determine the corresponding expression classification according to the facial image
  • the second determining module 33 is configured to determine a corresponding expression model to be adjusted based on the expression classification
  • the adjustment module 34 is configured to use the facial image and the eyeball feature information to adjust the corresponding parameters of the expression model to be adjusted to obtain a target expression model.
  • the aforementioned device when used to determine the corresponding expression classification according to the facial image, it is specifically used for:
  • the aforementioned device when used to use the facial image and the eyeball feature information to adjust the corresponding parameters of the expression model to be adjusted to obtain the target expression model, it is specifically used for:
  • the facial feature information includes any one or more of the following: distance information between the corners of the mouth on both sides, vertical distance information between the corners of the mouth and the lip peak on one side or both sides;
  • the aforementioned device uses the facial feature information and the eyeball feature information to adjust the corresponding parameters of the expression model to be adjusted to obtain the target expression model, it is specifically used for:
  • the aforementioned device when used to determine the corresponding expression model to be adjusted based on the expression classification, it is specifically used for:
  • the expression model to be adjusted is determined based on the selection instruction.
  • the aforementioned device when used to determine a plurality of corresponding optional expression models using the expression classification, it is specifically used for:
  • the plurality of optional expression models are determined according to the expression model library corresponding to the facial proportion information and the expression classification.
  • An exemplary embodiment of the present application also provides a data processing device, the device includes a collection module and a sending module; wherein:
  • the collection module is used to collect the user's facial image and determine the user's eyeball feature information
  • a sending module configured to send the facial image and the eyeball characteristic information to a data processing device, so that the data processing device determines the corresponding expression classification according to the facial image; and determines the corresponding expression to be adjusted based on the expression classification.
  • An expression model using the facial image and the eyeball feature information to adjust corresponding parameters of the expression model to be adjusted to obtain a target expression model.
  • the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, details are not repeated here.
  • the device can execute the above-mentioned method embodiments, and the aforementioned and other operations and/or functions of the modules in the device are respectively for the corresponding processes in the methods in the above-mentioned method embodiments, and for the sake of brevity, are not repeated here repeat.
  • each step of the method embodiment in the embodiment of the present application can be completed by an integrated logic circuit of the hardware in the processor and/or instructions in the form of software, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as hardware
  • the execution of the decoding processor is completed, or the combination of hardware and software modules in the decoding processor is used to complete the execution.
  • the software module may be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, and registers.
  • the storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps in the above method embodiments in combination with its hardware.
  • Fig. 4 is a schematic block diagram of an electronic device provided by an embodiment of the present application, the electronic device may include:
  • a memory 401 and a processor 402 the memory 401 is used to store computer programs and transmit the program codes to the processor 402 .
  • the processor 402 can invoke and run a computer program from the memory 401, so as to implement the method in the embodiment of the present application.
  • the processor 402 can be used to execute the above-mentioned method embodiments according to the instructions in the computer program.
  • the processor 402 may include but not limited to:
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field Programmable Gate Array
  • the memory 401 includes but is not limited to:
  • non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electronically programmable Erase Programmable Read-Only Memory (Electrically EPROM, EEPROM) or Flash.
  • the volatile memory can be Random Access Memory (RAM), which acts as external cache memory.
  • RAM Static Random Access Memory
  • SRAM Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • Synchronous Dynamic Random Access Memory Synchronous Dynamic Random Access Memory
  • SDRAM double data rate synchronous dynamic random access memory
  • Double Data Rate SDRAM, DDR SDRAM double data rate synchronous dynamic random access memory
  • Enhanced SDRAM, ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory
  • Direct Rambus RAM Direct Rambus RAM
  • the computer program can be divided into one or more modules, and the one or more modules are stored in the memory 401 and executed by the processor 402 to complete the method.
  • the one or more modules may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.
  • the electronic device may also include:
  • a transceiver 403 , the transceiver 403 can be connected to the processor 402 or the memory 401 .
  • the processor 402 can control the transceiver 403 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices.
  • Transceiver 403 may include a transmitter and a receiver.
  • the transceiver 403 may further include antennas, and the number of antennas may be one or more.
  • bus system includes not only a data bus, but also a power bus, a control bus and a status signal bus.
  • the present application also provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a computer, the computer can execute the methods of the above method embodiments.
  • the embodiments of the present application further provide a computer program product including instructions, and when the instructions are executed by a computer, the computer executes the methods of the foregoing method embodiments.
  • the computer program product includes one or more computer instructions.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g. (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) to another website site, computer, server or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media.
  • the available medium may be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a digital video disc (digital video disc, DVD)), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)), etc.
  • a method for determining an expression model including:
  • determining the corresponding expression classification according to the facial image includes:
  • the target expression model obtained includes:
  • the facial feature information includes any one or more of the following: distance information between the corners of the mouth on both sides, vertical distance information between the corners of the mouth and the lip peak on one side or both sides;
  • the target expression model obtained includes:
  • determining the corresponding expression model to be adjusted based on the expression classification includes:
  • the expression model to be adjusted is determined based on the selection instruction.
  • using the expression classification to determine a plurality of corresponding optional expression models includes:
  • the plurality of optional expression models are determined according to the expression model library corresponding to the facial proportion information and the expression classification.
  • a data processing method including:
  • a device for determining an expression model including:
  • An acquisition module configured to acquire the user's facial image and eyeball feature information
  • the first determination module is used to determine the corresponding expression classification according to the facial image
  • the second determination module is used to determine the corresponding expression model to be adjusted based on the expression classification
  • An adjustment module configured to use the facial image and the eyeball feature information to adjust corresponding parameters of the expression model to be adjusted to obtain a target expression model.
  • the aforementioned device when used to determine the corresponding expression classification according to the facial image, it is specifically used for:
  • the aforementioned device when used to use the facial image and the eyeball feature information to adjust the corresponding parameters of the expression model to be adjusted to obtain a target expression model, it is specifically used for:
  • the facial feature information includes any one or more of the following: distance information between the corners of the mouth on both sides, vertical distance information between the corners of the mouth and the lip peak on one side or both sides;
  • the aforementioned device uses the facial feature information and the eyeball feature information to adjust the corresponding parameters of the expression model to be adjusted to obtain the target expression model, it is specifically used for:
  • the aforementioned device when used to determine the corresponding expression model to be adjusted based on the expression classification, it is specifically used for:
  • the expression model to be adjusted is determined based on the selection instruction.
  • the aforementioned device when used to determine a plurality of corresponding optional expression models using the expression classification, it is specifically used for:
  • the plurality of optional expression models are determined according to the expression model library corresponding to the facial proportion information and the expression classification.
  • a data processing device includes a collection module and a sending module; wherein:
  • the collection module is used to collect the user's facial image and determine the user's eyeball feature information
  • a sending module configured to send the facial image and the eyeball characteristic information to a data processing device, so that the data processing device determines the corresponding expression classification according to the facial image; and determines the corresponding expression to be adjusted based on the expression classification.
  • An expression model using the facial image and the eyeball feature information to adjust corresponding parameters of the expression model to be adjusted to obtain a target expression model.
  • an electronic device including:
  • a memory for storing executable instructions of the processor
  • the processor is configured to execute the above methods by executing the executable instructions.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above methods are implemented.
  • modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
  • the disclosed systems, devices and methods may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the modules is only a logical function division. In actual implementation, there may be other division methods.
  • multiple modules or components can be combined or can be Integrate into another system, or some features may be ignored, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.
  • a module described as a separate component may or may not be physically separated, and a component displayed as a module may or may not be a physical module, that is, it may be located in one place, or may also be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)

Abstract

本申请公开了一种表情模型确定方法、装置、设备及计算机可读存储介质，其中，方法包括：获取用户的面部图像与眼球特征信息；根据面部图像确定对应的表情分类；基于表情分类确定对应的待调整表情模型；利用面部图像与眼球特征信息调整待调整表情模型的对应参数，得到目标表情模型。

Description

表情模型确定方法、装置、设备及计算机可读存储介质
相关申请的交叉引用
本申请要求于2021年12月21日提交的，申请名称为“表情模型确定方法、装置、设备及计算机可读存储介质”的、中国专利申请号为“202111572188.3”的优先权，该中国专利申请的全部内容通过引用结合在本申请中。
技术领域
本申请属于虚拟现实技术领域，尤其涉及一种表情模型确定方法、装置、设备及计算机可读存储介质。
背景技术
相关技术中，VR设备可基于对数据的运算模拟虚拟世界，提供用户关于视觉、听觉、触觉等感官的模拟，在VR虚拟场景中，人物形象的展示，一般基于静态模型，虽然可以支持用户对自身形象的设置功能，但是一般设置完成后的形象为固定形象，在场景中不可更改。即使在场景中涉及动态形象的展示，也需依赖预先设置的动作及表情，并且仅能在场景中的某些情节显示，或者依赖用户使用按键触发显示，用户体验较差。
技术解决方案
本申请实施例提供一种与现有技术不同的实现方案，以解决现有技术中的形象确定方式导致的用户体验较差的技术问题。
第一方面，本申请实施例提供一种表情模型确定方法，包括：
获取用户的面部图像与眼球特征信息；
根据所述面部图像确定对应的表情分类；
基于所述表情分类确定对应的待调整表情模型；
利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
第二方面，本申请实施例提供一种数据处理方法，包括：
采集用户的面部图像，以及确定用户的眼球特征信息；
将所述面部图像与所述眼球特征信息发送至数据处理设备，以使所述数据处理设备根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
第三方面，本申请实施例提供一种表情模型确定装置，包括：
获取模块，用于获取用户的面部图像与眼球特征信息；
第一确定模块，用于根据所述面部图像确定对应的表情分类；
第二确定模块，用于基于所述表情分类确定对应的待调整表情模型；
调整模块，用于利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
第四方面，本申请实施例提供一种电子设备，包括：
处理器；以及
存储器，用于存储所述处理器的可执行指令；
其中，所述处理器配置为经由执行所述可执行指令来执行第一方面、第二方面，第一方面各可能的实施方式，或第二方面各可能的实施方式中的任一方法。
第五方面，本申请实施例提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现第一方面、第二方面，第一方面各可能的实施方式，或第二方面各可能的实施方式中的任一方法。
第六方面，本申请实施例提供一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现第一方面、第二方面，第一方面各可能的实施方式，或第二方面各可能的实施方式中的任一方法。
本申请实施例通过获取用户的面部图像与眼球特征信息；根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型的方案，可以实时根据获取到的用户的面部图像与眼球特征信息合成新的三维表情形象，可以动态对三维表情模型进行局部更新，生成新的形象，提高了形象的真实感，提高了用户体验。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。在附图中：
图1为本申请一实施例提供的表情模型确定系统的结构示意图；
图2a为本申请一实施例提供的表情模型确定方法的流程示意图；
图2b为本申请一实施例提供的几何模型的示意图；
图2c为本申请一实施例提供的多个可选表情模型的示意图；
图3为本申请实施例提供的一种表情模型确定装置的结构示意图；
图4为本申请实施例提供的一种电子设备的结构示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
本申请实施例的说明书、权利要求书及附图中的术语“第一”和“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请实施例的实施例例如能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
首先，下面对本申请实施例中的部分用语进行解释说明，以便于本领域技术人员理解。
VR：Virtual Reality，虚拟现实技术，是20世纪发展起来的一项全新的实用技术。虚拟现实技术囊括计算机、电子信息、仿真技术，其基本实现方式是计算机模拟虚拟环境从而给人以环境沉浸感。
下面以具体的实施例对本申请的技术方案以及本申请的技术方案如何解决上述技术问题进行详细说明。下面这几个具体的实施例可以相互结合，对于相同或相似的概念或过程可能在某些实施例中不再赘述。下面将结合附图，对本申请的实施例进行描述。
图1为本申请一示例性实施例提供的一种表情模型确定系统的结构示意图，该结构包括：数据处理设备11与头戴设备12，其中：头戴设备12用于采集用户的面部图像，以及确定用户的眼球特征信息；将所述面部图像与所述眼球特征信息发送至数据处理设备11；
数据处理设备11接收用户的面部图像与用户的眼球特征信息；根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
在一些实施例中，前述数据处理设备11可以为PC、移动终端设备等。
在一些实施例中，前述数据处理设备11还可以为服务端设备，当数据处理设备11为服务端设备时，头戴设备12可以通过PC或移动终端设备将面部图像与眼球特征信息传输至服务端设备。
进一步地，数据处理设备11在确定出目标表情模型后，将根据该目标表情模型所对应的场景的场景数据，合成待显示的场景画面数据，并将待显示的场景画面数据发送至头戴设备13供其显示，其中头戴设备13与头戴设备12可以为同一个设备，头戴设备13也可以为其它设备。
在本申请的另一些实施例中，前述头戴设备12在采集用户的面部图像，以及确定用户的眼球特征信息后，自身可根据面部图像确定对应的表情分类；根据所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整待调整表情模型的对应参数，得到目标表情模型，并将该目标表情模型通过前述数据处理设备11推送至服务端设备，供服务端设备根据该目标表情模型与目标表情模型所在的场景数据，合成待显示的场景画面数据，并将待显示的场景画面数据发送至头戴设备13供其显示，其中头戴设备13与头戴设备12可以为同一个设备，头戴设备13也可以为其它设备。
通过前述方案，可以实时对场景中展示的用户的形象进行更新，提高了VR场景的真实性以及用户体验。
本系统实施例中的各组成单元，如数据处理设备11以及头戴设备12中具体功能的执行原理及交互过程可参见如下各方法实施例的描述。
图2a为本申请一示例性实施例提供的一种表情模型确定方法的流程示意图，该方法可适用于前述数据处理设备、服务端设备，或头戴设备，该方法至少包括以下步骤：
S201、获取用户的面部图像与眼球特征信息；
S202、根据所述面部图像确定对应的表情分类；
S203、基于所述表情分类确定对应的待调整表情模型；
S204、利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
具体地，前述面部图像可以由设置于头戴设备的拍摄装置采集得到，具体地，拍摄装置的安装位置，以及拍摄装置的数量本申请不做限定。
进一步地，前述眼球特征信息包括眼球位置信息和/或视线方向等，其中，眼球位置信息可以为眼球中心相对于眼球所在眼睛的中心坐标的距离信息，或者眼球中心相对于眼球所在眼睛的中心坐标的距离信息与眼睛宽度(即同一只眼睛的内眼角与外眼角的间距)的比例信息。
具体地，当眼球位置信息为眼球中心相对于眼球所在眼睛的中心坐标的距离信息时，眼球位置信息的确定方式可基于图2b所示的几何模型确定，具体地，可依据眼球与VR头戴设备的实时位置构建几何模型，计算视线与VR头戴设备中屏幕的交点，即眼球在屏幕上的注视位置；进一步根据该注视位置确定眼球位置信息。
具体地，头戴设备中可设置有显示屏，具体地，可仅包含一个用于显示供双眼观看的画面的显示屏，也可以包含两个显示屏，该两个显示屏分别展示左右眼观看的画面。
在本申请的一些实施例中，可参见图2b所示，若头戴设备中设置一个显示屏，该显示屏包含有两个显示区域，该两个显示区域可分别展示供左右眼观看的画面，以任一眼睛对应 的显示区域为例，如左眼对应的显示区域为例，可以屏幕左上角为原点o，屏幕所在平面为xoy，已知眼球距离屏幕距离为d，瞳距为L，屏幕高度为h，屏幕宽度为w，左眼坐标R1则为(h/2，w/2-L/2，d)。若根据眼球追踪模块确定出过点R1的眼球追踪向量为RA(a，b，c)，可计算出过R1的向量RA与xoy面的交点，即屏幕注视点的坐标。具体地，过R1的向量RA与xoy面的交点坐标的确定方式可根据以下公式确定：
S＝R1+(RN*(R0-R1))/(RN*RA)*RA
其中，R1为左眼坐标、RA为视线的方向向量、RN为xoy平面法矢、R0为xoy平面上一点；S为计算得到的交点坐标。其中，R1、RA、RN、R0为已知信息，已知信息可直接自眼球追踪模块获取，或者根据现有技术确定，此处不再赘述。
进一步地，确定出过R1的向量RA与xoy面的交点坐标后，可进一步根据左眼对应的显示区域的中心坐标与该交点坐标确定出中心坐标与交点坐标的第一距离信息；获取左眼的眼睛宽度信息；根据所述第一距离信息、左眼对应的显示区域的区域宽度信息，与所述眼睛宽度信息确定出眼球中心距离左眼中心的第二距离信息。
具体地，据所述第一距离信息、左眼对应的显示区域的区域宽度信息，与所述眼睛宽度信息确定出眼球中心距离左眼中心的第二距离信息包括：
确定显示区域的区域宽度信息与所述眼睛宽度信息的比例值；
将所述第一距离信息与所述比例值的比值作为所述第二距离信息。
进一步地，左眼对应的眼球位置信息则为第二距离信息，相应地，右眼对应的眼球位置信息的确定方式与左眼对应的眼球位置信息的确定方式类似，此处不再赘述。
需要说明的是，前述显示区域的区域宽度信息可以根据已知的显示屏的屏幕参数信息确定。
进一步地，前述步骤S202中，根据所述面部图像确定对应的表情分类包括：
将所述面部图像输入预设的表情识别模型，确定所述表情分类，其中，所述表情识别模型可以为基于对多组样本数据的训练确定出的机器学习模型，具体可以为神经网络模型。
具体地，前述表情识别模型可识别多种面部表情，如：愤怒、开心、惊喜、难过等；前述将所述面部图像输入预设的表情识别模型后确定出的表情分类可以为前述多种面部表情中的任一种。
进一步地，前述步骤S203中，基于所述表情分类确定对应的待调整表情模型可具体包括：根据表情分类与预设对应关系确定所述待调整表情模型。
进一步地，前述预设对应关系为表情分类与其对应的表情模型的对应关系，具体地，表情分类与其对应的表情模型可以一一对应，也可以一对多。
在一些实施例中，当表情分类与其对应的表情模型为一一对应时，可直接将根据预设 对应关系确定出的表情分类对应的表情模型作为待调整表情模型；
当表情分类与其对应的表情模型的对应关系为一对多时，基于所述表情分类确定对应的待调整表情模型，即根据表情分类与预设对应关系确定所述待调整表情模型可具体包括：
利用所述表情分类确定其对应的多个可选表情模型；
控制展示所述多个可选表情模型；
获取用户针对从所述多个可选表情模型中选出所述待调整表情模型的选择指令；
基于所述选择指令确定所述待调整表情模型。
具体地，前述多个可选表情模型可以根据包含表情分类与其对应的多个可选表情模型的预设关系库确定。多个可选表情模型中，不同的可选表情模型对应的模型特征可以不同，模型特征可以包括以下任一种或多种：性别、发型、衣着、饰品，例如图2c所示。另外，模型特征还可以包括其他信息，以进一步提高用户的个性化体验。
在本申请的另一些实施例中，利用所述表情分类确定其对应的多个可选表情模型还包括：
根据所述面部图像确定面部比例信息；
依据所述面部比例信息与所述表情分类对应的表情模型库确定所述多个可选表情模型。
具体地，前述面部比例信息包括面部宽度比例信息与面部长度比例信息；在一些实施例中，面部长度比例信息包括以下任一种或多种：上庭占面部长度的比例、中庭占面部长度的比例，以及下庭占面部长度的比例；面部宽度比例信息包括以下一种或多种：左眼外侧至左侧发际线的距离占面部宽度的比例、左眼长度占面部宽度的比例、左眼内眼角与右眼内眼角的距离占面部宽度的比例、右眼长度占面部宽度的比例、右眼外侧至右侧发际线的距离占面部宽度的比例。
进一步地，前述依据所述面部比例信息与所述表情分类对应的表情模型库确定出的多个可选表情模型中的各可选表情模型对应的模型面部比例信息与面部图像对应的面部比例信息完全一致，或部分一致。相应地，模型面部比例信息包括相应的模型面部宽度比例信息与模型面部长度比例信息；在一些实施例中，模型面部长度比例信息包括以下任一种或多种：模型中上庭占模型面部长度的比例、模型中中庭占模型面部长度的比例，以及模型中下庭占模型面部长度的比例；模型面部宽度比例信息包括以下一种或多种：模型中左眼外侧至左侧发际线的距离占模型面部宽度的比例、模型中左眼长度占模型面部宽度的比例、模型中左眼内眼角与右眼内眼角的距离占模型面部宽度的比例、模型中右眼长度占模型中面部宽度的比例、模型中右眼外侧至右侧发际线的距离占模型面部宽度的比例。
在一些实施例中，本申请中涉及到的表情模型可以二维模型，也可以为三维模型。
进一步地，前述步骤S204中，利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型包括：
S2041、根据所述面部图像确定对应的面部特征信息，所述面部特征信息包括以下任一种或多种：两侧嘴角之间的距离信息、单侧或双侧嘴角与唇峰的垂直距离信息；
S2042、利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
在一些实施例中，前述S2041中，根据所述面部图像确定对应的面部特征信息的方式可以基于图像识别技术实现，也可以将面部图像输入经训练得出的特征识别模型，从而确定出面部特征信息，对此，本申请不做限定。
进一步地，面部特征信息除了包含前述两侧嘴角之间的距离信息、单侧或双侧嘴角与唇峰的垂直距离信息外，还可以进一步包含唇部的特征点信息，其中唇部的特征点信息可以为唇部的特征点坐标，具体地，唇部的特征点可以包括两侧嘴角点、两唇峰、嘴唇最低点等。唇部的特征点的坐标可以通过相关的神经网络模型根据面部图像确定。
在一些实施例中，用户的面部图像可以为二维图像，也可以为三维图像。
进一步地，前述步骤S2042中，利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型包括：
将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息匹配，以及将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息匹配，得到所述目标表情模型。
具体地，待调整面部特征参数的参数类型与面部特征信息的参数类型相同，如：当面部特征信息为两侧嘴角之间的距离信息时，待调整面部特征参数也为两侧嘴角之间的距离信息。相应地，所述待调整表情模型的眼球特征参数的参数类型，与眼球特征信息的参数类型也相同，如：眼球特征信息为眼球位置信息时，待调整表情模型的眼球特征参数也为眼球位置信息。
将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息匹配，指将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息相同，以及将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息匹配，指将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息相同。
进一步地，本申请在依据所述面部特征信息与所述眼球特征信息调整待调整表情模型的对应参数时，还可将待调整表情模型中其它的面部信息与面部特征信息和所述眼球特征信息的相应参数联合调整，以使模型更平滑。
另外，连续执行前述步骤S201至S204，可实现对目标表情模型的连续更新，进而将 用户的眼动信息与表情变化实时反映至VR场景中的3维形象模型中，使虚拟现实中三维形象更加生动，逼真，可以让VR使用者在虚拟世界中，感受到对方真实的动作和表情。
本申请实施例通过获取用户的面部图像与眼球特征信息；根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型的方案，可以实时根据获取到的用户面部图像与眼球特征信息合成新的三维表情形象，可以动态对三维表情模型进行局部更新，生成新的形象，提高了形象的真实感，提高了用户体验。
进一步地，本申请还提供一种数据处理方法，可适用于头戴设备，具体该方法可包括以下步骤：
采集用户的面部图像，以及确定用户的眼球特征信息；
将所述面部图像与所述眼球特征信息发送至数据处理设备，以使所述数据处理设备根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
本实施例对应的具体实施方式可参见前述内容，此处不再赘述。
进一步地，本申请还提供一种数据处理装置，该装置可具体包括：获取模块、调用模块，以及合成与生成模块；其中：
获取模块，用于进行眼球追踪、表情识别，以及面部特征信息获取；
调用模块，用于根据眼球追踪结果确定出眼球位置，以及根据表情识别结果确定出表情分类；
合成模块，用于根据眼球位置、面部特征信息，以及所述表情分类对应的二维或三维模型，合成新的形象。
本实施例对应的具体实施方式可参见前述内容，此处不再赘述。
图3为本申请一示例性实施例提供的一种表情模型确定装置的结构示意图；该装置包括：获取模块31、第一确定模块32、第二确定模块33，以及调整模块34；其中：
获取模块31，用于获取用户的面部图像与眼球特征信息；
第一确定模块32，用于根据所述面部图像确定对应的表情分类；
第二确定模块33，用于基于所述表情分类确定对应的待调整表情模型；
调整模块34，用于利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
在一些实施例中，前述装置在用于根据所述面部图像确定对应的表情分类时，具体用于：
将所述面部图像输入预设的表情识别模型，确定所述表情分类，其中，所述表情识别模型为神经网络模型。
在一些实施例中，前述装置在用于利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型时，具体用于：
根据所述面部图像确定对应的面部特征信息，所述面部特征信息包括以下任一种或多种：两侧嘴角之间的距离信息、单侧或双侧嘴角与唇峰的垂直距离信息；
利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
在一些实施例中，前述装置在利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型时，具体用于：
将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息匹配，以及将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息匹配，得到所述目标表情模型。
在一些实施例中，前述装置在用于基于所述表情分类确定对应的待调整表情模型时，具体用于：
利用所述表情分类确定对应的多个可选表情模型；
控制展示所述多个可选表情模型；
获取用户针对从所述多个可选表情模型中选出所述待调整表情模型的选择指令；
基于所述选择指令确定所述待调整表情模型。
在一些实施例中，前述装置在用于利用所述表情分类确定对应的多个可选表情模型时，具体用于：
根据所述面部图像确定对应的面部比例信息；
依据所述面部比例信息与所述表情分类对应的表情模型库确定所述多个可选表情模型。
本申请一示例性实施例还提供一种数据处理装置，该装置包括采集模块与发送模块；其中：
采集模块，用于采集用户的面部图像，以及确定用户的眼球特征信息；
发送模块，用于将所述面部图像与所述眼球特征信息发送至数据处理设备，以使所述数据处理设备根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
应理解的是，装置实施例与方法实施例可以相互对应，类似的描述可以参照方法实施 例。为避免重复，此处不再赘述。具体地，该装置可以执行上述方法实施例，并且该装置中的各个模块的前述和其它操作和/或功能分别为了上述方法实施例中的各个方法中的相应流程，为了简洁，在此不再赘述。
上文中结合附图从功能模块的角度描述了本申请实施例的装置。应理解，该功能模块可以通过硬件形式实现，也可以通过软件形式的指令实现，还可以通过硬件和软件模块组合实现。具体地，本申请实施例中的方法实施例的各步骤可以通过处理器中的硬件的集成逻辑电路和/或软件形式的指令完成，结合本申请实施例公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。在一些实施例中，软件模块可以位于随机存储器，闪存、只读存储器、可编程只读存储器、电可擦写可编程存储器、寄存器等本领域的成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法实施例中的步骤。
图4是本申请实施例提供的电子设备的示意性框图，该电子设备可包括：
存储器401和处理器402，该存储器401用于存储计算机程序，并将该程序代码传输给该处理器402。换言之，该处理器402可以从存储器401中调用并运行计算机程序，以实现本申请实施例中的方法。
例如，该处理器402可用于根据该计算机程序中的指令执行上述方法实施例。
在本申请的一些实施例中，该处理器402可以包括但不限于：
通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等等。
在本申请的一些实施例中，该存储器401包括但不限于：
易失性存储器和/或非易失性存储器。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synch link DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DR RAM)。
在本申请的一些实施例中，该计算机程序可以被分割成一个或多个模块，该一个或者 多个模块被存储在该存储器401中，并由该处理器402执行，以完成本申请提供的方法。该一个或多个模块可以是能够完成特定功能的一系列计算机程序指令段，该指令段用于描述该计算机程序在该电子设备中的执行过程。
如图4所示，该电子设备还可包括：
收发器403，该收发器403可连接至该处理器402或存储器401。
其中，处理器402可以控制该收发器403与其他设备进行通信，具体地，可以向其他设备发送信息或数据，或接收其他设备发送的信息或数据。收发器403可以包括发射机和接收机。收发器403还可以进一步包括天线，天线的数量可以为一个或多个。
应当理解，该电子设备中的各个组件通过总线系统相连，其中，总线系统除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。
本申请还提供了一种计算机存储介质，其上存储有计算机程序，该计算机程序被计算机执行时使得该计算机能够执行上述方法实施例的方法。或者说，本申请实施例还提供一种包含指令的计算机程序产品，该指令被计算机执行时使得计算机执行上述方法实施例的方法。
当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。该计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行该计算机程序指令时，全部或部分地产生按照本申请实施例该的流程或功能。该计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。该计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，该计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line，DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。该计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。该可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如数字视频光盘(digital video disc，DVD))、或者半导体介质(例如固态硬盘(solid state disk，SSD))等。
根据本申请的一个或多个实施例，提供一种表情模型确定方法，包括：
获取用户的面部图像与眼球特征信息；
根据所述面部图像确定对应的表情分类；
基于所述表情分类确定对应的待调整表情模型；
利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
根据本申请的一个或多个实施例，根据所述面部图像确定对应的表情分类包括：
将所述面部图像输入预设的表情识别模型，确定所述表情分类，其中，所述表情识别模型为神经网络模型。
根据本申请的一个或多个实施例，利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型包括：
根据所述面部图像确定对应的面部特征信息，所述面部特征信息包括以下任一种或多种：两侧嘴角之间的距离信息、单侧或双侧嘴角与唇峰的垂直距离信息；
利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
根据本申请的一个或多个实施例，利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型包括：
将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息匹配，以及将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息匹配，得到所述目标表情模型。
根据本申请的一个或多个实施例，基于所述表情分类确定对应的待调整表情模型包括：
利用所述表情分类确定对应的多个可选表情模型；
控制展示所述多个可选表情模型；
获取用户针对从所述多个可选表情模型中选出所述待调整表情模型的选择指令；
基于所述选择指令确定所述待调整表情模型。
根据本申请的一个或多个实施例，利用所述表情分类确定对应的多个可选表情模型包括：
根据所述面部图像确定对应的面部比例信息；
依据所述面部比例信息与所述表情分类对应的表情模型库确定所述多个可选表情模型。
根据本申请的一个或多个实施例，提供一种数据处理方法，包括：
采集用户的面部图像，以及确定用户的眼球特征信息；
将所述面部图像与所述眼球特征信息发送至数据处理设备，以使所述数据处理设备根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
根据本申请的一个或多个实施例，提供一种表情模型确定装置，包括：
获取模块，用于获取用户的面部图像与眼球特征信息；
第一确定模块，用于根据所述面部图像确定对应的表情分类；
第二确定模块，用于基于所述表情分类确定对应的待调整表情模型；
调整模块，用于利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
根据本申请的一个或多个实施例，前述装置在用于根据所述面部图像确定对应的表情分类时，具体用于：
将所述面部图像输入预设的表情识别模型，确定所述表情分类，其中，所述表情识别模型为神经网络模型。
根据本申请的一个或多个实施例，前述装置在用于利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型时，具体用于：
根据所述面部图像确定对应的面部特征信息，所述面部特征信息包括以下任一种或多种：两侧嘴角之间的距离信息、单侧或双侧嘴角与唇峰的垂直距离信息；
利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
根据本申请的一个或多个实施例，前述装置在利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型时，具体用于：
将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息匹配，以及将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息匹配，得到所述目标表情模型。
根据本申请的一个或多个实施例，前述装置在用于基于所述表情分类确定对应的待调整表情模型时，具体用于：
利用所述表情分类确定对应的多个可选表情模型；
控制展示所述多个可选表情模型；
获取用户针对从所述多个可选表情模型中选出所述待调整表情模型的选择指令；
基于所述选择指令确定所述待调整表情模型。
根据本申请的一个或多个实施例，前述装置在用于利用所述表情分类确定对应的多个可选表情模型时，具体用于：
根据所述面部图像确定对应的面部比例信息；
依据所述面部比例信息与所述表情分类对应的表情模型库确定所述多个可选表情模型。
根据本申请的一个或多个实施例，提供一种数据处理装置，该装置包括采集模块与发送模块；其中：
采集模块，用于采集用户的面部图像，以及确定用户的眼球特征信息；
发送模块，用于将所述面部图像与所述眼球特征信息发送至数据处理设备，以使所述数据处理设备根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
根据本申请的一个或多个实施例，提供一种电子设备，包括：
处理器；以及
存储器，用于存储所述处理器的可执行指令；
其中，所述处理器配置为经由执行所述可执行指令来执行上述各方法。
根据本申请的一个或多个实施例，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述各方法。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的模块及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，该模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个模块或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或模块的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的模块可以是或者也可以不是物理上分开的，作为模块显示的部件可以是或者也可以不是物理模块，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。例如，在本申请各个实施例中的各功能模块可以集成在一个处理模块中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个模块中。
以上仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以该权利要求的保护范围为准。

Claims (11)

  1. 一种表情模型确定方法，包括：
    获取用户的面部图像与眼球特征信息；
    根据所述面部图像确定对应的表情分类；
    基于所述表情分类确定对应的待调整表情模型；
    利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
  2. 根据权利要求1所述的方法，其中，根据所述面部图像确定对应的表情分类包括：
    将所述面部图像输入预设的表情识别模型，确定所述表情分类，其中，所述表情识别模型为神经网络模型。
  3. 根据权利要求1所述的方法，其中，利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型包括：
    根据所述面部图像确定对应的面部特征信息，所述面部特征信息包括以下任一种或多种：两侧嘴角之间的距离信息、单侧或双侧嘴角与唇峰的垂直距离信息；
    利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
  4. 根据权利要求3所述的方法，其中，利用所述面部特征信息与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型包括：
    将所述待调整表情模型的待调整面部特征参数调整为与所述面部特征信息匹配，以及将所述待调整表情模型的眼球特征参数调整为与所述眼球特征信息匹配，得到所述目标表情模型。
  5. 根据权利要求1所述的方法，其中，基于所述表情分类确定对应的待调整表情模型包括：
    利用所述表情分类确定对应的多个可选表情模型；
    控制展示所述多个可选表情模型；
    获取用户针对从所述多个可选表情模型中选出所述待调整表情模型的选择指令；
    基于所述选择指令确定所述待调整表情模型。
  6. 根据权利要求5所述的方法，其中，利用所述表情分类确定对应的多个可选表情模型包括：
    根据所述面部图像确定对应的面部比例信息；
    依据所述面部比例信息与所述表情分类对应的表情模型库确定所述多个可选表情模型。
  7. 一种数据处理方法，包括：
    采集用户的面部图像，以及确定用户的眼球特征信息；
    将所述面部图像与所述眼球特征信息发送至数据处理设备，以使所述数据处理设备根据所述面部图像确定对应的表情分类；基于所述表情分类确定对应的待调整表情模型；利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
  8. 一种表情模型确定装置，包括：
    获取模块，用于获取用户的面部图像与眼球特征信息；
    第一确定模块，用于根据所述面部图像确定对应的表情分类；
    第二确定模块，用于基于所述表情分类确定对应的待调整表情模型；
    调整模块，用于利用所述面部图像与所述眼球特征信息调整所述待调整表情模型的对应参数，得到目标表情模型。
  9. 一种电子设备，包括：
    处理器；以及
    存储器，用于存储所述处理器的可执行指令；
    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1-6或权利要求7中任一项所述的方法。
  10. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1-6或权利要求7中任一项所述的方法。
  11. 一种计算机程序产品，包括计算机程序，该计算机程序被处理器执行时实现权利要求1-6或权利要求7中任一项所述的方法。
PCT/CN2022/125570 2021-12-21 2022-10-17 表情模型确定方法、装置、设备及计算机可读存储介质 WO2023116145A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111572188.3 2021-12-21
CN202111572188.3A CN116311407A (zh) 2021-12-21 2021-12-21 表情模型确定方法、装置、设备及计算机可读存储介质

Publications (1)

Publication Number Publication Date
WO2023116145A1 true WO2023116145A1 (zh) 2023-06-29

Family

ID=86831023

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/125570 WO2023116145A1 (zh) 2021-12-21 2022-10-17 表情模型确定方法、装置、设备及计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN116311407A (zh)
WO (1) WO2023116145A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105868694A (zh) * 2016-03-24 2016-08-17 中国地质大学（武汉） 基于面部表情和眼球动作的双模态情感识别方法及系统
CN108537881A (zh) * 2018-04-18 2018-09-14 腾讯科技（深圳）有限公司 一种人脸模型处理方法及其设备、存储介质
CN110096925A (zh) * 2018-01-30 2019-08-06 普天信息技术有限公司 人脸表情图像的增强方法、获取方法和装置
CN110807364A (zh) * 2019-09-27 2020-02-18 中国科学院计算技术研究所 三维人脸与眼球运动的建模与捕获方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105868694A (zh) * 2016-03-24 2016-08-17 中国地质大学（武汉） 基于面部表情和眼球动作的双模态情感识别方法及系统
CN110096925A (zh) * 2018-01-30 2019-08-06 普天信息技术有限公司 人脸表情图像的增强方法、获取方法和装置
CN108537881A (zh) * 2018-04-18 2018-09-14 腾讯科技（深圳）有限公司 一种人脸模型处理方法及其设备、存储介质
CN110807364A (zh) * 2019-09-27 2020-02-18 中国科学院计算技术研究所 三维人脸与眼球运动的建模与捕获方法及系统

Also Published As

Publication number Publication date
CN116311407A (zh) 2023-06-23

Similar Documents

Publication Publication Date Title
US20210383586A1 (en) Artificial intelligence-based animation character drive method and related apparatus
WO2021208648A1 (zh) 虚拟对象调整方法、装置、存储介质与增强现实设备
CN112379812B (zh) 仿真3d数字人交互方法、装置、电子设备及存储介质
CN107103801B (zh) 远程三维场景互动教学系统及控制方法
WO2017215295A1 (zh) 一种摄像机参数调整方法、导播摄像机及系统
WO2022105519A1 (zh) 音效调整方法、装置、设备、存储介质及计算机程序产品
US20220237812A1 (en) Item display method, apparatus, and device, and storage medium
US20230274471A1 (en) Virtual object display method, storage medium and electronic device
CN115909015B (zh) 一种可形变神经辐射场网络的构建方法和装置
CN104536579A (zh) 交互式三维实景与数字图像高速融合处理系统及处理方法
US11595615B2 (en) Conference device, method of controlling conference device, and computer storage medium
US20220375258A1 (en) Image processing method and apparatus, device and storage medium
CN106648098A (zh) 一种自定义场景的ar投影方法及系统
CN112669422A (zh) 仿真3d数字人生成方法、装置、电子设备及存储介质
WO2023116145A1 (zh) 表情模型确定方法、装置、设备及计算机可读存储介质
WO2023207174A1 (zh) 显示方法、装置、显示设备、头戴式设备及存储介质
US20230106330A1 (en) Method for creating a variable model of a face of a person
WO2022253094A1 (zh) 图像生成方法、装置、设备和介质
CN112767520A (zh) 数字人生成方法、装置、电子设备及存储介质
CN114416237A (zh) 显示状态切换方法、装置及系统、电子设备、存储介质
EP4176644A1 (en) Connection assessment system
WO2021041428A1 (en) Method and device for sketch-based placement of virtual objects
US20240013404A1 (en) Image processing method and apparatus, electronic device, and medium
CN115714888B (zh) 视频生成方法、装置、设备与计算机可读存储介质
CN110719455A (zh) 视频投影方法及相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22909463

Country of ref document: EP

Kind code of ref document: A1