WO2023237023A1 - Procédé et appareil de traitement d'image, support de stockage, et dispositif d'affichage monté sur la tête - Google Patents

Procédé et appareil de traitement d'image, support de stockage, et dispositif d'affichage monté sur la tête Download PDF

Info

Publication number
WO2023237023A1
WO2023237023A1 PCT/CN2023/098980 CN2023098980W WO2023237023A1 WO 2023237023 A1 WO2023237023 A1 WO 2023237023A1 CN 2023098980 W CN2023098980 W CN 2023098980W WO 2023237023 A1 WO2023237023 A1 WO 2023237023A1
Authority
WO
WIPO (PCT)
Prior art keywords
eye
image
user
head
display device
Prior art date
Application number
PCT/CN2023/098980
Other languages
English (en)
Chinese (zh)
Inventor
陈才
李伟哲
潘定龙
Original Assignee
广州视源电子科技股份有限公司
广州视源人工智能创新研究院有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州视源电子科技股份有限公司, 广州视源人工智能创新研究院有限公司 filed Critical 广州视源电子科技股份有限公司
Publication of WO2023237023A1 publication Critical patent/WO2023237023A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris

Definitions

  • the present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and head-mounted display device.
  • VR Virtual Reality, virtual reality
  • AR Augmented Reality, augmented reality
  • the user obtains VR or AR content by wearing a head-mounted display device, such as obtaining VR video content or game content.
  • a head-mounted display device such as obtaining VR video content or game content.
  • the head-mounted display device covers the eyes of the user, the interaction between the user wearing the head-mounted display device and ordinary users not wearing the head-mounted display device is affected.
  • Embodiments of the present application provide an image processing method, device, storage medium and head-mounted display device, which can reduce the impact of the head-mounted display device on the interaction between the wearer and ordinary users.
  • the technical solutions are as follows:
  • inventions of the present application provide an image processing method applied to a head-mounted display device.
  • the head-mounted display device includes an external display module and an eye tracking module.
  • the method includes:
  • the stylized eye image is displayed on the external display module.
  • inventions of the present application provide an image processing device applied to a head-mounted display device.
  • the head-mounted display device includes an external display module and an eye tracking module.
  • the method includes:
  • An image acquisition module configured to acquire eye images of the user wearing the head-mounted display device at a predetermined frequency through the eye tracking module
  • a feature extraction unit configured to extract eye features of the user from the eye image
  • a stylized rendering module configured to perform stylized rendering based on the eye characteristics and a predetermined style image, and generate a stylized eye image
  • a display module for displaying the stylized eye image on the external display module.
  • embodiments of the present application provide a computer storage medium that stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the above method.
  • a head-mounted display device which includes:
  • An eye tracking module communicatively connected to the processor, used to obtain eye images of the user wearing the head-mounted display device at a predetermined frequency
  • the processor is configured to: extract eye features of the user from the eye image, perform stylized rendering based on the eye features and a predetermined style image, and generate a stylized eye image;
  • An external display module is communicatively connected to the processor and is used to display the stylized eye image.
  • embodiments of the present application provide a head-mounted display device, including: a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the above method. step.
  • the eye image of the user wearing the head-mounted display device is obtained through the eye tracking module, and the eye image is stylized and displayed on the external display module, which enables ordinary users to interact with the head-mounted device through the external display screen.
  • the wearer can have richer interactions and reduce the impact of the head-mounted display device on the interaction between the wearer and ordinary users;
  • the wearing user can be rendered in a stylized manner based on the eye characteristics of the wearing user, the user's personalized visual output can be achieved.
  • Figure 1 shows a schematic diagram of an application scenario of an image processing method provided according to an embodiment of the present application
  • Figure 2 shows a schematic structural diagram of a head-mounted display device provided according to an embodiment of the present application
  • FIG. 3 shows a schematic flowchart of an image processing method provided according to some embodiments of the present application.
  • Figure 4 shows a schematic diagram after stylization provided according to some embodiments of the present application.
  • Figure 5 shows a schematic flowchart of an image processing method provided according to further embodiments of the present application.
  • Figure 6 shows a schematic flowchart of an image processing method provided according to other embodiments of the present application.
  • Figure 7 shows a schematic diagram of an eye image collected by an eye tracking module provided according to some embodiments of the present application.
  • Figure 8 shows a schematic diagram of identifying eye shapes provided according to some embodiments of the present application.
  • Figure 9 shows a schematic diagram of a gaze point position provided according to some embodiments of the present application.
  • Figure 10 shows a schematic structural diagram of an image processing device provided by an embodiment of the present application.
  • Figure 11 shows a schematic structural diagram of a head-mounted display device provided by an embodiment of the present application.
  • Figure 12 shows a schematic structural diagram of another head-mounted display device provided by an embodiment of the present application.
  • Figure 13 shows a schematic structural diagram of yet another head-mounted display device provided by an embodiment of the present application.
  • Figure 14 shows a schematic structural diagram of an eye tracking module provided by an embodiment of the present application.
  • Figure 15 shows a schematic structural diagram of yet another head-mounted display device provided by an embodiment of the present application.
  • Figure 16 shows a schematic structural diagram of another head-mounted display device provided by an embodiment of the present application.
  • Head-mounted display device Head-mounted display device, such as AR glasses.
  • Stylized rendering It can also be called style transfer, which refers to the rendering process of converting the style of the stylized image to the style of the image to be rendered.
  • Stylized rendering includes the following content: adjustment of details of the eyes (such as the distance between the eyes, eye size, size and color of the eye pupils, etc.) and surroundings (such as the height and shape of the eyebrows, the shape of the nose, etc.); changing makeup (such as eye shadow) , mascara, concealer) or accessories (such as sunglasses, eye nails, etc.); style changes (such as transforming into real or fictional characters, animals, etc.).
  • Eye movement behavior type refers to the type of eye movement behavior of the user wearing the head-mounted display device.
  • the eye movement behavior type can include closing eyes, blinking and opening eyes. Opening eyes can include looking down, looking up, looking to the left, Look to the right, look forward, and eye movement behaviors such as squinting, staring, and raising eyebrows combined with structured light sensors.
  • the deep network is used to process the eye area image to identify the key point positions of the eyes, eyebrows and pupil positions, and then the key point positions are input into the depth map network model for category judgment and output confidence, which includes two types of actions: One type is continuous actions, such as looking around, which not only needs to output category labels, but also needs to output the direction and angle value of distance; the other type is discrete actions, such as squinting, raising eyebrows, etc., which need to output category labels.
  • Eye movement special effects Eye special effects corresponding to the type of eye movement behavior.
  • Figure 1 shows a schematic diagram of an application scenario of an image processing method provided according to an embodiment of the present application.
  • the image processing method can be applied to a wearable head-mounted display device 100 .
  • the head-mounted display device 100 includes an external display module 110 placed outside the head-mounted display device 100 .
  • the external display module 110 can be a display screen 110 , the display screen 110 may be a flat surface or a curved surface, and there may be one or more display areas on the display screen 110 of the head-mounted display device 110, and these areas may be rectangular, circular or other shapes.
  • the display screen 110 may be an OLED (Organic Light-Emitting Diode, organic light-emitting diode), LCD (Liquid Crystal Display, liquid crystal display), MicroLED (Micro-Light-Emitting Diode, micro light-emitting diode), naked-eye 3D screen, etc. used for display Display and naked-eye 3D screen technical implementation solutions include light barriers and lenticular lenses.
  • the display screen 110 can also be a touch screen that supports touch operations, and more interactive functions can be implemented through the touch screen.
  • the logic of the image processing method in the embodiment of the present application can be implemented by installing software and applications in the memory used with the head-mounted display device, or by writing a program in a corresponding device inside the head-mounted display device.
  • the logic of the image processing method in the embodiment of this application can be implemented by installing software and applications in the memory used with the head-mounted display device, or by writing a program in a corresponding device inside the head-mounted display device.
  • Figure 2 shows a schematic structural diagram of a head-mounted display device provided according to an embodiment of the present application.
  • the head-mounted display device 200 includes a sensor 205 , a processor 210 and a memory 215 , an external display module 220 and an internal display module 225 .
  • the head-mounted display device 200 may also include other appropriate modules or units, such as a power supply, a support structure, an input-output unit, etc.
  • the sensor 205 is used to collect data related to the user wearing the head-mounted device 200 or the environment around the head-mounted device 200.
  • the sensor 205 may include multiple sensors.
  • the sensor 205 may be located on the head-mounted device 200 or may be located on the head-mounted device 200. Externally, the sensor 205 sends the collected raw data or processed data to the processor 210 of the head-mounted display device 200 in a wired or wireless manner.
  • Multiple sensors can include: an eye tracking module used to collect user eye characteristics.
  • the eye tracking module includes sensors such as MEMS (Micro-Electro-Mechanical System) sensors or event cameras.
  • the eye tracking module includes a transmitting end and a receiving end.
  • the transmitting end can be an infrared laser transmitting end or other appropriate color laser transmitting end, and the receiving end is a receiver corresponding to the light source.
  • the infrared laser receiving end if the eye tracking module is an event camera, the transmitting end can be an infrared transmitting end, and the receiving end can be an event camera.
  • the infrared transmitting end emits infrared light at a predetermined frequency to illuminate the eyes of the user wearing the head-mounted display device 200, and collects the reflection information of the eyes through the infrared receiving device to generate an eye image of the user.
  • the infrared transmitter and infrared receiver By using the infrared transmitter and infrared receiver, interference in the visible light band can be eliminated.
  • eye tracking can also be achieved in dark environments.
  • the sensor 205 also includes a three-dimensional image sensor.
  • the three-dimensional image sensor is used to acquire a three-dimensional eye image, such as an eye image with z-axis information.
  • the three-dimensional image sensor can be a structured light sensor or a binocular. Stereo depth camera. Taking the three-dimensional image sensor as a structured light sensor as an example, the structured light sensor includes a transmitting end and a receiving end.
  • the infrared transmitting end emits a speckle laser, which is copied and diffused by diffractive optical elements (DOE, Diffractive Optical Elements) to form a
  • DOE diffractive optical elements
  • the dot matrix projected by the infrared transmitter is reflected by the 3D contour of the eye and then received by the receiving end; the receiving end compares the characteristics of the received reflected signal to obtain the three-dimensional contour near the eye.
  • eye images with 3-dimensional information can be generated through feature point matching and reconstruction.
  • the sensor 205 may also include: a physiological data collection sensor for collecting the user's physiological data, such as a sensor for body temperature, heartbeat, blood pressure or blood sugar.
  • the physiological data collection sensor is located on the outer frame of the head-mounted device at a location in contact with the user's skin.
  • Sensors used to sense the movement trend of the head-mounted display device 200 such as accelerometers, magnetometers, IMU (Inertial Measurement Unit, inertial measurement unit) sensors, etc.
  • Sensors used to sense the peripheral environment of the head-mounted display device 200 such as cameras, Depth camera, millimeter wave, ultrasonic and other sensors, etc.
  • the processor 210 and the memory 215 are devices used to store and process signals from various sensors.
  • the processor 210 and the memory 215 can be located in the head-mounted display device 200. Part or all of the processing of the processor 210 and the memory 215 can also be located in the cloud.
  • the processor 210 and memory 215 are connected to the head-mounted display device 200 through wired or wireless communication.
  • the external display module 220 is used to display the stylized rendering of the eye image of the user wearing the head-mounted display device 200.
  • the external display module 220 can be a display screen, and the display content on the external display module 220 can be a static picture or a dynamic video; If the external display module 220 includes two display screens, the same content can be displayed on the two display screens, or different content can be displayed respectively. For example, the rendered left eye and right eye pictures or videos can be displayed on the left and right display screens respectively.
  • the internal display module 225 is used to display AR content or VR content, such as VR video content or VR game content, watched by the user wearing the head-mounted display device 200 .
  • the sensor 205 includes an eye tracking module such as an infrared camera, through which the eye image of the user wearing the head-mounted display device 200 is acquired at a predetermined frequency; and the user's eye image is extracted from the eye image. eye features; perform stylized rendering based on the eye features and the predetermined style image to obtain a stylized eye image; display the stylized eye image on the external display module 220.
  • the external display module 220 is an external display screen, rendering the stylized eye image displayed on the external display module 220. Cartoon eyes, eyebrows and other shapes on the external display.
  • users outside the head-mounted display device can have richer interactions with users wearing the head-mounted display device through the external display screen, which reduces the impact of the head-mounted display device on the interaction between the wearing user and ordinary users. .
  • FIG. 3 shows a schematic flowchart of an image processing method provided according to some embodiments of the present application.
  • the execution subject of the image processing method may be a computing device with computing processing functions, such as the processor of the above-mentioned head-mounted display device.
  • the image processing method includes steps S310 to S340. Below, the image processing method in the example embodiment will be described in detail with reference to the accompanying drawings.
  • step S310 the eye image of the user wearing the head-mounted display device is acquired at a predetermined frequency through the eye tracking module.
  • the head-mounted display device includes an eye tracking module.
  • the eye tracking module is an infrared eye tracking module.
  • the infrared eye tracking module includes an infrared transmitter and an infrared receiver.
  • the infrared transmitter emits infrared light at a predetermined frequency. Illuminating the eyes of the user wearing the head-mounted display device; receiving the reflected light from the user's eyes through the infrared receiver to generate an image of the user's eyes.
  • the predetermined frequency may be 50 Hz or other suitable frequency.
  • the predetermined frequency can be adaptively adjusted according to different scenarios. For example, in a scenario where the wearing user and an ordinary user are interacting, the predetermined frequency is increased; in a scenario where the wearing user is not interacting, the predetermined frequency is reduced.
  • the value of the predetermined frequency can also be set according to the processing performance of the display device.
  • the eye tracking module can also be other appropriate optical sensors such as near-infrared sensors and RGBD cameras with depth information. Or a camera with a light source, etc.
  • step S320 the user's eye features are extracted from the eye image.
  • the eye features may include one or more of gaze point position features, pupil features, eyebrow features, and eye shape features.
  • the gaze point position feature can represent the user's current gaze direction; the pupil feature includes the pupil diameter. If the pupil is smaller than a certain threshold, it is judged that the eyes are closed; the eye shape feature is used to represent the eye shapes of different users.
  • the user's eye features are extracted from the eye image through feature extraction operations, such as extracting the user's pupil status and gaze point position.
  • the feature extraction operation may include: filtering algorithm or structured light algorithm.
  • the filtering algorithm is the Kalman filter algorithm
  • feature extraction is performed on the eye image through the Kalman filter algorithm to obtain the user's Eye characteristics.
  • the specific eye feature recognition algorithm first uses the facial key point extraction algorithm to extract the key points of the eye area, and then uses the multi-camera recognition results to perform 3D fusion, and obtains 3D with higher perceptual accuracy by minimizing the reprojection error.
  • Eye key point information including pupil position, gaze direction, etc.
  • the speckle laser with a specific code is emitted at the transmitting end, it is reflected by the surface of the object. After receiving the reflected signal at the receiving end, it undergoes signal processing such as filtering to form a speckle pattern ( Including information such as spatial phase and light intensity); and then use image processing methods to extract image features under different brightness scenes for comparison. Compare the operation, perform local or global feature comparison, find the position of the feature point in the space of the eye image, and/or compare it with the previous frame of eye image or the previous frames of eye images, so as to obtain the eye image. Images of contours near the head and changes in contours.
  • the feature comparison operation is to construct a cost function of matching confidence at different depths and at different times, such as cosine confidence.
  • the user's eye movements when opening their eyes. For example, if the eye contour in multiple consecutive frames is flat and the height is smaller than the normal size, and the visible pupil area is small, the user can be considered to be squinting; similarly, if the contour height is larger than normal, the user can be considered to be staring. Based on the changes in the user's eyebrow contours, the user's eyebrow movements, such as raising eyebrows, frowning, etc., can also be judged.
  • the filtering algorithm and the structured light extraction algorithm are used as examples to illustrate feature extraction, those skilled in the art should understand that the feature extraction operation can also be other appropriate feature extraction models such as target detection models or Recurrent neural network models, etc., are also within the scope of the embodiments of this application.
  • step S330 stylized rendering is performed based on the eye features and the predetermined style image to generate a stylized eye image.
  • stylized rendering refers to a rendering process that converts the style of a stylized image to the style of an image to be rendered.
  • converting the style of the cartoon eye image to the style of the eye image of the user wearing the head-mounted device is to transfer the style of the cartoon eye image to the eye image of the user wearing the head-mounted device.
  • the style of the predetermined style image includes but is not limited to: cartoon style, anthropomorphic style, realistic style or exaggerated style, etc.
  • the cartoon style may include the style of cartoon eyes, and the anthropomorphic style may include the style of the virtual character's eyes.
  • a predetermined style image that needs to be transformed can be set in advance, for example, in response to the user's style selection operation, a predetermined style image of the corresponding eye image is set.
  • Use styletransferGAN to generate the eye image of the user's eye image to the target style.
  • the input of the network is the user's real-time eye image and the eye image of the target style.
  • the output of the network is 3D eye structure parameters and texture map information, such as Geometric parameters such as eyeballs, pupil size, and eyebrow position.
  • the 3D model can be driven by the key points mentioned above. The number of parameters of the driven model is small, and the style transfer and action simulation of eye movements can be achieved at a low cost of calculation and delay.
  • the predetermined style image is adjusted and rendered to generate a stylized eye image based on the eye characteristics of the eye image of the user wearing the head-mounted device.
  • the eye characteristics may include gaze point position characteristics and pupil characteristics. , one or more of iris position characteristics, eyebrow characteristics, eyelid characteristics, and eye shape.
  • the eye features include eye shape and pupil diameter
  • the predetermined style image is adjusted and rendered to generate a stylized eye image.
  • the rendering of this style can be adjusted to some extent. For example, the distance between the eyes can be adjusted during rendering. One way to achieve this is for the user to manually input the desired distance, and another way to achieve it is by dragging the selection button. Change the size of the spacing.
  • the eye image can be rendered in a stylized manner through a style transformation neural network model, which is used to implement multiple styles of transformation of the eye image.
  • a pre-selected predetermined style image and the acquired user's eye image are input into the style transformation neural network, and the style of the style image is transferred to the user's eye image to generate a stylized eye image.
  • the style transformation neural network includes a style inference network and a style transfer network.
  • the style inference network is used to obtain corresponding style features from the style image, and the style transfer network transfers the acquired style features to the eye image.
  • the style reasoning network is the styletransferGAN mentioned above.
  • the style transfer network is a driving model network. For cartoon styles, 3D eye modeling can be used to achieve direct driving. For real or detailed styles, we use conditionalGAN to generate eyes. Image, the network input is the eye image of the corresponding style and the user's real-time eye feature key points.
  • step S340 the stylized eye image is generated and displayed on the external display module.
  • the stylized rendered eye image is displayed on an external display module, such as an external display screen.
  • Figure 4 shows a schematic diagram after stylization provided according to some embodiments of the present application.
  • the external display module is an external display screen, and 6 groups of eye images generated after different stylized renderings are displayed on the external display screen.
  • the eye image of the user wearing the head-mounted device is acquired through the eye tracking module, and the eye image is rendered in a stylized manner and then displayed on the external display module. It enables external personnel to have richer interactions with users wearing the head-mounted display through the external display, reducing the impact of the head-mounted device on the interaction between the wearing user and ordinary users; on the other hand, because it can be based on the user's eyes.
  • Features provide stylized rendering for the wearer, thereby enabling the user's personalized visual output.
  • the eye features include the user's eye angle
  • the above-mentioned image processing method also includes: collecting an external image of the head-mounted device through an image acquisition module; if the external image includes an external person, identifying the external The person's facial position; dynamically adjusts the user's eye angle based on the external person's facial position.
  • the camera collects the external image of the head-mounted display device, identifies the position of the external person in the external image, determines the facial position of the external person, and dynamically adjusts the rendering on the external screen of the head-mounted device to generate the user's eye angle based on the facial position of the external person. and/or location.
  • 2D rendered eye images with eye angle information can be displayed; if the display is a 3D display, such as a naked-eye 3D display, it can render eye images with 3D special effects from different angles. .
  • the eye angle of the user wearing the head-mounted display device is adjusted based on the facial position of an external person, and the eye angle and/or position of the user can be generated by adjusting the rendering in the external screen of the head-mounted display device according to the external environment. This allows outsiders to intuitively know whether the wearer is communicating with him/her.
  • the head-mounted device further includes a physiological data collection sensor
  • the physiological data collection sensor includes a sensor that measures physiological data such as body temperature, heartbeat, blood pressure, or blood sugar.
  • the physiological data collection sensor is located on the outside of the head-mounted device where it is in contact with the user's skin.
  • the physiological data collection sensor can be located on the outer frame of the head-mounted device where it is in contact with the user's skin.
  • the physiological data stimulation sensor can also be a part of the head-mounted display. Wearable devices outside the device such as bracelets, etc.
  • the image processing method also includes: collecting the user's physiological data through a physiological data collection sensor; extracting the user's physiological characteristics from the collected user's physiological data; and adjusting and rendering the predetermined style image based on the eye characteristics and physiological characteristics.
  • the predetermined style image can be adjusted and rendered based on the heartbeat data and eye characteristics.
  • a sensor that detects heart rate near the headset, such as a pulse wave sensor based on an optical solution. After passing through the signal processing unit, the sensor can output the heart rate at a fixed frequency, such as outputting the detected heart rate at a frequency of 10 Hz. For example, if the heart rate at a certain moment or the average heart rate within a certain period of time is greater than a set threshold, combined with the eye opening action captured by the eye movement sensor, the width of the eye opening in the predetermined style image will be increased.
  • the user's personalized visual output can be further realized.
  • the head-mounted display device further includes: a motion sensor, which may include an accelerometer, a magnetometer, an IMU sensor, etc.
  • the image processing method further includes: collecting motion data of the head-mounted display device through the motion sensor; Based on the motion data of the head-mounted device point, the position offset of the head-mounted device is determined, and based on the eye characteristics and position offset, the predetermined style image is adjusted and rendered.
  • the predetermined style image will be adjusted and rendered based on the positional offset and eye characteristics. For example, if the user takes off the head-mounted display device and puts it on again, the head-mounted display device has a certain positional offset from the previous eye position, then the position of the rendered stylized image is adjusted based on the positional offset.
  • the predetermined style image is adjusted and rendered based on the motion data of the head-mounted display device. Even if the wearer's head-mounted display device is positionally shifted or is repeatedly taken off and put on, the display on the external display module will not be affected. The stability and consistency of stylized images.
  • Figure 5 shows a schematic flowchart of an image processing method provided according to further embodiments of the present application.
  • step S510 the user's eye features are extracted from the eye image of the user wearing the head-mounted display device.
  • step S510 and step S320 are basically similar, and will not be repeated here. Again.
  • step S520 the user's eye movement behavior type is determined based on the user's eye characteristics.
  • the eye movement behavior type includes blinking type and eye opening type.
  • the user's eye movement behavior type can be determined through a neural network model. For example, assuming that the neural network model is a classification network model, the user's eye characteristics are input into This classification network model determines the type of user's eye movement behavior.
  • the eye features include pupil features. Eye key point information at different moments is used and input into the LSTM network based on the graph model in time sequence to detect actions such as blinking and eye closing and the corresponding time. Based on the pupil characteristics of the wearer, determine the closed eye images in the eye images; count the closed eye images in multiple eye images acquired within a predetermined time to obtain the number of closed eye images; if the number of closed eye images is greater than or equal to If the number of eyes-closed images is less than the predetermined threshold, it is determined that the user's eye-movement behavior type is the eyes-open type.
  • the next eye image is obtained; if the next eye image is an open eye image, it is determined that the user's eye movement type is a blink type. That is to say, when the number of closed eye images is greater than or equal to the predetermined threshold, the eyelids are determined to be closed. After the blink is closed, until the next open eye image is collected, it is determined that the eyelids are open. The eyelid closure and eyelid opening together constitute a blink.
  • the number of closed-eye images of the user in the closed-eye state recorded in the cache is LD.
  • step S530 based on the eye movement type and eye characteristics, the predetermined style image is adjusted and rendered to generate a stylized eye image.
  • the user's eye movement behavior type is a blink type
  • the user's blink duration is determined based on the number of eye-closed images; based on the blink duration and the eye characteristics, the predetermined style image is adjusted and rendered to generate a blink animation .
  • the user's eye movement type is an eye-opening type
  • determine the user's gaze point position determines the user's eye movement direction based on the user's gaze point position; adjust and adjust the predetermined style image based on the user's eye movement direction and pupil diameter. Rendering,generating stylized eye images.
  • eye movement directions include, but are not limited to, looking downward, upward, left, right, forward, etc.
  • Figure 4 shows the stylized eye images corresponding to different eye movement directions.
  • step S540 the stylized rendered eye image is displayed on the external display module.
  • step S540 and step S340 are basically similar, and will not be described again here.
  • the eye image of the user wearing the head-mounted device is obtained through the eye tracking module, and the predetermined style image is adjusted and rendered externally according to the eye movement type and eye characteristics. Displayed on the display module, it can display stylized eye images corresponding to different eye movement types, allowing external personnel to have richer interactions with users wearing the head-mounted display device through the external display.
  • the image processing method further includes: determining eye movement special effects corresponding to the eye movement type according to the eye movement type and a preset correspondence, where the preset correspondence is the eye movement type and the eye movement special effect.
  • the corresponding relationship is based on eye characteristics, eye movement behavior types and corresponding eye movement special effects, and the predetermined style image is adjusted and rendered.
  • the correspondence between the eye movement behavior type and the eye movement special effects is stored in the database in advance. If the eye movement behavior type is a blink type, the corresponding blink special effects are obtained. Based on the eye image and the blink special effects corresponding to the blink type, the Pre-styled images are adjusted and rendered.
  • the expression of the eye image can be enhanced and exaggerated eye movement special effects can be output.
  • Figure 6 shows a schematic flowchart of an image processing method provided according to other embodiments of the present application.
  • step S605 the most recently collected eye image is obtained.
  • the eye image of the user wearing the head-mounted display device is collected by the eye tracking module at a predetermined frequency, such as 50 Hz, to obtain the most recently collected eye image.
  • the number LD of closed-eye images is stored in the cache of the memory, and the number LD of closed-eye images is 0 in the initial state.
  • Figure 7 shows a schematic diagram of an eye image collected by an eye tracking module provided according to some embodiments of the present application.
  • step S610 it is determined whether the collected eye image is a closed eye image.
  • whether the acquired eye image is a closed-eye image is determined based on pupil data of the eye image, such as pupil diameter. If the collected eye image is a closed-eye image, that is, no pupil data is collected, then the number of closed-eye images is increased by 1, and return to step S605 to continue to obtain the most recently collected closed-eye image; if the collected eye image is not a closed-eye image. , then proceed to S615.
  • step S615 it is determined whether the number of closed-eye images LD is greater than or equal to a predetermined number.
  • the process proceeds to step S620; if the number of closed-eye images LD is less than the predetermined number, then Proceed to step S635.
  • a predetermined number such as 50
  • the predetermined quantity can be adjusted according to the hardware processing capability of the head-mounted display device. For example, if the hardware processing capability is strong, the predetermined quantity can be set larger; if the hardware processing capability is weak, the predetermined quantity can be set smaller. Small.
  • step S620 the eye closing duration is calculated and a stylized blink animation is generated.
  • the eye-closed duration is calculated based on the number of eye-closed images LD, and the eye images are rendered in a stylized manner based on the different eye shapes identified in the eye images during the blinking process to generate a stylized blinking animation.
  • step S625 the blink animation is displayed through the external display module.
  • a blink animation is displayed on an external display screen of the head-mounted device, thereby outputting a blink special effect close to a real eye state.
  • step S630 LD is set to 0.
  • the number LD of eyes-closed images stored in the cache of the memory is set to 0.
  • step S635 the eye shape is recognized.
  • the eye shape is identified from the user's eye image, for example, the user's eye shape feature is extracted from the eye image through a feature extraction operation.
  • Figure 8 shows a schematic diagram of identifying eye shapes provided according to some embodiments of the present application. Referring to Figure 8, three selection boxes are used to represent the eye shape, iris shape and pupil shape respectively. Among them, the oval box represents the eye selection box of the eye shape, the large round frame represents the iris selection box of the iris shape, and the small round frame represents the pupil shape. pupil selection box.
  • step S640 the gaze point position and pupil diameter are calculated.
  • the user's gaze point location and pupil diameter are calculated based on eye shape characteristics of the wearing user.
  • the user's gaze point position and pupil diameter are calculated based on the three selection boxes of the eye shape in Figure 8
  • the pupil diameter is calculated based on the size of the pupil selection box
  • the pupil selection box is calculated based on the size of the pupil selection box.
  • the position in the eye selection box determines the location of the user's gaze point. For example, determine the pitch angle pitch of the pupil selection box in Figure 9 to be 15 degrees and the yaw angle yaw to be 9 degrees.
  • step S645 stylized rendering is performed.
  • the predetermined style image is adjusted and rendered based on eye characteristics of the eye image of a user wearing the head-mounted display device. For example, assuming that eye features include eye shape, gaze point position, and pupil diameter, a predetermined style image is adjusted and rendered based on the eye shape, gaze point position, and pupil diameter of the eye image of the user wearing the head-mounted display device.
  • the style reasoning network is the styletransferGAN mentioned above.
  • the style transfer network is a driving model network. For cartoon styles, 3D eye modeling can be used to achieve direct driving. For real or detailed styles, we use conditionalGAN to generate eyes. image, whose network input is the eye image of the corresponding style and the user’s real-time eye Key points of local characteristics.
  • step S650 the stylized rendered eye image is output.
  • stylized rendering is performed based on the eye characteristics and the stylized image.
  • eye characteristics such as eye geometry, gaze point position, and pupil diameter
  • stylized rendering is performed based on the eye characteristics and the stylized image.
  • styletransferGAN Since we use styletransferGAN to generate a style-transferred image, and the user's eye image is used as its input, a more natural and user-friendly transferred image can be generated through the neural network, and some features of the user's eyes, such as skin color, eye color, etc., can be retained as much as possible. part shape.
  • Figure 10 shows a schematic structural diagram of an image processing device provided by an exemplary embodiment of the present application.
  • the image processing device 1000 can be implemented as all or part of the device through software, hardware, or a combination of the two.
  • the image processing device 1000 is applied to a head-mounted display device, and the head-mounted display device includes an external display module and Eye tracking module.
  • the image processing device 1000 includes an image acquisition module 1010, a feature extraction module 1020, a stylized rendering module 1030 and a display module 1040. in:
  • Image acquisition module 1010 configured to acquire eye images of the user wearing the head-mounted display device at a predetermined frequency through the eye tracking module;
  • Feature extraction module 1020 used to extract eye features of the user from the eye image
  • the stylized rendering module 1030 is configured to perform stylized rendering based on the eye characteristics and a predetermined style image, and generate a stylized eye image;
  • a display module 1040 is used to display the stylized eye image on the external display module.
  • the head-mounted display device further includes a physiological data collection sensor
  • the device 1000 further includes:
  • a physiological data collection module configured to collect the physiological data of the user through the physiological data collection sensor
  • a physiological feature extraction module used to extract the physiological features of the user from the collected physiological data of the user
  • the stylized rendering module 1030 is also used to:
  • the predetermined style image is adjusted and rendered based on the eye characteristics and the physiological characteristics.
  • the head-mounted display device further includes: a motion sensor
  • the device 1000 further includes:
  • a motion data collection module configured to collect motion data of the head-mounted display device through the motion sensor
  • a position offset determination module configured to determine the position offset of the head-mounted display device based on the motion data of the head-mounted display device point
  • the stylized rendering module 1030 is also used to:
  • the predetermined style image is adjusted and rendered based on the eye characteristics and the position offset.
  • the stylized rendering module 1030 includes: a wind migration unit for adjusting and rendering the predetermined style image based on the eye characteristics.
  • the style transfer unit includes:
  • An eye behavior type determination unit configured to determine the eye movement type of the user based on the eye characteristics
  • An adjustment and rendering unit configured to adjust and render the predetermined style image based on the eye movement type and the eye characteristics.
  • the eye characteristics include pupil characteristics
  • the eye behavior type determination unit is configured to:
  • the number of eye-closed images is greater than or equal to a predetermined threshold, it is determined that the user's eye movement behavior type is the eye-closed type
  • the user's eye movement behavior type is an eye-open type.
  • the eye behavior type determination unit is further configured to:
  • next eye image is an eye-open image, it is determined that the user's eye movement behavior type is a blink type.
  • the adjustment and rendering unit is configured as:
  • the user's eye movement behavior type is a blink type, determine the blink duration of the user based on the number of eye-closed images;
  • the predetermined style image is adjusted and rendered to generate a blink animation.
  • the display module 1040 is configured to:
  • the blink animation is displayed on the external display module.
  • the device 1000 further includes:
  • a fixation point determination module configured to determine the user's fixation point position based on the eye characteristics if the user's eye movement behavior type is an eye-opening type
  • An eye movement direction determination module is configured to determine the eye movement direction of the user based on the position of the user's gaze point.
  • the adjustment and rendering unit is configured as:
  • the predetermined style image is adjusted and rendered based on the eye movement direction and the pupil diameter.
  • the device further includes:
  • a feature determination module configured to determine eye movement special effects corresponding to the eye movement type according to the eye movement type and a preset correspondence, where the preset correspondence is the eye movement type and the eye movement Correspondence between special effects.
  • the adjustment and rendering unit is configured as:
  • the predetermined style image is adjusted and rendered based on the eye characteristics, the eye movement behavior type and the corresponding eye movement special effects.
  • the eye tracking module includes a transmitter and a receiver
  • the image acquisition module 1010 is configured as:
  • the incident light is emitted at a predetermined frequency through the transmitting end to illuminate the eyes of the user wearing the head-mounted display device;
  • the receiving end receives the reflected light of the user's eyes to generate an eye image of the user.
  • the head-mounted display device further includes an image acquisition module, the eye characteristics include the user's eye angle, and the device further includes:
  • An external image acquisition module configured to acquire external images of the head-mounted display device through the image acquisition module
  • a facial position recognition module configured to identify the facial position of the external person if the external image includes an external person
  • a position adjustment module configured to adjust the eye angle of the user based on the facial position of the external person.
  • the image processing device when the image processing device provided in the above embodiments performs the image processing method, only the division of the above functional modules is used as an example. In actual applications, the above function allocation can be completed by different functional modules as needed. , that is, dividing the internal structure of the device into different functional modules to complete all or part of the functions described above.
  • Figure 11 shows a schematic structural diagram of a head-mounted display device provided by an embodiment of the present application.
  • the head-mounted display device 1100 includes: a processor 1110 , a memory, an eye tracking module 1120 and an external display module 1130 . in,
  • the eye tracking module 1120 is communicatively connected to the processor 1110 and is used to obtain eye images of the user wearing the head-mounted display device at a predetermined frequency;
  • the processor 1110 is configured to: extract eye features of the user from the eye image, perform stylized rendering based on the eye features and a predetermined style image, and generate a stylized eye image;
  • the external display module 1130 is communicatively connected with the processor 1110 and is used to display the stylized eye image.
  • the head-mounted display device 1100 further includes:
  • the physiological data collection sensor 1210 is communicatively connected with the processor 1110 and is used to collect the physiological data of the user.
  • the processor 1110 is further configured to: extract physiological characteristics of the user from the collected physiological data of the user, and adjust and render the predetermined style image based on the eye characteristics and the physiological characteristics.
  • the physiological data collection sensor is located on the outer frame of the head-mounted display device at a location in contact with the user's skin.
  • the head-mounted display device 1100 further includes:
  • a motion sensor 1310 is communicatively connected to the processor and used to collect motion data of the head-mounted display device
  • the processor 1110 is further configured to: determine the position offset of the head display device based on the motion data of the head display device point, and calculate the predetermined style image based on the eye characteristics and the position offset. Make adjustments and render.
  • the eye tracking module 1120 includes:
  • the transmitting end 1410 is used to emit incident light at a predetermined frequency to illuminate the eyes of the user wearing the head-mounted display device;
  • the receiving end 1420 is configured to receive reflected light from the user's eyes and generate an eye image of the user.
  • the head-mounted display device 1100 further includes:
  • the external image collection module 1510 is communicatively connected with the processor 1110 and is used to collect external images of the head-mounted display device;
  • the processor 1110 is further configured to: if the external image includes an external person, identify the facial position of the external person; and dynamically adjust the eye angle of the user based on the facial position of the external person.
  • the processor 1110 specifically performs the following operations when performing stylized rendering based on the eye features and the predetermined style image:
  • the style of the predetermined style image is transferred to the eye image.
  • the processor 1110 specifically performs the following operations when performing the adjustment and rendering of the predetermined style image based on the eye characteristics:
  • the predetermined style image is adjusted and rendered based on the eye movement type and the eye characteristics.
  • the eye characteristics include pupil characteristics.
  • the processor 1110 determines the eye movement type of the user based on the eye characteristics, the processor 1110 specifically performs the following: operate:
  • the number of eye-closed images is greater than or equal to a predetermined threshold, it is determined that the user's eye movement behavior type is the eye-closed type
  • the user's eye movement behavior type is an eye-open type.
  • the processor 1110 is also used to:
  • next eye image is an eye-open image, it is determined that the user's eye movement behavior type is a blink type.
  • the processor 1110 specifically performs the following operations when performing the adjustment and rendering of the predetermined style image based on the eye movement type and the eye characteristics. :
  • the user's eye movement behavior type is a blink type, determine the blink duration of the user based on the number of eye-closed images;
  • the predetermined style image is adjusted and rendered to generate a blink animation
  • the displaying the stylized eye image on the external display module includes:
  • the blink animation is displayed on the external display module.
  • the processor 1110 is also used to:
  • the user's eye movement behavior type is the eyes-open type, determine the user's gaze point position based on the eye characteristics
  • the eye movement direction of the user is determined.
  • the processor 1110 specifically performs the following operations when performing the adjustment and rendering of the predetermined style image based on the eye movement type and the eye characteristics. :
  • the predetermined style image is adjusted and rendered based on the eye movement direction and the pupil diameter.
  • the processor 1110 is also used to:
  • the eye movement special effects corresponding to the eye movement behavior types are determined according to the correspondence between the eye movement behavior types and the preset correspondence, where the preset correspondence is the correspondence between the eye movement behavior types and the eye movement special effects.
  • the predetermined style image is adjusted and rendered based on the eye characteristics, the eye movement behavior type and the corresponding eye movement special effects.
  • the head-mounted display device provided in the above embodiments and the image processing method embodiments belong to the same concept, and their implementation process and implementation effects are detailed in the method embodiments, which will not be described again here.
  • Embodiments of the present application also provide a computer storage medium.
  • the computer storage medium can store multiple instructions.
  • the instructions are suitable for the processor to load and execute the image processing method as described in the above embodiments.
  • the specific execution process Reference may be made to the detailed description of the above embodiments, which will not be described again here.
  • This application also provides a computer program product.
  • the computer program product stores at least one instruction.
  • the at least one instruction is loaded by the processor and executes the image processing method as in the above embodiment.
  • the specific execution process can be found in The specific description of the above embodiments will not be described again here.
  • the head-mounted display device 1600 may include: at least one processor 1601, at least one communication module 1604, an input and output interface 1603, a memory 1605, and at least one communication bus 1602.
  • the communication bus 1602 is used to realize connection communication between these components.
  • the input and output interface 1603 may include a display screen (Display) and a camera (Camera).
  • the optional input and output interface 1603 may also include an external display module such as an external display screen.
  • the communication module 1604 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the processor 1601 may include one or more processing cores.
  • the processor 1601 uses various interfaces and lines to connect various parts of the entire head-mounted display device 1600, by running or executing instructions, programs, code sets or instruction sets stored in the memory 1605, and calling data stored in the memory 1605, Perform various functions of the head-mounted display device 1600 and process data.
  • the processor 1601 can use at least one of digital signal processing (Digital Signal Processing, DSP), field-programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). implemented in hardware form.
  • DSP Digital Signal Processing
  • FPGA Field-Programmable Gate Array
  • PLA programmable logic array
  • Processor 1601 can be integrated into One or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU), a modem, etc.
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • the CPU mainly handles the operating system, user interface, and applications
  • the GPU is responsible for rendering and drawing the content that needs to be displayed on the display
  • the modem is used to handle wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1601 and may be implemented by a separate chip.
  • the memory 1605 may include random access memory (RAM) or read-only memory (Read-Only Memory).
  • the memory 1605 includes non-transitory computer-readable storage medium.
  • Memory 1605 may be used to store instructions, programs, codes, sets of codes, or sets of instructions.
  • the memory 1605 may include a program storage area and a data storage area, where the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), Instructions, etc., used to implement each of the above method embodiments; the storage data area can store data, etc. involved in each of the above method embodiments.
  • the memory 1605 may optionally be at least one storage device located remotely from the aforementioned processor 1601. As shown in Figure 16, memory 1605, which is a computer storage medium, may include an operating system, a communication module, an input and output interface module, and an image processing application program.
  • the input and output interface 1603 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1601 can be used to call the image processing program stored in the memory 1605, so that The processor 1601 performs steps in the image processing method according to various exemplary embodiments of the present disclosure.
  • the processor 1601 can be used to call the image processing application stored in the memory 1605, and specifically perform the following operations:
  • the stylized eye image is displayed on the external display module.
  • the processor 1601 performs stylized rendering of the eye image based on the eye characteristics and the predetermined style image, specifically performing the following operations:
  • the predetermined style image is adjusted and rendered.
  • the processor 1601 performs the adjustment and rendering of the predetermined style image based on the eye characteristics, specifically performing the following operations:
  • the predetermined style image is adjusted and rendered based on the eye movement type and the eye characteristics.
  • the eye characteristics include pupil characteristics
  • the processor 1601 performs the following operations when performing the step of determining the user's eye movement behavior type based on the eye characteristics:
  • the user's eye movement behavior type is a blink type
  • the user's eye movement behavior type is an eye-open type.
  • the processor 1601 performs the following operations when performing the adjustment and rendering of the predetermined style image based on the eye movement type and the eye characteristics:
  • the user's eye movement behavior type is a blink type, determine the blink duration of the user based on the number of eye-closed images;
  • the predetermined style image is adjusted and rendered to generate a blink animation
  • Displaying the stylized rendered eye image on the external display module includes:
  • the blink animation is displayed on the external display module.
  • the processor 1601 also performs the following operations:
  • the user's eye movement behavior type is the eyes-open type, determine the user's gaze point position based on the eye characteristics
  • the eye movement direction of the user is determined.
  • the processor 1601 performs the following operations when performing the adjustment and rendering of the predetermined style image based on the eye movement type and the eye characteristics:
  • the predetermined style image is adjusted and rendered based on the eye movement direction and the pupil diameter.
  • the processor 1601 also performs the following operations:
  • the predetermined style image is adjusted and rendered based on the eye characteristics, the eye movement behavior type and the corresponding eye movement special effects.
  • the eye tracking module includes an infrared transmitter and an infrared receiver.
  • the processor 1601 performs the step of obtaining the eyes of the user wearing the head-mounted display device through the eye tracking module. image, specifically perform the following operations:
  • the infrared transmitting end emits infrared light at a predetermined frequency to illuminate the eyes of the user wearing the head-mounted display device;
  • the infrared receiving end receives the reflected light of the user's eyes to generate an eye image of the user.
  • the head-mounted display device further includes an image acquisition module such as a camera
  • the eye features include the user's gaze position
  • the processor 1601 also performs the following operations:
  • the external image includes an external person, identifying the facial position of the external person
  • the user's gaze position is adjusted based on the external person's facial position.
  • the above is a schematic solution of a head-mounted display device according to the embodiment of this specification. It should be noted that the technical solution of the head-mounted display device and the technical solution of the above-mentioned image processing method belong to the same concept. For details that are not described in detail in the technical solution of the head-mounted display device, please refer to the technical solution of the above-mentioned image processing method. description of.
  • plural means two or more unless otherwise specified.
  • “And/or” describes the relationship between related objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. The character “/” generally indicates that the related objects are in an "or” relationship.
  • the program can be stored in a computer-readable storage medium.
  • the program can be stored in a computer-readable storage medium.
  • the process may include the processes of the embodiments of each of the above methods.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Multimedia (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Des modes de réalisation de la présente demande divulguent un procédé et un appareil de traitement d'image, un support de stockage, et un dispositif d'affichage monté sur la tête. Le procédé consiste à : acquérir, au moyen d'un module de suivi d'œil à une fréquence prédéterminée, une image d'œil d'un utilisateur qui porte le dispositif d'affichage monté sur la tête ; extraire des caractéristiques d'œil de l'utilisateur à partir de l'image d'œil ; effectuer un rendu stylisé sur la base des caractéristiques d'œil et d'une image de style prédéterminée pour générer une image d'œil stylisée ; et afficher l'image d'œil stylisée sur un module d'affichage externe. Selon la solution technique des modes de réalisation de la présente demande, l'influence du dispositif d'affichage monté sur la tête sur l'interaction entre un porteur et un utilisateur ordinaire peut être réduite, et la sortie visuelle personnalisée de l'utilisateur peut être réalisée.
PCT/CN2023/098980 2022-06-08 2023-06-07 Procédé et appareil de traitement d'image, support de stockage, et dispositif d'affichage monté sur la tête WO2023237023A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210639122.XA CN117234325A (zh) 2022-06-08 2022-06-08 图像处理方法、装置、存储介质及头显设备
CN202210639122.X 2022-06-08

Publications (1)

Publication Number Publication Date
WO2023237023A1 true WO2023237023A1 (fr) 2023-12-14

Family

ID=89081348

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/098980 WO2023237023A1 (fr) 2022-06-08 2023-06-07 Procédé et appareil de traitement d'image, support de stockage, et dispositif d'affichage monté sur la tête

Country Status (2)

Country Link
CN (1) CN117234325A (fr)
WO (1) WO2023237023A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117978864A (zh) * 2024-03-18 2024-05-03 东莞莱姆森科技建材有限公司 一种具备信息推送功能的智能镜子及镜柜

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110531516A (zh) * 2019-07-12 2019-12-03 上海大学 一种头戴式眼动跟踪操作辅助的智能装置
US20200167968A1 (en) * 2018-11-28 2020-05-28 International Business Machines Corporation Displaying a virtual eye on a wearable device
CN112989904A (zh) * 2020-09-30 2021-06-18 北京字节跳动网络技术有限公司 风格图像生成方法、模型训练方法、装置、设备和介质

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200167968A1 (en) * 2018-11-28 2020-05-28 International Business Machines Corporation Displaying a virtual eye on a wearable device
CN110531516A (zh) * 2019-07-12 2019-12-03 上海大学 一种头戴式眼动跟踪操作辅助的智能装置
CN112989904A (zh) * 2020-09-30 2021-06-18 北京字节跳动网络技术有限公司 风格图像生成方法、模型训练方法、装置、设备和介质

Also Published As

Publication number Publication date
CN117234325A (zh) 2023-12-15

Similar Documents

Publication Publication Date Title
US11598957B2 (en) Facial expressions from eye-tracking cameras
EP3807710B1 (fr) Affichage à réalité augmentée avec fonctionnalité de modulation d'image
US11656680B2 (en) Technique for controlling virtual image generation system using emotional states of user
CN112034977B (zh) Mr智能眼镜内容交互、信息输入、应用推荐技术的方法
US20130335405A1 (en) Virtual object generation within a virtual environment
CN109086726A (zh) 一种基于ar智能眼镜的局部图像识别方法及系统
TW201228380A (en) Comprehension and intent-based content for augmented reality displays
AU2021290132C1 (en) Presenting avatars in three-dimensional environments
US11645823B2 (en) Neutral avatars
KR20120060978A (ko) 시선 위치 추적 기반의 3차원 인간-컴퓨터 상호작용 장치 및 방법
WO2023237023A1 (fr) Procédé et appareil de traitement d'image, support de stockage, et dispositif d'affichage monté sur la tête
JP2022537817A (ja) 動的オクルージョンのための高速の手のメッシュ化
JP7128473B2 (ja) キャラクター表示方法
US20230171484A1 (en) Devices, methods, and graphical user interfaces for generating and displaying a representation of a user
Bhaskaran et al. Immersive User Experiences: Trends and Challenges of Using XR Technologies
WO2023096940A2 (fr) Dispositifs, procédés et interfaces utilisateur graphiques pour générer et afficher une représentation d'un utilisateur
KR20240091224A (ko) 사용자의 표현을 생성 및 디스플레이하기 위한 디바이스들, 방법들, 및 그래픽 사용자 인터페이스들
CN117765049A (zh) 对齐用户表示

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23819188

Country of ref document: EP

Kind code of ref document: A1