CN112181152B

CN112181152B - Advertisement pushing management method, device and application based on MR (magnetic resonance) glasses

Info

Publication number: CN112181152B
Application number: CN202011267167.6A
Authority: CN
Inventors: 陈涛; 朱若晴
Original assignee: Magic Scorpion Technology Wuhan Co ltd
Current assignee: Magic Scorpion Technology Wuhan Co ltd
Priority date: 2020-11-13
Filing date: 2020-11-13
Publication date: 2023-05-26
Anticipated expiration: 2040-11-13
Also published as: CN112181152A

Abstract

The invention belongs to the technical field of advertisement pushing, and discloses an advertisement pushing management method, equipment and application based on MR (magnetic resonance) glasses, wherein a display module displays images to a user by using HMD equipment; the image sensing module is used for acquiring image data of surrounding and/or front real scenes; the eye tracking module is used for detecting the sensor data of the gaze direction of each eye of the user; the position module is used for acquiring current position data of the equipment; the voice recognition module is used for detecting and recognizing voice commands of a user by utilizing audio hardware; the motion control module is used for measuring and acquiring user motion data by utilizing one or more motion sensors; according to the invention, the scene is detected and identified through the hardware forms such as position positioning, time, acceleration sensor and the like, so that the stepped scene identification is realized, the effect is accurate, and the power consumption is greatly lower.

Description

Advertisement pushing management method, device and application based on MR (magnetic resonance) glasses

Technical Field

The invention belongs to the technical field of advertisement pushing, and particularly relates to an advertisement pushing management method, equipment and application based on MR (magnetic resonance) glasses.

Background

At present, the AR/MR/XR glasses have privacy security risks, the front-facing camera is required to acquire and analyze the current surrounding environment of a user in the advertisement/content recommendation technology of the current MR/AR/VR intelligent glasses, so that advertisement content related to a scene is pushed, the front-facing camera is required to be started at any time in the technical link, the camera which records data at any time can unconsciously infringe the privacy of other people, and the privacy problem can become a commercial first enemy of an AR advertisement recommendation system in the future. Not only start the battery duration of camera to MR glasses all the time like this also brings not little challenge, reduces the whole experience of MR glasses.

The internet age of information explosion is realized nowadays, a large amount of advertisement information is invisible, people's life is filled, and people are increasingly bored with useless advertisements. As is well known, accurate marketing is an important direction of various large marketing means, but along with further improvement of the living standard of people, the personalized demands of consumers on products are continuously highlighted, the difficulty of accurate marketing is also continuously improved, the marketing means of the traditional outdoor media advertisement are difficult to quantify, and the final effect is difficult to guarantee. The above problems can be predicted to be more serious in the future smart glasses age.

Compared with an advertisement pushing method based on eye tracking technology applied to AR intelligent glasses of the invention patent CN 108960937, the method is used for pushing information only based on user interests, and the pushed information is incomplete.

In some of the invention patents disclosed at home and abroad at present, the interest degree of the user on the things in front of the current vision is judged by collecting and analyzing the physiological data of the user, and then content information related to the things is presented to the user according to the interest degree. However, the above-described method is inaccurate. The reason is that when a person sees something interesting/aversive in front of the vision, a biological reaction (such as heart beat acceleration, skin temperature change, electroencephalogram change, etc.) may be caused, but the biological reaction is delayed, the interesting event is about to elapse, it is possible that when the MR glasses judge that the user is interested in something, and then the front camera is started to acquire an interesting image for recognition analysis, the interesting object has walked away or disappeared, it is conceivable that only the local image of the interesting object at the original position obtained by the camera is invalid, and there is no chance of restarting for the second time. For example, a user wearing MR glasses walks on the street, suddenly a smart car is designed to drive in front of the eyes, the eyes of the user catch up with the car, the pupils are slightly open, and the heartbeat accelerates, but the car is far away just when the MR eyes start the camera and recognize the car, and the MR glasses cannot acquire effective car images.

Through the above analysis, the problems and defects existing in the prior art are as follows: the prior AR/MR/XR glasses push information with privacy security risks; the battery endurance capacity is low due to the fact that the information is obtained by the camera to push advertisements by the MR glasses, and the overall experience of the MR glasses is reduced; meanwhile, the existing advertisement pushing method by using MR glasses is incomplete in information pushing and inaccurate in data of interest of users.

The difficulty of solving the problems and the defects is as follows:

in order to solve the above-mentioned privacy security risk due to the use of the camera, the same technical scheme is used to identify the current scene of the user?

In order to provide a better product experience for the user, what technical solutions are used to identify the status of the user to ensure that the user does not become objectionable to the pushing of advertisements, and also to ensure that the advertisements are popped up in a secure situation to prevent the user from creating a security risk due to being attracted to the attention.

In practice AR virtual advertising would be space-covered, then by what technical algorithms to filter out advertising information that some users do not like?

In order to better make the computing mechanism solve the emotion of the user and further provide more humanized content information, the sensor needs to acquire the biological information of the user, but the biological information acquisition of the user is delayed, so how to timely acquire the camera event corresponding to the biological information time?

The meaning of solving the problems and the defects is as follows: in the embodiment of the invention, the current scene of the user is comprehensively analyzed through scene identification, object detection and space environment identification, so that privacy security risks are eliminated; the acceleration sensor is used for identifying the current motion state and scene of the user to analyze the proper push time.

In the invention, the front-end camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses. When the biological data of the user reach the threshold value, the cached images/videos/audios before and after the time of the 'biological index of interest' of the user are retrieved and analyzed, and the AI recognition technology recognizes objects or events in the images/videos, recognizes the saliency of the images and simultaneously and frames the image objects.

In summary, the invention provides a more humanized advertisement pushing mode with better user experience, balances the conflict between consumers and merchants, and provides an innovative method for effective flow rate change of future AR/VR glasses. Effectively promote the development of AR/VR industry.

Disclosure of Invention

Aiming at the problems existing in the prior art, the invention provides an advertisement pushing management method, equipment and application based on MR glasses.

The invention is realized in such a way that the advertisement pushing management method based on the MR glasses comprises the following steps:

step one, identifying a scene, and confirming the current scene of a user through an MR (magnetic resonance) glasses positioning system;

step two, the current motion state of the user is identified, and whether the current motion state of the user is suitable for ejecting/pushing content information is judged through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

Performing object detection, namely performing AI object recognition on physical world image data shot by a front-end camera by using a local computing power operation object detector of the MR glasses, simultaneously recognizing and framing a plurality of specific objects in the images by using an AI recognition technology, and performing recognition on interesting objects in the images by using a pre-trained feature database by using a local processor of an MR glasses terminal, wherein the images shot by the front-end camera of the MR glasses are configured to be consistent with physical world image pictures seen by eyes of a user; after the object is detected, setting a boundary block diagram at the periphery of the object according to the image recognition and image segmentation results;

step four, identifying the space environment, wherein the space analysis module analyzes the current physical environment of the user according to the detection results of the plurality of objects in the step three, and further confirms the identified environment and privacy security in the step one according to the environment;

step five, content screening is displayed, advertisement information preset and promoted by the entity store merchant is received according to object identification and classification results, each object attribute/label in the graph is matched with user personal information, and object image information which is disliked/not needed by the user is screened out, or emotion content which is not suitable for the current user is screened out;

Step six, presenting information prompt, and rendering and presenting the content prompt notice screened in the step four by an MR glasses optical display in the form of a holographic image;

selecting the object subjected to information screening, intercepting the partial image selected by the frame after the object detection in the step three from the large image, uploading the partial image to a cloud advertisement server, carrying out matching identification on the received partial image of the object by the cloud server, searching relevant content information in a database, and returning a search result to a terminal MR (magnetic resonance) glasses;

and step eight, the MR glasses download the server content data to the storage, and the simple advertisement content information is rendered and presented by the MR glasses optical display in the form of a hologram.

In the first step, positioning location information is obtained through MR intelligent glasses positioning technology including but not limited to GPS/beidou satellite, wiFi module, and communication signal base station, and the advertisement recommendation system is started in shopping malls, commercial streets, and places with consumption behaviors for entertainment.

Further, the first step further includes:

judging whether the current moment is suitable for popup content information or not according to daily work and rest habits of a user, wherein the method specifically comprises the following steps:

the system calls the matching of the current time and the user work and rest rule data to judge whether the advertisement recommendation system can be started;

For privacy security risks, a front camera is needed to acquire and analyze the current surrounding environment of a user in the advertisement/content recommendation technology of the current MR/AR/VR intelligent glasses, so that advertisement contents related to scenes are pushed.

Further, in the second step, data of acceleration sensors and gyroscopic sensors are collected; the acceleration sensor data and the gyroscope sensor data respectively comprise data of three sub dimensions of an x axis, a Y axis and a z axis;

the acceleration sensor and the gyroscope sensor can be arranged in intelligent electronic equipment, and the intelligent electronic equipment comprises intelligent MR glasses; the data to be identified are sensor data acquired by an acceleration sensor and a gyroscope sensor; the acceleration sensor and the gyroscope sensor can record various data in the using process of the intelligent MR glasses.

In the third step, the physical world image data shot by the front camera is subjected to a preprocessing program, and the filtering and the correction of the image data are executed; an image pre-processor segments image data into regions based on visual and/or depth image data;

AI recognition and classification by an object detector of MR glasses of objects in a real world image and determining a position and/or pose of the real world objects in a scene observed by the MR device;

Further, various known computer object recognition techniques are utilized to identify objects in images of real world scenes; object recognition uses an appearance-based method, comparing an image with a reference image of a known object to identify the object; the appearance-based method comprises edge matching, gray matching and receiving a histogram of field response;

object recognition uses feature-based methods that rely on matching object features and image features; the feature-based method comprises gesture clustering, geometric hashing, scale-invariant feature transformation and interpretation tree;

the one or more object detection/classification models include: parameters or random forests for a machine learning algorithm, one or more decision trees, are provided to real world objects captured in image data of the machine learning algorithm through training detection and/or classification.

Further, the step four specifically includes:

the spatial analysis module is configured to identify/classify and determine boundaries of physical space observed by the MR device; the space analysis module is configured to compare the AI object recognition result of the object detector with the scene feature library and analyze whether the current environment of the user is a privacy security risk area; each or several object combinations in the scene feature library correspond to an environmental scene; identifying different sites by one or more characteristic objects; if the current environment of the user is a privacy security high risk area, the system closes the camera or prohibits the camera from recording image data such as photos/videos, and if the current environment of the user is a privacy security low/no risk area, the camera is continuously started, a content recommendation system is started, the environment identified in the step one through a positioning technology is secondarily confirmed, and the privacy sensitive area is eliminated.

Further, the fifth step further includes: after the MR glasses recognize/classify the object, the object is matched with personal information of the user according to the object characteristics, and a part of the information is filtered out and displayed, wherein the personal information comprises, but is not limited to, a personal interest library, an interest tag, a user portrait, a historical eye-gaze point interest library, a user name, a personal data picture, contact information, date of birth, gender, marital status, family status, work, educational background, visual preference, interest or other demographic information.

Further, the step six further includes:

the system presents a prompt notification at the target object location, the notification being one or more of: pop-up notifications, voice notifications, vibrations, notification text or icons in an object-based content sharing user interface, notification messages in a social media service, text messages, upon addition of a new content item, the system will cause the user's device to sound or vibrate;

the user individually configures the type of notification to receive each object or class of objects associated with the user; the user individually configuring whether to receive notifications for each object and/or class of objects associated with the user;

A notification module is also used for presenting a prompt notification for the target object position, the notification module enables the OS of the MR glasses system to match the object recognition result with the application program, and matches the notification with the application program object registered in the OS, so that the object is identified as the target of the notification; the OS performs this step by, for example, comparing the context Identifier (ID) contained in the notification payload with the context IDs of all objects registered in the system, and finding an object with a matching context ID; after identifying the object as the target of the notification, the operating system may call a special notification handler associated with the application of the object, but separate from the main process of the application;

the notification handler is an executable file that is different from the main executable file of the application; the notification handler may then execute code to process and present the notification in context with respect to the object in the system UI.

Further, the step seven further includes:

searching according to the position information, scene information and local images returned by the MR glasses of the terminal at the cloud server, and searching whether advertisement information corresponding to the target object exists in a database or not; the cloud server is configured to be an advertisement delivery platform for receiving/collecting advertisement delivery demands of merchants in physical stores, and the merchants can set introduction contents, coupons, discount information and any advertisement contents on the advertisement delivery platform and even can link with self-owned applets of the merchants.

Further, the step eight further includes:

the system determines a direction and a three-dimensional position to display holographic content based on the location of the associated physical object in the environment map; when a certain holographic content is displayed on a physical object-a chair, the position of the surface or the vicinity of the chair in the environment map is determined as the three-dimensional position of the holographic content display; the position and posture of the imaging device acquired from the environment map generating unit, and then converting the determined three-dimensional position into a planar position and a holographic content depth on the optical imaging screen, the calculated position in the input image to generate an output image to be displayed on an optical display of the MR glasses;

the user movement causes a change in the user's perspective to the physical object, the electronic device detects the user's movement and adjusts the user's view of the asset to correspond to the user's movement; detecting movement of the user by a sensor or by analyzing an image captured by the camera;

adjusting a user's perspective on the asset using simultaneous localization and mapping techniques; in SLAM, a set of points is tracked by successive camera frames; using these trajectories, triangulating the 3D position of the points, while calculating the camera pose of the viewpoint using the estimated point positions; using the 3D position and camera pose, the user's asset view is also adjusted.

Further, the interaction instruction of the user specifically includes: eye movement interaction, gesture recognition interaction, peripheral controller, voiced/unvoiced speech recognition, head movement interaction;

(1) The eye movement tracking device of the MR glasses captures eye movement data, detects eye movements, calculates pupil size, iris images, eye jump track, gazing time length and the like of eyes, and when the gazing time length exceeds a certain time length, the eye movement tracking device is regarded as performing behavior movements corresponding to eyeballs at a certain position, for example, performing movements of gazing corresponding to clicking operations (eye behavior movements) on UI buttons in a telescopic mode in an interactive interface, namely displaying content information;

(2) The gesture recognition module of the MR glasses detects that the hand makes a specific action, for example, the gesture recognition device receives the motion information of the hand of the user or recognizes the specific shape of the hand, and the gesture recognition module is associated with control options such as up, down, left and right sliding, zooming in, zooming out, clicking and closing, so that content information is displayed;

(3) The method comprises the steps that through a mobile controller containing wireless signal transmission of a control device, a key-press controller, a wearable glove and a fingerstall controller are held, control signals are transmitted to MR glasses, and a UI button is clicked to enter a telescopic mode through an operation interaction interface, wherein the telescopic mode comprises a 6DoF acceleration sensor tracking technology and a 6DoF electromagnetic tracking technology handle;

(4) Receiving the voice with sound or soundless sound sent by the user through the voice recognition device with sound or soundless sound of the MR glasses, analyzing the voice instruction of the user through the system, and controlling the equipment to display the content information;

(5) The button of the interactive interface is selected through the head tracking device, for example, the motion of the head of the user is calculated through an acceleration sensor, a gyroscope and a magnetometer of the MR glasses, a cursor fixed relative to the head is arranged in the center of the visual field of the MR eyes, and the head is moved to control the cursor selection position to perform click confirmation with the holographic UI interface, namely, content information is displayed.

Further, whether the current object is interested is judged by identifying the biological information of the user, then the system pushes the content information to the user, and instant information is obtained by caching video, and the advertisement pushing management method based on the MR glasses further comprises the following steps:

step S211, identifying a scene: confirming the current scene of the user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for popup content information or not through daily work and rest habits of the user;

step S212: identifying a current motion state of a user: judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

Step S213: object detection: performing AI object recognition on physical world image data shot by a front camera by using a local computing power operation object detector of the MR glasses, and recognizing and framing a plurality of objects in the image by using an AI recognition technology;

step S204: and (3) spatial environment identification: the space analysis module analyzes the current physical environment of the user according to the plurality of object detection results of S213, and further confirms the environment and privacy safety identified in S101 according to the environment;

step S215, circularly recording and caching: the front camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses;

step S216, detecting the current biological information of the user, including eye movement behavior data, brain electrical data, heartbeat, blood pressure and respiratory rate, and judging whether the user is possibly interested in the current image.

Step S217: retrieving and analyzing cached images/videos/audios before and after the time of the 'biological index of interest' of the user, identifying objects or events in the images/videos by an AI (analog to digital) identification technology, identifying the saliency of the images, and simultaneously, selecting the image objects in a frame mode;

step S218, the MR glasses intercept the partial images selected by the frames after the object detection in the step S213 from the large images, upload the partial images to the cloud advertisement server, the cloud server carries out matching identification on the received partial images of the objects, search relevant content information in a database, and the search result is returned to the terminal MR glasses; wherein the content information includes text, pictures, audio, video, and applications; meanwhile, a content prompt notice is marked and presented for a target object or an event position, and the content prompt notice is rendered and presented by an MR glasses optical display in the form of a hologram;

Step S219, the MR glasses wait for the interactive behavior of the user on the content prompt notice, and simultaneously the MR glasses download the server content data to the cache; if the interaction instruction of the user is received, the content information returned by the server is further presented; if the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

And step S220, presenting the content information after receiving the interaction instruction of the user.

Further, the step S215 loops recording and caching, further includes:

the front-end camera of the MR glasses circularly records video, photos and audio in a period of time, and the video, the photos and the audio are cached in the local memory of the MR glasses. Specifically recording video/audio to a buffer; the emotion/cognitive state triggered recording system writes the video/audio data to the buffer as it is captured by the camera and microphone. The buffer may be implemented as a circular buffer, such as a buffer that stores data for a recent period of time in a circular manner, the storage capacity of the buffer may be manually set, and when the capacity is full, the video at the earliest time is circularly deleted, and new video data is written.

Further, the step S216 detects current biometric information of the user, and further includes:

eye movement behavior data, brain electrical data, heart beat, blood pressure and respiratory rate, and judging whether the user is possibly interested in the current image. In particular, a sensor data analysis module that detects and receives sensor data receives data from one or more sensing devices. The sensor data may include, but is not limited to, galvanic skin response data, skin temperature data, galvanic skin data, brain wave activity, heart electrical activity data, eye movement data, facial expression data, pupil dilation and/or constriction data, voice volume data or voice rate indicating data.

Further, the step S217 extracts the buffered video before and after the emotion stimulating time for image recognition, and further includes:

further, the method further comprises the steps of:

the emotion/cognitive state triggered recording system comprises a buffer area, a sensor data analysis module, a recording module, a staring detection module and a video segment storage area;

the buffer is configured to store video and audio data when video and audio data is received from the camera and microphone;

the sensor data analysis module receives sensor data from the sensing device and analyzes the received sensor data to determine an emotional or cognitive state of the user; the sensor data analysis module provides available data indicative of the emotional or cognitive state of the user;

The recording module determines whether to record the video clip based on data indicative of an emotional or cognitive state of the user. The recording module is configured to begin recording video clips based on changes in the emotional or cognitive state of the user. The recording module may be configured to begin recording the video clip when the emotional or cognitive state of the user changes from a neutral state to a non-neutral state; the recording module is configured to begin recording the video clip when a value representing an emotional or cognitive state of the user exceeds a threshold;

the gaze detection module tracks a user's gaze to determine a direction of the user's gaze; the gaze detection module may be configured as a video overlay that includes colored points, highlighting or other visual indicators regarding the gaze direction of the user at the time the video was captured;

the video clip store is configured to store video clips recorded based on emotional or cognitive states of the user. When the recording module starts recording, copying the video and audio data stored in the buffer to a new video segment, which is stored in the video segment memory; the recording module directs the additional video and audio data to be recorded to the video clip as described above; the video clip store also stores metadata associated with the video clip, which may include, but is not limited to, indicators of the emotion or cognitive state of the user triggering the recording, one or more indicators of the emotion of the user; or a cognitive state at the time of video recording, and for providing a visual indicator.

Further, whether the current object is interested is judged by identifying the limb actions of the user, and the advertisement pushing management method based on the MR glasses further comprises the following steps:

s301, identifying a scene: confirming the current scene of the user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for popup content information or not through daily work and rest habits of the user;

s302: identifying a current motion state of a user: judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

s303: privacy security validation: carrying out AI identification on the image shot by the front camera by utilizing the local calculation force of the MR glasses, analyzing the current scene of the user by the attribute of a plurality of objects in the environment, judging whether the personal privacy potential safety hazard exists in the scene, and executing S104 if the personal privacy potential hazard does not exist in the scene;

s304, circularly recording and caching: the front camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses;

s305, detecting the current body motion of a user, and detecting the body motion of the user by using an acceleration sensor, a gyroscope, an IMU, an eyeball tracking device, a gesture recognition device and the like of the MR glasses;

S306: retrieving and analyzing cached images/videos/audios before and after the time of the user's interesting physical behaviors', and identifying objects or events in the images/videos by an AI identification technology; the user 'S body action and the information of the image, video and audio recognition event are combined to judge the interest degree of the user in the event, if the user' S body action is judged to be interested, the S307 is executed;

s307, the MR glasses intercept the partial images selected by the frames after the object detection in the S303 from the large images, upload the partial images to the cloud advertisement server, the cloud server carries out matching identification on the received partial images of the objects, search relevant content information in the database, and return the search result to the terminal MR glasses. Wherein the content information includes text, pictures, audio, video, and applications. Meanwhile, a content prompt notice is marked and presented for a target object or an event position, and the content prompt notice is rendered and presented by an MR glasses optical display in the form of a hologram;

s308, the MR glasses wait for the interaction of the user on the content prompt notification, and meanwhile the MR glasses download the server content data to the cache. And if the interaction instruction of the user is received, the content information returned by the server is further presented. If the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the visual field picture of the user, and emptying the cache information;

And S309, presenting the content information after receiving the interaction instruction of the user.

Further, the step S305 of detecting the current body motion of the user includes:

eye movement is detected by an eye movement tracking device: the saccade distance exceeds/equals a preset threshold, and/or the smooth movement behavior duration of the eye exceeds/equals a preset threshold, and/or the movement speed of the eye exceeds/equals a preset threshold, and/or the eye suddenly stops moving unordered, and/or the fixation duration of the eye at a certain point exceeds/equals a preset threshold, and/or the eye reciprocates between two points/ranges, and/or the eye tremors behavior, and/or the change of the pupil diameter of the eye exceeds/equals a preset threshold, vestibular eye movement, converging/diverging movement, smooth chasing, long saccades and correcting saccades; in normal daily activities, the eye can be tracked for voluntary and involuntary movements; the user's intent is discerned when the user's eyes routinely perform both types of movements. This requires distinguishing between voluntary and involuntary eye movements for communicating intent;

detecting head movement by a head tracking device: in further aspects, the eye signals may be combined with other modalities to convey user intent; the eye signal may be combined with voluntary head movements as part of the eye signal "language"; detecting the speed, acceleration and rotation angle of a user in the process of turning the head;

The head position tracking system includes position and motion sensors (e.g., compasses, gyroscopes), accelerometer light sources and/or other devices for acquiring the position, orientation and motion of the MR glasses, detected by detecting motion of the entire scene in an externally facing camera; head position information may be received from a head position tracking system; head position information is provided to a head position tracking system of an eye movement interaction intent prediction model. Forming an Inertial Measurement Unit (IMU) on the MR glasses together; the head position information may help determine the head movement, rotation or direction of the user;

the motion tracking device detects body motion: detecting the dodging, jumping and trembling body behaviors of a user; to the extent that the head tracking device of an MR eye can detect body movement on behalf of the movement tracking device, body movement can also be detected by any wearable smart hardware such as a smart watch, smart bracelet, controller handle with spatial positioning, etc.

Another object of the present invention is to provide an MR glasses-based advertisement push management apparatus comprising:

a display module for presenting images to a user using the HMD device;

An image sensing module for acquiring image data of a surrounding and/or front real scene using one or more externally facing image sensors, imaging cameras or depth infrared cameras and other devices;

an eye tracking module comprising one or more scintillation sources, an image sensor, and a MEMS scanning mirror; sensor data for detecting gaze directions of each eye of the user;

the position module is used for acquiring the current position data of the equipment;

the voice recognition module is used for detecting and recognizing voice commands of users by utilizing audio hardware;

a motion control module for measuring and acquiring user motion data using one or more motion sensors;

the control module comprises a logic unit, a data holding unit and a communication unit; the device is used for data storage and communication control;

the biological characteristic acquisition module is used for acquiring brain electrical activity, heart electrical activity, voice volume, speech speed and other biological characteristic data of the user by using one or more biological characteristic sensors;

the buffer module is used for recording the triggering video/audio data supporting emotion/cognitive state;

an input/output device includes one or more input interfaces for signal or data interaction with the electronic device.

Further, the control module includes:

a logic unit to execute instructions and communicate with one or more processors;

a data holding unit for storing data using one or more storage devices;

and the communication unit is used for carrying out system communication with other computers.

Further, the MR glasses-based advertisement push management device is provided in a head-mounted display device HMD worn by a user, and tracks an eye gaze direction, and a user head gaze direction and head pose motion by:

a sensor included in the HMD converts the detected head gaze direction and/or motion defining gaze gestures into user input in a virtual environment generated by the HMD.

Further, the MR glasses comprise a position subsystem arranged to determine a current position based on signals received from a navigation satellite system; the location subsystem is arranged to determine a location based on radio frequency signals identifying the transmitting device and the location determined for such device;

the MR glasses include audio hardware including one or more microphones arranged to detect sound;

the MR glasses comprise one or more motion sensors arranged to measure and report motion data;

The MR glasses further include a controller including a logic system, a data retention system, and a communication system;

the logic subsystem includes one or more processors that are configured to execute instructions and communicate.

The data-holding subsystem includes one or more storage devices and/or one or more storage devices;

the data-holding subsystem also includes one or more media on which instructions executable by the logic subsystem are stored;

the communication subsystem is arranged to allow communication with other computer systems; the communication is performed via Wi-Fi, cellular data communication and/or bluetooth;

the MR glasses comprise a biological characteristic sensor, including one or more biological characteristic sensors, an electroencephalogram device for measuring brain electrical activity, an electrocardiogram for measuring heart electrical activity, a microphone or other sound sensor for measuring voice volume, speech speed and the like, a light sensor and an optical scanner;

the MR glasses include a buffer configured to record video/audio to support mood/cognitive state triggered video recording;

the MR glasses include an input/output device including a keyboard, a keypad, a mouse, a microphone, a touch sensor, a touch screen, a joystick, control buttons, scroll buttons, a camera, one or more input interfaces.

Another object of the present invention is to provide a computer device, characterized in that the computer device comprises a memory and a processor, the memory storing a computer program, which when executed by the processor, causes the processor to perform the advertisement push management method based on MR glasses.

Another object of the present invention is to provide a computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the MR glasses-based advertisement push management method.

The invention further provides an information data processing terminal which is used for realizing the advertisement pushing management method based on the MR glasses.

The invention further provides an information pushing system, and the information data processing terminal is used for realizing the advertisement pushing management method based on the MR glasses.

By combining all the technical schemes, the invention has the advantages and positive effects that: the invention can provide more diversified information for the user, and push advertisement information except for content information which the user does not like, rather than what information the user likes to push only; in the aspect of protecting a user privacy mechanism, the invention assists in judging the scene through daily work and rest time of the user, further judges the scene and safety through an acceleration sensor and an IMU for identifying the motion state (such as walking, running, speaking, sitting and the like) of the user, and finally identifies the scene through a camera. The invention detects and identifies the scene through the hardware forms such as position location, time (user work and rest), acceleration sensor, etc., is equivalent to the use of hardware detection scene with lower power consumption, compared with the existing technical scheme that only the scene is identified by a camera, the invention stepwise identifies the scene, and not only is accurate in effect, but also the power consumption is greatly lower.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description will briefly explain the drawings needed in the embodiments of the present application, and it is obvious that the drawings described below are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of an advertisement push management method based on MR glasses according to an embodiment of the present invention.

Fig. 2 is a schematic structural diagram of an advertisement push management device based on MR glasses according to an embodiment of the present invention;

in the figure: 100. an HMD device; 102. a portable handheld electronic device or controller.

FIG. 3 is a schematic diagram of a user wearing an HMD device provided by an embodiment of the present invention;

in the figure: 110. a controller; 122. an optical lens; 130. a front camera; 142. a flicker source; 144. an image sensor; 150. a location subsystem; 160. a motion sensor; 170. a microphone; 180. and a speaker.

Fig. 4 is a schematic diagram of an HMD device according to an embodiment of the present invention.

Fig. 5 is a diagram of detection data of head movement of a user in a resting state provided by an embodiment of the present invention.

Fig. 6 is a diagram of detection data of a head action in a user session according to an embodiment of the present invention.

Fig. 7 is a diagram of detection data of a head motion during walking of a user according to an embodiment of the present invention.

Fig. 8 is a schematic diagram of a video buffering system in a second method according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the following examples in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

Aiming at the problems in the prior art, the invention provides an advertisement pushing management method and equipment based on MR glasses, and the invention is described in detail below with reference to the accompanying drawings.

As shown in fig. 1, the advertisement push management method based on MR glasses provided by the embodiment of the invention includes:

s101, confirming a current scene of a user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for starting an advertisement recommendation program or not through daily work and rest habits of the user and the current scene of the user; judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

S102, performing AI object recognition on physical world image data shot by a front camera by using a local computing power operation object detector of MR glasses; analyzing the current physical environment of the user; identifying and framing a plurality of objects in the image by using an AI (advanced identification) identification technology, matching each object attribute/label in the image with a user personal interest library label, and screening out object image information disliked by the user;

s103, rendering and presenting the content prompt notice after information screening by an MR glasses optical display in the form of a holographic image; meanwhile, the MR glasses intercept the framed object image from the large image, upload the large image to the cloud server, the cloud server carries out matching identification on the received object partial image, searches relevant content information in the database, and feeds back the search result;

s104, the MR glasses download server content data to a cache; judging whether the user has an interaction instruction or not; if an interaction instruction of a user is received, content information returned by the server is presented; if the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

If an interaction instruction of a user is received, content information returned by a presentation server provided by the embodiment of the invention comprises:

The interactive instruction of the user specifically comprises: eye movement interaction, gesture recognition interaction, peripheral controller, voiced/unvoiced speech recognition, head movement interaction;

the content information comprises characters, pictures, audio, video and application programs, and can also be used for editing corresponding relations for users independently in advance;

and if the interaction instruction of the user is received, the content information returned by the presentation server comprises:

the eye movement tracking device of the MR glasses captures eye movement data, detects eye movements, calculates pupil sizes, iris images, eye jump tracks, gazing time length and the like of eyes, and when the gazing time length exceeds a certain time length, the eye movement tracking device is regarded as performing behavior movements corresponding to eyeballs at a certain position, namely displaying content information;

or: the gesture recognition module of the MR glasses detects that the hand makes a specific action, and is associated with up, down, left and right sliding, zooming in, zooming out, clicking and closing control options, so that content information is displayed;

or: transmitting a control signal to the MR glasses through a mobile controller containing wireless signal transmission of the control equipment, and clicking a UI button to enter a telescopic mode through operating an interactive interface;

or: receiving the voice with sound or soundless sound sent by the user through the voice recognition device with sound or soundless sound of the MR glasses, analyzing the voice instruction of the user through the system, and controlling the equipment to display the content information;

Or: and selecting a button of the interactive interface through the head tracking device, namely displaying content information.

The advertisement pushing management method based on the MR glasses provided by the embodiment of the invention comprises the following steps:

(1) Confirming a current scene of a user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for starting an advertisement recommendation program or not through daily work and rest habits of the user and the current scene of the user; judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

(2) Performing AI object recognition on physical world image data shot by a front camera by using a local computing power operation object detector of the MR glasses, and recognizing and framing a plurality of objects in the image by using an AI recognition technology; analyzing the current physical environment of the user according to the detection results of the plurality of objects; confirming the identified environment and privacy security according to the environment;

(3) The front camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses; detecting biological information of the current eye movement behavior data, brain electricity data, heart beat, blood pressure and respiratory frequency of the user, and judging whether the user is possibly interested in the current image;

(4) The method comprises the steps of calling and analyzing cached images/videos/audios before and after the time of the biological index of interest of a user, identifying objects or events in the images/videos by using an AI identification technology, identifying the saliency of the images, and simultaneously selecting the image objects in a frame mode;

(5) The MR glasses intercept the frame selected object image from the large image, upload the frame selected object image to the cloud server, the cloud server carries out matching identification on the received object local image, searches relevant content information in the database, and feeds back the search result;

(6) The MR glasses download the server content data to the cache; judging whether the user has an interaction instruction or not; if an interaction instruction of a user is received, content information returned by the server is presented; if the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

In step (4), the method provided by the embodiment of the invention can be used for retrieving and analyzing the cached images/videos/audios before and after the time of the biological index of interest of the user, identifying objects or events in the images/videos and identifying the saliency of the images by using the AI identification technology, and simultaneously, the steps of combining and framing the image objects include:

first, analyzing the received sensor data to determine whether an emotional/cognitive state of the user is reached; the emotional state includes, but is not limited to, happiness, sadness, anger, fear, disappointment, or pride; the cognitive state includes, but is not limited to, concentration, participation, distraction, boring, drowsiness, confusion, or depression;

Secondly, marking a time axis according to the activated emotion or cognitive state, and acquiring cache images/video/audio before and after the moment according to the time mark;

finally, identifying objects or events in the images/videos by using an AI identification technology, identifying the saliency of the images, and simultaneously framing and selecting the image objects;

the detecting the saliency of the object in the image comprises: detecting an image through an image analysis algorithm; or predicting the object significance in the image shot by the front camera through a pre-trained significance prediction model;

the detecting the image by the image analysis algorithm comprises:

the method specifically comprises the following steps: detecting the position of an object in the image, wherein the closer to the center, the more remarkable; detecting a region with higher color contrast in an image; detecting dynamic objects in the video, and calculating the moving speed of the objects; detecting whether the direction of the object movement is from far to near or from near to far; and calculating the duty ratio of the target object in the whole picture.

The advertisement pushing management method based on the MR glasses provided by the embodiment of the invention further comprises the following steps:

1) Confirming the current scene of the user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for popup content information or not through daily work and rest habits of the user; judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

2) Carrying out AI identification on the image shot by the front camera by utilizing the local computing power of the MR glasses, analyzing the current scene of the user by the attribute of a plurality of objects in the environment, judging whether the personal privacy potential safety hazard exists in the scene, and executing the step 3 if the personal privacy potential hazard does not exist in the scene;

3) The front camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses; detecting body actions of a user by using an acceleration sensor, a gyroscope, an IMU, an eyeball tracking device and a gesture recognition device of the MR glasses;

4) The method comprises the steps of calling and analyzing cached images/videos/audios before and after the time of the interesting physical behaviors of a user, and identifying objects or events in the images/videos by an AI identification technology; the body actions of the user and the images, the videos and the audios are integrated to identify the event so as to judge the interest degree of the user in the event, and if the user is interested, the step 5) is executed;

5) The MR glasses intercept the partial images selected by the frames after the object detection from the large images, upload the partial images to the cloud advertisement server, the cloud server carries out matching identification on the received partial images of the objects, search relevant content information in the database, and return the search result to the terminal MR glasses;

6) The MR glasses download the server content data to the cache; judging whether the user has an interaction instruction or not; if an interaction instruction of a user is received, content information returned by the server is presented; if the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

As shown in fig. 2, an advertisement push management device based on MR glasses according to an embodiment of the present invention includes:

a display module for presenting images to a user using the HMD device;

The control module provided by the embodiment of the invention comprises:

a data holding unit for storing data using one or more storage devices;

The technical effects of the present invention will be further described with reference to specific examples.

According to embodiments of the present invention, an eye-tracking device may track gaze direction and detect eye-movement behavior of a user. In some implementations, the eye-tracking device may be provided in a head-mounted display (HMD) device worn by the user, and may track eye gaze direction, and user head gaze direction and head pose motion, by: for example, sensors (e.g., acceleration sensors, gyroscopes, IMUs, etc.) included in the HMD may convert detected head gaze directions and/or movements defining gaze gestures into user inputs in the virtual environment generated by the HMD. In some implementations, the eye movement tracking device may include an optical tracking device, e.g., an optical gaze tracking device included in the HMD, for detecting and/or tracking a gaze direction, eye movement, eye behavior of the user, and algorithmically mapping the eye movement on a holographic three-dimensional space or two-dimensional screen to form a gaze point cursor by which text input of the virtual keyboard is performed in a virtual environment generated by the HMD.

In the example shown in fig. 3, as shown in fig. 3, a user wearing the HMD 100 holds a portable handheld electronic device or controller 102, which portable handheld electronic device 102 may interoperate and communicate (e.g., wiFi or bluetooth connection) with the HMD 100 through a wired or wireless connection. This may provide for communication between the handheld electronic device 102 and the HMD 100 as well as for the exchange of data, allowing the handheld electronic device 102 to act as a controller in communication with the HMD 100, whereby the controller 102 interacts in a virtual environment generated by the HMD 100. For example, input received on a touch surface of the handheld electronic device 102, the space of the handheld electronic device 102 moves. The handheld electronic device 102 is actually a peripheral hardware that controls a virtual target by hand, so the handheld electronic device 102 may also be a control device such as a data glove (for gesture tracking), a 6DOF controller handle, a smart bracelet, a smart ring, a smart watch, a mobile phone, etc.

Fig. 4 shows an example of a Mixed Reality (MR) device 100, and fig. 2 shows an example of a Mixed Reality (MR) device 100. Fig. 2 shows a block diagram of the MR device 100 shown in fig. 4. The MR device 100 is a head-mounted MR device intended to be worn on the head of a user during ordinary use, including a head-mounted display (HMD) device. However, it is apparent that the invention is not limited to head-mounted MR devices, and that the techniques described herein may be used with other types of MR devices, including but not limited to smart phone devices, tablet computers, notebook computers and/or systems, including heads-up displays configured to provide mixed reality functionality. Mixed reality refers to an experience that allows a virtual image to be mixed with a real world physical environment in a display. For example, real world objects and/or real world space may be identified and enhanced with corresponding virtual objects. Mixed reality may be implemented using, for example, virtual reality or augmented reality techniques.

The MR device 100 comprises a display subsystem 120 for displaying images to a user of the MR device 100. As shown in fig. 3, the display subsystem 120 is intended to be close to the eye of the user and includes a transparent HMD device including one or more transparent or translucent optical lenses 122 arranged such that images may be projected onto the transparent lenses. The hologram is generated by an image generating element (e.g., see-through OLED display) located within the see-through lens 122. A user wearing MR device 100 may see an actual direct view of the real world (rather than an image) through transparent lens 122 and simultaneously view virtual objects (which may be referred to as virtual images or holograms) that may enhance the user's direct view of the real space. Note that the invention is explicitly not limited to MR devices using a see-through display device. The optical lens 122 may control illumination and/or acquisition of an image of the eye through various waveguide techniques. More specifically, switchable Bragg Gratings (SBGs), such as those produced by Digilens Inc., electrically switchable liquid crystal gratings, surface Relief Gratings (SRGs), switchable waveguide displays or MEMS-based mirror (i.e., beam scanning) devices may be used to control the position of the light source and/or spatially select the light collected from the eye as the light is directed toward the eye. Illumination sources, including single light sources (e.g., light emitting diodes, LEDs) and/or detectors, including single detectors (e.g., photodiodes), may be used to illuminate and/or acquire images. The illumination source may be a incoherent (e.g., LED) or coherent light source, including an ultra-low power (to ensure eye safety) diode laser. Spatial resolution may be provided by electronically selective waveguides (e.g., SBG, SRG, liquid crystal, MEMS mirrors). Illumination and image capture may additionally or alternatively involve reflection or refraction by so-called "hot mirrors", fraunhoffer lenses, microlens arrays, waveguide combiners, total Internal Reflection (TIR) holographic waveguides and/or holographic lenses or other optics.

In some implementations, the display subsystem 120 may be opaque and not allow the user to view the scene directly through the display subsystem 120 in all or part of the user's field of view. Such implementations include, but are not limited to, these augmented reality devices being arranged to display visual images of the physical space in front of the MR device 100, and adding virtual objects to augment the display of the physical space, the virtual reality devices and the virtual reality devices being arranged to display virtual objects at locations corresponding to real world objects in the physical space in front of the MR device 100. In some implementations, the MR device 100 may be embodied in a handheld mobile device for display as an object. Enhancement on display devices such as LCD panels. In such an embodiment, the handheld mobile device may be arranged to provide a "window" in which virtual augmentation for the scene surrounding the MR device 100 may be displayed.

The MR device 100 further comprises one or more outwardly facing image sensors (front facing cameras) 130 configured to acquire image data of a real scene surrounding and/or in front of the MR device 100. The front facing camera 130 may comprise one or more further digital imaging cameras 132 arranged to capture two-dimensional visual images. In some embodiments, two imaging cameras 132 may be used to capture stereoscopic images. The front-facing camera 130 may also include one or more depth cameras 134, such as, but not limited to, a time-of-flight depth camera, arranged to capture depth image data, such as to provide an estimated and/or depth map. Or measured distances from the MR device 100 to the respective fields of view (FOV) of the depth camera 134. Depth image data obtained via the depth camera 134 may be registered to other image data, such as images captured simultaneously via the imaging camera 132. The outward facing image sensor 130 may be configured to capture individual images and/or sequences. The frame rate or frame rate at which the images may be configured. In some embodiments, the front-facing camera 130 or other sensor associated with the MR device 100 may be configured to evaluate and/or identify external conditions, including, but not limited to, time of day, direction of illumination, environment, temperature, and others. External conditions may provide additional factors to the MR device 100 to determine the type of virtual graphical element to be displayed to the user.

The MR device 100 may further comprise an eye-tracking system 140, which eye-tracking system 140 is configured to detect sensor data of gaze directions of each eye of the user. The eye tracking system 140 may be arranged to determine the gaze direction of each eye of the user in any suitable way. Referring to fig. 2 and 4, the eye tracking system 140 includes one or more scintillation sources 142, such as infrared light sources, arranged to reflect the flickering light from each eyeball of the user, and one or more image sensors 144. An image of each eye of the user is captured. The flicker variation from the user's eye, determined from image data collected via one or more image sensors 144, may be used to determine a direction of gaze. Further, the location at which the gaze point projected from the user's eyes intersects the external display may be used to determine an object or location at which the user gazes (e.g., a virtual object displayed by display subsystem 120). Eye tracking system 140 may have any suitable number and arrangement of flash sources and image sensors. In one non-limiting example embodiment, four scintillation sources and one image sensor are used per eye. Furthermore, in some embodiments, the eye tracking system 140 may be configured to assist the MR device 100 in more accurately identifying real world objects of interest and associating those objects with virtual applications. The eye movement data are captured through the eye movement tracking device, eye movements are detected, the pupil size, iris images, blink frequency, single-eye blink, eye face closure, sight line, eye jump track and gazing duration of eyes are calculated, and when gazing duration exceeds a certain duration or blinks, clicking confirmation instructions are carried out on UI positions of a certain function.

Further, the eye tracking method of the eye tracking device includes, but is not limited to:

1) And (3) by receiving the infrared light reflected by the eyes, calculating the connecting line between the pupil center and the cornea center for eye movement tracking. The hardware comprises at least one invisible infrared light source, a miniature camera, a reflecting heat mirror and an optical waveguide lens.

2) Another example of an eye tracking technique is eye tracking by capturing an image of the eye or calculating the image of the retina (capillaries, fovea) or the intensity of the light reflected from the retina. The hardware includes invisible infrared light sources, photosensitive sensors, MEMS micro-mechanical system mirrors, optical light guide elements (e.g., optical waveguide mirrors).

3) Another embodiment of an eye tracking technique is to create a model of the eye by emitting structured light, calculate the visual center of the eye model, and perform eye tracking. The structured light infrared pattern projector projects a certain number of invisible light spots onto the measured object, and then the human eye depth map can be calculated according to the reflected light spots received by the infrared camera. The 2D face shot by the RGB camera is combined with the calculated depth human eye information, and the accurate and fine 3D target object can be drawn through algorithm processing.

4) Another embodiment of eye tracking technology performs eye tracking by receiving reflected light from the cornea of an eye, and calculating the time at which the intensity of reflected light from the center of the cornea is maximum. The hardware comprises an invisible infrared light source, a photosensitive sensor, a reflecting mirror of the MEMS micro-mechanical system and an optical waveguide lens, wherein the optical waveguide lens can be configured to be close to one or more layers of optical waveguides of eyes for acquiring and conducting eye images, an input diffraction coupler in the waveguide is used for coupling the eye images, eye image light rays are conducted in the optical waveguides, and meanwhile, the image sensor is configured at a light ray outgoing end of an optical waveguide output diffraction coupler.

5) Additionally or alternatively, the device for tracking eye movement may be incorporated within or operate in conjunction with a contact lens. In addition, the device may be implanted in the eye, in the orbital, or attached to the optic nerve and/or muscle to track movement.

As additional examples, the eye-tracking device may be embedded in a smart phone, tablet, computer, smartwatch, or any other remote object or location, including a computer, display, dashboard, sign, or wearable device worn by other people. In these cases, imaging of the eye is typically performed at a greater distance from the eye than with a head-mounted wearable device. One or more cameras or sensing devices are used to monitor eye movement from such distances when the user looks in the general direction of the device.

The MR device 100 may comprise a position subsystem 150, which position subsystem 150 is arranged to provide the position of the MR device 100. The location subsystem 150 may be arranged to determine the current location based on signals received from the navigation satellite system, such as but not limited to GPS (united states), GLONASS (russia), galileo (europe) and CNSS (china), and technologies that enhance such signals, such as but not limited to enhanced GPS (a-GPS). The location subsystem 150 may be arranged to determine a location based on Radio Frequency (RF) signals identifying the transmitting device and the location determined for such device. For example, wi-Fi, bluetooth, zigbee, RFID, NFC, and cellular communications include device identifiers that can be used for location determination. The MR device 100 may be arranged to use the position provided by the position subsystem 150 as an approximate position, which is refined based on data collected by other sensors. The MR device 100 may comprise audio hardware comprising one or more microphones 170 arranged to detect sound, such as verbal commands from a user of the MR device 100, and/or one or more speakers 180 arranged to output sound to the speakers. The user, for example, spoken queries, replies, descriptions and/or information.

The MR device 100 may comprise one or more motion sensors 160 arranged to measure and report the motion of the MR device 100 as motion data. In some implementations, the motion sensor 160 may include an Inertial Measurement Unit (IMU) including an accelerometer (e.g., a 3-axis gyroscope), a gyroscope (e.g., a 3-axis gyroscope), and/or a magnetometer (e.g., a 3-axis magnetometer). The MR device 100 may be arranged to use the motion data to determine a change in position and/or orientation of the MR device 100 and/or a respective change in position and/or orientation of an object in a scene relative to the MR device 100. The facing image sensor 130, the image sensor 144, the sensors comprised in the position subsystem 150, the motion sensor 160 and the microphone 170, which are comprised in or coupled to the head-mounted MR device 100, may be used individually or together as a head-mounted sensor. The data collected by such head-mounted sensors reflects the position and orientation of the user's head.

The MR device 100 further comprises a controller 110, which controller 110 comprises a logic system 112, a data retention system 114 and a communication system 116. Logic subsystem 112 may include, for example, one or more processors that are configured to execute instructions and communicate. The other elements of the MR device 100 shown in fig. 2 to 4 have the same structure. Fig. 2 to 4 implement aspects of the invention relating to an MR device 100 according to such instructions. These aspects include, but are not limited to, configuring and controlling devices, processing sensor inputs, communicating with other computer systems, and/or displaying virtual devices. The data-holding subsystem 114 includes one or more storage devices (e.g., without limitation, DRAM devices) and/or one or more storage devices (e.g., without limitation, the following). Flash memory device). The data-holding subsystem 114 includes one or more media having stored thereon instructions executable by the logic subsystem 112, which cause the logic subsystem 112 to implement aspects of the present invention involving the MR device 100. Such instructions may be included as part of. An operating system, an application program, or other executable program. The communication subsystem 116 is arranged to allow the MR device 100 to communicate with other computer systems. Such communication may be performed via Wi-Fi, cellular data communication, and/or bluetooth, for example. These instructions cause the logic subsystem 112 to implement aspects of the present invention relating to the MR device 100. Such instructions may be included as part of an operating system, application program, or other executable program.

The biometric sensor 190 includes one or more biometric sensors (e.g., a galvanic skin response sensor for measuring galvanic skin response, a heart rate monitor, a skin temperature sensor for measuring skin surface temperature). An electroencephalogram (EEG) device for measuring brain electrical activity, an electrocardiogram (ECG or EKG) device for measuring heart electrical activity, a microphone (e.g., microphone 108) or other sound sensor for measuring voice volume, speech rate, etc., a light sensor, an optical scanner, etc.

The buffer 195 may be configured to record video/audio to support emotional/cognitive state triggered video recording.

In addition, the input/output device 748 may include one or more input interfaces, which may include a keyboard, a keypad, a mouse, a microphone, a touch sensor, a touch screen, a joystick, control buttons, scroll buttons, a camera, a neural interface, or any other suitable device. To generate signals and/or data defining the user's interactions with the electronic device 706. By way of example, and not limitation, the input/output devices 748 may include a display (e.g., holographic display, heads-up display, protector, touch screen, liquid Crystal Display (LCD), etc.), speakers, haptic interface, and the like.

It will be appreciated that the MR device 100 is provided as an example and is therefore not meant to be limiting. It will thus be appreciated that the MR device 100 may include additional and/or alternative sensors, cameras, microphones, input devices, output devices, etc. in addition to those shown without departing from the scope of the invention. Furthermore, the physical configuration of the MR device and its various sensors and subassemblies may take a variety of different forms without departing from the scope of the invention.

The method comprises the following steps: push messages, not targeted, are passive to receive content as long as the user is not bothersome. The user may not notice some real-world objects, however, the AR glasses recognize real-world objects, analyze and predict the content that the user may be interested in, and in fact, filter out some information that the user is not interested in, and prompt the user for the content that the user may be interested in. The information is received passively by the user, and is not pushed to the user when the user actively wants to know. While filtering out information that is not liked by the user, rather than merely pushing information that is liked by the user.

S101, identifying a scene: confirming the current scene of the user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for starting an advertisement recommendation program or not through two aspects of daily work and rest habits of the user;

s102: identifying a current motion state of a user: judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

s103: object detection: and carrying out AI object identification on the physical world image data shot by the front camera by using a local computing power operation object detector of the MR glasses.

S104: and (3) spatial environment identification: the spatial analysis module analyzes the current physical environment of the user according to the plurality of object detection results of S103.

S105: displaying screening content: and meanwhile, the AI recognition technology recognizes and frames a plurality of objects in the image, matches each object attribute/label in the image with the user personal interest library label, and screens out the image information of the objects which are disliked by the user.

S106, presenting information prompt: and rendering and presenting the content prompt notice screened by the S104 information by the MR glasses optical display in the form of a hologram.

S107, simultaneously with S106, the MR glasses intercept the object image (local image) selected by the frame in S104 from the large image, upload the image to the cloud server, the cloud server performs matching identification on the received object local image, search the related content information in the database,

S108: the MR glasses wait for the interactive action of the user on the content prompt notification, and simultaneously the MR glasses download the server content data to the cache. And if the interaction instruction of the user is received, the content information returned by the server is further presented. If the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

S109: and when receiving the interaction instruction of the user, presenting the content information, wherein the content information comprises characters, pictures, audio, video and application programs, and also can edit the corresponding relation for the user independently in advance.

S101, identifying a scene: the 'confirm the current scene of the user through the MR glasses positioning system' is specifically: the MR intelligent glasses positioning technology comprises, but is not limited to, GPS/Beidou satellite and WiFi module, and a communication signal base station to obtain positioning position information, and if the MR intelligent glasses positioning technology is used in places with consumption behaviors such as shopping malls, entertainment and the like, an advertisement recommendation system can be started.

Further, the step of judging whether the current moment is suitable for popup content information or not according to the daily work and rest habits of the user is specifically that the system calls the matching of the current time and the user work and rest rule data to judge whether the advertisement recommendation system can be started or not. For example, the user is working at 9-11 am, and pushing messages during this time is often annoying to the user; for example, by judging that the 9 to 11 am point of the user is in the office of the company through the positioning technology and the time rule, starting the front-facing camera to acquire the surrounding environment information in the current scene, the user cannot be attracted by the advertisement which is pushed by the object identification in the office environment (the advertisement conversion rate is low), and privacy security risks to other people exist. Otherwise, if the user is on the off-duty home road of 6-7 pm, the user is on a bus stop or the like, and the advertisement information is pushed back to achieve a better effect.

The privacy security risk is specifically that a front-facing camera is required to acquire and analyze the current surrounding environment of a user in the advertisement/content recommendation technology of the current MR/AR/VR intelligent glasses so as to push advertisement content related to a scene, the front-facing camera is required to be started at any time in the technical link, the camera which records data at any time can unconsciously infringe the privacy of other people, and the privacy problem can become a commercial first-order enemy of an AR advertisement recommendation system in the future. Not only start the battery duration of camera to MR glasses all the time like this also brings not little challenge, reduces the whole experience of MR glasses.

In particular, in another embodiment, the MR glasses detect a time, waking up the positioning system to confirm the scene at a time that the user is likely to be entertaining in general. The method can further save power consumption and enable the user to obtain better experience.

S102: identifying a current motion state of a user: judging whether the current motion state of the user is suitable for ejecting/pushing content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

the method specifically comprises the steps of collecting data of acceleration sensors and gyroscope sensors. The acceleration sensor data and the gyroscope sensor data respectively comprise data of three sub dimensions of an x axis, a Y axis and a z axis, so that the acquired data to be identified are 6-dimensional data.

The acceleration sensor and the gyro sensor may be arranged in an intelligent electronic device, which may be, for example, an intelligent MR glasses. The data to be identified may be sensor data acquired by an acceleration sensor and a gyro sensor. The acceleration sensor and the gyroscope sensor can record various data in the using process of the intelligent MR glasses.

For example, when a user is talking to a person, it is obviously not a good opportunity to push advertisements, which may be overly attractive to the user and may not be annoying to the user. Or because the user focuses attention on the opponent, the user is free to pay attention to the advertisement information, and the conversion rate of advertisement pushing is reduced.

For example, when a user is running, the conventional optical display system of MR glasses cannot effectively present virtual holograms in such high-frequency and largely-swaying wearing state, so that a "rainbow" image dispersion situation occurs, and presenting any holographic content is not ideal in the current situation. Still others may ignore any information that is not relevant to the goal while running in a hurry. It is not suitable to pop up/push content information in a running state.

For example, it can be determined that the user is in the elevator when the user is in the running state of the elevator, and these repetitive motion characteristics of the acceleration motion, the uniform motion, and the deceleration motion of the elevator can be detected by the acceleration sensor of the MR glasses. It is conceivable that the better the scene is when pushing/popping messages in the elevator, the higher the conversion rate is.

Acceleration sensor data and gyroscope sensor data of MR glasses used by a user can be distributed in disorder, and the sensor data is relatively large. Figures 5 to 7, for example, show graphs of the head movement tracking sensor signals in resting, speaking, running states;

s103: object detection: the object detector is operated by utilizing the local computing power of the MR glasses to perform AI object recognition on the physical world image data shot by the front-facing camera, meanwhile, the AI recognition technology is used for recognizing and framing a plurality of objects in the image, specifically, the feature database trained in advance is operated by utilizing the local processor of the MR glasses terminal to recognize interesting objects in the image, wherein the image shot by the front-facing camera of the MR glasses is configured to be consistent with the physical world image picture seen by eyes of a user (the camera field angle is consistent with the eye field angle and calibrated). After the object detection, a boundary block diagram is set at the periphery of the object according to the image recognition and image segmentation results.

Wherein physical world image data (such as visual image data and/or depth image data) captured by the front-facing camera has been subjected to a certain preprocessing procedure, such as filtering, correction or other preprocessing of the image data. In some examples, the image pre-processor may segment the image data into regions based on visual and/or depth image data.

Further, AI identification and classification is performed by the object detector of the MR glasses 100 of objects in the real world image and the position and/or pose of the real world objects in the scene observed by the MR device 100 are determined. In some real world, a visual tag (e.g., a QR code or a bar code) or other indicia is not identified to mark or attach to the real object. The object classifier is configured to receive image data from the MR device 100 and to identify real world objects present in the provided image data using the trained model data or model data. The object classification system is configured to identify and/or classify real-world objects based on image data using machine learning and reasoning techniques.

Further, the system may utilize various known computer object recognition techniques to identify objects in images of real world scenes. In some embodiments, object recognition may use an appearance-based method that compares an image to a reference image of a known object to identify the object. Examples of appearance-based methods include edge matching, gray matching, receiving histograms of field responses, and the like. In some embodiments, object recognition may use feature-based methods that rely on matching object features and image features. Examples of feature-based methods include gesture clustering, geometric hashing, scale-invariant feature transforms, interpretation trees, and the like. The system may use one or more object recognition methods in combination to enhance the accuracy of object recognition.

Wherein the "one or more object detection/classification models 222" may each specify parameters (e.g., weight values) or random forests for a machine learning algorithm (e.g., convolutional Neural Network (CNN)), for one or more decision trees that are trained to detect and/or classify real world objects captured in image data provided to the machine learning algorithm.

S104: and (3) spatial environment identification: the spatial analysis module analyzes the current physical environment of the user according to the plurality of object detection results of S103, and further confirms the environment and privacy security identified in S101 according to the environment, specifically,

the spatial analysis module is configured to identify/classify and determine boundaries of a physical space observed by the MR device 100, such as, but not limited to, the physical space 200 shown in fig. 4. The spatial analysis module is configured to compare the AI object recognition result of the object detector with a "scene feature library" so as to analyze whether the current environment of the user is a privacy security risk area. Each or several object combinations in the "scene feature library" corresponds to an environmental scene. For example, when the object detector identifies objects such as a refrigerator, a microwave oven, a water tank and the like in an image acquired by the front camera, the current environment of the user is judged to be a kitchen according to a weight algorithm; if the image data identifies a "bed", the user's current environment may be in a bedroom; if the image data identifies a desk, a bookcase, and an office computer, the user's current environment may be in the office; if the image data identifies a toilet, urinal, the user's current environment may be in a bathroom. In addition, places such as shops, restaurants, gas stations, fitting rooms and the like can be identified through one or more characteristic objects. If the current environment of the user is a privacy security high risk area, the system turns off the camera or prohibits the camera from recording image data such as photos/videos, and if the current environment of the user is a privacy security low/no risk area, the camera is continuously started, and the content recommendation system is started.

In the design of the step, the purpose is not to identify the current scene of the user, but to judge whether the current environment of the user has privacy potential safety hazard or not through a visual identification technology. For example, if the current physical environment is "toilet" or "office" or "fitting room" by identifying the feature object, it is a privacy security high risk area, and it is not preferable to start the camera in real time or prohibit the camera from recording image data such as photos/videos.

Another purpose of this step is also to make a secondary confirmation of the environment identified by the positioning technique of step S101, since existing position positioning techniques may malfunction indoors. For example, it may be determined by a satellite/base station or other positioning technique that a user is in a restaurant, but the user's location inside the restaurant cannot be further confirmed, the user may be in a hall, possibly in a restroom, a bathroom, a kitchen, etc., not only because the signals used and positioned indoors are obscured, but also because there is no such detailed indoor floor in the WeChat map. Further confirmation of the environment is therefore required, excluding privacy sensitive areas.

S105: display content screening: and receiving advertisement information preset and popularized by the entity store merchant according to the object identification and classification results, matching each object attribute/label in the graph with the personal information of the user, screening out the dislike/unnecessary object image information of the user, or screening out the emotional content unsuitable for the current user.

For example, if a current user is a list of male users wearing MR glasses to meet with a mall (e.g. Mo Da square), various physical shops such as lady's clothes, women's shoes, lady's bags, gold and silver jewelry, men/women's cosmetics, unmanned body inspection shops (large-area unmanned aerial vehicle), brand mobile experience shops (Hua as mobile phones), children entertainment areas, sports brands (nike, addies), restaurants (sichuan dish, barbecue, daily feed, western-style food) and movie theatres may be shot and identified by the camera of the MR glasses on the premise of confirming no privacy risk, and the physical shops may have corresponding advertisement information, but without a content screening system, the user may see full screen holographic advertisement information through the MR glasses, which not only easily causes the user to generate boring emotion and poor user experience, but also has low conversion rate of advertisement, and excessively attracts the user's attention to the content to generate attraction fatigue.

In contrast, according to the present invention, after the MR glasses recognize/classify the object, the object is matched with the personal information of the user according to the characteristics of the object, and then a part of the information is filtered out and displayed. For example, the above-mentioned women's clothes, women's shoes, women's bags, gold and silver jewelry, women's cosmetics, and advertisement information related to children's entertainment areas are not suitable for pushing to a list of men, and the opposite unmanned body inspection shop (the man unmanned aerial vehicle), brand mobile experience shop (the Hua mobile phone), sports brands (the Kappy, addi, outdoor) can better draw the attention of the male users. In another example, the male user may like to make up, such as purchasing male cosmetics/skin care products in a history, the system allows male-related male cosmetic/skin care product advertising cues to pop up. Wherein the personal information includes, but is not limited to, a personal interest library, interest tags, user portraits, historical eye gaze point interest libraries, user names, profile pictures, contact information, birth date, gender, marital status, family status, work, educational background, visual preferences, interests, or other demographic information.

The system presents a prompt notification at the target object location, which may be one or more of the following: pop-up notifications, voice notifications, vibrations, notification text or icons in an object-based content sharing user interface, notification messages in a social media service, and once a new content item is added to a text message, the system will cause the user's device to sound or vibrate. In some embodiments, the user may individually configure the type of notification to receive each object and/or object category associated with the user. In some embodiments, the user may individually configure whether to receive notifications for each object and/or class of objects associated with the user. The user may configure the type of notification to be received.

In another embodiment, a notification module may also be used to present a prompt notification of the target object location, which allows the OS of the MR eyeglass system to match the object recognition results to the application, and then to match the notification to the application object that has been registered in the OS, thereby identifying the object as the target of the notification. The OS may perform this step by, for example, comparing the context Identifier (ID) contained in the notification payload with the context IDs of all objects registered in the system, and finding the object with the matching context ID. After identifying the object as the target of the notification, the operating system may call a special notification handler associated with the application of the object (or with the object in the image), but separate from the main process of the application. For example, the notification handler may be an executable file that is different from the main executable file of the application. The notification handler may then execute code (e.g., defined by an application developer) to process and present the notification in the system UI (e.g., alongside, above, etc.) relative to the object in the context. .

S107, selecting the object subjected to information screening, intercepting the partial image selected by the frame after the object detection in S103 from the large image, uploading the partial image to a cloud advertisement server, carrying out matching identification on the received partial image of the object by the cloud server, searching relevant content information in a database, and returning the search result to the MR glasses of the terminal. Wherein the content information includes text, pictures, audio, video, and applications.

Further, the cloud server searches according to the position information, scene information and local images (target objects) returned by the terminal MR glasses, and whether advertisement information corresponding to the target objects exists in the database or not is searched. The cloud server is configured as an advertisement delivery platform capable of receiving/collecting advertisement delivery demands of merchants in physical stores, and the merchants can set any advertisement content such as introduction content, coupons, discount information and the like on the advertisement delivery platform, and even can link with self-owned applets (such as meal ordering applets) of the merchants and the like.

For example, when the user is at a mall and walks through a hot pot, the MR glasses upload the location information of the mall to the server to retrieve the commercial merchant in the mall. In the process of passing through the hot pot store, the MR glasses acquire local images of the plaque logo and the door finish pattern of the hot pot store, and upload the local images to the server to determine the advertisement release content of the specific advertiser. The advertisement delivery content is transmitted back to the MR glasses terminal.

S108: the MR glasses download the server content data to the storage, and the simple advertisement content information is rendered and presented by the MR glasses optical display in the form of a hologram. And a button instruction for acquiring further information can be arranged on the UI interface for displaying the simple content, and if the interactive instruction of the user is received, the content information returned by the server is further presented. If the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

First the system determines the direction and three-dimensional position in which the holographic content is to be displayed based on the position of the relevant physical object in the environment map. For example, when a certain holographic content (or program interface) is displayed on a physical object-a chair, the position of the surface or vicinity of the chair in the environment map is determined as the three-dimensional position at which the holographic content is displayed. The position and posture of the imaging device acquired from the environment map generating unit 110 are then converted into a planar position and a hologram content depth on the optical imaging screen, and the calculated position in the input image generates an output image to be displayed on the optical display of the MR glasses.

If the user movement causes the user's perspective to the physical object to change, the electronic device may detect the user's movement and adjust the user's view of the asset to correspond to the user's movement. The movement of the user may be detected by the sensor 160 or by analyzing images captured by the camera 130. In one embodiment, simultaneous localization and mapping (SLAM) techniques may be used to adjust a user's perspective on an asset. For example, in SLAM, a set of points is tracked by consecutive camera frames. Using these trajectories, the 3D position of the point may be triangulated while using the estimated point position to calculate the camera pose of the viewpoint. Using the 3D position and camera pose, the user's asset view may also be adjusted.

The interactive instruction of the user specifically comprises: eye movement interactions, gesture recognition interactions, peripheral controllers, voiced/unvoiced speech recognition, and head movement interactions.

(1) The eye movement tracking device of the MR glasses captures eye movement data, detects eye movements, calculates pupil sizes, iris images, eye jump tracks, gazing time length and the like of eyes, and when the gazing time length exceeds a certain time length, the eye movement tracking device is regarded as performing behavior movements corresponding to eyeballs at a certain position, for example, performing movements of gazing corresponding to clicking operations (eye behavior movements) on UI buttons in a telescopic mode in an interactive interface, namely displaying content information.

(2) The gesture recognition module of the MR glasses detects that the hand makes a specific action, for example, the gesture recognition device receives motion information of the hand of the user, or recognizes a specific shape of the hand, and the gesture recognition module is associated with control options such as up, down, left, right sliding, zooming in, zooming out, clicking and closing, so that content information is displayed.

(3) Through the mobile controller (3 degrees of freedom or 6 degrees of freedom) containing wireless signal transmission of the control device, such as a holding button controller, a wearable glove, a fingerstall controller and the like, control signals are transmitted to the MR glasses, and a UI button is clicked to enter a telescopic mode through an operation interaction interface, wherein the telescopic mode comprises a 6DoF acceleration sensor tracking technology and a 6DoF electromagnetic tracking technology handle.

(4) The voice recognition device of the MR glasses receives the voice or the soundless voice sent by the user, and the voice instruction of the user is analyzed through the system, so that the control equipment displays the content information.

The second method is as follows: and pushing the information, namely pertinently judging whether the current object is interested in or not by identifying the biological information of the user, pushing the content information by the system, and passively receiving the content by the user.

S211, identifying a scene: confirming the current scene of the user through an MR glasses satellite positioning system, and judging whether the current moment is suitable for popup content information or not through daily work and rest habits of the user;

s212: identifying a current motion state of a user: judging whether the current motion state of the user is suitable for popup of content information or not through an acceleration sensor, a gyroscope and an IMU of the MR glasses;

s213: object detection: and (3) performing AI object recognition on the physical world image data shot by the front camera by using a local computing power operation object detector of the MR glasses, and simultaneously recognizing and framing a plurality of objects in the image by using an AI recognition technology.

S204: and (3) spatial environment identification: the space analysis module analyzes the current physical environment of the user according to the plurality of object detection results of S213, and further confirms the environment and privacy security identified in S101 according to the environment.

S215, circularly recording and caching: the front-end camera of the MR glasses circularly records video, photos and audio in a period of time, and the video, the photos and the audio are cached in the local memory of the MR glasses.

S216, detecting current biological information of the user, including eye movement behavior data, brain electricity data, heart beat, blood pressure and respiratory frequency, and judging whether the user is possibly interested in the current image.

S217: the method comprises the steps of retrieving and analyzing cached images/videos/audios before and after the time of the 'biological index of interest' of a user, and identifying objects or events in the images/videos and the saliency of the images by an AI identification technology, and simultaneously framing and selecting the image objects.

And S218, the MR glasses intercept the partial images selected by the frames after the object detection in the S213 from the large images, upload the partial images to the cloud advertisement server, carry out matching identification on the received partial images of the objects by the cloud server, search relevant content information in the database, and return the search result to the terminal MR glasses. Wherein the content information includes text, pictures, audio, video, and applications. And meanwhile, a content prompt notice is presented for the target object or the event position coordinate, and the content prompt notice is rendered and presented by the MR glasses optical display in the form of a hologram.

S219, the MR glasses wait for the interaction of the user on the content prompt notification, and meanwhile the MR glasses download the server content data to the cache. And if the interaction instruction of the user is received, the content information returned by the server is further presented. If the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

And S220, presenting the content information after receiving the interaction instruction of the user.

The specific implementation is as follows:

in contrast to the first embodiment, the difference is S215-S217, and other steps are the same as those of the first embodiment, so that the description thereof will not be repeated.

In the first embodiment of the invention, the reason for designing S215-S217 is that in some of the invention patents disclosed at home and abroad, the degree of interest of the user in the current vision front object is judged by collecting and analyzing the physiological data of the user, and then the content information related to the object is presented to the user according to the degree of interest. However, the above-described method is inaccurate. The reason is that when a person sees something interesting/aversive in front of the vision, a biological reaction (such as heart beat acceleration, skin temperature change, electroencephalogram change, etc.) may be caused, but the biological reaction is delayed, the interesting event is about to elapse, it is possible that when the MR glasses judge that the user is interested in something, and then the front camera is started to acquire an interesting image for recognition analysis, the interesting object has walked away or disappeared, it is conceivable that only the local image of the interesting object at the original position obtained by the camera is invalid, and there is no chance of restarting for the second time. For example, a user wearing MR glasses walks on the street, suddenly a smart car is designed to drive in front of the eyes, the eyes of the user catch up with the car, the pupils are slightly open, and the heartbeat accelerates, but the car is far away just when the MR eyes start the camera and recognize the car, and the MR glasses cannot acquire effective car images.

In order to solve the technical defects: the front-end camera of the MR glasses circularly records video, photos and audio in a period of time, and the video, the photos and the audio are cached in the local memory of the MR glasses. When the biological data of the user reach the threshold value, the cached images/videos/audios before and after the time of the 'biological index of interest' of the user are retrieved and analyzed, and the AI recognition technology recognizes objects or events in the images/videos, recognizes the saliency of the images and simultaneously and frames the image objects.

MR glasses record video continuously to a buffer. Any number of various types of sensors may be used to determine the emotional or cognitive state of the user. When a specific emotion or cognitive state is detected or when the degree of emotion or cognitive state exceeding a threshold is detected, video segment contents corresponding to the moment in a buffer area are extracted for image recognition.

S215, circularly recording and caching: the front-end camera of the MR glasses circularly records video, photos and audio in a period of time, and the video, the photos and the audio are cached in the local memory of the MR glasses. In particular video/audio is recorded to a buffer. For example, when video/audio data is captured by camera 106 and microphone 108, emotion/cognitive state triggered recording system 102 writes the video/audio data to buffer 202. The buffer 202 may be implemented as a circular buffer, such as the buffer 202 circularly storing data for the last period of time, the storage capacity of the buffer may be manually set, and when the capacity is full, the video at the earliest time is circularly deleted, and new video data is written.

S216, detecting current biological information of the user, including eye movement behavior data, brain electricity data, heart beat, blood pressure and respiratory frequency, and judging whether the user is possibly interested in the current image. In particular to detect and receive sensor data. For example, the sensor data analysis module 204 receives data from one or more sensing devices. The sensor data may include, but is not limited to, galvanic skin response data, skin temperature data, galvanic skin data, brain wave activity, heart electrical activity data, eye movement data, facial expression data, pupil dilation and/or constriction data, voice volume data or voice rate indicating data.

S217: the method comprises the steps of receiving and analyzing cached images/videos/audios before and after emotion/cognitive state time of a user, and identifying objects or events in the images/videos and identifying image saliency by an AI identification technology, and simultaneously selecting image objects in a frame mode.

And in particular to analyzing the received sensor data to determine whether an emotional/cognitive state of the user is achieved. For example, the sensor data analysis module determines an emotional/cognitive state of the user based on the received sensor data. For example, the sensor data analysis module may utilize a deep neural network to analyze the received sensor data to determine an emotional or cognitive state of the user. The system then marks the time axis according to the activated emotion or cognitive state and obtains the cached images/video/audio before and after this time according to the time mark. Further, the AI identification technology identifies objects or events in the image/video, and identifies the image saliency, and simultaneously frames the image object, wherein the object detection process is as described in the object detection S103 technology in the first embodiment. The emotional state may include, but is not limited to, happiness, sadness, anger, fear, disappointment, or pride. Similarly, cognitive states may include, but are not limited to, concentration, participation, distraction, boring, drowsiness, confusion, or depression.

Further, the detecting the salience of the object in the image, specifically, the image can be detected by any image analysis algorithm, including but not limited to the following salience detection method;

1. detecting the position of an object in the image, wherein the closer to the center, the more remarkable;

2. detecting a region with higher color contrast in an image;

3. detecting dynamic objects in the video, and calculating the moving speed of the objects;

4. detecting whether the direction of the object movement is from far to near or from near to far;

5. calculating the duty ratio of the target object in the whole picture;

the significance detection mode can be any one or any several of the significance detection modes, and the significance can be judged by a certain weight combination.

In another embodiment of the detecting the saliency of the object in the image, the saliency of the object in the image captured by the front camera may be predicted by a pre-trained saliency prediction model.

Various methods for analyzing user viewing activity determine (or predict) user-specific points of interest in a given video. For example, in some embodiments, user-specific saliency models may be trained using content items viewed by users and individual user-specific heat map data. As mentioned, in some embodiments, the heat map data may identify regions of interest to a user in a given content item on a frame-by-frame basis. As described above, the heat map data may be determined based on user interactions with the content item during presentation (e.g., sensor data, gesture data, input device data, headset motion data, eye tracking data, etc.). In some embodiments, a user-specific model may be used to predict which points of interest a user may be interested in other content items accessed by the user.

In some embodiments, one or more frames are extracted in the real-time picture of the front camera to create additional content. For example, a portion of content (e.g., one or more frames) that has been determined to be of interest may be extracted from the content item, e.g., as one or more images or short videos. In some embodiments, these portions of interesting content may be identified based on user interactions during presentation of the content item. Such user interactions may be measured using sensor data, gesture data, input device data, headset motion data, eye tracking data, to name a few. One or more significance prediction models may be generated. For example, in some embodiments, as described above, an aggregate heat map may be used to train a general saliency prediction model that describes user view tracking data for various content items. In some embodiments, the prominence prediction model may be used to predict content (e.g., points of interest) that may be of interest to a user during presentation of the content item.

Further, the actions, the gestures and the expressions of the environmental person in the front-facing camera picture can be detected through the related image algorithm, for example, but not limited to, waving hands, shouting, smiling, jumping, delivering (smoke delivering) and the like of a person in a scene to the user of the intelligent glasses, and the environmental person can be judged to be a target object potentially to be focused by the user.

Fig. 8 shows a schematic diagram of a video caching system in method two, the selection component of the emotion/cognitive state trigger recording system 102, which may be implemented on a single device or may be distributed across multiple devices, such as HMD device 112, cell phone 116, and computing device 118. The exemplary emotion/cognitive state triggered recording system 102 includes a buffer 202, a sensor data analysis module 204, a recording module 206, a gaze detection module 208, and a video segment storage area 210.

The buffer 202 is configured to store video and audio data as it is received from the camera 106 and microphone 108. In an example, buffer 202 is implemented as a five minute circular buffer.

The sensor data analysis module 204 receives sensor data from the sensing device 104 and analyzes the received sensor data to determine an emotional or cognitive state of the user. In an example embodiment, the received sensor data is indicative of an emotional or cognitive state. In alternative embodiments, sensor data is analyzed to determine the emotional or cognitive state of the user using, for example, a Deep Neural Network (DNN). The sensor data analysis module 204 provides available data indicative of the emotional or cognitive state of the user.

The recording module 206 determines whether to record the video clip based on data indicative of the emotional or cognitive state of the user. In an example, the recording module 206 may be configured to begin recording video clips based on a change in the emotional or cognitive state of the user. For example, the recording module 206 may be configured to begin recording video clips when the emotional or cognitive state of the user changes from a neutral state to a non-neutral state. In another example, the recording module 206 may be configured to begin recording video clips when a value representing an emotional or cognitive state of the user exceeds a threshold.

The recording module 206 also determines when to stop recording a particular video clip. For example, the recording module 206 may record video clips based on predefined time periods, changes in the emotional or cognitive state of the user, or a combination of both. For example, when the recording module 206 begins recording video clips, the recording module 206 may continue recording for a predetermined period of time (e.g., 10 minutes). Alternatively, the recording module 206 may continue recording until the sensor data indicates that the emotional or cognitive state of the user has changed or fallen below a threshold. In another example, the predefined time period may indicate a minimum or maximum length of a video clip to be recorded. In this example, if the predefined time period indicates a minimum recording time, the recording module 206 may cause the video clip to be recorded until the predefined time period is met or until the emotional or cognitive state of the user changes, whichever occurs later. If the predetermined period of time indicates a maximum recording time, the recording module 206 may cause the video clip to be recorded until the predetermined period of time is met or until the emotional or cognitive state of the user changes.

The gaze detection module 208 tracks the user's gaze to determine the direction of the user's gaze. The gaze detection module 208 may be configured to generate, for example, a video overlay that includes colored points, highlights, or other visual indicators regarding the gaze direction of the user at the time the video was captured.

Video clip storage 210 is configured to store video clips recorded based on the emotional or cognitive state of the user. For example, when the recording module 206 initiates recording, the video and audio data stored in the buffer 202 is copied to a new video segment, which is stored in the video segment memory 210. In addition, the recording module 206 directs the additional video and audio data to be recorded to the video clip as described above. In an example embodiment, video clip store 210 also stores metadata associated with the video clip, which may include, but is not limited to, an indicator of the emotion or cognitive state of the user triggering the recording, one or more indicators of the emotion of the user. Or a cognitive state at the time of video recording, and for providing a visual indicator.

And a third method: and pushing information, namely pertinently, judging whether the current object is interested in or not by identifying the limb action of the user, pushing the content information by the system, and passively receiving the content by the user.

s303: privacy security validation: and (4) carrying out AI identification on the image shot by the front camera by utilizing the local computing power of the MR glasses, analyzing the current scene of the user by the attribute of a plurality of objects in the environment, judging whether the personal privacy potential safety hazard exists in the scene, and executing S104 if the personal privacy potential hazard does not exist.

S304, circularly recording and caching: the front-end camera of the MR glasses circularly records video, photos and audio in a period of time, and the video, the photos and the audio are cached in the local memory of the MR glasses.

S305, detecting the current body motion of the user, and detecting the body motion of the user by using an acceleration sensor, a gyroscope, an IMU, an eyeball tracking device, a gesture recognition device and the like of the MR glasses.

S306: the buffered images/videos/audios before and after the time of the user's "body behavior of interest" are retrieved and analyzed, and AI recognition technology recognizes objects or events in the images/videos. The user' S body motion and the image, video and audio recognition event are combined to judge the interest degree of the user in the event, if the interest degree is judged, S307 is executed

S307, the MR glasses intercept the partial images selected by the frames after the object detection in the S303 from the large images, upload the partial images to the cloud advertisement server, the cloud server carries out matching identification on the received partial images of the objects, search relevant content information in the database, and return the search result to the terminal MR glasses. Wherein the content information includes text, pictures, audio, video, and applications. And meanwhile, a content prompt notice is presented for the target object or the event position coordinate, and the content prompt notice is rendered and presented by the MR glasses optical display in the form of a hologram.

S308, the MR glasses wait for the interaction of the user on the content prompt notification, and meanwhile the MR glasses download the server content data to the cache. And if the interaction instruction of the user is received, the content information returned by the server is further presented. If the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the user visual field picture, and emptying the cache information.

Specifically described as

Further, eye movement is detected by an eye movement tracking device: the saccade distance exceeds/equals a preset threshold, and/or the smooth movement behavior duration of the eye exceeds/equals a preset threshold, and/or the movement speed of the eye exceeds/equals a preset threshold, and/or the eye suddenly stops moving unordered, and/or the fixation duration of the eye at a certain point exceeds/equals a preset threshold, and/or the eye reciprocates between two points/ranges, and/or the eye tremors behavior, and/or the change of the pupil diameter of the eye exceeds/equals a preset threshold, vestibular eye movement, converging/diverging movement, smooth chasing, long saccades and correcting saccades; during normal daily activities, the eye can be tracked for both voluntary and involuntary activities. An aspect of such an embodiment is to discern the intent of the user as the user's eyes routinely perform both types of movements. This requires distinguishing between voluntary and involuntary eye movements for communicating intent.

Voluntary exercise may be associated with normal daily activities, and may also be used to convey purposeful intent based on the eye signal "language". Thus, an algorithm "filter" and/or other programming method is used to identify intent based on video images of one or both eyes. The algorithm may be encoded as software, firmware, embedded in a silicon-based logic device, or a combination of these methods. The processing may also be performed locally, remotely, or in combination with such methods including so-called "cloud computing".

Voluntary movements that might convey purposeful intent include glances with well-defined angular velocities and ballistic contours (based on eye physiology) for astigmatic movements observed at different distances from the observer, vestibular eye movements associated with head movements and smooth tracking movements for following specific (real or virtual) objects. Other information of the user's intent may be determined based on other factors such as pupil dilation, lens accommodation, eyelid movement, facial muscle movement, body and head movement, hand movement, and related contextual information.

In other aspects, the tracked eye movements and geometries may be used to discern physiological and/or emotional states of an individual in a continuous manner. For example, the extent of pupil dilation may be used to discern emotional states such as fear, interest, or cognitive load. The anti-saccadic motion may be an indication of viewing offensive things. When combined with information about the real or virtual object the user is looking at, an indication of the class of object that is, for example, a startle, attractive or interesting item can be discerned. Such information may be used, for example, for subsequent display of customized information.

Further, head movement is detected by the head tracking device: in further aspects, the eye signals may be combined with other modalities to convey user intent. For example, the eye signal may be combined with voluntary head movements (e.g., head nodding) as part of the eye signal "language".

The speed, acceleration and rotation angle during the turning process of the user are detected, such as a jerky turning head and a smooth turning head. In practice, for example, eye movement and head movement may be detected simultaneously, and when user a calls the name of user on the side of user B, the user may quickly turn his head in the direction of user a, with eye jump over a long distance of the eyes, the eyeball turns in the direction of user a, and when an object/event of interest occurs outside the field of view of the person, the person subconsciously turns his head and eyes to find the target object. In another practical embodiment, a sports car is driven from a distance, in front of the user, through which the person is attracted by the sports car, the head and eyes moving smoothly following the movement of the car.

Involuntary eye movements are often associated with voluntary head movements (so-called vestibular eye movements). Thus, vestibular eye movements that occur in a reverse direction to head movements may be included in eye signal control during voluntary head movements. Other forms that may be used in conjunction with the eye signaling language include gestures, other body part movements, voice, switch activation, facial expressions,

The head position tracking system includes position and motion sensors (e.g., compasses, gyroscopes), accelerometer light sources and/or other devices for acquiring the position, direction and motion of the MR glasses, detected by detecting motion of the entire scene in an externally facing camera. Head position information may be received from a head position tracking system. Head position information is provided to a head position tracking system of an eye movement interaction intent prediction model. An Inertial Measurement Unit (IMU) is formed on the MR glasses. The head position information may help determine the head movement, rotation or direction of the user.

Further, body movement is detected by a movement tracking device: body behaviors such as dodging, jumping, trembling and the like of the user are detected. The head tracking device of the MR eye may to some extent detect body movements on behalf of the movement tracking device, but in another embodiment body movements may also be detected by any wearable smart hardware such as a smart watch, a smart bracelet, a controller handle with spatial positioning, etc. For example, when a user is stimulated by an incident, the user is scared around, jumped around, and shaken around, the incident including, but not limited to, a head-on drive, a head-on run, a dog, and a loud outside window. It is possible to detect a reaction in which the user stops working when a keyword or the like of interest enters the ear and turns the line of sight to the sound source with high accuracy. From these physical behaviors, the MR computer can determine the user's interest level.

For example, physical actions include, but are not limited to, turning head, turning body, stopping walking suddenly, shaking/hiding (next jump), eye smoothing, pupil diameter variation, long distance jumps, turning head with jumps, and the like. And judging that the interest event exists in the current environment through the physical behaviors.

In contrast to the third embodiment, the present embodiment determines the interest level of the object in front of the vision by recognizing some physical behaviors of the user, and uses the detected physical behaviors as marks to synchronize the video in the cache, where the physical behaviors include, but are not limited to, physical behaviors such as turning, suddenly stopping walking, shaking/hiding (jumping down), smooth eye tracking, pupil diameter change, long distance jumping, jumping with the jump of the head, and the like, and the physical behaviors include, but are not limited to, brain electrical data, heartbeat, blood pressure, and respiratory rate.

The foregoing is merely illustrative of specific embodiments of the present invention, and the scope of the invention is not limited thereto, but any modifications, equivalents, improvements and alternatives falling within the spirit and principles of the present invention will be apparent to those skilled in the art within the scope of the present invention.

Claims

1. The advertisement pushing management method based on the MR glasses is characterized in that whether the current object is interested in is judged by identifying biological information of the user, then the system pushes content information to the user, instant information is obtained through a cache video, and the advertisement pushing management method based on the MR glasses further comprises the following steps:

step S214: and (3) spatial environment identification: the space analysis module analyzes the current physical environment of the user according to the plurality of object detection results of S213, and further confirms the environment and privacy safety identified in S211 according to the environment;

Step S215, circularly recording and caching: the front camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses; the front camera of the MR glasses circularly records video, photos and audio within a period of time and is cached in the local memory of the MR glasses; specifically recording video/audio to a buffer; when the video/audio data is captured by the camera and the microphone, the recording system triggered by the emotion/cognition state writes the video/audio data into the buffer; the buffer is realized as a circular buffer, the buffer circularly stores data in the last time period, the storage capacity of the buffer is manually set, when the capacity is full, the video in the earliest time is circularly deleted, and new video data is written;

step S216, detecting current biological information of a user, including eye movement behavior data, brain electricity data, heart beat, blood pressure and respiratory frequency, and judging whether the user is possibly interested in the current image;

Step S218, the MR glasses intercept the partial images selected by the frames after the object detection in the step S213 from the large images, upload the partial images to the cloud advertisement server, the cloud server carries out matching identification on the received partial images of the objects, search relevant content information in a database, and the search result is returned to the terminal MR glasses; wherein the content information includes text, pictures, audio, video, and applications; meanwhile, a content prompt notice is presented at the position coordinates of a target object or an event, and the content prompt notice is rendered and presented by an MR glasses optical display in the form of a hologram;

step S219, the MR glasses wait for the interactive behavior of the user on the content prompt notice, and simultaneously the MR glasses download the server content data to the cache; if the interaction instruction of the user is received, the content information returned by the server is further presented; if the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the visual field picture of the user, and emptying the cache information;

step S220, presenting content information after receiving an interaction instruction of a user;

further comprises: the emotion/cognitive state triggering recording system comprises a buffer, a sensor data analysis module, a recording module, a staring detection module and a video segment memory;

the recording module determines whether to record the video clip based on data indicative of an emotional or cognitive state of the user; the recording module is configured to begin recording the video clip based on a change in the emotional or cognitive state of the user, or the recording module may be configured to begin recording the video clip when the emotional or cognitive state of the user changes from a neutral state to a non-neutral state, or the recording module may be configured to begin recording the video clip when a value representing the emotional or cognitive state of the user exceeds a threshold value;

The video segment memory is configured to store video segments recorded based on emotional or cognitive states of the user; when the recording module starts recording, copying the video and audio data stored in the buffer to a new video segment, which is stored in the video segment memory; the recording module records the additional video and audio data to the video clip; the video segment memory also stores metadata associated with the video segments including an indicator of an emotion or cognitive state of the user triggering the recording, one or more indicators of the emotion of the user; or a cognitive state at the time of video recording, and for providing a visual indicator.

2. The method for managing advertisement pushing based on MR glasses according to claim 1, wherein in step S211, the positioning location information is obtained by the MR intelligent glasses positioning technology including GPS/beidou satellite, wiFi module and communication signal base station, and the advertisement recommendation system is started in shopping malls, commercial streets and places with consumption behaviors for entertainment.

3. The method for advertisement push management based on MR glasses according to claim 1, wherein the step S211 further comprises:

the system calls the matching of the current time and the user work and rest rule data to judge whether to start the advertisement recommendation system;

for privacy security risk, a front-end camera is required to acquire and analyze the current surrounding environment of the user, so that advertisement content related to a scene is pushed.

4. The advertisement push management method based on MR glasses according to claim 1, wherein in the step S212, acceleration sensor data and gyroscope sensor data are collected; the acceleration sensor data and the gyroscope sensor data respectively comprise data of three sub dimensions of an x axis, a Y axis and a z axis;

5. The advertisement push management method based on MR glasses according to claim 1, wherein in step S213, the physical world image data captured by the front camera is subjected to a preprocessing procedure to perform filtering and correction of the image data; an image pre-processor segments image data into regions based on visual and/or depth image data;

The object detector through the MR glasses performs AI recognition and classification of objects in the real world image and determines the position and/or pose of the real world objects in the scene observed by the MR device.

6. The MR glasses-based advertisement push management method of claim 5, wherein objects in an image of a real world scene are identified; object recognition uses an appearance-based method, comparing an image with a reference image of a known object to identify the object; the appearance-based method comprises edge matching, gray matching and receiving a histogram of field response;

object recognition uses feature-based methods that rely on matching object features and image features; the feature-based method comprises gesture clustering, geometric hashing, scale-invariant feature transformation and interpretation tree.

7. The method for managing advertisement pushing based on MR glasses according to claim 1, wherein the step S214 specifically comprises:

the spatial analysis module is configured to identify/classify and determine boundaries of physical space observed by the MR device; the space analysis module is configured to compare the AI object recognition result of the object detector with the scene feature library and analyze whether the current environment of the user is a privacy security risk area; each or several object combinations in the scene feature library correspond to an environmental scene; identifying different sites by one or more characteristic objects; if the current environment of the user is a privacy security high risk area, the system turns off the camera or prohibits the camera from recording photo/video image data, if the current environment of the user is a privacy security low/no risk area, the camera is continuously started, the content recommendation system is started, and the environment identified by the positioning technology in step S211 is secondarily confirmed to exclude the privacy sensitive area.

8. The advertisement push management method based on MR glasses according to claim 1, wherein the step S216 detects the current biological information of the user, further comprising:

judging whether the user is possibly interested in the current image or not; a sensor data analysis module that detects and receives sensor data receives data from one or more sensing devices; the sensor data may include data of galvanic skin response, data of skin temperature, data of galvanic skin, data of brain wave activity, data of electrical activity of the heart, data of eye movement, data of facial expression, data of pupil dilation and/or constriction, data of speech volume or data indicative of speech speed.

9. The method for advertisement push management based on MR glasses according to claim 1, wherein the step S217 extracts the buffered video before and after the emotion stimulating time for image recognition, further comprising:

analyzing the received sensor data to determine if an emotional/cognitive state of the user is reached; the sensor data analysis module determines an emotional/cognitive state of the user based on the received sensor data; the sensor data analysis module analyzes the received sensor data using the deep neural network to determine an emotional or cognitive state of the user; then the system marks the time axis according to the activated emotion or cognitive state, and acquires the cache images/video/audio before and after the moment according to the time mark; further, the AI recognition technology recognizes objects or events in the images/videos, recognizes the significance of the images, and simultaneously frames and selects the image objects; emotional states include happiness, sadness, anger, fear, disappointment, or pride; cognitive states include concentration, participation, distraction, boring, drowsiness, confusion, or depression.

10. The method for advertisement push management based on MR glasses according to claim 1, wherein the step S218 further comprises:

11. The method for managing advertisement push based on MR glasses according to claim 1, wherein the step S218 presents an information prompt, further comprising:

a notification module is also used for presenting a prompt notification for the target object position, the notification module enables an Operating System (OS) of the MR glasses system to match an object identification result with an application program, and matches the notification with an application program object registered in the OS, so that the object is identified as a notification target; the OS performs this step by, for example, comparing the context Identifier (ID) contained in the notification payload with the context IDs of all objects registered in the system, and finding an object with a matching context ID; after identifying the object as the target of the notification, the operating system may call a special notification handler associated with the application of the object, but separate from the main process of the application;

12. The method for advertisement push management based on MR glasses according to claim 1, wherein the step S218 further comprises:

13. The advertisement pushing management method based on MR glasses as claimed in claim 1, wherein the interactive instruction of the user in step S219 specifically comprises: eye movement interaction, gesture recognition interaction, peripheral controller, voiced/unvoiced speech recognition, head movement interaction;

(1) The eye movement tracking device of the MR glasses captures eye movement data, detects eye movements, calculates the pupil size, iris images, eye jump track and gazing duration of eyes, and when the gazing duration exceeds a certain duration, the eye movement tracking device is regarded as performing behavior movements of corresponding eyeballs on a certain position, namely displaying content information;

(2) The gesture recognition module of the MR glasses detects that the hand makes a specific action, and then content information is displayed;

(5) The button of the interactive interface is selected through the head tracking device, the motion of the head of the user is calculated through the acceleration sensor, the gyroscope and the magnetometer of the MR glasses, a cursor fixed relative to the head is arranged in the center of the visual field of the MR eyes, and the head is moved to control the cursor selection position to perform click confirmation with the holographic UI interface, namely content information is displayed.

14. The MR glasses-based advertisement push management method as set forth in claim 1, wherein the MR glasses-based advertisement push management method further comprises determining whether the current object is of interest by recognizing a user's limb motion:

s303: privacy security validation: AI identification is carried out on the image shot by the front camera by utilizing the local calculation force of the MR glasses, the current scene of the user is analyzed by the attribute of a plurality of objects in the environment, whether the personal privacy potential safety hazard exists in the scene is judged, and if not, S304 is executed;

s305, detecting the current body motion of a user, and detecting the body motion of the user by using an acceleration sensor, a gyroscope, an IMU, an eyeball tracking device and a gesture recognition device of the MR glasses;

s306: retrieving and analyzing cached images/videos/audios before and after the time of the user's interesting physical behaviors', and identifying objects or events in the images/videos by an AI identification technology; the user 'S body action and the information of the image, video and audio recognition event are combined to judge the interest degree of the user in the event, if the user' S body action is judged to be interested, S307 is executed;

s307, the MR glasses intercept the partial images selected by the frames after the object detection in the S303 from the large images, upload the partial images to the cloud advertisement server, the cloud server carries out matching identification on the received partial images of the objects, search relevant content information in the database, and the search result is transmitted back to the terminal MR glasses; wherein the content information includes text, pictures, audio, video, and applications; meanwhile, a content prompt notice is marked and presented for a target object or an event position, and the content prompt notice is rendered and presented by an MR glasses optical display in the form of a hologram;

S308, the MR glasses wait for the interactive behavior of the user on the content prompt notification, and simultaneously the MR glasses download the server content data to a cache; if the interaction instruction of the user is received, the content information returned by the server is further presented; if the interaction instruction of the user is not received, automatically disappearing a period of time after the content prompt notification leaves the visual field picture of the user, and emptying the cache information;

15. The method for advertisement push management based on MR glasses according to claim 14, wherein the step S305 of detecting the current body action of the user comprises:

eye movement is detected by an eye movement tracking device: the saccade distance exceeds/equals a preset threshold, and/or the smooth movement behavior duration of the eye exceeds/equals a preset threshold, and/or the movement speed of the eye exceeds/equals a preset threshold, and/or the eye suddenly stops moving unordered, and/or the fixation duration of the eye at a certain point exceeds/equals a preset threshold, and/or the eye reciprocates between two points/ranges, and/or the eye tremors behavior, and/or the change of the pupil diameter of the eye exceeds/equals a preset threshold, vestibular eye movement, converging/diverging movement, smooth chasing, long saccades and correcting saccades; in normal daily activities, the eye can be tracked for voluntary and involuntary movements; distinguishing the user's intent when the user's eyes routinely perform both types of movements; this requires distinguishing between voluntary and involuntary eye movements for communicating intent;

Detecting head movement by a head tracking device: the eye signal may be combined with voluntary head movements as part of the eye signal "language"; detecting the speed, acceleration and rotation angle of a user in the process of turning the head;

the head position tracking system includes position and motion sensors, accelerometer light sources and/or other devices for acquiring the position, direction and motion of the MR glasses, by detecting motion of the entire scene in an externally facing camera; head position information may be received from a head position tracking system; a head position tracking system that provides head position information to the eye movement interaction intent prediction model; forming an Inertial Measurement Unit (IMU) on the MR glasses together; the head position information may help determine the head movement, rotation or direction of the user;

the motion tracking device detects body motion: detecting the dodging, jumping and trembling body behaviors of a user; the head tracking device of the MR eye can detect body movements on behalf of the movement tracking device, and also by smart watches, smart bracelets, any wearable smart hardware with spatially positioned controller handles.

16. An MR glasses-based advertisement push management apparatus, comprising:

a display module for presenting images to a user using the HMD device;

an eye tracking module comprising one or more scintillation sources, an image sensor, and a MEMS scanning mirror; sensor data for detecting gaze directions of each eye of the user; pupil cornea reflex method;

The buffer module is used for recording the triggering video/audio data supporting emotion/cognitive state; the system comprises a buffer, a sensor data analysis module, a recording module, a gaze detection module and a video segment memory;

The video segment memory is configured to store video segments recorded based on emotional or cognitive states of the user; when the recording module starts recording, copying the video and audio data stored in the buffer to a new video segment, which is stored in the video segment memory; the recording module records the additional video and audio data to the video clip; the video segment memory also stores metadata associated with the video segments, which may include an indicator of an emotion or cognitive state of the user triggering the recording, one or more indicators of the emotion of the user; or a cognitive state at the time of video recording and for providing a visual indicator;

17. The MR glasses-based advertisement push management apparatus of claim 16, wherein the control module comprises:

a data holding unit for storing data using one or more storage devices;

18. The MR glasses-based advertisement push management device of claim 16, wherein the MR glasses-based advertisement push management device is provided in a head-mounted display device, HMD, worn by a user, by tracking eye gaze direction, and user head gaze direction and head pose motion, comprising:

19. The MR glasses-based advertisement push management device of claim 16, wherein the MR glasses comprise a location subsystem arranged to determine a current location based on signals received from a navigation satellite system; the location subsystem is arranged to determine a location based on radio frequency signals identifying the transmitting device and the location determined for such device;

the MR glasses further include a controller including a logic subsystem, a data-holding subsystem, and a communication subsystem;

The logic subsystem includes one or more processors that are configured to execute instructions and communicate;

the data-holding subsystem includes one or more storage devices;

the MR glasses comprise a biological characteristic sensor, one or more biological characteristic sensors, an electroencephalogram device for measuring brain electrical activity, an electrocardiogram for measuring heart electrical activity, a microphone or other sound sensor for measuring voice volume and speech speed, a light sensor and an optical scanner;

the input/output devices include a keyboard, a keypad, a mouse, a microphone, a touch sensor, a touch screen, a joystick, control buttons, scroll buttons, a camera.

20. A computer device, characterized in that the computer device comprises a memory and a processor, the memory stores a computer program, and the computer program when executed by the processor causes the processor to execute the advertisement push management method based on MR glasses according to any one of claims 1 to 15.

21. A computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the MR glasses-based advertisement push management method according to any one of claims 1 to 15.

22. An information data processing terminal, which is characterized in that the information data processing terminal is used for realizing the advertisement pushing management method based on the MR glasses according to any one of claims 1-15.