WO2020125499A1 - 一种操作提示方法及眼镜 - Google Patents

一种操作提示方法及眼镜 Download PDF

Info

Publication number
WO2020125499A1
WO2020125499A1 PCT/CN2019/124352 CN2019124352W WO2020125499A1 WO 2020125499 A1 WO2020125499 A1 WO 2020125499A1 CN 2019124352 W CN2019124352 W CN 2019124352W WO 2020125499 A1 WO2020125499 A1 WO 2020125499A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
image
semantic map
item
eye
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/124352
Other languages
English (en)
French (fr)
Other versions
WO2020125499A9 (zh
Inventor
程俊
王鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Institute of Advanced Technology of CAS
Publication of WO2020125499A1 publication Critical patent/WO2020125499A1/zh
Publication of WO2020125499A9 publication Critical patent/WO2020125499A9/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods

Definitions

  • the invention belongs to the technical field of data processing, and particularly relates to an operation prompting method and glasses.
  • the embodiments of the present invention provide an operation prompt method and glasses to solve the problems in the prior art that users need a lot of manual operations to view and compare to be able to determine the errors in their own operations, and the operations are cumbersome and inefficient.
  • a first aspect of an embodiment of the present invention provides an operation prompt method, including:
  • the operation mode includes an operation step for one or more of the items
  • an operation prompt corresponding to the operation step is output.
  • a second aspect of an embodiment of the present invention provides glasses that include a memory and a processor, and a computer program that can run on the processor is stored on the memory, and the processor executes the computer program To implement the steps of the operation prompt method as described above.
  • the beneficial effects of the embodiments of the present invention are: by constructing a 3D semantic map of the environment in which the user is located and identifying the user's gaze behavior of the item, that is, determining that the user has an operation prompt requirement, according to the 3D semantic map
  • the actual situation intelligently identifies the user's required operation mode (that is, the operation tutorial on the item), and monitors and prompts the user to operate according to the user's actual operation of the item, so that the user does not need to enter any manual operation input, nor Watching the screen proactively, you can learn about the problems in your operation in a timely manner, which realizes the intelligent matching of user operation tutorials and greatly improves the user's efficiency of learning about operation errors.
  • FIG. 1 is a schematic flowchart of an implementation of the operation prompt method provided in Embodiment 1 of the present invention
  • FIG. 2 is a schematic diagram of an implementation process of an operation prompt method provided in Embodiment 2 of the present invention.
  • FIGS. 3A and 3B are schematic diagrams of an implementation process of the operation prompt method provided in Embodiment 3 of the present invention.
  • FIGS. 4A and 4B are schematic flowcharts of an operation prompt method provided by Embodiment 4 of the present invention.
  • Embodiment 5 is a schematic diagram of an implementation process of an operation prompt method provided in Embodiment 5 of the present invention.
  • FIG. 6 is a schematic diagram of an implementation process of an operation prompt method provided in Embodiment 6 of the present invention.
  • Embodiment 7 is a schematic structural diagram of an operation prompt device provided in Embodiment 7 of the present invention.
  • Embodiment 8 is a schematic diagram of eyeglasses provided by Embodiment 8 of the present invention.
  • the embodiments of the present invention will be briefly described here.
  • the embodiments of the present invention will first construct a corresponding 3D semantic map based on the image of the user environment to determine the user The actual situation of the environment and the objects contained in the environment, but considering the actual situation, even if the user is in an environment, there may not be operational requirements, for example, the user may only pass a certain location, or the user may be in In a certain location, but no operation is required, so in order to prevent false triggering, in this embodiment of the present invention, an eye movement analysis is performed on the user to determine whether the user has gaze behavior on the item, and when the user has gaze behavior, It is determined that the user has a demand for operation prompts.
  • the operation tutorial required by the user is intelligently matched.
  • the user's real-time operation of the item is analyzed, and the user operation does not meet the operation tutorial
  • corresponding prompts for correct operation steps should be provided, so that the user does not need to perform tedious manual operations, and can also obtain the required operation tutorials in time, and timely know the problems and correct operation methods in their own operations.
  • the specific execution subject in the embodiments of the present invention may be different according to different requirements of actual application situations, for example, it may be a smart device such as a wearable device (such as smart glasses) or a device such as a server,
  • the execution subject is a wearable device
  • all data collection, processing, and output operations in the embodiments of the present invention are performed by the wearable device
  • the execution subject is a device such as a server that cannot directly collect and output data to the user
  • the work of collecting and outputting user data is performed by other devices, that is, the direct objects of data acquisition and output in the embodiments of the present invention are not users, but other devices that can perform data collection and output to users to achieve
  • the purpose of the user's operation prompts for example, the smart glasses are responsible for collecting and sending the user's data to the server for processing, and the server sends the processed prompts to the smart glasses, and then the smart glasses finally output to the user, and the specific execution of the main body It can be a smart device such as a wearable device
  • FIG. 1 shows an implementation flowchart of the operation prompt method provided in Embodiment 1 of the present invention. Details are as follows:
  • S101 Acquire an image of the environment in which the user is located, and construct a 3D semantic map of the environment in which the user is located based on the image.
  • image collection is first performed on the environment where the user is located, and a 3D semantic map is constructed.
  • the 3D semantic map is a three-dimensional map of the environment that contains semantic information.
  • a wide-angle camera can be set in smart glasses to collect environmental images. After acquiring the environmental images, a three-dimensional environmental map is constructed based on these images, and the items contained therein and the related items are identified. Attribute data.
  • methods for constructing 3D semantic maps include, but are not limited to, stereoscopic vision-based 3D semantic map algorithms or other construction algorithms, which are not limited here, and can be selected or set by technicians according to actual needs, or refer to the present invention Other relevant embodiments.
  • S102 Perform eye movement recognition on the user, and determine whether the user gazes at the items included in the 3D semantic map.
  • the embodiment of the present invention will recognize whether the user has gaze behavior on the item, and will only perform subsequent operation tutorial matching and other operations when it is determined that the user has gaze behavior, specifically, the embodiment of the invention
  • devices for taking eye movements of the user such as eye trackers, are built in, and these devices collect eye image data of the user and perform eye movement analysis on the collected eye images to determine the user What is the eye movement situation, such as whether the user blinks, where is the gaze area, and whether there are gaze items, etc.
  • the specific eye movement analysis and tracking method can be used to determine the user What is the eye movement situation, such as whether the user blinks, where is the gaze area, and whether there are gaze items, etc.
  • the embodiment of the present invention will analyze the 3D semantic map of the user environment to determine the actual environment Situation, such as kitchen or equipment operation room, etc., then according to the environment to estimate the user's possible operations and corresponding required operation tutorials.
  • the operation tutorial may be preset by the technician, and then selected according to the 3D semantic map scene category, etc., or may be preset by the technician to set a certain operation tutorial generation rule, and then based on the 3D semantic map Scene category, etc. to generate the operation tutorial.
  • the operation mode when the operation mode is determined, it includes:
  • the actual possible needs of the user will be determined according to the scene category corresponding to the user's 3D semantic map and the actual contained items, for example, the user is in the kitchen, And it contains a lot of food, it means that the user is very likely to need to cook, at this time, some corresponding cooking tutorials can be generated.
  • the specific scene category recognition method and item recognition method are not limited here, and can be set by technicians according to actual needs.
  • the operation mode when the operation mode is determined, it includes:
  • the items and user data contained in the 3D semantic map obtain the corresponding operation mode.
  • the actual operation requirements of the user are determined according to the scene category of the environment, the actual contained items, and some user data of the user, and the corresponding operation tutorial is selected.
  • the specific content of the user data can be set by the technician or the user, including but not limited to, such as the user's personal information and hobby data, or some operation requirements preset by the user, for example, the user presets his own dietary requirements.
  • S104 Monitor whether the user's operation on the item meets the requirements of the operation steps.
  • the embodiment of the present invention After obtaining the operation tutorial required by the user, the embodiment of the present invention starts monitoring and identifying the actual operation of the user, and compares with the operation steps in the operation tutorial to determine whether the user's actual operation meets the requirements of the operation tutorial, thereby Determine the user's operational defects.
  • the camera in the smart glasses will photograph the user's behavior, and conduct behavior analysis on the obtained image/video, that is, analyze the user's behavior and behavior Whether the order etc. meet the requirements in the operation tutorial.
  • the embodiment of the present invention will prompt the user for the operation, that is, inform the user that there is an error in the current operation, and will also inform the user of the correct Operation steps, for example, when operating the device, after pressing the power button, you need to use the mode key to select the device mode, and then press the start key to proceed, and if the user presses the power button, directly press the start key, this
  • the output method of the operation prompt includes, but is not limited to, prompts such as audio/video/text, which can be set by the technical personnel.
  • Embodiment 1 of the present invention includes:
  • Augmented Reality (AR) format technology is used to output prompt information, so that users can more intuitively know their own problems and corresponding correct operations.
  • the operation mode required by the user is intelligently identified according to the actual situation of the 3D semantic map ( It is the operation tutorial of the item), and monitors and prompts the user's operation according to the actual operation of the item by the user, so that the user does not need to enter any manual operation input, nor need to actively watch the screen all the time, you can know in time
  • the problems in their own operation have realized the intelligent matching of user operation tutorials, which has greatly improved the efficiency of users' knowledge of operation errors.
  • the environment image to be collected includes a color image and a depth image of the environment, as shown in FIG. 2,
  • Embodiment 2 of the present invention include:
  • the smart glasses use RGB-D cameras as sensors to acquire color images and depth images, and use the visual SLAM algorithm to complete the autonomous positioning and posture estimation and optimization of smart glasses (that is, the user's position information and posture information acquisition ), at the same time carry out item detection to obtain the semantic information of the environment, and then use RGB-D segmentation algorithm to segment the perceived items, thereby constructing a 3D semantic map of the environment.
  • the visual SLAM basic framework consists of four parts: visual odometry, back-end optimization, loopback detection, and 3D mapping.
  • the visual odometer needs to complete the motion estimation between two adjacent frames, and roughly estimate the current pose of the camera.
  • the back-end optimization is to globally and consistently optimize the estimation of the visual odometer to eliminate noise interference.
  • the constraints of the loop detection are used to optimize the pose, so that the positioning and pose estimation are more accurate.
  • Loopback detection is to eliminate the cumulative error of the process when returning to the original passing position.
  • the construction of the map creates a three-dimensional map of the environment based on the estimated motion and pose of the first three parts.
  • the overall design of the visual SLAM algorithm realizes the global consistent estimation of the current position and posture through the sensor input data, which is clear for its own positioning and movement.
  • the visual SLAM algorithm is coordinated by three parts: visual odometer, back-end optimization and loopback detection.
  • the visual odometer extracts the ORB feature points, uses the FLANN algorithm to match the two adjacent ORB features, and uses the matching results to use the PnP algorithm and the RANSAC algorithm to complete a rough estimate of the position and pose of smart glasses.
  • the task of the visual odometer is to estimate the changes in the pose of the smart glasses between the two images, and to estimate the posture and movement trajectory of the smart glasses over a period of time. It consists of the following processes:
  • ORB feature extraction is composed of FAST key points and BRIEF descriptors, and artificially endows rotation and scale invariance.
  • Scale invariance is realized by the principle of pyramid.
  • the image pyramid is down-sampled to obtain the image pyramid, and the above four-step feature extraction is completed for each layer to achieve scale invariance of FAST key points.
  • Rotation invariance is achieved by the gray-scale centroid method, calculating the centroid of the image block U with the key point as the center, defining the vector from the center to the centroid as the direction of the key point, and achieving the rotation invariance of the FAST key point
  • Descriptor Make improvements based on BRIEF. One is to consider all the points in the 31 ⁇ 31 neighborhood of the key point, after Gaussian smooth filtering of the image, select the gray average value in the 5 ⁇ 5 neighborhood instead of the single point gray for calculation, and the noise resistance is strong. The second is to use an uncorrelated greedy search algorithm with a mean value close to 0.5 when selecting 5 ⁇ 5 neighborhoods, which ensures the representativeness and uniqueness of the descriptor and makes it distinguishable.
  • the core idea is to use the derived index structure to locate the search range in the vicinity of the feature, and complete the feature matching in the neighboring area, which can effectively speed up the matching speed.
  • the local sensitive hash is used as the derived index structure. Projecting features into the hash space in the same way, the probability that two adjacent features are still adjacent in the hash space after projection will be very high, and the two features that are not adjacent before are projected in the hash space. The probability of neighboring will be very small, so that it can be matched in the neighborhood of the hash space, effectively reducing the scope.
  • the matching relationship is used to estimate the motion and pose of the smart glasses.
  • the process of imaging smart glasses is also called the observation process, that is, the process of reflecting or emitting light from a point in three-dimensional space, projecting onto the imaging plane of the smart glasses through the optical center of the smart glasses.
  • the invention adopts the PnP algorithm to roughly estimate the pose between two frames, and then uses the RANSAC algorithm to optimize the consistency of the pose between frames, so as to avoid that the mismatching feature has a serious impact on the pose estimation.
  • the RANSAC algorithm is used to optimize this problem.
  • the ORB feature points of the image are extracted, and the fast nearest neighbor algorithm is used to match the features of the two adjacent image frames. Finally, the PnP algorithm and the RANSAC algorithm are used to complete the camera motion and pose estimation.
  • Back-end optimization is to globally and consistently optimize the estimation of the visual odometer, eliminate noise interference, and make the pose estimation more accurate.
  • the system detects a loopback, it passes the information to the back-end to eliminate accumulated errors.
  • the embodiment of the present invention introduces a key frame mechanism, selects representative image frames for pose optimization, and reduces unnecessary calculations.
  • a bundle adjustment algorithm is used to optimize the pose and feature points of the camera currently in the convergence process.
  • the back-end uses a pose pattern optimization method to obtain a globally consistent trajectory and pose.
  • the back-end optimization receives the posture and feature points from the visual odometer, and optimizes it by means of bundle adjustment.
  • the work flow is: checking the queue, processing key frames, culling, generating and fusing map points, and adjusting and optimizing the position of the local bundling.
  • the current frame has enough feature points and matching to ensure the feature richness.
  • Loopback detection mainly solves the problem of gradually accumulating errors in pose estimation of smart glasses. When you come to the place where you have been once again, this part confirms and establishes the connection between the current pose and the historical pose, and transmits it to the back end for optimization processing, which eliminates the accumulated error of the system for a long time and obtains the overall Consistent trajectory and posture.
  • loopback detection provides the correlation between current data and all historical data. When visual odometry loses track of features, loopback detection can be used to relocate, enhancing the robustness of pose estimation.
  • Loopback detection is continuously detected while the system is running, and the cumulative error of the pose estimation of smart glasses is eliminated by the occurrence of loopback constraints.
  • the closed-loop constraint relationship when the camera returns to a certain previous position is passed to the back end for posture optimization.
  • the workflow of loopback detection includes: Step 1. Detection of loopback candidate frames. Step 2. Establish contact with the previous key frame. Step 3. Check whether loopback occurs. If it does not occur, return to step 1. If it occurs, go to step 4. Step 4. Pose image optimization.
  • Treating features as words pre-training a dictionary containing all feature types, and generating a set of corresponding words according to the dictionary for each feature of the image, that is, a bag of words. Therefore, when judging the similarity of images, it is only necessary to compare their word bags, which greatly accelerates the speed of loop detection.
  • Feature clustering uses the unsupervised machine learning K-means++ algorithm, and uses the K-d tree structure to improve search efficiency.
  • the training process of the dictionary is:
  • K-mean++ is also used to cluster the samples belonging to the node into k categories to obtain a new layer.
  • the leaf layer is the word corresponding to the feature.
  • TF-IDF TF-IDF algorithm. If the similarity between the current frame and a previous key frame exceeds 3 times the similarity between the current frame and the previous key frame, it is considered that a loopback may have occurred. However, a verification step is still needed to set up a cache mechanism for loopback detection. A single high degree of similarity is not enough to determine loopback. When the similarity of consecutive frames is high, the loopback is confirmed. After confirming that the loopback occurs, the loopback detection part sends this information to the back-end. The back-end uses graph optimization to optimize the pose to eliminate the cumulative error and obtain a globally consistent trajectory and pose.
  • the visual SLAM algorithm design consists of three parts: visual odometer, back-end optimization, and loopback detection.
  • the visual odometer part first extracts the ORB features of the sensor input frame, and then uses the fast nearest neighbor algorithm to match the features of the adjacent two frames of the image.
  • the PnP algorithm and the RANSAC algorithm are used to complete the camera pose estimation.
  • the rear part optimizes the posture and feature points from the visual odometer by means of bundle adjustment.
  • Loopback detection uses the bag of words model and the trained dictionary to detect whether the camera has returned to the previous location. If loopback occurs, this constraint information is sent to the backend.
  • the backend uses graph optimization to optimize the pose and eliminate accumulated errors. , To ensure the global consistency of the camera trajectory and pose.
  • the item detection will be performed from the color image first, wherein the detection algorithm includes but is not limited to the Yolv V3 object detection algorithm, which is not limited here, and from Item segmentation in color images is detailed below:
  • the depth map of the corresponding pixel is also acquired, that is, the three-dimensional point cloud. Therefore, before building a semantic map, it is necessary to segment the detected objects, and then use the visual SLAM module to estimate and optimize the camera pose to project pixels to the position in space, thereby constructing a three-dimensional semantic map.
  • CPF Constrained Plane Fitting
  • S202 Construct a 3D semantic map according to the user's position information and posture information, and item information and position information of the item.
  • the optimized key frame's position and posture of the smart glasses that is, the user's position information and posture information
  • the category of the item in the key frame that is, the item information
  • its position and Three-dimensional segmentation of the detected target object The next thing to do is to integrate the acquired information to build a globally consistent three-dimensional semantic map.
  • the general process is divided into three steps: firstly perform consistent fusion of data and update the target object model, then build a three-dimensional map of the environment, and finally merge the semantic information Go to the three-dimensional map to get a three-dimensional semantic map with rich information.
  • an improved RGB-D segmentation algorithm of GrabCut is designed, which combines the information of GrabCut segmented color map and CPF segmented depth map, segmented the detected target, and completed the target extraction of the semantic map in units of objects jobs. Mark them according to the object category, and then use the consistent fusion of the target object to update the model of the object object to avoid modeling the same target multiple times. Furthermore, a three-dimensional semantic map with rich information is constructed and stored with a colorful octree map structure.
  • the role of data association is to determine whether the target is already in the map when the result of the RGB-D segmentation of the target is obtained, whether new objects need to be added or existing objects are maintained, to avoid modeling the same object multiple times and thus in the map ghosting appears.
  • a set of boundary markers of candidate objects is selected based on the Euclidean distance of each centroid of the point cloud after segmentation. Then, the nearest neighbor search is performed on the existing boundary marker of the map and the current target's 3D point, and the Euclidean distance of the relevant point pair is calculated.
  • the Euclidean distance of two three-dimensional points is the 2-norm of the two points.
  • the target is considered to be the same as the existing target of the map, and the current target information is related to the existing target of the map for common maintenance
  • the target model when the environment is more complicated, the nearest neighbor search for three-dimensional points is accelerated by seeking a k-d tree structure. In order to ensure that the data association can obtain the latest information, as long as the object is detected, it is segmented using the improved RGB-D target segmentation algorithm designed above. In this way, each object in the map retains three pieces of information: the target model obtained through data association, the key frame pose of the observed target, and the probability of each category given by the target detection module.
  • the probability of the target object in the map can be updated according to the probability value given by the target detection module, suppose a total of class C objects are currently detected, Sc represents a vector consisting of the probability of each category of the target, n is the number of key frames detected by this target, Update calculation of target detection probability:
  • the invention adopts a map form that is flexible, occupies a small amount of memory and supports real-time update: octree map
  • the entire space is used as the root node, which is divided into eight child nodes according to the space coordinate system, and then each child node is further divided into eight child nodes until it reaches the required resolution, that is, the leaf node.
  • the difference between the voxel model of the octree map and the point cloud map is that when all points in a block are occupied or unoccupied, there is no need to expand this node, which occupies relatively little memory space.
  • the speed of searching for leaf nodes is very fast, and the time complexity of the octree in layer d is O(d).
  • the octree map can support the setting of the color of each node, that is, the form of a color octree map. It also supports updating and correcting information at any time, which is very suitable for constructing a three-dimensional semantic map. Therefore, this paper chooses the color octree map as the construction and storage form of the three-dimensional semantic map.
  • a three-dimensional semantic map To build a three-dimensional semantic map, first establish a three-dimensional map of the environment and continuously update it, and then integrate the semantic information into the three-dimensional map in real time, that is, a three-dimensional semantic map containing rich information can be constructed. In the process of camera movement, information is continuously acquired and processed, and the semantic map is continuously updated.
  • the task of 3D mapping is to map the depth information captured by the RGB-D camera to the 3D space according to the estimated and optimized position and posture of the key frame in the visual SLAM algorithm, and establish a 3D map of the environment. Because the RGB-D camera used in the present invention can obtain the depth of each pixel in the field of view, the depth map can be used directly for dense construction, and the depth map is mapped to a point cloud and stitched according to the optimized camera pose to obtain a three-dimensional map .
  • the semantic map is a three-dimensional map containing semantic information, that is, the semantic information of the environment is marked when the octree map is established.
  • the semantic information is also integrated into the octree structure to obtain a three-dimensional semantic map with rich information.
  • the semantic mapping system of the present invention uses an RGB-D camera as a visual sensor to capture color information and depth information.
  • the visual SLAM algorithm is used to complete the autonomous positioning, pose estimation and optimization of AR smart glasses, and obtain a globally consistent trajectory and pose.
  • the convolutional neural network YOLOv3 model is used for target detection, which detects the object type, probability and position of the key frame, and obtains the semantic information of the environment.
  • the RGB-D segmentation algorithm is used to segment the sensed objects, and the three-dimensional map of the environment is constructed in the form of an octree map.
  • the semantic information is integrated into the octree map to complete the construction of the three-dimensional semantic map of the environment.
  • the embodiment of the present invention designs a visual SLAM algorithm using an RGB-D camera as a sensor.
  • the present invention selects the ORB feature as the basis of the algorithm, which not only provides the spatial geometric relationship in the pose estimation of the visual odometer, but also serves as a standard for judging the similarity of images during loopback detection, and achieves the unity of the system to a certain extent.
  • the PnP algorithm is used instead of the ICP algorithm.
  • the PnP algorithm uses the optimized camera coordinates of the previous frame and the pixel coordinates of the current frame to calculate the pose to avoid the interference of camera measurement errors.
  • the back-end optimization part is processed by targeted algorithms.
  • the optimization of the pose and map points transmitted by the visual odometer uses the local bundling adjustment algorithm.
  • the closed-loop constraint optimization from the loop detection is optimized by the graph optimization method. .
  • the overall structure of the semantic mapping system is designed.
  • This paper designed an improved RGB-D segmentation algorithm of GrabCut, combined with the CPF segmentation result using depth point cloud information to modify the GrabCut segmentation using only color image information, to achieve the complementarity of the two performances.
  • the objects detected by the target are segmented and marked according to the object category, the camera pose and target detection and segmentation results obtained by the visual SLAM algorithm are integrated, and a three-dimensional semantic map of the environment is constructed and stored in a color octree map structure.
  • Running the system in a laboratory environment the system builds a readable and accurate three-dimensional semantic map while positioning itself, estimating the pose, and semantic perception, which verifies the feasibility of the semantic mapping solution of the present invention.
  • the third embodiment of the present invention includes:
  • S301 Acquire a user's eye image, perform pupil positioning on the eye image, and determine the user's gaze area in the 3D semantic map based on the obtained pupil position information.
  • the user's line of sight is tracked based on the user's pupil and Purchin spots, and the area where the user gazes is determined. Therefore, first, the position of the pupil in the eye needs to be determined.
  • the specific pupil recognition algorithm can be set by the technicians according to their needs, including but not limited to, such as using a neural network model to train the sample data of the pupil image and identify the pupil in the eye image, or refer to Embodiment 4 of the present invention To identify. Since the technique of tracking the user's line of sight based on the pupil and Purchin spot is relatively mature, it will not be repeated here.
  • the duration threshold can be set by the technical personnel.
  • the embodiments of the present invention considering that the user's head will move, the line of sight will change when performing the above line of sight tracking.
  • the embodiments of the present invention include:
  • the natural movement of the head includes two basic movements: vertical pitch movement and horizontal left and right movement.
  • the general mapping model based on the quadratic polynomial is obtained by the user through multi-point calibration while keeping the head still. When the head position changes, the error of the estimated fixation point obtained by the model greatly increases. Based on this, this paper proposes a head motion solution algorithm based on polynomial mapping. The algorithm needs to use head motion estimation equipment to obtain head motion information in real time.
  • the coordinate of the estimated point is combined with the initial eye position to establish a three-dimensional line of sight direction, and then use the head motion information to rotate and translate the line of sight direction, and use the intersection point of the currently compensated line of sight and the screen as the final The estimated coordinates of the fixation point.
  • the effect of head movement is due to head movement that changes the position of the eye, we can compensate for the line of sight as long as we know the change in the position of the eye.
  • a line-of-sight estimation model under natural movement of the head is established.
  • FIG. 3B the figure shows a side plan view of the line-of-sight estimation principle
  • O1 represents the initial eye position
  • O2 represents the eye position after exercise.
  • the corresponding pupil corneal vector pccr1 is obtained.
  • the eye moves to position O2, assuming that pccr2 is equal to pccr1, that is, the eye pattern features do not change.
  • the corresponding line of sight will be deflected to the right.
  • the intersection with the screen is S2.
  • the error of line of sight estimation with pccr2 is large.
  • this point as the initial estimation point of sight estimation, establish the sight direction g1 from this point and the initial position of the eye, and then use the angle information of the head movement to correct g1, and the intersection of the corrected sight direction and the screen It is the estimated point of the current fixation point.
  • Embodiment 4 of the present invention includes:
  • S401 Divide the eye image into N ⁇ M area images, and binarize the grayscale of all the area images to obtain corresponding N ⁇ M eye grayscale values, where N and M are both positive integers.
  • the eye features in the embodiment of the present invention are described in detail as follows.
  • the basic rectangular blocks in the figure have the same size, ABCD is the most original rectangular feature in the figure, E is composed of 3 basic rectangles, and F is 9 A rectangle, G is a rectangle, H and I are composed of 4 basic rectangles, J is composed of 12 rectangles, K and L are composed of 4 rectangles, the calculation of each rectangle feature is the black part of the picture and the subtraction To remove the pixel sum of the white part, the feature G here is a single rectangular feature, so only the pixel sum within the rectangle is calculated.
  • the eye features are designed based on the structure of the eye. Due to the change in the brightness of the eye corners and the surroundings, the eye corners are darker than the surrounding pixels.
  • Feature F can express this feature very well.
  • the center point of the eyeball is basically black. Therefore, a simple rectangle G represents this characteristic of the eyeball.
  • Two more obvious pixel mutations will occur in the horizontal direction of the eye. From the sclera to the iris, and then from the iris to the sclera, the feature H can reflect this change.
  • the vertical direction of the eye also has similar gray-scale change characteristics. The I characteristics are generated based on this characteristic. Together with C and E, they can enhance the description of the horizontal and vertical gray-scale changes of the eye.
  • J represents the corner of the eye.
  • K and L represent the edge information of the corner of the eye with the change of the gray level of the part between the eyeballs. Adding new features reduces the number of eye features used for classification, making eye detection easier.
  • the pupil positioning in the embodiment of the present invention adopts a frame-independent design idea.
  • Careful observation of the human eye screenshot reveals that many parts of the human eye image signal will change with the incident light, the mirror reflection of different people and eyes, and the relative positions appearing in the screenshot are different with the different angles of the face.
  • the effect of direct classification training is not satisfactory.
  • we found that the pupil area is relatively stable image information in the human eye screenshot, and this part of the feature is obvious when the eyes are opened. Therefore, we first locate the pupil to simplify the complexity of the problem.
  • N and M are both positive integers, and the specific value is set by the technician himself.
  • S402. Acquire the skin image of the user, and calculate the average gray value after the skin image gray binarization process.
  • the embodiment of the present invention also collects the user's skin image and calculates the corresponding average grayscale value.
  • the skin image is preferably a skin image around the eye.
  • the embodiment of the present invention calculates the eye gray value of each area image separately from the average gray value of the skin
  • the absolute value of the difference, and the area images that meet the preset grayscale threshold requirements will be selected in order from small to large, and after each area image is filtered, the pixels contained in all the filtered area images are counted once The number of points until the number of pixels included in the selected area image is within a preset number range to ensure accurate and reliable pupil recognition.
  • the specific values of the preset gray threshold and the preset number threshold can be set by a technician.
  • the embodiment of the present invention includes:
  • the Places205-AlexNet network model follows the same network architecture as AlexNet but it is specifically trained on scene classification tasks.
  • the training data set contains about 2.5 million pictures of 205 semantic classes, and each class has at least 5000 sample pictures. These pictures come from various network resources, such as Google pictures, Bing, Flickr, etc., and classify the pictures.
  • the number and types of samples in the training data set ensure that the resulting classifier can be generalized well, and it is not necessary to perform secondary training or fine-tuning when it is applied in an environment where it has never been trained. This ensures that the semantic mapping system in this paper is portable and can be used by users working in multiple environments.
  • the input to the Places205-AlexNet network model is an RGB image adjusted to 224 ⁇ 224 ⁇ 3 pixels, regardless of their initial size.
  • the Places205-AlexNet convolutional neural network model has a total of 8 layers of networks, including the first 5 convolutional layers and the next 3 fully connected layers. Given the picture It of the current scene, the output layer of the network soft-max outputs the discrete probability points p(o t
  • the classifier in this paper uses the output of the fc7 layer of the Places205-AlexNet network model as a feature vector. The fc7 layer is the last universal (ie class-independent) fully connected layer in the network. Since the purpose of the Places205-AlexNet network model design is to identify 205 scene categories in the Places205 scene dataset, the final fc8 layer and prob layer have 205 output neuron nodes.
  • the output layer prob of the network outputs a discrete probability distribution p(o t
  • the probability that the image It at time t belongs to the scene category Ci is independent of each other. Since the two adjacent pictures acquired by the camera are continuous in time, the recursive Bayesian filtering technique can be applied.
  • This paper describes the robot's scene classification problem as a probability estimation problem and estimates the discrete probability distribution on all possible scene labels Ci. All the observation pictures from the past to the present are known. Assuming that the first-order Markov property is satisfied, the following Bayesian filter formula can be obtained:
  • the embodiments of the present invention include:
  • S502 Acquire a mode selection instruction of a user pair, and select one operation mode from a plurality of operation modes based on the mode selection instruction.
  • the user can select one of them to perform subsequent operations.
  • the input method of the mode selection instruction includes, but is not limited to, for example, the user inputs by means of voice/eye movement/head posture transformation, for example, relevant information of various operation tutorials is displayed in the smart glasses, and the user selects by voice , Eye movements (such as blinking, staring, etc.) or changes in head posture (such as shaking the head switching operation tutorial, nodding to confirm, to select the command input to achieve the mode), the specific needs to be set by the technical staff according to the actual needs. .
  • the user in the embodiment of the present invention can perform various eye movement operations to perform information interaction operations with the smart glasses, including:
  • the main gaze point used to mark the reading behavior (the gaze target has been confirmed and no more recognition time is required) is within a preset range of time length, and the preset range is preferably 600-1100 milliseconds.
  • the glance data can be used to define the lookback, that is, the fixation point coordinates fall in the space formed by the first 5 fixation points as the center of the circle and 1 fixation area as the radius, but the first fixation point of the fixation point is not counted.
  • the overall focus change is that the saccade length exceeds 3 fixation areas and more after the last main fixation point (duration is longer than the preset duration, preferably 1100 milliseconds or more).
  • the local attention point change is that after the last main fixation point (the duration is longer than the preset duration, preferably 1100 milliseconds or more), the length of multiple saccades does not exceed 3 fixation areas, but the total distance of its position change exceeds 3 fixation areas. behavior.
  • Search behavior When there is an overall focus change or a local focus change behavior, the user may be considered to be searching.
  • the overall focus change is a typical search behavior, while the local focus change is that the user believes that the general area has been found and is continuing to look for a specific target.
  • Persistent search The behavior when 10 saccades and more saccades appear continuously after the last main fixation point (the duration is longer than the preset duration, preferably 1100 milliseconds or more), it can be considered that the user is thinking, distracting, or resting.
  • the user can perform eye movement control on the smart glasses through the above method, so as to realize various types of human-computer interaction operations required in other embodiments of the present invention.
  • Embodiment 6 of the present invention includes:
  • S602 Acquire a preset attribute threshold corresponding to the item involved in the operation mode, and identify whether the item attribute data of the item meets the preset attribute threshold during the operation of the item by the user.
  • the embodiment of the present invention not only monitors the user's operation behavior when monitoring the user's operation, but also checks the attributes of the user's operation items. According to monitoring, to prevent the occurrence of such errors but unrecognized situations.
  • a corresponding attribute threshold value is set for each item involved in the operation tutorial.
  • the baking time of the oven is set to 1 hour and the temperature is 120 degrees, and Identify the attribute data of the items during the user's operation, and determine whether the changes in the item attribute data caused by the operation are within the threshold requirement range.
  • determine whether the baking time and temperature of the oven set by the user are 1 hour and 120 degrees. (It can be recognized according to the image taken by the smart glasses when the user sets the oven).
  • the embodiments of the present invention will determine the user's The operation of the item does not meet the requirements of the step in the operation tutorial.
  • FIG. 7 shows a structural block diagram of an operation prompt device provided by an embodiment of the present invention.
  • the operation prompting device illustrated in FIG. 7 may be an execution subject of the operation prompting method provided in the foregoing first embodiment.
  • the operation prompting device includes:
  • the map building module 71 is used to obtain an image of the environment in which the user is located, and construct a 3D semantic map of the environment in which the user is located based on the image.
  • the gaze recognition module 72 is used to perform eye movement recognition on the user and determine whether the user gazes at the items contained in the 3D semantic map.
  • the mode obtaining module 73 is configured to obtain an operation mode corresponding to the 3D semantic map if the user looks at an item included in the 3D semantic map, and the operation mode includes operation steps for one or more of the items .
  • the operation monitoring module 74 is configured to monitor whether the user's operation on the item meets the requirements of the operation step.
  • the operation prompt module 75 is configured to output an operation prompt corresponding to the operation step if the user's operation on the article does not meet the requirements of the operation step.
  • map building module 71 includes:
  • position information and posture information of the user, and position information and item information of the article in the environment where the user is located are obtained.
  • the 3D semantic map is constructed according to the user's position information and posture information and the item information and position information of the item.
  • the gaze recognition module 72 includes:
  • the pupil positioning module is used to acquire the eye image of the user, perform pupil positioning on the eye image, and determine the gaze area of the user in the 3D semantic map based on the obtained pupil position information.
  • the duration statistics module is used to identify the items contained in the gaze area and count the continuous duration of the items contained in the gaze area in the gaze area.
  • the gaze confirmation module is configured to determine that the user gazes at the items contained in the 3D semantic map if there is a duration longer than a preset duration threshold in the continuous duration.
  • the pupil positioning module includes:
  • image filtering is performed from the region image corresponding to the difference between the absolute value of the gray value of the eye and the average gray value less than the preset gray threshold, until The number of pixels included in the area image obtained by the screening is within a preset number threshold range, and the area image corresponding to the pupil is obtained to determine the position of the pupil in the eye image.
  • the mode acquisition module 73 includes:
  • the mode acquisition module 73 further includes:
  • the scene recognition module is used to recognize the scene category corresponding to the 3D semantic map and obtain the user data of the user.
  • the mode determination module is used to obtain the corresponding operation mode based on the scene category, the items included in the 3D semantic map and the user data.
  • the mode determination module includes:
  • the operation monitoring module 74 includes:
  • the behavior does not satisfy the requirements of the operation step, and/or the item attribute data does not satisfy the preset attribute threshold, it is determined that the user's operation of the item does not meet the requirements of the operation step.
  • the operation prompt module 75 includes:
  • first, second, etc. are used in the text to describe various elements in some embodiments of the present invention, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.
  • the first table may be named the second table, and similarly, the second table may be named the first table without departing from the scope of the various described embodiments. Both the first form and the second form are forms, but they are not the same form.
  • the glasses 8 of this embodiment include: a processor 80 and a memory 81, and the memory 81 stores a computer program 82 that can run on the processor 80.
  • the processor 80 executes the computer program 82, the steps in the above embodiments of each operation prompt method are implemented, for example, steps 101 to 105 shown in FIG. 1.
  • the processor 80 executes the computer program 82, the functions of each module/unit in the foregoing device embodiments are realized, for example, the functions of the modules 71 to 75 shown in FIG. 7.
  • the so-called processor 70 can be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory 71 may be an internal storage unit of the glasses 7, such as a hard disk or a memory of the glasses 7.
  • the memory 71 may also be an external storage device of the glasses 7, for example, a plug-in hard disk equipped on the glasses 7, a smart memory card (Smart) Media (SMC), a secure digital (SD) card, Flash card (Flash Card), etc.
  • the memory 71 may include both an internal storage unit of the glasses 7 and an external storage device.
  • the memory 71 is used to store the computer program and other programs and data required by the glasses.
  • the memory 71 can also be used to temporarily store data that has been sent or is about to be sent.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or software function unit.
  • the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.
  • the present invention can realize all or part of the processes in the methods of the above embodiments, and can also be completed by a computer program instructing relevant hardware.
  • the computer program can be stored in a computer-readable storage medium. When the program is executed by the processor, the steps of the foregoing method embodiments may be implemented.
  • the computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form, etc.
  • the computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (Read-Only Memory, ROM) , Random Access Memory (Random Access Memory, RAM), electrical carrier signals, telecommunications signals and software distribution media, etc.
  • ROM Read-Only Memory
  • RAM Random Access Memory
  • electrical carrier signals telecommunications signals and software distribution media, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种操作提示方法、装置及眼镜，适用于数据处理技术领域，该方法包括：获取用户所处环境的图像，并基于图像构建用户所处环境的3D语义地图（S101）；对用户进行眼动识别，判断用户是否注视3D语义地图中包含的物品（S102）；若用户注视3D语义地图中包含的物品，获取3D语义地图对应的操作模式，操作模式中包含对一个或多个物品的操作步骤（S103）；监测用户对物品的操作是否满足操作步骤的要求（S104）；若用户对物品的操作不满足操作步骤的要求，输出操作步骤对应的操作提示。用户无需再进行任何手动操作输入，也无需一直主动对着屏幕观看，即可以及时得知自己操作中的问题，实现了对用户操作教程的智能匹配，极大地提升了用户对操作错误的确定效率。

Description

一种操作提示方法及眼镜 技术领域
本发明属于数据处理技术领域，尤其涉及操作提示方法及眼镜。
背景技术
实际生活和工作中，用户经常需要上网搜索一些操作教程来指导自己的一些操作是否正确，例如搜索一些做饭图文教程并根据做饭图文教程来看看自己烹饪方法是否有误，或者搜索一些设备操作教程来判断设备操作是否有误，现有技术中，用户都是通过电脑手机等设备手动输入一些关键词来进行搜索，得到对应的图文视频教程，再根据这些图文视频教程来逐步进行对比，但这些都需要用户进行大量的手动操作输入，且需要用户一直主动对着屏幕进行对比，确定操作的错误，操作繁琐且效率低下。
技术问题
有鉴于此，本发明实施例提供了一种操作提示方法及眼镜，以解决现有技术中用户需要大量手动操作查看和对比，才能确定自己操作中存在的错误，操作繁琐效率低下的问题。
技术解决方案
本发明实施例的第一方面提供了一种操作提示方法，包括：
获取用户所处环境的图像，并基于所述图像构建用户所处环境的3D语义地图；
对用户进行眼动识别，判断所述用户是否注视所述3D语义地图中包含的物品；
若所述用户注视所述3D语义地图中包含的物品，获取所述3D语义地图对应的操作模式，所述操作模式中包含对一个或多个所述物品的操作步骤；
监测所述用户对所述物品的操作是否满足所述操作步骤的要求；
若所述用户对所述物品的操作不满足所述操作步骤的要求，输出所述操作步骤对应的操作提示。
本发明实施例的第二方面提供了一种眼镜，所述眼镜包括存储器、处理器，所述存储器上存储有可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如上所述的操作提示方法的步骤。
有益效果
本发明实施例与现有技术相比存在的有益效果是：通过构建用户所处环境的3D语义地图，并在识别出用户存在物品注视行为，即确定用户存在操作提示需求时，根据3D语义地图实际情况智能识别出用户所需的操作模式(即对物品的操作教程)，并根据用户对物品的实际操作情况，来进行监测和提示用户操作，使得用户无需再进行任何手动操作输入，也无需一直主动对着屏幕观看，即可以及时得知自己操作中的问题，实现了对用户操 作教程的智能匹配，极大地提升了用户对操作错误的获知效率。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本发明实施例一提供的操作提示方法的实现流程示意图；
图2是本发明实施例二提供的操作提示方法的实现流程示意图；
图3A和图3B是本发明实施例三提供的操作提示方法的实现流程示意图；
图4A和图4B是本发明实施例四提供的操作提示方法的实现流程示意图；
图5是本发明实施例五提供的操作提示方法的实现流程示意图；
图6是本发明实施例六提供的操作提示方法的实现流程示意图；
图7是本发明实施例七提供的操作提示装置的结构示意图；
图8是本发明实施例八提供的眼镜的示意图。
本发明的实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本发明实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本发明。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本发明的描述。
为了说明本发明所述的技术方案，下面通过具体实施例来进行说明。
为了便于理解本发明，此处先对本发明实施例进行简要说明，为了便于用户即使发现自己的操作问题，本发明实施例首先会基于用户环境的图像来构建对应的3D语义地图，从而确定出用户实际所处环境的情况以及环境内包含的物体等，但考虑到实际情况中即使用户身处某一环境中也不一定就有操作需求，例如用户可能只是经过某一地点，或者用户虽然身处某一地点之中但并不需要进行操作，因此为了防止误触发，本发明实施例中还会对用户进行眼动分析，判断用户是否存在对物品的注视行为，并在用户存在注视行为时，判定用户存在操作提示的需求，此时在根据用户实际所处的3D语义地图情况来智能匹配用户所需的操作教程，最后对用户对物品的实时操作进行分析，并在用户操作不满足操作教程的要求时，对应进行正确的操作步骤提示，从而使得用户无需进行繁琐的手动操作，也可以及时获得所需的操作教程，并及时得知自己操作中存在的问题及正确的操作方式。
应当明确地，在本发明实施例中的具体执行主体可以根据实际应用情况需求的不同而 不同，例如，其既可以是可穿戴设备等智能设备(如智能眼镜)，也可以是服务器等设备，当执行主体为可穿戴设备时，本发明实施例中所有的数据采集、处理及输出操作均由可穿戴设备完成，而当执行主体为服务器等无法直接对用户进行数据采集输出的设备时，本发明实施例中对用户数据采集和输出的工作均由其他设备完成，即本发明实施例中数据获取和输出的直接对象不是用户，而是可以对用户进行数据采集输出的其他设备，以实现对用户的操作提示目的，例如，由智能眼镜负责对用户的数据采集并发送给服务器进行处理，由服务器将处理后生成的提示发送至智能眼镜，再由智能眼镜最终输出至用户，具体执行主体的确定，可由研发人员根据实际研发情况和需求来进行选定和设计，此处以执行主体为智能眼镜为例来进行说明，详述如下：
图1示出了本发明实施例一提供的操作提示方法的实现流程图，详述如下：
S101，获取用户所处环境的图像，并基于图像构建用户所处环境的3D语义地图。
为了实现对用户需求的智能准确识别，本发明实施例中首先会对用户所处环境进行图像采集，并进行3D语义地图的构建。其中，3D语义地图就是包含语义信息的环境三维地图。本发明实施例中，可以在智能眼镜中设置广角摄像头来进行环境图像的采集，在获取到环境图像之后，再基于这些图像进行环境三维地图的构建，并识别其中包含的物品以及每种物品相关属性数据。具体而言，3D语义地图构建的方法包括但不限于如基于立体视觉的3D语义地图算法或者其他构建算法，此处不予限定，具体可由技术人员根据实际需求选取或设定，或者参考本发明的其他相关实施例。
S102，对用户进行眼动识别，判断用户是否注视3D语义地图中包含的物品。
当用户身处某一环境内时，并不能说明其就存在操作教程指导的需求，例如用户可能只是经过某一地点，或者只是处于某一地点但并不需要进行操作等，因此，若直接根据用户所处环境来进行操作教程匹配及提示等操作，可能会导致对用户的误提示，但当用户身处某一环境，且持续性地注视一些物品时，说明用户极有可能是需要进行物品操作的，因此，本发明实施例会对用户是否存在对物品的注视行为进行识别，并会在确定出用户存在注视行为时，才进行后续的操作教程匹配等操作，具体而言，本发明实施例中智能眼镜内会内置对用户进行眼动拍摄的装置，例如眼动仪等装置，并由这些装置来对用户进行眼睛图像数据的采集并对采集的眼睛图像进行眼动分析，以确定出用户的眼动情况如何，如用户是否眨眼、注视区域在哪以及是否有注视物品等，在本发明实施例中，具体的眼动分析追踪方法可由技术人员自行选定，既可以使用一些现有的算法，也可根据需求自行设置，或者参考本发明其他相关实施例方法。
在本发明实施例中，需要确认用户是否存在注视物品的行为，即在进行眼动识别时不 仅需要识别出用户看的物品，还需要进一步地确认是否存在持续性的看某个物品(即注视物品)的行为，因此本发明实施例中，在确定出用户看的物品的基础上，还需要统计用户看物品的时间，并依此判断是否注视物品，具体可参考本发明实施例三。
S103，若用户注视3D语义地图中包含的物品，获取3D语义地图对应的操作模式，操作模式中包含对一个或多个物品的操作步骤。
当用户身处某一环境且持续性地注视一些物品时，说明用户极有可能是需要进行物品操作的，此时，本发明实施例会对用户环境的3D语义地图进行分析确定其实际所处环境情况，如是厨房还是设备操作室等，再根据环境情况来估计用户可能进行的操作以及对应所需的操作教程。本发明实施例中，操作教程既可以是技术人员预先设置多个，再根据3D语义地图场景类别等来进行选取，也可以由技术人员预设设置一定的操作教程生成规则，再根据3D语义地图场景类别等来操作教程的生成得到。
作为本发明的一个实施例，在进行操作模式确定时，包括：
识别3D语义地图对应的场景类别。
基于场景类别及3D语义地图中包含的物品，获取对应的操作模式。
在本发明实施例中，考虑到用户在不同实际场景下的不同需求，会同时根据用户3D语义地图对应的场景类别和实际包含的物品来判断用户的实际可能需求，例如用户处于厨房之中，且包含大量食物，则说明用户极有可能需要进行烹饪，此时可以生成对应的一些做饭的教程。其中，具体的场景类别识别方法以及物品识别方法此处不予限定，可由技术人员根据实际需求设定。
作为本发明的另一个实施例，在进行操作模式确定时，包括：
识别3D语义地图对应的场景类别，并获取用户的用户数据。
基于场景类别、3D语义地图中包含的物品及用户数据，获取对应的操作模式。
考虑到即使是在相同的场景，具有相同的物品的情况下，不同用户的实际需求还是可能存在一定区别，例如即使都是处于厨房之中且包含大量食物，但每个用户的口味可能不同，其对应的烹饪教程也必然存在较大差别，因此，仅根据用户所处环境的场景类别以及包含的物品情况有时难以实现对用户操作教程需求的准确识别。在本发明实施例中，会同时根据环境的场景类别、实际包含的物品以及用户的一些用户数据来确定出用户的实际操作需求，并筛选出对应的操作教程。其中用户数据的具体内容可以由技术人员或用户自行设定，包括但不限于如用户的个人信息和爱好数据等，或者用户自己预先设置的一些操作需求，例如用户预先设置自己的饮食需求。
S104，监测用户对物品的操作是否满足操作步骤的要求。
在获取到用户所需的操作教程之后，本发明实施例开始对用户的实际操作进行监测识别，并与操作教程之中的操作步骤进行比对，判断用户实际操作是否符合操作教程的要求，从而确定出用户存在的操作缺陷。其中，为了实现对用户操作的监测识别，本发明实施例中，智能眼镜中的摄像头会对用户的行为进行拍摄，并会对得到的图像/视频进行行为分析，即分析用户的行为动作和行为顺序等是否满足操作教程中的要求。
S105，若用户对物品的操作不满足操作步骤的要求，输出操作步骤对应的操作提示。
当用户出现不满足操作教程中操作步骤要求的操作时，说明用户出现了操作错误，因此此时本发明实施例会对用户进行操作提示，即告知用户当前操作存在错误，并会同时告知用户正确的操作步骤，例如，对设备进行操作时，在按下开机键之后，需要使用模式键选取设备模式后，再按开始键进行，而若用户在按下开机键之后，直接就按开始键，此时本发明实施例发现用户操作不满足操作教程的要求，就会提示用户，不应按开始键，而是需要按模式键选取设备模式。其中，操作提示的输出方式包括但不限于如音频/视频/文字等提示，具体可由技术人员自行设定。
作为本发明实施例一中进行提示输出的一种具体实现方式，包括：
识别用户在3D语义地图中的注视区域，并基于注视区域进行操作提示的增强现实输出。
本发明实施例中，会利用增强现实(Augmented Reality，AR)格式技术来进行提示信息的输出，以使得用户可更为直观地获知自己的问题以及对应的正确操作。
本发明实施例通过构建用户所处环境的3D语义地图，并在识别出用户存在物品注视行为，即确定用户存在操作提示需求时，根据3D语义地图实际情况智能识别出用户所需的操作模式(即对物品的操作教程)，并根据用户对物品的实际操作情况，来进行监测和提示用户操作，使得用户无需再进行任何手动操作输入，也无需一直主动对着屏幕观看，即可以及时得知自己操作中的问题，实现了对用户操作教程的智能匹配，极大地提升了用户对操作错误的获知效率。
作为本发明实施例一中3D语义地图构建的一种具体实现方式，在本发明实施例中，需要采集的环境图像包括环境的彩色图像和深度图像，如图2所示，本发明实施例二，包括：
S201，基于彩色图像和深度图像，得到用户的位置信息和姿态信息，以及用户所处环境内物品的位置信息和物品信息。
在本发明实施例中，智能眼镜以RGB-D相机作为传感器，获取彩色图像和深度图像，利用视觉SLAM算法完成智能眼镜的自主定位和位姿估计及优化(即用户的位置信息和 姿态信息获取)，同时进行物品检测获取环境的语义信息，然后采用RGB-D分割算法将感知到的物品分割出来，从而构建环境的3D语义地图。具体而言，视觉SLAM基本框架由视觉里程计、后端优化、回环检测和三维建图四个部分组成。视觉里程计要完成相邻两帧图像间运动的估计，粗略估计出相机当前的位姿。后端优化就是要对视觉里程计的估计进行全局一致的优化，消除噪声干扰，另外利用回环检测的约束优化位姿，使定位和位姿估计更加准确。回环检测是在重新回到原来经过的位置时，消除过程的累计误差。地图的构建基于前三部分估计的运动和位姿，创建环境的三维地图。
视觉SLAM算法整体设计，通过传感器输入数据实现对当前位置和姿态的全局一致估计，是对自身的定位和运动的明确。视觉SLAM算法由视觉里程计、后端优化和回环检测三个部分协作完成。
视觉里程计通过提取ORB特征点，利用FLANN算法匹配相邻两帧ORB特征，利用匹配结果采用PnP算法和RANSAC算法结合完成智能眼镜位置和姿态的粗略估计。视觉里程计的任务是估计出智能眼镜在两帧图像之间的位姿变化，估计出智能眼镜一段时间内的姿态和运动轨迹。其由以下进程构成：
1、特征提取
我们采用ORB特征提取的方式提取图像特征。ORB特征由FAST关键点和BRIEF描述子构成，并人为地赋予了旋转和尺度不变性。
特征提取的过程：
1)粗提取，对于某一点p，其像素值为Ip，以p为中心半径为3的圆上的16个点，如果存在12个以上的点的像素值与Ip相差在阈值内，则认为该p是一个候选FAST关键点。
2)计算每个子集的信息熵。以信息增益为评价标准，其值最高的像素点设定为决策树的根节点，并继续对其子集进行迭代，直到确定该点的性质，即FAST关键点或非FAST关键点。于是便生成了ID3决策树，利用该树筛选出最优的FAST关键点。
3)利用非极大值抑制的思想，在局部范围内，保留得分最高的FAST关键点，删掉其他得分较低的FAST关键点，遍历一遍，即可完成筛选。
4)，赋予特征尺度和旋转不变性。尺度不变性利用金字塔原理实现，将图像降采样处理得到图像金字塔，对其每一层都完成上述四步特征提取，实现FAST关键点的尺度不变性。旋转不变性通过灰度质心法实现，计算以关键点为中心图像块U的质心，把从中心到质心的向量定义为关键点的方向，实现FAST关键点的旋转不变性
描述子：在BRIEF基础上做出改进。一是考虑关键点的31×31邻域内所有点， 将图像进行高斯平滑滤波后，选取5×5邻域内灰度均值代替单个点灰度进行计算，抗噪性强。二是选取5×5邻域时采用均值接近0.5的不相关贪婪搜索算法，保证了描述子具有代表性和独特性，使其具备区分性。
2、特征匹配
选用快速近似最近邻(FLANN)算法，核心思想是利用导出索引结构将搜索范围定位在特征邻近区域，在邻近区域内完成特征匹配，可以有效地加快匹配速度。利用BRIEF描述子由0和1组成的特点，使用局部敏感哈希作为导出索引结构。将特征按照同样的方式投影到哈希空间，本来相邻的两个特征投影后仍然在哈希空间相邻的几率也会非常高，而之前不相邻的两个特征投影之后在哈希空间相邻的几率会很小，这样即可在哈希空间邻域内进行匹配，有效的缩小了范围。
3、位姿估计算法设计
在提取并匹配两个相邻帧的特征之后，使用匹配关系来估计智能眼镜的运动和姿态。为了定量的估计智能眼镜运动及位姿，首先必须清楚智能眼镜成像与空间点的数学几何关系。智能眼镜成像的过程也被称为观测过程，即三维空间中的点反射或发射光线，经过智能眼镜的光心，投影到智能眼镜的成像平面上的过程。
本发明采用PnP算法进行两帧间位姿的粗略估计，再利用RANSAC算法进行该位姿的帧间一致性优化，避免误匹配特征对位姿估计造成严重影响。采用RANSAC算法对该问题进行优化。
首先提取图像的ORB特征点，利用快速最近邻算法进行相邻两帧图像的特征匹配，最后使用PnP算法和RANSAC算法完成相机运动和姿态估计。
4、后端优化
由于数据噪声、误匹配、计算误差等因素影响，会造成位姿估计的误差，长时间运行时误差逐渐累积，会严重影响系统性能。后端优化就是要对视觉里程计的估计进行全局一致的优化，消除噪声干扰，使位姿估计更加准确。另外，当系统检测到回环后，将信息传递给后端，消除累计误差。
本发明实施例引入关键帧机制，选择有代表性的图像帧进行位姿优化，减少不必要的计算。对于局部位姿优化问题，采用捆集调整算法，优化相机目前处于收敛过程的姿态和特征点。当系统检测出回环时，后端采用位姿图优化方式，获取全局一致的轨迹和位姿。后端优化接收视觉里程计传来的位姿和特征点，采用捆集调整的方式进行优化。工作流程为：检查队列、处理关键帧，地图点剔除、生成和融合，局部捆集调整优化位姿。
满足以下四条原则即将图像设定为关键帧：
(1)由于相机获取数据帧频较高，当前关键帧和上一帧关键帧之间必须经过一定的序列间隔。
(2)后端优化部分不在工作状态下。
(3)当前关键帧与之前选择地所有关键帧相互的共视区域低于一定范围。
(4)当前帧拥有足够的特征点和匹配，保证特征的丰富性。
5、回环检测
回环检测，主要解决智能眼镜位姿估计误差逐渐积累的问题。当相于再次来到曾经到过的地方时，该部分确认并建立当前位姿和历史位姿间的连接关系，传到后端进行优化处理，将系统长时间运行的累积误差消除，得到全局一致的轨迹和位姿。另一方面，回环检测提供了当前数据与所有历史数据的关联，在视觉里程计对特征的跟踪丢失时，可以利用回环检测进行重定位，增强位姿估计的鲁棒性。
回环检测在系统运行时持续检测，通过发生回环的约束来消除智能眼镜位姿估计的累计误差。将相机回到某个曾经来过的位置时的闭环约束关系传到后端进行位姿图优化。回环检测的工作流程包括：步骤1、回环候选帧检测。步骤2、与前关键帧建立联系。步骤3、检测是否发生回环，若未发生，返回值步骤1，若发生进入步骤4。步骤4、位姿图优化。
5.1、磁带模型的建立
把特征看作一个个单词，预先训练包含所有特征类型的字典，对每个图像的特征按照该字典生成一个对应词的集合，也就是词袋。于是，判断图像的相似程度时只要对比它们的词袋即可，大大加快了回环检测的速度。特征聚类使用无监督机器学习K-means++算法，使用K-d树的结构提高搜索效率。
字典的训练过程为：
1)在根节点，用上述K-mean++算法将所有样本分成k类，得到第一层。
2)对每一层的每个节点，同样用K-mean++将属于该节点的样本再聚类为k类，得到新的一层。
3)以此类推，直到最后的叶子节点层。该叶子层就是特征对应的单词。
5.2相似度计算方法
引入TF-IDF算法。如果当前帧与之前的某个关键帧的相似度超过当前帧与上一关键帧的相似度的3倍，就认为可能发生了回环。但仍需要一个验证步骤，设立回环检测的缓存机制，单次相似度高不足以判断为回环，当连续帧的相似度都很高时，才确认发生回环。确认发生回环后，回环检测部分将此信息发送给后端，后端采用图优化方式对位姿进行优化，消除累计误差，得到全局一致的轨迹和位姿。
视觉SLAM算法整体流程的总结：
视觉SLAM算法设计由三部分组成：视觉里程计、后端优化、回环检测。视觉里程计部分首先提取传感器输入帧的ORB特征，之后使用快速最近邻算法进行相邻两帧图像的特征匹配，采用PnP算法和RANSAC算法结合完成相机位姿估计。后端部分对视觉里程计传来的位姿和特征点采用捆集调整的方式进行优化。回环检测利用词袋模型和训练好的字典检测相机是否回到之前到过的位置，如果发生回环，将此约束信息发送到后端，后端采用图优化方式对位姿进行优化，消除累计误差，保证了相机轨迹和位姿的全局一致性。
为了实现对物品的物品信息和位置信息获取，本发明实施例中，首先会从彩色图像中进行物品检测，其中检测算法包括但不限于如Yolv V3物体检测算法，此处不予限定，并从彩色图像中进行物品分割，详述如下：
在获取彩色图像的同时也获取了相应像素的深度图，也就是三维点云。因此，建立语义地图前需要将检测到的物体分割出来，再利用视觉SLAM模块估计并优化出的相机位姿将像素点投影到空间中的位置，从而构建三维的语义地图。我们通过一种改进GrabCut算法实现目标RGB-D分割，该算法结合CPF(Constrained Plane Fitting)算法分割的几何平面信息，提升了GrabCut的分割效果，实现三维目标的RGB-D分割。首先采用GrabCut算法分割图像，再利用CPF算法分割三维点云，最后将点云分割结果作为滤波器，剔除图像分割结果中不符合目标空间几何关系的像素点，完成目标的RGB-D分割。
S202，根据用户的位置信息及姿态信息以及物品的物品信息和位置信息，构建3D语义地图。
至此已经获取构建语义地图的全部所需信息，包括优化过的关键帧的智能眼镜位置及姿态(即用户的位置信息和姿态信息)、关键帧中物品的类别(即物品信息)及其位置和检测到的目标物体的三维分割。接下来要做的就是整合获取的信息构建全局一致的三维语义地图，大致过程分为三步：首先进行数据的一致性融合并更新目标物体模型，然后构建环境的三维地图，最后将语义信息融合到三维地图中，得到包含丰富信息的三维语义地图。
语义地图的构建算法研究与设计。首先设计了一种改进GrabCut的目标RGB-D分割算法，将GrabCut分割彩色图与CPF分割深度图的信息相结合，将检测到的目标分割出来，完成以对象为单位的语义建图的目标提取工作。并按照物体类别做好标记，再利用目标对象的一致性融合更新物体对象的模型，避免对同一目标的多次建模。进而以彩色八叉树地图结构来构建并存储包含丰富信息的三维语义地图。
1、数据关联与模型更新算法设计
数据关联的作用是得到目标经RGB-D分割后的结果时确定该目标是否已经在地图中， 需要添加新的对象还是对现有对象进行维护，避免对同一对象多次建模从而在地图中出现重影。
首先对于每一次检测，基于分割后点云的各质心的欧氏距离选择一组候选对象的边界标志。然后对地图现有边界标志和当前目标的三维点进行最近邻搜索，并计算相关点对的欧氏距离。两个三维点的欧氏距离也就是两点的2-范数。
如果目标中有一半以上的三维点都与地图现有的目标距离小于一定阈值时，认为该目标与地图现有目标是同一个，将当前目标信息与地图的现有目标相关联，以共同维护目标模型。另外，当所处环境较复杂时，三维点的最近邻搜索通过寻求k-d树结构来加速。为了保证数据关联能获取到最新的信息，只要检测到物体就利用上文设计改进的RGB-D目标分割算法将其分割出来。这样地图中的每个对象都保留三项信息：通过数据关联得到的目标模型、观测到该目标的关键帧位姿和目标检测模块给出的各类别概率。地图中目标物体的概率可以根据目标检测模块给出的概率值更新，设当前共检测C类物体，Sc表示该目标的各类别概率组成的向量，n为检测到此目标的关键帧个数，目标检测概率更新计算：
Figure PCTCN2019124352-appb-000001
则地图中该目标的所属类别为max(Sc)，其置信度为p＝max(Sc)ln，为语义地图中目标类别及概率的标注提供信息。
2、语义地图的构建与存储形式
本发明采用灵活、占内存量小又支持实时更新的地图形式：八叉树地图
整个空间作为根节点，按照空间坐标系分为八个子节点，再继续将每个子节点分为八个子节点，一直分到所需的分辨率，即叶子节点。八叉树地图与点云图的体素模型不同点在于，当某方块中所有点都被占据或不占据时，就没有必要展开这个节点，相对来说占用内存空间非常小。而且搜索叶子节点的速度非常快，d层的八叉树时间复杂度为O(d)。
另外，八叉树地图能够支持设置每个节点的颜色，也就是彩色八叉树地图形式，同时支持随时更新和修正信息，非常适用于构建三维语义地图。因此，本文选择彩色八叉树地图作为三维语义地图的构建与存储形式。
构建三维语义地图，首先建立环境的三维地图并持续更新，再实时地将语义信息融合到三维地图中，即可以构建包含丰富信息的三维的语义地图。在相机运动过程中不断地获取信息并处理，进而不断地更新语义地图。
三维建图的工作是根据视觉SLAM算法中对关键帧估计并优化后的位置和姿态，将RGB-D相机捕捉到的深度信息映射到三维空间中，建立环境的三维地图。由于本发明采用 的RGB-D相机可以获取视野中每个像素的深度，可以直接使用深度图进行稠密建图，根据优化的相机位姿，将深度图映射为点云后进行拼接，得到三维地图。
语义地图就是包含语义信息的三维地图，也就是在建立八叉树地图时标注出环境的语义信息。在建立三维八叉树地图时，同时将语义信息融入八叉树结构中，就得到包含丰富信息的三维语义地图。
本发明语义建图系统以RGB-D相机作为视觉传感器，捕捉彩色信息和深度信息。利用视觉SLAM算法完成AR智能眼镜的自主定位和位姿估计及优化，得到全局一致的轨迹和位姿。同时利用卷积神经网络YOLOv3模型进行目标检测，检测关键帧中出现的物体类别、概率及其位置，获取环境的语义信息。然后采用RGB-D分割算法将感知到的物体分割出来，选择八叉树地图形式构建环境的三维地图。最后，将语义信息融入该八叉树地图，完成环境的三维语义地图的构建。
本发明实施例设计了以RGB-D相机为传感器的视觉SLAM算法。本发明选择ORB特征作为算法的基础，不仅在视觉里程计的位姿估计中提供空间几何关系，还作为回环检测时图像相似度判断的标准，一定程度上实现了系统的统一性。本发明实施例设计的位姿估计算法中，采用PnP算法替代ICP算法，PnP算法利用前一帧优化好的相机坐标和当前帧像素坐标计算位姿，避免相机测量误差的干扰。后端优化部分采用针对性的算法处理，对视觉里程计传来的位姿和地图点的优化采用局部捆集调整算法，对回环检测传来的闭环约束优化时采用图优化的方式优化位姿。
设计了语义建图系统的整体结构。本文设计了一种改进GrabCut的目标RGB-D分割算法，结合利用深度点云信息的CPF分割结果修正仅利用彩色图像信息的GrabCut分割，实现二者性能的互补。将目标检测的对象分割出来，并按照物体类别做好标记，综合视觉SLAM算法获取的相机位姿和目标的检测和分割结果，以彩色八叉树地图结构构建并存储环境的三维语义地图。在实验室环境下运行系统，系统在自身定位、姿态估计、语义感知的同时，构建出可读且准确的三维语义地图，验证了本发明语义建图方案的可行性。
作为本发明实施例一中对用户进行注视行为识别的一种具体实现方式，如图3A所示，本发明实施例三，包括：
S301，获取用户的眼部图像，对眼部图像进行瞳孔定位，并基于得到的瞳孔位置信息确定用户在3D语义地图中的注视区域。
本发明实施例中，会基于用户的瞳孔和普尔钦斑来进行用户视线的追踪，并确定出用户注视的区域，因此首先需要确定出眼睛中瞳孔的位置。其中，具体的瞳孔识别算法可由技术人员根据需求自行设定，包括但不限于如使用神经网络模型来进行瞳孔图像的样本数 据训练，并识别眼部图像中的瞳孔，或者参考本发明实施例四来进行识别。由于基于瞳孔和普尔钦斑来进行用户视线的追踪的技术已较为成熟，因此此处不予赘述。
S302，识别注视区域内包含的物品，并统计注视区域内包含的物品在注视区域内连续存在时长。
S303，若连续存在时长中存在大于预设时长阈值的时长，判定用户注视3D语义地图中包含的物品。
在确定出用户的注视区域之后，还需要进一步地确定出注视区域中包含的物品，以及每个物品在注视区域存在的连续时长(即用户对物品的连续注视时长)，若该时长较长，则说明用户存在注视某一物品的行为。其中，预设时长阈值的具体值大小，可由技术人员自行设定。
作为本发明的一个实施例，考虑到用户头部会运动，因此在进行上述视线追踪时，视线会发生变化，为了实现更为准确的视线追踪，本发明实施例中包括：
头部的自然运动包括两项基本运动:垂直方向的俯仰运动和水平方向的左右运动。一般的基于二次多项式的映射模型是用户在保持头部静止不动的情况下通过多点校准得到，在头部位置发生变化时，该模型得到的估计注视点误差大大增加。本文在此基础上提出一种基于多项式映射的头动解决算法，该算法需要利用头部运动估计设备实时得到头部运动的信息，在头动的情况下首先利用校准时得到的多项式映射模型进行注视点位置初步估计，将该估计点坐标结合初始的眼睛位置建立三维视线方向，再利用头部运动信息对该视线方向进行旋转和平移补偿，将当前补偿后的视线方向与屏幕的交点作为最终的注视点估计坐标。另外，考虑到其实头部运动的影响是源于头部运动使得眼睛的位置发生了变化，所以我们只要知道的眼睛位置的变化情况就可以对视线进行补偿。
根据以上几点说明建立头部自然运动下的视线估计模型，如图3B所示，图中表示一个视线估计原理的侧平面图，O1表示初始的眼睛位置，O2表示运动后眼睛的位置。在初始位置当注视屏幕上点S1时，对应得到瞳孔角膜矢量pccr1，头部向右绕Y轴旋转α角度后，眼睛移到位置O2，假设pccr2等于pccr1，即眼图特征不发生任何变化，则对应的视线将向右发生偏转，与屏幕的交点我们假设为S2，这是此时的注视点位置，但由于瞳孔角膜矢量未发生变化，用pccr2进行视线估计时误差较大。对此我们将该点作为视线估计的初步估计点，将该点和眼睛的初始位置建立视线方向g1，再利用头部运动的角度信息对g1进行修正，将修正后的视线方向与屏幕的交点即为当前的注视点估计点。
作为本发明实施例三中对瞳孔进行定位的一种实现方式，如图4A所示，本发明实施例四，包括：
S401，将眼部图像划分为N×M个区域图像，并对所有区域图像灰度二值化处理，得到对应的N×M个眼部灰度值，其中，N和M均为正整数。
如图4B所示，本发明实施例中的眼部特征详述如下，图中基本的矩形块大小一致，图中ABCD是最原始的矩形特征，E是由3个基本矩形组成，F由9个矩形组成，G是一个矩形，H与I由4个基本矩形构成，J由12个矩形构成，K和L由4个矩形构成，各矩形特征的计算都是图中的黑色部分像素和减去白色部分像素和，这里的特征G是一个单一的矩形特征，所以只计算矩形内的像素和。眼部特征是基于眼睛结构而设计的，由于眼角与周围的亮度有所变化，眼角相对于周围的像素较暗，特征F可以很好的表现这个特点，眼球的中心点基本上是呈现黑色，所以简单的矩形G就表示了眼球的这一特性，眼睛的水平方向会发生两次较明显的像素突变，从巩膜到虹膜，再由虹膜到巩膜，特征H就可以反应出这种变化，同理眼睛的垂直方向上也有类似的灰度变化特点，I特征就是基于这一特点而产生的，它们与C和E合起来能够增强对眼睛水平和垂直灰度变化特点的描述，J表示了眼角与眼球间部分的灰度变化情况，K、L表现了眼角的边缘信息。增加新特征后使得用于分类的眼睛特征数量减少，使眼睛检测变得更加容易。
本发明实施例的瞳孔定位采用了帧独立的设计思想。仔细观察人眼截图发现，人眼图像信号的很多部分会随着入射光线、不同人、眼睛的镜面反射而改变，而且，随着人脸转角的不同，出现在截图中的相对位置也不同，直接进行分类训练效果不尽如人意。经过多次对比，我们发现瞳孔区域是人眼截图中比较稳定的图像信息，在睁眼时候这部分的特征明显。因此我们先定位瞳孔进而简化问题的复杂性。
本发明实施例中，N和M均为正整数，其具体值大小由技术人员自行设定，以M＝N＝10为例进行说明，此时会将眼部图像划分为10×10个区域图像，并会计算每个区域图像的灰度值，得到10×10个眼部灰度值。
S402，获取用户的皮肤图像，并计算皮肤图像灰度二值化处理后的平均灰度值。
为了实现肤色灰度与最低灰度之间的权衡，本发明实施例还会采集用户的皮肤图像，并计算其对应的平均灰度值。其中皮肤图像优选地，可以为眼部周围的皮肤图像。
S403，按照差值绝对值从小到大的顺序，从对应的眼部灰度值与平均灰度值的差值绝对值小于预设灰度阈值的区域图像中进行图像筛选，直至筛选得到的区域图像包含的像素点数处于预设数量阈值范围内，得到瞳孔对应的区域图像，以确定瞳孔在眼部图像中的位置。
在得到每个区域图像分别对应的眼部灰度值以及计算得到的皮肤的平均灰度值之后，本发明实施例会分别计算每个区域图像的眼部灰度值，与皮肤的平均灰度值的差值绝对值， 并会按照从小到大的顺序依次筛选出符合预设灰度阈值要求的区域图像，并在每次筛选出一个区域图像之后，统计一次所有筛选出的区域图像包含的像素点数，直至筛选出的区域图像包含的像素点数处于预设数量范围内位置，以保证对瞳孔识别的准确可靠。其中，预设灰度阈值和预设数量阈值的具体值大小，可有技术人员自行设定。
作为本发明实施例一中对3D语义地图进行场景识别的一种具体实现方法，本发明实施例包括：
为了实现对每张图片进行单独分类，本文使用了Places205-AlexNet网络模型，该模型在各种基准数据集上的性能表现均超过其他方法。Places205-AlexNet网络模型遵循与AlexNet相同的网络架构但其在场景分类任务上进行了针对性的训练。该训练数据集包含了205个语义类的约250万张图片，且每一类至少有5000张样本图片。这些图片来源于各种网络资源，如谷歌图片、必应(Bing)、Flickr等，并将图片进行分类标注。训练数据集的样本数量和种类确保了由此产生的分类器可很好地泛化，且在其从未训练过的环境中应用时不必进行二次训练或微调。这确保了本文的语义建图系统是可移植的、能被在多种环境下作业的用户使用。
Places205-AlexNet网络模型的输入是被调整为224×224×3像素的RGB图片，而与它们的初始尺寸无关。Places205-AlexNet卷积神经网络模型共有8层网络，其中包括前面5个卷积层和后面3个全连接层。给定当前场景的图片It，该网络的输出层soft-max输出在205个已知场景类上的离散概率分p(o t|I t)。本文的分类器使用Places205-AlexNet网络模型的fc7层的输出作为一个特征向量，fc7层是网络中最后一个通用的(即与类无关的)全连接层。由于Places205-AlexNet网络模型设计的目的是为了对Places205场景数据集中205个场景类别进行识别，因此最后的fc8层和prob层具有205个输出神经元节点。
假设给定当前图片It，该网络的输出层prob输出在205个已知类Ci上的离散概率分布p(o t|I t)。令
Figure PCTCN2019124352-appb-000002
表示已知场景类标签的组合向量。然后定义相应的组合似然为：
Figure PCTCN2019124352-appb-000003
其中表示t时刻的图像It属于场景类Ci的概率，且Ci之间相互独立。由于相机获取的相邻两张图片在时间上是连续的，故可以应用递归的贝叶斯滤波技术。本文将机器人的场景分类问题描述为一个概率估计问题并估计在所有可能的场景标签Ci上的离散概率分布其中所有从过去到现在的观测图片是已知的。假设满足一阶马尔科夫性质，则可得到如下 的贝叶斯滤波公式：
Figure PCTCN2019124352-appb-000004
在将场景分类问题视为一个贝叶斯估计问题后，可以很自然地整合其他信息资源。例如，由于室内服务机器人工作在室内，因此205类场景中的室外场景基本上不可能被观测到。
作为本发明实施例五，考虑到实际情况中，满足场景类型、物品情况以及用户数据要求的操作教程有时可能有多个，此时，为了确保最终确定的操作教程是用户实际所需，如图5所示，本发明实施例包括：
S501，获取多种操作模式。
S502，获取用户对的模式选取指令，并基于模式选取指令，从多种操作模式筛选出一种操作模式。
当有多种满足要求的操作教程时，本发明实施例可由用户自行选取其所需的一个进行后续操作。其中，模式选取指令的输入方式，包括但不限于如用户通过语音/眼动/头部姿态变换等方式来进行输入，例如在智能眼镜中显示出多种操作教程的相关信息，用户通过语音选取、眼动操作(如眨眼、注视等)或者头部姿态变化(如摇头切换操作教程，点头确认，以实现模式选取指令输入)，具体需由技术人员根据实际需求设定可进行指令交互的方式。
作为本发明实施例中用户与智能眼镜的一直眼动信息交互实现方式，本发明实施例中用户可以进行多种眼动操作，以与智能眼镜进行信息交互操作，包括：
1)判断是否是阅读行为：
用来标注阅读行为(已确认注视目标，不需要更多的识别时间)的主要注视点，其时间长度预设范围内，预设范围优选为600-1100毫秒。
2)判断用户回视行为：
借助扫视数据可以定义回视，即注视点坐标落在前5个注视点为圆心，1个注视区域为半径构成的空间中，但该注视点的前1个注视点不算。
3)判断是否是改变全局关注点的行为：
整体关注点改变是在上一个主要注视点(持续时间预设时长以上，优选为1100毫秒以上)后扫视长度超过3个注视区域及以上。
4)判断局部关注点改变的行为：
局部关注点改变是在上一个主要注视点(持续时间预设时长以上，优选为1100毫秒以 上)后多个扫视长度不超过3个注视区域，但其位置改变总距离超过了3个注视区域的行为。
5)对搜索行为的判断：
搜索行为：当出现整体关注点改变或局部关注点改变行为时，可以认为用户正在进行搜索。整体关注点改变是典型的搜索行为，而局部关注点改变则是用户认为大致区域已经找到，正在继续寻找特定目标。
持久搜索：在上一次主要注视点(持续时间预设时长以上，优选为1100毫秒以上)后10个注视区域及以上的扫视连续出现时的行为，可以认为用户在思考、走神或休息。
本发明实施例中，用户可以通过上述方法来对智能眼镜进行眼动控制，以实现本发明其他实施例中所需的各类人机交互操作。
作为本发明实施例一中进行用户操作监测的一种具体实现方法，如图6所示，本发明实施例六，包括：
S601，识别用户对物品操作时的行为，并判断行为是否满足操作步骤的要求。
S602，获取操作模式涉及的物品对应的预设属性阈值，并识别用户对物品进行操作过程中，物品的物品属性数据是否满足预设属性阈值。
S603，若行为不满足操作步骤的要求，和/或物品属性数据不满足预设属性阈值，判定用户对物品的操作不满足操作步骤的要求。
考虑到即使用户操作行为上看似无误，但其中具体每一步操作是否完全准确有时也是难以识别的，例如用户的确是先开机-再选模式-再开始，但在模式选取时选错了模式，此时仍会导致整个操作出现问题，因此，为了及时确定出用户的操作问题，本发明实施例对用户进行操作监测时，不仅仅只是监测用户的操作行为，同时还会对用户操作物品的属性据进行监测，以防止此类错误发生了却又无法识别的情况出现。
具体而言，本发明实施例中会对操作教程内涉及的每个物品均设置对应的属性阈值，例如某个做饭教程中，设定烤箱的烤制时间为1小时温度为120度，并对用户操作过程中物品的属性数据进行识别，判断其操作导致的物品属性数据变化是否处于阈值要求范围，如上述的，判断用户设定的烤箱的烤制时间和温度是否为1小时和120度(可根据智能眼镜拍摄到的用户设置烤箱时的图像来进行识别得到)。
当用户行为出现错误(如按错键)，或者用户操作导致的物品的属性数据不满足要求(如烤箱温度设置错误)时都说明用户的操作是错误的，因此本发明实施例都会判定用户对物品的操作不满足操作教程中草在步骤的要求。
对应于上文实施例的方法，图7示出了本发明实施例提供的操作提示装置的结构框图， 为了便于说明，仅示出了与本发明实施例相关的部分。图7示例的操作提示装置可以是前述实施例一提供的操作提示方法的执行主体。
参照图7，该操作提示装置包括：
地图构建模块71，用于获取用户所处环境的图像，并基于所述图像构建用户所处环境的3D语义地图。
注视识别模块72，用于对用户进行眼动识别，判断所述用户是否注视所述3D语义地图中包含的物品。
模式获取模块73，用于若所述用户注视所述3D语义地图中包含的物品，获取所述3D语义地图对应的操作模式，所述操作模式中包含对一个或多个所述物品的操作步骤。
操作监测模块74，用于监测所述用户对所述物品的操作是否满足所述操作步骤的要求。
操作提示模块75，用于若所述用户对所述物品的操作不满足所述操作步骤的要求，输出所述操作步骤对应的操作提示。
进一步地，地图构建模块71，包括：
基于所述彩色图像和所述深度图像，得到所述用户的位置信息和姿态信息，以及所述用户所处环境内所述物品的位置信息和物品信息。
根据所述用户的位置信息及姿态信息以及所述物品的物品信息和位置信息，构建所述3D语义地图。
进一步地，注视识别模块72，包括：
瞳孔定位模块，用于获取所述用户的眼部图像，对所述眼部图像进行瞳孔定位，并基于得到的瞳孔位置信息确定所述用户在所述3D语义地图中的注视区域。
时长统计模块，用于识别所述注视区域内包含的物品，并统计所述注视区域内包含的物品在所述注视区域内连续存在时长。
注视确认模块，用于若所述连续存在时长中存在大于预设时长阈值的时长，判定所述用户注视所述3D语义地图中包含的物品。
进一步地，瞳孔定位模块，包括：
将所述眼部图像划分为N×M个区域图像，并对所有所述区域图像灰度二值化处理，得到对应的N×M个眼部灰度值，其中，N和M均为正整数。
获取所述用户的皮肤图像，并计算所述皮肤图像灰度二值化处理后的平均灰度值。
按照差值绝对值从小到大的顺序，从对应的所述眼部灰度值与所述平均灰度值的差值绝对值小于预设灰度阈值的所述区域图像中进行图像筛选，直至筛选得到的所述区域图像 包含的像素点数处于预设数量阈值范围内，得到瞳孔对应的所述区域图像，以确定所述瞳孔在所述眼部图像中的位置。
进一步地，模式获取模块73，包括：
识别所述3D语义地图对应的场景类别。
基于所述场景类别及所述3D语义地图中包含的物品，获取对应的所述操作模式。
进一步地，模式获取模块73，还包括：
场景识别模块，用于识别所述3D语义地图对应的场景类别，并获取所述用户的用户数据。
模式确定模块，用于基于所述场景类别、所述3D语义地图中包含的物品及所述用户数据，获取对应的所述操作模式。
进一步地，模式确定模块，包括：
获取多种所述操作模式。
获取所述用户对的模式选取指令，并基于所述模式选取指令，从所述多种所述操作模式筛选出一种所述操作模式。
进一步地，操作监测模块74，包括：
识别所述用户对所述物品操作时的行为，并判断所述行为是否满足所述操作步骤的要求。
获取所述操作模式涉及的所述物品对应的预设属性阈值，并识别所述用户对所述物品进行操作过程中，所述物品的物品属性数据是否满足所述预设属性阈值。
若所述行为不满足所述操作步骤的要求，和/或所述物品属性数据不满足所述预设属性阈值，判定所述用户对所述物品的操作不满足所述操作步骤的要求。
进一步地，操作提示模块75，包括：
识别所述用户在所述3D语义地图中的注视区域，并基于所述注视区域进行所述操作提示的增强现实输出。
本发明实施例提供的操作提示装置中各模块实现各自功能的过程，具体可参考前述图1所示实施例一的描述，此处不再赘述。
应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本发明实施例的实施过程构成任何限定。
还应理解的是，虽然术语“第一”、“第二”等在文本中在一些本发明实施例中用来描述各种元素，但是这些元素不应该受到这些术语的限制。这些术语只是用来将一个元素与另一元素区分开。例如，第一表格可以被命名为第二表格，并且类似地，第二表格可以 被命名为第一表格，而不背离各种所描述的实施例的范围。第一表格和第二表格都是表格，但是它们不是同一表格。
图8是本发明一实施例提供的眼镜的示意图。如图8所示，该实施例的眼镜8包括：处理器80、存储器81，所述存储器81中存储有可在所述处理器80上运行的计算机程序82。所述处理器80执行所述计算机程序82时实现上述各个操作提示方法实施例中的步骤，例如图1所示的步骤101至105。或者，所述处理器80执行所述计算机程序82时实现上述各装置实施例中各模块/单元的功能，例如图7所示模块71至75的功能。
所称处理器70可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
所述存储器71可以是所述眼镜7的内部存储单元，例如眼镜7的硬盘或内存。所述存储器71也可以是所述眼镜7的外部存储设备，例如所述眼镜7上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。进一步地，所述存储器71还可以既包括所述眼镜7的内部存储单元也包括外部存储设备。所述存储器71用于存储所述计算机程序以及所述眼镜所需的其他程序和数据。所述存储器71还可以用于暂时地存储已经发送或者将要发送的数据。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的模块/单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明实现上述实施例方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、电载波信号、电信信号以及软件分发介质等。以上所述实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本 发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使对应技术方案的本质脱离本发明各实施例技术方案的精神和范围，均应包含在本发明的保护范围之内。

Claims (10)

  1. 一种操作提示方法，其特征在于，包括：
    获取用户所处环境的图像，并基于所述图像构建用户所处环境的3D语义地图；
    对用户进行眼动识别，判断所述用户是否注视所述3D语义地图中包含的物品；
    若所述用户注视所述3D语义地图中包含的物品，获取所述3D语义地图对应的操作模式，所述操作模式中包含对一个或多个所述物品的操作步骤；
    监测所述用户对所述物品的操作是否满足所述操作步骤的要求；
    若所述用户对所述物品的操作不满足所述操作步骤的要求，输出所述操作步骤对应的操作提示。
  2. 如权利要求1所述的操作提示方法，其特征在于，所述图像包括彩色图像和深度图像，所述获取用户所处环境的图像，并基于所述图像构建构建用户所处环境的3D语义地图，包括：
    基于所述彩色图像和所述深度图像，得到所述用户的位置信息和姿态信息，以及所述用户所处环境内所述物品的位置信息和物品信息；
    根据所述用户的位置信息及姿态信息以及所述物品的物品信息和位置信息，构建所述3D语义地图。
  3. 如权利要求1所述的操作提示方法，其特征在于，所述对用户进行眼动识别，判断所述用户是否注视所述3D语义地图中包含的物品，包括：
    获取所述用户的眼部图像，对所述眼部图像进行瞳孔定位，并基于得到的瞳孔位置信息确定所述用户在所述3D语义地图中的注视区域；
    识别所述注视区域内包含的物品，并统计所述注视区域内包含的物品在所述注视区域内连续存在时长；
    若所述连续存在时长中存在大于预设时长阈值的时长，判定所述用户注视所述3D语义地图中包含的物品。
  4. 如权利要求3所述的操作提示方法，其特征在于，所述对所述眼部图像进行瞳孔定位，包括：
    将所述眼部图像划分为N×M个区域图像，并对所有所述区域图像灰度二值化处理，得到对应的N×M个眼部灰度值，其中，N和M均为正整数；
    获取所述用户的皮肤图像，并计算所述皮肤图像灰度二值化处理后的平均灰度值；
    按照差值绝对值从小到大的顺序，从对应的所述眼部灰度值与所述平均灰度值的差值绝对值小于预设灰度阈值的所述区域图像中进行图像筛选，直至筛选得到的所述区域图像 包含的像素点数处于预设数量阈值范围内，得到瞳孔对应的所述区域图像，以确定所述瞳孔在所述眼部图像中的位置。
  5. 如权利要求1所述的操作提示方法，其特征在于，所述获取所述3D语义地图对应的操作模式，包括：
    识别所述3D语义地图对应的场景类别；
    基于所述场景类别及所述3D语义地图中包含的物品，获取对应的所述操作模式。
  6. 如权利要求1所述的操作提示方法，其特征在于，所述获取所述3D语义地图对应的操作模式，包括：
    识别所述3D语义地图对应的场景类别，并获取所述用户的用户数据；
    基于所述场景类别、所述3D语义地图中包含的物品及所述用户数据，获取对应的所述操作模式。
  7. 如权利要求6所述的操作提示方法，其特征在于，获取所述操作模式的过程包括，包括：
    获取多种所述操作模式；
    获取所述用户对的模式选取指令，并基于所述模式选取指令，从所述多种所述操作模式筛选出一种所述操作模式。
  8. 如权利要求1所述的操作提示方法，其特征在于，所述监测所述用户对所述物品的操作是否满足所述操作步骤的要求，包括：
    识别所述用户对所述物品操作时的行为，并判断所述行为是否满足所述操作步骤的要求；
    获取所述操作模式涉及的所述物品对应的预设属性阈值，并识别所述用户对所述物品进行操作过程中，所述物品的物品属性数据是否满足所述预设属性阈值；
    若所述行为不满足所述操作步骤的要求，和/或所述物品属性数据不满足所述预设属性阈值，判定所述用户对所述物品的操作不满足所述操作步骤的要求。
  9. 如权利要求1所述的操作提示方法，其特征在于，所述输出所述操作步骤对应的操作提示，包括：
    识别所述用户在所述3D语义地图中的注视区域，并基于所述注视区域进行所述操作提示的增强现实输出。
  10. 一种眼镜，其特征在于，所述眼镜包括存储器、处理器，所述存储器上存储有可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如权利要求1 至9任意一项所述方法的步骤。
PCT/CN2019/124352 2018-12-17 2019-12-10 一种操作提示方法及眼镜 Ceased WO2020125499A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811543901.X 2018-12-17
CN201811543901.XA CN109782902A (zh) 2018-12-17 2018-12-17 一种操作提示方法及眼镜

Publications (2)

Publication Number Publication Date
WO2020125499A1 true WO2020125499A1 (zh) 2020-06-25
WO2020125499A9 WO2020125499A9 (zh) 2020-12-17

Family

ID=66497395

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/124352 Ceased WO2020125499A1 (zh) 2018-12-17 2019-12-10 一种操作提示方法及眼镜

Country Status (2)

Country Link
CN (1) CN109782902A (zh)
WO (1) WO2020125499A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111753755A (zh) * 2020-06-28 2020-10-09 刘晨 一种智能眼镜
CN112263217A (zh) * 2020-08-27 2021-01-26 上海大学 一种基于改进卷积神经网络的非黑素瘤皮肤癌病理图像病变区域检测方法
CN112860826A (zh) * 2021-01-15 2021-05-28 北京小米移动软件有限公司 一种功能控制方法、功能控制装置及存储介质
SE2051359A1 (en) * 2020-11-20 2022-05-21 Wiretronic Ab Method and system for compliance determination
CN114565815A (zh) * 2022-02-25 2022-05-31 包头市迪迦科技有限公司 一种基于三维模型的视频智能融合方法及系统
CN115097903A (zh) * 2022-05-19 2022-09-23 深圳智华科技发展有限公司 Mr眼镜控制方法、装置、mr眼镜及存储介质
CN115439544A (zh) * 2022-09-06 2022-12-06 北京理工大学 一种多个机器人鲁棒协同定位方法

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109782902A (zh) * 2018-12-17 2019-05-21 中国科学院深圳先进技术研究院 一种操作提示方法及眼镜
JP6651189B1 (ja) 2019-03-29 2020-02-19 株式会社 情報システムエンジニアリング 機械学習用のデータ構造、学習方法及び情報提供システム
CN110689622B (zh) * 2019-07-05 2021-08-27 电子科技大学 一种基于点云分割匹配闭环校正的同步定位与构图方法
CN110908768B (zh) * 2019-10-31 2022-07-05 北京浪潮数据技术有限公司 一种虚拟化平台运维方法、装置、设备及可读存储介质
CN111507195B (zh) * 2020-03-20 2023-10-03 北京万里红科技有限公司 虹膜分割神经网络模型的训练方法、虹膜分割方法及装置
JP6846765B1 (ja) 2020-03-26 2021-03-24 株式会社 情報システムエンジニアリング 情報処理プログラム
CN111652155A (zh) * 2020-06-04 2020-09-11 北京航空航天大学 一种人体运动意图的识别方法及系统
CN114037738B (zh) * 2021-11-18 2025-01-17 湖北三闾智能科技有限公司 一种人类视觉驱动的上肢辅助机器人控制方法
CN114330726B (zh) * 2021-12-29 2026-01-02 广东电网有限责任公司 追踪定位方法、装置、电子设备及存储介质
WO2023189218A1 (ja) * 2022-04-01 2023-10-05 ソニーグループ株式会社 情報処理装置、情報処理方法、記録媒体
CN115145668A (zh) * 2022-06-10 2022-10-04 杭州网易云音乐科技有限公司 一种应用的引导方法、装置、电子设备及存储介质
CN119576126B (zh) * 2024-11-12 2025-10-03 中国运载火箭技术研究院 基于稠密地图语义分割的混合现实眼动交互系统及方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140266988A1 (en) * 2013-03-15 2014-09-18 Eyecam, LLC Autonomous computing and telecommunications head-up displays glasses
CN104965314A (zh) * 2015-08-03 2015-10-07 陈丽晓 一种智能眼镜
CN106557168A (zh) * 2016-11-23 2017-04-05 上海擎感智能科技有限公司 智能眼镜及其操控方法、控制装置
CN108351522A (zh) * 2015-08-07 2018-07-31 托比股份公司 注视方向映射
CN109782902A (zh) * 2018-12-17 2019-05-21 中国科学院深圳先进技术研究院 一种操作提示方法及眼镜

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8860760B2 (en) * 2010-09-25 2014-10-14 Teledyne Scientific & Imaging, Llc Augmented reality (AR) system and method for tracking parts and visually cueing a user to identify and locate parts in a scene
CN103838378B (zh) * 2014-03-13 2017-05-31 广东石油化工学院 一种基于瞳孔识别定位的头戴式眼睛操控系统
CN106202269A (zh) * 2016-06-28 2016-12-07 广东欧珀移动通信有限公司 一种获取增强现实操作指导的方法、装置及移动终端
CN106774876B (zh) * 2016-12-12 2020-07-28 快创科技（大连）有限公司 基于ar增强现实技术和菜谱生成的烹饪辅助系统
CN107066507B (zh) * 2017-01-10 2019-09-17 中国人民解放军国防科学技术大学 一种基于云机器人混合云架构的语义地图构建方法
CN107564012B (zh) * 2017-08-01 2020-02-28 中国科学院自动化研究所 面向未知环境的增强现实方法及装置
CN107656505A (zh) * 2017-08-21 2018-02-02 杭州太若科技有限公司 使用增强现实设备控制人机协作的方法、装置和系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140266988A1 (en) * 2013-03-15 2014-09-18 Eyecam, LLC Autonomous computing and telecommunications head-up displays glasses
CN104965314A (zh) * 2015-08-03 2015-10-07 陈丽晓 一种智能眼镜
CN108351522A (zh) * 2015-08-07 2018-07-31 托比股份公司 注视方向映射
CN106557168A (zh) * 2016-11-23 2017-04-05 上海擎感智能科技有限公司 智能眼镜及其操控方法、控制装置
CN109782902A (zh) * 2018-12-17 2019-05-21 中国科学院深圳先进技术研究院 一种操作提示方法及眼镜

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111753755B (zh) * 2020-06-28 2024-06-07 刘晨 一种智能眼镜
CN111753755A (zh) * 2020-06-28 2020-10-09 刘晨 一种智能眼镜
CN112263217B (zh) * 2020-08-27 2023-07-18 上海大学 一种基于改进卷积神经网络的非黑素瘤皮肤癌病理图像病变区域检测方法
CN112263217A (zh) * 2020-08-27 2021-01-26 上海大学 一种基于改进卷积神经网络的非黑素瘤皮肤癌病理图像病变区域检测方法
CN116670565A (zh) * 2020-11-20 2023-08-29 瓦尔卓尼克公司(瑞典) 用于确定符合性的方法和系统
WO2022108509A1 (en) * 2020-11-20 2022-05-27 Wiretronic Ab Method and system for compliance determination
SE2051359A1 (en) * 2020-11-20 2022-05-21 Wiretronic Ab Method and system for compliance determination
US12322082B2 (en) 2020-11-20 2025-06-03 Wiretronic Ab Method and system for compliance determination
SE547671C2 (en) * 2020-11-20 2025-11-04 Wiretronic Ab Method and system for compliance determination
CN112860826A (zh) * 2021-01-15 2021-05-28 北京小米移动软件有限公司 一种功能控制方法、功能控制装置及存储介质
CN114565815A (zh) * 2022-02-25 2022-05-31 包头市迪迦科技有限公司 一种基于三维模型的视频智能融合方法及系统
CN114565815B (zh) * 2022-02-25 2023-11-03 包头市迪迦科技有限公司 一种基于三维模型的视频智能融合方法及系统
CN115097903A (zh) * 2022-05-19 2022-09-23 深圳智华科技发展有限公司 Mr眼镜控制方法、装置、mr眼镜及存储介质
CN115097903B (zh) * 2022-05-19 2024-04-05 深圳智华科技发展有限公司 Mr眼镜控制方法、装置、mr眼镜及存储介质
CN115439544A (zh) * 2022-09-06 2022-12-06 北京理工大学 一种多个机器人鲁棒协同定位方法

Also Published As

Publication number Publication date
WO2020125499A9 (zh) 2020-12-17
CN109782902A (zh) 2019-05-21

Similar Documents

Publication Publication Date Title
WO2020125499A1 (zh) 一种操作提示方法及眼镜
US20210073953A1 (en) Method for applying bokeh effect to image and recording medium
CN111353506B (zh) 自适应的视线估计方法和设备
US11715231B2 (en) Head pose estimation from local eye region
JP5016175B2 (ja) 顔画像処理システム
US6757571B1 (en) System and process for bootstrap initialization of vision-based tracking systems
CN108427503A (zh) 人眼追踪方法及人眼追踪装置
CN119847330B (zh) 一种ar设备的智能控制方法以及系统
CN107004275A (zh) 用于确定实物的至少一部分的处于绝对空间比例的3d重构件的空间坐标的方法和系统
CN117474948A (zh) 基于视觉算法的人体运动轨迹提取方法及系统
WO2022267653A1 (zh) 图像处理方法、电子设备及计算机可读存储介质
CN106934351B (zh) 手势识别方法、装置及电子设备
CN110084192B (zh) 基于目标检测的快速动态手势识别系统及方法
KR102160128B1 (ko) 인공지능 기반 스마트 앨범 생성 방법 및 장치
CN105912126B (zh) 一种手势运动映射到界面的增益自适应调整方法
CN116820251B (zh) 一种手势轨迹交互方法、智能眼镜及存储介质
CN113192132A (zh) 眼神捕捉方法及装置、存储介质、终端
JP2019191981A (ja) 行動認識装置、モデル構築装置及びプログラム
CN112700568A (zh) 一种身份认证的方法、设备及计算机可读存储介质
WO2024045350A1 (zh) 一种基于深度学习的眼动活体检测方法和系统
CN114022514B (zh) 一种融合头部姿态和眼球跟踪的实时视线推断方法
CN104036238B (zh) 基于主动光的人眼定位的方法
Perra et al. Adaptive eye-camera calibration for head-worn devices
WO2021203368A1 (zh) 图像处理方法、装置、电子设备和存储介质
JP2014064083A (ja) 監視装置及び方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19900930

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 10/11/2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19900930

Country of ref document: EP

Kind code of ref document: A1