CN115097903B - MR glasses control method and device, MR glasses and storage medium - Google Patents
MR glasses control method and device, MR glasses and storage medium Download PDFInfo
- Publication number
- CN115097903B CN115097903B CN202210547141.XA CN202210547141A CN115097903B CN 115097903 B CN115097903 B CN 115097903B CN 202210547141 A CN202210547141 A CN 202210547141A CN 115097903 B CN115097903 B CN 115097903B
- Authority
- CN
- China
- Prior art keywords
- user
- glasses
- information
- target object
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000011521 glass Substances 0.000 title claims abstract description 164
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000002372 labelling Methods 0.000 claims abstract description 28
- 230000000007 visual effect Effects 0.000 claims abstract description 22
- 210000003128 head Anatomy 0.000 claims description 41
- 210000001508 eye Anatomy 0.000 claims description 23
- 238000010030 laminating Methods 0.000 claims description 14
- 210000005252 bulbus oculi Anatomy 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 230000004418 eye rotation Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 229910052709 silver Inorganic materials 0.000 description 1
- 239000004332 silver Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/1613—Constructional details or arrangements for portable computers
- G06F1/163—Wearable computers, e.g. on a belt
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention belongs to the technical field of equipment control, and discloses an MR (magnetic resonance) glasses control method and device, an MR glasses and a storage medium. The method comprises the following steps: collecting a visual field image in the display range of the MR glasses; determining object information of a target object appointed to be searched according to input information of a user; and labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object. Through the mode, through the input information of the user, the object information of the target object appointed by the user is accurately judged, then the target object which the user wants to search in the field of view of the current MR glasses is determined and marked by combining the field of view image and the object information, so that the user can search objects more conveniently and intuitively, the object searching time of the user is reduced, and the use experience of the user is improved.
Description
Technical Field
The present invention relates to the field of device control technologies, and in particular, to a method and an apparatus for controlling MR glasses, an MR glasses, and a storage medium.
Background
In the current technological development, many forms of user virtual interaction modes appear, wherein VR is a pure virtual digital picture, mixed Reality including AR is a virtual digital picture+naked eye Reality, MR is a digital reality+virtual digital picture, and MR technology combines advantages of VR and AR, so that AR technology can be better represented. MR glasses are intelligent wearable devices carrying MR technology.
In life, people often want to find something or objects, but the fields of vision of people are disordered, and it is difficult to find something which the people want to find from the background and a plurality of objects stacked together. MR glasses also only enable information that helps the user identify the type and parameters of the object at the focus of the line of sight, and do not assist the user in object finding.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide an MR (magnetic resonance) glasses control method, an MR glasses control device, MR glasses and a storage medium, and aims to solve the technical problem that in the prior art, a user cannot conveniently and intuitively find things when wearing the MR glasses.
To achieve the above object, the present invention provides an MR glasses control method, comprising the steps of:
collecting a visual field image in the display range of the MR glasses;
determining object information of a target object appointed to be searched according to input information of a user;
and labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object.
Optionally, the determining object information of the target object specified to be searched according to the input information of the user includes:
collecting input information sent by a user to the MR glasses;
determining the type of the object to be determined according to the input information;
and displaying the types of the objects to be determined to a user through the MR glasses, and selecting object information of a target object according to the head gesture of the user fed back by the user.
Optionally, the collecting input information sent by the user to the MR glasses includes:
acquiring eyeball images of a user through an inner side camera of the MR glasses to obtain eye information of the user;
acquiring hand images of a user through an outer side camera of the MR glasses to obtain gesture information of the user;
invoking head posture information recorded by a gyroscope;
and taking the eye information of the user, the gesture information of the user and the head gesture information as input information.
Optionally, the determining the object type to be determined according to the input information includes:
determining the attribute of the target object described by the user according to the input information;
inquiring a preset database according to the target object attribute to obtain a plurality of candidate object types;
determining the matching degree of each candidate object type and the target object attribute according to the target object attribute;
and determining a plurality of object types to be determined according to the matching degree.
Optionally, the determining, according to the target object attribute, the matching degree between each candidate object type and the target object attribute includes:
determining each matching parameter according to the target object attribute;
determining the coincidence rate of each candidate object type corresponding to the target object attribute according to the matching parameters;
determining the omission ratio of each candidate object type corresponding to the target object attribute according to the matching parameters;
and determining the matching degree according to the coincidence rate and the omission rate.
Optionally, the displaying each object type to be determined to the user through the MR glasses, and selecting object information of the target object according to the user head gesture fed back by the user, includes:
displaying the types of the objects to be determined to a user through the MR glasses so as to enable the user to feed back a selection instruction through the head gesture;
collecting a selection instruction fed back by a user through the head gesture through a gyroscope;
determining the type of the object to be determined selected by the user according to the selection instruction, and taking the type of the object to be determined selected by the user as a target object;
and acquiring attribute information corresponding to the target object, and taking the attribute information as object information of the target object.
Optionally, the labeling the target object in the MR glasses according to the view image and the object information includes:
performing image recognition on the visual field image based on the object information to obtain a plurality of prediction similar objects;
calculating attribute laminating degree of each predicted similar object and the object information;
selecting a target searching object from the predicted similar objects according to the attribute fitting degree;
determining a plurality of areas to be marked according to the target searching object and the view image;
generating a glasses display strategy according to the region to be marked;
and labeling a target object to a user in the MR glasses according to the glasses display strategy.
In addition, to achieve the above object, the present invention also proposes an MR glasses control apparatus including:
the image acquisition module is used for acquiring a view image in the display range of the MR glasses;
the tracking module is used for determining object information of a target object appointed to be searched according to input information of a user;
and the display module is used for labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object.
In addition, to achieve the above object, the present invention also proposes an MR glasses comprising: a memory, a processor, and an MR glasses control program stored on the memory and executable on the processor, the MR glasses control program configured to implement the steps of the MR glasses control method as described above.
In addition, to achieve the above object, the present invention also proposes a storage medium having stored thereon an MR glasses control program which, when executed by a processor, implements the steps of the MR glasses control method as described above.
The invention collects the view field image in the display range of the MR glasses; determining object information of a target object appointed to be searched according to input information of a user; and labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object. Through the mode, the object information of the target object appointed by the user is accurately judged through the input information of the user, then the target object which the user wants to search in the field of view of the current MR glasses is determined and marked by combining the field image and the object information, so that the user can search objects more conveniently and intuitively, the object searching time of the user is reduced, and the use experience of the user is improved.
Drawings
FIG. 1 is a schematic diagram of the structure of MR glasses of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart of a first embodiment of the MR glasses control method of the present invention;
FIG. 3 is a schematic diagram of an MR glasses apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an MR glasses labeling target object according to an embodiment of the MR glasses control method of the present invention;
FIG. 5 is a flowchart of a second embodiment of the MR glasses control method of the present invention;
fig. 6 is a block diagram showing the structure of a first embodiment of the MR glasses control device of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic diagram of MR glasses structure of a hardware operation environment according to an embodiment of the present invention.
As shown in fig. 1, the MR glasses may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the structure shown in fig. 1 is not limiting of MR glasses and may include more or fewer components than shown, or certain components in combination, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and an MR glasses control program may be included in the memory 1005 as one storage medium.
In the MR glasses shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the MR glasses of the present invention may be disposed in the MR glasses, and the MR glasses call the MR glasses control program stored in the memory 1005 through the processor 1001, and execute the MR glasses control method provided by the embodiment of the present invention.
An embodiment of the present invention provides an MR glasses control method, referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of an MR glasses control method according to the present invention.
In this embodiment, the MR glasses control method includes the following steps:
step S10: a field of view image is acquired within the display range of the MR glasses.
It should be noted that, the execution body of the embodiment is MR glasses, and may be any type and any type of MR glasses, which is not limited in this embodiment.
In a specific implementation, a schematic diagram of a module of the MR glasses is shown in fig. 3, wherein the module includes three modules of an MR glasses control device in the MR glasses respectively, the image acquisition module 10 and the tracking module 20 acquire visual field images and input information, the tracking module 20 determines object information of a target object, finally the object information and the visual field images are transmitted to the display module 30, and the display module 30 displays and marks the object information and the visual field images for a user to help the user to find the target object.
It should be understood that acquiring a view image within the display range of MR glasses refers to: the acquisition module directly acquires images in a visual field which can be observed by eyes of a user through the MR glasses. The field image is an image within the field of view of the lens display region of the MR glasses.
Step S20: object information specifying the target object to be found is determined based on the input information of the user.
In a specific implementation, the input information refers to instruction information input to the MR glasses by the user through eyes, gestures or voices.
It should be understood that the target object refers to an object that the user is currently looking for, and may be any object, for example: keys, cups, etc., which are not limited in this embodiment.
In a specific implementation, after receiving the input information, determining a target object designated to be searched by a user according to the input information sent by the user, and then acquiring object information corresponding to the target object. The object information includes, but is not limited to, object size, contour, color, etc. of the target object.
Step S30: and labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object.
Note that, labeling the target object in the MR glasses according to the view image and the object information means that: according to the visual field image and the object information, determining an object closest to the target object and an area where the object is located on the visual field image, and then circling the object on a lens screen of the MR eye by a labeling method, so that the object searching of the auxiliary user is realized.
In a specific implementation, the solution of this embodiment may also be implemented in a scenario other than searching, for example: based on the object locking of the MR glasses for providing image recognition for the user in the field of view, the object is always marked on the lens display screen of the MR glasses, so that virtual information (video, animation, characters and the like) generated in real time is superimposed, visual operation guidance, recognition detection, remote support and other works are provided for the user, the user is assisted to complete business works with strong specialization and high complexity, and the aims of reducing the work difficulty and improving the work quality are achieved.
It should be noted that, as shown in fig. 4, a schematic diagram of an MR glasses labeling target object is shown, the rectangles, the pentagons and the triangles in fig. 4 are all different objects in the field of view of the MR glasses, when the target object is determined to be a pentagon, the pentagon object is circled by a method of labeling on a lens screen and is prompted to a user, and a dotted line box in the figure is a labeling manner and can be circled by lines. Fig. 4 is merely schematic and does not limit the description of the present embodiment in detail.
Further, in order to accurately mark the closest target object, step S30 further includes: performing image recognition on the visual field image based on the object information to obtain a plurality of prediction similar objects; calculating attribute laminating degree of each predicted similar object and the object information; selecting a target searching object from the predicted similar objects according to the attribute fitting degree; determining a plurality of areas to be marked according to the target searching object and the view image; generating a glasses display strategy according to the region to be marked; and labeling a target object to a user in the MR glasses according to the glasses display strategy.
In a specific implementation, performing image recognition on the view image based on the object information, and obtaining a plurality of predicted similar objects refers to: and determining keywords of image recognition according to the object information, and then carrying out image recognition in the visual field image based on the keywords to obtain a plurality of objects with recognition results, namely, predicting similar objects.
It should be noted that, calculating attribute fitting degrees of each predicted similar object and the object information; selecting a target object from the predicted similar objects according to the attribute fitting degree refers to: and calculating the attribute fitting degree of the predicted similar objects and the object information, namely, predicting the similarity degree of the attributes of the similar objects and the attributes described by the object information, and then selecting a preset number of predicted similar objects with the highest similarity degree as targets to search for the objects. The preset number is any number preset, which is not limited in this embodiment.
In specific implementation, the specific calculation method of the attribute laminating degree is as follows: and comparing the predicted similar object with specific information of the same attribute described by the object information to obtain the similarity of each attribute, and finally calculating the attribute laminating degree of each predicted similar object and the object information based on the similarity of each attribute. For example: and comparing the predicted similar object with the color attribute described by the object information, so as to obtain the similarity degree of the color attribute.
Specifically, the calculation of the attribute fitting degree includes the similarity degree of three attributes in the object information, which are respectively: color attributes and image attributes.
The method for calculating the similarity degree of the color attributes comprises the following steps: obtaining a color value mean value a of a predicted similar object in an RGB color mode, extracting a color value mean value b of the predicted similar object in the RGB color mode, and then calculating a similarity degree calculation formula of color attributes as follows:
where W is the degree of similarity of the color attributes.
The method for calculating the similarity degree of the image attributes comprises the following steps: carrying out gray processing on the outline of the predicted similar object in the view image and the outline of the target object described by the object attribute to obtain a predicted similar object gray image and a target object gray image, then respectively calculating gray average values of pixels of each row of the predicted similar object gray image and the target object gray image in sequence, wherein each gray average value corresponds to a characteristic of one row, then calculating variances of all the obtained gray average values, wherein the obtained variances are characteristic values of the image, so that a predicted similar object characteristic value c and a target object characteristic value d are obtained, and a similarity degree calculation formula of the image attribute is as follows:
wherein R is the similarity of image attributes.
Finally, the attribute laminating degree is calculated by the following steps:
the attribute fit depends on the values of W and R, the higher the attribute fit of each predicted similar object to the target object.
It should be understood that determining a plurality of regions to be annotated from the target finding object and the field of view image refers to: after the target finding objects are determined, determining the areas where the target finding objects are located and need to be marked in the visual field image, wherein each target finding object corresponds to one area to be marked as the area to be marked.
In a specific implementation, generating the glasses display policy according to the region to be marked refers to: and determining a glasses display strategy according to the region to be marked in combination with a preset marking mode, wherein the glasses display strategy is to mark the region to be marked in the lens display region of the glasses through the preset marking mode. The preset labeling mode is preset by a user, and specific labeling modes include but are not limited to: the region to be marked is circled in the lens display region of the MR glasses through a virtual circular track, or the region to be marked is circled in the lens display region of the MR glasses through a virtual arrow track.
By the method, the object is accurately marked in the visual field of the user according to the object information and the visual field image, so that the user is assisted to finish object searching work with strong specialization and high complexity, and the aims of reducing the work difficulty and improving the work quality are achieved.
The embodiment acquires the view field image in the display range of the MR glasses; determining object information of a target object appointed to be searched according to input information of a user; and labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object. Through the mode, the object information of the target object appointed by the user is accurately judged through the input information of the user, then the target object which the user wants to search in the field of view of the current MR glasses is determined and marked by combining the field image and the object information, so that the user can search objects more conveniently and intuitively, the object searching time of the user is reduced, and the use experience of the user is improved.
Referring to fig. 5, fig. 5 is a flowchart of a second embodiment of an MR glasses control method according to the present invention.
Based on the first embodiment, the MR glasses control method of the present embodiment includes, at step S20:
step S201: and collecting input information sent by a user to the MR glasses.
It should be noted that the MR glasses include a video collector, an audio collector, and a gyroscope. The video collector can collect eye images of the user and gesture information of the user in front of the MR glasses; the audio collector can collect voice instructions sent by a user, and the gyroscope can collect the head gesture of the user.
Further, in order to accurately collect various types of input information of the user, step S201 includes: acquiring eyeball images of a user through an inner side camera to obtain eye information of the user; acquiring hand images of a user through an outer side camera to obtain gesture information of the user; invoking head posture information recorded by a gyroscope; and taking the eye information of the user, the gesture information of the user and the head gesture information as input information.
It should be understood that the inner camera is disposed inside the frame of the MR glasses, collects an eye image of the user, and then determines eye information of the user through the eye image, wherein the eye information includes an eye focus, an eye rotation direction and a rotation track of the user.
In specific implementation, an outside camera is used for collecting hand images of a user, wherein the outside camera is arranged outside a glasses frame of the MR glasses, and collects front images of the MR glasses, namely hand images of the user, then image recognition is carried out based on the hand images of the user to obtain user gestures, then the user gestures are compared with preset instruction gestures, gesture instructions of the user are determined, and finally gesture information of the user is generated based on the gesture instructions.
It should be noted that, the head posture information recorded by the gyroscope is called: invoking a gyroscope built in the MR glasses, wherein the gyroscope can be 6dof, and then determining head posture information based on information acquired by the gyroscope, wherein the head posture information can comprise related information of deviation and rotation direction of the head of the user relative to a standard pose. The standard pose refers to the head pose of the user when looking at the front of the head-up.
It should be understood that the user eye information, the user gesture information, and the head pose information are referred to as input information: and determining relevant information such as the attribute of an object which is searched by the user through the specification of eye movement, gesture and voice description according to the eye information, the gesture information and the head gesture information of the user, and the like, and taking the information as input information.
By the method, the root can accurately acquire the input information of the user through various hardware and acquisition equipment, so that the judgment of the target object is more accurate.
Step S202: and determining the type of the object to be determined according to the input information.
It should be noted that, after receiving the input information, analysis is performed according to the input information, the types of the candidate objects are determined, then the attribute of the target object is determined, the matching degree of each candidate object type and the attribute of the target object is determined, and then the type of the object to be determined is obtained based on the matching degree.
Further, in order to be able to accurately determine the kind of the object to be determined, step S202 includes: determining the attribute of the target object described by the user according to the input information; inquiring a preset database according to the target object attribute to obtain a plurality of candidate object types; determining the matching degree of each candidate object type and the target object attribute according to the target object attribute; and determining a plurality of object types to be determined according to the matching degree.
It should be noted that, the target object attribute refers to a description attribute corresponding to the target object that the user specifies to find, for example: color, shape, size, etc. Specifically, the target object attribute can be obtained from input information, wherein the user eye information in the input information can be that the MR glasses provide a plurality of alternative description attributes on a display screen, and then the user selects the description attributes on the screen through eyeball focus and eyeball rotation; the method can also be user gesture information, and the user selects description attributes on the display screen of the MR glasses through gesture operation of a cursor; or the user selects the description attribute on the display screen of the MR glasses by moving the head in the head posture information.
It should be understood that, according to the target object attribute, querying a preset database to obtain a plurality of candidate object types refers to: after determining the target object attribute, the target object attribute is queried in a preset database, and a plurality of candidate object types matched with the target object attribute can be determined from the preset database. The preset database stores attributes corresponding to various candidate object types, wherein the attributes corresponding to the candidate object types are the same as descriptions of the attributes of the target object, for example: when the candidate object type is a key, the attribute corresponding to the candidate object type is silver, black, 5 cm in length, or the like.
In a specific implementation, determining the matching degree of each candidate object type and the target object attribute according to the target object attribute refers to: and performing text comparison on the description attribute corresponding to the candidate object type and the description attribute corresponding to the target object type to obtain a coincidence rate, then calculating the part of the description attribute corresponding to the target object type, which does not appear in the description attribute corresponding to each candidate object type, wherein the part is the percentage of the total target object attribute, namely the omission rate, and finally calculating the matching degree corresponding to each candidate object type by combining the coincidence rate and the omission rate.
It should be noted that, determining a plurality of object types to be determined according to the matching degree means: the sorting is performed according to the matching degree, then the sorting result is taken as the object type to be determined from front to back and the candidate object types with preset number of ranks, specifically, the preset number of ranks is any number of ranks, which can be set by the user, and the embodiment is not limited to this. For example: when the preset number ranking is 2, the sorting result is: when the key, the USB flash disk and the coin are used, the first 2 bits are taken, and the key and the USB flash disk are used as the types of objects to be determined.
By the method, the fact that a plurality of object types to be determined which are more in line with the description of the user are accurately judged according to the input information is displayed to the user, and the object which the user appointed to search for can be accurately determined.
Further, in order to accurately calculate the matching degree, the step of determining the matching degree of each candidate object type and the target object attribute according to the target object attribute includes: determining each matching parameter according to the target object attribute; determining the coincidence rate of each candidate object type corresponding to the target object attribute according to the matching parameters; determining the omission ratio of each candidate object type corresponding to the target object attribute according to the matching parameters; and determining the matching degree according to the coincidence rate and the omission rate.
It should be noted that, determining each matching parameter according to the target object attribute refers to: each description attribute is determined according to the target object attribute, and as a matching parameter, for example: size, color, contour, material, etc.
It should be understood that determining, according to the matching parameter, the coincidence ratio of each candidate object type to the target object attribute refers to: and performing text comparison on the matching parameters corresponding to the candidate object types and the matching parameters corresponding to the target object attributes to obtain the coincidence rate.
In a specific implementation, determining, according to the matching parameter, a omission ratio of each candidate object type corresponding to the target object attribute refers to: and calculating the part of the description attribute corresponding to the object attribute, which does not appear in the description attribute corresponding to each candidate object type, wherein the percentage of the description attribute corresponding to the overall object attribute is taken as the missing rate.
It should be noted that, the calculation formula of the matching degree is: a=q-P, where a is the degree of matching, Q is the coincidence rate, and P is the leak rate.
In this way, an accurate calculation of the matching degree of each candidate object type is achieved, so that the most pertinent candidate object type is selected as the object type to be determined.
Step S203: and displaying the types of the objects to be determined to a user through the MR glasses, and selecting object information of a target object according to the head gesture of the user fed back by the user.
It should be understood that determining object information of the target object selected to be found by the user based on the object type to be determined and the head pose of the user refers to: after the object types to be determined are determined, the object types to be determined are displayed on the MR glasses one by one, then one object is selected as a target object through the head gesture of the user, and the related information of the target object is used as object information.
Further, in order to be able to accurately determine the object information, step S203 includes: displaying the types of the objects to be determined to a user through the MR glasses so as to enable the user to feed back a selection instruction through the head gesture; collecting a selection instruction fed back by a user through the head gesture through a gyroscope; determining the type of the object to be determined selected by the user according to the selection instruction, and taking the type of the object to be determined selected by the user as a target object; and acquiring attribute information corresponding to the target object, and taking the attribute information as object information of the target object.
It should be noted that, displaying each object type to be determined to a user through the MR glasses, so that the user can feedback the selection instruction through the head gesture refers to: and displaying the object types to be determined on a display screen of the MR glasses in a list form, and prompting a user to select a stroke selection instruction through a virtual cursor on a swing control screen of the head.
It should be understood that the step of acquiring, by the gyroscope, a selection instruction fed back by the user through the head gesture refers to detecting, by the gyroscope, the head gesture of the user, thereby receiving the selection instruction of the user.
In a specific implementation, determining the type of the object to be determined selected by the user according to the selection instruction, and taking the type of the object to be determined selected by the user as the target object refers to: and determining one selected by the user from the various object types to be determined according to the selection instruction, and then taking the selected object as a target object.
It should be noted that, acquiring the attribute information corresponding to the target object, and taking the attribute information as the object information of the target object refers to: and acquiring related information of the description attribute corresponding to the target object as attribute information, and then storing the attribute information as object information for subsequent calling.
By the method, the target object is accurately determined according to the selection instruction of the head gesture of the user, so that the user can be more accurately assisted in object searching.
The embodiment collects the input information sent by the user to the MR glasses; determining the type of the object to be determined according to the input information; according to the object type to be determined and the head gesture of the user, the object information of the target object selected and searched by the user is determined, the object type to be determined, which is possibly designated by the user, is screened out according to the input information sent by the user, and then the object information of the target object is determined based on the head gesture of the user, so that the labeling of the display module on the target object is more accurate.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium stores an MR glasses control program, and the MR glasses control program realizes the steps of the MR glasses control method when being executed by a processor.
The storage medium adopts all the technical solutions of all the embodiments, so that the storage medium has at least all the beneficial effects brought by the technical solutions of the embodiments, and is not described in detail herein.
Referring to fig. 6, fig. 6 is a block diagram showing the structure of a first embodiment of the MR glasses control device of the present invention.
As shown in fig. 6, an MR glasses control apparatus according to an embodiment of the present invention includes:
the image acquisition module 10 is used for acquiring the view image in the display range of the MR glasses.
The tracking module 20 is configured to determine object information of the target object specified to be searched according to input information of a user.
And a display module 30, configured to annotate the target object in the MR glasses according to the view image and the object information, so as to assist a user in finding the target object.
The embodiment acquires the view field image in the display range of the MR glasses; determining object information of a target object appointed to be searched according to input information of a user; and labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object. Through the mode, the object information of the target object appointed by the user is accurately judged through the input information of the user, then the target object which the user wants to search in the field of view of the current MR glasses is determined and marked by combining the field image and the object information, so that the user can search objects more conveniently and intuitively, the object searching time of the user is reduced, and the use experience of the user is improved.
In an embodiment, the tracking module 20 is further configured to collect input information sent by the user to the MR glasses; determining the type of the object to be determined according to the input information; and displaying the types of the objects to be determined to a user through the MR glasses, and selecting object information of a target object according to the head gesture of the user fed back by the user.
In an embodiment, the tracking module 20 is further configured to acquire an eyeball image of the user through an inner side camera of the MR glasses, so as to obtain eye information of the user; acquiring hand images of a user through an outer side camera of the MR glasses to obtain gesture information of the user; invoking head posture information recorded by a gyroscope; and taking the eye information of the user, the gesture information of the user and the head gesture information as input information.
In one embodiment, the tracking module 20 is further configured to determine a target object attribute described by the user according to the input information; inquiring a preset database according to the target object attribute to obtain a plurality of candidate object types; determining the matching degree of each candidate object type and the target object attribute according to the target object attribute; and determining a plurality of object types to be determined according to the matching degree.
In an embodiment, the tracking module 20 is further configured to determine each matching parameter according to the target object attribute; determining the coincidence rate of each candidate object type corresponding to the target object attribute according to the matching parameters; determining the omission ratio of each candidate object type corresponding to the target object attribute according to the matching parameters; and determining the matching degree according to the coincidence rate and the omission rate.
In an embodiment, the tracking module 20 is further configured to display each of the object types to be determined to the user through the MR glasses, so that the user feeds back a selection instruction through the head gesture; collecting a selection instruction fed back by a user through the head gesture through a gyroscope; determining the type of the object to be determined selected by the user according to the selection instruction, and taking the type of the object to be determined selected by the user as a target object; and acquiring attribute information corresponding to the target object, and taking the attribute information as object information of the target object.
In an embodiment, the display module 30 is further configured to perform image recognition on the view image based on the object information to obtain a plurality of predicted similar objects; calculating attribute laminating degree of each predicted similar object and the object information; selecting a target searching object from the predicted similar objects according to the attribute fitting degree; determining a plurality of areas to be marked according to the target searching object and the view image; generating a glasses display strategy according to the region to be marked; and labeling a target object to a user in the MR glasses according to the glasses display strategy.
It should be understood that the foregoing is illustrative only and is not limiting, and that in specific applications, those skilled in the art may set the invention as desired, and the invention is not limited thereto.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.
In addition, technical details not described in detail in the present embodiment may refer to the MR glasses control method provided in any embodiment of the present invention, and are not described herein.
Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory)/RAM, magnetic disk, optical disk) and including several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (9)
1. An MR glasses control method, characterized in that the MR glasses control method comprises:
collecting a visual field image in the display range of the MR glasses;
determining object information of a target object appointed to be searched according to input information of a user, wherein the input information is instruction information input to the MR glasses by the user through eyes, gestures or voices, and the target object refers to the object which the user is searching for currently;
labeling the target object in the MR glasses according to the view image and the object information so as to assist a user to find the target object, wherein labeling the target object in the MR glasses according to the view image and the object information means that: according to the visual field image and the object information, determining an object closest to the target object and an area where the object is located on the visual field image, and then circling the object on a lens screen of the MR glasses for a user by a labeling method;
the labeling the target object in the MR glasses according to the view image and the object information includes:
performing image recognition on the visual field image based on the object information to obtain a plurality of prediction similar objects;
calculating attribute laminating degrees of each predicted similar object and the object information, wherein the attribute laminating degrees comprise laminating degrees of color attributes and image attributes, gray level average values of each row of pixel points of gray level images of each predicted similar object and gray level images of each target object are calculated in sequence, variance is calculated on each gray level average value to obtain a predicted similar object characteristic value and a target object characteristic value, and the laminating degrees of the image attributes are calculated according to the predicted similar object characteristic values and the target object characteristic values;
selecting a target searching object from the predicted similar objects according to the attribute fitting degree;
determining a plurality of areas to be marked according to the target searching object and the view image;
generating a glasses display strategy according to the region to be marked;
and labeling a target object to a user in the MR glasses according to the glasses display strategy.
2. The method of claim 1, wherein determining object information specifying a sought target object based on input information of a user comprises:
collecting input information sent by a user to the MR glasses;
determining the type of the object to be determined according to the input information;
and displaying the types of the objects to be determined to a user through the MR glasses, and selecting object information of a target object according to the head gesture of the user fed back by the user.
3. The method of claim 2, wherein the capturing input from the user to the MR glasses comprises:
acquiring eyeball images of a user through an inner side camera of the MR glasses to obtain eye information of the user;
acquiring hand images of a user through an outer side camera of the MR glasses to obtain gesture information of the user;
invoking head posture information recorded by a gyroscope;
and taking the eye information of the user, the gesture information of the user and the head gesture information as input information.
4. The method of claim 2, wherein said determining the type of object to be determined from said input information comprises:
determining the attribute of the target object described by the user according to the input information;
inquiring a preset database according to the target object attribute to obtain a plurality of candidate object types;
determining the matching degree of each candidate object type and the target object attribute according to the target object attribute;
and determining a plurality of object types to be determined according to the matching degree.
5. The method of claim 4, wherein said determining a degree of match of each of said candidate object categories with said target object attribute based on said target object attribute comprises:
determining each matching parameter according to the target object attribute;
determining the coincidence rate of each candidate object type corresponding to the target object attribute according to the matching parameters;
determining the omission ratio of each candidate object type corresponding to the target object attribute according to the matching parameters;
and determining the matching degree according to the coincidence rate and the omission rate.
6. The method of claim 2, wherein the presenting each of the object types to be determined to the user through the MR glasses, and selecting object information of the target object according to the user's head pose fed back by the user, comprises:
displaying the types of the objects to be determined to a user through the MR glasses so as to enable the user to feed back a selection instruction through the head gesture;
collecting a selection instruction fed back by a user through the head gesture through a gyroscope;
determining the type of the object to be determined selected by the user according to the selection instruction, and taking the type of the object to be determined selected by the user as a target object;
and acquiring attribute information corresponding to the target object, and taking the attribute information as object information of the target object.
7. An MR glasses control apparatus, characterized in that the MR glasses control apparatus comprises:
the image acquisition module is used for acquiring a view image in the display range of the MR glasses;
the tracking module is used for determining object information of a target object appointed to be searched according to input information of a user, wherein the input information is instruction information input to the MR glasses by the user through eyes, gestures or voices, and the target object refers to an object which the user is searching for currently;
the display module is configured to label the target object in the MR glasses according to the view image and the object information, so as to assist a user to find the target object, and labeling the target object in the MR glasses according to the view image and the object information means that: according to the visual field image and the object information, determining an object closest to the target object and an area where the object is located on the visual field image, and then circling the object on a lens screen of the MR glasses for a user by a labeling method;
the display module is further used for carrying out image recognition on the visual field image based on the object information to obtain a plurality of prediction similar objects; calculating attribute laminating degrees of each predicted similar object and the object information, wherein the attribute laminating degrees comprise laminating degrees of color attributes and image attributes, gray level average values of each row of pixel points of gray level images of each predicted similar object and gray level images of each target object are calculated in sequence, variance is calculated on each gray level average value to obtain a predicted similar object characteristic value and a target object characteristic value, and the laminating degrees of the image attributes are calculated according to the predicted similar object characteristic values and the target object characteristic values; selecting a target searching object from the predicted similar objects according to the attribute fitting degree; determining a plurality of areas to be marked according to the target searching object and the view image; generating a glasses display strategy according to the region to be marked; and labeling a target object to a user in the MR glasses according to the glasses display strategy.
8. An MR glasses, characterized in that the MR glasses comprise: memory, a processor and an MR glasses control program stored on the memory and executable on the processor, the MR glasses control program configured to implement the MR glasses control method according to any one of claims 1 to 6.
9. A storage medium, wherein an MR glasses control program is stored on the storage medium, which when executed by a processor, implements the MR glasses control method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210547141.XA CN115097903B (en) | 2022-05-19 | 2022-05-19 | MR glasses control method and device, MR glasses and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210547141.XA CN115097903B (en) | 2022-05-19 | 2022-05-19 | MR glasses control method and device, MR glasses and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115097903A CN115097903A (en) | 2022-09-23 |
| CN115097903B true CN115097903B (en) | 2024-04-05 |
Family
ID=83289420
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210547141.XA Active CN115097903B (en) | 2022-05-19 | 2022-05-19 | MR glasses control method and device, MR glasses and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115097903B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109298780A (en) * | 2018-08-24 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Information processing method, device, AR equipment and storage medium based on AR |
| CN109782902A (en) * | 2018-12-17 | 2019-05-21 | 中国科学院深圳先进技术研究院 | A kind of operation indicating method and glasses |
| CN111552076A (en) * | 2020-05-13 | 2020-08-18 | 歌尔科技有限公司 | Image display method, AR glasses and storage medium |
| CN111625091A (en) * | 2020-05-14 | 2020-09-04 | 佳都新太科技股份有限公司 | Label overlapping method and device based on AR glasses |
| CN112034977A (en) * | 2019-06-04 | 2020-12-04 | 陈涛 | Method for MR intelligent glasses content interaction, information input and recommendation technology application |
| CN112507799A (en) * | 2020-11-13 | 2021-03-16 | 幻蝎科技(武汉)有限公司 | Image identification method based on eye movement fixation point guidance, MR glasses and medium |
| CN112788583A (en) * | 2020-12-25 | 2021-05-11 | 深圳酷派技术有限公司 | Equipment searching method and device, storage medium and electronic equipment |
| CN113941138A (en) * | 2020-08-06 | 2022-01-18 | 黄得锋 | AR interaction control system, device and application |
| CN114119171A (en) * | 2021-12-02 | 2022-03-01 | 浙江科顿科技有限公司 | MR/AR/VR shopping and retrieval scene control method, mobile terminal and readable storage medium |
| CN114285984A (en) * | 2020-09-27 | 2022-04-05 | 宇龙计算机通信科技(深圳)有限公司 | AR (augmented reality) -glasses-based camera switching method and device, storage medium and terminal |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180150791A1 (en) * | 2016-11-29 | 2018-05-31 | Wal-Mart Stores, Inc. | Augmented reality-assisted modular set-up and product stocking systems and methods |
-
2022
- 2022-05-19 CN CN202210547141.XA patent/CN115097903B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109298780A (en) * | 2018-08-24 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Information processing method, device, AR equipment and storage medium based on AR |
| CN109782902A (en) * | 2018-12-17 | 2019-05-21 | 中国科学院深圳先进技术研究院 | A kind of operation indicating method and glasses |
| WO2020125499A1 (en) * | 2018-12-17 | 2020-06-25 | 中国科学院深圳先进技术研究院 | Operation prompting method and glasses |
| CN112034977A (en) * | 2019-06-04 | 2020-12-04 | 陈涛 | Method for MR intelligent glasses content interaction, information input and recommendation technology application |
| CN111552076A (en) * | 2020-05-13 | 2020-08-18 | 歌尔科技有限公司 | Image display method, AR glasses and storage medium |
| CN111625091A (en) * | 2020-05-14 | 2020-09-04 | 佳都新太科技股份有限公司 | Label overlapping method and device based on AR glasses |
| CN113941138A (en) * | 2020-08-06 | 2022-01-18 | 黄得锋 | AR interaction control system, device and application |
| CN114285984A (en) * | 2020-09-27 | 2022-04-05 | 宇龙计算机通信科技(深圳)有限公司 | AR (augmented reality) -glasses-based camera switching method and device, storage medium and terminal |
| CN112507799A (en) * | 2020-11-13 | 2021-03-16 | 幻蝎科技(武汉)有限公司 | Image identification method based on eye movement fixation point guidance, MR glasses and medium |
| CN112788583A (en) * | 2020-12-25 | 2021-05-11 | 深圳酷派技术有限公司 | Equipment searching method and device, storage medium and electronic equipment |
| CN114119171A (en) * | 2021-12-02 | 2022-03-01 | 浙江科顿科技有限公司 | MR/AR/VR shopping and retrieval scene control method, mobile terminal and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115097903A (en) | 2022-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110084775B (en) | Image processing method and device, electronic equipment and storage medium | |
| US10133951B1 (en) | Fusion of bounding regions | |
| US8274578B2 (en) | Gaze tracking apparatus and method using difference image entropy | |
| US9350913B2 (en) | Assisted text input for computing devices | |
| US10540378B1 (en) | Visual search suggestions | |
| US20090110245A1 (en) | System and method for rendering and selecting a discrete portion of a digital image for manipulation | |
| EP2854074A1 (en) | Method and device for implementing augmented reality | |
| EP2378393A2 (en) | Image processing system, image processing apparatus, image processing method, and program | |
| CN107229741B (en) | Information searching method, device, equipment and storage medium | |
| CN110741377A (en) | Face image processing method and device, storage medium and electronic equipment | |
| US20210141826A1 (en) | Shape-based graphics search | |
| US20100128118A1 (en) | Identification of visual fixations in a video stream | |
| US10705720B2 (en) | Data entry system with drawing recognition | |
| CN105894362A (en) | Method and device for recommending related item in video | |
| TW201448585A (en) | Real time object scanning using a mobile phone and cloud-based visual search engine | |
| US7869633B2 (en) | Person image retrieval apparatus | |
| CN113037925B (en) | Information processing method, information processing apparatus, electronic device, and readable storage medium | |
| CN115097903B (en) | MR glasses control method and device, MR glasses and storage medium | |
| CN111953907B (en) | Composition method and device | |
| CN113298593A (en) | Commodity recommendation and image detection method, commodity recommendation and image detection device, commodity recommendation and image detection equipment and storage medium | |
| CN117146828B (en) | Method and device for guiding picking path, storage medium and computer equipment | |
| CN117315756A (en) | Image recognition method, device, electronic equipment and storage medium | |
| CN116433939B (en) | Sample image generation method, training method, recognition method and device | |
| KR102401626B1 (en) | Method and system for image-based product search | |
| US20230051112A1 (en) | Apparatus of selecting video content for augmented reality, user terminal and method of providing video content for augmented reality |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |