CN116048276A - Virtual remote control method based on head-mounted display equipment and head-mounted display equipment - Google Patents
Virtual remote control method based on head-mounted display equipment and head-mounted display equipment Download PDFInfo
- Publication number
- CN116048276A CN116048276A CN202310101835.5A CN202310101835A CN116048276A CN 116048276 A CN116048276 A CN 116048276A CN 202310101835 A CN202310101835 A CN 202310101835A CN 116048276 A CN116048276 A CN 116048276A
- Authority
- CN
- China
- Prior art keywords
- virtual
- remote controller
- user
- mounted display
- head
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F3/0227—Cooperation and interconnection of the input arrangement with other functional units of a computer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/04847—Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a virtual remote control method based on a head-mounted display device and the head-mounted display device, comprising the following steps: shooting a real object held by a user by using a camera arranged on the head-mounted display device; calibrating the image of the real object to form a remote controller carrier; synthesizing a virtual control interface and the remote controller carrier to form a virtual remote controller, and projecting the virtual remote controller to the front of the eyes of a user; acquiring key actions of a user finger on the real object by using the camera, and identifying the key positions pressed by the finger; and identifying the operation of the user according to the key position pressed by the finger and responding. According to the invention, a real object which is randomly grabbed by a user can be used as a remote controller carrier, a virtual remote controller can be formed after the virtual control interface is projected, and the user can execute remote control operation on the head-mounted display device, so that the problem that the user cannot conveniently find the remote controller after wearing the head-mounted display device is solved, the increase of the number of devices and the increase of hardware cost are avoided, and the head-mounted display device is convenient for the user to carry and use.
Description
Technical Field
The invention belongs to the technical field of wearable equipment, relates to head-mounted display equipment, and particularly relates to a remote control method based on the head-mounted display equipment.
Background
The head-mounted display device is a video playing device which can amplify an image on an ultra-micro display screen through an optical system, project the image on the retina of a user, and then display a large-screen image in the eyes of the user, so that different display effects such as Virtual Reality (VR), augmented Reality (AR), mixed Reality (MR) and the like can be realized.
In the present head-mounted display device, as shown in fig. 1, control keys 1 are generally arranged on a peripheral side surface 2 of the head-mounted display device, for example, on the top of the peripheral side surface 2, or on the left side or right side of the peripheral side surface 2. The number of control keys 1 is generally more than two, including but not limited to an on/off key, a volume adjustment key, a program switch key, a screen brightness adjustment key, etc. After wearing the head-mounted display device, the user cannot see the control keys 1 on the peripheral side surface 2, so that the user can only touch the key positions by hand during operation, and misoperation is easy to occur.
In order to solve the above-described problems, some head-mounted display devices are configured with a dedicated small remote controller, and a user changes the operating state of the head-mounted display device by operating the remote controller. The solution of the special remote controller can lead to the increase of the number of devices, not only the hardware cost is increased, but also the user needs to carry one more peripheral, so that the use is inconvenient; on the other hand, since the head-mounted display device is generally used in an indoor or home environment, and many household appliances are configured with different remote controllers, it is difficult for a user to accurately and easily grasp the remote controllers of the head-mounted display device from the plurality of remote controllers after wearing the head-mounted display device, so that a situation that the remote controllers are misplaced or cannot be found often occurs, resulting in interruption of immersion experience of the user and affecting the satisfaction degree of the user.
Disclosure of Invention
The invention aims to provide a virtual remote control method based on a head-mounted display device, which is used for solving the problem that a configuration entity remote controller is likely to be easy to take by mistake or the remote controller cannot be found.
In order to solve the technical problems, the invention is realized by adopting the following technical scheme:
in one aspect, the present invention provides a virtual remote control method based on a head-mounted display device, including the following virtual remote control procedures:
shooting a real object held by a user by using a camera arranged on the head-mounted display device;
calibrating the image of the real object to form a remote controller carrier;
synthesizing a virtual control interface and the remote controller carrier to form a virtual remote controller, and projecting the virtual remote controller to the front of the eyes of a user;
acquiring key actions of a user finger on the real object by using the camera, and identifying the key positions pressed by the finger;
and identifying the operation of the user according to the key position pressed by the finger and responding.
In some embodiments of the present application, in order to be able to select a suitable real object as a remote controller carrier, the following steps may be configured in the calibration process of the real object image:
firstly, identifying the length and width dimensions of the real object;
then, judging whether the real object can be used as a remote controller carrier according to the length and width dimensions of the virtual control interface; that is to say,
if the length and width dimensions of the real object exceed the length and width dimensions of the virtual control interface, calibrating the image of the real object as a remote controller carrier;
if the length and width dimensions of the real object are smaller than those of the virtual control interface, the real object is indicated to be unsuitable as a remote controller carrier, and a user can be prompted to replace the real object at the moment.
In some embodiments of the present application, to reduce the probability of occurrence of a misoperation, it is preferable to configure a virtual remote controller generating key on a head-mounted display device, and execute the virtual remote control process when the virtual remote controller generating key is triggered; and when the object moves out of the shooting range of the camera, ending the virtual remote control process, and eliminating the virtual remote control image projected in front of the eyes of the user.
In some embodiments of the present application, in order to enable a user to implement remote control operation on all devices in a home by using a virtual remote controller, a remote control signal receiving device is further configured on the head-mounted display device, where the remote control signal receiving device is configured to receive remote control codes and corresponding function key names sent by remote controllers of other devices; generating a virtual remote controller interface for controlling the other equipment according to the function key name, and projecting the virtual remote controller interface on a remote controller carrier to form a virtual remote controller for remotely controlling the other equipment; and then, the remote control codes corresponding to the keys pressed by the user are sent to the other equipment by identifying the key operation of the user on the virtual remote controller and utilizing a remote control signal transmitting device arranged on the head-mounted display equipment, so that the other equipment can be controlled.
In some embodiments of the present application, in order to switch the control interface projected on the remote control carrier, virtual keys related to the other devices may be generated on a virtual control interface for controlling the head-mounted display device; and when the virtual key is identified to be pressed by the finger of the user, projecting a virtual remote controller interface of the other equipment on the remote controller carrier to form a virtual remote controller for remotely controlling the other equipment.
In another aspect, the present invention also provides a head-mounted display device, including:
the camera is used for shooting a real object held by a user;
the real object calibration module is used for calibrating the shot real object image to form a remote controller carrier;
the synthesis module is used for synthesizing the virtual control interface and the remote controller carrier to form a virtual remote controller;
an optical system for projecting the virtual remote control in front of the eyes of a user;
the gesture recognition module is used for recognizing the key position pressed by the finger according to the key action of the user finger on the real object acquired by the camera;
the instruction generation module is used for identifying a user instruction according to the key position pressed by the finger;
and the control module is used for controlling the head-mounted display device to respond to the user operation according to the user instruction.
In some embodiments of the present application, in order to select a suitable real object as a remote controller carrier, the real object calibration module identifies a length-width dimension of the real object according to a real object image captured by the camera, and if the length-width dimension of the real object exceeds a length-width dimension of the virtual control interface, the image of the real object may be calibrated as the remote controller carrier.
In some embodiments of the present application, a speaker may be further configured in the head-mounted display device for playing audio signals; when the real object calibration module judges that the length and width dimensions of the real object are smaller than those of the virtual control interface, the real object is considered unsuitable to be used as a remote controller carrier, and the loudspeaker can be controlled to play prompt information at the moment so as to remind a user of changing the real object.
In some embodiments of the present application, in order to reduce the probability of occurrence of misoperation, a virtual remote controller generating key may be disposed on a peripheral side surface of the head-mounted display device, and electrically connected to the control module; when the control module detects that the key generated by the virtual remote controller is triggered, the camera is started to shoot a real object held by a user, and a virtual remote control process is started; and when the real object is not detected by the camera, the control module ends the virtual remote control process and controls the optical system to eliminate the virtual remote control image projected in front of the eyes of the user.
In some embodiments of the present application, in order to implement the function of the universal remote controller, a remote control signal receiving device and a remote control signal transmitting device may be configured on the head-mounted display device; receiving remote control codes and corresponding function key names sent by remote controllers of other devices by using a remote control signal receiving device, and storing the remote control codes and the corresponding function key names in a memory in an associated manner; the control module generates a virtual remote controller interface for controlling the other equipment according to the function key name received by the remote control signal receiving device and stores the virtual remote controller interface into the memory; the control module is used for adding virtual keys related to other devices on a virtual control interface for controlling the head-mounted display device, projecting virtual remote controller interfaces of the other devices on the remote controller carrier when the gesture recognition module detects that the virtual keys are pressed, so as to form a virtual remote controller for remotely controlling the other devices, recognizing key operation of a user on the virtual remote controller through the gesture recognition module, and then sending remote control codes corresponding to the keys pressed by the user to the remote control signal emitting device, and sending the remote control codes to the other devices through the remote control signal emitting device so as to control the other devices.
Compared with the prior art, the invention has the advantages and positive effects that:
the virtual remote control method can take the real objects which are randomly grabbed by the user as the remote controller carrier, and the virtual remote controller can be formed after the virtual control interface is projected, so that the user can execute remote control operation on the head-mounted display device, the problem that the user is inconvenient to find the remote controller after wearing the head-mounted display device is solved, the increase of the number of devices and the increase of hardware cost are avoided, and the head-mounted display device is convenient to carry and use.
According to the invention, the remote control signal receiving device and the remote control signal transmitting device are configured on the head-mounted display equipment, so that the head-mounted display equipment has a remote control self-learning function, and a virtual universal remote controller can be formed by projecting a virtual remote controller interface associated with other equipment onto a real object grabbed by a user, so that remote control operation of other equipment is realized.
Other features and advantages of the present invention will become more apparent from the following detailed description of embodiments of the present invention, which is to be read in connection with the accompanying drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic structural diagram of one embodiment of a prior art head mounted display device;
FIG. 2 is a schematic diagram of one embodiment of a head mounted display device in accordance with the present invention;
FIG. 3 is a schematic block diagram of a system circuit of one embodiment of a head mounted display device in accordance with the present invention;
FIG. 4 is an effect diagram of one embodiment of a virtual manipulation interface for manipulating a head-mounted display device;
FIG. 5 is an effect diagram of another embodiment of a virtual manipulation interface for manipulating a head-mounted display device;
FIG. 6 is a schematic diagram of an embodiment of a virtual remote control;
FIG. 7 is a schematic diagram of an alternative embodiment of a virtual remote control;
FIG. 8 is a flow chart of one embodiment of a head mounted display device based virtual remote control method in accordance with the present invention;
FIG. 9 is a flow chart of a method of one embodiment of generating a universal virtual remote control.
Description of the embodiments
The following describes the embodiments of the present invention in further detail with reference to the drawings.
It should be noted that, in the description of the present invention, terms such as "front", "rear", "inner", "outer", and the like, which indicate directions or positional relationships are based on directions or positional relationships shown in the drawings, are merely for convenience of description, and do not indicate or imply that the apparatus or elements must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.
In order to facilitate the remote control operation of the head-mounted display devices such as VR, AR, MR and the like by a user, the virtual remote control technology is adopted to provide a virtual remote controller for the user wearing the display device, and the virtual remote controller is used for replacing the traditional special remote controller, so that the problem that the user cannot conveniently find the remote controller after wearing the display device can be solved; the virtual remote controller is used for replacing keys arranged on the peripheral side face of the head-mounted display device to control the head-mounted display device, so that the problem that false triggering is easily caused because a user cannot see the keys can be solved.
The specific mechanical structural design and circuit layout of the head mounted display device of the present embodiment will be described in detail with reference to fig. 2 and 3.
As shown in fig. 2 and 3, in order to implement a virtual remote control function on the head-mounted display device, the present embodiment sets a camera 11 on the head-mounted display device, preferably disposed on the front housing 10 of the head-mounted display device, for collecting a physical image captured by a user and an operation gesture of the user on the physical object. A key 13 may be provided on the peripheral side 12 of the head-mounted display device, and is dedicated to turning on the virtual remote controller generation function, and this key 13 is referred to as a virtual remote controller generation key in this embodiment. Since there is only one key on the peripheral side 12, the problem of false triggering caused by the fact that the user cannot see the key after wearing the display device does not occur. An optical system for projecting the generated virtual remote control in front of the eyes of the user is provided on the inner side of the front case 10.
A central control unit CPU and a memory are arranged on an internal circuit board of the head-mounted display device as shown in fig. 3. The CPU can be provided with a real object calibration module, a synthesis module, a gesture recognition module, an instruction generation module, a control module and the like. The real object calibration module is used for calibrating the real object image shot by the camera 11 to form a remote controller carrier. The synthesizing module accesses the memory, invokes the virtual control interface stored in the memory, synthesizes the virtual control interface with the calibrated remote controller carrier to form a virtual remote controller, and sends the virtual remote controller to the control module. The control module drives the optical system to project the formed virtual remote controller in front of the eyes of a user so as to facilitate the user to execute remote control operation. Then, the user can use the object grabbed by the hand as a remote controller to remotely control the head-mounted display device.
When the finger of the user moves on the real object, the camera 11 is used for collecting the key actions of the finger of the user on the real object and transmitting the key actions to the gesture recognition module so as to recognize the key positions pressed by the finger. The instruction generation module recognizes a user instruction according to the key position pressed by the finger and sends the user instruction to the control module so as to control the head-mounted display equipment to respond to the user operation and complete the remote control function.
The virtual remote control method of the present embodiment will be described in detail with reference to the head-mounted display device shown in fig. 2 and 3.
Referring to fig. 8, the virtual remote control method of the present embodiment mainly includes the following procedures:
s801, starting a virtual remote control function.
In this embodiment, the virtual remote control function of the head-mounted display device may be turned on by a key manner. Specifically, the control module may be configured to scan the virtual remote controller for the triggered state of the generation key 13. When the control module detects that the virtual remote controller generation key 13 is triggered, the virtual remote control function is started, and the following virtual remote control process is executed.
Of course, other modes of starting the virtual remote control function, such as a voice recognition mode, may be adopted, and the present embodiment is not limited to the above example.
S802, starting the camera 11 to shoot a real object held by a user.
After the virtual remote control function is started, the control module starts the camera 11 to shoot a scene in front of the head-mounted display device. The user can grasp and place the objects accessible by his surrounding tentacles in front of the head-mounted display device for easy acquisition by the camera 11. For example, the user can grasp nearby cola bottles, tissue boxes, even radishes and other foods to the front as a remote control carrier.
S803, calibrating the physical image shot by the camera 11 to form a remote controller carrier.
Because a plurality of keys are arranged on the operation interface of the remote controller, the operation interface has a certain length and a certain width. If the object grabbed by the user is very narrow or very short, the situation that the operation interface of the remote controller is not laid on the object can occur. In order to prevent the situation, the embodiment needs to calibrate the physical size grabbed by the user, and meets the physical layout size requirement of the remote control operation interface, so that the physical layout size requirement can be used as a remote control carrier for the subsequent virtual remote control generation process.
The process of calibrating the physical image shot by the camera 11 in the embodiment is completed in a physical calibration module, and specifically includes the following steps:
identifying the length and width of the object;
judging whether the length-width size of the real object is larger than the length-width size of the virtual control interface according to the length-width size of the virtual control interface stored in the memory in advance;
if the length and width dimensions of the real object exceed those of the virtual control interface, calibrating the image of the real object as a remote controller carrier;
if the length and width dimensions of the real object are smaller than those of the virtual control interface, the real object cannot be used as a remote controller carrier, and prompt information can be sent out to prompt a user to replace the real object.
In this embodiment, the reminding may be performed by projecting the reminding information in front of the eyes of the user through the optical system of the head-mounted display device; the voice broadcasting mode can also be adopted, and the prompting voice is played to the user by using a loudspeaker arranged in the head-mounted display device.
S804, combining the virtual control interface with the remote controller carrier to form a virtual remote controller, and projecting the virtual remote controller to the front of the eyes of a user.
This process may be accomplished in a synthesis module, specifically comprising the following processes:
retrieving a pre-stored virtual control interface from a memory, wherein the virtual control interface can be a key interface, as shown in fig. 4; a knob-type interface is also possible as shown in fig. 5. The interface in various forms can be provided for the user to select; or randomly calling an interface for direct use.
And synthesizing the selected virtual control interface with the generated remote controller carrier image to form a virtual remote controller, such as a cola bottle remote controller shown in fig. 6, and sending the virtual remote controller to the control module.
The control module controls the optical system in the head-mounted display device to project the generated virtual remote controller in front of the eyes of the user so as to facilitate the user to execute subsequent remote control operation.
S805, the camera 11 is used for collecting the key actions of the user finger on the real object, and the key positions pressed by the finger are identified.
After the virtual remote controller is generated, the camera 11 continues to collect the key actions of the user fingers on the real object and transmits the key actions to the gesture recognition module so as to recognize the key positions pressed by the user fingers.
The key action of the user's finger on the real object can be projected in front of the eyes of the user by utilizing the optical system, so that the user can observe the moving position of the user's finger conveniently, and then correct key operation can be made.
S806, identifying the operation of the user according to the key position pressed by the finger.
This process may be accomplished in an instruction generation module.
Specifically, since the size of the virtual control interface, the position of each key on the interface and the function corresponding to each key are all known, only the key position pressed by the finger of the user is identified, which key is triggered by the user can be analyzed, and a corresponding operation instruction can be generated according to the function corresponding to the key.
S807, controlling the head-mounted display device in response to the user operation.
The control module receives the user operation instruction identified by the instruction generation module, and the control system responds to the user operation. For example, program switching, volume adjustment, display of a function menu, and the like are performed.
And S808, ending the virtual remote control process when the object moves out of the shooting range of the camera 11, and eliminating the virtual remote control image projected in front of the eyes of the user.
After the user finishes the remote control operation, the user can put down the grabbed object, so that the object moves out of the range shot by the camera 11. When the real object is not detected by the camera 11, the control module ends the virtual remote control process and controls the optical system to eliminate the virtual remote controller image projected in front of the eyes of the user so as to prevent the visual interference to the program enjoyed in front of the eyes of the user due to the fact that the virtual remote controller image always exists.
It is contemplated that a user may need to operate other electrical devices in the home, such as turning on an air conditioner or the like, while wearing a head mounted display device to play a game or to view other video programs. At this time, the user is required to find a remote controller of another device to perform remote control operation on the device. Under the condition that a user wears the display device, searching things is not easy, and for some eye-shield type head-mounted display devices, the remote controllers of other devices can be searched even if the head-mounted display device is taken off, so that inconvenience is brought to the use of the user.
In order to solve the above problems, the present embodiment further expands the functions of the virtual remote controller, so that the virtual remote controller has the functions of a universal remote controller, and realizes the compatible control of one virtual remote controller on different peripheral devices.
In order to realize the function of the virtual universal remote controller, the present embodiment firstly improves the hardware of the head-mounted display device, and combines with the fig. 2 and 3 to show that the remote control signal transceiver 14 is added on the front shell 10 of the head-mounted display device, and the remote control signal receiving device and the remote control signal transmitting device are added on the internal circuit board of the head-mounted display device, and are respectively connected with the control module on the circuit board.
In terms of software design, as shown in fig. 9, the following process is mainly included:
s901, a remote control signal receiving device on the head-mounted display device is matched with a remote control signal transceiver 14 to receive remote control codes and corresponding function key names sent by remote controllers of other devices.
In this process, the remote controller of the other device can be controlled to emit a remote control signal, such as an infrared signal, by pressing a set key on the remote controller of the other device. The remote control signal receiving device in the head-mounted display equipment receives the remote control signal through the remote control signal transceiver 14, recognizes the remote control code and the corresponding function key name, and associates and stores the remote control code and the corresponding function key name in a memory in the head-mounted display equipment.
S902, generating a virtual remote controller interface for controlling other devices according to the function key names.
This process may be accomplished in the control module. And the control module generates a virtual remote controller interface according to the received function key name and stores the virtual remote controller interface in the memory. In the process, the size of the virtual remote controller interface, the position of each key on the interface and the corresponding function of each key can be obtained, and the command generating module can be directly written.
S903, generating virtual keys related to the other devices on a virtual control interface for controlling the head-mounted display device.
In order to be able to retrieve the virtual remote control interface, virtual keys associated with the other devices may be generated on a virtual manipulation interface of the head mounted display device. For example, if the other device is an air conditioner, a virtual key "air conditioner" may be generated on the virtual manipulation interface of the head-mounted display device. When the user triggers the virtual key of the air conditioner, the system invokes the virtual remote controller interface corresponding to the air conditioner and projects the virtual remote controller interface on the calibrated remote controller carrier to form a virtual remote controller for controlling the air conditioner.
The process can be completed in a control module, which can save the modified virtual control interface to a memory.
S904, starting a virtual remote control function.
This process can be described with reference to the correlation in the above-described process S801.
S905, starting the camera 11 to shoot a real object held by a user.
This process may be described with reference to the correlation in process S802 described above.
S906, calibrating the physical image shot by the camera 11 to form a remote controller carrier.
This process can be described with reference to the correlation in the above-described process S803.
S907, combining the virtual control interface with the remote controller carrier to form a virtual remote controller, and projecting the virtual remote controller to the front of the eyes of the user.
This process may be described with reference to the correlation in process S804 described above.
In this case, virtual buttons related to other devices are formed on the virtual control interface, for example, the virtual button "air conditioner" is provided, as shown in fig. 7.
S908, the camera 11 is used for collecting the key actions of the user finger on the real object, and the key positions pressed by the finger are identified.
This process may be described with reference to the correlation in the above-described process S805.
S909, identifying the operation of a user according to the key position pressed by the finger, and projecting the virtual remote controller interface of other equipment on a remote controller carrier when the user presses the virtual key related to the other equipment so as to form a virtual remote controller for remotely controlling the other equipment.
This process may be accomplished in an instruction generation module.
Specifically, since the size of the virtual control interface, the position of each key on the interface and the function corresponding to each key are all known, only the key position pressed by the finger of the user is identified, which key is triggered by the user can be analyzed, and a corresponding operation instruction can be generated according to the function corresponding to the key.
When the instruction generating module recognizes that the user's finger presses a virtual key related to other equipment (for example, when the user presses an air conditioner key), the control module accesses the memory, invokes a virtual remote controller interface corresponding to the other equipment (for example, invokes an air conditioner remote controller interface), projects the virtual remote controller interface onto a generated remote controller carrier through the optical system, replaces the previous virtual control interface, and forms a virtual remote controller for remotely controlling the other equipment (for example, forms a virtual remote controller for remotely controlling an air conditioner).
S910, recognizing key operation of a user on the virtual remote controller, and transmitting remote control codes corresponding to the keys pressed by the user to other devices by utilizing a remote control signal transmitting device arranged on the head-mounted display device so as to control the other devices.
The camera 11 is used for continuously collecting finger actions of a user on a real object, and a gesture recognition module and an instruction generation module are used for recognizing which key is pressed by the user, and then the function key name corresponding to the key is sent to the control module. The control module retrieves the remote control code corresponding to the function key name in the memory according to the received function key name, and then drives the remote control signal transmitting device on the head-mounted display device to transmit the remote control signal to other devices, such as an air conditioner, in cooperation with the remote control signal transceiver 14, so as to realize remote control operation on the other devices. For example, temperature setting, wind speed and direction adjustment, operation mode switching, and the like are performed.
And S911, ending the virtual remote control process when the object moves out of the shooting range of the camera 11, and eliminating the virtual remote control image projected in front of the eyes of the user.
After the user finishes the remote control operation, the user can put down the grabbed real object, and the system automatically finishes the virtual remote control process.
The virtual remote controller is generated by adopting the method of projecting the virtual control interface on the real object beside the user, so that the control requirement of the head-mounted display equipment can be met, the number of peripheral devices is not increased, and the immersion experience of the user is not interrupted. The function of the universal remote controller is realized on the virtual remote controller, so that a user can finish remote control operation on different electrical equipment without taking off the head-mounted display equipment to find a plurality of remote controllers, and the remote control device is convenient for the user to use.
It should be understood that the above description is not intended to limit the invention to the particular embodiments disclosed, but to limit the invention to the particular embodiments disclosed, and that other variations, modifications, additions and substitutions are possible, without departing from the scope of the invention as disclosed in the accompanying claims.
Claims (10)
1. A virtual remote control method based on a head-mounted display device, comprising the following virtual remote control processes:
shooting a real object held by a user by using a camera arranged on the head-mounted display device;
calibrating the image of the real object to form a remote controller carrier;
synthesizing a virtual control interface and the remote controller carrier to form a virtual remote controller, and projecting the virtual remote controller to the front of the eyes of a user;
acquiring key actions of a user finger on the real object by using the camera, and identifying the key positions pressed by the finger;
and identifying the operation of the user according to the key position pressed by the finger and responding.
2. The head-mounted display device-based virtual remote control method according to claim 1, wherein the process of calibrating the image of the real object comprises:
identifying the length and width dimensions of the object;
judging whether the real object can be used as a remote controller carrier according to the length and width dimensions of the virtual control interface;
if the length and width dimensions of the real object exceed the length and width dimensions of the virtual control interface, calibrating the image of the real object as a remote controller carrier;
and if the length and width dimensions of the real object are smaller than those of the virtual control interface, prompting a user to replace the real object.
3. The head mounted display device based virtual remote control method of claim 1,
configuring a virtual remote controller generation key on the head-mounted display device, and executing the virtual remote control process when the virtual remote controller generation key is triggered;
and when the object moves out of the shooting range of the camera, ending the virtual remote control process, and eliminating the virtual remote control image projected in front of the eyes of the user.
4. A virtual remote control method based on a head-mounted display device according to any one of claim 1 to 3,
receiving remote control codes and corresponding function key names sent by remote controllers of other devices by utilizing a remote control signal receiving device arranged on the head-mounted display device;
generating a virtual remote controller interface for controlling the other equipment according to the function key name;
projecting the virtual remote controller interface on the remote controller carrier to form a virtual remote controller for remotely controlling the other equipment;
and identifying key operation of a user on the virtual remote controller, and transmitting remote control codes corresponding to the keys pressed by the user to the other equipment by utilizing a remote control signal transmitting device arranged on the head-mounted display equipment so as to control the other equipment.
5. The method for virtual remote control based on a head mounted display device as claimed in claim 4, wherein,
generating virtual keys related to the other devices on a virtual control interface for controlling the head-mounted display device;
and when the virtual key is identified to be pressed by the finger of the user, projecting a virtual remote controller interface of the other equipment on the remote controller carrier to form a virtual remote controller for remotely controlling the other equipment.
6. A head-mounted display device, comprising:
the camera is used for shooting a real object held by a user;
the real object calibration module is used for calibrating the shot real object image to form a remote controller carrier;
the synthesis module is used for synthesizing the virtual control interface and the remote controller carrier to form a virtual remote controller;
an optical system for projecting the virtual remote control in front of the eyes of a user;
the gesture recognition module is used for recognizing the key position pressed by the finger according to the key action of the user finger on the real object acquired by the camera;
the instruction generation module is used for identifying a user instruction according to the key position pressed by the finger;
and the control module is used for controlling the head-mounted display device to respond to the user operation according to the user instruction.
7. The head-mounted display device according to claim 6, wherein the real object calibration module identifies the length and width dimensions of the real object according to the real object image shot by the camera, and if the length and width dimensions of the real object exceed the length and width dimensions of the virtual control interface, the real object image is calibrated as a remote controller carrier.
8. The head mounted display device of claim 7, further comprising:
a speaker for playing an audio signal;
and when the physical calibration module judges that the length and width of the physical object is smaller than the length and width of the virtual control interface, the physical calibration module controls the loudspeaker to play prompt information to remind a user of changing the physical object.
9. The head mounted display device of claim 6, further comprising:
the virtual remote controller generates keys which are distributed on the peripheral side surface of the head-mounted display device and are electrically connected with the control module;
when the control module detects that the key generated by the virtual remote controller is triggered, the camera is started to shoot a real object held by a user, and a virtual remote control process is started; and when the real object is not detected by the camera, the control module ends the virtual remote control process and controls the optical system to eliminate the virtual remote control image projected in front of the eyes of the user.
10. The head mounted display device of any one of claims 6 to 9, further comprising:
the remote control signal receiving device is used for receiving remote control codes and corresponding function key names sent by remote controllers of other devices;
the memory is used for storing the remote control codes and the corresponding function key names in an associated mode;
remote control signal transmitting means for transmitting a remote control signal to the other device;
the control module generates a virtual remote controller interface for controlling the other equipment according to the function key name received by the remote control signal receiving device and stores the virtual remote controller interface into the memory; the control module is used for adding virtual keys related to other devices on a virtual control interface for controlling the head-mounted display device, projecting virtual remote controller interfaces of the other devices on the remote controller carrier when the gesture recognition module detects that the virtual keys are pressed, so as to form a virtual remote controller for remotely controlling the other devices, recognizing key operation of a user on the virtual remote controller through the gesture recognition module, further sending remote control codes corresponding to the keys pressed by the user to the remote control signal emitting device, and sending the remote control codes to the other devices through the remote control signal emitting device to control the other devices.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310101835.5A CN116048276A (en) | 2023-02-09 | 2023-02-09 | Virtual remote control method based on head-mounted display equipment and head-mounted display equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310101835.5A CN116048276A (en) | 2023-02-09 | 2023-02-09 | Virtual remote control method based on head-mounted display equipment and head-mounted display equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116048276A true CN116048276A (en) | 2023-05-02 |
Family
ID=86131382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310101835.5A Pending CN116048276A (en) | 2023-02-09 | 2023-02-09 | Virtual remote control method based on head-mounted display equipment and head-mounted display equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116048276A (en) |
-
2023
- 2023-02-09 CN CN202310101835.5A patent/CN116048276A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6229096B2 (en) | Smart device operation control method, apparatus, program, and storage medium | |
| US7721207B2 (en) | Camera based control | |
| US20100302357A1 (en) | Gesture-based remote control system | |
| TWI491224B (en) | Control apparatus, remote control apparatus and method capable of controlling interface provided by tv | |
| EP2237131A1 (en) | Gesture-based remote control system | |
| CN111580661A (en) | Interaction method and augmented reality device | |
| CN111970456B (en) | Shooting control method, device, equipment and storage medium | |
| WO2018120768A1 (en) | Remote control method and terminal | |
| EP2803188B1 (en) | Features for use with a multi-sided controlling device | |
| JP2007286677A (en) | Character input program and character input device | |
| CN109164908B (en) | Interface control method and mobile terminal | |
| WO2014155885A1 (en) | Input device | |
| CN109597544A (en) | Input exchange method, device, equipment and storage medium | |
| CN116048276A (en) | Virtual remote control method based on head-mounted display equipment and head-mounted display equipment | |
| TW201138437A (en) | System and method for remotely switching TV channels | |
| JP2009064364A (en) | Program, storage medium and computer | |
| JP2014526183A (en) | Sensor means for a television receiver. | |
| CN105788231B (en) | Remotely controlled method, apparatus and terminal electronic device | |
| US11449451B2 (en) | Information processing device, information processing method, and recording medium | |
| TW201001931A (en) | Remote controller and operating method thereof | |
| US20200147483A1 (en) | Interactive gaming system | |
| CN112316410A (en) | Control method, system, equipment and readable storage medium based on television game | |
| TWI816447B (en) | Method for pairing devices, electronic system, and computer readable storage medium | |
| JP2010283549A (en) | Gesture-based remote control system | |
| TWI762178B (en) | Remote control method and control system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |