CN114779924A - Head-mounted display device, method for controlling household device and storage medium - Google Patents

Abstract

The disclosure relates to a head-mounted display device, a method for controlling a household device and a storage medium. The method for controlling the household equipment comprises the following steps: determining target household equipment in at least one household equipment, wherein the target household equipment is the household equipment concerned by a wearer of the head-mounted display equipment; projecting a virtual control interface corresponding to the target household equipment through the head-mounted display equipment; detecting a hand motion of the wearer; determining the operation of the wearer on the virtual control interface according to the hand action of the wearer, and controlling the target household equipment according to the operation of the wearer on the virtual control interface.

Description

Head-mounted display device, method for controlling household device and storage medium

Technical Field

The disclosure relates to an intelligent electronic device, and in particular, to a head-mounted display device, a method of controlling a home device, and a storage medium.

Background art

The head-mounted display device gradually becomes a common article for daily life of people, but the current head-mounted display device is difficult to adapt to different application scenes, and particularly is difficult to be jointly applied with the intelligent household device, so that the head-mounted display device which can be jointly applied with the intelligent household device and is suitable for various application scenes is needed to be provided.

Disclosure of Invention

The embodiment of the disclosure provides a head-mounted display device, a method for controlling a household device and a storage medium, so that the head-mounted display device and the intelligent household device can be jointly applied.

According to a first aspect of the present disclosure, there is provided a method for controlling a household device based on a head-mounted display device, including: determining target household equipment in at least one household equipment, wherein the target household equipment is the household equipment concerned by a wearer wearing the head-mounted display equipment; projecting a virtual control interface corresponding to the target household equipment through the head-mounted display equipment; detecting a hand movement of the wearer; determining the operation of the wearer on the virtual control interface according to the hand action of the wearer, and controlling the target household equipment according to the operation of the wearer on the virtual control interface.

According to a second aspect of the present disclosure, there is provided a head-mounted display device, the head-mounted display device comprising a memory, a processor and a control household device program stored on the memory and executable on the processor, the control household device program being configured to implement the steps of the method for controlling household devices according to any one of the first aspect of the present disclosure.

According to a third aspect of the present disclosure, there is provided a storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of controlling home equipment according to any one of the first aspect of the present disclosure.

The beneficial effects of the embodiment of the disclosure are that the head-mounted display device can determine the target household device concerned by the wearer by oneself, and then switches the control object of the head-mounted display device, so that the head-mounted display device can be suitable for various application scenes, and the requirements of the user can be better met. Moreover, the head-mounted display device can project a virtual control interface corresponding to the target household device, so that the wearer can conveniently control the target household device.

Other features of embodiments of the present disclosure and advantages thereof will become apparent from the following detailed description of exemplary embodiments thereof, which is to be read in connection with the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the embodiments of the disclosure.

Fig. 1 shows a flowchart of a method for controlling a home device based on a head-mounted display device according to an embodiment of the present disclosure;

2-5 illustrate schematic diagrams of examples of head mounted display devices according to embodiments of the present disclosure;

fig. 6 illustrates a block diagram of a head mounted display device of an embodiment of the present disclosure.

Detailed Description

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless it is specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

In all examples shown and discussed herein, any particular value should be construed as exemplary only and not as limiting. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

The embodiment of the application discloses a head-mounted display device which can be jointly applied to household equipment.

In this embodiment, the home device may be a multimedia device such as a smart television, a projector, a tablet computer, and a game machine, or an intelligent home device such as an air conditioner, an oven, and a washing machine. The head-mounted display device can be in communication connection with the household equipment, so that information transmission can be performed between the head-mounted display device and the household equipment. The communication connection can be realized through wireless communication modes such as bluetooth, WiFi, (Ultra Wide Band ) and the like, and can also be realized through a wired communication mode.

In the embodiments of the present disclosure, the head-mounted display device may be a head-mounted display device with an augmented reality function, that is, the head-mounted display device is provided with a light engine having the capability of projecting a virtual interface in front of the eyes of the wearer. Head-mounted display devices are capable of directing images through optical techniques to the retina of a wearer to present a "vision" to project the images into the visual space of the wearer. The head mounted display device may be smart glasses.

In the embodiment of the present disclosure, at least one acoustic module may be disposed in the head-mounted display device, and the audio may be played through the acoustic module. The acoustic module may include a speaker or an earphone.

In the embodiment of the disclosure, at least one microphone may be disposed in the head-mounted display device to collect sound information around the head-mounted display device. The head-mounted display device can have a voice recognition function, and a user can control the head-mounted display device in a voice mode.

In one example of the present disclosure, a head mounted display device includes a first detection module. The first detection module is used for determining household equipment concerned by a wearer. The first detection module may include one or more of a UWB module, a camera, a laser sensor, an infrared sensor, an ultrasonic sensor, an acceleration sensor, and a gyro sensor. In one example of the present disclosure, in order to support the function of the first detection module, an auxiliary device may be provided in the home appliance in the room. For example, in a room or a home appliance, one or more of a UWB module, a laser sensor, an infrared sensor, and an ultrasonic sensor are provided and used as an auxiliary device.

In one example of the present disclosure, information of each room, for example, a size, a shape, an image, and the like of each room are stored in advance in the head-mounted display device.

In one example of the disclosure, the head-mounted display device stores information of the home devices in advance, wherein the information at least comprises identification of the home devices and function information of the home devices. The information of the home devices stored in the head-mounted display device may further include image information of multimedia, information of a room to which the home devices belong, information of positions of the home devices in the room to which the home devices belong, and information of identifiers of auxiliary devices in the home devices.

In one example of the present disclosure, information of the auxiliary device is stored in advance in the head-mounted display apparatus. For example, the head-mounted display apparatus stores the identifier of the auxiliary device in advance, the signal sent by the auxiliary device to the head-mounted display apparatus contains the identifier of the auxiliary device, and the head-mounted display apparatus can distinguish the signals sent by the auxiliary devices by the identifier. For example, position information of the auxiliary device is stored in the head-mounted display apparatus in advance.

In one example, the first detection module includes a UWB module, the UWB module is also mounted in the home device, and the position of the head-mounted display device relative to the home device can be measured by a UWB positioning method. For example, a UWB module in a home device transmits a UWB signal, a UWB module in a head-mounted display device receives the UWB signal, and a distance and a direction of the head-mounted display device with respect to the home device are determined based on the UWB signal. For example, a UWB module in the head-mounted display device transmits a first UWB signal in a first direction (for example, to the front of the head-mounted display device), and assuming that the first home device receives the first UWB signal, the first home device sends back a second UWB signal in response, and after receiving the second UWB signal, the head-mounted display device acquires an identifier of the auxiliary device from the second UWB signal, so that it can be determined that the first home device is located in the first direction of the head-mounted display device.

In one example, the first detection module comprises a UWB module, and the UWB modules are respectively arranged at least three different positions in the room, so that the position of the head-mounted display device in the room can be accurately determined. For example, a plurality of UWB modules in a room emit UWB signals, and the UWB module in the head mounted display device receives these UWB signals, and determines the position of the head mounted display device in the room based on triangulation based on positional information of the UWB modules in the room stored in advance.

In one example, the first detection module includes a UWB module, and at least 3 pieces of household equipment are respectively provided with the UWB module in a room, so that the relative positions of the head-mounted display equipment and the at least 3 pieces of household equipment can be accurately determined. For example, UWB modules of at least 3 household devices in a room respectively emit UWB signals, and a UWB module in the head-mounted display device receives the UWB signals, and determines the relative position of the head-mounted display device with respect to the at least 3 household devices based on a triangulation method.

In one example, the first detection module includes a camera, and the image captured by the camera is compared with a pre-stored room image and a pre-stored home device image to identify, so as to determine a room where the wearer is currently located, a position of the wearer in the room, whether a home device is present in front of the wearer, and which home device is present.

In one example, the first detection module includes an acceleration sensor or a gyroscope sensor, such as a three-axis gyroscope, a six-axis gyroscope, a nine-axis gyroscope, an accelerometer, an angle sensor, etc., which can determine the current position of the wearer by detecting the movement of the wearer.

In one example, the first detection module includes a positioning module, such as a GPS positioning module, an inertial navigation positioning module, or the like, by which the current position of the wearer can be determined.

In one example, the first detection module includes one or more of a laser sensor, an infrared sensor, and an ultrasonic sensor. Based on laser sensor/infrared sensor/ultrasonic sensor, can detect the house equipment in front of wearing display device, also can detect the information in room. For example, the first detection module includes a laser sensor, the laser sensor emits laser in multiple directions, and measures a distance according to the laser reflected from each direction, so as to obtain data of a current room where the wearer is located, and the data is compared with pre-stored room information, so that the room where the wearer is located can be determined. For example, the head-mounted display device sends an infrared signal to the front, an infrared module in the home equipment sends a feedback signal after receiving the infrared signal, and the head-mounted display device may determine the home equipment located in front of the head-mounted display device according to the feedback signal.

In one example of the present disclosure, a head mounted display device includes a second detection module. The second detection module is a camera.

In one example, the second detection module comprises only one camera, and based on an image shot by the camera, the home equipment within the field of view of the camera can be determined.

In one example, the second detection module includes a first camera disposed to the left of the head mounted display device and a second camera disposed to the right of the head mounted display device. The field range of the first camera is a first field range, the field range of the second camera is a second field range, and the overlapping area of the first field range and the second field range is a current focus area. Based on the images shot by the first camera and the second camera, the household equipment in the current focus area can be determined.

In one example of the present disclosure, a head mounted display device includes providing a third detection module. The third detection module is used for assisting in determining the target household equipment. The third detection module may include one or more of a laser sensor, an infrared sensor, and an ultrasonic sensor.

In one example of the present disclosure, a head mounted display device includes a fourth detection module. The fourth detection module can be used for detecting the hand motion of the wearer. The fourth detection module may include one or more of a camera, a laser sensor, an infrared sensor, an ultrasonic sensor, and a millimeter wave sensor. For example, a camera may be positioned on a frame of the head mounted display device to capture images in front of the head mounted display device, and analysis of the images may determine the gestures and hand positions of the wearer. For example, a millimeter wave sensor is arranged on a frame of the head-mounted display device, and the position of the hand of the wearer can be determined according to sensing data.

In one example of the disclosure, the head-mounted display device and the home device are pre-bound, the head-mounted display device stores therein an identifier, function information, and the like of the home device, and the home device allows the head-mounted display device to control the home device.

The embodiment of the application provides a method for controlling household equipment based on a head-mounted display device, and as shown in fig. 1, the method comprises steps S11-S14.

Step S11, determining target household equipment in at least one household equipment, wherein the target household equipment is the household equipment concerned by a wearer wearing the head-mounted display equipment.

And step S12, projecting a virtual control interface corresponding to the target household equipment through the head-mounted display equipment.

In one example of the present disclosure, projecting a virtual control interface corresponding to a target home device includes: acquiring a function list corresponding to target household equipment, wherein the function list comprises at least one function item; and projecting a virtual control interface corresponding to the target household equipment according to a preset size at a preset distance right in front of the head-mounted display equipment, wherein the virtual control interface is provided with controls corresponding to the function items one to one.

In one example, the predetermined distance may be a distance that is convenient for the wearer to handle, such as between 15 cm and 30 cm. In one example, the preset distance may be set by the wearer.

In one example, the predetermined size may be a size of an area that is conveniently manipulated by the wearer, for example, the predetermined size is 30 cm long by 25 cm wide. In one example, the preset size may be set by the wearer.

In one example, the preset distance is adapted to the preset size, and the larger the size of the virtual control interface is, the farther the preset distance is relatively. For example, the predetermined distance is 100 cm, and the size of the virtual control interface is 100 cm in length and 60 cm in width. For example, the predetermined distance is 30 cm, and the virtual control interface has a length of 30 cm and a width of 15 cm.

In one example, the virtual control interface is rectangular. In one example, the shape of the virtual control interface may also be set by the wearer.

Step S13 detects the movement of the hand of the wearer.

In one example, the hand motion of the wearer is detected within a target area, which may be an area directly in front of the head mounted display device.

In one example of the present disclosure, the head-mounted display device further includes a fourth detection module, by which a hand motion of the wearer is detected within the target area. In one example, the hand motion of the wearer is detected by the fourth detection module within a target area, which is an area directly in front of the head mounted display device. The fourth detection module may include one or more of a millimeter wave sensor, an ultrasonic sensor, a camera, a laser sensor, and an infrared sensor. For example, a camera is positioned on the frame of the head mounted display device to capture images in front of the head mounted display device, and analyzing the images may determine the gestures and hand positions of the wearer. For example, a millimeter wave sensor is arranged on a frame of the head-mounted display device, and the position of the hand of the wearer can be determined according to sensing data. In one example, detecting the hand motion of the wearer may be detecting a position to which a finger of the wearer moves, and determining the operation of the virtual interface by the wearer based on the position to which the finger of the wearer moves. The person skilled in the art can flexibly set the hand action according to the actual need, for example, a click action, a drag action, etc.

Step S14, determining the operation of the wearer on the virtual control interface according to the hand motion of the wearer, and controlling the target household equipment according to the operation of the wearer on the virtual control interface.

For example, as shown in fig. 2, a user is wearing a head mounted display device and the user is focusing on a smart air conditioner that is determined to be a target home device. And acquiring a function list corresponding to the intelligent air conditioner, wherein the function list comprises 2 function items, namely, the temperature is adjusted to be high and the temperature is adjusted to be low. And projecting a virtual control interface corresponding to the intelligent air conditioner in front of the user, wherein the right area of the virtual control interface comprises 2 controls in the form of buttons, the upper button is used for increasing the set temperature of the intelligent air conditioner, the lower button is used for decreasing the set temperature of the intelligent air conditioner, and the middle area of the virtual control interface is used for displaying the current set temperature of the intelligent air conditioner. And when the user clicks the button on the virtual control interface according to the hand action of the user, controlling the intelligent air conditioner to increase the set temperature.

In one example of the disclosure, the head-mounted display device may be in bluetooth, WIFI, UWB communication connection with the target household device for information transmission between the two. Through the transmission of information, the head-mounted display device can control the target household device.

In an example of the disclosure, the head-mounted display device may determine a target home device that a wearer pays attention to by himself, and then switch a control object of the head-mounted display device, so that the head-mounted display device may be applicable to various application scenarios, and better meet the requirements of the user. Moreover, the head-mounted display device can project a virtual control interface corresponding to the target household device, so that the wearer can conveniently control the target household device.

In the disclosed embodiments, the rear of the head mounted display device is the direction toward the wearer, and the front of the head mounted display device is the opposite direction to the rear of the head mounted display device. The front of the head-mounted display device in the embodiment of the present disclosure means in front of the head-mounted display device and within a preset section in the left-right direction, for example, within a section from 60 degrees to the right of the head-mounted display device in front of the head-mounted display device.

In one example of the present disclosure, determining a target home device among at least one home device includes: the target household equipment is determined in at least one household equipment through a first detection module arranged in the head-mounted display equipment. The first detection module includes at least any one of: UWB module, camera, laser sensor, infrared sensor, ultrasonic sensor, acceleration sensor, gyroscope.

In this example, the head-mounted display device can determine the household device concerned by the wearer through the first detection module, and then switch the sound source of the head-mounted display device, so that the interactive experience is improved, and the requirements of the customer are better met.

In one example of the present disclosure, determining a target home device among at least one home device includes: and detecting the relative position between the head-mounted display equipment and the household equipment. And determining the household equipment positioned right in front of the head-mounted display equipment according to the relative position between the head-mounted display equipment and the household equipment. And determining the household equipment right in front of the head-mounted display equipment as target household equipment.

For example, as shown in fig. 3, two home devices, namely, an intelligent television 1 and an intelligent television 2, are in a room 1, a user wears a head-mounted display device, and when it is detected that the home device right in front of the head-mounted display device is the intelligent television 1, the intelligent television 1 is determined as a target home device.

In this example, the head-mounted display device can determine the home equipment concerned by the wearer through the relative position between the head-mounted display device and the home equipment, and then switches the control object of the head-mounted display device, so that the interactive experience is improved, and the requirements of the customer are better met.

In one example of the present disclosure, determining a target home device among at least one home device includes: the method comprises the steps of detecting a room where the head-mounted display device is located at present and detecting the relative position between the head-mounted display device and the household device. And taking the household equipment in the room where the head-mounted display equipment is currently located as candidate household equipment. And under the condition that the number of the candidate household equipment is one, determining the candidate household equipment as target household equipment. And under the condition that the number of the candidate household devices is multiple, determining the candidate household devices which are positioned right in front of the head-mounted display device in the multiple candidate household devices according to the relative positions of the head-mounted display device and the candidate household devices, and determining the candidate household devices which are positioned right in front of the head-mounted display device as the target household devices.

In one example, as shown in fig. 4, a home device is located in the room 2, and the home device is a smart television 3. The user wears the head-mounted display device, detects that a home device (intelligent television 3) is in a room where the user is located, and directly determines the intelligent television 3 as a target home device by taking the detected intelligent television 3 as a candidate home device because the number of the candidate home devices is one.

In another example, as shown in fig. 3, there are two home devices in a room 1, which are a smart television 1 and a smart television 2, respectively, a user wears a head-mounted display device, detects that there are 2 home devices (i.e., the smart television 1 and the smart television 2) in the room where the wearer is located, takes the 2 home devices as candidate home devices, and at the same time, detects the relative positions of the wearer and the candidate home devices, determines, according to the detected relative positions, that the candidate home device located right in front of the wearer is the smart television 1, and determines the smart television 1 as a target home device.

In this example, the target home device may be determined in different manners according to the number of home devices in the room where the head-mounted display device is located, so that the head-mounted display device controls the target home device.

In one example of the present disclosure, determining a target home device among at least one home device includes: and performing auxiliary positioning through a second detection module arranged in the head-mounted display equipment to determine the target household equipment. That is to say, in the embodiment of the present disclosure, the target home equipment may be determined in at least one home equipment by combining the detection results of the first detection module and the second detection module.

In one example of the disclosure, the first detection module is a UWB module, the second detection module is a camera, and the target home device is determined in at least one home device by combining detection results of the first detection module and the second detection module.

In one example, the second detection module includes only one camera, and the current focus area is the field of view range of the camera. Based on the image shot by the camera, the household equipment in the field range of the camera can be determined. For example, the camera of the second detection module is located at the center of the frame of the head-mounted display device, the UWB module determines that a plurality of home devices are located in front of the wearer, and the home device located at the center of the image among the home devices is determined as the target home device in combination with the image captured by the second detection module. For example, the camera of the second detection module is located at the center of the frame of the head-mounted display device, a plurality of household devices in front of the wearer are determined according to the shot images, and the household device closest to the camera is determined as the target household device according to the distance between the head-mounted display device and the household devices measured by the UWB module of the first detection module.

In one example, the second detection module comprises a first camera and a second camera, the field of view range of the first camera is a first field of view range, and the field of view range of the second camera is a second field of view range; the current focus area is located within an overlap region of the first field of view range and the second field of view range. Based on the images shot by the first camera and the second camera, the household equipment in the current focus area can be determined. For example, a plurality of household devices in front of the wearer are determined through the UWB module, and if only one household device is determined to be located in the current focus area in combination with the image captured by the second detection module, the household device located in the current focus area is determined to be the target household device.

In the embodiment of the present disclosure, the head-mounted display device further includes a third detection module, and may further determine, by combining detection results of the first detection module, the second detection module, and the third detection module, that the target device is determined in at least one household device. The third detection module can be one or more of a laser sensor, an infrared sensor and an ultrasonic sensor.

In one example of the present disclosure, the first detection module is a UWB module and the second detection module is a camera. Determining a target home device in at least one home device includes: judging whether the current focus area comprises two or more pieces of household equipment; if yes, triggering a third detection module arranged in the head-mounted display equipment to acquire auxiliary information; and determining the target household equipment from two or more household equipment by combining the first detection module, the second detection module and the third detection module.

For example, a plurality of household devices (including a first household device) located in front of the wearer are determined by the UWB module, and when two or more household devices of the plurality of household devices are determined to be located in the current focus area in combination with the image captured by the second detection module, the third detection module is triggered to emit a laser signal. The head of the wearer rotates to enable the laser signal emitted by the third detection module to face the desired household equipment, if the wearer wants to use the first household equipment as the target household equipment, the wearer rotates the head to send the first laser signal to the first household equipment, the first household equipment receives the first laser signal and then responds to the first laser signal to send back the second laser signal, the head-mounted display equipment receives the second laser signal and then acquires the identifier of the first household equipment from the second laser signal, and the first household equipment is determined to be the target household equipment.

In one example, triggering the third detection module disposed in the head-mounted display device to acquire the auxiliary information may be an automatic trigger, that is, when it is determined that the current focus area includes two or more home devices, the third detection module is automatically triggered to acquire the auxiliary information.

In an example, the triggering of the third detection module arranged in the head-mounted display device to acquire the auxiliary information may be voice triggering, that is, when it is determined that the current focus area includes two or more home devices, the speaker of the head-mounted display device is controlled to emit a voice prompt to inquire whether a user starts the third detection module to acquire the auxiliary information, if the voice reply of the user is collected through the microphone of the head-mounted display device to be "start", the third detection module is triggered to acquire the auxiliary information, and a target home device is determined from the two or more home devices by combining the first detection module, the second detection module, and the third detection module.

In one example of the present disclosure, the head mounted display device further includes an acoustic module for playing audio. The acoustic module includes a speaker and/or an earphone. The method further comprises the following steps: and setting a sound source of the acoustic module of the head-mounted display device as target household equipment.

In one example of the disclosure, the head-mounted display device may be in bluetooth, WIFI, UWB communication connection with the target household device for information transmission between the two. Through information transmission, the head-mounted display equipment can receive audio information of the target household equipment, and a sound source of an acoustic module of the head-mounted display equipment is set as the target household equipment.

In one example of the disclosure, when the sound source of the acoustic module is set as the target household device, the loudspeaker of the target household device is also controlled to be turned off. That is to say, change target house equipment into silence, played the audio frequency by head mounted display device to save the resource, reduced the sound interference, promoted user's experience.

In one example of the disclosure, setting an audio source of a head-mounted display device as a target home device includes: and setting a sound source of the earphone of the head-mounted display device as target household equipment.

In one example of the disclosure, when the sound source of the headset of the head-mounted display device is set as the target home device, the speaker of the target home device is also turned off.

For example, a user wears the head-mounted display device in the present application, and when it is detected that the wearer pays attention to a certain home device, the home device is determined as a target home device, and is in communication connection with the home device to acquire a sound source. And closing the loudspeaker of the target household equipment, and controlling the earphone of the head-mounted display equipment to play the obtained sound source. Through the mode, a private playing mode can be realized, and the sound of the target household equipment is played through the earphone of the head-mounted display equipment. In another example, a plurality of users may wear the head-mounted display devices respectively, and the earphones of the plurality of head-mounted display devices all play the sound source of the home device, so that each head-mounted display device can perform personalized setting when playing the sound source of the home device, for example, adjusting the volume, even selecting different audio tracks, setting different languages for playing, and the like.

In this example, by turning off the speaker of the home device and playing the sound source of the home device in the headset wearing the display device, the headset wearing the display device can play the audio of the home device concerned by the user, so that the home device can be played in a private manner, and the method is suitable for scenes needing private playing, improves user experience, and meets user requirements.

In an example of the present disclosure, the setting of the sound source of the acoustic module as the target home device includes: under the condition that the target household equipment is changed from first household equipment to second household equipment, a sound source of the acoustic module is switched from the first household equipment to the second household equipment, wherein the first household equipment and the second household equipment are multimedia equipment.

For example, as shown in fig. 5, two home devices, namely, a smart television 4 and a smart television 5, are provided in a room 3, and a user wears a head-mounted display device, and when the wearer is at a location a, it is detected that the home device concerned by the wearer is the smart television 4, that is, the target home device is the smart television 4, and at this time, a sound source of an acoustic module of the head-mounted display device is the smart television 4. When the wearer moves from the position A to the position B, it is detected that the position of the wearer changes, and the household device concerned by the wearer is changed from the intelligent television 4 to the intelligent television 5, that is, the target household device is changed from the intelligent television 4 to the intelligent television 5, and then the sound source of the acoustic module of the head-mounted display device is switched to the intelligent television 5.

In this example, the sound source of the head-mounted display device is switched in time when the household device that the user pays attention to changes, that is, the head-mounted display device can switch the sound source according to the requirements of the user, so that the head-mounted display device can be suitable for various application scenes, and the requirements of the user can be better met.

In one example of the present disclosure, the method further comprises: and under the condition that the target household equipment is changed from the first household equipment to the second household equipment, controlling the first household equipment to stop playing.

For example, when the home device concerned by the wearer wearing the display device is changed from a first home device to a second home device, the sound source of the acoustic module of the head-mounted display device is switched from the first home device to the second home device, and the first home device is controlled to stop playing.

In this example, the head-mounted display device may automatically turn off the multimedia playing work of the home equipment before the user changes attention under the condition that the home equipment that the user pays attention to changes, thereby saving resources, reducing user operations, reducing interference, and improving user experience.

In one example of the present disclosure, switching an audio source of an acoustic module from a first household device to a second household device includes: the method includes the steps that playing information of first household equipment is obtained, and the playing information of the first household equipment comprises identification of multimedia content played by the first household equipment and a playing progress corresponding to the multimedia content. And controlling the second household equipment to continue playing the multimedia content according to the playing information, and switching the sound source of the acoustic module from the first household equipment to the second household equipment.

In an example of the present disclosure, the head-mounted display device may acquire, through communication connection with the first home device, playing information of the first home device, for example, a name of playing content, a progress of playing content, and the like. When the sound source of the acoustic module of the head-mounted display device is switched from the first home device to the second home device, the second home device can be controlled to continue playing the playing content according to the previously acquired playing information.

Taking a scene that may be implemented by this embodiment as an example, for example, when a wearer wearing the head-mounted display device watches a television program in a bedroom through a television in the bedroom, the head-mounted display device may obtain playing information of the television program, and when the wearer moves to the living room to continue watching, that is, when the wearer pays attention to the television in the living room, the television in the living room is controlled to continue playing contents played by the television in the bedroom. In one example, the head-mounted display device can also control the television in the bedroom to stop playing, so as to save resources.

In this example, the head-mounted display device can automatically enable the household device after the attention is changed to continuously play the content played by the household device before the attention is changed under the condition that the household device concerned by the user is changed, so that the user operation is simplified, and the use experience of the user is improved.

The embodiment of the application provides a head-mounted display device, the head-mounted display device comprises a memory, a processor and a control household device program which is stored in the memory and can run on the processor, the control household device program is configured to realize the steps of the method for controlling the household devices, the steps can achieve the same technical effects, and the repeated description is omitted here for avoiding repetition.

Referring to fig. 6, an embodiment of the present application provides a head-mounted display device, where the head-mounted display device includes a memory, a processor, an acoustic module, an optical engine, a first detection module, a second detection module, a third detection module, and a fourth detection module. The acoustic module includes a speaker and/or an earphone. The optical machine is used for projecting a virtual control interface. The functions of the first detection module, the second detection module, the third detection module and the fourth detection module are as described in the foregoing. The head-mounted display device further comprises a home control device program which is stored in the memory and can run on the processor, the home control device program is configured to implement the steps of any one of the methods for controlling home devices described above, and the same technical effects can be achieved.

The embodiment of the present application provides a storage medium, on which a program or an instruction is stored, where the program or the instruction is executed by a processor to implement the steps of the method for controlling a home device, and the same technical effects can be achieved, and are not described herein again to avoid repetition.

The embodiments in the disclosure are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the control method and the storage medium embodiment, since they are basically similar to the apparatus embodiment, the description is relatively simple, and the relevant points can be referred to the partial description of the method embodiment.

The foregoing description of specific embodiments of the present disclosure has been described. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Embodiments of the present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied therewith for causing a processor to implement aspects of embodiments of the disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

Computer program instructions for carrying out operations for embodiments of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry may execute computer-readable program instructions to implement aspects of embodiments of the present disclosure by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Aspects of embodiments of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or improvements to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (13)

1. A method for controlling household equipment based on a head-mounted display device is characterized by comprising the following steps:

determining target household equipment in at least one household equipment, wherein the target household equipment is the household equipment concerned by a wearer wearing the head-mounted display equipment;

projecting a virtual control interface corresponding to the target household equipment through the head-mounted display equipment;

detecting a hand movement of the wearer;

the method comprises the steps of determining the operation of a wearer on the virtual control interface according to the hand action of the wearer, and controlling the target household equipment according to the operation of the wearer on the virtual control interface.

2. The method according to claim 1, wherein the determining a target home device among the at least one home device comprises: determining target household equipment in at least one household equipment through a first detection module arranged in the head-mounted display equipment;

the first detection module at least comprises any one of the following: UWB module, camera, laser sensor, infrared sensor, ultrasonic sensor, acceleration sensor, gyroscope.

3. The method according to claim 2, wherein the determining the target home device in the at least one home device comprises:

detecting a relative position between the head-mounted display device and the household device;

determining the household equipment right in front of the head-mounted display equipment according to the relative position between the head-mounted display equipment and the household equipment;

and determining the household equipment right in front of the head-mounted display equipment as the target household equipment.

4. The method according to claim 2, wherein the determining a target home device among the at least one home device comprises:

detecting a room where the head-mounted display equipment is located currently and a relative position between the head-mounted display equipment and the household equipment;

taking the household equipment in the room where the head-mounted display equipment is currently located as candidate household equipment;

determining the candidate household equipment as target household equipment under the condition that the number of the candidate household equipment is one;

determining candidate household equipment which is located right in front of the head-mounted display equipment in the plurality of candidate household equipment according to the relative position between the head-mounted display equipment and the candidate household equipment under the condition that the number of the candidate household equipment is multiple; and determining candidate household equipment which is positioned right in front of the head-mounted display equipment as the target household equipment.

5. The method according to any one of claims 2 to 4, wherein the determining the target household device in the at least one household device comprises: and performing auxiliary positioning through a second detection module arranged in the head-mounted display equipment to determine the target household equipment.

6. The method of claim 5, wherein the first detection module is a UWB module and the second detection module is a camera;

the determining the target household equipment in the at least one household equipment comprises the following steps:

judging whether the current focus area comprises two or more pieces of household equipment;

if yes, triggering a third detection module arranged in the head-mounted display equipment to acquire auxiliary information;

and determining the target household equipment from the two or more household equipment by combining the first detection module, the second detection module and the third detection module.

7. The method of claim 6, wherein the second detection module comprises a first camera and a second camera, wherein the field of view range of the first camera is a first field of view range and the field of view range of the second camera is a second field of view range;

the current focus area is located within an overlap area of the first field of view range and the second field of view range.

8. The method according to claim 2, wherein the projecting a virtual control interface corresponding to the target household device comprises:

acquiring a function list corresponding to the target household equipment, wherein the function list comprises at least one function item;

and projecting a virtual control interface corresponding to the target household equipment at a preset distance right in front of the head-mounted display equipment according to a preset size, wherein the virtual control interface is provided with controls corresponding to the function items one to one.

9. A head-mounted display device comprising a memory, a processor and a control home device program stored on the memory and executable on the processor, the control home device program being configured to implement the steps of the method of controlling home devices as claimed in any one of claims 1 to 8.

10. The head mounted display device of claim 9, wherein the head mounted display device further comprises an acoustic module;

the acoustic module is used for playing audio;

the control home device program is configured to set the sound source of the acoustic module as the target home device.

11. The head-mounted display device of claim 10, wherein the setting of the audio source of the acoustic module as the target household device comprises:

and under the condition that the target household equipment is changed from first household equipment to second household equipment, switching a sound source of the acoustic module from the first household equipment to the second household equipment, wherein the first household equipment and the second household equipment are multimedia equipment.

12. The head-mounted display device of claim 11, wherein switching the audio source of the acoustic module from a first household device to a second household device comprises:

the method comprises the steps of obtaining playing information of first household equipment, wherein the playing information of the first household equipment comprises an identifier of multimedia content played by the first household equipment and a playing progress corresponding to the multimedia content;

and controlling the second household equipment to continue playing the multimedia content according to the playing information, and switching the sound source of the acoustic module from the first household equipment to the second household equipment.

13. A storage medium having stored thereon computer instructions which, when executed by a processor, carry out the steps of the method of controlling a household appliance according to any one of claims 1 to 8.

Images