WO2021169881A1

WO2021169881A1 - Display control method and device for head-mounted device, and computer storage medium

Info

Publication number: WO2021169881A1
Application number: PCT/CN2021/077112
Authority: WO
Inventors: 张复尧; 熊棋; 徐毅
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-02-24
Filing date: 2021-02-20
Publication date: 2021-09-02
Also published as: CN115053203A

Abstract

Disclosed are a display control method and device for a head-mounted device, and a computer storage medium, wherein same are applied to a head-mounted device. The method comprises: identifying the current application scenario; and on the basis of a display control command corresponding to the current application scenario, controlling a display state of a head-mounted device. As such, by means of determining an application scenario of a head-mounted device, and according to a display control command corresponding to the application scenario, the head-mounted device can automatically change a display state of a user interface according to a user behavior or a use state under different application scenarios, such that the workload of manual adjustments which are performed by a user can be reduced. Moreover, the interactive effect between the head-mounted device and a user can further be improved, and the user friendliness of the user interface is improved.

Description

Display control method, device and computer storage medium of head-mounted device

Cross-references to related applications

This application claims priority to the prior U.S. provisional patent application entitled "Method for Controlling Head-Mounted Display" filed on February 24, 2020 with application number 62/980,915, the entire content of which is incorporated herein by reference. Applying.

Technical field

The embodiments of the present application relate to the field of vision enhancement technology, and in particular, to a display control method of a head-mounted device, a device, and a computer storage medium.

Background technique

In recent years, with the development of visual enhancement technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), a virtual three-dimensional world can be simulated through a computer system, so that The user can interact with the virtual scene and give the user an immersive experience.

In the visual enhancement system, at least a head-mounted device and a mobile device are included, and the head-mounted device and the mobile device can be connected through wired or wireless communication. At present, although a set of user interfaces are also defined for head-mounted devices, these user interfaces cannot adapt to different usage scenarios, resulting in unfriendly display of the user interface, which is not conducive to user operations.

Summary of the invention

The embodiments of the present application provide a display control method, device, and computer storage medium of a head-mounted device, which can improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

The technical solutions of the embodiments of the present application can be implemented as follows:

In the first aspect, an embodiment of the present application provides a display control method of a head-mounted device, which is applied to the head-mounted device, and the method includes:

Identify the current application scenario;

Based on the display control command corresponding to the current application scenario, the display state of the head-mounted device is controlled.

In the second aspect, an embodiment of the present application provides a display control method of a head-mounted device, which is applied to a mobile device, and the method includes:

Identify the application scenarios of mobile devices;

The application scenario of the mobile device is sent to the head-mounted device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and to control the head-mounted device according to the display control command. The display status of the device is controlled.

In a third aspect, an embodiment of the present application provides a head-mounted device, which includes a scene analysis service module, a sending module, and a display module; wherein,

The scenario analysis service module is configured to identify the current application scenario;

The sending module is configured to send the display control command corresponding to the current application scene from the scene analysis service module to the display module;

The display module is configured to control the display state according to the display control command.

In a fourth aspect, an embodiment of the present application provides a head-mounted device, which includes a first memory and a first processor; wherein,

The first memory is configured to store a computer program that can run on the first processor;

The first processor is configured to execute the method described in the first aspect when the computer program is running.

In a fifth aspect, an embodiment of the present application provides a mobile device, which includes a scenario analysis service module and a sending module; wherein,

The scenario analysis service module is configured to identify the application scenario of the mobile device;

The sending module is configured to send the application scenario of the mobile device to the head-mounted device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and control the display according to the Command to control the display state of the head-mounted device.

In a sixth aspect, an embodiment of the present application provides a mobile device, which includes a second memory and a second processor; wherein,

The second memory is configured to store a computer program that can run on the second processor;

The second processor is configured to execute the method described in the second aspect when the computer program is running.

In a seventh aspect, an embodiment of the present application provides a computer storage medium that stores a computer program that, when executed by a first processor, implements the method described in the first aspect, or is executed by a second The processor implements the method described in the second aspect when executed.

The embodiments of the present application provide a display control method, device, and computer storage medium of a head-mounted device, which are applied to the head-mounted device, by identifying the current application scenario; based on the display control command corresponding to the current application scenario, Control the display state of the head-mounted device. In this way, by determining the application scenario of the head-mounted device, and then according to the display control instructions corresponding to the application scenario, the head-mounted device can automatically change the display of the user interface according to the user behavior or use state in different application scenarios. Status can reduce the workload of the user's manual adjustment; at the same time, it can also improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

Description of the drawings

FIG. 1 is a schematic diagram of the composition of a vision enhancement system provided by an embodiment of the application;

2 is a schematic flowchart of a display control method of a head-mounted device according to an embodiment of the application;

3 is a schematic flowchart of another display control method of a head-mounted device according to an embodiment of the application;

4 is a schematic flowchart of a detailed flow chart of a display control method of a head-mounted device according to an embodiment of the application;

FIG. 5 is a detailed flowchart of another display control method of a head-mounted device according to an embodiment of the application;

FIG. 6 is a detailed flowchart of another display control method of a head-mounted device according to an embodiment of the application;

FIG. 7 is a detailed flowchart of yet another display control method of a head-mounted device according to an embodiment of the application;

FIG. 8 is a schematic diagram of the composition structure of a head-mounted device provided by an embodiment of the application;

FIG. 9 is a schematic diagram of a specific hardware structure of a head-mounted device provided by an embodiment of the application;

FIG. 10 is a schematic diagram of the composition structure of a mobile device provided by an embodiment of this application;

FIG. 11 is a schematic diagram of a specific hardware structure of a mobile device provided by an embodiment of this application;

FIG. 12 is a schematic diagram of the architecture structure of a vision enhancement system provided by an embodiment of the application.

Detailed ways

In order to have a more detailed understanding of the characteristics and technical content of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The attached drawings are for reference and explanation purposes only, and are not used to limit the embodiments of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of this application. The terminology used herein is only for the purpose of describing the embodiments of the application, and is not intended to limit the application.

In the following description, “some embodiments” are referred to, which describe a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and Can be combined with each other without conflict. It should also be pointed out that the term "first\second\third" referred to in the embodiments of this application only distinguishes similar objects, and does not represent a specific order for the objects. Understandably, "first\second\third" "Three" may be interchanged in specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein.

Refer to FIG. 1, which shows a schematic diagram of the composition of a vision enhancement system provided by an embodiment of the present application. As shown in FIG. 1, the visual enhancement system 10 may include a head-mounted device 110 and a mobile device 120. Wherein, the head-mounted device 110 and the mobile device 120 are in a wired or wireless communication connection.

Here, the head-mounted device 110 may specifically refer to a monocular or binocular head-mounted display (Head-Mounted Display, HMD), such as AR glasses. In FIG. 1, the head-mounted device 110 may include one or more display modules 111 placed near the position of the user's single eye or both eyes. Among them, through the display module 111 of the head-mounted device 110, the content displayed therein can be presented in front of the user's eyes, and the displayed content can fill or partially fill the user's field of vision. It should also be noted that the display module 111 may refer to one or more organic light-emitting diode (OLED) modules, liquid crystal display (LCD) modules, laser display modules, and the like.

In addition, in some embodiments, the head-mounted device 110 may also include one or more sensors and one or more cameras. For example, the head-mounted device 110 may include one or more sensors such as an inertial measurement unit (IMU), an accelerometer, a gyroscope, a proximity sensor, and a depth camera.

The mobile device 120 may be wirelessly connected to the head-mounted device 110 according to one or more wireless communication protocols (for example, Bluetooth, WIFI, etc.). Alternatively, the mobile device 120 may also be wired to the head-mounted device 110 via a data cable according to one or more data transmission protocols such as Universal Serial Bus (USB). Here, the mobile device 120 may be implemented in various forms. For example, the mobile devices described in the embodiments of the present application may include smart phones, tablet computers, notebook computers, laptop computers, palmtop computers, personal digital assistants (Personal Digital Assistant, PDA), smart watches, and so on.

In some embodiments, a user operating on the mobile device 120 can control the operations at the head-mounted device 110 via the mobile device 120. In addition, the data collected by the sensors in the head-mounted device 110 may also be sent back to the mobile device 120 for further processing or storage.

It should also be noted that the current HMD solution usually defines a set of unchanging user interfaces (User Interface, UI), but these user interfaces cannot adapt to different usage scenarios. However, adaptive system behavior has been widely adopted by other devices and applications. Among them, some devices will sense the environment and adjust their functions; for example, noise-canceling headphones will detect the voice of people in the environment and adjust the level of noise reduction. Some applications will sense the user's behavior/posture and adjust functions accordingly; for example, a fitness application on a smart watch will start tracking the user's heartbeat when it detects that the user is exercising. Some systems sense the environment and user behavior at the same time; for example, a car display system may detect the movement of the car and the Bluetooth connection of the user's phone, and in this case display a set of specific UI layouts and/or components. In other words, although the head-mounted device also defines a set of user interfaces, these user interfaces cannot adapt to different usage scenarios, resulting in unfriendly display of the user interface, which is not conducive to user operations.

The embodiment of the present application provides a display control method of a head-mounted device, which is applied to the head-mounted device. The basic idea of the method is: to identify the current application scenario; based on the display control command corresponding to the current application scenario, Control the display state of the head-mounted device.

The embodiment of the present application also provides a display control method of a head-mounted device, which is applied to a mobile device. The basic idea of the method is to identify the application scenario of the mobile device; and send the application scenario of the mobile device to the head-mounted device. Device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and to control the display state of the head-mounted device according to the display control command.

In this way, by determining the application scenario of the head-mounted device, and then according to the display control instructions corresponding to the application scenario, the head-mounted device can automatically change the display of the user interface according to the user behavior or use state in different application scenarios. Status can reduce the workload of the user's manual adjustment; at the same time, it can also improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

Hereinafter, each embodiment of the present application will be described in detail with reference to the accompanying drawings.

In an embodiment of the present application, refer to FIG. 2, which shows a schematic flowchart of a display control method of a head-mounted device provided in an embodiment of the present application. As shown in Figure 2, the method may include:

S101: Identify the current application scenario.

It should be noted that the method of the embodiment of the present application is applied to a head-mounted device, and the head-mounted device includes a display module. Specifically for the display module, a user interface is usually defined on the display module, but these user interfaces are currently unable to adapt to different application scenarios. The embodiments of the present application mainly provide a way to control users according to related factors such as user behavior or usage status. The method of the interface; more specifically, the method provided in the embodiment of the present application can enable the user interface to adapt to different application scenarios of the head-mounted device.

In the embodiment of the present application, the head-mounted device includes a scene analysis service module, which can be used to identify the application scenario of the head-mounted device in a variety of ways, for example, the application scenario of the head-mounted device can be identified according to the user's posture information , The application scenario of the head-mounted device can also be identified according to the foreground application event, and even the application scenario of the head-mounted device can be identified according to external events (such as external sensor events, application scenarios of the mobile device, etc.), which is not limited here.

In a possible implementation manner, the identifying the current application scenario may include:

Obtain the user's posture information;

Comparing the posture information with a predefined posture;

If the posture information matches the predefined posture, application scenario recognition is performed according to the posture information, and the current application scenario is determined.

It should be noted that after comparing the posture information with the predefined posture, if the posture information matches the predefined posture, then the current application scenario can be determined according to the matched posture information; if the posture information is not the same as the predefined posture If it matches, it will return to the step of obtaining the user's posture information.

It should also be noted that the predefined posture may include at least one preset posture. For the matching of the posture information with the predefined posture, the method may further include: if a preset posture matching the posture information is queried in the predefined posture, determining that the posture information matches the predefined posture .

In other words, you can query in the pre-defined posture. If the preset posture that matches the posture information can be queried, it can indicate that the posture information matches the pre-defined posture; at this time, it can be based on the posture information (or relative Matching preset posture) to determine the current application scenario.

In addition, the head-mounted device may also include a user status monitoring service module, which is used to obtain user sensor data to determine the user's posture information. Therefore, in some embodiments, the obtaining the user's posture information may include:

Obtain user sensor data through the user status monitoring service module of the head-mounted device;

The sensor data of the user is analyzed to determine the posture information of the user.

In other words, after the user sensor data is acquired through the user status monitoring service module of the head-mounted device, signal processing technology and/or machine learning technology can be used to determine the user's posture information from the user sensor data.

Specifically, the machine learning technology can be Support Vector Machine (SVM) technology, Artificial Neural Network (ANN) technology, Deep Learning (DL) technology, etc., but there is no limitation here.

In another possible implementation manner, the identifying the current application scenario may include:

Get the foreground application event;

Search the foreground application event in a predefined foreground application event list;

If the foreground application event matches the predefined foreground application event list, application scenario identification is performed according to the foreground application event, and the current application scenario is determined.

It should be noted that for foreground application events, if the foreground application event matches the predefined foreground application event list, then the current application scenario can be determined according to the matched foreground application event; if the foreground application event matches the predefined foreground application event If the list does not match, it will return to the step of obtaining the foreground application event.

It should also be noted that the predefined foreground application event list may include at least one preset foreground application event. For foreground application events that match a predefined foreground application event list, the method may further include: if a preset foreground application event that matches the foreground application event is searched in the preset foreground application event list, determining The foreground application event matches the predefined foreground application event list.

In other words, you can search in the predefined foreground application event list. If you can search for the preset foreground application event that matches the foreground application event, then it can indicate that the foreground application event matches the predefined foreground application event list; At this time, the current application scenario can be determined according to the foreground application event (or the matching preset foreground application event).

In addition, the head-mounted device may also include a foreground application monitoring service module for acquiring foreground application events. Therefore, in some embodiments, the obtaining the foreground application event may include: obtaining the foreground application event through the foreground application monitoring service module of the head-mounted device.

It should be noted that the foreground application event may be an active application of the user running on the head-mounted device. In other words, the foreground application monitoring service module can monitor the user's active applications running on the head-mounted device to obtain foreground application events.

Obtain external sensor events;

Perform application scenario recognition according to the external sensor event, and determine the current application scenario.

It should be noted that the embodiments of the present application may also identify the current application scenario according to external sensor events. Among them, the external sensor event may be detected by an external sensor device (or called: externally extended sensor device).

In some embodiments, the acquiring external sensor events may include:

Establishing a communication connection between the external sensor device and the event receiver of the head-mounted device;

Obtain the external sensor event sent by the external sensor device through the event receiver.

It should be noted that the external sensor device includes a sensor monitoring service module. The external sensor event can be obtained through the sensor monitoring service module, and then the external sensor device sends the external sensor event to the event receiver of the head-mounted device, thereby In this type of device, the external sensor event can be obtained from the event receiver.

It should also be noted that the communication connection between the external sensor device and the event receiver of the head-mounted device may be a wired communication connection established through a data cable, or may be a wireless communication connection established according to a wireless communication protocol.

In the embodiment of the present application, the wireless communication protocol may include at least one of the following: Bluetooth (Bluetooth) protocol, Wireless Fidelity (WIFI) protocol, Infrared Data Association (IrDA) protocol, and short-distance transmission ( Near Field Communication, NFC) protocol.

Obtain application scenarios of mobile devices;

Perform application scenario identification according to the application scenario of the mobile device, and determine the current application scenario.

Further, in some embodiments, the obtaining the application scenario of the mobile device may include:

Establishing a communication connection between the mobile device and the event receiver of the head-mounted device;

Obtain the application scenario of the mobile device sent by the mobile device through the event receiver.

Further, in some embodiments, the performing application scenario identification according to the application scenario of the mobile device to determine the current application scenario may include:

The application scenario of the mobile device is determined as the current application scenario.

It should be noted that in a mobile device, user sensor data (or user posture information) can be obtained through the user status monitoring service module inside the mobile device, or the foreground application can be obtained through the foreground application monitoring service module inside the mobile device. Event; then through the analysis of these user sensor data or foreground application events, the application scenario of the mobile device can be determined; in addition, the mobile device can even obtain external sensor events through external sensor devices to determine the application of the mobile device Scenes. Here, it should be noted that the mobile device uses its internal user status monitoring service module or foreground application monitoring service module, or uses external sensor devices to determine the application scenario of the mobile device. The application scenarios of the user status monitoring service module or the foreground application monitoring service module, or the use of external sensor devices to determine the head-mounted device are similar, and will not be detailed here.

It should also be noted that after determining the application scenario of the mobile device, the mobile device can send it to the event receiver of the head-mounted device, so that in the head-mounted device, the mobile device can be obtained from the event receiver The application scenarios of the headsets, and then determine the application scenarios of the head-mounted device. Since the mobile device and the head-mounted device are in the same visual enhancement system, in a specific example, the application scenario of the mobile device can be directly determined as the current application scenario of the head-mounted device.

In the embodiment of the present application, the communication connection between the mobile device and the event receiver of the head-mounted device may be a wired communication connection established through a data cable, or may be a wireless communication connection established according to a wireless communication protocol. Here, the wireless communication protocol may include at least one of the following: Bluetooth protocol, WIFI protocol, IrDA protocol, and NFC protocol.

S102: Control the display state of the head-mounted device based on the display control command corresponding to the current application scenario.

It should be noted that after the current application scenario of the head-mounted device is determined, the display control command corresponding to the current application scenario can be obtained. Specifically, in some embodiments, after the identification of the current application scenario, the method may further include:

Comparing the current application scenario with a predefined scenario;

If the current application scene matches the predefined scene, the display control command corresponding to the current application scene is determined.

In the embodiment of the present application, the predefined scenes may include at least one of the following: driving scenes, walking scenes, cycling scenes, standing scenes, sitting scenes, holding mobile devices scenes, putting down mobile devices scenes, and geographic location scenes ; Display control commands can include at least one of the following: open display command, close display command, open partial display command, close partial display command, adjust display area size command, and adjust display element arrangement command.

It should be noted that for the current application scenario of the headset, if the current application scenario matches the predefined scenario, then the corresponding display control command can be determined according to the matching scenario at this time to control the interior of the headset The display status of the display module, such as changing the display, UI layout, rearranging UI elements, UI depth, brightness, power consumption in AR, etc.; if the current application scene does not match the predefined scene, then it will return to execution at this time Steps to identify the current application scenario.

It should also be noted that the predefined scene may include at least one preset scene, and each preset scene corresponds to a display control command with a different display state. For matching the application scene with the predefined scene, the method may further include: comparing the application scene with the predefined scene; if a preset scene matching the application scene is queried in the predefined scene, determining all The application scenarios described match the predefined scenarios.

That is to say, you can query in a predefined scene. If you can find a preset scene that matches the application scene, then it can indicate that the application scene matches the predefined scene; at this time, it can be based on the application scene (or , Matching preset scene) to determine the corresponding display control command to control the display state of the display module inside the head-mounted device.

This embodiment provides a display control method of a head-mounted device, which is applied to the head-mounted device. Identify the current application scenario; control the display state of the head-mounted device based on the display control command corresponding to the current application scenario. In this way, by determining the application scenario of the head-mounted device, and then according to the display control instructions corresponding to the application scenario, the head-mounted device can automatically change the display of the user interface according to the user behavior or use state in different application scenarios. State, to reduce the workload of manual adjustment by the user; at the same time, it can also improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

In another embodiment of the present application, refer to FIG. 3, which shows a schematic flowchart of another display control method for a head-mounted device provided in an embodiment of the present application. As shown in Figure 3, the method may include:

S201: Identify the application scenario of the mobile device.

It should be noted that the method in the embodiment of the present application is applied to a mobile device. In the embodiment of the present application, the mobile device may also include a scene analysis service module to analyze the application scene of the mobile device, and then send the application scene to the head-mounted device, and the head-mounted device includes a display Module, in order to be able to adaptively control the display state of the display module in different application scenarios.

It should also be noted that the embodiment of the application can also have multiple ways to identify the application scenario of the mobile device. For example, the application scenario of the mobile device can be identified based on the user's posture information, or the application of the mobile device can be identified based on the foreground application event. Scenarios can even identify the application scenarios of the mobile device based on external events (such as external sensor events, etc.), which is not limited here.

In a possible implementation manner, the application scenario of identifying a mobile device may include:

Obtain the user's posture information;

Comparing the posture information with a predefined posture;

If the posture information matches the predefined posture, application scenario recognition is performed according to the posture information, and the application scenario of the mobile device is determined.

It should be noted that after comparing the posture information with the predefined posture, if the posture information matches the predefined posture, then the application scenario of the mobile device can be determined according to the matched posture information; if the posture information matches the predefined posture If it does not match, it will return to the step of obtaining the user's posture information.

It should also be noted that, in the mobile device, the predefined posture may include at least one preset posture. For the matching of the posture information with the predefined posture, the method may further include: if a preset posture matching the posture information is queried in the predefined posture, determining that the posture information matches the predefined posture .

In other words, you can query in the pre-defined posture. If the preset posture that matches the posture information can be queried, it can indicate that the posture information matches the pre-defined posture; at this time, it can be based on the posture information (or relative Matching preset posture) to determine the application scenario of the mobile device.

In addition, the mobile device may also include a user status monitoring service module, which is used to obtain user sensor data to determine the user's posture information. Therefore, in some embodiments, the obtaining the user's posture information may include:

Obtain user sensor data through the user status monitoring service module of the mobile device;

That is to say, after the user sensor data is obtained through the user status monitoring service module of the mobile device, signal processing technology and/or machine learning technology can be used to determine the user's posture information from the user sensor data. For example, the machine learning technology can be SVM technology, ANN technology, deep learning technology, etc., but it is not limited in any way.

In another possible implementation manner, the application scenario of identifying a mobile device may include:

Get the foreground application event;

If the foreground application event matches the predefined foreground application event list, application scenario identification is performed according to the foreground application event, and the application scenario of the mobile device is determined.

It should be noted that for foreground application events, if the foreground application event matches the predefined foreground application event list, then the application scenario of the mobile device can be determined according to the matched foreground application event; if the foreground application event matches the predefined foreground application If the event list does not match, it will return to the step of obtaining the foreground application event.

It should also be noted that, in a mobile device, the predefined foreground application event list may include at least one preset foreground application event. For foreground application events that match a predefined foreground application event list, the method may further include: if a preset foreground application event that matches the foreground application event is searched in the preset foreground application event list, determining The foreground application event matches the predefined foreground application event list.

In other words, you can search in the predefined foreground application event list. If you can search for the preset foreground application event that matches the foreground application event, then it can indicate that the foreground application event matches the predefined foreground application event list; At this time, the application scenario of the mobile device can be determined according to the foreground application event (or the matching preset foreground application event).

In addition, the mobile device may also include a foreground application monitoring service module for acquiring foreground application events. Therefore, in some embodiments, the obtaining the foreground application event may include: obtaining the foreground application event through the foreground application monitoring service module of the mobile device.

It should be noted that the foreground application event may be an active application of the user running on the mobile device. In other words, the foreground application monitoring service module can monitor the user's active applications running on the mobile device to obtain foreground application events.

Obtain external sensor events;

The application scenario identification is performed according to the external sensor event, and the application scenario of the mobile device is determined.

It should be noted that the embodiments of the present application may also identify application scenarios of the mobile device based on external sensor events. Among them, the external sensor event may be detected by an external sensor device (or called: externally extended sensor device).

In some embodiments, the acquiring external sensor events may include:

Establishing a communication connection between the external sensor device and the event receiver of the mobile device;

It should be noted that the external sensor device includes a sensor monitoring service module. The external sensor event can be obtained through the sensor monitoring service module, and then the external sensor device sends the external sensor event to the event receiver of the mobile device, so that in the mobile device, The external sensor event can be obtained from the event receiver.

It should also be noted that the communication connection between the external sensor device and the event receiver of the mobile device may be a wired communication connection established through a data cable, or may be a wireless communication connection established according to a wireless communication protocol. Here, the wireless communication protocol may include at least one of the following: Bluetooth protocol, WIFI protocol, IrDA protocol, and NFC protocol.

S202: Send the application scenario of the mobile device to the head-mounted device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and to control the head according to the display control command. The display status of the wearable device is controlled.

It should be noted that after the application scenario of the mobile device is determined, the application scenario of the mobile device can be sent to the head-mounted device, so that the head-mounted device can determine the display control command and control the head-mounted device according to the display control command The display status.

In the embodiment of the present application, the application scenario of the mobile device may include at least one of the following: driving scenario, walking scenario, cycling scenario, standing scenario, sitting scenario, holding mobile device scenario, lowering the mobile device scenario, and geographic location Location scene; display control commands can include at least one of the following: open display command, close display command, open partial display command, close partial display command, adjust display area size command, and adjust display element arrangement command.

Specifically, after the application scenario of the mobile device is determined, the mobile device can send it to the event receiver of the head-mounted device, so that in the subsequent head-mounted device, the mobile device's information can be obtained from the event receiver. Application scenarios, and then determine the application scenarios of the head-mounted device. Since the mobile device and the head-mounted device are in the same visual enhancement system, in a specific example, the application scenario of the mobile device can be directly determined as the current application scenario of the head-mounted device.

In this way, the head-mounted device can determine the corresponding display control command according to the application scenario of the mobile device to control the display status of the display module inside the head-mounted device, such as changing the display, UI layout, rearranging UI elements, and UI depth, brightness, power consumption, etc.

This embodiment provides a display control method of a head-mounted device, which is applied to a mobile device. Identify the application scenario of the mobile device; send the application scenario of the mobile device to the head-mounted device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine the display control command and control the display according to the Command to control the display state of the head-mounted device. In this way, after the head-mounted device receives the application scenario of the mobile device, it can display control instructions corresponding to the application scenario, so that in different application scenarios, the head-mounted device can automatically change the user according to user behavior or usage status. The display status of the interface can reduce the workload of manual adjustment by the user; at the same time, it can also improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

In another embodiment of the present application, refer to FIG. 4, which shows a detailed flowchart of a display control method for a head-mounted device provided in an embodiment of the present application. As shown in Figure 4, the method may include:

S301: monitor the user's posture information;

S302: Compare with a predefined posture;

S303: Judge whether it matches;

S304: If the judgment result is yes, determine the current application scenario and compare it with the predefined scenario;

It should be noted that for step S303, if the judgment result is yes, then step S304 is executed; if the judgment result is no, then step S301 is executed back.

It should also be noted that the head-mounted device may include a user status monitoring service module, a foreground application monitoring service module, and a scene analysis server, while a mobile device may also include a user status monitoring service module, a foreground application monitoring service module, and scene analysis. server. In this way, the monitoring of the user's posture information, foreground application events, etc., can be performed by a head-mounted device or a mobile device, and there is no limitation here.

In the embodiment of the present application, for step S301, it may be obtained by monitoring by the user status monitoring service module; for step S302-S304, it may be obtained by analyzing by the scene analysis server.

S305: monitor foreground application events;

S306: Search in the predefined foreground application event list;

S307: Judge whether it matches;

It should be noted that, for step S307, if the judgment result is yes, then step S304 is executed; if the judgment result is no, then step S305 is executed back.

It should also be noted that steps S301-S303 and steps S305-S307 can be executed in parallel, and there is no order of execution to determine the current application scenario.

In addition, for step S305, it can be obtained by monitoring by the foreground application monitoring service module; for step S306-S307 and S304, it can be obtained by analyzing the scene analysis server.

S308: Obtain external sensor events, and execute S304 according to the external sensor events;

It should be noted that, in addition to mobile devices and head-mounted devices, the vision enhancement system may also include external sensor devices, and the external sensor devices include a sensor monitoring service module. Here, step S308 may be to run a sensor monitoring service module on the external sensor device to detect the external sensor event.

It should also be noted that steps S308, steps S301-S303, and steps S305-S307 can be executed in parallel, and there is no order of execution; and this also shows three ways to identify the current application scenario, after which, Step S304 is all performed.

S309: Determine whether it matches;

S310: If the judgment result is yes, send a display control command based on the matching scene.

It should be noted that, for step S309, if the judgment result is yes, then step S310 is executed; if the judgment result is no, then step S308, S305, and S301 are executed back.

It should also be noted that S309 and S310 may be performed by a head-mounted device. The head-mounted device may also include a display module (also referred to as a “screen”). In this way, the scene analysis server in the head-mounted device can control the display state of the display module after sending a display control command to the display module.

Further, after step S310, the next stage of application scene recognition and its corresponding display control command can be performed, that is, it is also necessary to return to steps S308, S305, and S301 at this time.

In addition, the matching scenario described in the embodiment of the present application may specifically refer to a preset scenario (or, the current application scenario) that matches the current application scenario. Here, in the case where the judgment result is yes, the current application scene or the preset scene matching it may be referred to as "matching scene" for short. At this time, inside the head-mounted device, the scene analysis server sends a display control command to the display module to control the display state of the display module.

In short, the embodiments of the present application relate to a method for controlling a user interface based on factors related to user behavior and usage. Specifically, the embodiment of the present application relates to a user interface, which can adapt to application scenarios of a head-mounted device in different use cases. For example, the embodiment of the present application can autonomously sense the user's state and turn off the screen display of the head-mounted device or hide a part of UI components on the screen of the head-mounted device.

It should be noted that the embodiments of the present application are mainly used to improve the interaction effect between the head-mounted device and the user. More specifically, the embodiments of the present application enable the head-mounted device to automatically change its UI according to the user's behavior. This can reduce the workload of the user to manually adjust the UI. For example, for a pair of AR glasses that work with a mobile device (such as a smart phone), when the system senses that the user is watching the screen of the mobile device, the AR glasses will close its display module or avoid presenting any content to reduce visual interference. In this way, in a specific example, the optical see through (OST) of the AR glasses is turned off, so that the user can better view the real world. In another specific example, the system may resize and/or reorganize the UI components on the display (for example, when the user looks down, the UI may be rearranged to the upper half).

Exemplarily, when a user uses a head-mounted device to play video games or watch a movie, the user may use a wireless controller to control the head-mounted device. When the user's smart phone receives a text message or a call, the user puts down the wireless controller and picks up the smart phone. The embodiment of the present application detects this series of events, and at this time, the screen of the head-mounted device will be turned off, so that the user can better watch the screen of the smart phone. The detection of events can be based on one or more of the following: sensor data from the wireless controller (for example, whether the user holds the wireless controller), events from the smartphone (for example, the user picks up the smartphone and/or unlocks the smartphone) To read text), and sensor data from the head-mounted device (for example, whether the device is pointing down), etc. As shown in the foregoing FIG. 4, it shows a possible implementation manner provided by an embodiment of the present application.

In addition, FIG. 5 shows another possible implementation manner provided by an embodiment of the present application. At this time, only user behavior can cause the screen of the head-mounted device to turn off. For example, the user suddenly enters a running state from walking at a slow speed. Refer to FIG. 5, which shows a detailed flowchart of another display control method of a head-mounted device provided by an embodiment of the present application. As shown in Figure 5, the method may include:

S401: monitor the user's posture information;

S402: Compare with a predefined posture;

S403: Judge whether it matches;

S404: If the judgment result is yes, determine the current application scenario and compare it with the predefined scenario;

S405: When the current application scene matches the predefined scene, send a display control command based on the matching scene.

It should be noted that, for step S403, if the judgment result is yes, then step S404 is executed; if the judgment result is no, then step S401 is executed back.

Further, after step S405, the next stage of application scene recognition and its corresponding display control command can be performed, that is, it is also necessary to return to step S401 at this time.

In the embodiment of the present application, the predefined scene includes at least one preset scene, and each preset scene corresponds to a display control command with a different display state. Here, in the case that the current application scene matches the predefined scene, the current application scene or the preset scene matching it may be referred to as a “matching scene” for short. At this time, inside the head-mounted device, the scene analysis server sends a display control command to the display module to control the display state of the display module.

FIG. 6 shows another possible implementation manner provided by an embodiment of the present application. At this time, only a special application event can cause the screen of the head-mounted device to turn off. For example, a smart phone receives an incoming video call, and when the user accepts the video call, the screen of the head-mounted device is turned off to facilitate the video call on the smart phone. Refer to FIG. 6, which shows a detailed flowchart of yet another display control method of a head-mounted device provided by an embodiment of the present application. As shown in Figure 6, the method may include:

S501: Monitor foreground application events;

S502: Search in a list of predefined foreground application events;

S503: Judge whether it matches;

S504: If the judgment result is yes, determine the current application scenario and compare it with the predefined scenario;

S505: When the current application scene matches the predefined scene, send a display control command based on the matching scene.

It should be noted that, for step S503, if the judgment result is yes, then step S504 is executed; if the judgment result is no, then step S501 is executed back.

Further, after step S505, the next stage of application scene recognition and its corresponding display control command can be performed, that is, it is also necessary to return to step S501 at this time.

FIG. 7 shows another possible implementation manner provided by an embodiment of the present application. At this time, a single external sensor event may cause the screen of the head-mounted device to turn off. For example, when a user approaches a smart door lock and starts typing a password on the lock disk. Refer to FIG. 7, which shows a detailed flowchart of another method for display control of a head-mounted device provided by an embodiment of the present application. As shown in Figure 7, the method may include:

S601: Obtain external sensor events;

S602: Compare with a predefined scene;

S603: Determine whether it matches;

S604: If the judgment result is yes, if the current application scene matches the predefined scene, send a display control command based on the matching scene.

It should be noted that, for step S603, if the judgment result is yes, then step S604 is executed; if the judgment result is no, then step S601 is executed back.

Further, after step S604, the next stage of application scene recognition and its corresponding display control command can be performed, that is, it is also necessary to return to step S601 at this time.

In the embodiment of the present application, the vision enhancement system may further include an external sensor device, and the external sensor event can be acquired through the external sensor device. After acquiring external sensor events, the current application scenario can be identified; then the current application scenario is compared with the predefined scenario, so that when the current application scenario matches the predefined scenario, the matching scenario can be determined by The scene analysis server in the head-mounted device sends the display control command to the display module, so as to control its display state.

This embodiment provides a display control method of a head-mounted device. The specific implementation of the foregoing embodiment is described in detail through this embodiment. It can be seen from this that the embodiment of the application significantly improves the user experience of the head-mounted device. . More specifically, the embodiments of the present application can enable the head-mounted device to automatically change its UI according to user behavior, thereby reducing the workload of the user to manually adjust the UI. In this way, the embodiments of the present application can develop a head-mounted device product with a better user experience, and make the head-mounted device product more intelligent.

In yet another embodiment of the present application, based on the same inventive concept as the foregoing embodiment, refer to FIG. 8, which shows a schematic diagram of the composition structure of a head-mounted device 70 provided by an embodiment of the present application. As shown in FIG. 8, the head-mounted device 70 may include: a scene analysis service module 701, a sending module 702, and a display module 703; among them,

The scenario analysis service module 701 is configured to identify the current application scenario;

The sending module 702 is configured to send the display control command corresponding to the current application scene from the scene analysis service module to the display module 703;

The display module 703 is configured to control the display state according to the display control command.

In some embodiments, the scene analysis service module 701 is specifically configured to obtain posture information of the user; and compare the posture information with a predefined posture; The posture information is used to identify the application scenario, and the current application scenario is determined.

In some embodiments, referring to FIG. 8, the head-mounted device 70 may further include a user status monitoring service module 704;

The user status monitoring service module 704 is configured to obtain user sensor data; and analyze the user sensor data to determine the posture information of the user;

The sending module 702 is further configured to send the posture information of the user to the scene analysis service module 701.

Further, the scene analysis service module 701 is specifically configured to determine that the posture information matches the predefined posture if a preset posture matching the posture information is queried in the predefined posture; The predefined posture includes at least one preset posture.

In some embodiments, the scene analysis service module 701 is specifically configured to obtain foreground application events; and search for the foreground application events in a list of predefined foreground application events; and if the foreground application event matches the predefined If the foreground application event list matches, the application scenario identification is performed according to the foreground application event, and the current application scenario is determined.

In some embodiments, referring to FIG. 8, the head-mounted device 70 may further include a foreground application monitoring service module 705;

The foreground application monitoring service module 705 is configured to obtain the foreground application event;

The sending module 702 is further configured to send the foreground application event to the scene analysis service module 701.

Further, the scene analysis service module 701 is specifically configured to determine that the foreground application event matches the foreground application event if a preset foreground application event matching the foreground application event is searched in the preset foreground application event list. The predefined foreground application event list matches; wherein, the predefined foreground application event list includes at least one preset foreground application event.

In some embodiments, the scene analysis service module 701 is specifically configured to acquire external sensor events; and perform application scene recognition according to the external sensor events, and determine the current application scene.

In some embodiments, referring to FIG. 8, the head-mounted device 70 may further include a communication module 706 and an event receiver 707;

The communication module 706 is configured to establish a communication connection between the external sensor device and the event receiver 707;

The event receiver 707 is configured to receive the external sensor event sent by the external sensor device.

In some embodiments, the scenario analysis service module 701 is further configured to obtain the application scenario of the mobile device; and determine the application scenario of the mobile device as the current application scenario.

In some embodiments, the communication module 706 is further configured to establish a communication connection between the mobile device and the event receiver 707;

The event receiver 707 is further configured to receive the application scenario of the mobile device sent by the mobile device.

In some embodiments, the scene analysis service module 701 is further configured to compare the current application scene with a predefined scene; and if the current application scene matches the predefined scene, determine the current The display control commands corresponding to the application scenarios.

In some embodiments, the predefined scene includes at least one of the following: a driving scene, a walking scene, a riding scene, a standing scene, a sitting scene, a holding mobile device scene, a lowering mobile device scene, and a geographic location scene;

The display control command includes at least one of the following: a display opening command, a display closing command, a partial display opening command, a partial display closing command, a display area size adjustment command, and a display element arrangement adjustment command.

It can be understood that, in the embodiments of the present application, a "module" may be a part of a circuit, a part of a processor, a part of a program, or software, etc., of course, may also be a unit, or may also be non-modular. Moreover, the various components in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or It is said that the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can It is a personal computer, a server, or a network device, etc.) or a processor (processor) that executes all or part of the steps of the method described in this embodiment. The aforementioned storage media include: U disk, mobile hard disk, read only memory (Read Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes.

Therefore, the embodiments of the present application provide a computer storage medium, which is applied to the head-mounted device 70, and the computer storage medium stores a computer program that, when executed by a processor, implements any one of the foregoing embodiments. Methods.

Based on the composition of the above-mentioned head-mounted device 70 and the computer storage medium, refer to FIG. 9, which shows a schematic diagram of the specific hardware structure of the head-mounted device 70 provided by the embodiment of the present application. As shown in FIG. 9, it may include: a first communication interface 801, a first memory 802, and a first processor 803; various components are coupled together through a first bus system 804. It can be understood that the first bus system 804 is used to implement connection and communication between these components. In addition to the data bus, the first bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for clarity of description, various buses are marked as the first bus system 804 in FIG. 9. in,

The first communication interface 801 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

The first memory 802 is configured to store a computer program that can run on the first processor 803;

The first processor 803 is configured to execute: when the computer program is running:

Identify the current application scenario;

It can be understood that the first memory 802 in the embodiment of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), and electrically available Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be a random access memory (Random Access Memory, RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced SDRAM, ESDRAM), Synchronous Link Dynamic Random Access Memory (Synchlink DRAM, SLDRAM) And Direct Rambus RAM (DRRAM). The first memory 802 of the system and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

The first processor 803 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the first processor 803 or instructions in the form of software. The aforementioned first processor 803 may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (ASIC), a ready-made programmable gate array (Field Programmable Gate Array, FPGA) Or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the first memory 802, and the first processor 803 reads the information in the first memory 802, and completes the steps of the foregoing method in combination with its hardware.

It can be understood that the embodiments described in this application can be implemented by hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, and others for performing the functions described in this application Electronic unit or its combination. For software implementation, the technology described in this application can be implemented through modules (for example, procedures, functions, etc.) that perform the functions described in this application. The software codes can be stored in the memory and executed by the processor. The memory can be implemented in the processor or external to the processor.

Optionally, as another embodiment, the first processor 803 is further configured to execute the method described in any one of the foregoing embodiments when the computer program is running.

This embodiment provides a head-mounted device, which may include a scene analysis service module, a sending module, and a display module. In this way, the embodiments of the present application can develop a head-mounted device product with a better user experience, and make the head-mounted device product more intelligent. This allows the head-mounted device to automatically change its UI according to user behavior, reducing the user's manual adjustment of the UI; at the same time, it can also improve the interaction between the head-mounted device and the user, and improve the user-friendliness of the user interface. .

In yet another embodiment of the present application, based on the same inventive concept as the foregoing embodiment, refer to FIG. 10, which shows a schematic diagram of the composition structure of a mobile device 90 provided by an embodiment of the present application. As shown in FIG. 10, the mobile device 90 may include: a scene analysis service module 901 and a sending module 902; among them,

The scenario analysis service module 901 is configured to identify the application scenario of the mobile device;

The sending module 902 is configured to send the application scenario of the mobile device to the head-mounted device; wherein, the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and according to the display control command The display state of the head-mounted device is controlled.

In some embodiments, the scene analysis service module 901 is specifically configured to obtain posture information of the user; and compare the posture information with a predefined posture; The posture information is used to identify the application scenario, and the application scenario of the mobile device is determined.

In some embodiments, referring to FIG. 10, the mobile device 90 may further include a user status monitoring service module 903;

The user status monitoring service module 903 is configured to obtain user sensor data; and analyze the user sensor data to determine the posture information of the user;

The sending module 902 is further configured to send the posture information of the user to the scene analysis service module 901.

Further, the scene analysis service module 901 is specifically configured to determine that the posture information matches the predefined posture if a preset posture matching the posture information is queried in the predefined posture; The predefined posture includes at least one preset posture.

In some embodiments, the scene analysis service module 901 is specifically configured to obtain foreground application events; and search for the foreground application events in a list of predefined foreground application events; and if the foreground application event matches the predefined foreground application event If the foreground application event list matches, the application scenario identification is performed according to the foreground application event, and the application scenario of the mobile device is determined.

In some embodiments, referring to FIG. 10, the mobile device 90 may also include a foreground application monitoring service module 904;

The foreground application monitoring service module 904 is configured to obtain the foreground application event;

The sending module 902 is further configured to send the foreground application event to the scene analysis service module 901.

Further, the scene analysis service module 901 is specifically configured to determine that the foreground application event matches the foreground application event if a preset foreground application event matching the foreground application event is searched in the preset foreground application event list. The predefined foreground application event list matches; wherein, the predefined foreground application event list includes at least one preset foreground application event.

In some embodiments, the scene analysis service module 901 is specifically configured to acquire external sensor events; and perform application scene recognition according to the external sensor events, and determine the application scene of the mobile device.

In some embodiments, referring to FIG. 10, the mobile device 90 may further include a communication module 905 and an event receiver 906;

The communication module 905 is configured to establish a communication connection between the external sensor device and the event receiver 906;

The event receiver 906 is configured to receive the external sensor event sent by the external sensor device.

It is understandable that, in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course, it may also be a module, or it may also be non-modular. Moreover, the various components in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function module.

If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, this embodiment provides a computer storage medium applied to the mobile device 90. The computer storage medium stores a computer program that, when executed by the second processor, implements any of the foregoing embodiments. The method described.

Based on the above composition of the mobile device 90 and the computer storage medium, refer to FIG. 11, which shows a schematic diagram of a specific hardware structure of the mobile device 90 provided by an embodiment of the present application. As shown in FIG. 11, it may include: a second communication interface 1001, a second memory 1002, and a second processor 1003; various components are coupled together through a second bus system 1004. It can be understood that the second bus system 1004 is used to implement connection and communication between these components. In addition to the data bus, the second bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear description, various buses are marked as the second bus system 1004 in FIG. 11. in,

The second communication interface 1001 is used for receiving and sending signals in the process of sending and receiving information with other external network elements;

The second memory 1002 is configured to store a computer program that can run on the second processor 1003;

The second processor 1003 is configured to execute: when running the computer program:

Identify the application scenarios of mobile devices;

Optionally, as another embodiment, the second processor 1003 is further configured to execute the method described in any one of the foregoing embodiments when the computer program is running.

It can be understood that the hardware functions of the second memory 1002 and the first memory 802 are similar, and the hardware functions of the second processor 1003 and the first processor 803 are similar; the details are not described herein again.

This embodiment provides a mobile device, which may include a scene analysis service module and a sending module. In this way, after the head-mounted device receives the application scenario of the mobile device, it can display control instructions corresponding to the application scenario, so that in different application scenarios, the head-mounted device can automatically change its UI according to user behavior , Reducing the workload of the user to manually adjust the UI; at the same time, it can also improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

In yet another embodiment of the present application, refer to FIG. 12, which shows a schematic structural structure diagram of a vision enhancement system provided by an embodiment of the present application. As shown in FIG. 12, the visual enhancement system may include a head-mounted device 1101, a mobile device 1102, and an external sensor device 1103. Wherein, the head-mounted device 1101 is the head-mounted device described in any one of the preceding embodiments, the mobile device 1102 is the mobile device described in any one of the preceding embodiments, and the external sensor device 1103 is the one described in any one of the preceding embodiments. The external sensor device mentioned.

In the embodiment of the present application, the head-mounted device 1101 may include a user status monitoring service module 1, a scene analysis service module 2, an event receiver 3, a foreground application monitoring service module 4, and a display module 5 (that is, the white-filled in FIG. 12 Module); the external sensor device 1103 may include a sensor monitoring service module 6 (that is, the module filled with black in Figure 12); the mobile device 1102 may include a user status monitoring service module 7, a scene analysis service module 8, and a foreground application monitoring service module 9 (That is, the module filled with gray in Figure 12).

Based on the above architectural example, on the head-mounted device 1101: The user status monitoring service module 1 monitors related sensor data (Sensor Data), and uses signal processing technology and/or machine learning technology to detect certain events from the sensor data ( Such as the user's posture information). Then, the service module 1 sends the event to the scene analysis service module 2 on the head-mounted device 1101. In addition, the event receiver 3 can receive external sensor events from the sensor monitoring service module 6 in the external sensor device 1103, that is, the sensor monitoring service module 6 runs on the external sensor device 1103 (such as an IoT device) to detect external sensors. event. The event receiver 3 may also receive user state events from the scene analysis service module 8 on the mobile device 1102, and then the event receiver 3 forwards the event to the scene analysis service module 2 on the head-mounted device 1101.

The foreground application monitoring service module 4 monitors the user's active applications running on the head-mounted device 1101. The service module sends the foreground application event (or called the foreground task event) to the scene analysis service module 2. For example, when the user starts or stops a particular application.

The scene analysis service module 2 collects sensor data from internal sensors of the head-mounted device 1101, collects sensor events from external sensor devices, and collects foreground application events. Then, the module analyzes the data and determines a suitable display control command, and then the display control command will be sent to the display module 5.

The display control commands are predefined for different scenarios. Here, the display control commands include, but are not limited to, turn on/off the display, turn on/off part of the display, brightness control, adjust the display size, rearrange UI elements, and so on.

On the mobile device 1102: The user status monitoring service module 7 monitors real-time sensor data on the mobile device and detects events from the sensor data. The user status monitoring service module 7 sends the sensor event to the scene analysis service module 8 on the mobile device. For example, the mobile device changes to landscape mode or portrait mode. The foreground application monitoring service module 9 monitors the user's active applications running on the mobile device. For example, when the user starts or stops a specific application, the service sends the foreground application event to the scene analysis service module 8. It should be noted that the mobile device 1102 may also include an event receiver (not shown in the figure), and the event receiver may also be used to receive external sensor events from the sensor monitoring service module 6 in the external sensor device 1103.

Specifically, the scene analysis service module 8 collects sensor data from internal sensors of the mobile device, and collects foreground application events from the foreground application monitoring service module 9. Then, the scene analysis service module 8 analyzes the data, determines the usage scene on the mobile device, and sends the scene event of the mobile device to the event receiver 3 on the head-mounted device 1101 through a data cable or WiFi or other wireless communication signals.

The application scenarios that can be detected by the vision enhancement system of the embodiment of the present application include, but are not limited to, driving, walking, cycling, standing, sitting, holding a mobile device, putting down a mobile device, geographic location, and so on. The system response includes but is not limited to changing the display, UI layout, UI depth, brightness, power consumption, etc. in AR.

In this way, the embodiments of the present application significantly improve the user experience of the head-mounted device. More specifically, the embodiments of the present application can enable the head-mounted device to automatically change its UI according to user behavior, thereby reducing the workload of the user to manually adjust the UI. In this way, the embodiments of the present application can develop a head-mounted device product with a better user experience, and make the head-mounted device product more intelligent.

It should be noted that in this application, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements , And also include other elements not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element.

The serial numbers of the foregoing embodiments of the present application are for description only, and do not represent the superiority or inferiority of the embodiments.

The methods disclosed in the several method embodiments provided in this application can be combined arbitrarily without conflict to obtain new method embodiments.

The features disclosed in the several product embodiments provided in this application can be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or device embodiments provided in this application can be combined arbitrarily without conflict to obtain a new method embodiment or device embodiment.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Industrial applicability

In the embodiment of the present application, the current application scenario is identified; based on the display control command corresponding to the current application scenario, the display state of the head-mounted device is controlled. In this way, by determining the application scenario of the head-mounted device, and then according to the display control instructions corresponding to the application scenario, the head-mounted device can automatically change the display of the user interface according to the user behavior or use state in different application scenarios. Status can reduce the workload of the user's manual adjustment; at the same time, it can also improve the interaction effect between the head-mounted device and the user, and improve the user-friendliness of the user interface.

Claims

A display control method of a head-mounted device is applied to the head-mounted device, and the method includes:

Identify the current application scenario;

Based on the display control command corresponding to the current application scenario, the display state of the head-mounted device is controlled.
The method according to claim 1, wherein said identifying the current application scenario comprises:

Obtain the user's posture information;

Comparing the posture information with a predefined posture;

If the posture information matches the predefined posture, application scenario recognition is performed according to the posture information, and the current application scenario is determined.
The method according to claim 2, wherein said obtaining the user's posture information comprises:

Obtain user sensor data through the user status monitoring service module of the head-mounted device;

The sensor data of the user is analyzed to determine the posture information of the user.
The method according to claim 2, wherein the predefined posture includes at least one preset posture, and the method further comprises:

If a preset posture matching the posture information is queried in the predefined posture, it is determined that the posture information matches the predefined posture.
The method according to claim 1, wherein said identifying the current application scenario comprises:

Get the foreground application event;

Search the foreground application event in a predefined foreground application event list;

If the foreground application event matches the predefined foreground application event list, application scenario identification is performed according to the foreground application event, and the current application scenario is determined.
The method according to claim 5, wherein said obtaining foreground application events comprises:

Obtain the foreground application event through the foreground application monitoring service module of the head-mounted device.
The method according to claim 5, wherein the predefined foreground application event list includes at least one preset foreground application event, and the method further comprises:

If a preset foreground application event that matches the foreground application event is searched in the preset foreground application event list, it is determined that the foreground application event matches the predefined foreground application event list.
The method according to claim 1, wherein said identifying the current application scenario comprises:

Obtain external sensor events;

Perform application scenario recognition according to the external sensor event, and determine the current application scenario.
The method according to claim 8, wherein said acquiring external sensor events comprises:

Establishing a communication connection between the external sensor device and the event receiver of the head-mounted device;

Obtain the external sensor event sent by the external sensor device through the event receiver.
The method according to claim 1, wherein said identifying the current application scenario comprises:

Obtain application scenarios of mobile devices;

The application scenario of the mobile device is determined as the current application scenario.
The method according to claim 10, wherein said obtaining the application scenario of the mobile device comprises:

Establishing a communication connection between the mobile device and the event receiver of the head-mounted device;

Obtain the application scenario of the mobile device sent by the mobile device through the event receiver.
The method according to any one of claims 1 to 11, wherein, after the identifying the current application scenario, the method further comprises:

Comparing the current application scenario with a predefined scenario;

If the current application scene matches the predefined scene, the display control command corresponding to the current application scene is determined.
The method according to claim 12, wherein the predefined scene includes at least one of the following: driving scene, walking scene, cycling scene, standing scene, sitting scene, holding mobile device scene, putting down the mobile device Scenes and geographic locations;

The display control command includes at least one of the following: a display opening command, a display closing command, a partial display opening command, a partial display closing command, a display area size adjustment command, and a display element arrangement adjustment command.
A display control method of a head-mounted device, applied to a mobile device, and the method includes:

Identify the application scenarios of mobile devices;

The application scenario of the mobile device is sent to the head-mounted device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and to control the head-mounted device according to the display control command. The display status of the device is controlled.
The method according to claim 14, wherein said identifying the application scenario of the mobile device comprises:

Obtain the user's posture information;

Comparing the posture information with a predefined posture;

If the posture information matches the predefined posture, application scenario recognition is performed according to the posture information, and the application scenario of the mobile device is determined.
The method according to claim 15, wherein said obtaining the posture information of the user comprises:

Obtain user sensor data through the user status monitoring service module of the mobile device;

The sensor data of the user is analyzed to determine the posture information of the user.
The method according to claim 15, wherein the predefined posture includes at least one preset posture, and the method further comprises:

If a preset posture matching the posture information is queried in the predefined posture, it is determined that the posture information matches the predefined posture.
The method according to claim 14, wherein said identifying the application scenario of the mobile device comprises:

Get the foreground application event;

Search the foreground application event in a predefined foreground application event list;

If the foreground application event matches the predefined foreground application event list, application scenario identification is performed according to the foreground application event, and the application scenario of the mobile device is determined.
The method according to claim 18, wherein said obtaining foreground application events comprises:

Obtain the foreground application event through the foreground application monitoring service module of the mobile device.
The method according to claim 18, wherein the predefined foreground application event list includes at least one preset foreground application event, and the method further comprises:

If a preset foreground application event that matches the foreground application event is searched in the preset foreground application event list, it is determined that the foreground application event matches the predefined foreground application event list.
The method according to claim 14, wherein said identifying the application scenario of the mobile device comprises:

Obtain external sensor events;

The application scenario identification is performed according to the external sensor event, and the application scenario of the mobile device is determined.
The method according to claim 21, wherein said acquiring external sensor events comprises:

Establishing a communication connection between the external sensor device and the event receiver of the mobile device;

Obtain the external sensor event sent by the external sensor device through the event receiver.
A head-mounted device, wherein the head-mounted device includes a scene analysis service module, a sending module, and a display module;

The scenario analysis service module is configured to identify the current application scenario;

The sending module is configured to send the display control command corresponding to the current application scene from the scene analysis service module to the display module;

The display module is configured to control the display state according to the display control command.
A head-mounted device, wherein the head-mounted device includes a first memory and a first processor;

The first memory is configured to store a computer program that can run on the first processor;

The first processor is configured to execute the method according to any one of claims 1 to 13 when running the computer program.
A mobile device, wherein the mobile device includes a scene analysis service module and a sending module;

The scenario analysis service module is configured to identify the application scenario of the mobile device;

The sending module is configured to send the application scenario of the mobile device to the head-mounted device; wherein the application scenario of the mobile device is used to instruct the head-mounted device to determine a display control command and control the display according to the Command to control the display state of the head-mounted device.
A mobile device, wherein the mobile device includes a second memory and a second processor;

The second memory is configured to store a computer program that can run on the second processor;

The second processor is configured to execute the method according to any one of claims 14 to 22 when running the computer program.
A computer storage medium, wherein the computer storage medium stores a computer program that, when executed by a first processor, implements the method according to any one of claims 1 to 13, or is executed by a second processor When executed, the method according to any one of claims 14 to 22 is realized.