WO2023124972A1

WO2023124972A1 - Display state switching method, apparatus and system, electronic device and storage medium

Info

Publication number: WO2023124972A1
Application number: PCT/CN2022/138689
Authority: WO
Inventors: 池卓哲
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-12-28
Filing date: 2022-12-13
Publication date: 2023-07-06
Also published as: CN114416237A

Abstract

A display state switching method, apparatus and system, an electronic device and a storage medium. The display state switching method comprises: acquiring a display state of a display device, the display state being an augmented reality state comprising virtual information and real information, the virtual information comprising a virtual audio and a virtual picture, and the real information comprising a real environment audio and a real picture (step S310); in response to an interaction operation acting on an audio device or the display device, switching mode information of the audio device to determine target mode information (step S320); and adjusting the real environment audio in the real information according to the target mode information, and adjusting the real picture according to the target mode information, so as to perform switching between the augmented reality state and a virtual reality state (step S330). The present method can improve the accuracy and comprehensiveness of mode switching.

Description

Display state switching method, device and system, electronic equipment, storage medium

Cross References to Related Applications

This application claims the priority of the Chinese patent application with application number 202111630842.1 and titled "Display state switching method, device and system, electronic equipment, storage medium" filed on December 28, 2021, and the entire content of the Chinese patent application Incorporated herein by reference in its entirety.

technical field

The present disclosure relates to the technical fields of augmented reality and virtual reality, and in particular, relates to a display state switching method, a display state switching device, a display state switching system, electronic equipment, and a computer-readable storage medium.

Background technique

Virtual reality and augmented reality can bring users a strong sense of immersion, allow users to experience the digital world more realistically, and provide unlimited possibilities for the development of the digital world in the future. But virtual reality and augmented reality can only provide information at the visual level.

In related technologies, some devices can provide real information on the premise of providing virtual information, but can only realize screen switching. In the above method, only partial switching of states can be performed, which has certain limitations, so switching between different states cannot be accurately completed. Moreover, there are many operation steps and the convenience is poor.

Contents of the invention

According to one aspect of the present disclosure, a display state switching method is provided, including: acquiring a display state of a display device, the display state being an augmented reality state including virtual information and real information; the virtual information includes virtual audio and virtual picture , the real information includes real environment audio and real picture; in response to the interactive operation acting on the audio device or the display device, the mode information of the audio device is switched to determine the target mode information; according to the target mode information, the real The real environment audio in the information is adjusted, and the real picture is adjusted according to the target mode information, so as to switch between the augmented reality state and the virtual reality state.

According to an aspect of the present disclosure, a display state switching device is provided, including: a state acquisition module configured to acquire a display state of a display device, the display state being an augmented reality state including virtual information and real information; the virtual The information includes virtual audio and virtual screen, and the real information includes real environment audio and real screen; the mode switching module is configured to switch and determine the mode information of the audio device in response to an interactive operation acting on the audio device or the display device Target mode information; an audio picture adjustment module, configured to adjust the real environment audio in the real information according to the target mode information, and adjust the real picture according to the target mode information, so as to adjust all Switch between the above augmented reality state and virtual reality state.

According to an aspect of the present disclosure, there is provided a display state switching system, including: an audio device configured to determine target mode information in response to an interactive operation, and perform real-world audio in real-world information adjusted according to the target mode information. Play; a communication device configured to connect the audio device to the display device; a display device connected to the audio device configured to determine target mode information in response to an interactive operation, and The real picture in the real information adjusted by the information is displayed, so as to switch between the augmented reality state and the virtual reality state.

According to one aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the executable instructions to Execute the display state switching method described in any one of the above.

According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the display state switching method described in any one of the above items is implemented.

Description of drawings

FIG. 1 shows a schematic diagram of a system architecture to which a display state switching method or a display state switching method according to an embodiment of the present disclosure can be applied.

Fig. 2 shows a schematic diagram of a device in the related art.

Fig. 3 schematically shows a schematic diagram of a display state switching method in an embodiment of the present disclosure.

Fig. 4 schematically shows a schematic diagram of an audio device in an embodiment of the present disclosure.

Fig. 5 schematically shows a flow chart of adjusting real environment audio in an embodiment of the present disclosure.

Fig. 6 schematically shows a flow chart of adjusting a real picture in an embodiment of the present disclosure.

FIG. 7 schematically shows a comparative schematic diagram of state switching in an embodiment of the present disclosure.

Fig. 8 schematically shows a schematic diagram of an interface of a see-through mode in an embodiment of the present disclosure.

Fig. 9 schematically shows a schematic interface of a weak noise reduction mode in an embodiment of the present disclosure.

Fig. 10 schematically shows a schematic diagram of an interface of a strong noise reduction mode in an embodiment of the present disclosure.

Fig. 11 schematically shows a block diagram of a display state switching device in an embodiment of the present disclosure.

Fig. 12 schematically shows a block diagram of a display state switching system in an embodiment of the present disclosure.

Fig. 13 shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details being omitted, or other methods, components, devices, steps, etc. may be adopted. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and/or processor means and/or microcontroller means.

Embodiments of the present disclosure provide a method for switching display states, which can be applied to an application scenario of switching modes of a display device. Fig. 1 shows a schematic diagram of a display state switching method and device or a system architecture of a display state switching method and device to which an embodiment of the present disclosure can be applied.

As shown in FIG. 1 , the system architecture 100 may include a display device 101 , a network 102 , and an audio device 103 . Wherein, the display device may be a terminal with a display screen, such as a smart phone, a computer, and a tablet computer. The display device is used for switching mode information of the display device in response to an interactive operation. The network 102 is used as a medium for providing a communication link between the display device 101 and the audio device 103 . Network 102 may include various connection types, such as wired communication links, wireless communication links, and so on. In the embodiment of the present disclosure, the network 102 between the display device 101 and the audio device 103 can be a wired communication link, for example, a communication link can be provided through a serial port connection line; it can also be a wireless communication link, and communication can be provided through a wireless network. link. The audio device 103 may be used to play audio and trigger mode switching, for example, may be a device such as an earphone.

In the embodiment of the present disclosure, the display status of the display device is acquired. The display state may be an augmented reality state including virtual information and real information. Further, when the audio device or the display device receives the interactive operation, the target mode information is determined and the real environment audio in the real information is adjusted. At the same time, the real picture is adjusted according to the target mode information, so as to realize mutual switching between the augmented reality state and the virtual reality state.

The device 1 shown in Diagram A in FIG. 2 is VR glasses, which can provide the user with almost completely immersive visual experience. Among them, the external images captured by multiple cameras outside the VR can be synthesized and mapped to the internal display device. Users can see the external environment and even superimpose some virtual information to form an AR effect. The device 2 shown in Figure B in FIG. 2 has an augmented reality function, and at the same time, the AR/VR form of the virtual object can be switched through the label on the top of the touch screen.

In order to solve the problems in related technologies, an embodiment of the present disclosure provides a display state switching method. Next, the display state switching method in the embodiment of the present disclosure will be described in detail with reference to FIG. 3 .

In step S310, the display state of the display device is obtained, the display state is an augmented reality state including virtual information and real information; the virtual information includes virtual audio and virtual picture, and the real information includes real environment audio and real picture .

In the embodiment of the present disclosure, the display device may be any type of electronic device, such as a mobile phone, a tablet computer or a wearable device. The wearable device may be, for example, VR/AR glasses or the like. The display device consists of a display screen, an environment understanding system, and an interactive system. Wherein, the display screen is arranged on a side close to the user, and the display screen is used to project the digital information calculated in the display device to the eyes of the user in real time. Based on this, the types of display screens include, but are not limited to, mobile phone screens, tablet screens, and display devices projected from VR/AR glasses to the user's glasses.

The environment understanding system is set on the side away from the user, that is, the side close to the external environment. The environment understanding system can be a number of cameras or 3D scanning sensors, and the environment understanding system can capture real environment information according to actual needs, and superimpose virtual information on the captured real information based on SLAM (Simultaneous localization and mapping) technology on to generate the display state of the display device.

The display device is equipped with an interactive system in addition to displaying visual information, and the interactive system is used to perform interactive operations. The type of interactive operation can be determined in combination with the type of display device, and different types of display devices have different types of interactive operation. Based on this, the types of interactive operations can be of various types, thus increasing the diversity of operation types.

The display state refers to the current display state of the display device, and the display state may be an augmented reality state. Specifically, the environment understanding system captures real information including real environment audio and real pictures, and superimposes virtual information including virtual audio and virtual pictures on the real information based on SLAM technology to generate an augmented reality state. Therefore, the display information also includes virtual information and real information, and both virtual information and real information include two dimensions of audio and picture.

Continuing to refer to FIG. 3 , in step S320 , in response to an interactive operation acting on the audio device or the display device, the mode information of the audio device is switched to determine target mode information.

In the embodiment of the present disclosure, the audio device may be an earphone, specifically a noise-cancelling earphone, for example, the active noise-cancelling earphone shown in FIG. 4 . Active noise-cancelling headphones may include in-ear noise-cancelling headphones or head-mounted noise-cancelling headphones, which are not specifically limited here. Active noise canceling headphones can achieve active noise cancellation. Active noise reduction refers to sending out completely opposite sound waves according to the specified noise, so that the two sound waves cancel each other out when they meet, thereby isolating the noise. In the embodiment of the present disclosure, the active noise reduction earphone collects external noise signals through the microphone, and then transmits sound waves opposite to the noise through the earphone in real time, thereby canceling the noise falling into the ear.

Audio devices can be used to play virtual audio and real environment audio, in addition to receiving interactive operations. The mode information of an audio device refers to its noise canceling mode. Referring to FIG. 4 , the noise reduction mode may include four modes: a strong noise reduction mode, a weak noise reduction mode, a transparent mode, and an off mode. The strong noise reduction mode refers to the highest level of reverse sound waves, so as to block the influence of external environmental sounds to the greatest extent and ensure the greatest sense of immersion. The weak noise reduction mode refers to a hybrid mode. In the weak noise reduction mode, the level of the reverse sound wave is weakened, and the real audio of the external environment can be vaguely heard when the virtual audio can be heard clearly. In the transparent mode, the reverse sound wave is not only turned off, but also through the mutual cooperation of the internal and external microphones, it can even eliminate the passive noise reduction effect brought by the silicone earplugs, so that the user can more intuitively collect the sound information of the surrounding environment. In the transparent mode, even with noise-cancelling headphones, you can clearly hear the sound of the external environment.

Based on the interactive operation on the audio device, the mode switching of the display device can be realized. If the audio device is a noise-canceling headset, the interactive operation may be determined according to the type of buttons included in the noise-canceling headset. Because the key types in audio equipment include touch-sensitive areas, physical keys, and physical knobs. Therefore, the interactive operation may include but not limited to a first audio control operation on the touch sensitive area, a second audio control operation on a physical button, and a third audio control operation on a physical knob.

When the interactive operation is received, the mode information of the audio device corresponding to the interactive operation can be determined, so as to complete the mode switching. Specifically, the mode switching interaction may be completed in response to the first audio control operation on the touch sensitive area. The first audio control operation may be, for example, single click, double click, continuous click (triple click), long press, or slide up and down. Wherein, the corresponding relationship between the first audio control operation and the mode may be: single click corresponds to open, double click is strong noise reduction mode, continuous click is weak noise reduction mode, long press or slide up and down is transparent mode. The mode switching interaction may also be accomplished in response to a second audio control operation on the physical key. The number of physical keys can be one or more. When there are multiple physical keys, each mode corresponds to one physical key, and the second audio control operation may be a pressing operation on each physical key. When the number of physical buttons is one, mode switching can be completed by single-clicking, double-clicking, continuous clicking, or long-pressing the physical button. That is, different mode switching is realized for different operation modes of the same physical key. For example, click to enable, double-click for strong noise reduction mode, continuous click for weak noise reduction mode, and long press for transparent mode. A mode switching interaction may also be implemented in response to a third audio control operation acting on the physical knob. The third audio control operation may be to rotate the physical knob clockwise or counterclockwise. For example, turn clockwise for strong noise reduction mode and weak noise reduction mode, and turn counterclockwise for transparency mode. In addition, the interaction mode and switching order of switching modes may also be determined according to actual needs and personal preferences, which are not limited here.

The display device may be any type of electronic device with a screen, such as a mobile phone, a tablet, and VR/AR glasses. The display device is used for displaying screen information and receiving interactive operations. The screen information may include virtual screens and real screens. The interactive operation is used to switch mode information, and the interactive operation can be specifically determined according to the type of the display device. Since the types of display devices may include touch devices, wearable devices, and peripheral devices, interactive operations may include, but are not limited to, the first screen control operation on the display screen of the touch device, the display of wearable devices The second picture control operation of the screen, the third picture control operation of the peripheral device, and the fourth picture control operation of the peripheral device.

The first picture control operation may be a touch operation on the picture control control in the display screen of the touch device. The screen control control may be a UI control located at any position on the display screen and used for switching the execution mode. The touch operation may include but not limited to a click operation or a press operation and the like. The second picture control operation may be a gesture operation on a picture control control in the display screen of the wearable device. The gesture operation may be a 3D gesture, such as waving or clapping hands. When the gesture operation is captured and understood by the environment understanding system, the mode switching interaction can be completed. The second screen control operation is suitable for wearable devices (such as VR/AR glasses). The third screen control operation may be a pressing operation of a physical button in the peripheral device. The peripheral device is connected with the display device, and the picture control controls on the display screen of the display device are controlled through the physical keys of the peripheral device. Specifically, the peripheral device may be an operation handle, and types of the operation handle include hand-held handles and wearable handles. The operating handle can output a ray pointer. When the ray pointer is located on the screen control control on the display screen, the user can complete the mode switching interaction by pressing the physical button on the operating handle connected to the display device. It is suitable for wearable devices (VR/AR) . The fourth screen control operation may be a pressing operation of a physical button in a peripheral device combined with the user's line of sight. The peripheral device is connected with the display device, and the picture control controls on the display screen of the display device are controlled through the physical keys of the peripheral device. Specifically, the display device worn on the user's head has an eye-tracking device on the side close to the user. The eye-tracking device locates the user's precise line of sight direction by capturing the infrared reflection of the user's pupil, and can output an image that matches the user's line of sight by capturing the user's eye tracking. Pointers in the same direction. When the pointer is located on the corresponding screen control control on the display screen, if it is detected that the user presses the physical button on the operating handle connected to the display device, the mode switching interaction is completed, which is suitable for wearable devices (VR/AR).

In the embodiments of the present disclosure, through interactive operations acting on the audio device and the display device, the display state of the display device can be switched between modes to obtain target mode information. The interactive operation of display devices and audio devices can complement each other, avoiding the limitation of mode switching in only one way, and can switch modes according to the interactive operations on other types of devices when it is inconvenient to operate display devices or audio devices , thus improving the convenience of mode switching. Moreover, compared with related technologies, the target mode information can include a variety of different modes, not just the virtual reality state and the augmented reality state, but also the weak noise reduction mode, so it can realize multi-mode switching, improve diversity and operation Comprehensiveness increases the scope of application.

Continuing to refer to FIG. 3 , in step S330, the real environment audio in the real information is adjusted according to the target mode information, and the real picture is adjusted according to the target mode information, so as to The augmented reality state and the virtual reality state are switched.

In the embodiment of the present disclosure, since the actual information corresponding to each mode information is different, when the mode information of the display device is switched in response to the interactive operation acting on the audio device or the display device to determine the target mode information, it is necessary to determine the target mode information according to the target The mode information simultaneously adjusts the real picture and the real environment audio in the real information. When adjusting the audio of the real environment, its volume information can be adjusted. When adjusting the real picture, the display brightness and definition of the real picture can be adjusted at the same time.

Figure 5 schematically shows a flow chart for adjusting real-world audio. Referring to Figure 5, it mainly includes the following steps:

In step S510, if the level information is the first level, adjust the volume information to a maximum volume value;

In step S520, if the level information is the second level, reduce the volume information to an intermediate volume value;

In step S530, if the level information is the third level, adjust the volume information to a minimum volume value.

In the embodiment of the present disclosure, after the target mode information is determined according to the interactive operation, the real environment audio in the real information may be adjusted according to the degree information of the target mode information. The degree information refers to the proportion of the actual information in the target mode information, and the first degree is greater than the second degree and greater than the third degree. Volume information is positively correlated with degree information. When the target mode information is the transparent mode, the degree information is the first degree, that is, the actual information occupies the largest proportion. When the target mode information is the weak noise reduction mode, the degree information is the second degree, that is, the proportion occupied by the actual information decreases. When the target mode information is the strong noise reduction mode, the degree information is the third degree, that is, the proportion occupied by the actual information is the smallest. For real environment audio, the volume information of the real environment audio can be adjusted.

Based on this, when the target mode information is the transparent mode, the volume information of the real environment audio can be increased to the maximum volume value because the proportion occupied by the real information is the largest. When the target mode information is the weak noise reduction mode, since the proportion occupied by the real information is smaller than the maximum value, the volume information of the real environment audio can be reduced to an intermediate volume value. The intermediate volume value may be greater than the minimum volume value and less than the maximum volume value, and the intermediate volume value may be any value between 25% and 75% of the maximum volume value, for example, 50%. When the target mode information is the strong noise reduction mode, since the real information occupies the smallest proportion, the volume information of the real environment audio can be reduced to a minimum volume value, and the minimum volume value can be 0. In the strong noise reduction mode, only the virtual audio in the virtual information can be heard, but the real environment audio cannot be heard.

Figure 6 schematically shows a flow chart for adjusting the real picture, referring to Figure 6, it mainly includes the following steps:

In step S610, if the degree information is the first degree, determine the display brightness as a maximum brightness value, and determine the clarity as a maximum clarity;

In step S620, if the degree information is the second degree, reduce the display brightness to an intermediate brightness value, and reduce the sharpness to obtain an intermediate sharpness;

In step S630, if the degree information is the third degree, reduce the display brightness to a minimum brightness value, and reduce the resolution to a minimum resolution.

In the embodiment of the present disclosure, both display brightness and clarity are positively correlated with degree information. When the target mode information is the transparent mode, the real information and the virtual information can be displayed simultaneously, and the occupation ratio of the real information can be greater than or equal to the occupation ratio of the virtual information. Since the real information occupies the largest proportion at this time, the display brightness of the real picture can be increased to the maximum brightness value, and the definition can be adjusted to the maximum definition. When the target mode information is the weak noise reduction mode, since the proportion occupied by the real information is reduced, the display brightness of the real environment audio can be reduced to an intermediate brightness value, and the definition can be reduced to an intermediate definition. The middle brightness value can be greater than the minimum brightness value and smaller than the maximum brightness value, and the middle resolution is larger than the minimum resolution and smaller than the maximum resolution. The intermediate brightness value may be any value between 25% and 75% of the maximum brightness value, for example, 50%. Intermediate resolution can be anywhere between 25% and 75% of maximum resolution. Also, the adjustment degrees of display brightness and clarity can be the same or different. In this mode, virtual images and real images can be displayed at the same time. When the target mode information is the strong noise reduction mode, since the proportion of the real information is the smallest, the display brightness of the real picture can be reduced to the minimum brightness value and the definition can be reduced to the minimum definition, the minimum brightness value and the minimum clarity degree can be 0. In the strong noise reduction mode, only the virtual picture is displayed, not the real picture. Because in the strong noise reduction mode, only virtual information can be displayed but not real information, so the display device is in a state of virtual reality. It should be noted that during the whole process of mode information switching, the state of the virtual audio and virtual picture contained in the virtual information remains unchanged, the volume information of the virtual audio can be the maximum volume value, and the display brightness of the virtual picture can be the maximum brightness. value, the resolution of the virtual screen can be the maximum resolution.

By adjusting the attribute information of the real environment audio and the real picture according to the target mode information switched by the audio device, the transition between the augmented reality state and the virtual reality state is realized from two dimensions of audio and video. It avoids the limitations of the related technologies that can only switch between different modes according to one dimension of the screen, and can accurately realize the mode switching between different modes from multiple dimensions, and realize the complete switching from the augmented reality state to the virtual reality state, which increases the overall and scope of application. In addition, adjusting the real-world audio and real-world images through interactive operations on audio devices or display devices can improve the accuracy of audio and image adjustments, and also improve the convenience and comprehensiveness of user operations. In addition, since the real audio and the real picture can be adjusted at the same time, the problem of out-of-sync audio and video can be avoided, the synchronous switching of the audio and picture can be realized, and the display effect can be improved.

In the embodiment of the present disclosure, if the display device communicates with the audio device through the Bluetooth communication protocol in the communication device, and the camera on the display device has started the recording function, it is used to transmit the real picture captured by the display device to the display screen . At the same time, the environment understanding system of the display device performs real-time three-dimensional modeling of the real environment through SLAM technology, and superimposes virtual information on the real picture to determine the display state of the display device as an augmented reality state. Based on this, it is possible to display real pictures and real environment audio in the augmented reality state. During the whole switching process, the virtual audio and the virtual picture corresponding to the virtual information remain unchanged. For example, the virtual audio can be the maximum volume value, and the virtual picture can keep the maximum brightness and maximum clarity unchanged.

Figure 7 schematically shows a comparison diagram of state switching. Referring to Figure 7, in terms of modes, device 1 and device 2 only provide two states of AR and VR, and a weak noise reduction mode is added to the embodiment of the present disclosure. , in the weak noise reduction mode, the user can focus on the virtual information while feeling the necessary real information. In terms of audio and video synchronization, device 1 and device 2 only provide the AR/VR mode switching of the screen. The embodiment of the present disclosure adds the dimension of sound, and at the same time, the screen and sound are in a one-to-one correspondence in different noise reduction modes, providing users with Multi-modal mode switching effect. In terms of interaction methods, device 1 only provides switching of VR glasses with pointing handles or 3D gestures, and device 2 only provides switching of screen touch methods, and neither of them considers the linkage of screen and sound when switching. Embodiments of the present disclosure provide a variety of switching interaction modes: including screen touch in the display device interaction system, 3D gesture capture, pointing handle, eye tracking + handle switching interaction mode, and touch sensing on the active noise reduction headset, The switching interaction mode of buttons and knobs increases the diversity of operations, improves the diversity and comprehensiveness of switching methods, increases the category of mode information, and can realize the complete switching of multiple modes.

If the user needs to pay attention to the real information, the target mode information is a transparent mode to strengthen the real information. Referring to FIG. 8 , the user can switch to the see-through mode 801 through the interaction system of the display device or the interaction system of the active noise reduction earphone 802 . In see-through mode, the display device generates virtual information (eg, a virtual puppy). The virtual information includes virtual audio 803 (the sound of the puppy) and virtual screen 804 (the image of the puppy). And the virtual information can be superimposed on the captured real information, and the real information includes real environment audio 805 (recorded real environment sound in real time) and real picture 806 (such as tables, sofas, etc.). Wherein, the display brightness and definition of the real environment audio and the real picture are the largest, for example, both are 100%.

If the user needs to focus on virtual information and real information at the same time, the target mode information is a weak noise reduction mode to weaken the real information. When the target mode information is the weak noise reduction mode, it is still in the augmented reality state, but the reality information needs to be weakened. Referring to FIG. 9 , the user can switch to the weak noise reduction mode 901 through the interaction system of the display device or the interaction system of the active noise reduction earphone 902 . In the weak noise reduction mode, the virtual audio 903 (the sound of the puppy) and the virtual picture 904 (the image of the puppy) can be superimposed on the captured real information, and the real information includes the real environment audio 905 (real environment sound recorded in real time) ) and a real picture 906 (for example, a table, a sofa, etc. therein). Wherein, the display brightness and definition of the audio in the real environment and the real picture are both reduced, for example, both are 25% to 75%.

If the user does not need to pay attention to real information but only needs to pay attention to virtual information, the target mode information is a strong noise reduction mode to completely shield the real information. When the target mode information is the strong noise reduction mode, it is in a virtual reality state, that is, no real information is displayed. Referring to FIG. 10 , the user can switch to the strong noise reduction mode 1001 through the interaction system of the display device or the interaction system of the active noise reduction earphone 1002 . In the strong noise reduction mode, the virtual audio 1003 (the sound of the puppy) and the virtual picture 1004 (the image of the puppy) can be superimposed on the captured real information, and the real information includes the real environment audio 1005 (real environment sound recorded in real time) ) and a real picture 1006 (for example, a table, a sofa, etc. therein). Among them, the display brightness and clarity of the real environment audio and the real picture are reduced to 0, and only virtual information can be displayed.

The technical solution provided by the embodiments of the present disclosure takes into account the picture-sound correspondence in different noise reduction states, and realizes the function of switching AR/VR modes by integrating visual and auditory modalities. With the multi-modal audio-visual transparency mode, the picture and sound information in the real environment are enhanced. Even if the user wears a heavy headset and earphones that block external sound, he can clearly see and hear the picture and sound information in the real environment. Sound for increased immersion. Cooperating with the multi-modal sound and picture weak noise reduction mode, the user can clearly see and hear the virtual generated picture and sound while weakening the picture and sound of the real environment, and can experience virtual information more focusedly while ensuring safe use . Cooperating with the multi-modal audio-visual strong noise reduction mode, it can completely shield the pictures and sounds in the real environment, and users can experience a more immersive and focused VR effect. It realizes the synchronous switching of picture and sound, and provides a convenient and comprehensive switching interaction method to switch between different levels of noise reduction modes.

In the embodiment of the present disclosure, a display state switching device is also provided. As shown in FIG. 11 , the display state switching device 1100 specifically includes:

The state acquiring module 1101 is configured to acquire the display state of the display device, the display state is an augmented reality state including virtual information and real information; the virtual information includes virtual audio and virtual picture, and the real information includes real environment audio and realistic images;

The mode switching module 1102 is configured to switch the mode information of the audio device and determine the target mode information in response to an interactive operation acting on the audio device or the display device;

The audio picture adjustment module 1103 is configured to adjust the real environment audio in the real information according to the target mode information, and adjust the real picture according to the target mode information, so as to adjust the enhanced Switch between real state and virtual reality state.

In an exemplary embodiment of the present disclosure, the adjusting the real environment audio in the real information according to the target mode information includes: a volume adjustment module configured to correspond to the audio volume according to the target mode information The level information of the real environment audio is adjusted.

In an exemplary embodiment of the present disclosure, the volume adjustment module includes: a first volume adjustment module configured to adjust the volume information to a maximum volume value if the level information is the first level; The adjustment module is configured to reduce the volume information to an intermediate volume value if the level information is the second level; the third volume adjustment module is configured to reduce the volume information to the third level if the level information is the third level. The information is reduced to the minimum volume value.

In an exemplary embodiment of the present disclosure, the picture adjustment module includes: a display information adjustment module configured to adjust the display brightness and clarity of the real picture according to the level information of the target mode information.

In an exemplary embodiment of the present disclosure, the display information adjustment module includes: a first display adjustment module configured to determine the display brightness as a maximum brightness value if the degree information is a first degree, and set The sharpness is determined to be the maximum sharpness; the second display adjustment module is configured to reduce the display brightness to an intermediate brightness value if the degree information is the second degree, and reduce the sharpness to obtain intermediate sharpness degree; a third display adjustment module configured to reduce the display brightness to a minimum brightness value and reduce the definition to a minimum definition if the degree information is a third degree.

In an exemplary embodiment of the present disclosure, the device further includes: a synchronous adjustment module configured to synchronously adjust the real environment audio and the real picture, and convert the virtual Audio and virtual picture remain unchanged.

It should be noted that the specific details of each module in the display state switching device have been described in detail in the corresponding display state switching method, so details will not be repeated here.

In the embodiment of the present disclosure, a display state switching system is also provided. As shown in FIG. 12 , the display state switching system 1200 specifically includes:

The audio device 1201 is configured to determine the mode information in response to the interactive operation, and adjust the audio of the real environment in the real information included in the display state according to the mode information;

a communication device 1202, configured to connect the audio device and the display device;

The display device 1203, connected to the audio device, is configured to display the augmented reality state including virtual information and real information, and adjust the real picture in the real information according to the mode information in response to the interactive operation, so as to enhance the augmented reality Switch between real state and virtual reality state.

In addition, the real state switching system further includes: a peripheral device 1204, connected to the display device through the communication device 1202, and configured to receive interactive operation determination target mode information.

FIG. 13 shows a schematic diagram of an electronic device suitable for implementing an exemplary embodiment of the present disclosure. The terminal of the present disclosure can be configured in the form of an electronic device as shown in FIG. 13. However, it should be noted that the electronic device shown in FIG. any restrictions.

The electronic device of the present disclosure includes at least a processor and a memory, and the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can realize the method of the exemplary embodiment of the present disclosure.

Specifically, as shown in Figure 13, the electronic device 1300 may include: a processor 1310, an internal memory 1321, an external memory interface 1322, a Universal Serial Bus (Universal Serial Bus, USB) interface 1330, a charging management module 1340, and a power management module 1341, battery 1342, antenna 1, antenna 2, mobile communication module 1350, wireless communication module 1360, audio module 1370, speaker 1371, receiver 1372, microphone 1373, earphone jack 1374, sensor module 1380, display screen 1390, camera module 1391 , an indicator 1392, a motor 1393, a button 1394, and a Subscriber Identification Module (Subscriber Identification Module, SIM) card interface 1395, etc. The sensor module 1380 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor.

It can be understood that, the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 1300 . In other embodiments of the present application, the electronic device 1300 may include more or fewer components than shown in the illustration, or combine certain components, or separate certain components, or arrange different components. The illustrated components may be realized in hardware, software, or a combination of software and hardware.

The processor 1310 may include one or more processing units, for example: the processor 1310 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor , a baseband processor and/or a neural network processor (Neural-network Processing Unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors. In addition, a memory may also be provided in the processor 1310 for storing instructions and data. The display state switching method in this exemplary embodiment may be executed by an application processor, a graphics processor or an image signal processor, and when the method involves processing related to a neural network, it may be executed by an NPU.

The internal memory 1321 may be used to store computer-executable program codes including instructions. The internal memory 1321 may include an area for storing programs and an area for storing data. The external memory interface 1322 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1300.

The communication function of the mobile terminal 1300 can be realized by the mobile communication module, the antenna 1, the wireless communication module, the antenna 2, the modem processor and the baseband processor. Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1400 . The wireless communication module can provide wireless communication solutions such as wireless local area network, bluetooth, and near field communication applied to the mobile terminal 200 .

The display screen is used to realize a display function, such as displaying a user interface, images, videos, and the like. The camera module is used to realize a shooting function, such as taking images, videos, and the like. The audio module is used to implement audio functions, such as playing audio, collecting voice, etc. The power module is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring the battery status.

The present application also provides a computer-readable storage medium. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or exist independently without being assembled into the electronic device.

A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The computer-readable storage medium may send, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wires, optical cables, RF, etc., or any suitable combination of the foregoing.

The computer-readable storage medium bears one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device is made to implement the methods described in the following embodiments.

Through the description of the above implementations, those skilled in the art can easily understand that the example implementations described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of software products, and the software products can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

In addition, the above-mentioned drawings are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not imply or limit the chronological order of these processes. In addition, it is also easy to understand that these processes may be executed synchronously or asynchronously in multiple modules, for example.

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. Actually, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be embodied by a plurality of modules or units.

Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with the true scope and spirit of the disclosure indicated by the appended claims. It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

A display state switching method, comprising:

Obtain the display state of the display device, the display state is an augmented reality state including virtual information and real information; the virtual information includes virtual audio and virtual picture, and the real information includes real environment audio and real picture;

In response to an interactive operation acting on the audio device or the display device, switch the mode information of the audio device to determine target mode information;

The real environment audio in the real information is adjusted according to the target mode information, and the real picture is adjusted according to the target mode information, so as to switch between the augmented reality state and the virtual reality state.
The display state switching method according to claim 1, wherein said adjusting said real environment audio in said real information according to said target mode information comprises:

The volume information of the real environment audio is adjusted according to the level information corresponding to the target mode information.
The display state switching method according to claim 2, wherein the adjusting the volume information of the real environment audio according to the degree information corresponding to the target mode information includes:

If the level information is the first level, adjusting the volume information to a maximum volume value;

If the level information is the second level, reduce the volume information to an intermediate volume value;

If the degree information is the third degree, reduce the volume information to a minimum volume value.
The display state switching method according to claim 1, wherein said adjusting said real picture according to said target mode information comprises:

According to the level information of the target mode information, the display brightness and definition of the real picture are adjusted.
The display state switching method according to claim 4, wherein the adjusting the display brightness and clarity of the real picture according to the level information of the mode information includes:

If the degree information is the first degree, determine the display brightness as a maximum brightness value, and determine the clarity as a maximum clarity;

If the degree information is the second degree, reduce the display brightness to an intermediate brightness value, and reduce the sharpness to obtain an intermediate sharpness;

If the degree information is the third degree, reduce the display brightness to a minimum brightness value, and reduce the definition to a minimum definition.
The display state switching method according to claim 1, wherein the interactive operation is determined according to the type of keys included in the audio device or according to the type of the display device.
The display state switching method according to claim 6, wherein the interactive operations of the display device and the audio device complement each other.
The display state switching method according to claim 6, wherein the key type in the audio device includes any one of a touch-sensitive area, a physical key, and a physical knob; the type of the display device includes a touch device, an adjustable Any one of wearable devices and peripheral devices.
The display state switching method according to claim 3, wherein the level information represents the proportion of the actual information in the target mode information, and the first level is greater than the second level and greater than the third level.
The display state switching method according to claim 2, wherein the volume information is positively correlated with the level information.
The display state switching method according to claim 4, wherein both the display brightness and the clarity are positively correlated with the level information.
The display state switching method according to claim 1, wherein the method further comprises:

The real environment audio and the real picture are adjusted synchronously, and the virtual audio and the virtual picture included in the virtual information remain unchanged.
The display state switching method according to claim 1, wherein the target mode information is any one of a transparent mode, a weak noise reduction mode, and a strong noise reduction mode.
The display state switching method according to claim 13, wherein in the transparent mode, the proportion of real information occupied is the largest, in the strong noise reduction mode, the proportion of real information is the smallest, and in the weak noise reduction mode, the proportion of real information is the largest. The occupied ratio is larger than the strong noise reduction mode and smaller than the transparent mode.
A display state switching device, comprising:

A state acquiring module configured to acquire the display state of the display device, the display state being an augmented reality state including virtual information and real information; the virtual information includes virtual audio and virtual picture, and the real information includes real environment audio and realistic picture;

The mode switching module is configured to switch the mode information of the audio device to determine the target mode information in response to an interactive operation acting on the audio device or the display device;

The audio picture adjustment module is configured to adjust the audio of the real environment in the reality information according to the target mode information, and adjust the real picture according to the target mode information, so as to adjust the augmented reality state and virtual reality state to switch.
The display state switching device according to claim 15, wherein the audio picture adjustment module comprises:

The volume adjustment module is configured to adjust the volume information of the real environment audio according to the level information corresponding to the target mode information.
The display state switching device according to claim 16, wherein the audio adjustment module comprises:

The first volume adjustment module is configured to adjust the volume information to a maximum volume value if the level information is the first level;

The second volume adjustment module is configured to reduce the volume information to an intermediate volume value if the level information is the second level;

The third volume adjustment module is configured to reduce the volume information to a minimum volume value if the level information is a third level.
A display state switching system, comprising:

An audio device configured to determine target mode information in response to an interactive operation, and play real environment audio in the real information adjusted according to the target mode information;

a communication device configured to connect the audio device and the display device;

The display device, connected to the audio device, is configured to determine the target mode information in response to the interactive operation, and display the reality picture in the reality information adjusted according to the target mode information, so as to show the augmented reality state and the virtual reality state to switch.
An electronic device comprising:

processor; and

a memory configured to store executable instructions of the processor;

Wherein, the processor is configured to execute the display state switching method described in any one of claims 1-14 by executing the executable instructions.
A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the display state switching method described in any one of claims 1-14 is implemented.