TWI715474B - Method for dynamically adjusting camera configuration, head-mounted display and computer device - Google Patents

Abstract

The invention provides a method for dynamically adjusting the camera configuration, a head-mounted display and a computer device. The method includes: obtaining a moving speed of the head-mounted display in one degree of freedom; and in response to determining that the moving speed has changed to be higher than a speed threshold, adjusting each front camera to a first mode, wherein each front camera in the first mode has a first resolution and a first frame rate; in response to determining that the moving speed has changed to be not higher than the speed threshold, adjusting each front camera to a second mode; controlling each adjusted front camera to shoot a front scene as visual content and display the visual content.

Description

Method for dynamically adjusting lens configuration, head-mounted display and computer device

The present invention relates to an augmented reality technology, and particularly relates to a method for dynamically adjusting lens configuration, a head-mounted display and a computer device.

With the development of technology, virtual reality (VR) technology has been widely used in simulation operations in various fields, such as aviation training, e-sports games, medical and industrial education and training, so as to help trainees Simulate and repeatedly operate technologies in various fields, thereby reducing difficult or costly work in the real world. However, even though a considerable number of manufacturers have invested a lot of resources in the research and development of VR technology, there are still many areas to be improved in VR technology, and the most criticized one is that the frame rate is too low, causing users to use it. It is easy to produce dizziness and nausea.

For example, when a user is using a VR head-mounted display, there is a 57 millisecond delay from when the head is moved to when photons are emitted to the user’s eyes (ie, the user actually sees the VR content) time. Since this delay time does not actually exist in the real environment, it may cause the user to feel unreal or uncomfortable. Studies have shown that to reduce this discomfort, the above-mentioned delay time should be less than 20 milliseconds.

However, with the current technology, most of the time is spent on GPU processing the picture and transmitting the picture back to the head-mounted display (about 26 milliseconds).

In addition, in VR-related applications, augmented reality (AR) is also one of the attention-grabbing directions, that is, the external environment is captured through the front lens of the head-mounted display and presented as visual content for users to watch. , And then present the virtual objects in the visual content. However, in AR related applications, the problems derived from the aforementioned delay time still exist. Therefore, for those skilled in the art, how to design a technology that can reduce the user's discomfort when using a head-mounted display is really an important issue.

In view of this, the present invention provides a method for dynamically adjusting lens configuration, a head-mounted display and a computer device, which can be used to solve the above technical problems.

The present invention provides a method for dynamically adjusting lens configuration, which is suitable for a head-mounted display with at least one front lens. The method includes: obtaining a moving speed of the head-mounted display in one degree of freedom; in response to determining that the moving speed is changed to be higher than a speed threshold, adjusting each front lens to a first mode, wherein each front lens is The first mode has a first resolution and a first frame rate; in response to determining that the movement speed has not changed above the speed threshold, each front lens is adjusted to a second mode, where each front lens is in the second mode It has a second resolution and a second frame rate, the second resolution is higher than the first resolution, and the second frame rate is lower than the first frame rate; control the adjusted front lens to shoot a front view as One visual content, and display visual content.

The invention provides a head-mounted display, which includes at least one front lens, a storage circuit and a processor. The storage circuit stores multiple modules. The processor is coupled to at least one front lens and a storage circuit, and accesses the aforementioned module to perform the following steps: obtain a moving speed of the head mounted display in one degree of freedom; respond to determining that the moving speed is changed to be higher than a speed Threshold, adjust each front lens to a first mode, where each front lens has a first resolution and a first frame rate in the first mode; in response to determining that the movement speed has not changed above the speed threshold, Adjust each front lens to a second mode, where each front lens has a second resolution and a second frame rate in the second mode, the second resolution is higher than the first resolution, and the second frame rate is lower At the first frame rate; control the adjusted front lenses to shoot a front view as a visual content, and display the visual content.

The invention provides a computer device, which includes a storage circuit and a processor. The storage circuit stores multiple modules. The processor is coupled to the storage circuit and accesses the aforementioned module to perform the following steps: obtain a moving speed of a head-mounted display in a degree of freedom, wherein the head-mounted display includes at least one front lens; When the speed is changed to be higher than a speed threshold, each front lens is adjusted to a first mode, where each front lens has a first resolution and a first frame rate in the first mode; in response to determining that the moving speed is changed to Not higher than the speed threshold, adjust each front lens to a second mode, where each front lens has a second resolution and a second frame rate in the second mode, and the second resolution is higher than the first resolution , And the second frame rate is lower than the first frame rate; each front lens after adjustment is controlled to shoot a front view as a visual content, and the visual content is displayed.

Based on the above, the present invention can dynamically switch between the first mode (high frame rate/low resolution) and the second mode (low frame rate/high resolution) according to the moving speed of the head-mounted display, thereby reducing Time delay and improve the effect of user experience.

Please refer to FIG. 1, which is a schematic diagram of a head-mounted display according to an embodiment of the present invention. In different embodiments, the head-mounted display 100 may be used to provide VR/AR content to the user, for example. In some embodiments, the head-mounted display 100 can be connected to a computer device, and the computer device can provide related VR/AR content to the head-mounted display 100, and then the head-mounted display 100 can be presented to the user for viewing . In other embodiments, the head-mounted display 100 can also be an all-in-one (AIO) head-mounted display, which can also generate and provide VR/AR content for users to watch, without additional Connect the above-mentioned computer device, but it is not limited to this.

As shown in FIG. 1, the head-mounted display 100 may include a storage circuit 102, a processor 104, and front lenses 1061 to 106N. The storage circuit 102 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hard disk Disk or other similar devices or a combination of these devices can be used to record multiple codes or modules.

The front lenses 1061 to 106N can be fixed on the head-mounted display 100 and can be used to shoot the scene in front of the head-mounted display 100. In some embodiments, the head-mounted display 100 can present the front view captured by the front lenses 1061 to 106N as visual content for the user to view. In addition, the head-mounted display 100 (or the aforementioned computer device) can also superimpose related AR objects/contents on the aforementioned visual content, thereby providing the AR experience required by the user, but it is not limited to this.

The processor 104 is coupled to the storage circuit 102 and the front lens 1061~106N, and can be an image signal processor (ISP), a general purpose processor, a special purpose processor, a traditional processor, and a digital signal processor , Multiple microprocessors (microprocessors), one or more microprocessors, controllers, microcontrollers, special application integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gates combined with digital signal processor core Field Programmable Gate Array (FPGA), any other types of integrated circuits, state machines, processors based on Advanced RISC Machines (ARM), and similar products.

In the embodiment of the present invention, the processor 104 can access the modules and program codes recorded in the storage circuit 102 to implement the method for dynamically adjusting the lens configuration proposed in the present invention. The details are described below.

Please refer to FIG. 2, which is a flowchart of a method for dynamically adjusting lens configuration according to an embodiment of the present invention. The method of this embodiment can be executed by the head-mounted display 100 in FIG. 1. The details of each step in FIG. 2 will be described below in conjunction with the components shown in FIG.

First, in step S210, the processor 104 may obtain the moving speed of the head-mounted display 100 in a degree of freedom (DOF). In different embodiments, the aforementioned degrees of freedom may include at least one of a plurality of translational degrees of freedom and a plurality of rotational degrees of freedom, and the processor 104 may use the inertial sensor and/or acceleration in the head-mounted display 100 The sensor obtains the aforementioned moving speed, but it is not limited to this.

Please refer to FIG. 3, which is a schematic diagram of 6DOF according to an embodiment of the present invention. As shown in FIG. 3, the degree of freedom considered in the present invention may be one or more of the three translational degrees of freedom and the three rotational degrees of freedom shown. Specifically, the three translational degrees of freedom are, for example, translational degrees of freedom on the X, Y, and Z axes, and the three rotational degrees of freedom are, for example, rotational degrees of freedom on the X, Y, and Z axes (commonly known as Roll, pitch and yaw), but not limited to this.

For ease of description, the following description will be based on only one degree of freedom, but the present invention may not be limited to this. Based on this, in step S210, assuming that the degree of freedom considered is the translational degree of freedom on the X axis, the processor 104 can detect the moving speed of the head-mounted display 100 in this degree of freedom, that is, the head-mounted display 100 The speed of moving forward and backward along the X axis. In addition, assuming that the degree of freedom considered is the rotational degree of freedom on the Y axis, the processor 104 may detect the moving speed of the head-mounted display 100 on this degree of freedom in step S210, that is, the head-mounted display 100 moves along the Y axis. The speed at which the axis pitches up and down, but it is not limited to this.

After obtaining the moving speed of the head-mounted display 100 in the considered degree of freedom, the processor 104 can determine whether the moving speed is changed from not higher than a speed threshold to higher than the above-mentioned speed threshold, or from higher than The above-mentioned speed threshold is changed to be not higher than the above-mentioned speed threshold. In different embodiments, the value of the adopted speed threshold can be set to different values according to the degree of freedom considered, but it is not limited to this.

In an embodiment, if the moving speed of the head-mounted display 100 in the aforementioned degrees of freedom is not higher than the aforementioned speed threshold, the head-mounted display 100 at this time can be said to be in a static state. Conversely, if the moving speed of the head-mounted display 100 in the aforementioned degree of freedom is higher than the aforementioned speed threshold, the head-mounted display 100 at this time can be said to be in a moving state. In this case, the above-mentioned speed threshold can be selected as a value for separating the static state/moving state according to the designer's knowledge, but it is not limited to this.

In one embodiment, in response to determining that the movement speed has changed to be higher than the speed threshold, the processor 104 may adjust each front lens 1061 to 106N to the first mode in step S220, wherein each front lens 1061 to 106N is in the first mode. The mode has the first resolution and the first frame rate.

In another embodiment, in response to determining that the movement speed has not changed above the speed threshold, the processor 104 may adjust each front lens to the second mode in step S230, wherein each front lens 1061 to 106N is in the second mode. Has a second resolution and a second frame rate, the second resolution is higher than the first resolution, and the second frame rate is lower than the first frame rate.

After that, in step S240, the processor 104 may control the adjusted front lenses 1061 to 106N to capture the front view as the visual content, and display the visual content.

In detail, when the user moves his head and causes the head-mounted display 100 to move at a high speed in the aforementioned degrees of freedom, each front lens 1061~106N needs a higher frame rate to ensure a lower delay time . At the same time, in order to effectively reduce the amount of data, the front lens 1061~106N can be used for shooting with a lower resolution at this time. Therefore, when the processor 104 determines that the head-mounted display 100 changes from a static state to a moving state, the processor 104 can switch the front lenses 1061 to 106N to the first mode, and then control the front lenses 1061 to 106N to be the first mode. The mode captures the front view as the visual content presented to the user. In this way, each of the front lenses 1061 to 106N can shoot in the first mode with a higher frame rate (for example, 120fps) and a lower resolution (for example, 720p), so as to prevent the user from being uncomfortable due to a long delay time.

On the other hand, when the user's head is relatively stationary, since the front view captured by the front lenses 1061 to 106N has a relatively low degree of change, a lower frame rate can be used for shooting. At the same time, in order to allow users to watch high-quality visual content in a static state, the front lenses 1061~106N can be used for shooting with higher resolution at this time. Therefore, when the processor 104 determines that the head mounted display 100 changes from the moving state to the static state, the processor 104 can switch the front lenses 1061 to 106N to the second mode, and then control the front lenses 1061 to 106N to perform the second mode. The mode captures the front view as the visual content presented to the user. In this way, each of the front lenses 1061 to 106N can shoot in the second mode with a lower frame rate (for example, 24fps) and a higher resolution (for example, 4000x3000), so that the user can watch more delicate visual content.

It can be seen from the above that the method of the present invention allows the head-mounted display to dynamically allow each front lens to shoot the front view in the first mode or the second mode according to its own moving speed in a certain degree of freedom as a view for the user. Visual content. In this way, the user can avoid discomfort due to the long delay time when moving the head, and allow the user to watch more delicate visual content when the head is stationary.

In one embodiment, after the front lenses 1061 to 106N are switched to the first mode, the processor 104 can continuously monitor the moving speed of the head mounted display 100 in the aforementioned degrees of freedom, if the moving speed is not changed to not high At the speed threshold, the processor 105 can maintain the front lenses 1061 to 106N in the first mode, and vice versa, it can switch the front lenses 1061 to 106N to the second mode. On the other hand, after the front lenses 1061 to 106N are switched to the second mode, the processor 104 can continuously monitor the moving speed of the head mounted display 100 in the aforementioned degrees of freedom, if the aforementioned moving speed has not changed to be higher than the speed threshold Value, the processor 105 can maintain the front lenses 1061 to 106N in the second mode, and vice versa, it can switch the front lenses 1061 to 106N to the first mode, but it is not limited to this.

In addition, in some embodiments, if the frame rate and/or resolution difference between the first mode and the second mode is large, the user may observe more abrupt visual changes. Therefore, the present invention also proposes a relatively smooth mode switching mechanism to avoid the above problems.

Specifically, in one embodiment, before the processor 104 switches the front lenses 1061 to 106N to the second mode, the processor 104 may first switch the front lenses 1061 to 106N to a third mode, and then switch the front lenses 1061 to 106N to a third mode. The front lenses 1061~106N switch to the second mode, where each front lens 1061~106N has a third resolution and a third frame rate in the third mode, and the third resolution is between the first resolution and the second resolution The third frame rate is between the first frame rate and the second frame rate.

Similarly, in an embodiment, before the processor 104 switches the front lenses 1061 to 106N to the first mode, the processor 104 may first switch the front lenses 1061 to 106N to a fourth mode, and then switch the front lenses 1061 to 106N to a fourth mode. The lenses 1061~106N switch to the first mode, where each front lens 1061~106N has a fourth resolution and a fourth frame rate in the fourth mode, and the fourth resolution is between the first resolution and the second resolution , And the fourth frame rate is between the first frame rate and the second frame rate.

Please refer to FIG. 4, which is a schematic diagram illustrating the resolution and frame rate of each mode according to an embodiment of the present invention. In this embodiment, it is assumed that the first resolution and first frame rate of the first mode can be characterized as "720P@120fps" in FIG. 4, and the second resolution and second frame rate of the second mode can be characterized as "4000x3000@24fps" in Figure 4.

In this case, in order to improve the smoothness of the mode switching, when the processor 104 switches the front lenses 1061~106N from the first mode to the second mode, the front lenses 1061~106N can be switched to the third mode first. (It can be characterized as 2248x1792@90fps), and then switch each front lens 1061~106N to the second mode to avoid allowing the user to observe more abrupt visual changes. In this embodiment, the third mode can be understood as a transition mode between the first mode and the second mode, but it may not be limited to this.

In addition, when the processor 104 switches each front lens 1061~106N from the second mode to the first mode, it can first switch each front lens 1061~106N to the fourth mode (which can have the same resolution as the third mode). And frame rate, but not limited to this), and then switch each front lens 1061~106N to the first mode to avoid allowing the user to observe more abrupt visual changes. In this embodiment, the fourth mode can be understood as a transition mode between the first mode and the second mode, but it may not be limited to this.

In addition, in other embodiments, more transition modes can be set between the first mode and the second mode to further improve the smoothness of the mode switching, but it is not limited to this.

In addition, the resolution and frame rate of each mode in FIG. 4 are only for example, and are not used to limit the possible implementation of the present invention. In different embodiments, the designer can determine the resolution and frame rate corresponding to each mode according to requirements. For example, in one embodiment, the first mode may use "1920x1080@90fps", the second mode may use "12M@20fps", and the third/fourth mode may use "2240x1792@60fps". But it is not limited to this.

In some embodiments, the processor 104 may further include synchronizing the shooting parameters of the front lenses 1061 to 106N during the movement of the head mounted display 100 to ensure that the front lenses 1061 to 106N have consistent shooting quality. In different embodiments, the aforementioned shooting parameters include at least one of an exposure value, a white balance value, and an auto focus parameter, but it may not be limited thereto.

In an embodiment, the processor 104 may first fix the shooting parameters of one of the front lenses 1061 to 106N, and set the shooting parameters of other front lenses accordingly to synchronize the shooting parameters of the front lenses 1061 to 106N, but it may not be limited to this. .

It should be understood that although the above embodiment only dynamically adjusts the shooting mode of the front lens based on the moving speed of a single degree of freedom, in other embodiments, the processor 104 can simultaneously monitor the head-mounted display 100 in multiple degrees of freedom. The shooting mode of each front lens is dynamically adjusted according to the moving speed corresponding to any degree of freedom, but it is not limited to this.

Please refer to FIG. 5, which is a schematic diagram of a computer device according to an embodiment of the present invention. In different embodiments, the computer device 500 is, for example, a personal computer, a notebook computer, a server or other similar devices that can provide required VR/AR content to the head-mounted display 100 through USB3.0 or other similar interfaces. But it is not limited to this. As shown in FIG. 4, the computer device 500 may include a storage circuit 502 and a processor 504. For various possible implementations of the storage circuit 502 and the processor 504, reference may be made to the relevant descriptions of the storage circuit 102 and the processor 104, which are not described here. Repeat.

In an embodiment, the processor 504 can access the modules and program codes recorded in the storage circuit 502 to implement the method in FIG. 2. In other words, the various steps/operations performed by the processor 104 of the head mounted display 100 described above can be performed by the processor 504 of the computer device 500 instead. For related details, please refer to the description in the previous embodiment, which will not be repeated here.

In summary, the method of the present invention allows the head-mounted display to dynamically allow each front lens to shoot the front view in the first mode or the second mode according to its own moving speed in one or more degrees of freedom. The visual content that the viewer views. For example, when the moving speed of the head-mounted display in a certain degree of freedom becomes higher than the corresponding speed threshold, the present invention can use the front lens of the head-mounted display to have a higher frame rate and a lower resolution. The first mode is for shooting to avoid discomfort caused by the long delay time when the user moves the head. In addition, when the moving speed of the head-mounted display in a certain degree of freedom becomes not higher than the corresponding speed threshold, the present invention can make each front lens of the head-mounted display have a lower frame rate and a higher resolution. The second mode of shooting allows the user to watch more delicate visual content with his head still.

In addition, the present invention further provides one or more transition modes between the first mode and the second mode, so as to improve the smoothness of the switching between the first mode and the second mode.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be determined by the scope of the attached patent application.

100: Head-mounted display 102, 502: storage circuit 104, 504: processor 1061~106N: Front lens 500: computer device S210~S240: steps

FIG. 1 is a schematic diagram of a head-mounted display according to an embodiment of the present invention. 2 is a flowchart of a method for dynamically adjusting lens configuration according to an embodiment of the invention. Fig. 3 is a schematic diagram of 6DOF drawn according to an embodiment of the present invention. FIG. 4 is a schematic diagram illustrating the resolution and frame rate of each mode according to an embodiment of the present invention. FIG. 5 is a schematic diagram of a computer device according to an embodiment of the present invention.

S210~S240: steps

Claims (9)

A method for dynamically adjusting lens configuration, suitable for a head-mounted display with at least one front lens, includes: obtaining a moving speed of the head-mounted display in a degree of freedom; responding to determining that the moving speed is changed to be higher than A speed threshold is used to adjust each of the front lenses to a first mode, wherein each of the front lenses has a first resolution and a first frame rate in the first mode; in response to determining that the moving speed is changed to no Above the speed threshold, first adjust each front lens to a third mode, and then adjust each front lens to a second mode, wherein each front lens has a second resolution in the second mode And a second frame rate, the second resolution is higher than the first resolution, and the second frame rate is lower than the first frame rate, each of the front lenses has a third resolution in the third mode And a third frame rate, the third resolution is between the first resolution and the second resolution, and the third frame rate is between the first frame rate and the second frame rate; Each of the adjusted front lenses is controlled to shoot a front view as a visual content, and the visual content is displayed. The method according to claim 1, wherein before the step of adjusting each of the front lenses to the first mode, the method further comprises: adjusting each of the front lenses to a fourth mode, wherein each of the front lenses is in the The fourth mode has a fourth resolution and a fourth frame rate, the fourth resolution is between the first resolution and the second resolution, and the fourth frame rate is between the first Between the frame rate and the second frame rate. The method according to claim 1, further comprising synchronizing at least one shooting parameter of each front lens. The method according to claim 3, wherein the at least one shooting parameter includes at least one of an exposure value, a white balance value, and an auto focus parameter. The method according to claim 3, wherein the at least one front lens includes a first front lens and a second front lens, and the step of synchronizing the at least one shooting parameter of each front lens includes: obtaining and maintaining the first front lens A first shooting parameter of the front lens; a second shooting parameter of the second front lens is adjusted to the first shooting parameter. The method according to claim 1, wherein after the step of adjusting each of the front lenses to the first mode, the method further comprises: in response to determining that the moving speed has not changed to not higher than the speed threshold, maintaining each of the front lenses The lens is the first mode. The method according to claim 1, wherein after the step of adjusting each of the front lenses to the second mode, the method further comprises: in response to determining that the moving speed has not changed to be higher than the speed threshold, maintaining each of the front lenses This is the second mode. A head-mounted display includes: at least one front lens; a storage circuit storing a plurality of modules; a processor, coupled to the at least one front lens and the storage circuit, and accessing the modules to execute the following step: Obtain a moving speed of the head-mounted display in a degree of freedom; in response to determining that the moving speed has changed to be higher than a speed threshold, adjust each of the front lenses to a first mode, wherein each of the front lenses is in the The first mode has a first resolution and a first frame rate; in response to determining that the moving speed has not changed above the speed threshold, first adjust each front lens to a third mode, and then adjust each The front lens is adjusted to a second mode, wherein each of the front lenses has a second resolution and a second frame rate in the second mode, the second resolution is higher than the first resolution, and the second The frame rate is lower than the first frame rate, each of the front cameras has a third resolution and a third frame rate in the third mode, and the third resolution is between the first resolution and the second resolution And the third frame rate is between the first frame rate and the second frame rate; each of the front lenses after adjustment is controlled to shoot a front view as a visual content, and the visual content is displayed. A computer device includes: a storage circuit storing a plurality of modules; a processor coupled to the storage circuit and accessing the modules to perform the following steps: obtaining a head-mounted display in a degree of freedom The head-mounted display includes at least one front lens; in response to determining that the moving speed has changed to be higher than a speed threshold, first adjust each front lens to a third mode, and then adjust each front lens The lens is adjusted to a first mode, and each of the front lenses has a first resolution and a first mode in the first mode A frame rate, each of the front lenses has a third resolution and a third frame rate in the third mode, the third resolution is between the first resolution and the second resolution, and the The third frame rate is between the first frame rate and the second frame rate; in response to determining that the moving speed has not changed above the speed threshold, each of the front lenses is adjusted to a second mode, wherein each The front lens has a second resolution and a second frame rate in the second mode, the second resolution is higher than the first resolution, and the second frame rate is lower than the first frame rate; controlling Each of the adjusted front lenses shoots a front view as a visual content, and displays the visual content.

Images