WO2022100288A1 - 显示设备、手柄以及虚拟目标定位追踪的校准方法 - Google Patents
显示设备、手柄以及虚拟目标定位追踪的校准方法 Download PDFInfo
- Publication number
- WO2022100288A1 WO2022100288A1 PCT/CN2021/119626 CN2021119626W WO2022100288A1 WO 2022100288 A1 WO2022100288 A1 WO 2022100288A1 CN 2021119626 W CN2021119626 W CN 2021119626W WO 2022100288 A1 WO2022100288 A1 WO 2022100288A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- handle
- shooting
- light
- time
- camera
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 230000001360 synchronised effect Effects 0.000 claims description 40
- 230000004397 blinking Effects 0.000 claims description 23
- 238000004891 communication Methods 0.000 claims description 18
- 230000001186 cumulative effect Effects 0.000 claims description 15
- 238000005516 engineering process Methods 0.000 abstract description 17
- 238000012545 processing Methods 0.000 description 27
- 238000010586 diagram Methods 0.000 description 20
- 230000008569 process Effects 0.000 description 17
- 238000004590 computer program Methods 0.000 description 14
- 239000000284 extract Substances 0.000 description 11
- 230000000694 effects Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000033001 locomotion Effects 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 239000004065 semiconductor Substances 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005401 electroluminescence Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
Definitions
- the embodiments of the present application relate to VR (Virtual Reality, virtual reality) technology and AR (Augmented Reality, augmented reality) technology.
- VR technology and AR technology came into being. These technologies are new technologies that seamlessly integrate real world information and virtual world information. They are physical information that is difficult to experience in a certain time and space in the real world, such as visual information, sound, taste, touch. And so on, through computer and other science and technology, simulation and then superimposition, to achieve the application of virtual information to the real world, perceived by the user's senses, so as to achieve a sensory experience beyond reality.
- VR equipment is applied, such as VR helmet, that is, VR head display (that is, virtual reality head-mounted display device).
- VR helmet that is, VR head display (that is, virtual reality head-mounted display device).
- VR headsets are increasingly active in the market, such as the education and training industry, fire drill industry, virtual driving industry and real estate industry are all using VR headsets.
- the VR handle In the use of the VR helmet, a VR handle needs to be equipped so that the user can control the virtual target in the virtual reality scene displayed by the VR helmet, and the VR handle is communicated with the VR helmet.
- the VR handle includes semiconductor light-emitting diodes (Light Emitting Diode, LED for short) arranged in a certain spatial structure, wherein the light-emitting color of the LED is visible light or infrared light with high saturation, and the camera on the VR helmet is used.
- LED Light Emitting Diode
- the image of the flashing LED light on the VR handle is obtained, and the VR helmet analyzes the flashing image of the LED light to realize the positioning and tracking of the target moving in the virtual space.
- some corresponding technical problems need to be solved.
- Some embodiments of the present application provide a display device, a handle, and a method for calibrating virtual target positioning and tracking, which can solve the problems of VR handle operation delay, failure or misoperation in actual VR application scenarios.
- the embodiment of the present application provides a display device, which is communicatively connected to a handle, including:
- a display which is used to display the interface
- a processor connected to the camera and the display, respectively, the processor is configured to:
- the time delay deviation value of at least one continuous shooting storage period is the difference between the lighting start time of the handle light in the shooting storage period and the shooting start time of the camera. difference; wherein, the duration of the shooting storage cycle is the same as the duration of the flashing cycle of the handle light state;
- a synchronization calibration instruction is sent to the handle, and the synchronization calibration instruction is used to instruct
- the handle light is calibrated as the start calibration time at the lighting start time of the next shooting and storage cycle, and N is greater than or equal to 1.
- the display device provided in this application may be a VR helmet.
- the light of the handle can be based on at least one continuous shooting storage period.
- the difference between the start time of the light and the start time of the camera's shooting generates a synchronous calibration instruction for calibrating the flickering of the handle light.
- the handle calibrates the lighting start time of the next shooting and storage cycle to be the same as the shooting start time of the next shooting and storage cycle.
- the camera can capture the image of the handle light in real time and completely within the shooting and storage period, thereby avoiding the inability to accurately and effectively track the handle position due to the asynchronous shooting and storage of the camera and the flickering of the handle light in the related art. It solves the problem of handle operation delay, failure or misoperation that occurs when the handle is applied.
- Some embodiments of the present application provide a display device, a control device, and a synchronous calibration method, which can realize accurate positioning of the control device and improve user experience.
- Embodiments of the present application provide a display device, including:
- the lighting delay time is sent to the control device through the communicator, and the control device is the device that is successfully paired with the display device.
- the display device is a VR helmet
- the control device is a handle
- the indicator light is an LED light.
- the display device counts the number of frames of the currently captured image.
- the display device determines to The lighting delay time of the synchronous calibration of the indicator light, and the lighting delay time is sent to the control device through the communicator, so as to realize the synchronization calibration of the exposure time of the display device and the lighting time of the control device, and then based on the light spot on the subsequent captured image.
- Some embodiments of the present application provide a virtual reality device and a handle positioning method, which can improve the accuracy of handle positioning.
- Embodiments of the present application provide a virtual reality device, including:
- a camera configured to collect multiple frames of images of a handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light;
- processor connected to the camera, the processor being configured to:
- the first encoded information and the second encoded information of the at least one indicator light remove the interference light spot in the at least one light spot, and determine the light spot corresponding to the at least one indicator light;
- the position of the handle is determined according to the position of the light spot corresponding to the at least one indicator light in the multi-frame images.
- the virtual reality device includes a display; a camera is configured to collect multiple frames of images of the handle, the handle is connected to the virtual reality device, and the handle is provided with at least one indicator light; A controller connected with the camera, the controller is configured to: extract at least one light spot from the multi-frame images of the handle; encode the at least one light spot to form first encoded information; The second encoded information removes the interference light spot in at least one light spot, and determines the light spot corresponding to at least one indicator light; and determines the position of the handle according to the position of the light spot corresponding to at least one indicator light in the multi-frame image.
- the interference light spot can be removed by encoding information when the handle is positioned, thereby improving the accuracy of the handle positioning.
- FIG. 1 exemplarily shows a schematic diagram of an operation scenario between a display device and a control device in some embodiments
- FIG. 2 exemplarily shows a block diagram of the hardware configuration of the display device provided by some embodiments
- FIG. 3 exemplarily shows a schematic diagram that the camera shooting and storage and the flashing of the handle light are not synchronized in some embodiments
- FIG. 4 exemplarily shows the solution flow for the camera shooting storage and the unsynchronized flashing of the handle light in some embodiments
- FIG. 5 exemplarily shows a schematic flowchart of a method for calibrating virtual target positioning and tracking provided by some embodiments
- FIG. 6 exemplarily shows a schematic flowchart of a method for calibrating virtual target positioning and tracking provided by some embodiments
- Fig. 7a exemplarily shows a hardware configuration block diagram of the control device provided by some embodiments.
- FIG. 8 exemplarily shows a schematic flowchart of a method for calibrating virtual target positioning and tracking provided by some embodiments
- FIG. 9 exemplarily shows a schematic diagram of a calibration device for virtual target positioning and tracking provided by some embodiments.
- FIG. 10 exemplarily shows a schematic diagram of a calibration device for virtual target positioning and tracking provided by some embodiments
- FIG. 11 exemplarily shows the timing diagram of the asynchronous exposure of the camera and the flickering of the LED light
- FIG. 12 exemplarily shows the timing diagram of the synchronization between the exposure of the camera and the blinking of the LED lights
- FIG. 13 is a flowchart of a synchronization calibration method provided by an embodiment of the present application.
- Figure 14 exemplarily shows a schematic diagram of the relationship between the exposure start time of the current frame image and the current system time
- FIG. 15 is a schematic structural diagram of a synchronization calibration device provided by an embodiment of the application.
- 16 is a schematic structural diagram of a synchronization calibration device provided by another embodiment of the present application.
- FIG. 17 exemplarily shows a schematic flowchart of handle positioning according to some embodiments.
- FIGS 18a-18b illustrate schematic diagrams of bright spots in accordance with some embodiments
- FIG. 19 exemplarily shows a flow chart of another handle positioning according to some embodiments.
- module refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware or/and software code capable of performing the function associated with that element.
- the display device is a VR helmet, that is, a VR head-mounted display, and also called VR glasses in the early days.
- VR head-mounted display is a kind of head-mounted display that closes people's vision and hearing to the outside world, and guides users to create a feeling of being in a virtual environment. After the eyes obtain this information with differences, a three-dimensional perception occurs in the mind.
- control device used in this application, in some embodiments of this application, is a handle, which is a portable device paired with a display device, which can usually be wired/wireless to control the display in a short distance. equipment.
- a display device which can usually be wired/wireless to control the display in a short distance. equipment.
- RF radio frequency
- Bluetooth is used to connect with the display device, and may also include functional modules such as WiFi modules, USB (Universal Serial Bus) communication modules, Bluetooth, and motion sensors.
- the handle is as important as a mouse is for a PC (Personal Computer).
- gesture used in this application refers to a user's behavior that is used by a user to express an expected thought, action, purpose/or result through an action such as a change of hand shape or hand movement.
- FIG. 1 exemplarily shows a schematic diagram of an operation scenario between the display device 10 and the handle 20 under the application scenario of 6 degrees of freedom (DOF for short).
- the handle 20 is provided with a semiconductor light-emitting diode (Light Emitting Diode, LED for short) light
- the camera 200 on the display device 10 captures the image of the handle 20 when the light is on, and then the display device 10 captures the image when the handle 20 is lit.
- the processor 300 on the above implements the positioning and tracking of the handle 20 after analyzing the image when the light is on, and further realizes the positioning and tracking of the moving target corresponding to the handle 20 within the virtual space range.
- the display device 10 includes a display 100 , a camera 200 , a processor 300 and a communicator 400 .
- the processor 300 is connected to the camera 200 and the display 100 respectively. connect.
- the display 100 is used to display an interface.
- the display device 10 may be a virtual reality (Virtual Reality, VR for short) helmet, and the display 100 may be understood as a display screen on the VR helmet, used for displaying VR helmet instructions displayed interface.
- the display 100 may be an organic electroluminescence display (Organic Electroluminescence Display, OLED for short), or may be other types of displays, which are not limited in this application.
- the camera 200 is arranged on the display device 10 for acquiring image data.
- the display device 10 may be a VR helmet
- the camera 200 may be a binocular camera
- the model of the binocular camera may be selected according to actual needs, which is not limited in this application.
- communicator 400 is a component for communicating with external devices or external servers according to various communication protocol types.
- the communicator 400 may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver.
- the processor 300 is connected to the display 100 and the camera 200 respectively, and the processor 300 is configured to obtain a delay deviation value of at least one continuous shooting storage period, and the delay deviation value is the handle 20 in the shooting storage period The difference between the lighting start time of the light and the shooting start time of the camera 200, wherein the shooting storage period is the same as the flashing period of the handle light state.
- the shooting and storage period includes a shooting period and a storage period, and the shooting period is the duration of the image capturing by the camera 200 , and specifically, in this application, refers to the duration of the shooting handle when the light is on.
- the storage period is the duration of image storage performed by the camera 200, and specifically, in this application, refers to the duration of storage of the image when the handle is lit.
- the flickering cycle includes the on-time and off-time of the light of the handle 20. During the flickering cycle, the light of the handle first turns on and then turns off. The light of the handle 20 turns on and turns off once.
- the durations are the on-light duration and the off-light duration respectively.
- the shooting frame rate of the camera 200 is 60 frames per second (frame per second, FPS for short)
- the shooting storage period is about 16.667 milliseconds (millisecond, ms for short)
- the shooting period in the shooting storage period is 8.33 ms
- the storage period in the shooting storage period is 8.33ms
- the corresponding light-on time and light-off time in the flashing period of the light of the handle 20 are 8.33ms and 8.33ms, respectively.
- the shooting period and the storage period are both 8ms
- the corresponding light-on duration and light-off duration are 8ms and 8ms, respectively.
- the shooting period and the storage period may be the same or different.
- the shooting period, the storage period, the light-on duration and the light-off duration are all the same.
- the shooting start time of the shooting cycle is equal to the lighting start time of the light of the handle 20
- the shooting storage of the camera 200 is synchronized with the flashing of the light of the handle 20, and the VR helmet is moving in the virtual space.
- the positioning tracking will not cause delay, failure or misoperation of the VR handle.
- the time delay deviation value of at least one continuous shooting storage cycle is acquired.
- the processor 300 is further configured to send a synchronization calibration instruction to the handle when the cumulative value of the delay deviation values of N shooting storage periods in the at least one continuous shooting storage period is greater than a preset deviation value, and the synchronization calibration
- the instruction is used to instruct the light of the handle 20 to be calibrated at the start time of the light of the next shooting and storage cycle as the start calibration time, and N is greater than or equal to 1.
- the preset deviation value may be a quarter of the duration of the shooting storage period, that is, if the shooting storage period is T, the preset deviation value is T/4.
- Figure 4 shows the solution for the camera's shooting storage and handle lights flashing out of sync, including:
- S401 Determine the flickering period of the handle light according to the shooting frame rate of the camera.
- the shooting storage period of the camera 200 and the flashing period of the light of the handle 20 need to be set to be the same.
- the shooting storage period of the camera 200 can be determined according to the shooting frame rate of the camera 200 . For example, assuming that the shooting frame rate of the camera 200 is 60 FPS, the shooting storage period and the blinking period are both approximately equal to 16.667 ms.
- S404 Acquire a delay deviation value of at least one continuous shooting storage period, and obtain an accumulated value of the delay deviation values of N shooting storage periods in at least one continuous shooting storage period, where the delay deviation value is the shooting storage period
- the difference between the lighting start time of the handle light and the shooting start time in the cycle; N is greater than or equal to 1.
- c1 represents the time delay deviation value of the initial shooting and storage period, and can also be understood as the time delay deviation value of the first shooting and storage period
- c1 tl-tc
- c2 represents the delay deviation value of the next shooting storage period of the initial shooting storage period, which can also be understood as the delay deviation value of the second shooting storage period
- c2 tl2-tc2 where tl2 represents the second shooting storage period.
- the lighting start time of the second shooting storage period, tc2 represents the shooting start time of the second shooting storage period.
- S405 Determine whether the accumulated value of the delay deviation value is greater than a preset deviation value, wherein the preset deviation is a quarter of the duration of the shooting storage period.
- the photographing storage period is T
- ⁇ t is greater than T/4. If ⁇ t>T/4, it is determined that the initial calibration time is t1- ⁇ t.
- the synchronization calibration instruction is used to instruct the handle light to be calibrated to be t1- ⁇ t at the lighting start time of the next shooting storage cycle.
- the lighting start time of the next shooting storage period is calibrated to be t1- ⁇ t.
- the shooting start time of the next shooting storage period is also t1- ⁇ t.
- the shooting storage of the camera 200 is synchronized with the flashing of the lights of the handle 20 .
- the process returns to step S404.
- the processor 300 in the display device 10 can continuously monitor the flickering of the light of the handle 20 .
- the flickering of the light of the handle 20 is out of sync with the shooting and storage of the camera 200 , a time delay is reached.
- the processor 300 can control the flashing of the light of the handle 20 , so that the flashing of the light of the handle 20 is synchronized with the shooting storage of the camera 200 .
- the display device 10 when the shooting storage of the camera 200 and the flashing of the light of the handle 20 are not synchronized, the display device 10 can be based on the lighting start time of the light of the handle 20 in at least one continuous shooting storage period and the The difference between the shooting start times of the camera 200 is synchronized with the shooting storage of the camera 200 and the flashing of the lights of the handle 20, so as to solve the problem of handle operation delay, failure or misoperation when the handle 20 is applied.
- the processor 300 before sending the synchronization calibration command to the handle 20, the processor 300 is further configured to:
- the processor 300 After the processor 300 sends a synchronization calibration instruction to the handle 20, and the handle 20 calibrates the flashing of the light of the handle 20 and the shooting and storage synchronization of the camera 200 according to the synchronization calibration instruction, the processor 300 It is also configured to acquire a photographed image of the light of the handle 20 , and the photographed image is an image obtained when the handle 20 is lit.
- the captured image refers to an image captured by the camera 200 in a plurality of capture storage periods when the handle 20 is lit.
- the processor 300 then performs positioning and tracking of the handle 20 in the virtual reality scene according to the captured image.
- a high definition multimedia interface (High Definition Multimedia Interface, HDMI for short) on the processor 300 receives an image captured by the camera 200 when the handle 20 is lit, and the processor 300 is responsible for the lighting In the image obtained at the time of shooting, the lights flashing in the image are encoded and then the feature points of the handle 20 are identified, so as to realize the tracking and positioning of the handle 20 .
- the processor 300 can also smooth and predict the determined position information of the handle 20 in the three-dimensional space, so as to realize the timeliness and fluency of the positioning and tracking of the handle 20 .
- the display device 10 can analyze the captured image of the light of the handle 20 to determine the position of the handle 20 in the three-dimensional space, thereby realizing the positioning and tracking of the handle in the virtual reality scene. Since the shooting and storage of the camera 200 is synchronized with the flashing of the lights of the handle 20, the camera 200 can synchronously capture all the images of the lights of the handle 20 within the shooting cycle, so that the positioning and tracking results of the handle 20 are more accurate, and no The problem of operation delay, failure or misoperation of the handle 20 is caused.
- some embodiments of the present application provide a method for calibrating virtual target positioning and tracking, which is applied to the aforementioned display device 10.
- the display device 10 is connected to the handle 20 in communication, and the calibration method for virtual target positioning and tracking Methods include:
- S501 Obtain a time delay deviation value of at least one continuous shooting storage period, where the time delay deviation value is the difference between the lighting start time of the handle light in the shooting storage period and the shooting start time of the camera .
- the duration of the shooting storage cycle is the same as the duration of the blinking cycle of the handle light state.
- the shooting storage period is about 16.667ms
- the shooting period in the shooting storage period is 8.33ms
- the shooting period is 8.33ms.
- the storage period in the storage period is 8.33ms
- the corresponding light-on time and light-off time in the flashing period of the light of the handle 20 are 8.33ms and 8.33ms, respectively. If it is assumed that the shooting period and the storage period are both 8ms, the corresponding light-on duration and light-off duration are 8ms and 8ms, respectively.
- the shooting period, the storage period, the light-on time period and the light-off time period is set to be the same.
- the shooting start time of the shooting cycle is equal to the lighting start time of the light of the handle 20
- the shooting storage of the camera 200 is synchronized with the flashing of the light of the handle 20, and the VR helmet is moving in the virtual space.
- the positioning tracking will not cause delay, failure or misoperation of the VR handle.
- the time delay deviation value of at least one continuous shooting storage cycle is acquired.
- the shooting start time of the shooting cycle is tc
- the lighting start time of the light of the handle 20 is t1
- the delay deviation value is c
- the delay deviation value of the at least one continuous shooting storage cycle is ⁇ t
- the preset deviation value may be a quarter of the duration of the shooting storage period, that is, if the shooting storage period is T, the preset deviation value is T/4.
- the processor 300 sends a synchronous calibration instruction to the handle 20, and the synchronous calibration instruction is used to instruct the handle 20 to be calibrated at the start of the lighting start time of the next shooting storage cycle to be the initial calibration time.
- the handle 20 receives the synchronization calibration instruction, it calibrates the lighting start time of the next shooting storage cycle to tr. At this time, the shooting start time of the next shooting storage period is also tr, and during the next shooting storage period, the shooting storage of the camera 200 is synchronized with the flashing of the light of the handle 20 .
- the method for calibrating virtual target positioning and tracking can start according to the lighting of the light of the handle 20 in at least one continuous shooting and storage period when the shooting storage of the camera 200 and the flashing of the light of the handle 20 are not synchronized.
- the difference between the time and the shooting start time of the camera 200, the shooting storage of the camera 200 is synchronized with the flashing of the light of the handle 20, so as to solve the problem that the handle operation delay, failure or misoperation occurs when the handle 20 is applied.
- the problem can start according to the lighting of the light of the handle 20 in at least one continuous shooting and storage period when the shooting storage of the camera 200 and the flashing of the light of the handle 20 are not synchronized.
- the difference between the time and the shooting start time of the camera 200, the shooting storage of the camera 200 is synchronized with the flashing of the light of the handle 20, so as to solve the problem that the handle operation delay, failure or misoperation occurs when the handle 20 is applied. The problem.
- some embodiments of the present application provide a calibration method for virtual target positioning and tracking, including:
- S601 Acquire a time delay deviation value of at least one continuous shooting storage period, where the delay deviation value is the difference between the lighting start time of the handle light in the shooting storage period and the shooting start time of the camera ; wherein, the duration of the shooting storage cycle is the same as the duration of the blinking cycle of the handle light state.
- step S501 for the specific implementation of this step, reference is made to the description of step S501 in some embodiments shown in FIG. 5 , which will not be explained in detail here.
- S602 Determine the difference between the start time when the handle light is first turned on within the N shooting storage periods and the accumulated value of the N shooting storage periods, as the starting calibration time.
- step S502 For the specific implementation of this step, reference is made to the description of step S502 in some embodiments shown in FIG. 5 , which will not be explained in detail here.
- S604 Acquire a shot image of the handle light, where the shot image is an image shot when the handle light is on.
- the high definition multimedia interface (High Definition Multimedia Interface, HDMI for short) on the processor 300 receives the image captured by the camera 200 when the handle 20 is on , the processor 300 encodes the light flickering in the image captured when the light is on, and then identifies the feature points of the handle 20 , so as to realize the tracking and positioning of the handle 20 .
- High Definition Multimedia Interface HDMI for short
- the calibration method for the positioning and tracking of the virtual target can analyze the captured images of the light of the handle 20 to determine the position of the handle 20 in the three-dimensional space, thereby realizing the positioning and tracking of the handle in the virtual reality scene. Since the shooting and storage of the camera 200 is synchronized with the flashing of the lights of the handle 20, the camera 200 can synchronously capture all the images of the lights of the handle 20 within the shooting cycle, so that the positioning and tracking results of the handle 20 are more accurate, and no The problem of operation delay, failure or misoperation of the handle 20 is caused.
- control device 20 specifically a handle 20 .
- the control device 20 may include at least one of a communicator 21, a controller 22, a physical function key 23, an indicator light 24 for being tracked, a power supply 25, a reset circuit 26, and a memory 27. in:
- the physical function keys 23 may include, but are not limited to, volume up/down keys, up/down/left/right movement keys, voice input keys, menu keys, power on/off keys, and a system for calling up the system menu. key, trigger (Trigger) key, etc.
- volume plus and minus keys are used to control the volume in the VR scene; the movement keys are used to control the movement of up, down, left and right in the VR scene.
- the indicator lights 24 are arranged on the housing of the control device 100 according to a certain structure.
- the luminous color may be a visible light color with high saturation, or may be an infrared light
- the position tracking of the control device 20 by the display device can be realized by the LED light.
- the number of indicator lights can be at least one, and the number and arrangement shape of indicator lights on the left-hand handle and the indicator lights on the right-hand handle can be the same, or the number and arrangement of indicator lights on the left-hand handle and the indicator lights on the right-hand handle Shapes can be different.
- the structure layout of the indicator light 22 is also different, as shown in Figs. 7b to 7d.
- the communicator 21 may include a WiFi module, a USB communication module, a Bluetooth module and other modules based on various communication protocols.
- the communicator 21 may provide a wireless communication solution including 2G/3G/4G/5G, etc. applied on the control device 20 .
- the communicator 21 can receive electromagnetic waves by an antenna, filter, amplify, etc. the received electromagnetic waves, and transmit them to a modulation and demodulation processor for demodulation.
- the communicator 21 can also amplify the signal modulated by the modulation and demodulation processor, and then convert it into electromagnetic waves and radiate it out through the antenna.
- at least part of the functional modules of the communicator 21 may be provided in the controller 22 .
- the communicator 21 further includes an NFC (Near Field Communication, near field communication) module to facilitate short-range communication.
- the NFC module can be based on RFID (Radio Frequency Identification, radio frequency identification) technology, IrDA (Infrared Data Association, Infrared Data Association) technology, UWB (Ultra Wideband, ultra-wideband) technology, BT (Bluetooth, Bluetooth) technology and other technologies to fulfill.
- the controller 22 generally controls the overall operation of the control device 20, such as operations associated with display, data communication, camera operation, and recording operations.
- the controller 22 may include one or more processing units, for example: the controller 22 may be an MCU (Micro Control Unit, micro control unit), a CPU (Central Processing Unit, central processing unit), or a DSP (Digital Signal Processor, digital signal processor), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), etc.
- a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
- the power supply 25 is used to provide stable power to the various functional circuits and modules of the control device 20 .
- Power supply 25 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to control device 20 .
- the power source 25 may be a battery and associated control circuitry.
- the reset circuit 26 enables the controller 22 to start working from the initial state at the moment when power is obtained. If the random access memory, counter and other circuits in the controller 22 start to work without being reset after receiving power supply, some interference may cause the controller 22 to fail to work normally due to disordered programs. For this reason, the controller 22 needs to be provided with a reset circuit.
- memory 27 includes storage of various software modules used to drive control device 20 .
- various software modules stored in the memory 27 include at least one of a basic module, a detection module, a display control module, a browser module, and various service modules.
- the basic module is used for the signal communication between various hardwares in the display device 20 and the bottom software module that sends processing and control signals to the upper layer module.
- the detection module is a management module used to collect various information from various sensors or user input interfaces, perform digital-to-analog conversion, and analyze and manage.
- the display control module is a module used to control the display to display image content, and can be used to play information such as multimedia image content and UI interface.
- the communication module is a module used for control and data communication with external devices.
- the browser module is a module for performing data communication between browsing servers. Service modules are used to provide various services and modules including various applications.
- the memory 27 is also used to store and receive external data and user data, images of various items in various user interfaces, and visual effect diagrams of focal objects.
- the handle 20 is provided with a handle light, that is, an LED light.
- the handle 20 includes a communicator 21 and a controller 22 .
- the communicator 21 is configured to receive a synchronization calibration instruction sent by the display device 10, where the synchronization calibration instruction is used to instruct the light of the handle 20 to be calibrated at the start of the lighting start time of the next shooting storage cycle as the start calibration time, wherein,
- the duration of the photographing storage cycle of the display device 100 is the same as the duration of the blinking cycle of the light state of the handle 20 .
- the controller 22 is configured to calibrate the lighting start time of the light of the handle 20 in the next shooting storage cycle according to the synchronization calibration instruction.
- the communicator 21 after receiving the synchronization calibration instruction, the communicator 21 sends the synchronization calibration instruction to the controller 22, and the controller 22 parses the start calibration time according to the synchronization calibration instruction, and stores the next shot.
- the lighting start time of the cycle is calibrated to the start calibration time, so as to realize the synchronization between the light flashing of the handle 20 and the shooting and storage of the camera 200 .
- the initial calibration time is t1- ⁇ t
- t1 is the lighting start time of the initial shooting storage period
- ⁇ t is the accumulated value of the delay deviation values of at least one continuous shooting storage period.
- the communicator 21 may be a Universal Serial Bus (Universal Serial Bus, USB for short).
- some embodiments of the present application provide a method for calibrating virtual target positioning and tracking, which is applied to the aforementioned handle 20 , and the handle 20 is communicatively connected to the display device 10 , and the method for calibrating virtual target positioning and tracking includes: :
- S801 Receive a synchronous calibration instruction sent by the display device, where the synchronous calibration instruction is used to instruct the handle light to be calibrated as the start calibration time at the lighting start time of the next shooting and storage cycle, wherein the shooting and storage cycle of the display device The duration is the same as the blinking cycle duration of the handle light state.
- the communicator 21 on the handle 20 receives the synchronization calibration instruction sent by the display device 10, and the communicator 21 is, for example, a USB.
- the handle 20 also includes a controller 22, and the communicator 21 sends the synchronization calibration instruction to the controller 22 after receiving the synchronization calibration instruction.
- the controller 22 calibrates the lighting start time of the handle 20 light in the next shooting storage cycle according to the synchronization calibration instruction as t1- ⁇ t, t1 is the lighting start time of the initial shooting storage cycle, and ⁇ t is at least one The cumulative value of the delay deviation value of the continuous shooting storage period.
- the calibration device 30 for virtual target positioning and tracking includes: an acquisition module 31 and a processing module 32 .
- the acquisition module 31 is configured to acquire the time delay deviation value of at least one continuous shooting storage period, where the delay deviation value is the difference between the lighting start time of the handle light in the shooting storage period and the shooting start time of the camera difference between.
- the duration of the shooting storage cycle is the same as the duration of the blinking cycle of the handle light state.
- the processing module 32 is configured to send a synchronization calibration instruction to the handle when the cumulative value of the delay deviation values of N shooting storage periods in the at least one continuous shooting storage period is greater than the preset deviation value, and the synchronization calibration instruction uses In order to instruct the start time of lighting of the handle light in the next shooting and storage cycle as the start calibration time, N is greater than or equal to 1.
- the preset deviation value is a quarter of the duration of the shooting storage period.
- the processing module 32 is further configured to determine the difference between the start time when the handle light is first turned on within the N shooting storage periods and the accumulated value of the N shooting storage periods, as the starting calibration time.
- the acquiring module 31 is further configured to acquire a photographed image of the handle light, where the photographed image is an image photographed when the handle is lit.
- the processing module 32 is further configured to perform positioning and tracking of the handle in the virtual reality scene according to the captured image.
- the apparatus provided in this embodiment can be used to perform the steps performed by the display device 10 in the embodiments shown in FIG. 2 to FIG. 4 , and the implementation principles and technical effects thereof are similar, and are not repeated here.
- the calibration device 40 for positioning and tracking a virtual target includes: a receiving module 41 and a processing module 42 .
- the receiving module 41 is configured to receive a synchronous calibration instruction sent by the display device, where the synchronous calibration instruction is used to instruct the handle light to be calibrated as the start calibration time at the lighting start time of the next shooting and storage cycle, wherein the display device
- the duration of the shooting storage cycle is the same as the flashing cycle duration of the handle light state.
- the processing module 42 is configured to calibrate the lighting start time of the handle light in the next shooting and storage cycle according to the synchronization calibration instruction.
- the device provided in this embodiment can be used to perform the steps performed by the handle 20 in some embodiments shown in FIG. 7 , and the implementation principle and technical effect thereof are similar, which will not be repeated here.
- the usual method is to hold a control device with LED lights on and off, use the binocular camera on the control device to obtain the image of the control device, and then extract and encode the light spot on the image to determine The ID number of the LED light, and the positioning tracking is realized according to the PNP (Pespective-N-Point) algorithm to complete the interaction in the VR scene.
- PNP Pulspective-N-Point
- this method has the problem of time offset between the exposure of the camera and the flickering of the LED light. Over time, camera exposure and LED light flickering cannot be tightly controlled in sync. As shown in FIG.
- the flashing of the indicator light on the control device is periodically synchronously calibrated by obtaining the camera shooting exposure start time stamp and the number of frames, so that the camera shooting exposure and the flashing indicator light are kept constant. Maintain strict synchronization.
- the display device 10 uses the camera to track the control device 20 , and the control device 20 is arranged with a certain spatial structure of indicator light cycles Blinks brightly and dimly. Specifically, the display device 10 uses the camera to acquire the image including the control device 20. Since the camera exposure and the flashing of the indicator light are strictly synchronized (as shown in FIG. 12 ), the bright spots in the image are extracted and encoded to remove interference noise. , identify the ID number of the indicator light on the control device, and calculate the position of the control device in space according to the PNP algorithm, which can achieve high-precision spatial positioning of the control device, ensure the effectiveness of user operations, and improve user experience.
- FIG. 13 is a flowchart of a synchronization calibration method provided by an embodiment of the present application. As shown in FIG. 13, the processor 300 in the display device 10 is configured to perform the following steps:
- the display device 10 shoots the light on the control device 20 through the camera 200 and flashes (when the indicator light 24 is lit), and obtains an image including the control device 20.
- the light spot in the image is the image corresponding to the indicator light 24, and the processor 300 counts the number of frames of the current frame image captured by the camera 200 during the capturing process.
- the number of frames can be counted by setting a counter, wherein the counter can be an up-counter or a down-counter or an up-down counter.
- the counter is an addition counter, and when the number of frames counted by the counter reaches a preset number of frames, it resets to 0 and starts counting again, wherein the preset number of frames is, for example, 200 frames; or, the counter continuously accumulates the number of image frames, Until it reaches the maximum value that can be counted by the counter itself, then reset to 0 and start counting again.
- the counter when the counter is a subtraction counter, its initial value can be set to the preset number of frames, and then decremented with the increase of the number of image frames in the statistical process until it returns to zero, and then reset to the preset number of frames, and restarts the statistics ;
- the initial value of the counter is the maximum value that can be counted by itself, and then decrements with the increase of the number of image frames in the statistical process until it returns to zero, and then resets to the maximum value and starts counting again.
- the counter when the counter is a reversible counter, for example, in the technical process, the process of accumulation is similar to that of the addition counter, and the process of accumulation and subtraction is similar to the implementation of the subtraction counter, which will not be repeated here. .
- the processor 300 determines the lighting delay time for synchronous calibration of the indicator lights on the control device according to the exposure start time of the current frame image and the current system time.
- the frame rate of the camera is fixed, and the period of the camera to capture a single frame of image can be obtained according to the frame rate.
- the frame rate of the camera is 60FPS (Frames Per Second, the number of frames transmitted per second)
- the exposure start time of the current frame image and the current system time can be used to determine the start exposure time of the subsequent image captured by the camera.
- the lighting delay time of the indicator light on the control device can be further determined according to the initial exposure time of the subsequent captured image of the camera.
- the lighting delay time is sent to the control device through the communicator.
- the control device 20 is a device successfully paired with the display device 10 .
- the lighting delay time can be carried in the synchronization instruction and sent to the control device 20 .
- the processor 300 performs the above operations periodically, and the exposure start timestamp and frame number of the image captured by the camera 200 are periodically synchronized to calibrate the blinking of the indicator light 24 to ensure strict synchronization between the image captured by the camera 200 and the blinking of the indicator light 24 control, so as to realize precise positioning and tracking of the control device 20 by the display device 10 .
- the display device determines the lighting delay time for synchronizing the calibration of the indicator lights on the control device, and sends the lighting delay time through the communicator.
- the light delay time is given to the control device to achieve synchronous calibration of the exposure time of the display device and the lighting time of the control device, so as to accurately locate and track the control device based on the light spot on the subsequent captured images, improving the user experience.
- the controller 22 in the control device 20 is configured to perform the following steps:
- the lighting start time of the indicator light on the control device is determined according to the lighting delay time.
- control indicator light is periodically flashed since the lighting start time.
- the controller 22 If the lighting delay time is carried in the synchronization command and sent to the control device 20, after the controller 22 receives the synchronization command through the communicator 21, it needs to parse the synchronization command to obtain the lighting delay time.
- the control device determines the lighting start time of the indicator light on the control device according to the lighting delay time sent by the display device, and starts from the lighting start time, the control indicator light flashes periodically, wherein the indication
- the flickering period of the light is determined according to the frame rate of the camera of the display device, so as to achieve synchronous calibration of the exposure time of the display device and the lighting time of the control device, and then accurately locate and track the control device based on the light spot on the subsequent captured images, improving user experience.
- the lighting delay time of synchronous calibration may include: determining the time deviation value between the exposure start time of the current frame image and the current system time; The lighting delay time for synchronization calibration.
- the processor 300 When the camera captures a frame of image and starts to expose, the processor 300 will allocate a buffer to the frame of image, and there will be a time stamp at this time, which is the time when the frame of image starts to be exposed.
- the cycle of the camera shooting a single frame image is T1
- the exposure start time of the next frame of image is t12
- the exposure start time of the Nth frame of image is t1n. If the next synchronization calibration is after the Nth frame image, the lighting delay time is:
- d is the adjustment time, which is obtained based on the experimental results. Considering that the transmission of the lighting delay time between the display device 10 and the control device 20 takes time, and the display device 10 may also have a delay in sending the lighting delay time, it is possible to balance the transmission delay and the transmission delay by setting d. required time.
- the processor 300 is further configured to: determine the blinking cycle of the indicator light on the control device according to the frame rate of the camera, and the product of the blinking cycle and the frame rate is 1; and send the blinking cycle to the control device.
- the time of one frame of image includes the exposure time of 8.33ms and the image storage time of 8.33ms. It can be assumed here that both are 8ms, and the corresponding indicator light flashes.
- the bright and dark times correspond to 8ms and 8ms, respectively, as shown in Figure 7.
- the processor 300 is further configured to: when the control device is lit, capture an image including the control device through a camera; and perform location tracking of the control device according to the image.
- the processor 300 calculates the position of the indicator light 24 in space, and calculates the position of the control device 20 in the real space according to the spatial layout of the indicator light 24 on the control device 20 in combination with the PNP algorithm, thereby realizing the control device Localization and tracking in 3D space.
- the processor 300 smoothes and predicts the position of the control device based on the calculated result, so as to ensure the timeliness and fluency of the positioning and tracking of the control device.
- the light emitted by the indicator light may be visible light or infrared light.
- FIG. 15 is a schematic structural diagram of a synchronization calibration apparatus provided by an embodiment of the present application.
- An embodiment of the present application provides a synchronization calibration device, which is applied to a display device.
- the synchronization calibration apparatus 110 includes: an acquisition module 111 , a processing module 112 and a sending module 113 . in:
- the obtaining module 111 is configured to obtain the frame number of the current frame image obtained by the camera during the shooting process.
- the processing module 112 is configured to determine the lighting delay time for synchronous calibration of the indicator lights on the control device according to the exposure start time of the current frame image and the current system time when the number of frames is an integer multiple of the preset number of frames.
- the sending module 113 is configured to send the lighting delay time to the control device through the communicator, and the control device is a device that is successfully paired with the display device.
- the apparatus provided in this embodiment of the present application can be used to perform the steps performed by the display device in the embodiment shown in FIG. 13 , and the implementation principle and technical effect thereof are similar, and are not repeated here.
- the processing module 112 may be specifically configured to: determine the time offset value between the exposure start moment of the current frame of image and the current system time. ; According to the time deviation value and the duration of acquiring the N frames of images, determine the lighting delay time for synchronous calibration of the indicator lights on the control device.
- the processing module 112 may also be used to: determine the blinking cycle of the indicator light on the control device according to the frame rate of the camera, and the product of the blinking cycle and the frame rate is 1; trigger the sending module 113 to send the blinking cycle to the control device.
- the processing module 112 may also be used to: when the control device is lit, capture an image including the control device through a camera; and perform positioning and tracking of the control device according to the image.
- FIG. 16 is a schematic structural diagram of a synchronization calibration apparatus provided by another embodiment of the present application.
- An embodiment of the present application provides a synchronization calibration device, which is applied to a control device, where the control device is a device that is successfully paired with the above-mentioned display device.
- the synchronization calibration apparatus 120 includes: a receiving module 121 and a processing module 122 . in:
- the receiving module 121 is configured to receive the lighting delay time sent by the display device. Among them, the lighting delay time is used to indicate the lighting delay time of the synchronous calibration of the indicator lights on the control device.
- the processing module 122 is configured to determine the lighting starting time of the indicator light on the control device according to the lighting delay time; and, starting from the lighting starting time, the control indicator light flashes periodically.
- the apparatus provided in this embodiment of the present application can be used to execute the steps executed by the control device in the embodiment shown in FIG. 13 , and the implementation principle and technical effect thereof are similar, and are not repeated here.
- the blinking period of the indicator light is determined according to the frame rate of the camera of the display device, and the product of the blinking period and the frame rate is 1.
- the camera on the VR device will capture the image of the handle (control device), extract and encode the light spot formed by the flashing indicator light in the image, and determine the corresponding brightness of the indicator light according to the encoded information. spot. Finally, the VR device completes the positioning of the handle according to the position of the light spot corresponding to the indicator light in the image.
- the handle control device
- the VR device completes the positioning of the handle according to the position of the light spot corresponding to the indicator light in the image.
- there may be other light sources around the handle which will cause interference light spots on the image of the handle, affecting the accuracy of the handle positioning.
- some embodiments of the present application provide a virtual reality device and a handle positioning method, which removes interference light spots in an image of the handle by using preset coding information of an indicator light on the handle, thereby improving the accuracy of handle positioning. .
- FIG. 17 exemplarily shows a schematic flowchart of a handle positioning according to some embodiments, and this embodiment relates to a specific process of how to position the handle.
- the execution subject of this embodiment is a virtual reality device, and the virtual reality device is connected with the handle.
- the method includes:
- S301 Collect multiple frames of images of the handle, and the handle is provided with at least one indicator light.
- the VR device after the handle is connected to the VR device, if the VR device receives an instruction for positioning the handle, it can collect multiple frames of images of the handle.
- a camera may be carried on the VR device, and multiple frames of images of the handle may be collected by invoking the camera carried on the VR device.
- the VR device may also be externally connected with other camera devices, and the VR device may call the external camera device to collect multiple frames of images of the handle by sending instruction information to the external camera device.
- the embodiment of the present application does not limit the type of the camera, which may be a monocular camera by way of example.
- At least one indicator light may be provided on the handle.
- the indicator light on the handle may flash, so that the collected multi-frame images include the corresponding indicator lights. light spot, so as to assist the VR device to position the handle.
- the embodiment of the present application also does not limit the type of the indicator light, which can be exemplarily a light-emitting diode (light-emitting diode light, LED).
- Figures 7b-7d exemplarily show schematic diagrams of the arrangement of indicator lights according to some embodiments. As shown in Figures 7b-7d, for handles of different shapes, the indicator lights can be arranged in sequence according to the edge of the handle. It should be understood that the colors of the indicator lights in the embodiments of the present application may be the same or different, and may be specifically set according to actual conditions.
- the VR device may periodically send synchronization information to the handle, and the synchronization information includes at least one indicator light. blinking cycle. After the handle receives the synchronization information, the indicator light can be controlled to flash according to the flashing cycle. At the same time, the VR device can record images synchronously, so that the image of the handle captured by the camera exposure is exactly the image when the indicator light of the handle is on.
- the controller in the VR device can extract at least one light spot from the multi-frame images of the handle.
- the controller in the VR device may sequentially input multiple frames of images of the handle into the image recognition algorithm model to obtain each frame input by the image recognition algorithm model. At least one spot extracted from the image.
- the image recognition algorithm model may be an opecv blob light blob extraction algorithm model.
- the opecv blob light spot extraction algorithm model can analyze the connected domain of the same pixel in the image, and extract the light spot in the image after binarizing the image.
- S303 Encode at least one light spot to form first encoded information.
- the controller in the VR device extracts at least one light spot from the multi-frame images of the handle, the at least one light spot can be encoded to form the first encoded information.
- the controller in the VR device may first determine changes in brightness and darkness of the at least one light spot in the multi-frame images according to the diameter change of the at least one light spot in the multi-frame images. Subsequently, the controller in the VR device encodes at least one light spot according to the light and dark changes in the multi-frame images to form first encoded information.
- FIGS. 18a-18b exemplarily show schematic diagrams of bright spots according to some embodiments, as shown in FIGS. 18a-18b, the diameter of the bright spots shown in FIG.
- the diameter of the bright spot can be set to R1
- Fig. 18a is the image of the previous frame of Fig. 18b.
- the code of the light spot in Fig. 18a can be set to 1, and in Fig. 18b
- the encoding can be set to 0.
- m is a constant, which can be set based on the actual situation, for example, m is 1.3, 1.5, etc.
- n is a constant, which can be specifically set based on the actual situation, for example, m is 1.3, 1.5, etc.
- the encoded information of the same bright spot may be combined in time sequence, thereby forming the first encoded information corresponding to the light spot. It should be understood that the embodiment of the present application does not limit the number of frames of the image included in the first encoding information, which may be five frames in an example.
- the first encoded information can be compared with the second encoded information of at least one indicator light preset in the VR device, so as to remove the interference light spot in the at least one light spot, And determine the light spot corresponding to at least one indicator light.
- Table 1 is the second coded information table of the indicator light.
- the light spot corresponding to the first encoded information is the light spot corresponding to at least one indicator light.
- the light spot corresponding to the first encoded information is an interference light spot, and then the interference light spot is removed.
- the controller in the VR device After the controller in the VR device removes the interference light spot in the at least one light spot and determines the light spot corresponding to the at least one indicator light, it can determine the light spot corresponding to the at least one indicator light according to the position of the light spot corresponding to the at least one indicator light in the multi-frame image. position of the handle.
- the controller in the VR device may determine the spatial layout of the indicator light on the handle according to the position of the light spot corresponding to at least one indicator light in the multi-frame image. Then, through the Perspective-n-Point (PnP) algorithm, the position of the handle in the three-dimensional control is determined, so as to realize the positioning and tracking of the handle.
- PnP Perspective-n-Point
- the positioning and tracking of the handle in this application may not only be used for 3DOF (coordinates), but also may be used for 6DOF, which is not limited in this embodiment of the application.
- the position of the handle can be further smoothed and predicted, thereby improving the timeliness and fluency of the positioning and tracking of the handle.
- the handle positioning method by collecting multiple frames of images of the handle, the handle is provided with at least one indicator light. Second, extract at least one light spot from the multi-frame images of the handle, and encode the at least one light spot to form first encoded information. Thirdly, according to the first encoded information and the second encoded information of the at least one indicator light, the interference light spot in the at least one light spot is removed, and the light spot corresponding to the at least one indicator light is determined. Finally, the position of the handle is determined according to the position of the light spot corresponding to at least one indicator light in the multi-frame image. Compared with the related art, in the embodiment of the present application, when the handle is positioned, the interference light spot can be removed through the preset coding information of the indicator light, thereby improving the accuracy of the handle positioning.
- FIG. 19 exemplarily shows a schematic flowchart of another handle positioning according to some embodiments. As shown in FIG. 19 , the method includes:
- S901. Collect multiple frames of images of the handle, and the handle is provided with at least one indicator light.
- S901-S902 can be understood with reference to S301-S302 shown in FIG. 17 , and repeated content will not be repeated here.
- S904 Encode at least one light spot according to the light and dark changes in the multi-frame images to form first encoded information.
- At least one light spot is encoded according to the light and dark changes in the multi-frame images and the corresponding relationship between the light and dark of the light spot and the bitmap information to form the first encoded information.
- the corresponding relationship between the brightness of the light spot and the bitmap information can be, for example, the same light spot in two adjacent frames of images, the brighter corresponds to 1, and the brighter corresponds to 0.
- S905. Determine whether there is second encoding information that is the same as the first encoding information.
- step S906 If yes, go to step S906, if not, go to step S907.
- the position of the handle is determined according to the position of the light spot corresponding to at least one indicator light in the multi-frame images.
- the handle positioning method by collecting multiple frames of images of the handle, the handle is provided with at least one indicator light. Second, extract at least one light spot from the multi-frame images of the handle, and encode the at least one light spot to form first encoded information. Thirdly, according to the first encoded information and the second encoded information of the at least one indicator light, the interference light spot in the at least one light spot is removed, and the light spot corresponding to the at least one indicator light is determined. Finally, the position of the handle is determined according to the position of the light spot corresponding to at least one indicator light in the multi-frame image. Compared with the related art, in the embodiment of the present application, when the handle is positioned, the interference light spot can be removed through the preset coding information of the indicator light, thereby improving the accuracy of the handle positioning.
- FIG. 2 exemplarily shows a schematic structural diagram of a display device according to some embodiments.
- the display device may be implemented by software, hardware or a combination of the two, so as to execute the handle positioning method in the above-mentioned embodiments.
- the virtual display device 10 includes: a display 100 , a camera 200 and a processor 300 .
- the display device is specifically a virtual reality device.
- the camera is configured to collect multi-frame images of the handle, the handle is connected with the virtual reality device, and the handle is provided with at least one indicator light;
- the first encoded information and the second encoded information of the at least one indicator light remove the interference light spot in the at least one light spot, and determine the light spot corresponding to the at least one indicator light;
- the position of the handle is determined according to the position of the light spot corresponding to at least one indicator light in the multi-frame images.
- the processor is specifically configured as:
- the diameter change of the at least one light spot in the multi-frame images determine the light-dark change of the at least one light spot in the multi-frame images
- At least one light spot is encoded according to the light and dark changes in the multi-frame images to form the first encoded information.
- the processor is specifically configured as:
- At least one light spot is encoded according to the light and dark changes in the multi-frame images and the corresponding relationship between the light and dark of the light spot and the bitmap information to form the first encoded information.
- the processor is specifically configured as:
- the processor is further configured to:
- the synchronization information includes the blinking period of at least one indicator light.
- each module of the above apparatus is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated.
- these modules can all be implemented in the form of software calling through processing elements; they can also all be implemented in hardware; some modules can also be implemented in the form of calling software through processing elements, and some modules can be implemented in hardware.
- the processing module may be a separately established processing element, or may be integrated into a certain chip of the above-mentioned device to be implemented, in addition, it may also be stored in the memory of the above-mentioned device in the form of program code, and a certain processing element of the above-mentioned device Call and execute the functions of the above processing modules.
- each step of the above-mentioned method or each of the above-mentioned modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.
- the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more ASIC (Application Specific Integrated Circuit, specific integrated circuit), or, one or more DSP (Digital Signal Processor) , digital signal processor), or, one or more FPGAs (Field Programmable Gate Array, Field Programmable Gate Array), etc.
- the processing element may be a general-purpose processor, such as a CPU or other processors that can invoke program codes.
- these modules can be integrated together and implemented in the form of SOC (System-on-a-Chip, system-on-chip).
- the above-mentioned embodiments it may be implemented in whole or in part by software, hardware, firmware or any combination thereof.
- software it can be implemented in whole or in part in the form of a computer program product.
- the computer program product includes one or more computer programs.
- the computer may be a general purpose computer, special purpose computer, computer network, or other programmable device.
- the computer program can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program can be transferred from a website site, computer, server or data center via wired (eg coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg infrared, wireless, microwave, etc.) to another website site, computer, server or data center.
- the computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media.
- the available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.
- Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method in any of the above method embodiments is implemented.
- the embodiment of the present application also provides a display system, including the display device 10 and the handle 20 as described above.
- An embodiment of the present application further provides a chip for running an instruction, where the chip is used to execute the method in any of the above method embodiments.
- Embodiments of the present application further provide a computer program product, where the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium, and at least one processor can read the computer program from the computer-readable storage medium, When the at least one processor executes the computer program, the method of any of the above method embodiments can be implemented.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
本申请提供实施例,属于显示技术,提供一种显示设备、手柄以及虚拟目标定位追踪的校准方法,该显示设备包括显示器,摄像头;分别与摄像头和显示器连接的处理器。该处理器被配置为获取至少一个连续的拍摄存储周期的时延偏差值,时延偏差值为拍摄存储周期内手柄灯的亮灯起始时刻与摄像头的拍摄起始时刻之间的差值;其中,拍摄存储周期时长与手柄灯状态的闪烁周期时长相同;当至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向手柄发送同步校准指令,同步校准指令用于指示将手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。由此可以解决VR手柄操作延时或误操作的问题。
Description
相关申请交叉引用
本申请要求于2020年11月12日提交中国专利局、申请号为202011260412.0、申请名称为“虚拟现实设备以及手柄定位方法”、2020年11月13日提交中国专利局、申请号为202011260735.X、申请名称为“显示设备、控制设备以及同步校准方法”以及2020年11月12日提交中国专利局、申请号为202011260409.9、申请名称为“显示设备、手柄以及虚拟目标定位追踪的校准方法”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请实施例涉及VR(Virtual Reality,虚拟现实)技术和AR(Augmented Reality,增强现实)技术。
随着科技的发展,VR技术和AR技术应运而生。该些技术均是将真实世界信息和虚拟世界信息无缝集成的新技术,是把原本在现实世界的一定时间空间范围内很难体验到的实体信息,例如,视觉信息,声音,味道,触觉等,通过电脑等科学技术,模拟仿真后再叠加,实现将虚拟的信息应用到真实世界,被用户感官所感知,从而达到超越现实的感官体验。
目前,随着虚拟现实(Virtual Reality,简称VR)和增强现实(Augmented Reality,简称AR)技术的发展,VR设备被应用,例如VR头盔,即VR头显(也即虚拟现实头戴式显示设备)越来越在市场上活跃,例如教育培训行业、消防演练行业、虚拟驾驶行业和房地产行业均在使用VR头盔。
而在VR头盔的使用中,需要配备VR手柄使用户在VR头盔显示的虚拟现实场景中可以进行虚拟目标的操控,VR手柄与VR头盔通信连接。比如在一些VR应用场景中,需要对在一个空间范围内运动的目标进行定位,如赛车游戏,需要对用户操控的赛车进行定位追踪。定位追踪的原理为:该VR手柄包括以一定空间结构排列的半导体发光二极管(Light Emitting Diode,简称LED),其中LED的发光色为饱和度较高的可见光或红外光,利用VR头盔上的摄像头获取VR手柄上LED灯闪烁为亮的图像,VR头盔对该LED灯闪烁的图像进行分析后实现虚拟空间范围内运动的目标的定位追踪。其中,在利用摄像头和带有灯的手柄进行定位时,需解决一些相应的技术问题。
发明内容
本申请某些实施方式中提供一种显示设备、手柄以及虚拟目标定位追踪的校准方法,可解决在实际VR应用场景中,VR手柄操作延时、失败或误操作的问题。
本申请实施例提供一种显示设备,与手柄通信连接,包括:
显示器,用于显示界面;
摄像头,用于获取图像数据;
分别与所述摄像头和所述显示器连接的处理器,所述处理器被配置为:
获取至少一个连续的拍摄存储周期的时延偏差值,所述时延偏差值为所述拍摄存储周期内所述手柄灯的亮灯起始时刻,与所述摄像头的拍摄起始时刻之间的差值;其中,所述拍摄存储周期时长与所述手柄灯状态的闪烁周期时长相同;
当所述至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向所述手柄发送同步校准指令,所述同步校准指令用于指示将所述手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。
本申请提供的该显示设备可以为VR头盔。在整个该摄像头的拍摄过程中,对于与该显示设备配对成功的该手柄,在该摄像头的拍摄存储和该手柄的灯闪烁不同步的时候,可以根据至少一个连续的拍摄存储周期内该手柄灯的亮灯起始时刻和该摄像头的拍摄起始时刻之间的差值,生成用于校准该手柄灯闪烁的同步校准指令。该手柄根据该同步校准指令进行下一个拍摄存储周期的亮灯起始时刻校准为和该下一个拍摄存储周期的拍摄起始时刻相同。此时该摄像头可以在拍摄存储周期内实时并且完整得拍摄到该手柄灯亮的图像,从而避免了相关技术中因为该摄像头的拍摄存储和该手柄灯的闪烁不同步造成的无法准确有效追踪手柄位置的缺陷,解决了手柄应用时发生的手柄操作延时、失败或误操作的问题。
本申请某些实施方式提供一种显示设备、控制设备以及同步校准方法,可实现对控制设备的准确定位,提升用户体验。
本申请实施例提供一种显示设备,包括:
显示器;
摄像头;
分别与摄像头和显示器连接的处理器,该处理器被配置为:
获取摄像头拍摄得到的当前帧图像在拍摄过程中的帧数;
在帧数为预设帧数的整数倍时,根据当前帧图像的曝光起始时刻和当前系统时间,确定对控制设备上指示灯进行同步校准的亮灯延时时间;
通过通信器发送亮灯延时时间给控制设备,控制设备为与显示设备配对成功的设备。
示例地,显示设备为VR头盔,控制设备为手柄,指示灯为LED灯。在拍摄过程中,对于与显示设备配对成功的控制设备,显示设备统计当前已拍摄图像的帧数,在已拍摄图像的帧数为预设帧数的整数倍时,显示设备确定对控制设备上指示灯进行同步校准的亮灯延时时间,并通过通信器发送亮灯延时时间给控制设备,以实现显示设备曝光时间与控制设备亮灯时间的同步校准,进而基于后续拍摄图像上的光斑准确定位追踪控制设备,提升用户体验。
本申请某些实施例提供一种虚拟现实设备以及手柄定位方法,可提升手柄定位的准确性。
本申请实施例提供一种虚拟现实设备,包括:
显示器;
摄像头,被配置为采集手柄的多帧图像,所述手柄与所述虚拟现实设备连接,所述手柄设置有至少一个指示灯;
与所述摄像头连接的处理器,所述处理器被配置为:
从所述手柄的多帧图像中提取至少一个光斑;
对所述至少一个光斑进行编码,形成第一编码信息;
根据所述第一编码信息以及所述至少一个指示灯的第二编码信息,去除所述至少一个光斑中的干扰光斑,并确定所述至少一个指示灯对应的光斑;
根据所述至少一个指示灯对应的光斑在所述多帧图像中的位置,确定所述的手柄的位置。
本申请实施例提供的虚拟现实设备以及手柄定位方法,虚拟现实设备包括显示器;摄像头,该摄像头被配置为采集手柄的多帧图像,该手柄与虚拟现实设备连接,手柄设置有至少一个指示灯;与摄像头连接的控制器,该控制器被配置为:从手柄的多帧图像中提取至少一个光斑;对至少一个光斑进行编码,形成第一编码信息;根据第一编码信息以及至少一个指示灯的第二编码信息,去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑;根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。与相关技术相比,本申请实施例在手柄定位时可以通过编码信息去除干扰光斑,从而提高了手柄定位的准确性。
为了更清楚地说明本申请实施例或相关技术中的实施方式,下面将对实施例或相关技术描述中所需要使用的附图作一简单地介绍,显而易见地,下面描述中的附图是本申请的一些实施例,对于本领域普通技术人员来讲,还可以根据这些附图获得其他的附图。
图1中示例性示出了一些实施例中显示设备与控制设备之间操作场景的示意图;
图2中示例性示出了某些实施例提供的显示设备的硬件配置框图;
图3中示例性示出了一些实施例中摄像头拍摄存储和手柄灯闪烁不同步的示意图;
图4中示例性示出了一些实施例中针对摄像头拍摄存储和手柄灯闪烁不同步的解决方法流程;
图5中示例性示出了某些实施例提供的虚拟目标定位追踪的校准方法的流程示意图;
图6中示例性示出了某些实施例提供的虚拟目标定位追踪的校准方法的流程示意图;
图7a中示例性示出了某些实施例提供的控制设备的硬件配置框图;
图7b至图7d中示例性示出了控制设备的结构示意图;
图8中示例性示出了某些实施例提供的虚拟目标定位追踪的校准方法的流程示意图;
图9中示例性示出了某些实施例提供的虚拟目标定位追踪的校准装置的示意图;
图10中示例性示出了某些实施例提供的虚拟目标定位追踪的校准装置的示意图;
图11中示例性示出了摄像头拍摄曝光与LED灯闪烁不同步的时序图;
图12中示例性示出了摄像头拍摄曝光与LED灯闪烁同步的时序图;
图13为本申请一实施例提供的同步校准方法的流程图;
图14中示例性示出了当前帧图像的曝光起始时刻和当前系统时间的关系示意图;
图15为本申请一实施例提供的同步校准装置的结构示意图;
图16为本申请另一实施例提供的同步校准装置的结构示意图;
图17示例性示出了根据一些实施例的一种手柄定位的流程示意图;
图18a-18b示例性示出了根据一些实施例的亮斑的示意图;
图19示例性示出了根据一些实施例的另一种手柄定位的流程示意图。
为使本申请的目的、实施方式和优点更加清楚,下面将结合本申请示例性实施例中的附图,对本申请示例性实施方式进行清楚、完整地描述,显然,所描述的示例性实施例仅是本申请一部分实施例,而不是全部的实施例。
基于本申请描述的示例性实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本申请所附权利要求保护的范围。此外,虽然本申请中公开内容按照示范性一个或几个实例来介绍,但应理解,可以就这些公开内容的各个方面也可以单独构成一个完整实施方式。
需要说明的是,本申请中对于术语的简要说明,仅是为了方便理解接下来描述的实施方式,而不是意图限定本申请的实施方式。除非另有说明,这些术语应当按照其普通和通常的含义理解。
本申请中说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”等是用于区别类似或同类的对象或实体,而不必然意味着限定特定的顺序或先后次序,除非另外注明(Unless otherwise indicated)。应该理解这样使用的用语在适当情况下可以互换,例如能够根据本申请实施例图示或描述中给出那些以外的顺序实施。
此外,术语“包括”和“具有”以及他们的任何变形,意图在于覆盖但不排他的包含,例如,包含了一系列组件的产品或设备不必限于清楚地列出的那些组件,而是可包括没有清楚地列出的或对于这些产品或设备固有的其它组件。
本申请中使用的术语“模块”,是指任何已知或后来开发的硬件、软件、固件、人工智能、模糊逻辑或硬件或/和软件代码的组合,能够执行与该元件相关的功能。
本申请中使用的术语“显示设备”,在本申请某些实施例中,显示设备为VR头盔,即VR头显,早期也有VR眼镜等称呼。VR头显是一种利用头戴式显示器将人的对外界的视觉、听觉封闭,引导用户产生一种身在虚拟环境中的感觉,其显示原理是左右眼屏幕分别显示左右眼的图像,人眼获取这种带有差异的信息后在脑海中产生立体感。
本申请中使用的术语“控制设备”,在本申请某些实施例中,控制设备为手柄,是与显示设备配对工作的一个便携式设备,通常可在较短的距离范围内有线/无线控制显示设备。一般使用红外线和/或射频(RF)信号和/或蓝牙与显示设备连接,也可以包括WiFi模块、USB(Universal Serial Bus,通用串行总线)通信模块、蓝牙、动作传感器等功能模块。对于VR设备而言,手柄相当于鼠标对于PC(Personal Computer,个人计算机)一样重要。
本申请中使用的术语“手势”,是指用户通过一种手型的变化或手部运动等动作,用于表达预期想法、动作、目的/或结果的用户行为。
在相关技术中,在利用摄像头和带有灯的手柄进行定位时,摄像头的拍摄手柄的曝光和VR手柄上的灯闪烁存在时间偏移的问题,即摄像头的拍摄曝光和VR手柄上的灯闪烁不能实现严格的同步控制,这造成手柄定位失败,由此造成在实际VR应用场景中,VR手柄操作延时、失败或误操作的问题,进而影响用户体验。
以下将介绍解决摄像头的拍摄曝光和VR手柄上的灯闪烁不能实现严格的同步控制的技术方案。
图1中示例性示出了6自由度(degree of freedom,简称DOF)应用场景下,显示设备10与手柄20之间操作场景的示意图。如图1中示出,该手柄20上设置有半导体发光二极管(Light Emitting Diode,简称LED)灯,该显示设备10上的摄像头200采集该手柄 20亮灯时的图像,再由该显示设备10上的处理器300对该亮灯时的图像进行分析后实现该手柄20的定位追踪,进而实现对虚拟空间范围内与手柄20对应的运动目标的定位追踪。
请参见图2,本申请某些实施例提供一种显示设备10,该显示设备10包括显示器100、摄像头200、处理器300和通信器400,该处理器300分别与该摄像头200和该显示器100连接。
该显示器100用于显示界面,一些实施例中,该显示设备10可以是虚拟现实(Virtual Reality,简称VR)头盔,该显示器100可以理解为该VR头盔上的显示屏,用于显示VR头盔指示显示的界面。一些实施例中,该显示器100可以为有机电致发光显示器(Organic Electroluminescence Display,简称OLED),也可以为其他类型的显示器,本申请不做限定。
该摄像头200设置在该显示设备10上,用于获取图像数据。一些实施例中,该显示设备10可以是VR头盔,该摄像头200可以是双目相机,该双目相机的型号可以根据实际需要选择,本申请不做限定。
在一些实施例中,通信器400是用于根据各种通信协议类型与外部设备或外部服务器进行通信的组件。通信器400可以包括Wifi模块,蓝牙模块,有线以太网模块等其他网络通信协议芯片或近场通信协议芯片,以及红外接收器中的至少一种。
该处理器300分别与该显示器100和该摄像头200连接,该处理器300被配置为获取至少一个连续的拍摄存储周期的时延偏差值,该时延偏差值为该拍摄存储周期内该手柄20灯的亮灯起始时刻,与该摄像头200的拍摄起始时刻之间的差值,其中,该拍摄存储周期时长与该手柄灯状态的闪烁周期时长相同。
该拍摄存储周期包括拍摄周期和存储周期,该拍摄周期为摄像头200进行图像拍摄的时长,具体的,在本申请中指的是拍摄手柄亮灯时的图像的时长。该存储周期为该摄像头200进行图像存储的时长,具体的,在本申请中指的是存储手柄亮灯时的图像的时长。该闪烁周期包括该手柄20灯的亮灯时长和灭灯时长,在该闪烁周期内该手柄灯先进行一次亮灯,再经历一次灭灯,该手柄20灯进行一次亮灯和进行一次灭灯的时长分别为该亮灯时长和该灭灯时长。
假设该摄像头200的拍摄帧率是60帧/秒(frame per second,简称FPS),则该拍摄存储周期大约为16.667毫秒(millisecond,简称ms),该拍摄存储周期中的拍摄周期为8.33ms,该拍摄存储周期中的存储周期为8.33ms,则对应的该手柄20灯的闪烁周期中亮灯时长和灭灯时长分别为8.33ms和8.33ms。若假设该拍摄周期和该存储周期均为8ms,则对应的该亮灯时长和灭灯时长分别为8ms和8ms。在该显示设备10启用之前,需要将该拍摄周期和该亮灯时长设置为相同、并将该存储周期和该灭灯时长设置为相同。
在本实施例中,该拍摄周期和该存储周期可以相同也可以不同,当该拍摄周期和该存储周期相同时,该拍摄周期、该存储周期、该亮灯时长和该灭灯时长均相同。当该拍摄周期的拍摄起始时刻和该手柄20灯的亮灯起始时刻相等时,该摄像头200的拍摄存储和该手柄20的灯闪烁同步,该VR头盔在进行虚拟空间范围内运动的目标的定位追踪时不会造成VR手柄操作延时、失败或误操作的问题。当该拍摄周期的拍摄起始时刻和该手柄20灯的亮灯起始时刻之间存在延迟时,获取至少一个连续的拍摄存储周期的时延偏差值。
该处理器300还被配置为当该至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向该手柄发送同步校准指令,该同步校准指令用 于指示将该手柄20灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。在一些实施例中,该预设偏差值可以为该拍摄存储周期的四分之一时长,即假设该拍摄存储周期为T,该预设偏差值即为T/4。当该摄像头200拍摄存储和该手柄20灯闪烁如图3所示出现不同步的状况时,执行如图4所示的方法流程图。
图4所示为针对该摄像头拍摄存储和手柄灯闪烁不同步的解决方法,包括:
S401,根据该摄像头的拍摄帧率确定该手柄灯的闪烁周期。
在该显示设备10启用之前,需要将该摄像头200的拍摄存储周期和该手柄20灯的闪烁周期设置为相同,该摄像头200的拍摄存储周期可以根据该摄像头200的拍摄帧率确定。举例说明,假设该摄像头200的拍摄帧率为60FPS,则该拍摄存储周期和该闪烁周期大约都等于16.667ms。
S403,获取该摄像头在起始的拍摄存储周期的拍摄起始时刻。
S402,获取该手柄灯在该起始的拍摄存储周期的亮灯起始时刻。
S404,获取至少一个连续的拍摄存储周期的时延偏差值,以及获取至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值,该时延偏差值为该拍摄存储周期内该手柄灯的亮灯起始时刻与该拍摄起始时刻之间的差值;N大于或等于1。
假设该拍摄起始时刻为tc,该亮灯起始时刻为tl,该时延偏差值为c,该时延偏差值的累计值为Δt。则c=tl-tc,Δt=c1+c2+…+cN,其中N大于或等于1。c1代表该起始的拍摄存储周期的时延偏差值,也可以理解为第1个拍摄存储周期的时延偏差值,c1=tl-tc。c2代表该起始的拍摄存储周期的下一个拍摄存储周期的时延偏差值,也可以理解为第2个拍摄存储周期的时延偏差值,c2=tl2-tc2,其中,tl2代表该第2个拍摄存储周期的亮灯起始时刻,tc2代表该第2个拍摄存储周期的拍摄起始时刻。c3可以理解为第3个拍摄存储周期的时延偏差值,c3=tl3-tc3,其中,tl3代表该第3个拍摄存储周期的亮灯起始时刻,tc3代表该第3个拍摄存储周期的拍摄起始时刻。cN可以理解为第N个拍摄存储周期的时延偏差值,cN=tlN-tcN,其中,tlN代表该第N个拍摄存储周期的亮灯起始时刻,tcN代表第N个拍摄存储周期的拍摄起始时刻。
S405,判断该时延偏差值的累计值是否大于预设偏差值,其中,该预设偏差为该拍摄存储周期的四分之一时长。
S406,若该时延偏差值的累计值大于该预设偏差值,则根据该起始的拍摄存储周期的亮灯起始时刻和该时延偏差值的累计值的差值确定该手柄灯在下一个拍摄存储周期的亮灯起始时刻为起始校准时刻。
S407,根据该起始校准时刻生成同步校准指令,并向该手柄发送该同步校准指令,该同步校准指令用于指示将该手柄灯将下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻。
假设该拍摄存储周期为T,则判断Δt是否大于T/4。若Δt>T/4,则确定该起始校准时刻为tl-Δt。根据该起始校准时刻生成该同步校准指令后,该同步校准指令用于指示该手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为tl-Δt。当该手柄20接收到该同步校准指令后,将该下一个拍摄存储周期的亮灯起始时刻校准为tl-Δt。此时,该下一个拍摄存储周期的拍摄起始时刻也为tl-Δt,在该下一个拍摄存储周期时,该摄像头200的拍摄存储和该手柄20的灯闪烁同步。关于N的取值,当Δt=c1时若Δt大于该预设偏差值,则N=1。当Δt=c2时若Δt大于该预设偏差值,则N=2。即,N的数值取决于Δt与该预设偏差值之间的关系。
在本申请某些实施例中,若该时延偏差值的累计值小于或等于该预设偏差值,则返回执行步骤S404。在该显示设备10的使用中,该显示设备10中的该处理器300可以不断得对该手柄20灯的闪烁进行监测,当该手柄20灯闪烁与该摄像头200的拍摄存储不同步达到时延偏差值大于该摄像头200的拍摄存储周期的四分之一时,该处理器300可以对该手柄20灯的闪烁进行控制,从而使该手柄20灯闪烁与该摄像头200的拍摄存储达到同步。
本实施例提供的该显示设备10在该摄像头200的拍摄存储和该手柄20的灯闪烁不同步的时候,可以根据至少一个连续的拍摄存储周期内该手柄20灯的亮灯起始时刻和该摄像头200的拍摄起始时刻之间的差值,对该摄像头200的拍摄存储和该手柄20的灯闪烁进行同步,从而解决在手柄20应用时发生手柄操作延时、失败或误操作的问题。
在一些实施例中,向该手柄20发送同步校准指令之前,该处理器300还被配置为:
确定N个拍摄存储周期中起始的拍摄存储周期中该手柄灯的亮灯起始时刻和该N个拍摄存储周期的时延偏差值的累计值之间的差值,为该起始校准时刻。
例如,N=3,则该起始的拍摄存储周期指的是3个连续的拍摄存储周期中起始的拍摄存储周期,假设该3个拍摄存储周期中起始的拍摄存储周期中该手柄20灯的亮灯起始时刻为t1,该起始校准时刻为tr,该3个拍摄存储周期的时延偏差值的累计值为Δt=c1+c2+c3,则t1=tr-Δt=tr-(c1+c2+c3)。
在一些实施例中,该处理器300在对该手柄20发送同步校准指令,且该手柄20根据该同步校准指令校准该手柄20灯的闪烁和该摄像头200的拍摄存储同步后,该处理器300还被配置为获取该手柄20灯的拍摄图像,该拍摄图像为该手柄20亮灯时拍摄得到的图像。该拍摄图像是指该摄像头200在多个拍摄存储周期拍摄到的该手柄20亮灯时的图像。该处理器300再根据该拍摄图像进行该手柄20在虚拟现实场景中的定位追踪。在一些实施例中,该处理器300上的高清多媒体接口(High Definition Multimedia Interface,简称HDMI)接收该摄像头200拍摄的该手柄20亮灯时拍摄得到的图像,由该处理器300对该亮灯时拍摄得到的图像中灯闪烁进行编码后识别该手柄20的特征点,实现该手柄20的追踪定位。该处理器300还可以对确定的该手柄20在三维空间中的位置信息进行平滑和预测,实现该手柄20的定位追踪的时效性和流畅性。
本实施例提供的该显示设备10可以对获取的该手柄20灯的拍摄图像进行分析,确定该手柄20在三维空间的位置,进而实现该手柄在虚拟现实场景中的定位追踪。由于该摄像头200的拍摄存储和该手柄20的灯闪烁同步,因此该摄像头200可以在拍摄周期内同步得拍摄到该手柄20灯亮的全部图像,使得该手柄20的定位追踪结果更加准确,不会造成该手柄20操作延时、失败或误操作的问题。
请参见图5,本申请某些实施例提供了一种虚拟目标定位追踪的校准方法,应用于前述的该显示设备10,该显示设备10与该手柄20通信连接,该虚拟目标定位追踪的校准方法包括:
S501,获取至少一个连续的拍摄存储周期的时延偏差值,该时延偏差值为该拍摄存储周期内该手柄灯的亮灯起始时刻,与该摄像头的拍摄起始时刻之间的差值。其中,该拍摄存储周期时长与该手柄灯状态的闪烁周期时长相同。
如上所述,假设该摄像头200的拍摄帧率是60帧/秒(frame per second,简称FPS),则该拍摄存储周期大约为16.667ms,该拍摄存储周期中的拍摄周期为8.33ms,该拍摄存储周期中的存储周期为8.33ms,则对应的该手柄20灯的闪烁周期中亮灯时长和灭灯时长分 别为8.33ms和8.33ms。若假设该拍摄周期和该存储周期均为8ms,则对应的该亮灯时长和灭灯时长分别为8ms和8ms。在该显示设备10启用之前,需要将该拍摄周期、该存储周期、该亮灯时长和该灭灯时长设置为相同。当该拍摄周期的拍摄起始时刻和该手柄20灯的亮灯起始时刻相等时,该摄像头200的拍摄存储和该手柄20的灯闪烁同步,该VR头盔在进行虚拟空间范围内运动的目标的定位追踪时不会造成VR手柄操作延时、失败或误操作的问题。当该拍摄周期的拍摄起始时刻和该手柄20灯的亮灯起始时刻之间存在延迟时,获取至少一个连续的拍摄存储周期的时延偏差值。假设该拍摄周期的拍摄起始时刻为tc,该手柄20灯的亮灯起始时刻为tl,该时延偏差值为c,该至少一个连续的拍摄存储周期的时延偏差值为Δt,则c=tl-tc,Δt=c1+c2+…+cN,其中N大于或等于1。
S502,当该至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向该手柄发送同步校准指令,该同步校准指令用于指示将该手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。
如上所述,在一些实施例中,该预设偏差值可以为该拍摄存储周期的四分之一时长,即假设该拍摄存储周期为T,该预设偏差值即为T/4。关于N的取值,当Δt=c1时,Δt大于该预设偏差值,则N=1。当Δt=c2时,Δt大于该预设偏差值,则N=2。即,N的数值取决于Δt与该预设偏差值之间的关系。当Δt大于该预设偏差值时,该处理器300向该手柄20发送同步校准指令,该同步校准指令用于指示该手柄20灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻。假设该起始校准时刻为tr,则tr=tl-Δt。当该手柄20接收到该同步校准指令后,将该下一个拍摄存储周期的亮灯起始时刻校准为tr。此时,该下一个拍摄存储周期的拍摄起始时刻也为tr,在该下一个拍摄存储周期时,该摄像头200的拍摄存储和该手柄20的灯闪烁同步。
本实施例提供的虚拟目标定位追踪的校准方法在该摄像头200的拍摄存储和该手柄20的灯闪烁不同步的时候,可以根据至少一个连续的拍摄存储周期内该手柄20灯的亮灯起始时刻和该摄像头200的拍摄起始时刻之间的差值,对该摄像头200的拍摄存储和该手柄20的灯闪烁进行同步,从而解决在手柄20应用时发生手柄操作延时、失败或误操作的问题。
请参见图6,本申请某些实施例提供了一种虚拟目标定位追踪的校准方法,包括:
S601,获取至少一个连续的拍摄存储周期的时延偏差值,该时延偏差值为该拍摄存储周期内该手柄灯的亮灯起始时刻,与该摄像头的拍摄起始时刻之间的差值;其中,该拍摄存储周期时长与该手柄灯状态的闪烁周期时长相同。
本步骤的具体实现方式参照图5所示的某些实施例中步骤S501的描述,此处不再进行详细解释。
S602,确定N个拍摄存储周期内该手柄灯首次亮灯起始时刻和该N个拍摄存储周期的累计值之间的差值,为该起始校准时刻。
例如,N=3,则该起始的拍摄存储周期指的是3个连续的拍摄存储周期中起始的拍摄存储周期,假设该3个拍摄存储周期中起始的拍摄存储周期中该手柄20灯的亮灯起始时刻为t1,该起始校准时刻为tr,该3个拍摄存储周期的时延偏差值的累计值为Δt=c1+c2+c3,则t1=tr-Δt=tr-(c1+c2+c3)。
S603,当该至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向该手柄发送同步校准指令,该同步校准指令用于指示将该手柄灯 在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。
本步骤的具体实现方式参照图5所示的某些实施例中步骤S502的描述,此处不再进行详细解释。
S604,获取该手柄灯的拍摄图像,该拍摄图像为该手柄亮灯时拍摄得到的图像。
S605,根据该拍摄图像进行该手柄在虚拟现实场景中的定位追踪。
关于步骤S604~S605,如上所述,在一些实施例中,该处理器300上的高清多媒体接口(High Definition Multimedia Interface,简称HDMI)接收该摄像头200拍摄的该手柄20亮灯时拍摄得到的图像,由该处理器300对该亮灯时拍摄得到的图像中灯闪烁进行编码后识别该手柄20的特征点,实现该手柄20的追踪定位。
本实施例提供的该虚拟目标定位追踪的校准方法可以对获取的该手柄20灯的拍摄图像进行分析,确定该手柄20在三维空间的位置,进而实现该手柄在虚拟现实场景中的定位追踪。由于该摄像头200的拍摄存储和该手柄20的灯闪烁同步,因此该摄像头200可以在拍摄周期内同步得拍摄到该手柄20灯亮的全部图像,使得该手柄20的定位追踪结果更加准确,不会造成该手柄20操作延时、失败或误操作的问题。
请参见图7a,本申请某些实施例提供一种控制设备20,具体为手柄20。如图7a所示,控制设备20可以包括通信器21、控制器22、物理功能键23、用于被追踪的指示灯24、电源25、复位电路26和存储器27等组件中的至少一个。其中:
在一些实施例中,物理功能键23可以包括但不限于音量加减键、上/下/左/右的移动按键、语音输入按键、菜单键、开关机按键、用于调出系统菜单的系统按键、扳机(Trigger)键等。对于各按键的功能可参考相关技术,此处仅作示例性说明,不再一一赘述。例如,音量加减键用于VR场景中音量的大小控制;移动按键用于控制VR场景中的上、下、左、右的移动。
在一些实施例中,指示灯24,按照一定结构布局在控制设备100的外壳上。以LED灯为例,其发光色可以是饱和度较高的可见光颜色,也可以是红外光,通过LED灯可以实现显示设备对控制设备20的定位追踪。其中,指示灯的个数可以为至少一个,并且左手手柄上指示灯和右手手柄上指示灯的数目和排列形状可以是相同的,或左手手柄上指示灯和右手手柄上指示灯的数目和排列形状可以是不同的。另外,随着控制设备22的形状的不同,指示灯22的结构布局也是不同的,如图7b至图7d所示。
在一些实施例中,通信器21可以包括WiFi模块、USB通信模块、蓝牙模块等基于各种通信协议的模块。示例地,通信器21可以提供应用在控制设备20上的包括2G/3G/4G/5G等无线通信的解决方案。通信器21可以由天线接收电磁波,并对接收的电磁波进行滤波,放大等处理,传送至调制解调处理器进行解调。通信器21还可以对经调制解调处理器调制后的信号放大,经天线转为电磁波辐射出去。在一些实施例中,通信器21的至少部分功能模块可以被设置于控制器22中。在一些实施例中,通信器21的至少部分功能模块可以与控制器22的至少部分模块被设置在同一个器件中。或者,通信器21还包括NFC(Near Field Communication,近场通信)模块,以促进短程通信。例如,在NFC模块可基于RFID(Radio Frequency Identification,射频识别)技术,IrDA(Infrared Data Association,红外数据协会)技术,UWB(Ultra Wideband,超宽带)技术,BT(Bluetooth,蓝牙)技术和其他技术来实现。
控制器22通常控制控制设备20的整体操作,诸如与显示,数据通信,摄像机操作和 记录操作相关联的操作。控制器22可以包括一个或多个处理单元,例如:控制器22可以是MCU(Micro Control Unit,微控制单元)、CPU(Central Processing Unit,中央处理单元),也可以是DSP(Digital Signal Processor,数字信号处理器)、ASIC(Application Specific Integrated Circuit,专用集成电路)等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请所公开的方法的步骤可以直接体现为硬件处理器执行完成,或者用处理器中的硬件及软件模块组合执行完成。
在一些实施例中,电源25用于为控制设备20的各功能电路和模块提供稳定的电力。电源25可以包括电源管理系统,一个或多个电源,及其他与为控制设备20生成、管理和分配电力相关联的组件。示例性地,电源25可以是电池及相关控制电路。
在一些实施例中,复位电路26是使控制器22在获得供电的瞬间,由初始状态开始工作。若控制器22内的随机存储器、计数器等电路获得供电后不经复位便开始工作,可能某种干扰会导致控制器22因程序错乱而不能正常工作,为此,控制器22需要设置复位电路。
在一些实施例中,存储器27,包括存储用于驱动控制设备20的各种软件模块。如:存储器27中存储的各种软件模块,包括:基础模块、检测模块、显示控制模块、浏览器模块、和各种服务模块等中的至少一种。
其中,基础模块用于显示设备20中各个硬件之间信号通信、并向上层模块发送处理和控制信号的底层软件模块。检测模块用于从各种传感器或用户输入接口中收集各种信息,并进行数模转换以及分析管理的管理模块。显示控制模块用于控制显示器进行显示图像内容的模块,可以用于播放多媒体图像内容和UI界面等信息。通信模块用于与外部设备之间进行控制和数据通信的模块。浏览器模块用于执行浏览服务器之间数据通信的模块。服务模块,用于提供各种服务以及各类应用程序在内的模块。同时,存储器27还用存储接收外部数据和用户数据、各种用户界面中各个项目的图像以及焦点对象的视觉效果图等。
该手柄20上设置有手柄灯,即LED灯。该手柄20包括通信器21和控制器22。该通信器21被配置为接收该显示设备10发送的同步校准指令,该同步校准指令用于指示将该手柄20灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,其中,该显示设备100的拍摄存储周期时长与该手柄20灯状态的闪烁周期时长相同。该控制器22被配置为根据该同步校准指令校准该手柄20灯在下一个拍摄存储周期的亮灯起始时刻。具体的,该通信器21接收到该同步校准指令后将该同步校准指令发送给该控制器22,该控制器22根据该同步校准指令解析出该起始校准时刻,并将该下一个拍摄存储周期的亮灯起始时刻校准为该起始校准时刻,从而实现该手柄20灯闪烁和该摄像头200拍摄存储的同步。如上所述,该起始校准时刻为tl-Δt,tl为该起始的拍摄存储周期的亮灯起始时刻,Δt为至少一个连续的拍摄存储周期的时延偏差值的累计值。一些实施例中,该通信器21可以为通用串行总线(UniversalSerialBus,简称USB)。
请参见图8,本申请某些实施例提供一种虚拟目标定位追踪的校准方法,应用于前述的该手柄20,该手柄20与该显示设备10通信连接,该虚拟目标定位追踪的校准方法包括:
S801,接收该显示设备发送的同步校准指令,该同步校准指令用于指示将该手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,其中,该显示设备的拍摄存储周期时长与该手柄灯状态的闪烁周期时长相同。
具体的,该手柄20上的通信器21接收该显示设备10发送的该同步校准指令,该通信器21例如USB。该手柄20还包括控制器22,该通信器21接收到该同步校准指令后将 该同步校准指令发送给该控制器22。
S802,根据该同步校准指令校准该手柄灯在下一个拍摄存储周期的亮灯起始时刻。
该控制器22根据该同步校准指令校准该手柄20灯在下一个拍摄存储周期的亮灯起始时刻为tl-Δt,tl为该起始的拍摄存储周期的亮灯起始时刻,Δt为至少一个连续的拍摄存储周期的时延偏差值的累计值。
请参见图9,本申请某些实施例提供一种虚拟目标定位追踪的校准装置30,应用于该显示设备10。如图9所示,该虚拟目标定位追踪的校准装置30包括:获取模块31和处理模块32。
该获取模块31用于获取至少一个连续的拍摄存储周期的时延偏差值,该时延偏差值为该拍摄存储周期内该手柄灯的亮灯起始时刻,与该摄像头的拍摄起始时刻之间的差值。其中,该拍摄存储周期时长与该手柄灯状态的闪烁周期时长相同。
该处理模块32用于当该至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向该手柄发送同步校准指令,该同步校准指令用于指示将该手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。其中,该预设偏差值为该拍摄存储周期的四分之一时长。
该处理模块32还用于确定N个拍摄存储周期内该手柄灯首次亮灯起始时刻和该N个拍摄存储周期的累计值之间的差值,为该起始校准时刻。
该获取模块31还用于获取该手柄灯的拍摄图像,该拍摄图像为该手柄亮灯时拍摄得到的图像。
该处理模块32还用于根据该拍摄图像进行该手柄在虚拟现实场景中的定位追踪。
本实施例提供的装置,可用于执行图2至图4所示的实施例中该显示设备10执行的步骤,其实现原理和技术效果类似,在此不再赘述。
请参见图10,本申请某些实施例提供一种虚拟目标定位追踪的校准装置40,应用于该手柄20。如图10所示,该虚拟目标定位追踪的校准装置40包括:接收模块41和处理模块42。
该接收模块41用于接收该显示设备发送的同步校准指令,该同步校准指令用于指示将该手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,其中,该显示设备的拍摄存储周期时长与该手柄灯状态的闪烁周期时长相同。
该处理模块42用于根据该同步校准指令校准该手柄灯在下一个拍摄存储周期的亮灯起始时刻。
本实施例提供的装置,可用于执行图7所示某些实施例中该手柄20执行的步骤,其实现原理和技术效果类似,在此不再赘述。
在一些VR应用场景中,需要对在一个空间范围内运动的目标进行定位。例如用户与VR环境交互过程中,通常采用的方式为手持带有LED灯亮灭闪烁的控制设备,利用控制设备上的双目摄像头获取控制设备的图像,然后对图像进行光斑提取并编码,判断出LED灯的ID编号,并根据PNP(Pespective-N-Point)算法实现定位追踪,完成VR场景中的交互。但此方式存在摄像头拍摄曝光和LED灯闪烁存在时间偏移的问题。随着时间的推移,摄像头拍摄曝光与LED灯闪烁不能实现严格的同步控制。如图11所示例,假如摄像头拍摄曝光时刻为tc,LED灯点亮时刻为tl,摄像头拍摄曝光时刻与LED灯点亮时刻的时间差为:c=tl-tc。这个时间差属于纳秒级别很小,但累加起来偏移会造成对控制设备的识别和 定位不准确,追踪失效进一步会导致操作延时、失败或误操作,用户体验效果差。
针对以上问题,本申请某些实施例中,根据实际使用环境,通过获取摄像头拍摄曝光起始时间戳和帧数定期同步校准控制设备上指示灯的闪烁,从而实现摄像头拍摄曝光与指示灯闪烁一直保持严格的同步。
在如图1所示的VR交互系统中,在佩戴显示设备的用户使用控制设备10020进入VR场景后,显示设备10利用摄像头追踪控制设备20,控制设备20上排布一定空间结构的指示灯周期亮暗闪烁。具体地,显示设备10利用摄像头获取包含控制设备20的图像,由于摄像头拍摄曝光和指示灯闪烁保持严格同步(如图12所示),因此,通过提取图像中亮斑进行编码,去除干扰杂点,识别出控制设备上指示灯的ID编号,并根据PNP算法计算出控制设备在空间中的位置,可实现对控制设备的高精度空间定位,保证用户操作的有效性,提升用户体验。
下面采用详细的实施例,来说明本申请如何进行同步校准。
图13为本申请一实施例提供的同步校准方法的流程图。如图13所示,显示设备10中处理器300被配置为执行以下步骤:
在S101中,获取摄像头拍摄得到的当前帧图像在拍摄过程中的帧数。
在实际应用过程中,显示设备10通过摄像头200拍摄控制设备20上灯闪(指示灯24点亮时),得到包含控制设备20的图像,图像中光斑为指示灯24对应影像,并通过处理器300统计摄像头200拍摄得到的当前帧图像在拍摄过程中的帧数。
在一些实施例中,可通过设置计数器来统计帧数,其中计数器可以是加法计数器或减法计数器或可逆计数器。一种实现中,计数器为加法计数器,当计数器统计的帧数达到预设帧数时,复位至0重新开始计数,其中预设帧数例如为200帧;或者,计数器不断累加图像的帧数,直至达到该计数器本身可统计的最大值,然后复位至0重新开始计数。类似的,当计数器为减法计数器时,其初始值可以设置为预设帧数,然后在统计过程中随图像帧数的增加而递减,直至归零,然后复位至预设帧数,重新开始统计;或者,计数器的初始值为其本身可统计的最大值,然后在统计过程中随图像帧数的增加而递减,直至归零,然后复位至最大值重新开始计数。或者,当计数器为可逆计数器,例如在技术过程中先加后减或者先减后加,其中累加的过程与加法计数器的实现类似,累减的过程与减法计数器的实现类似,此处不再赘述。
在S102中,在帧数为预设帧数的整数倍时,根据当前帧图像的曝光起始时刻和当前系统时间,确定对控制设备上指示灯进行同步校准的亮灯延时时间。
示例地,仍以预设帧数为200为例,具体实现时,处理器300获取帧数Num,当Num%200!=0时,继续获取帧数Num=Num+1,直至Num%200=0,即帧数为预设帧数的整数倍。在帧数为200或400等200的整数倍时,处理器300根据当前帧图像的曝光起始时刻和当前系统时间,确定对控制设备上指示灯进行同步校准的亮灯延时时间。
通常情况下,摄像头的帧率是固定的,根据帧率可获得摄像头拍摄单帧图像的周期。例如,摄像头的帧率是60FPS(Frames Per Second,每秒传输帧数),则摄像头拍摄单帧图像的周期为T=1/60≈16.667ms,其中可包含曝光时间8.33ms和图像存储时间8.33ms。在摄像头拍摄单帧图像的周期已知的情况下,通过当前帧图像的曝光起始时刻和当前系统时间可以确定摄像头后续拍摄图像的起始曝光时刻。
由于本申请同步校准的是摄像头拍摄曝光和指示灯闪烁,因此,可进一步根据摄像头 后续拍摄图像的起始曝光时刻确定控制设备上指示灯的亮灯延时时间。
在S103中,通过通信器发送亮灯延时时间给控制设备。
其中,控制设备20为与显示设备10配对成功的设备。一些实施例中,可以将亮灯延时时间携带在同步指令中发送给控制设备20。
一些实施例中,处理器300周期性地执行以上操作,通过摄像头200拍摄图像曝光起始时间戳和帧数定期同步校准指示灯24闪烁,以确保摄像头200拍摄图像和指示灯24闪烁的严格同步控制,从而实现显示设备10对控制设备20的精准定位追踪。
本申请实施例在显示设备的摄像头已拍摄图像的帧数为预设帧数的整数倍时,显示设备确定对控制设备上指示灯进行同步校准的亮灯延时时间,并通过通信器发送亮灯延时时间给控制设备,以实现显示设备曝光时间与控制设备亮灯时间的同步校准,进而基于后续拍摄图像上的光斑准确定位追踪控制设备,提升用户体验。
对应地,如图13所示,控制设备20中控制器22被配置为执行以下步骤:
在S201中,通过通信器接收显示设备发送的亮灯延时时间。
在S202中,根据亮灯延时时间,确定控制设备上指示灯的亮灯起始时刻。
在S203中,自亮灯起始时刻开始,控制指示灯周期性闪烁。
若亮灯延时时间是携带在同步指令中发送给控制设备20的,则控制器22通过通信器21接收同步指令后,还需解析该同步指令以获得亮灯延时时间。
本申请实施例中,控制设备根据显示设备发送的亮灯延时时间,确定控制设备上指示灯的亮灯起始时刻,并自亮灯起始时刻开始,控制指示灯周期性闪烁,其中指示灯的闪烁周期是根据显示设备的摄像头的帧率确定的,从而实现显示设备曝光时间与控制设备亮灯时间的同步校准,进而基于后续拍摄图像上的光斑准确定位追踪控制设备,提升用户体验。
在上述实施例的基础上,若在当前帧图像之后的N帧图像后进行同步校准,N为自然数,则根据当前帧图像的曝光起始时刻和当前系统时间,确定对控制设备上指示灯进行同步校准的亮灯延时时间,可以包括:确定当前帧图像的曝光起始时刻和当前系统时间的时间偏差值;根据时间偏差值与获取N帧图像的时长,确定对控制设备上指示灯进行同步校准的亮灯延时时间。
当摄像头拍摄一帧图像开始曝光时,处理器300会分配一个缓存区(buffer)给该帧图像,此时会有一个时间戳,就是该帧图像开始曝光时刻。
例如,参考图14,摄像头拍摄单帧图像的周期为T1,确定当前帧图像的曝光起始时刻t10和当前系统时间t11的时间偏差值为△t1=t11-t10。其中,下一帧图像的曝光起始时刻为t12,或者第N帧图像的曝光起始时刻为t1n。假如下一次同步校准为第N帧图像之后,则亮灯延时时间为:
dT=T1*N-△t1+d
其中,d为调整时间,是基于实验结果得到的。考虑到亮灯延时时间在显示设备10和控制设备20之间的传输需要时间,且显示设备10发送亮灯延时时间也可能存在延迟,因此,可以通过设置d来平衡发送延迟及传输所需的时间。
一些实施例中,处理器300还被配置为:根据摄像头的帧率,确定控制设备上指示灯的闪烁周期,闪烁周期与帧率的乘积为1;发送闪烁周期给控制设备。
基于摄像头的帧率设定指示灯的闪烁周期,以便使摄像头的拍摄曝光时刻与指示灯闪烁保持严格同步。假如摄像头的帧率是60FPS,则一帧图像时间大约为T=16.667ms,一帧 图像时间包含曝光时间8.33ms和图像存储时间8.33ms,这里可以假设都是8ms,则对应的指示灯闪烁的亮暗时间分别对应为8ms和8ms,如图7所示。
一些实施例中,处理器300还被配置为:在控制设备亮灯时,通过摄像头拍摄包含控制设备的图像;根据图像对控制设备进行定位追踪。
具体地,处理器300计算出指示灯24在空间中的位置,并根据指示灯24在控制设备20上的空间布局,结合PNP算法算处控制设备20在现实空间中的位置,从而实现控制设备在三维空间中的定位和追踪。
进一步地,处理器300基于计算出的结果对控制设备的位置进行平滑和预测,保证控制设备定位追踪的时效性和流畅性。
补充说明的是,本申请提供的方案中,指示灯发出的光可以是可见光或红外光等。
下述为本申请装置实施例,可以用于执行本申请方法实施例。对于本申请装置实施例中未披露的细节,请参照本申请方法实施例。
图15为本申请一实施例提供的同步校准装置的结构示意图。本申请实施例提供一种同步校准装置,应用于显示设备。如图15所示,该同步校准装置110包括:获取模块111、处理模块112和发送模块113。其中:
获取模块111,用于获取摄像头拍摄得到的当前帧图像在拍摄过程中的帧数。
处理模块112,用于在帧数为预设帧数的整数倍时,根据当前帧图像的曝光起始时刻和当前系统时间,确定对控制设备上指示灯进行同步校准的亮灯延时时间。
发送模块113,用于通过通信器发送亮灯延时时间给控制设备,控制设备为与显示设备配对成功的设备。
本申请实施例提供的装置,可用于执行图13所示实施例中显示设备执行的步骤,其实现原理和技术效果类似,在此不再赘述。
一些实施例中,若在当前帧图像之后的N帧图像后进行同步校准,N为自然数,则处理模块112可以具体用于:确定当前帧图像的曝光起始时刻和当前系统时间的时间偏差值;根据时间偏差值与获取所述N帧图像的时长,确定对控制设备上指示灯进行同步校准的亮灯延时时间。
一些实施例中,处理模块112还可以用于:根据摄像头的帧率,确定控制设备上指示灯的闪烁周期,闪烁周期与帧率的乘积为1;触发发送模块113发送闪烁周期给控制设备。
一些实施例中,处理模块112还可以用于:在控制设备亮灯时,通过摄像头拍摄包含控制设备的图像;根据图像对控制设备进行定位追踪。
图16为本申请另一实施例提供的同步校准装置的结构示意图。本申请实施例提供一种同步校准装置,应用于控制设备,该控制设备为与上述显示设备配对成功的设备。如图16所示,该同步校准装置120包括:接收模块121和处理模块122。其中:
接收模块121,用于接收显示设备发送的亮灯延时时间。其中,亮灯延时时间用于指示控制设备上指示灯进行同步校准的亮灯延时时间。
处理模块122,用于根据亮灯延时时间,确定控制设备上指示灯的亮灯起始时刻;以及,自亮灯起始时刻开始,控制指示灯周期性闪烁。
本申请实施例提供的装置,可用于执行图13所示实施例中控制设备执行的步骤,其实现原理和技术效果类似,在此不再赘述。
一些实施例中,指示灯的闪烁周期是根据显示设备的摄像头的帧率确定的,闪烁周期 与帧率的乘积为1。
在虚拟现实设备的使用过程中,VR设备上的摄像头会拍摄手柄(控制设备)的图像,并对图像中由于指示灯闪烁形成的光斑进行提取并编码,根据编码信息确定出指示灯对应的亮斑。最后,VR设备根据指示灯对应的光斑在图像中的位置,完成对手柄的定位。然而,在真实环境中,手柄周围可能会存在其他光源,从而在手柄的图像上形成干扰光斑,影响了手柄定位的准确性。
为解决上述问题,本申请某些实施例提供一种虚拟现实设备以及手柄定位方法,通过手柄上的指示灯预设的编码信息去除手柄的图像中的干扰光斑,从而提高了手柄定位的准确性。
可以理解,上述手柄定位方法可以通过本申请实施例提的虚拟现实设备实现。下面以集成或安装有相关执行代码的虚拟现实设备为例以具体地实施例对本申请实施例的技术方案进行详细说明。下面这几个具体的实施例可以相互结合,对于相同或相似的概念或过程可能在某些实施例不再赘述。
图17示例性示出了根据一些实施例的一种手柄定位的流程示意图,本实施例涉及的是如何定位手柄的具体过程。其中,本实施例的执行主体为虚拟现实设备,该虚拟现实设备与手柄连接。如图17所示,该方法包括:
S301、采集手柄的多帧图像,手柄设置有至少一个指示灯。
在申请实施例中,当手柄接入VR设备后,若VR设备接收到手柄进行定位的指示,则可以采集手柄的多帧图像。
本申请实施例对于如何采集手柄的多帧图像不做限制,在一些实施例中,VR设备上可以携带有摄像头,通过调用VR设备上携带的摄像头可以采集手柄的多帧图像。在另一些实施例中,VR设备也可以外接有其他摄像设备,VR设备通过向外接的摄像设备发送指示信息,可以调用外接的摄像设备采集手柄的多帧图像。其中,本申请实施例对于摄像头的类型也不做限制,示例性的,可以为单目相机。
在本申请中,手柄上设置可以有至少一个指示灯,在VR设备采集手柄的多帧图像时,手柄上的指示灯可以进行闪烁,以使采集到的多帧图像中包含有指示灯对应的光斑,从而辅助VR设备对手柄进行定位。其中,本申请实施例对于指示灯的类型也不做限制,示例性的,可以为发光二极管(light-emitting diode light,LED)。
需要说明的是,本申请实施例对于指示灯的数量以及指示灯在手柄上的排布方式均不作限制,可以根据实际情况具体设置。示例性的,图7b-7d示例性示出了根据一些实施例的指示灯排布示意图。如图7b-7d所示,针对不同形状的手柄,指示灯均可以依据手柄的边缘依次排布。应理解,本申请实施例中的指示灯的颜色可以为相同,也可以为不同,可以根据实际情况具体设置。
应理解,为了使指示灯按照预设的闪烁周期进行闪烁,在一些实施例中,在采集手柄的多帧图像之前,VR设备可以定期向手柄发送同步信息,该同步信息包括有至少一个指示灯的闪烁周期。手柄在接收到同步信息后,可以控制指示灯按照闪烁周期进行闪烁。与此同时,VR设备可以同步地进行图像录制,从而使得摄像头曝光拍摄到的手柄的图像正好是手柄的指示灯灯亮时的图像。
S302、从手柄的多帧图像中提取至少一个光斑。
在本步骤中,当VR设备采集手柄的多帧图像后,VR设备中的控制器可以从手柄的 多帧图像中提取至少一个光斑。
本申请实施例对于如何提取光斑不做限制,在一些实施例中,VR设备中的控制器可以将手柄的多帧图像依次输入到图像识别算法模型中,以获取图像识别算法模型输入的各帧图像中提取的至少一个光斑。示例性的,该图像识别算法模型可以为opecv blob光斑点提取算法模型。该opecv blob光斑点提取算法模型可以对图像中相同像素的连通域进行分析,并在对图像进行二值化处理后,提取出图像中的光斑。
S303、对至少一个光斑进行编码,形成第一编码信息。
在本步骤中,当VR设备中的控制器从手柄的多帧图像中提取至少一个光斑后,可以对至少一个光斑进行编码,形成第一编码信息。
在一些实施例中,VR设备中的控制器可以先根据至少一个光斑在多帧图像中的直径变化,确定至少一个光斑在多帧图像中的亮暗变化。随后,VR设备中的控制器再根据多帧图像中的亮暗变化,对至少一个光斑进行编码,形成第一编码信息。
示例性的,图18a-18b示例性示出了根据一些实施例的亮斑的示意图,如图18a-18b所示,图18a所示的亮斑的直径可以设置为R0,图18b所示的亮斑的直径可以设置为R1,图18a为图18b前一帧的图像。
若R0>m*R1,则确定该光斑的直径由大变小,其对应的指示灯由亮变暗,则相应的,该光斑在图18a中的编码可以设置为1,在图18b中的编码可以设置为0。其中,m为常数,可以根基实际情况具体设置,例如m为1.3、1.5等。
若R0<n*R1,则确定该光斑的直径由小变大,其对应的指示灯由暗变亮,则相应的,该光斑在图18a中的编码可以设置为0,在图18b中的编码可以设置为1。其中,n为常数,可以根基实际情况具体设置,例如m为1.3、1.5等。
在本申请实施例中,在完成各帧图像中各个亮斑的编码后,可以将相同亮斑的编码信息按照时间顺序组合,从而形成该光斑对应的第一编码信息。应理解,本申请实施例对于第一编码信息包含的图像的帧数不做限制,示例性的,可以为五帧。
S304、根据第一编码信息以及至少一个指示灯的第二编码信息,去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑。
在本步骤中,当VR设备确定第一编码信息后,可以将第一编码信息和VR设备中预先设置的至少一个指示灯的第二编码信息进行比较,从而去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑。
示例性的,表1为指示灯的第二编码信息表。
表1
在本申请中,存在与第一编码信息相同的第二编码信息,则确定第一编码信息对应的光斑为至少一个指示灯对应的光斑。相应的,若不存在与第一编码信息相同的第二编码信息,则确定第一编码信息对应的光斑为干扰光斑,随后,去除干扰光斑。
S305、根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。
在本步骤中,在VR设备中的控制器去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑后,可以根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。
在一些实施例中,VR设备中的控制器可以根据至少一个指示灯对应的光斑在多帧图像中的位置确定指示灯在手柄上的空间布局。随后,通过多点姿态(Perspective-n-Point,PnP)算法,确定手柄在三维控件中的位置,从而实现手柄的定位与追踪。
应理解,本申请中对于手柄的定位与追踪,不仅可以用于3DOF(坐标),也可以运用于6DOF,本申请实施例对此不做限制。
在另一些可以的实施方式中,在完成手柄的定位与追踪后,可以进一步对手柄位置进行平滑和预测,从而提高手柄的定位追踪的时效性和流畅性。
本申请实施例提供的手柄定位方法,通过采集手柄的多帧图像,手柄设置有至少一个指示灯。其次,从手柄的多帧图像中提取至少一个光斑,并对至少一个光斑进行编码,形成第一编码信息。再次,根据第一编码信息以及至少一个指示灯的第二编码信息,去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑。最后根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。与相关技术相比,本申请实施例在手柄定位时可以通过指示灯预设的编码信息去除干扰光斑,从而提高了手柄定位的准确性。
在上述实施例的基础上,下面对于虚拟现实设备如何去除手柄图像中的干扰光斑进行说明。图19示例性示出了根据一些实施例的另一种手柄定位的流程示意图,如图19所示,该方法包括:
S901、采集手柄的多帧图像,手柄设置有至少一个指示灯。
S902、从手柄的多帧图像中提取至少一个光斑。
S901-S902的技术名词、技术效果、技术特征,以及可选实施方式,可参照图17所示的S301-S302理解,对于重复的内容,在此不再累述。
S903、根据至少一个光斑在多帧图像中的直径变化,确定至少一个光斑在多帧图像中的亮暗变化。
S904、根据多帧图像中的亮暗变化,对至少一个光斑进行编码,形成第一编码信息。
在一些实施例中,根据多帧图像中的亮暗变化,以及光斑的亮暗与位图信息的对应关系,对至少一个光斑进行编码,形成第一编码信息。
示例性的,继续参考图18a-18b,光斑的亮暗与位图信息的对应关系,可以例如,相邻的两帧图像中的同一个光斑,较亮对应1,较案对应0。
S905、判断是否存在与第一编码信息相同的第二编码信息。
若是,则执行步骤S906,若否,则执行步骤S907。
S906、确定第一编码信息对应的光斑为至少一个指示灯对应的光斑。
S907、确定第一编码信息对应的光斑为干扰光斑。
S908、去除干扰光斑。
而后,根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。
本申请实施例提供的手柄定位方法,通过采集手柄的多帧图像,手柄设置有至少一个指示灯。其次,从手柄的多帧图像中提取至少一个光斑,并对至少一个光斑进行编码,形成第一编码信息。再次,根据第一编码信息以及至少一个指示灯的第二编码信息,去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑。最后根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。与相关技术相比,本申请实施例在手柄定位时可以通过指示灯预设的编码信息去除干扰光斑,从而提高了手柄定位的准确性。
图2示例性示出了根据一些实施例的显示设备的结构示意图。该显示设备可以通过软件、硬件或者两者的结合实现,以执行上述实施例中的手柄定位方法。如图2所示,该虚拟显示设备10包括:显示器100、摄像头200和处理器300。显示设备具体为虚拟现实设备。
摄像头,被配置为采集手柄的多帧图像,手柄与虚拟现实设备连接,手柄设置有至少一个指示灯;
与摄像头连接的处理器,处理器被配置为:
从手柄的多帧图像中提取至少一个光斑;
对至少一个光斑进行编码,形成第一编码信息;
根据第一编码信息以及至少一个指示灯的第二编码信息,去除至少一个光斑中的干扰光斑,并确定至少一个指示灯对应的光斑;
根据至少一个指示灯对应的光斑在多帧图像中的位置,确定的手柄的位置。
在本申请一些实施例中,处理器具体被配置为:
根据至少一个光斑在多帧图像中的直径变化,确定至少一个光斑在多帧图像中的亮暗变化;
根据多帧图像中的亮暗变化,对至少一个光斑进行编码,形成第一编码信息。
在本申请一些实施例中,处理器具体被配置为:
根据多帧图像中的亮暗变化,以及光斑的亮暗与位图信息的对应关系,对至少一个光斑进行编码,形成第一编码信息。
在本申请一些实施例中,处理器具体被配置为:
若存在与第一编码信息相同的第二编码信息,则确定第一编码信息对应的光斑为至少一个指示灯对应的光斑;
若不存在与第一编码信息相同的第二编码信息,则确定第一编码信息对应的光斑为干扰光斑;
去除干扰光斑。
在本申请一些实施例中,处理器还被配置为:
向手柄发送同步信息,同步信息包括有至少一个指示灯的闪烁周期。
需要说明的是,应理解以上装置的各个模块的划分仅仅是一种逻辑功能的划分,实际实现时可以全部或部分集成到一个物理实体上,也可以物理上分开。且这些模块可以全部以软件通过处理元件调用的形式实现;也可以全部以硬件的形式实现;还可以部分模块通过处理元件调用软件的形式实现,部分模块通过硬件的形式实现。例如,处理模块可以为 单独设立的处理元件,也可以集成在上述装置的某一个芯片中实现,此外,也可以以程序代码的形式存储于上述装置的存储器中,由上述装置的某一个处理元件调用并执行以上处理模块的功能。其它模块的实现与之类似。此外这些模块全部或部分可以集成在一起,也可以独立实现。这里该的处理元件可以是一种集成电路,具有信号的处理能力。在实现过程中,上述方法的各步骤或以上各个模块可以通过处理器元件中的硬件的集成逻辑电路或者软件形式的指令完成。
例如,以上这些模块可以是被配置成实施以上方法的一个或多个集成电路,例如:一个或多个ASIC(Application Specific Integrated Circuit,特定集成电路),或,一个或多个DSP(Digital Signal Processor,数字信号处理器),或,一个或者多个FPGA(Field Programmable Gate Array,现场可编程门阵列)等。再如,当以上某个模块通过处理元件调度程序代码的形式实现时,该处理元件可以是通用处理器,例如CPU或其它可以调用程序代码的处理器。再如,这些模块可以集成在一起,以SOC(System-on-a-Chip,片上系统)的形式实现。
在上述实施例中,可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时,可以全部或部分地以计算机程序产品的形式实现。该计算机程序产品包括一个或多个计算机程序。在计算机上加载和执行该计算机程序指令时,全部或部分地产生按照本申请实施例该的流程或功能。该计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。该计算机程序可以存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,该计算机程序可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。该计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。该可用介质可以是磁性介质,(例如,软盘、硬盘、磁带)、光介质(例如,DVD)、或者半导体介质(例如固态硬盘solid state disk(SSD))等。
本申请实施例还提供一种计算机可读存储介质,该计算机可读存储介质中存储有计算机程序,当计算机程序被处理器执行时实现如上任一方法实施例该的方法。
本申请实施例还提供一种显示系统,包括如上所述的该显示设备10和该手柄20。
本申请实施例还提供一种运行指令的芯片,芯片用于执行如上任一方法实施例该的方法。
本申请实施例还提供一种计算机程序产品,该计算机程序产品包括计算机程序,该计算机程序存储在计算机可读存储介质中,至少一个处理器可以从该计算机可读存储介质中读取计算机程序,该至少一个处理器执行该计算机程序时可实现如上任一方法实施例该的方法。
最后应说明的是:以上各实施例仅用以说明本申请的技术方案,而非对其限制;尽管参照前述各实施例对本申请进行了详细的说明,本领域的普通技术人员应当理解:其依然可以对前述各实施例所记载的技术方案进行修改,或者对其中部分或者全部技术特征进行等同替换;而这些修改或者替换,并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。
为了方便解释,已经结合具体的实施方式进行了上述说明。但是,上述示例性的讨论 不是意图穷尽或者将实施方式限定到上述公开的具体形式。根据上述的教导,可以得到多种修改和变形。上述实施方式的选择和描述是为了更好的解释原理以及实际的应用,从而使得本领域技术人员更好的使用该实施方式以及适于具体使用考虑的各种不同的变形的实施方式。
Claims (10)
- 一种显示设备,与手柄通信连接,其特征在于,包括:显示器,用于显示界面;摄像头,用于获取图像数据;分别与所述摄像头和所述显示器连接的处理器,所述处理器被配置为:获取至少一个连续的拍摄存储周期的时延偏差值,所述时延偏差值为所述拍摄存储周期内所述手柄灯的亮灯起始时刻,与所述摄像头的拍摄起始时刻之间的差值;其中,所述拍摄存储周期时长与所述手柄灯状态的闪烁周期时长相同;当所述至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向所述手柄发送同步校准指令,所述同步校准指令用于指示将所述手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。
- 根据权利要求1所述的设备,其特征在于,所述向所述手柄发送同步校准指令之前,所述处理器还被配置为:确定N个拍摄存储周期中起始的拍摄存储周期中所述手柄灯的亮灯起始时刻和所述N个拍摄存储周期的时延偏差值的累计值之间的差值,为所述起始校准时刻。
- 根据权利要求2所述的设备,其特征在于,所述预设偏差值为所述拍摄存储周期的四分之一时长。
- 根据权利要求1至3任一项所述的设备,其特征在于,所述处理器还被配置为:获取所述手柄灯的拍摄图像,所述拍摄图像为所述手柄亮灯时拍摄得到的图像;根据所述拍摄图像进行所述手柄在虚拟现实场景中的定位追踪。
- 一种手柄,其特征在于,所述手柄上设置有手柄灯,包括:收发器,被配置为接收显示设备发送的同步校准指令,所述同步校准指令用于指示将所述手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,其中,所述显示设备的拍摄存储周期时长与所述手柄灯状态的闪烁周期时长相同;控制器,被配置为根据所述同步校准指令校准所述手柄灯在下一个拍摄存储周期的亮灯起始时刻。
- 一种虚拟目标定位追踪的校准方法,其特征在于,应用于显示设备,所述显示设备与手柄通信连接,所述方法包括:获取至少一个连续的拍摄存储周期的时延偏差值,所述时延偏差值为所述拍摄存储周期内所述手柄灯的亮灯起始时刻,与所述摄像头的拍摄起始时刻之间的差值;其中,所述拍摄存储周期时长与所述手柄灯状态的闪烁周期时长相同;当所述至少一个连续的拍摄存储周期中的N个拍摄存储周期的时延偏差值的累计值大于预设偏差值时,向所述手柄发送同步校准指令,所述同步校准指令用于指示将所述手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,N大于或等于1。
- 根据权利要求6所述的方法,其特征在于,所述向所述手柄发送同步校准指令之前,还包括:确定N个拍摄存储周期内所述手柄灯首次亮灯起始时刻和所述N个拍摄存储周期的累计值之间的差值,为所述起始校准时刻。
- 根据权利要求7所述的方法,其特征在于,所述预设偏差值为所述拍摄存储周期 的四分之一时长。
- 根据权利要求6至8任一项所述的方法,其特征在于,所述方法还包括:获取所述手柄灯的拍摄图像,所述拍摄图像为所述手柄亮灯时拍摄得到的图像;根据所述拍摄图像进行所述手柄在虚拟现实场景中的定位追踪。
- 一种虚拟目标定位追踪的校准方法,应用于手柄,所述手柄与显示设备通信连接,其特征在于,包括:接收所述显示设备发送的同步校准指令,所述同步校准指令用于指示将所述手柄灯在下一个拍摄存储周期的亮灯起始时刻校准为起始校准时刻,其中,所述显示设备的拍摄存储周期时长与所述手柄灯状态的闪烁周期时长相同;根据所述同步校准指令校准所述手柄灯在下一个拍摄存储周期的亮灯起始时刻。
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011260412.0 | 2020-11-12 | ||
CN202011260735.XA CN114500979B (zh) | 2020-11-12 | 2020-11-12 | 显示设备、控制设备以及同步校准方法 |
CN202011260409.9A CN114500978B (zh) | 2020-11-12 | 2020-11-12 | 显示设备、手柄以及虚拟目标定位追踪的校准方法 |
CN202011260735.X | 2020-11-12 | ||
CN202011260412.0A CN114489310A (zh) | 2020-11-12 | 2020-11-12 | 虚拟现实设备以及手柄定位方法 |
CN202011260409.9 | 2020-11-12 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022100288A1 true WO2022100288A1 (zh) | 2022-05-19 |
Family
ID=81600831
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/119626 WO2022100288A1 (zh) | 2020-11-12 | 2021-09-22 | 显示设备、手柄以及虚拟目标定位追踪的校准方法 |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2022100288A1 (zh) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112206511A (zh) * | 2020-10-15 | 2021-01-12 | 网易(杭州)网络有限公司 | 游戏中的动作同步方法、装置、电子设备及存储介质 |
WO2024017045A1 (zh) * | 2022-07-21 | 2024-01-25 | 华为技术有限公司 | 定位方法、系统及电子设备 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170131774A1 (en) * | 2015-11-10 | 2017-05-11 | Oculus Vr, Llc | Control for a virtual reality system including opposing portions for interacting with virtual objects and providing tactile feedback to a user |
CN106768361A (zh) * | 2016-12-19 | 2017-05-31 | 北京小鸟看看科技有限公司 | 与vr头戴设备配套的手柄的位置追踪方法和系统 |
US20190325274A1 (en) * | 2018-04-24 | 2019-10-24 | Microsoft Technology Licensing, Llc | Handheld object pose determinations |
CN110568753A (zh) * | 2019-07-30 | 2019-12-13 | 青岛小鸟看看科技有限公司 | 一种手柄、头戴设备、头戴系统及其时间同步方法 |
CN111355897A (zh) * | 2018-12-24 | 2020-06-30 | 海信视像科技股份有限公司 | 一种灯光控制方法及装置 |
-
2021
- 2021-09-22 WO PCT/CN2021/119626 patent/WO2022100288A1/zh active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170131774A1 (en) * | 2015-11-10 | 2017-05-11 | Oculus Vr, Llc | Control for a virtual reality system including opposing portions for interacting with virtual objects and providing tactile feedback to a user |
CN106768361A (zh) * | 2016-12-19 | 2017-05-31 | 北京小鸟看看科技有限公司 | 与vr头戴设备配套的手柄的位置追踪方法和系统 |
US20190325274A1 (en) * | 2018-04-24 | 2019-10-24 | Microsoft Technology Licensing, Llc | Handheld object pose determinations |
CN111355897A (zh) * | 2018-12-24 | 2020-06-30 | 海信视像科技股份有限公司 | 一种灯光控制方法及装置 |
CN110568753A (zh) * | 2019-07-30 | 2019-12-13 | 青岛小鸟看看科技有限公司 | 一种手柄、头戴设备、头戴系统及其时间同步方法 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112206511A (zh) * | 2020-10-15 | 2021-01-12 | 网易(杭州)网络有限公司 | 游戏中的动作同步方法、装置、电子设备及存储介质 |
WO2024017045A1 (zh) * | 2022-07-21 | 2024-01-25 | 华为技术有限公司 | 定位方法、系统及电子设备 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP4083900B1 (en) | Virtual reality experience sharing | |
WO2022100288A1 (zh) | 显示设备、手柄以及虚拟目标定位追踪的校准方法 | |
EP3598274B1 (en) | System and method for hybrid eye tracker | |
US11463628B1 (en) | Systems and methods for synchronizing image sensors | |
WO2020082902A1 (zh) | 视频的音效处理方法及相关产品 | |
CN107079565B (zh) | 照明设备 | |
CN107656635B (zh) | 视觉系统及控制视觉系统的方法 | |
JP6276394B2 (ja) | 画像キャプチャ入力および投影出力 | |
US10628711B2 (en) | Determining pose of handheld object in environment | |
CN111654746A (zh) | 视频的插帧方法、装置、电子设备和存储介质 | |
TWI644235B (zh) | Wearable instant interaction system | |
US20180241941A1 (en) | Image processing apparatus, image processing method, and image pickup apparatus | |
WO2021061326A1 (en) | Automated video capture and composition system | |
JP2020173670A (ja) | 複数のマーカを備えたデバイス | |
KR20130129174A (ko) | 다중 3d 디스플레이들/장치들을 위한 셔터 신호들의 동기화 | |
TWI781357B (zh) | 三維影像處理方法、攝影裝置及非暫態電腦可讀取儲存媒體 | |
CN113721767A (zh) | 一种手柄的追踪方法、装置、系统及介质 | |
WO2019187801A1 (ja) | 情報処理装置、情報処理方法、および、プログラム | |
CN114500978B (zh) | 显示设备、手柄以及虚拟目标定位追踪的校准方法 | |
CN114500979B (zh) | 显示设备、控制设备以及同步校准方法 | |
KR101519030B1 (ko) | 감성광고 기능을 구비한 스마트 tv | |
CN112153442A (zh) | 播放方法、装置、终端、电视设备、存储介质及电子设备 | |
JP2015159381A (ja) | 情報処理装置、データ生成装置、情報処理方法、および情報処理システム | |
EP4210318A2 (en) | Data processing system, method for determining coordinates, and computer readable storage medium | |
EP3629140A1 (en) | Displaying method, animation image generating method, and electronic device configured to execute the same |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21890816 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21890816 Country of ref document: EP Kind code of ref document: A1 |