WO2017173735A1 - Video see-through-based smart eyeglasses system and see-through method thereof - Google Patents

Video see-through-based smart eyeglasses system and see-through method thereof Download PDF

Info

Publication number
WO2017173735A1
WO2017173735A1 PCT/CN2016/086348 CN2016086348W WO2017173735A1 WO 2017173735 A1 WO2017173735 A1 WO 2017173735A1 CN 2016086348 W CN2016086348 W CN 2016086348W WO 2017173735 A1 WO2017173735 A1 WO 2017173735A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
module
unit
video
video image
Prior art date
Application number
PCT/CN2016/086348
Other languages
French (fr)
Chinese (zh)
Inventor
艾韬
Original Assignee
深圳市易瞳科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市易瞳科技有限公司 filed Critical 深圳市易瞳科技有限公司
Publication of WO2017173735A1 publication Critical patent/WO2017173735A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Definitions

  • the invention relates to the technical field of smart glasses, in particular to a smart glasses system based on video perspective and a seeing method thereof.
  • Augmented Reality is a technology that combines the position and angle of real-time camera images with corresponding images.
  • Smart glasses based on AR technology superimpose and interact digital content on the near-eye screen in the real world.
  • the AR smart glasses not only display the real world information, but also display the virtual information at the same time, and the two kinds of information complement each other and superimpose.
  • AR technology not only has a wide range of applications in applications similar to Virtual Reality (VR) technology, such as aircraft development and development, data model visualization, virtual training, entertainment and art, but also because of its
  • VR Virtual Reality
  • the ability to enhance the display output of the real environment has more obvious advantages than VR technology in medical research and anatomical training, precision instrument manufacturing and maintenance, military aircraft navigation, engineering design and remote robot control.
  • users can use the helmet display to combine the real world with computer graphics to see the real world around it. As the computing power of portable electronic products increases, the use of augmented reality is expected to become more widespread.
  • optical see-through realizes the translucent display effect through the beam splitting prism, so that the wearer can see the digital picture at the same time while seeing the real world.
  • Video perspective glasses combine video and digital images generated by the camera through video synthesis technology, which enables a larger range of viewing angles than optical perspective.
  • the smart glasses system based on video perspective captures the reality picture through the binocular camera placed in front of the glasses, and the glasses need to be displayed in real time without distortion in the high-definition screen without distortion.
  • the information that the wearer perceives the position, size, and delay of the scene through the glasses is basically consistent with the external scene information directly perceived by the naked eye. Based on this, the present invention proposes a smart glasses system based on video perspective, thereby maximally reducing the delay of the video picture and improving the viewing experience of the user.
  • the invention provides a smart glasses system based on video perspective and a see-through method thereof, aiming at solving the technical problem of video picture delay existing in the existing AR smart glasses.
  • a smart glasses system based on video perspective, comprising smart glasses and a data processing device, the smart glasses being connected with a data processing device, the smart glasses including a camera module , image preprocessing module, image fusion module and display
  • the camera module is configured to collect a video image, and transmit the captured video image to an image preprocessing module
  • the image preprocessing module is configured to preprocess the video image, and then lower the clear video image and the high definition video.
  • the images are respectively transmitted to the data processing device and the image fusion module;
  • the data processing device is configured to generate a virtual digital image by an algorithm, and transmit the virtual digital image to the image fusion module;
  • the image fusion module is configured to use the virtual digital image and the high definition image
  • the video image is fused and output to the display module for display.
  • the technical solution adopted by the embodiment of the present invention further includes: an attitude sensing module, wherein the posture sensing module is disposed on the smart glasses, and is connected to the data processing device for collecting posture information of the smart glasses wearer, and The attitude information is transmitted to the data processing device.
  • the technical solution adopted by the embodiment of the present invention further includes: an optical device disposed on a side of the display module close to the human eye for assisting the human eye to focus the display image of the display module.
  • the technical solution adopted by the embodiment of the present invention further includes: the camera module is a binocular camera, the binocular cameras are respectively disposed on left and right lenses of the smart glasses, and the front end of the binocular camera is further mounted with a wide-angle lens.
  • the image preprocessing module includes a preprocessing unit, a distortion correcting unit, a stereo correcting unit, a depth calculating unit, and an image splicing unit;
  • the pre-processing unit is configured to process and process a video image transmitted by the camera module, and obtain configuration parameters of the camera module by processing the result;
  • the distortion correcting unit is configured to repair the distortion of the camera module by using a configuration parameter
  • the stereo correcting unit is configured to pull the screen of the binocular camera back to a plane by the configuration parameter, and align the binocular cameras with each other;
  • the depth calculation unit is configured to calculate an optimal binocular matching result using a local matching algorithm, and calculate a depth video image by using the parallax;
  • the image splicing unit is configured to splicing the unscaled high-definition video images to the image fusion module, and splicing the scaled low-definition video images, the depth video images, and the auxiliary images to the data processing device.
  • the data processing device includes an image splitting unit, a scene analyzing unit, a map building unit, a map updating unit, a posture analyzing unit, a model rendering unit, and an image superimposing unit;
  • the image splitting unit is configured to split the stitched video image
  • the scene analysis unit is configured to analyze an observation scenario
  • the map construction unit is configured to construct a map to achieve autonomous positioning
  • the map update unit is configured to store map information of the location environment, and construct an incremental map on the constructed map
  • the posture analysis unit is configured to analyze the wearer's perspective posture by integrating the visual algorithm with the attitude sensing module;
  • the model rendering unit is configured to render a digital model through a perspective gesture
  • the image superimposing unit is configured to superimpose the angle of view gesture and the rendered digital model on the video image, generate a virtual digital image with pose information, and output the virtual digital image to the image fusion module.
  • the image fusion module further includes an expansion unit, a synchronization unit, and a fusion unit;
  • the expansion unit is configured to expand the virtual digital image by the difference calculation, so that the resolution and the frame rate of the virtual digital image and the high-definition video image are consistent;
  • the synchronization unit is configured to align the virtual digital image with the two-way video of the high-definition video image by the synchronization signal of the high-definition video image, and analyze the high-definition video image portion to be covered by the virtual digital image when merging;
  • the merging unit is configured to superimpose the virtual digital image into the corresponding frame of the high-definition video image according to the analysis result of the synchronization unit, fuse the virtual digital image and the high-definition video image, and output the fused video image to the display module for display. .
  • Another technical solution adopted by the embodiment of the present invention is: a method for seeing a smart glasses based on video perspective, comprising the following steps:
  • Step 100 Acquire a video image through the camera module, and transmit the collected video image to an image preprocessing module;
  • Step 200 Preprocessing the video image by the image preprocessing module, and transmitting the low clear video image and the high definition video image to the data processing device and the image fusion module respectively;
  • Step 300 The data processing device generates a virtual digital image by an algorithm, and transmits the virtual digital image to the image fusion module.
  • Step 400 The virtual digital image is merged with the high-definition video image through the image fusion module, and then output to the display module for display.
  • the technical solution adopted by the embodiment of the present invention further includes: in step 200, the image preprocessing module performs preprocessing on the video image by processing: processing the video image transmitted by the camera module, and obtaining the processing result by using the processing result
  • the configuration parameters of the camera module the distortion of the camera module is repaired by the configuration parameters, and the screen of the binocular camera is pulled back to a plane to align the binocular cameras; the local matching algorithm is used to calculate the matching result with the optimal binocular, through the parallax
  • the depth video image is calculated; the unscaled high-definition video image is spliced and directly output to the image fusion module, and the scaled low-definition video image, the depth video image, and the auxiliary image are spliced and output to the data.
  • the image preprocessing module performs preprocessing on the video image by processing: processing the video image transmitted by the camera module, and obtaining the processing result by using the processing result
  • the configuration parameters of the camera module the distortion of the camera module is
  • the technical solution adopted by the embodiment of the present invention further includes: the step 300 further includes: acquiring the posture information of the wearer through the attitude sensing module, and transmitting the posture information to the data processing device; the data processing device generating the virtual digital image
  • the method includes: splitting the spliced video image; analyzing the observed scene; constructing the map to realize autonomous positioning; storing map information of the location environment, constructing an incremental map on the existing map; and transmitting the visual algorithm and the gesture
  • the sensor module is integrated to analyze the wearer's perspective posture; the digital model is rendered by the angle of view gesture; the perspective pose and the rendered digital model are superimposed on the video image to generate a virtual digital image with pose information.
  • the present invention has the beneficial effects that the video perspective-based smart glasses system and the perspective method thereof according to the embodiments of the present invention transmit the high-definition video image directly from the image pre-processing module to the image fusion module, and The low-resolution low-frame rate low-definition video image is output to the data processing device for augmented reality processing, and the virtual digital image with slight delay is merged into the high-definition video image through the image fusion module, thereby reducing the data processing device augmented reality algorithm.
  • the amount of calculation increases the image quality while minimizing the delay of the video picture, greatly reducing the vertigo of the video perspective and improving the wearer's comfort.
  • FIG. 1 is a schematic structural diagram of a smart glasses system based on video perspective according to an embodiment of the present invention
  • FIG. 2 is a circuit diagram of a smart glasses system based on video perspective according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of image processing of an image preprocessing module according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a stitching image of an image stitching unit according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of image processing of a data processing apparatus according to an embodiment of the present invention.
  • FIG. 6 is a flow chart of a method for seeing a smart glasses based on video perspective according to an embodiment of the present invention.
  • FIG. 1 is a schematic structural diagram of a smart glasses system based on video perspective according to an embodiment of the present invention
  • FIG. 2 is a circuit diagram of a smart glasses system based on video perspective according to an embodiment of the present invention.
  • the video perspective-based smart glasses system of the embodiment of the present invention includes smart glasses and a data processing device.
  • the smart glasses include a camera module, an image preprocessing module, an attitude sensing module, an image fusion module, a display module and an optical component; wherein the camera module is connected with the image preprocessing module for acquiring video images in real time and collecting the captured video.
  • the image is transmitted to the image preprocessing module; the image preprocessing module is respectively connected with the image fusion module and the data processing device, and is configured to receive the video image transmitted by the camera module, and preprocess the video image by the chip algorithm, and then LD (low)
  • the video image and the HD (high definition) video image are respectively transmitted to the data processing device and the image fusion module; the attitude sensing module is disposed on the smart glasses and is connected with the data processing device for collecting the posture information of the smart glasses wearer.
  • the data processing device is connected to the image fusion module, and is configured to receive the LD video image transmitted by the image preprocessing module and the posture information transmitted by the attitude sensing module, and the digital content and the posture information are obtained through a correlation algorithm.
  • superimposed on the LD video image to generate virtual with pose information Digital images, digital and virtual image transfer to the image integration module; which pose clear coordinates information between the virtual digital images and high-definition video images.
  • the image fusion module is connected to the display module for respectively receiving the HD video image (substantially no delay) transmitted by the image preprocessing module and the virtual digital image transmitted by the data processing device (with a little delay), and the virtual digital image and the HD
  • the video image is fused and output to the display module for display;
  • the optical device is disposed on the side of the display module close to the human eye, and is used to help the human eye focus on the display image of the display module.
  • the camera module is a binocular camera, and the binocular cameras are respectively disposed on the left and right lenses of the smart glasses, and the distance between the binocular cameras can be adjusted according to the distance of the wearer; the front end of the binocular camera A wide-angle lens (not shown) is also installed for collecting external scenes of a large scene; the display module is a smart glasses display, and the optical components are composed of achromatic lens groups, the purpose of which is to help the human eye focus on the near-eye screen content, Significantly improve image quality without the need for color correction on the software.
  • the image preprocessing module uses an FPGA (Field-Programmable Gate Array) as the main computing device, and processes the video image captured by the camera module through a chip-based algorithm, and can pass related algorithms.
  • the parameters of the camera module are configured.
  • FIG. 3 is a schematic diagram of image processing of the image preprocessing module according to the embodiment of the present invention.
  • the image preprocessing module further includes a preprocessing unit, a distortion correction unit, a stereo correction unit, and a depth calculation.
  • the unit and the image splicing unit, the preprocessing unit, the distortion correcting unit, the stereo correcting unit, the depth calculating unit and the image splicing unit are sequentially connected;
  • the preprocessing unit is configured to process and process the video image transmitted by the camera module, and obtain configuration parameters of the camera module by processing the result;
  • the configuration parameters of the camera module specifically include: color filter arrangement, white balance adjustment, automatic exposure, and auto focus High dynamic range image generation and mapping.
  • the distortion correcting unit is configured to eliminate the distortion of the camera module by using the configuration parameters obtained by the calibration;
  • the stereo correction unit is configured to pull the screen of the binocular camera back to one by using the configuration parameters obtained by the calibration Plane and align the binocular cameras;
  • the depth calculation unit is configured to calculate the optimal binocular matching result with the line using the local matching algorithm, thereby calculating the depth video image by the parallax; the depth video image can also be directly obtained by adding a depth camera, and the obtained depth video image needs to be corrected. It aligns with the original video image.
  • the image splicing unit is configured to splicing the unscaled high-definition video image and output it directly to the image fusion module, and splicing the scaled low-definition video image, the depth video image and other auxiliary images into the data processing device.
  • the low-definition video image comes from the downsampling of the high-definition video image.
  • FIG. 4 it is a schematic diagram of a stitching image of an image splicing unit according to an embodiment of the present invention.
  • the data processing device uses a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit) as an arithmetic device, and implements various applications of augmented reality through a software algorithm.
  • FIG. 5 is a schematic diagram of image processing of a data processing apparatus according to an embodiment of the present invention.
  • the data processing device further includes an image splitting unit, a scene analyzing unit, a map building unit, a map updating unit, a posture analyzing unit, a model rendering unit and an image superimposing unit, an image splitting unit, a scene analyzing unit, a map building unit, and a map updating unit.
  • the posture analysis unit, the model rendering unit, and the image superimposing unit are sequentially connected;
  • the image splitting unit is configured to split the stitched video image
  • the scene analysis unit is configured to analyze the observation scene by an image recognition algorithm or a position estimation algorithm during the movement of the wearer;
  • the map building unit is configured to construct a map based on its own positioning to achieve autonomous positioning
  • the map update unit is configured to store map information of the location environment and construct an incremental map on the existing map
  • the attitude analysis unit is configured to analyze the wearer's perspective posture by integrating the visual algorithm with the attitude sensing module;
  • the model rendering unit is configured to render a corresponding digital model by using the obtained perspective gesture
  • the image superimposing unit is configured to superimpose the angle of view gesture and the rendered digital model on the position corresponding to the environment of the video image, generate a virtual digital image with pose information, and output the virtual digital image to the image fusion module.
  • the image fusion module further includes an expansion unit, a synchronization unit, and a fusion unit, and the extension unit, the synchronization unit, and the fusion unit are sequentially connected;
  • the expansion unit is configured to expand the low-resolution low frame rate virtual digital image by the difference calculation, so that the virtual digital image and the high-definition video image have the same resolution and frame rate;
  • the synchronization unit is configured to align the virtual digital image with the two-way video of the high-definition video image by the synchronization signal of the high-definition video image, and analyze the high-definition video image portion to be covered by the virtual digital image during the fusion;
  • the merging unit is configured to superimpose the virtual digital image into the corresponding frame of the high-definition video image according to the analysis result of the synchronization unit, fuse the virtual digital image and the high-definition video image, and output the fused video image to the display module for display.
  • Augmented reality smart glasses will have a strong sense of vertigo due to the delay of the video screen, making the wearer feel uncomfortable.
  • the high-definition video image is directly sent from the image pre-processing module to the image fusion module, and the low-resolution low-frame rate low-definition video image is output to
  • the data processing device performs augmented reality processing, and the virtual digital image with slight delay is merged into the high-definition video image through the image fusion module, thereby reducing the calculation amount of the augmented reality algorithm of the data processing device.
  • the overlay image is slightly floating because the high-definition video image is updated earlier than the virtual digital image, the two-way (high-definition video image and virtual digital image) separation scheme can greatly reduce the vertigo of the video perspective.
  • FIG. 6 is a flowchart of a method for seeing a smart glasses based on video perspective according to an embodiment of the present invention.
  • the video perspective-based smart glasses perspective method of the embodiment of the invention comprises the following steps:
  • Step 100 Wear smart glasses, collect video images in real time through the camera module of the smart glasses, and transmit the collected video images to the image preprocessing module;
  • the camera module is a binocular camera, and the binocular cameras are respectively disposed on the left and right lenses of the smart glasses, and the distance between the binocular cameras can be adjusted according to the distance of the wearer; the front end of the binocular camera is also installed.
  • Step 200 Receiving, by the image preprocessing module, the video image transmitted by the camera module, and preprocessing the video image by using a chip algorithm, and transmitting the LD video image and the HD video image to the data processing device and the image fusion module respectively;
  • the image preprocessing module uses the FPGA as the main computing device, and processes the video image captured by the camera module through a chip algorithm, and can configure the parameters of the camera module through a correlation algorithm.
  • the processing method for preprocessing the video image by the image preprocessing module is specifically: processing and processing the video image transmitted by the camera module, and obtaining configuration parameters of the camera module through the processing result; and distorting the camera module by using the calibration configuration parameter Eliminate; the configuration parameters obtained by calibration pull the screen of the binocular camera back to a plane, and align the binocular camera peers; use the local matching algorithm to calculate the optimal binocular matching result with the row, thereby calculating the depth by parallax Video image; the unscaled high-definition video image is spliced and directly output to the image fusion module, and the scaled low-definition video image, the depth video image and other auxiliary images are spliced and output to the data processing device;
  • the low-definition video image comes from the downsampling of
  • Step 300 Receive an LD video image transmitted by the image preprocessing module through the data processing device, superimpose the digital content and the posture information into the LD video image by using a correlation algorithm, generate a virtual digital image with posture information, and transmit the virtual digital image.
  • the image fusion module To the image fusion module;
  • the smart glasses are further provided with an attitude sensing module, which acquires the posture information of the wearer through the attitude sensing module, and transmits the posture information to the data processing device.
  • the data processing device uses a CPU or a GPU as an arithmetic device, and implements various applications of augmented reality through software algorithms.
  • the specific manner of the data processing device for generating a virtual digital image includes: splitting the stitched video image; analyzing the observed scene by an image recognition algorithm or a position estimation algorithm during the movement of the wearer; Constructing a map to realize autonomous positioning; storing map information of the location environment, and constructing an incremental map on the existing map; analysing the wearer's perspective posture through the fusion of the visual algorithm and the attitude sensing module; Corresponding digital model; superimposing the angle of view and the rendered digital model on the position corresponding to the environment of the video image to generate a virtual digital image with pose information.
  • Step 400 The HD video image transmitted by the image preprocessing module and the virtual digital image transmitted by the data processing device are respectively received by the image fusion module, and the virtual digital image is merged with the HD video image and output to the display module.
  • the image fusion module merges the virtual digital image with the HD video image by specifically expanding the virtual digital image of the low resolution and low frame rate by using the difference calculation to make the virtual digital image and the high definition video image.
  • the resolution and the frame rate are consistent; the two-way video of the virtual digital image and the high-definition video image are aligned by the synchronization signal of the high-definition video image, and the high-definition video image portion to be covered by the virtual digital image is analyzed; according to the analysis result
  • the virtual digital image is superimposed into the corresponding frame of the high-definition video image to fuse the virtual digital image and the high-definition video image.
  • Step 500 Display a video image through the display module, and focus the display image through the optical component.
  • the video perspective-based smart glasses system and the perspective method thereof transmit the high-definition video image directly from the image pre-processing module to the image fusion module, and output the low-resolution low-frame rate low-definition video image to the data.
  • the processing device performs augmented reality processing, and the virtual digital image with slight delay is merged into the high-definition video image through the image fusion module, thereby reducing the calculation amount of the augmented reality algorithm of the data processing device, and reducing the delay of the video image to the greatest extent.
  • the image quality is improved, the vertigo of the video perspective is greatly reduced, and the wearer's comfort is improved.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

The present invention relates to a video see-through-based smart eyeglasses system and see-through method thereof. The system comprises a smart eyeglasses and a data processing device; said smart eyeglasses comprises a camera module, an image preprocessing module, an image fusion module, and a display module; said camera module is used for acquiring a video image and transmitting the acquired video image to the image preprocessing module; said image preprocessing module is used for preprocessing the video image, then transmitting a low-definition video image to the data processing device and a high-definition video image to the image fusion module; the data processing device generates a virtual digital image by means of an algorithm; the image fusion module is used for fusing the virtual digital image with the high-definition video image, then outputting to the display module to be displayed. The present invention reduces video picture delay to the greatest possible extent while also improving imaging quality, significantly reducing the feeling of vertigo from video see-through and improving the comfort of the wearer.

Description

一种基于视频透视的智能眼镜系统及其透视方法Intelligent glasses system based on video perspective and perspective method thereof 技术领域Technical field
本发明涉及智能眼镜技术领域，尤其涉及一种基于视频透视的智能眼镜系统及其透视方法。The invention relates to the technical field of smart glasses, in particular to a smart glasses system based on video perspective and a seeing method thereof.
背景技术Background technique
增强现实(Augmented Reality,简称AR)是一种实时摄影机影像的位置及角度并加上相应图像的技术，基于AR技术的智能眼镜通过在近眼屏幕上把数字内容叠加在现实世界并进行互动。AR智能眼镜不仅展现了真实世界的信息，而且将虚拟的信息同时显示出来，两种信息相互补充、叠加。AR技术不仅在与虚拟现实(Virtual Reality,简称VR)技术相类似的应用领域具有广泛的应用，诸如飞行器的研制与开发、数据模型的可视化、虚拟训练、娱乐与艺术等领域，而且由于其具有能够对真实环境进行增强显示输出的特性，在医疗研究与解剖训练、精密仪器制造和维修、军用飞机导航、工程设计和远程机器人控制等领域，具有比VR技术更加明显的优势。在视觉化的增强现实中，用户利用头盔显示器，把真实世界与电脑图形多重合成在一起，便可以看到真实的世界围绕着它。随着随身电子产品运算能力的提升，预期增强现实的用途将会越来越广。Augmented Reality (AR) is a technology that combines the position and angle of real-time camera images with corresponding images. Smart glasses based on AR technology superimpose and interact digital content on the near-eye screen in the real world. The AR smart glasses not only display the real world information, but also display the virtual information at the same time, and the two kinds of information complement each other and superimpose. AR technology not only has a wide range of applications in applications similar to Virtual Reality (VR) technology, such as aircraft development and development, data model visualization, virtual training, entertainment and art, but also because of its The ability to enhance the display output of the real environment has more obvious advantages than VR technology in medical research and anatomical training, precision instrument manufacturing and maintenance, military aircraft navigation, engineering design and remote robot control. In visual augmented reality, users can use the helmet display to combine the real world with computer graphics to see the real world around it. As the computing power of portable electronic products increases, the use of augmented reality is expected to become more widespread.
目前AR智能眼镜有两种类型：光学透视型和视频透视型。光学透视型智能眼镜通过分光棱镜实现半透明显示效果，使佩戴者在看到现实世界的同时可以同时看到数字画面。视频透视型眼镜通过视频合成技术，将相机采集到的现实画面和计算机生成的数字画面融合在一起，它能实现比光学透视更大的视角范围。基于视频透视的智能眼镜系统通过放置在眼镜前的双目摄像头捕捉现实画面，眼镜需要在高清屏幕中无延时，无畸变地实时呈现。在进行虚拟画面叠加之前，首先要保证佩戴者通过眼镜感知到景物的位置、大小以及延时等信息与裸眼直接感知的外界景物信息基本一致。基于此，本发明提出一种基于视频透视的智能眼镜系统，从而最大幅度地减小视频画面的延时，提高用户的观看体验度。There are currently two types of AR smart glasses: optical see-through and video see-through. The optical see-through smart glasses realize the translucent display effect through the beam splitting prism, so that the wearer can see the digital picture at the same time while seeing the real world. Video perspective glasses combine video and digital images generated by the camera through video synthesis technology, which enables a larger range of viewing angles than optical perspective. The smart glasses system based on video perspective captures the reality picture through the binocular camera placed in front of the glasses, and the glasses need to be displayed in real time without distortion in the high-definition screen without distortion. Before performing virtual image superimposition, firstly, the information that the wearer perceives the position, size, and delay of the scene through the glasses is basically consistent with the external scene information directly perceived by the naked eye. Based on this, the present invention proposes a smart glasses system based on video perspective, thereby maximally reducing the delay of the video picture and improving the viewing experience of the user.
发明内容Summary of the invention
本发明提供了一种基于视频透视的智能眼镜系统及其透视方法，旨在解决现有AR智能眼镜存在的视频画面延时的技术问题。The invention provides a smart glasses system based on video perspective and a see-through method thereof, aiming at solving the technical problem of video picture delay existing in the existing AR smart glasses.
为了解决以上提出的问题，本发明采用的技术方案为：一种基于视频透视的智能眼镜系统，包括智能眼镜和数据处理装置，所述智能眼镜与数据处理装置连接，所述智能眼镜包括摄像模块、图像预处理模块、图像融合模块和显示 模块；所述摄像模块用于采集视频图像，并将采集到的视频图像传输至图像预处理模块；所述图像预处理模块用于对视频图像进行预处理后，将低清视频图像和高清视频图像分别传输至数据处理装置及图像融合模块；所述数据处理装置用于通过算法生成虚拟数字图像，并将虚拟数字图像传输至图像融合模块；所述图像融合模块用于将虚拟数字图像与高清视频图像进行融合后输出至显示模块进行显示。In order to solve the above problems, the technical solution adopted by the present invention is: a smart glasses system based on video perspective, comprising smart glasses and a data processing device, the smart glasses being connected with a data processing device, the smart glasses including a camera module , image preprocessing module, image fusion module and display The camera module is configured to collect a video image, and transmit the captured video image to an image preprocessing module; the image preprocessing module is configured to preprocess the video image, and then lower the clear video image and the high definition video. The images are respectively transmitted to the data processing device and the image fusion module; the data processing device is configured to generate a virtual digital image by an algorithm, and transmit the virtual digital image to the image fusion module; the image fusion module is configured to use the virtual digital image and the high definition image The video image is fused and output to the display module for display.
本发明实施例采取的技术方案还包括：还包括姿态传感模块，所述姿态传感模块设于智能眼镜上，并与数据处理装置连接，用于采集智能眼镜佩戴者的姿态信息，并将姿态信息传输至数据处理装置。The technical solution adopted by the embodiment of the present invention further includes: an attitude sensing module, wherein the posture sensing module is disposed on the smart glasses, and is connected to the data processing device for collecting posture information of the smart glasses wearer, and The attitude information is transmitted to the data processing device.
本发明实施例采取的技术方案还包括：还包括光学器件，所述光学器件设于显示模块靠近人眼的一侧，用于帮助人眼对显示模块的显示图像进行对焦。The technical solution adopted by the embodiment of the present invention further includes: an optical device disposed on a side of the display module close to the human eye for assisting the human eye to focus the display image of the display module.
本发明实施例采取的技术方案还包括：所述摄像模块为双目摄像头，所述双目摄像头分别设置于智能眼镜的左右镜片上，所述双目摄像头的前端还安装有广角镜头。The technical solution adopted by the embodiment of the present invention further includes: the camera module is a binocular camera, the binocular cameras are respectively disposed on left and right lenses of the smart glasses, and the front end of the binocular camera is further mounted with a wide-angle lens.
本发明实施例采取的技术方案还包括：所述图像预处理模块包括预处理单元、畸变矫正单元、立体校正单元、深度计算单元和图像拼接单元；The technical solution adopted by the embodiment of the present invention further includes: the image preprocessing module includes a preprocessing unit, a distortion correcting unit, a stereo correcting unit, a depth calculating unit, and an image splicing unit;
所述预处理单元用于对摄像模块传输的视频图像进行处理加工，并通过加工结果得到摄像模块的配置参数；The pre-processing unit is configured to process and process a video image transmitted by the camera module, and obtain configuration parameters of the camera module by processing the result;
所述畸变矫正单元用于通过配置参数对摄像模块畸变进行修复；The distortion correcting unit is configured to repair the distortion of the camera module by using a configuration parameter;
所述立体校正单元用于通过配置参数将双目摄像头的画面拉回一个平面，并使双目摄像头同行对齐；The stereo correcting unit is configured to pull the screen of the binocular camera back to a plane by the configuration parameter, and align the binocular cameras with each other;
所述深度计算单元用于使用局部匹配算法计算最优双目匹配结果，并通过视差计算出深度视频图像；The depth calculation unit is configured to calculate an optimal binocular matching result using a local matching algorithm, and calculate a depth video image by using the parallax;
所述图像拼接单元用于将未经缩放的高清视频图像进行拼接后输出到图像融合模块，并将经过缩放的低清视频图像、深度视频图像和辅助图像进行拼接后输出到数据处理装置。The image splicing unit is configured to splicing the unscaled high-definition video images to the image fusion module, and splicing the scaled low-definition video images, the depth video images, and the auxiliary images to the data processing device.
本发明实施例采取的技术方案还包括：所述数据处理装置包括图像拆分单元、场景分析单元、地图构建单元、地图更新单元、姿态分析单元、模型渲染单元和图像叠加单元；The technical solution adopted by the embodiment of the present invention further includes: the data processing device includes an image splitting unit, a scene analyzing unit, a map building unit, a map updating unit, a posture analyzing unit, a model rendering unit, and an image superimposing unit;
所述图像拆分单元用于将拼接好的视频图像进行拆分；The image splitting unit is configured to split the stitched video image;
所述场景分析单元用于对观测场景进行分析；The scene analysis unit is configured to analyze an observation scenario;
所述地图构建单元用于构造地图，实现自主定位； The map construction unit is configured to construct a map to achieve autonomous positioning;
所述地图更新单元用于储存位置环境的地图信息，并在构造的地图上建造增量式地图；The map update unit is configured to store map information of the location environment, and construct an incremental map on the constructed map;
所述姿态分析单元用于通过视觉算法与姿态传感模块融合，分析佩戴者的视角姿态；The posture analysis unit is configured to analyze the wearer's perspective posture by integrating the visual algorithm with the attitude sensing module;
所述模型渲染单元用于通过视角姿态渲染数字模型；The model rendering unit is configured to render a digital model through a perspective gesture;
所述图像叠加单元用于将视角姿态及渲染的数字模型叠加到视频图像上，生成带位姿信息的虚拟数字图像，并将虚拟数字图像输出至图像融合模块。The image superimposing unit is configured to superimpose the angle of view gesture and the rendered digital model on the video image, generate a virtual digital image with pose information, and output the virtual digital image to the image fusion module.
本发明实施例采取的技术方案还包括：所述图像融合模块还包括扩展单元、同步单元和融合单元；The technical solution adopted by the embodiment of the present invention further includes: the image fusion module further includes an expansion unit, a synchronization unit, and a fusion unit;
所述扩展单元用于通过差值计算将虚拟数字图像进行扩展，使虚拟数字图像与高清视频图像的分辨率和帧率一致；The expansion unit is configured to expand the virtual digital image by the difference calculation, so that the resolution and the frame rate of the virtual digital image and the high-definition video image are consistent;
所述同步单元用于通过高清视频图像的同步信号使虚拟数字图像与高清视频图像的两路视频每帧对齐，并分析出融合时将要被虚拟数字图像覆盖的高清视频图像部分；The synchronization unit is configured to align the virtual digital image with the two-way video of the high-definition video image by the synchronization signal of the high-definition video image, and analyze the high-definition video image portion to be covered by the virtual digital image when merging;
所述融合单元用于根据同步单元的分析结果将虚拟数字图像叠加到高清视频图像的对应帧中，使虚拟数字图像和高清视频图像进行融合，并将融合后的视频图像输出至显示模块进行显示。The merging unit is configured to superimpose the virtual digital image into the corresponding frame of the high-definition video image according to the analysis result of the synchronization unit, fuse the virtual digital image and the high-definition video image, and output the fused video image to the display module for display. .
本发明实施例采取的另一技术方案为：一种基于视频透视的智能眼镜透视方法，包括以下步骤：Another technical solution adopted by the embodiment of the present invention is: a method for seeing a smart glasses based on video perspective, comprising the following steps:
步骤100：通过摄像模块采集视频图像，并将采集到的视频图像传输至图像预处理模块；Step 100: Acquire a video image through the camera module, and transmit the collected video image to an image preprocessing module;
步骤200：通过图像预处理模块对视频图像进行预处理，并将低清视频图像和高清视频图像分别传输至数据处理装置及图像融合模块；Step 200: Preprocessing the video image by the image preprocessing module, and transmitting the low clear video image and the high definition video image to the data processing device and the image fusion module respectively;
步骤300：数据处理装置通过算法生成虚拟数字图像，并将虚拟数字图像传输至图像融合模块；Step 300: The data processing device generates a virtual digital image by an algorithm, and transmits the virtual digital image to the image fusion module.
步骤400：通过图像融合模块将虚拟数字图像与高清视频图像进行融合后输出至显示模块进行显示。Step 400: The virtual digital image is merged with the high-definition video image through the image fusion module, and then output to the display module for display.
本发明实施例采取的技术方案还包括：在步骤200中，所述图像预处理模块对视频图像进行预处理的处理方式具体为：对摄像模块传输的视频图像进行处理加工，并通过加工结果得到摄像模块的配置参数；通过配置参数对摄像模块畸变进行修复，并将双目摄像头的画面拉回一个平面，使双目摄像头同行对齐；使用局部匹配算法计算与最优双目匹配结果，通过视差计算出深度视频图像；将未经缩放的高清视频图像进行拼接后直接输出到图像融合模块中，并将经过缩放的低清视频图像、深度视频图像和辅助图像进行拼接后输出到数据处 理装置中。The technical solution adopted by the embodiment of the present invention further includes: in step 200, the image preprocessing module performs preprocessing on the video image by processing: processing the video image transmitted by the camera module, and obtaining the processing result by using the processing result The configuration parameters of the camera module; the distortion of the camera module is repaired by the configuration parameters, and the screen of the binocular camera is pulled back to a plane to align the binocular cameras; the local matching algorithm is used to calculate the matching result with the optimal binocular, through the parallax The depth video image is calculated; the unscaled high-definition video image is spliced and directly output to the image fusion module, and the scaled low-definition video image, the depth video image, and the auxiliary image are spliced and output to the data. In the device.
本发明实施例采取的技术方案还包括：所述步骤300还包括：通过姿态传感模块获取佩戴者的姿态信息，并将姿态信息传输至数据处理装置；所述数据处理装置生成虚拟数字图像的方式包括：将拼接好的视频图像进行拆分；对观测场景进行分析；构造地图，实现自主定位；储存位置环境的地图信息，在既有地图上建造增量式地图；通过视觉算法与姿态传感模块融合，分析佩戴者的视角姿态；通过视角姿态渲染数字模型；将视角姿态及渲染的数字模型叠加到视频图像上，生成带位姿信息的虚拟数字图像。The technical solution adopted by the embodiment of the present invention further includes: the step 300 further includes: acquiring the posture information of the wearer through the attitude sensing module, and transmitting the posture information to the data processing device; the data processing device generating the virtual digital image The method includes: splitting the spliced video image; analyzing the observed scene; constructing the map to realize autonomous positioning; storing map information of the location environment, constructing an incremental map on the existing map; and transmitting the visual algorithm and the gesture The sensor module is integrated to analyze the wearer's perspective posture; the digital model is rendered by the angle of view gesture; the perspective pose and the rendered digital model are superimposed on the video image to generate a virtual digital image with pose information.
与现有技术相比，本发明的有益效果在于：本发明实施例的基于视频透视的智能眼镜系统及其透视方法通过将高清视频图像直接从图像预处理模块发送到图像融合模块中，并将低分辨率低帧率的低清视频图像输出至数据处理装置进行增强现实处理，有轻微延时的虚拟数字图像再通过图像融合模块融合到高清视频图像中，从而减少数据处理装置增强现实算法的计算量，在最大幅度地减小视频画面延时的同时提高成像质量，使视频透视的眩晕感大大降低，提高佩戴者的舒适感。Compared with the prior art, the present invention has the beneficial effects that the video perspective-based smart glasses system and the perspective method thereof according to the embodiments of the present invention transmit the high-definition video image directly from the image pre-processing module to the image fusion module, and The low-resolution low-frame rate low-definition video image is output to the data processing device for augmented reality processing, and the virtual digital image with slight delay is merged into the high-definition video image through the image fusion module, thereby reducing the data processing device augmented reality algorithm. The amount of calculation increases the image quality while minimizing the delay of the video picture, greatly reducing the vertigo of the video perspective and improving the wearer's comfort.
附图说明DRAWINGS
图1为本发明实施例的基于视频透视的智能眼镜系统的结构示意图；1 is a schematic structural diagram of a smart glasses system based on video perspective according to an embodiment of the present invention;
图2是本发明实施例的基于视频透视的智能眼镜系统的电路图；2 is a circuit diagram of a smart glasses system based on video perspective according to an embodiment of the present invention;
图3是本发明实施例的图像预处理模块图像处理示意图；3 is a schematic diagram of image processing of an image preprocessing module according to an embodiment of the present invention;
图4是本发明实施例的图像拼接单元拼接图像示意图；4 is a schematic diagram of a stitching image of an image stitching unit according to an embodiment of the present invention;
图5是本发明实施例的数据处理装置图像处理示意图；FIG. 5 is a schematic diagram of image processing of a data processing apparatus according to an embodiment of the present invention; FIG.
图6是本发明实施例的基于视频透视的智能眼镜透视方法的流程图。6 is a flow chart of a method for seeing a smart glasses based on video perspective according to an embodiment of the present invention.
具体实施方式detailed description
为了便于理解本发明，下面将参照相关附图对本发明进行更全面的描述。附图中给出了本发明的较佳实施例。但是，本发明可以以许多不同的形式来实现，并不限于本文所描述的实施例。相反地，提供这些实施例的目的是使对本发明的公开内容的理解更加透彻全面。In order to facilitate the understanding of the present invention, the present invention will be described more fully hereinafter with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention may be embodied in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the understanding of the present disclosure will be more fully understood.
除非另有定义，本文所使用的所有的技术和科学术语与属于本发明的技术领域的技术人员通常理解的含义相同。本文中在本发明的说明书中所使用的术语只是为了描述具体的实施例的目的，不是旨在于限制本发明。All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. The terminology used in the description of the present invention is for the purpose of describing particular embodiments and is not intended to limit the invention.
请参阅图1和图2，图1是本发明实施例的基于视频透视的智能眼镜系统的结构示意图，图2是本发明实施例的基于视频透视的智能眼镜系统的电路图。本发明实施例的基于视频透视的智能眼镜系统包括智能眼镜和数据处理装置， 智能眼镜包括摄像模块、图像预处理模块、姿态传感模块、图像融合模块、显示模块和光学组件；其中，摄像模块与图像预处理模块连接，用于实时采集视频图像，并将采集到的视频图像传输至图像预处理模块；图像预处理模块分别与图像融合模块及数据处理装置连接，用于接收摄像模块传输的视频图像，并通过芯片化算法对视频图像进行预处理后，将LD(低清)视频图像和HD(高清)视频图像分别传输至数据处理装置及图像融合模块；姿态传感模块设于智能眼镜上，并与数据处理装置连接，用于采集智能眼镜佩戴者的姿态信息，并将姿态信息传输至数据处理装置；数据处理装置与图像融合模块连接，用于接收图像预处理模块传输的LD视频图像和姿态传感模块传输的姿态信息，通过相关算法将数字内容和姿态信息叠加至LD视频图像中，生成带位姿信息的虚拟数字图像，并将虚拟数字图像传输至图像融合模块；其中，位姿信息明确了虚拟数字图像和高清视频图像的坐标关系。图像融合模块与显示模块连接，用于分别接收图像预处理模块传输的HD视频图像(基本无延时)以及数据处理装置传输的虚拟数字图像(有少许延时)，并将虚拟数字图像与HD视频图像进行融合后输出至显示模块进行显示；光学器件设于显示模块靠近人眼的一侧，用于帮助人眼对显示模块的显示图像进行对焦。1 and FIG. 2, FIG. 1 is a schematic structural diagram of a smart glasses system based on video perspective according to an embodiment of the present invention, and FIG. 2 is a circuit diagram of a smart glasses system based on video perspective according to an embodiment of the present invention. The video perspective-based smart glasses system of the embodiment of the present invention includes smart glasses and a data processing device. The smart glasses include a camera module, an image preprocessing module, an attitude sensing module, an image fusion module, a display module and an optical component; wherein the camera module is connected with the image preprocessing module for acquiring video images in real time and collecting the captured video. The image is transmitted to the image preprocessing module; the image preprocessing module is respectively connected with the image fusion module and the data processing device, and is configured to receive the video image transmitted by the camera module, and preprocess the video image by the chip algorithm, and then LD (low) The video image and the HD (high definition) video image are respectively transmitted to the data processing device and the image fusion module; the attitude sensing module is disposed on the smart glasses and is connected with the data processing device for collecting the posture information of the smart glasses wearer. And transmitting the attitude information to the data processing device; the data processing device is connected to the image fusion module, and is configured to receive the LD video image transmitted by the image preprocessing module and the posture information transmitted by the attitude sensing module, and the digital content and the posture information are obtained through a correlation algorithm. Superimposed on the LD video image to generate virtual with pose information Digital images, digital and virtual image transfer to the image integration module; which pose clear coordinates information between the virtual digital images and high-definition video images. The image fusion module is connected to the display module for respectively receiving the HD video image (substantially no delay) transmitted by the image preprocessing module and the virtual digital image transmitted by the data processing device (with a little delay), and the virtual digital image and the HD The video image is fused and output to the display module for display; the optical device is disposed on the side of the display module close to the human eye, and is used to help the human eye focus on the display image of the display module.
在本发明实施例中，摄像模块为双目摄像头，双目摄像头分别设置于智能眼镜的左右镜片上，且双目摄像头之间的距离可根据佩戴者的瞳距进行调节；双目摄像头的前端还安装有广角镜头(图未示)，用于采集较大场景的外界景物；显示模块为智能眼镜显示屏，光学组件由消色差透镜组构成，目的是帮助人眼对近眼屏幕内容进行对焦，在极大提升成像质量的同时，不需要在软件上进行颜色修正。In the embodiment of the present invention, the camera module is a binocular camera, and the binocular cameras are respectively disposed on the left and right lenses of the smart glasses, and the distance between the binocular cameras can be adjusted according to the distance of the wearer; the front end of the binocular camera A wide-angle lens (not shown) is also installed for collecting external scenes of a large scene; the display module is a smart glasses display, and the optical components are composed of achromatic lens groups, the purpose of which is to help the human eye focus on the near-eye screen content, Significantly improve image quality without the need for color correction on the software.
在本发明实施例中，图像预处理模块以FPGA((Field－Programmable Gate Array，现场可编程门阵列，)为主要运算器件，通过芯片化算法处理摄像模块采集的视频图像，并可通过相关算法对摄像模块的参数进行配置；具体请一并参阅图3，是本发明实施例的图像预处理模块图像处理示意图。图像预处理模块还包括预处理单元、畸变矫正单元、立体校正单元、深度计算单元和图像拼接单元,预处理单元、畸变矫正单元、立体校正单元、深度计算单元和图像拼接单元依次连接；In the embodiment of the present invention, the image preprocessing module uses an FPGA (Field-Programmable Gate Array) as the main computing device, and processes the video image captured by the camera module through a chip-based algorithm, and can pass related algorithms. The parameters of the camera module are configured. For details, please refer to FIG. 3 , which is a schematic diagram of image processing of the image preprocessing module according to the embodiment of the present invention. The image preprocessing module further includes a preprocessing unit, a distortion correction unit, a stereo correction unit, and a depth calculation. The unit and the image splicing unit, the preprocessing unit, the distortion correcting unit, the stereo correcting unit, the depth calculating unit and the image splicing unit are sequentially connected;
预处理单元用于对摄像模块传输的视频图像进行处理加工，并通过加工结果得到摄像模块的配置参数；所述摄像模块的配置参数具体包括：色彩过滤排列、白平衡调节、自动曝光、自动对焦、高动态范围图像生成和映射。The preprocessing unit is configured to process and process the video image transmitted by the camera module, and obtain configuration parameters of the camera module by processing the result; the configuration parameters of the camera module specifically include: color filter arrangement, white balance adjustment, automatic exposure, and auto focus High dynamic range image generation and mapping.
畸变矫正单元用于通过标定得到的配置参数对摄像模块畸变进行消除；The distortion correcting unit is configured to eliminate the distortion of the camera module by using the configuration parameters obtained by the calibration;
立体校正单元用于通过标定得到的配置参数将双目摄像头的画面拉回一个 平面，并使双目摄像头同行对齐；The stereo correction unit is configured to pull the screen of the binocular camera back to one by using the configuration parameters obtained by the calibration Plane and align the binocular cameras;
深度计算单元用于使用局部匹配算法计算与所在行的最优双目匹配结果，从而通过视差计算出深度视频图像；深度视频图像也可以通过添加深度相机直接获得，所得深度视频图像需通过校正使其与原始视频图像对齐。The depth calculation unit is configured to calculate the optimal binocular matching result with the line using the local matching algorithm, thereby calculating the depth video image by the parallax; the depth video image can also be directly obtained by adding a depth camera, and the obtained depth video image needs to be corrected. It aligns with the original video image.
图像拼接单元用于将未经缩放的高清视频图像进行拼接后直接输出到图像融合模块中，并将经过缩放的低清视频图像、深度视频图像和其他辅助图像进行拼接后输出到数据处理装置中；其中，低清视频图像来自于高清视频图像的下采样。具体如图4所示，是本发明实施例的图像拼接单元拼接图像示意图。The image splicing unit is configured to splicing the unscaled high-definition video image and output it directly to the image fusion module, and splicing the scaled low-definition video image, the depth video image and other auxiliary images into the data processing device. Among them, the low-definition video image comes from the downsampling of the high-definition video image. Specifically, as shown in FIG. 4, it is a schematic diagram of a stitching image of an image splicing unit according to an embodiment of the present invention.
在本发明实施例中，数据处理装置以CPU(Central Processing Unit，中央处理器)或GPU(Graphic Processing Unit，图形处理器)等作为运算器件，通过软件算法实现增强现实的各类应用。具体请一并参阅图5，是本发明实施例的数据处理装置图像处理示意图。数据处理装置还包括图像拆分单元、场景分析单元、地图构建单元、地图更新单元、姿态分析单元、模型渲染单元和图像叠加单元，图像拆分单元、场景分析单元、地图构建单元、地图更新单元、姿态分析单元、模型渲染单元和图像叠加单元依次连接；In the embodiment of the present invention, the data processing device uses a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit) as an arithmetic device, and implements various applications of augmented reality through a software algorithm. For details, please refer to FIG. 5, which is a schematic diagram of image processing of a data processing apparatus according to an embodiment of the present invention. The data processing device further includes an image splitting unit, a scene analyzing unit, a map building unit, a map updating unit, a posture analyzing unit, a model rendering unit and an image superimposing unit, an image splitting unit, a scene analyzing unit, a map building unit, and a map updating unit. , the posture analysis unit, the model rendering unit, and the image superimposing unit are sequentially connected;
图像拆分单元用于将拼接好的视频图像进行拆分；The image splitting unit is configured to split the stitched video image;
场景分析单元用于在佩戴者移动过程中，通过图像识别算法或位置估计算法对观测场景进行分析；The scene analysis unit is configured to analyze the observation scene by an image recognition algorithm or a position estimation algorithm during the movement of the wearer;
地图构建单元用于在自身定位的基础上构造地图，实现自主定位；The map building unit is configured to construct a map based on its own positioning to achieve autonomous positioning;
地图更新单元用于储存位置环境的地图信息，并在既有地图上建造增量式地图；The map update unit is configured to store map information of the location environment and construct an incremental map on the existing map;
姿态分析单元用于通过视觉算法与姿态传感模块融合，分析佩戴者的视角姿态；The attitude analysis unit is configured to analyze the wearer's perspective posture by integrating the visual algorithm with the attitude sensing module;
模型渲染单元用于通过所获视角姿态渲染相应的数字模型；The model rendering unit is configured to render a corresponding digital model by using the obtained perspective gesture;
图像叠加单元用于将视角姿态及渲染的数字模型叠加到视频图像与环境相对应的位置上，生成带位姿信息的虚拟数字图像，并将虚拟数字图像输出至图像融合模块。The image superimposing unit is configured to superimpose the angle of view gesture and the rendered digital model on the position corresponding to the environment of the video image, generate a virtual digital image with pose information, and output the virtual digital image to the image fusion module.
图像融合模块还包括扩展单元、同步单元和融合单元，扩展单元、同步单元和融合单元依次连接；The image fusion module further includes an expansion unit, a synchronization unit, and a fusion unit, and the extension unit, the synchronization unit, and the fusion unit are sequentially connected;
扩展单元用于通过差值计算将低分辨率低帧率的虚拟数字图像进行扩展，使虚拟数字图像与高清视频图像的分辨率和帧率一致； The expansion unit is configured to expand the low-resolution low frame rate virtual digital image by the difference calculation, so that the virtual digital image and the high-definition video image have the same resolution and frame rate;
同步单元用于通过高清视频图像的同步信号使虚拟数字图像与高清视频图像的两路视频每帧对齐，并分析出融合时将要被虚拟数字图像覆盖的高清视频图像部分；The synchronization unit is configured to align the virtual digital image with the two-way video of the high-definition video image by the synchronization signal of the high-definition video image, and analyze the high-definition video image portion to be covered by the virtual digital image during the fusion;
融合单元用于根据同步单元的分析结果将虚拟数字图像叠加到高清视频图像的对应帧中，使虚拟数字图像和高清视频图像进行融合，并将融合后的视频图像输出至显示模块进行显示。The merging unit is configured to superimpose the virtual digital image into the corresponding frame of the high-definition video image according to the analysis result of the synchronization unit, fuse the virtual digital image and the high-definition video image, and output the fused video image to the display module for display.
增强现实智能眼镜因为视频画面延时的原因，会产生较强的眩晕感，使佩戴者感到不适。在本发明实施例中，为了保证现实视频画面的最小延时，通过将高清视频图像直接从图像预处理模块发送到图像融合模块中，并将低分辨率低帧率的低清视频图像输出至数据处理装置进行增强现实处理，有轻微延时的虚拟数字图像再通过图像融合模块融合到高清视频图像中，从而减少数据处理装置增强现实算法的计算量。虽然因为高清视频图像比虚拟数字图像的提早更新会造成叠加画面轻微浮动，但双路(高清视频图像和虚拟数字图像)分离的方案可以使视频透视的眩晕感大大降低。Augmented reality smart glasses will have a strong sense of vertigo due to the delay of the video screen, making the wearer feel uncomfortable. In the embodiment of the present invention, in order to ensure the minimum delay of the real video picture, the high-definition video image is directly sent from the image pre-processing module to the image fusion module, and the low-resolution low-frame rate low-definition video image is output to The data processing device performs augmented reality processing, and the virtual digital image with slight delay is merged into the high-definition video image through the image fusion module, thereby reducing the calculation amount of the augmented reality algorithm of the data processing device. Although the overlay image is slightly floating because the high-definition video image is updated earlier than the virtual digital image, the two-way (high-definition video image and virtual digital image) separation scheme can greatly reduce the vertigo of the video perspective.
请参阅图6，是本发明实施例的基于视频透视的智能眼镜透视方法的流程图。本发明实施例的基于视频透视的智能眼镜透视方法包括以下步骤：Please refer to FIG. 6 , which is a flowchart of a method for seeing a smart glasses based on video perspective according to an embodiment of the present invention. The video perspective-based smart glasses perspective method of the embodiment of the invention comprises the following steps:
步骤100：佩戴智能眼镜，通过智能眼镜的摄像模块实时采集视频图像，并将采集到的视频图像传输至图像预处理模块；Step 100: Wear smart glasses, collect video images in real time through the camera module of the smart glasses, and transmit the collected video images to the image preprocessing module;
在步骤100中，摄像模块为双目摄像头，双目摄像头分别设置于智能眼镜的左右镜片上，且双目摄像头之间的距离可根据佩戴者的瞳距进行调节；双目摄像头的前端还安装有广角镜头，用于采集较大场景的外界景物。In step 100, the camera module is a binocular camera, and the binocular cameras are respectively disposed on the left and right lenses of the smart glasses, and the distance between the binocular cameras can be adjusted according to the distance of the wearer; the front end of the binocular camera is also installed. There is a wide-angle lens for capturing outside scenes of larger scenes.
步骤200：通过图像预处理模块接收摄像模块传输的视频图像，并通过芯片化算法对视频图像进行预处理后，将LD视频图像和HD视频图像分别传输至数据处理装置及图像融合模块；Step 200: Receiving, by the image preprocessing module, the video image transmitted by the camera module, and preprocessing the video image by using a chip algorithm, and transmitting the LD video image and the HD video image to the data processing device and the image fusion module respectively;
在步骤200中，图像预处理模块以FPGA为主要运算器件，通过芯片化算法处理摄像模块采集的视频图像，并可通过相关算法对摄像模块的参数进行配置。所述图像预处理模块对视频图像进行预处理的处理方式具体为：对摄像模块传输的视频图像进行处理加工，并通过加工结果得到摄像模块的配置参数；通过标定得到的配置参数对摄像模块畸变进行消除；通过标定得到的配置参数将双目摄像头的画面拉回一个平面，并使双目摄像头同行对齐；使用局部匹配算法计算与所在行的最优双目匹配结果，从而通过视差计算出深度视频图像；将未经缩放的高清视频图像进行拼接后直接输出到图像融合模块中，并将经过缩放的低清视频图像、深度视频图像和其他辅助图像进行拼接后输出到数据处理装置中；其中，低清视频图像来自于高清视频图像的下采样。所述摄像模块的配 置参数具体包括：色彩过滤排列、白平衡调节、自动曝光、自动对焦、高动态范围图像生成和映射。In step 200, the image preprocessing module uses the FPGA as the main computing device, and processes the video image captured by the camera module through a chip algorithm, and can configure the parameters of the camera module through a correlation algorithm. The processing method for preprocessing the video image by the image preprocessing module is specifically: processing and processing the video image transmitted by the camera module, and obtaining configuration parameters of the camera module through the processing result; and distorting the camera module by using the calibration configuration parameter Eliminate; the configuration parameters obtained by calibration pull the screen of the binocular camera back to a plane, and align the binocular camera peers; use the local matching algorithm to calculate the optimal binocular matching result with the row, thereby calculating the depth by parallax Video image; the unscaled high-definition video image is spliced and directly output to the image fusion module, and the scaled low-definition video image, the depth video image and other auxiliary images are spliced and output to the data processing device; The low-definition video image comes from the downsampling of the HD video image. The camera module The parameters include: color filter arrangement, white balance adjustment, auto exposure, auto focus, high dynamic range image generation and mapping.
步骤300：通过数据处理装置接收图像预处理模块传输的LD视频图像，通过相关算法将数字内容和姿态信息叠加至LD视频图像中，生成带位姿信息的虚拟数字图像，并将虚拟数字图像传输至图像融合模块；Step 300: Receive an LD video image transmitted by the image preprocessing module through the data processing device, superimpose the digital content and the posture information into the LD video image by using a correlation algorithm, generate a virtual digital image with posture information, and transmit the virtual digital image. To the image fusion module;
在步骤300中，智能眼镜上还设有姿态传感模块，通过姿态传感模块获取佩戴者的姿态信息，并将姿态信息传输至数据处理装置。数据处理装置以CPU或GPU等作为运算器件，通过软件算法实现增强现实的各类应用。所述数据处理装置生成虚拟数字图像的具体方式包括：将拼接好的视频图像进行拆分；在佩戴者移动过程中，通过图像识别算法或位置估计算法对观测场景进行分析；在自身定位的基础上构造地图，实现自主定位；储存位置环境的地图信息，并在既有地图上建造增量式地图；通过视觉算法与姿态传感模块融合，分析佩戴者的视角姿态；通过所获视角姿态渲染相应的数字模型；将视角姿态及渲染的数字模型叠加到视频图像与环境相对应的位置上，生成带位姿信息的虚拟数字图像。In step 300, the smart glasses are further provided with an attitude sensing module, which acquires the posture information of the wearer through the attitude sensing module, and transmits the posture information to the data processing device. The data processing device uses a CPU or a GPU as an arithmetic device, and implements various applications of augmented reality through software algorithms. The specific manner of the data processing device for generating a virtual digital image includes: splitting the stitched video image; analyzing the observed scene by an image recognition algorithm or a position estimation algorithm during the movement of the wearer; Constructing a map to realize autonomous positioning; storing map information of the location environment, and constructing an incremental map on the existing map; analysing the wearer's perspective posture through the fusion of the visual algorithm and the attitude sensing module; Corresponding digital model; superimposing the angle of view and the rendered digital model on the position corresponding to the environment of the video image to generate a virtual digital image with pose information.
步骤400：通过图像融合模块分别接收图像预处理模块传输的HD视频图像以及数据处理装置传输的虚拟数字图像，并将虚拟数字图像与HD视频图像进行融合后输出至显示模块；Step 400: The HD video image transmitted by the image preprocessing module and the virtual digital image transmitted by the data processing device are respectively received by the image fusion module, and the virtual digital image is merged with the HD video image and output to the display module.
在步骤400中，图像融合模块将虚拟数字图像与HD视频图像进行融合的方式具体为：通过差值计算将低分辨率低帧率的虚拟数字图像进行扩展，使虚拟数字图像与高清视频图像的分辨率和帧率一致；通过高清视频图像的同步信号使虚拟数字图像与高清视频图像的两路视频每帧对齐，并分析出融合时将要被虚拟数字图像覆盖的高清视频图像部分；根据分析结果将虚拟数字图像叠加到高清视频图像的对应帧中，使虚拟数字图像和高清视频图像进行融合。In step 400, the image fusion module merges the virtual digital image with the HD video image by specifically expanding the virtual digital image of the low resolution and low frame rate by using the difference calculation to make the virtual digital image and the high definition video image. The resolution and the frame rate are consistent; the two-way video of the virtual digital image and the high-definition video image are aligned by the synchronization signal of the high-definition video image, and the high-definition video image portion to be covered by the virtual digital image is analyzed; according to the analysis result The virtual digital image is superimposed into the corresponding frame of the high-definition video image to fuse the virtual digital image and the high-definition video image.
步骤500：通过显示模块显示视频图像，并通过光学组件对显示图像进行对焦。Step 500: Display a video image through the display module, and focus the display image through the optical component.
本发明实施例的基于视频透视的智能眼镜系统及其透视方法通过将高清视频图像直接从图像预处理模块发送到图像融合模块中，并将低分辨率低帧率的低清视频图像输出至数据处理装置进行增强现实处理，有轻微延时的虚拟数字图像再通过图像融合模块融合到高清视频图像中，从而减少数据处理装置增强现实算法的计算量，在最大幅度地减小视频画面延时的同时提高成像质量，使视频透视的眩晕感大大降低，提高佩戴者的舒适感。The video perspective-based smart glasses system and the perspective method thereof according to the embodiment of the present invention transmit the high-definition video image directly from the image pre-processing module to the image fusion module, and output the low-resolution low-frame rate low-definition video image to the data. The processing device performs augmented reality processing, and the virtual digital image with slight delay is merged into the high-definition video image through the image fusion module, thereby reducing the calculation amount of the augmented reality algorithm of the data processing device, and reducing the delay of the video image to the greatest extent. At the same time, the image quality is improved, the vertigo of the video perspective is greatly reduced, and the wearer's comfort is improved.
上述实施例为本发明较佳的实施方式，但本发明的实施方式并不受上述实施例的限制，其他的任何未背离本发明的精神实质与原理下所作的改变、修饰、替代、组合、简化，均应为等效的置换方式，都包含在本发明的保护范围之内。 The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and combinations thereof may be made without departing from the spirit and scope of the invention. Simplifications should all be equivalent replacements and are included in the scope of the present invention.

Claims (10)

  1. 一种基于视频透视的智能眼镜系统，包括智能眼镜和数据处理装置，其特征在于，所述智能眼镜包括摄像模块、图像预处理模块、图像融合模块和显示模块；所述摄像模块用于采集视频图像，并将采集到的视频图像传输至图像预处理模块；所述图像预处理模块用于对视频图像进行预处理后，将低清视频图像和高清视频图像分别传输至数据处理装置及图像融合模块；所述数据处理装置用于通过算法生成虚拟数字图像，并将虚拟数字图像传输至图像融合模块；所述图像融合模块用于将虚拟数字图像与高清视频图像进行融合后输出至显示模块进行显示。A smart glasses system based on video perspective, comprising smart glasses and a data processing device, wherein the smart glasses comprise a camera module, an image preprocessing module, an image fusion module and a display module; and the camera module is configured to collect video And transmitting the captured video image to an image preprocessing module; the image preprocessing module is configured to preprocess the video image, and respectively transmit the low clear video image and the high definition video image to the data processing device and the image fusion a module; the data processing device is configured to generate a virtual digital image by an algorithm, and transmit the virtual digital image to an image fusion module; the image fusion module is configured to fuse the virtual digital image with the high-definition video image, and output the image to the display module. display.
  2. 根据权利要求1所述的基于视频透视的智能眼镜系统，其特征在于：还包括姿态传感模块，所述姿态传感模块设于智能眼镜上，并与数据处理装置连接，用于采集智能眼镜佩戴者的姿态信息，并将姿态信息传输至数据处理装置。The smart glasses system based on video perspective according to claim 1, further comprising an attitude sensing module, wherein the posture sensing module is disposed on the smart glasses and connected to the data processing device for collecting smart glasses. The wearer's posture information is transmitted to the data processing device.
  3. 根据权利要求2所述的基于视频透视的智能眼镜系统，其特征在于：还包括光学器件，所述光学器件设于显示模块靠近人眼的一侧，用于帮助人眼对显示模块的显示图像进行对焦。The smart glasses system based on the video perspective according to claim 2, further comprising an optical device disposed on a side of the display module close to the human eye for assisting the human eye to display the display image of the display module. Focus.
  4. 根据权利要求1所述的基于视频透视的智能眼镜系统，其特征在于：所述摄像模块为双目摄像头，所述双目摄像头分别设置于智能眼镜的左右镜片上，所述双目摄像头的前端还安装有广角镜头。The smart glasses system based on video perspective according to claim 1, wherein the camera module is a binocular camera, and the binocular cameras are respectively disposed on left and right lenses of the smart glasses, and the front end of the binocular camera A wide-angle lens is also installed.
  5. 根据权利要求4所述的基于视频透视的智能眼镜系统，其特征在于：所述图像预处理模块包括预处理单元、畸变矫正单元、立体校正单元、深度计算单元和图像拼接单元；The smart glasses system based on video perspective according to claim 4, wherein the image preprocessing module comprises a preprocessing unit, a distortion correcting unit, a stereo correcting unit, a depth calculating unit and an image splicing unit;
    所述预处理单元用于对摄像模块传输的视频图像进行处理加工，并通过加工结果得到摄像模块的配置参数；The pre-processing unit is configured to process and process a video image transmitted by the camera module, and obtain configuration parameters of the camera module by processing the result;
    所述畸变矫正单元用于通过配置参数对摄像模块畸变进行修复；The distortion correcting unit is configured to repair the distortion of the camera module by using a configuration parameter;
    所述立体校正单元用于通过配置参数将双目摄像头的画面拉回一个平面，并使双目摄像头同行对齐；The stereo correcting unit is configured to pull the screen of the binocular camera back to a plane by the configuration parameter, and align the binocular cameras with each other;
    所述深度计算单元用于使用局部匹配算法计算最优双目匹配结果，并通过视差计算出深度视频图像；The depth calculation unit is configured to calculate an optimal binocular matching result using a local matching algorithm, and calculate a depth video image by using the parallax;
    所述图像拼接单元用于将未经缩放的高清视频图像进行拼接后输出到图像融合模块，并将经过缩放的低清视频图像、深度视频图像和辅助图像进行拼接后输出到数据处理装置。The image splicing unit is configured to splicing the unscaled high-definition video images to the image fusion module, and splicing the scaled low-definition video images, the depth video images, and the auxiliary images to the data processing device.
  6. 根据权利要求2或5所述的基于视频透视的智能眼镜系统，其特征在于：所述数据处理装置包括图像拆分单元、场景分析单元、地图构建单元、地图更新单元、姿态分析单元、模型渲染单元和图像叠加单元； The smart glasses system based on video perspective according to claim 2 or 5, wherein the data processing device comprises an image splitting unit, a scene analyzing unit, a map building unit, a map updating unit, a posture analyzing unit, and a model rendering. Unit and image overlay unit;
    所述图像拆分单元用于将拼接好的视频图像进行拆分；The image splitting unit is configured to split the stitched video image;
    所述场景分析单元用于对观测场景进行分析；The scene analysis unit is configured to analyze an observation scenario;
    所述地图构建单元用于构造地图，实现自主定位；The map construction unit is configured to construct a map to achieve autonomous positioning;
    所述地图更新单元用于储存位置环境的地图信息，并在构造的地图上建造增量式地图；The map update unit is configured to store map information of the location environment, and construct an incremental map on the constructed map;
    所述姿态分析单元用于通过视觉算法与姿态传感模块融合，分析佩戴者的视角姿态；The posture analysis unit is configured to analyze the wearer's perspective posture by integrating the visual algorithm with the attitude sensing module;
    所述模型渲染单元用于通过视角姿态渲染数字模型；The model rendering unit is configured to render a digital model through a perspective gesture;
    所述图像叠加单元用于将视角姿态及渲染的数字模型叠加到视频图像上，生成带位姿信息的虚拟数字图像，并将虚拟数字图像输出至图像融合模块。The image superimposing unit is configured to superimpose the angle of view gesture and the rendered digital model on the video image, generate a virtual digital image with pose information, and output the virtual digital image to the image fusion module.
  7. 根据权利要求6所述的基于视频透视的智能眼镜系统，其特征在于：所述图像融合模块还包括扩展单元、同步单元和融合单元；The smart glasses system based on video perspective according to claim 6, wherein the image fusion module further comprises an expansion unit, a synchronization unit, and a fusion unit;
    所述扩展单元用于通过差值计算将虚拟数字图像进行扩展，使虚拟数字图像与高清视频图像的分辨率和帧率一致；The expansion unit is configured to expand the virtual digital image by the difference calculation, so that the resolution and the frame rate of the virtual digital image and the high-definition video image are consistent;
    所述同步单元用于通过高清视频图像的同步信号使虚拟数字图像与高清视频图像的两路视频每帧对齐，并分析出融合时将要被虚拟数字图像覆盖的高清视频图像部分；The synchronization unit is configured to align the virtual digital image with the two-way video of the high-definition video image by the synchronization signal of the high-definition video image, and analyze the high-definition video image portion to be covered by the virtual digital image when merging;
    所述融合单元用于根据同步单元的分析结果将虚拟数字图像叠加到高清视频图像的对应帧中，使虚拟数字图像和高清视频图像进行融合，并将融合后的视频图像输出至显示模块进行显示。The merging unit is configured to superimpose the virtual digital image into the corresponding frame of the high-definition video image according to the analysis result of the synchronization unit, fuse the virtual digital image and the high-definition video image, and output the fused video image to the display module for display. .
  8. 一种基于视频透视的智能眼镜透视方法，其特征在于：包括以下步骤：A smart glasses perspective method based on video perspective, comprising: the following steps:
    步骤100：通过摄像模块采集视频图像，并将采集到的视频图像传输至图像预处理模块；Step 100: Acquire a video image through the camera module, and transmit the collected video image to an image preprocessing module;
    步骤200：通过图像预处理模块对视频图像进行预处理，并将低清视频图像和高清视频图像分别传输至数据处理装置及图像融合模块；Step 200: Preprocessing the video image by the image preprocessing module, and transmitting the low clear video image and the high definition video image to the data processing device and the image fusion module respectively;
    步骤300：数据处理装置通过算法生成虚拟数字图像，并将虚拟数字图像传输至图像融合模块；Step 300: The data processing device generates a virtual digital image by an algorithm, and transmits the virtual digital image to the image fusion module.
    步骤400：通过图像融合模块将虚拟数字图像与高清视频图像进行融合后输出至显示模块进行显示。Step 400: The virtual digital image is merged with the high-definition video image through the image fusion module, and then output to the display module for display.
  9. 根据权利要求8所述的基于视频透视的智能眼镜透视方法，其特征在于：在步骤200中，所述图像预处理模块对视频图像进行预处理的处理方式具体为：对摄像模块传输的视频图像进行处理加工，并通过加工结果得到摄像模块的配置参数；通过配置参数对摄像模块畸变进行修复，并将双目摄像头的画面拉回 一个平面，使双目摄像头同行对齐；使用局部匹配算法计算与最优双目匹配结果，通过视差计算出深度视频图像；将未经缩放的高清视频图像进行拼接后直接输出到图像融合模块中，并将经过缩放的低清视频图像、深度视频图像和辅助图像进行拼接后输出到数据处理装置中。The method according to claim 8, wherein in the step 200, the image preprocessing module preprocesses the video image by using a video image transmitted by the camera module. Processing and processing, and obtaining the configuration parameters of the camera module through the processing result; repairing the distortion of the camera module through the configuration parameters, and pulling back the screen of the binocular camera A plane is used to align the binocular cameras; the local matching algorithm is used to calculate the optimal binocular matching result, and the depth video image is calculated by the parallax; the unscaled high-definition video image is spliced and directly output to the image fusion module. The scaled low-definition video image, the depth video image, and the auxiliary image are spliced and output to the data processing device.
  10. 根据权利要求8或9所述的基于视频透视的智能眼镜透视方法，其特征在于：所述步骤300还包括：通过姿态传感模块获取佩戴者的姿态信息，并将姿态信息传输至数据处理装置；所述数据处理装置生成虚拟数字图像的方式包括：将拼接好的视频图像进行拆分；对观测场景进行分析；构造地图，实现自主定位；储存位置环境的地图信息，在既有地图上建造增量式地图；通过视觉算法与姿态传感模块融合，分析佩戴者的视角姿态；通过视角姿态渲染数字模型；将视角姿态及渲染的数字模型叠加到视频图像上，生成带位姿信息的虚拟数字图像。 The method according to claim 8 or 9, wherein the step 300 further comprises: acquiring the posture information of the wearer through the posture sensing module, and transmitting the posture information to the data processing device. The manner in which the data processing apparatus generates the virtual digital image includes: splitting the stitched video image; analyzing the observed scene; constructing the map to realize autonomous positioning; and storing map information of the location environment, constructing on the existing map Incremental map; through the fusion of visual algorithm and attitude sensing module, analyzing the wearer's perspective posture; rendering the digital model through the angle of view; superimposing the angle of view and the rendered digital model on the video image to generate virtual with pose information digital image.
PCT/CN2016/086348 2016-04-07 2016-06-20 Video see-through-based smart eyeglasses system and see-through method thereof WO2017173735A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610218731.2 2016-04-07
CN201610218731.2A CN107277495B (en) 2016-04-07 2016-04-07 A kind of intelligent glasses system and its perspective method based on video perspective

Publications (1)

Publication Number Publication Date
WO2017173735A1 true WO2017173735A1 (en) 2017-10-12

Family

ID=60000809

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/086348 WO2017173735A1 (en) 2016-04-07 2016-06-20 Video see-through-based smart eyeglasses system and see-through method thereof

Country Status (2)

Country Link
CN (1) CN107277495B (en)
WO (1) WO2017173735A1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109062407A (en) * 2018-07-27 2018-12-21 江西省杜达菲科技有限责任公司 Remote mobile terminal three-dimensional display & control system and method based on VR technology
CN109714589A (en) * 2019-02-22 2019-05-03 上海北冕信息科技有限公司 Input/output unit, equipment for augmented reality
CN110244837A (en) * 2019-04-26 2019-09-17 北京圣威特科技有限公司 Augmented reality and the experience glasses and its imaging method being superimposed with virtual image
CN111047713A (en) * 2019-12-30 2020-04-21 复旦大学 Augmented reality interaction system based on multi-view visual positioning
WO2020192458A1 (en) * 2019-03-25 2020-10-01 华为技术有限公司 Image processing method and head-mounted display device
CN111770363A (en) * 2020-07-10 2020-10-13 陕西师范大学 Low-delay high-resolution mobile augmented reality system based on context awareness
CN112489138A (en) * 2020-12-02 2021-03-12 中国船舶重工集团公司第七一六研究所 Target situation information intelligent acquisition system based on wearable equipment
CN112907756A (en) * 2021-03-10 2021-06-04 中国石油天然气集团有限公司 Emergency AR back end auxiliary operation system and auxiliary operation method
CN115065816A (en) * 2022-05-09 2022-09-16 北京大学 Real geospatial scene real-time construction method based on panoramic video technology
CN116126150A (en) * 2023-04-13 2023-05-16 北京千种幻影科技有限公司 Simulated driving system and method based on live-action interaction

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108881852A (en) * 2018-07-27 2018-11-23 中船电子科技有限公司 A kind of intelligent glasses control method for customs preventive law enforcement
CN109147059B (en) * 2018-09-06 2020-09-25 联想(北京)有限公司 Method and equipment for determining delay value
CN110414101B (en) * 2019-07-15 2023-08-04 中国商用飞机有限责任公司北京民用飞机技术研究中心 Simulation scene measurement method, accuracy measurement method and system
CN111930233B (en) * 2020-08-05 2023-07-21 聚好看科技股份有限公司 Panoramic video image display method and display device
CN112558302B (en) * 2020-12-08 2022-12-20 恒玄科技(上海)股份有限公司 Intelligent glasses for determining glasses posture and signal processing method thereof
CN114363489B (en) * 2021-12-29 2022-11-15 珠海惠中智能技术有限公司 Augmented reality system with camera and eye display device direct coupling
CN114500977A (en) * 2022-01-22 2022-05-13 深圳市帝泰光电有限公司 Intelligent wearable device with enhanced vision and method for realizing stereoscopic vision transposition
CN115047624B (en) * 2022-05-24 2023-06-27 北京领为军融科技有限公司 Intelligent glasses control system
CN115661418A (en) * 2022-12-22 2023-01-31 灯影科技有限公司 Mixed reality display device, method, equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005135052A (en) * 2003-10-29 2005-05-26 Canon Inc Method for realizing fog effect in mixed reality space
US20070242086A1 (en) * 2006-04-14 2007-10-18 Takuya Tsujimoto Image processing system, image processing apparatus, image sensing apparatus, and control method thereof
CN101311893A (en) * 2007-05-23 2008-11-26 佳能株式会社 Mixed reality presentation apparatus and control method
CN103105174A (en) * 2013-01-29 2013-05-15 四川长虹佳华信息产品有限责任公司 AR (augmented reality)-based vehicle-mounted live-action safe navigation method
CN104883556A (en) * 2015-05-25 2015-09-02 深圳市虚拟现实科技有限公司 Three dimensional display method based on augmented reality and augmented reality glasses

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103530881B (en) * 2013-10-16 2016-03-09 北京理工大学 Be applicable to the Outdoor Augmented Reality no marks point Tracing Registration method of mobile terminal

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005135052A (en) * 2003-10-29 2005-05-26 Canon Inc Method for realizing fog effect in mixed reality space
US20070242086A1 (en) * 2006-04-14 2007-10-18 Takuya Tsujimoto Image processing system, image processing apparatus, image sensing apparatus, and control method thereof
CN101311893A (en) * 2007-05-23 2008-11-26 佳能株式会社 Mixed reality presentation apparatus and control method
CN103105174A (en) * 2013-01-29 2013-05-15 四川长虹佳华信息产品有限责任公司 AR (augmented reality)-based vehicle-mounted live-action safe navigation method
CN104883556A (en) * 2015-05-25 2015-09-02 深圳市虚拟现实科技有限公司 Three dimensional display method based on augmented reality and augmented reality glasses

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109062407A (en) * 2018-07-27 2018-12-21 江西省杜达菲科技有限责任公司 Remote mobile terminal three-dimensional display & control system and method based on VR technology
CN109714589A (en) * 2019-02-22 2019-05-03 上海北冕信息科技有限公司 Input/output unit, equipment for augmented reality
WO2020192458A1 (en) * 2019-03-25 2020-10-01 华为技术有限公司 Image processing method and head-mounted display device
CN110244837A (en) * 2019-04-26 2019-09-17 北京圣威特科技有限公司 Augmented reality and the experience glasses and its imaging method being superimposed with virtual image
CN111047713B (en) * 2019-12-30 2023-05-30 复旦大学 Augmented reality interaction system based on multi-vision positioning and operation method thereof
CN111047713A (en) * 2019-12-30 2020-04-21 复旦大学 Augmented reality interaction system based on multi-view visual positioning
CN111770363A (en) * 2020-07-10 2020-10-13 陕西师范大学 Low-delay high-resolution mobile augmented reality system based on context awareness
CN112489138A (en) * 2020-12-02 2021-03-12 中国船舶重工集团公司第七一六研究所 Target situation information intelligent acquisition system based on wearable equipment
CN112489138B (en) * 2020-12-02 2024-02-20 中国船舶集团有限公司第七一六研究所 Target situation information intelligent acquisition system based on wearable equipment
CN112907756A (en) * 2021-03-10 2021-06-04 中国石油天然气集团有限公司 Emergency AR back end auxiliary operation system and auxiliary operation method
CN112907756B (en) * 2021-03-10 2023-11-03 中国石油天然气集团有限公司 Auxiliary operation system and auxiliary operation method for rescue AR rear end
CN115065816A (en) * 2022-05-09 2022-09-16 北京大学 Real geospatial scene real-time construction method based on panoramic video technology
US11836878B2 (en) 2022-05-09 2023-12-05 Peking University Method and apparatus for constructing real-geographic-space scene in real time
CN115065816B (en) * 2022-05-09 2023-04-07 北京大学 Real geospatial scene real-time construction method and real-time construction device
CN116126150B (en) * 2023-04-13 2023-06-27 北京千种幻影科技有限公司 Simulated driving system and method based on live-action interaction
CN116126150A (en) * 2023-04-13 2023-05-16 北京千种幻影科技有限公司 Simulated driving system and method based on live-action interaction

Also Published As

Publication number Publication date
CN107277495B (en) 2019-06-25
CN107277495A (en) 2017-10-20

Similar Documents

Publication Publication Date Title
WO2017173735A1 (en) Video see-through-based smart eyeglasses system and see-through method thereof
US9699438B2 (en) 3D graphic insertion for live action stereoscopic video
US20160295194A1 (en) Stereoscopic vision system generatng stereoscopic images with a monoscopic endoscope and an external adapter lens and method using the same to generate stereoscopic images
WO2016115872A1 (en) Binocular ar head-mounted display device and information display method thereof
US11200646B2 (en) Compensation for deformation in head mounted display systems
WO2013175923A1 (en) Simulation device
WO2016115873A1 (en) Binocular ar head-mounted display device and information display method therefor
US11962746B2 (en) Wide-angle stereoscopic vision with cameras having different parameters
CN104345454B (en) Head-mounted vision auxiliary system and imaging method thereof
CN108093244B (en) Remote follow-up stereoscopic vision system
CN107037584B (en) Intelligent glasses perspective method and system
JP2000354257A (en) Image processor, image processing method and program provision medium
WO2019041614A1 (en) Head-mounted immersive virtual reality display device and immersive virtual reality display method
CN114401414A (en) Immersive live broadcast information display method and system and information push method
US20230239457A1 (en) System and method for corrected video-see-through for head mounted displays
JP4580678B2 (en) Gaze point display device
TWI589150B (en) Three-dimensional auto-focusing method and the system thereof
WO2016157923A1 (en) Information processing device and information processing method
US20120162775A1 (en) Method for Correcting Hyperstereoscopy and Associated Helmet Viewing System
JP2021039444A (en) Image processing device, control method and program thereof
JP2017098596A (en) Image generating method and image generating apparatus
CN111629194B (en) Method and system for converting panoramic video into 6DOF video based on neural network
JP2000308090A (en) Video displaying and processing device, video photographing and processing device, and camera controller
JPH02291787A (en) Wide visual field display device and wide visual field photography device
JP2006340017A (en) Device and method for stereoscopic video image display

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16897668

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 16897668

Country of ref document: EP

Kind code of ref document: A1