WO2021137348A1 - Procédé pour générer une carte d'espace afin de partager des informations d'espace tridimensionnel entre une pluralité de terminaux, et pour lire un point d'exécution de commande - Google Patents

Procédé pour générer une carte d'espace afin de partager des informations d'espace tridimensionnel entre une pluralité de terminaux, et pour lire un point d'exécution de commande Download PDF

Info

Publication number
WO2021137348A1
WO2021137348A1 PCT/KR2020/000608 KR2020000608W WO2021137348A1 WO 2021137348 A1 WO2021137348 A1 WO 2021137348A1 KR 2020000608 W KR2020000608 W KR 2020000608W WO 2021137348 A1 WO2021137348 A1 WO 2021137348A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
terminals
spatial information
space
server
Prior art date
Application number
PCT/KR2020/000608
Other languages
English (en)
Korean (ko)
Inventor
한상준
Original Assignee
엔센스코리아주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 엔센스코리아주식회사 filed Critical 엔센스코리아주식회사
Publication of WO2021137348A1 publication Critical patent/WO2021137348A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration

Definitions

  • the present invention is based on a real-virtual matching technology capable of synthesizing a virtual object in a real space to increase the presence of a virtual object by a plurality of people in the same space through their respective terminals and to provide the same user experience.
  • augmented reality technology is a technology field derived from virtual reality technology that synthesizes and superimposes virtual objects on real space and shows them. It can increase the presence of a virtual object by creating the illusion that it actually exists in real space.
  • a 3D point cloud map is generated from a depth image obtained using a depth camera, an object in a real space on which augmented content is to be projected with the 3D point cloud map is tracked, and virtual on the real space using a display device such as a projector.
  • a display device such as a projector.
  • the position value of the target object in the virtual space is calculated from the depth image obtained using the depth camera, and the reference position database and the There is a method of generating an event execution signal by comparison.
  • all of the above examples are accompanied by a device capable of acquiring the same depth as a depth camera in order to configure a three-dimensional real space as spatial information or to receive an input for interaction with a user in a three-dimensional real space.
  • a device capable of acquiring the same depth as a depth camera in order to configure a three-dimensional real space as spatial information or to receive an input for interaction with a user in a three-dimensional real space.
  • the present invention implements an augmented reality technology in which a plurality of users use their individual terminals in an arbitrary space, and a method in which a plurality of users interact with a virtual object and share it with each other.
  • augmented reality technology that allows you to feel the same presence in one space
  • by implementing the augmented reality technology using each individual terminal the limitation of the installation space is eliminated, and the 3D point cloud map is shared with each other and a single camera is used in the real space. It compensates for the position error, which is a difficult point in tracking , and when each other executes a command using each individual terminal, the same point in the three-dimensional space can be read as the command execution point.
  • the purpose is to provide a way to share the command execution point.
  • a spatial recognition method based on depth information acquired through a conventional infrared ToF camera, an RGB-D type depth camera, a stereo depth camera, etc. is avoided, and spatial information is constructed using only image information. It implements a SLAM (Simultaneous Localization and Mapping) method and provides a method for sharing it by a plurality of terminals.
  • SLAM Simultaneous Localization and Mapping
  • the present invention is characterized in that a plurality of users use their respective terminals to simultaneously use the augmented reality technology in an arbitrary space
  • the terminals include a camera and a color image acquisition unit for inputting images in a real space
  • the An image processing unit for extracting depth information from a color image and constructing a 3D point cloud map
  • a display and image output unit that can synthesize and overlap real space and virtual objects and output them, and a user input by touching the display
  • It consists of a touch screen capable of processing and a touch input unit capable of processing it
  • a server for a plurality of terminals to share a 3D point cloud map
  • a connection control unit for communicating with the terminal in a socket communication method in the server
  • a space map storage unit It consists of an event transmitter.
  • a 3D point cloud map generated based on the spatial information connecting the 3D point cloud map to a server to share the 3D point cloud map with each other using socket communication; receiving a role assigned to a master node or a slave node from the server connected through the socket communication;
  • the present invention generates a 3D point cloud map by recognizing a real space by using each individual terminal in a single space by a plurality of users at the same time, and creates a virtual map that can be shared by a plurality of users through a server, , a number of users share a virtual map from the server so that each user can experience the same augmented reality, thereby enhancing the sense of presence, as well as a number of users operating virtual objects through the terminal based on the virtual map
  • This is a method that reads the same three-dimensional spatial coordinates and shares them, so that event information related to manipulation of virtual objects in each augmented reality experience can be simultaneously executed.
  • the present invention can be used for tasks such as application examples such as
  • augmented reality technology can be applied in place of mock-ups in construction, architecture, interior design, 3D product design, etc. in order to increase the understanding of multilateral.
  • the present invention is applied to multiple users simultaneously synthesizing and superimposing virtual objects in the same real space, and has the same effect as having a meeting while viewing the real thing using a mockup. It is effective because you can easily experience augmented reality without being limited by time.
  • a method of using a model for urban construction planning or architectural design using a diorama it is possible to substitute the augmented reality technology using a plurality of terminals of the present invention instead of making an existing model. , it reduces the production cost or production time of the diorama, and in the presentation using this real model, modifications or changes of the model cannot be immediately reflected during the presentation, whereas the instruction execution point reading method through the present invention is used to present Through a method in which the user in charge of the demonstration touches the display and executes commands on the virtual object, for example, the existing building before the redevelopment and the new building after the redevelopment that are displayed overlaid on the real space as a virtual object are shown. or, similarly, changing or relocating trees or sculptures in green space can be shown in real time.
  • the application of the present invention is not limited to the above embodiment, and the present invention can be applied to fields other than the above embodiment without changing the essential content of the present invention.
  • 1 is a system structural diagram including a server capable of sharing spatial information with a plurality of terminals;
  • FIG. 2 is a flowchart specifically illustrating a method for generating 3D spatial information in a terminal and interworking with a server
  • FIG. 3 is a flowchart specifically illustrating a method of outputting an augmented reality image by reflecting spatial information in connection with a server in a terminal;
  • FIG. 4 is a flowchart specifically illustrating a method of sharing spatial information with a plurality of terminals in a server and reading a command execution point;
  • FIG. 5 is a flowchart specifically illustrating a method of controlling a virtual object in a server and sharing the virtual object by a plurality of terminals;
  • a method for sharing through and reading a command execution point that converts a user's two-dimensional input information using a touch screen into three-dimensional information in one terminal to experience the same augmented reality even if multiple users use their own terminals is the structural diagram of
  • one or more users each use the terminal 100 and acquire continuous images from the color image acquisition unit 120 using the camera 110 .
  • This is transmitted to the image processing unit 130 to generate and track a spatial map by the method of FIG. 2 .
  • the 3D point cloud map is transmitted to the server 200 through the data transceiver 140 and is received by the access control unit 210 .
  • the access control unit classifies the data operation command sent by the terminal by the method of FIG. 4 and controls the execution of the space map storage unit 220 , the space map search unit 230 , and the command execution point reader 240 according to the method of FIG. 4 .
  • the space map generated by the space map storage unit is stored in the space map DB 250 .
  • the two-dimensional image of the real space obtained by the color image acquisition unit and the 3D modeling of the virtual object stored in the virtual object DB 270 of the server and the location and direction information of the virtual object are shown in FIG. In the same way, they are synthesized and superimposed, and real-virtual images are generated and output to the display 160 .
  • the user can experience augmented reality synthesized by superimposing virtual objects in real space through the above process, and at this time, touch the touch screen 161 to manipulate, create, or delete the virtual objects to execute commands.
  • a touch is performed, and a two-dimensional coordinate input value is derived from the touch input unit 170 .
  • the coordinate input value input by the user through the above process is branched from the access control unit 210 of the server 200 through the data transmission/reception unit 140 and transmitted to the command execution point reader 240 .
  • the command execution point obtained in the same way as in FIG. 4 is transmitted to the event transmitter 260, and information about the virtual object is reflected in the virtual object DB 270, and the event transmitter is all nodes connected to the server through the access controller.
  • the information of the virtual object is transmitted to the terminal of
  • the plurality of terminals receiving the virtual object information through the data transceiver repeats the process of being updated with the new virtual object information in the image output unit and outputting the information to the display as in the above process.
  • Each terminal acquires an image of real space from a camera and extracts feature points using an algorithm such as SIFT, SURF, or ORB. Thereafter, it is determined whether the current state is the initialized state of the spatial map, and if the spatial map is not initialized, the camera posture and origin are estimated using the feature points extracted from the acquired image. At this time, two consecutive images are required to estimate the camera posture and origin. To compare the similarity between the feature points extracted from each image, the Euclidean distance is obtained to obtain a feature point pair matching the two feature points with the shortest distance.
  • an algorithm such as SIFT, SURF, or ORB.
  • the relative position of the t frame may be obtained from the t-1 frame using the geometric relationship of the matched feature point pair.
  • the spatial map is initialized by estimating the initial position of the spatial map by finding the camera position that best matches the spatial map while moving the camera image in 3D space based on the matching pair of feature points between the previous frame and the current frame.
  • a frame in which a large change occurs among image frames is determined as a keyframe for fast and stable position tracking, and a 3D point cloud map with a high computational load
  • the generation is executed only for keyframes in which a large change occurs, and spatial map registration and tracking, which does not have a high computational load, is branched to be executed for all frames.
  • the corresponding frame is determined as a keyframe and added to the keyframe group, and the extracted feature points are matched with the feature points of the spatial map initialized and generated in the above step in the three-dimensional space of each feature point.
  • a 3D point cloud map is generated so that the coordinates are optimal solutions.
  • the 3D spatial information is transmitted to the server, and the receiving server collects all the 3D spatial information received from the plurality of terminals to obtain the 3D spatial information of each feature point. Process the bundle adjustment to sum the 3D point cloud map so that the spatial coordinates are the optimal solution.
  • each terminal receives the bundle-adjusted server spatial map, so that a plurality of terminals can share all of the three-dimensional spatial information in real space.
  • the 3D spatial information generated by each terminal can obtain the simultaneity shared by a plurality of terminals.
  • the plurality of terminals may make a socket connection with the server and share the command execution point by exchanging the user's input information from the terminal with each other.
  • a terminal that initially accesses the server or creates a new shared group is assigned as a master node from the server, and a terminal that accesses the same shared group thereafter is assigned as a slave node, maintaining socket connection and command give and receive
  • the terminal assigned as the master node may newly add a virtual object to be augmented and expressed in real space, or may change three-dimensional space coordinates.
  • the user touches an arbitrary point on the touch screen of the terminal, and the 2D coordinates are projected in 3D to obtain a line segment that can orthogonalize the camera coordinates and the 2D coordinates.
  • the coordinates of the three-dimensional space obtained in this way become the spatial coordinates for outputting the virtual object, and the spatial coordinates and object information are transmitted to each terminal connected to the server and the socket, so that a plurality of terminals can display the same object on the same real space.
  • It can be augmented, so that it is possible to realize a method of generating a spatial map and reading out a command execution point for sharing 3D spatial information in a plurality of terminals, which is intended to be achieved in the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention concerne un procédé permettant à une pluralité de terminaux de générer une carte d'espace afin de partager des informations d'espace tridimensionnel entre la pluralité des terminaux, et de lire un point d'exécution de commande, et, plus spécifiquement, un procédé qui combine des caractéristiques d'image acquises à partir d'images couleur continues et d'informations de capteur acquises par un capteur gyroscopique, de façon à générer des informations d'espace tridimensionnel; transmet/reçoit celles-ci à/d'un serveur par l'intermédiaire d'une communication d'interface de connexion afin de synthétiser des informations fragmentaires d'espace obtenues à partir d'une pluralité de terminaux, pour générer ainsi et partager des informations d'espace étendu; obtient une continuité en vue d'augmenter l'espace réel, projette des coordonnées bidimensionnelles introduites par un utilisateur par l'intermédiaire d'un écran tactile sur chaque terminal sur des coordonnées d'espace tridimensionnel, de façon à transformer les coordonnées d'espace tridimensionnel projetées en coordonnées tridimensionnelles d'espace réel, partage celles-ci entre la pluralité des terminaux afin que la pluralité des terminaux puissent augmenter simultanément le même objet virtuel dans le même lieu.
PCT/KR2020/000608 2019-12-31 2020-01-13 Procédé pour générer une carte d'espace afin de partager des informations d'espace tridimensionnel entre une pluralité de terminaux, et pour lire un point d'exécution de commande WO2021137348A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR20190179781 2019-12-31
KR10-2019-0179781 2019-12-31

Publications (1)

Publication Number Publication Date
WO2021137348A1 true WO2021137348A1 (fr) 2021-07-08

Family

ID=76686609

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/000608 WO2021137348A1 (fr) 2019-12-31 2020-01-13 Procédé pour générer une carte d'espace afin de partager des informations d'espace tridimensionnel entre une pluralité de terminaux, et pour lire un point d'exécution de commande

Country Status (1)

Country Link
WO (1) WO2021137348A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20130136569A (ko) * 2011-03-29 2013-12-12 퀄컴 인코포레이티드 각각의 사용자의 시점에 대해 공유된 디지털 인터페이스들의 렌더링을 위한 시스템
KR20160048874A (ko) * 2013-08-30 2016-05-04 퀄컴 인코포레이티드 물리적 장면을 표현하기 위한 방법 및 장치
US20160189432A1 (en) * 2010-11-18 2016-06-30 Microsoft Technology Licensing, Llc Automatic focus improvement for augmented reality displays
US20180045963A1 (en) * 2016-08-11 2018-02-15 Magic Leap, Inc. Automatic placement of a virtual object in a three-dimensional space
KR101989969B1 (ko) * 2018-10-11 2019-06-19 대한민국 증강현실 기반 건축 유적의 실감형 콘텐츠 체험시스템

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160189432A1 (en) * 2010-11-18 2016-06-30 Microsoft Technology Licensing, Llc Automatic focus improvement for augmented reality displays
KR20130136569A (ko) * 2011-03-29 2013-12-12 퀄컴 인코포레이티드 각각의 사용자의 시점에 대해 공유된 디지털 인터페이스들의 렌더링을 위한 시스템
KR20160048874A (ko) * 2013-08-30 2016-05-04 퀄컴 인코포레이티드 물리적 장면을 표현하기 위한 방법 및 장치
US20180045963A1 (en) * 2016-08-11 2018-02-15 Magic Leap, Inc. Automatic placement of a virtual object in a three-dimensional space
KR101989969B1 (ko) * 2018-10-11 2019-06-19 대한민국 증강현실 기반 건축 유적의 실감형 콘텐츠 체험시스템

Similar Documents

Publication Publication Date Title
CN110140099B (zh) 用于跟踪控制器的系统和方法
US20230392934A1 (en) Mapping in autonomous and non-autonomous platforms
US20230071839A1 (en) Visual-Inertial Positional Awareness for Autonomous and Non-Autonomous Tracking
CN110582798B (zh) 用于虚拟增强视觉同时定位和地图构建的系统和方法
CN110073313B (zh) 使用母设备和至少一个伴随设备与环境交互
US11948369B2 (en) Visual-inertial positional awareness for autonomous and non-autonomous mapping
EP2343882B1 (fr) Dispositif de traitement d'images, procédé de sélection d'objets et programme
US10547974B1 (en) Relative spatial localization of mobile devices
JP6348741B2 (ja) 情報処理システム、情報処理装置、情報処理プログラム、および情報処理方法
WO2023056544A1 (fr) Système de localisation et procédé de localisation d'objet et de caméra pour la cartographie du monde réel
CN110264509A (zh) 确定图像捕捉设备的位姿的方法、装置及其存储介质
WO2019221800A1 (fr) Système et procédé d'enregistrement spatial de multiples dispositifs de réalité augmentée
CN106304842A (zh) 用于定位和地图创建的增强现实系统和方法
KR20160138729A (ko) 영상에서 동작 인식을 위한 특징점 추출 방법 및 골격 정보를 이용한 사용자 동작 인식 방법
AU2015248967B2 (en) A method for localizing a robot in a localization plane
KR101639161B1 (ko) 골격 정보를 이용한 사용자 인증 방법
Kim et al. Recursive estimation of motion and a scene model with a two-camera system of divergent view
WO2021196973A1 (fr) Procédé et appareil d'affichage de contenu virtuel, dispositif électronique et support de stockage
WO2021137348A1 (fr) Procédé pour générer une carte d'espace afin de partager des informations d'espace tridimensionnel entre une pluralité de terminaux, et pour lire un point d'exécution de commande
WO2023088127A1 (fr) Procédé de navigation en intérieur, serveur, appareil et terminal
CA3172195A1 (fr) Systeme de localisation d'objet et de camera et methode de cartographie du monde reel
KR20230095197A (ko) 3차원 공간에서 생성한 인지적 매쉬 정보와 가상 객체들간의 상호작용 방법 및 그 장치
Sheng et al. Review on SLAM algorithms for Augmented Reality
KR20210048798A (ko) 단말에 구비된 카메라의 포즈를 결정하는 방법 및 이를 수행하는 위치 계산 서버
Jianjun et al. A direct visual-inertial sensor fusion approach in multi-state constraint Kalman filter

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20909054

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20909054

Country of ref document: EP

Kind code of ref document: A1