KR20230162292A - Method for generating map target data using 3D scanner - Google Patents

Abstract

A method of generating map target data using a 3D scanner according to an embodiment of the present invention is a method of generating map target data using a 3D scanner performed by a map target program running on at least one processor of a computing device. Obtaining depth data and image data sensed at the scanning position from; generating at least one keyframe based on the acquired depth data and image data; and generating map target data including the generated keyframe.

Description

Method and system for generating map target data using a 3D scanner {Method for generating map target data using 3D scanner}

The present invention relates to a method and system for generating map target data using a 3D scanner.

Industrial fields that are in the spotlight in the era of the 4th Industrial Revolution, such as augmented reality (AR), smart cities and smart factories, self-driving cars, drones, metaverse (virtual world), and digital twin (digital copies of real places and objects) are all ' ‘Spatial information’ is the core.

Until now, two-dimensional (2D) spatial information has had limitations in capturing the characteristics of the three-dimensional (3D) space in which we live. For example, in the case of building information, only area can be displayed in 2D, but in 3D, height can be added to express volume (scale) close to real life.

This makes it possible to simulate interior furniture arrangements and interiors, greatly increasing the value of spatial information in 3D. For this reason, building more sophisticated 3D spatial information is considered an essential infrastructure that will drive future technological development beyond the Fourth Industrial Revolution.

In particular, in order to implement augmented reality and autonomous driving, 3D spatial information is needed for the device to understand the surrounding environment and at the same time determine the location of the device.

SLAM (Simultaneous Localization And Map-Building) technology has been studied for a long time to sense 3D spatial information of the surrounding environment on devices, and recently, map target data has been developed at low cost through the visual slam algorithm using a camera device. It is being created.

However, this visual slam technology has a problem with the accumulation of position estimation errors, which causes significant deviation from the actual value, and when position estimation fails due to movement of the camera device, discontinuous position estimation values are continuously generated, so that the camera There is a problem of losing the location of the device.

Meanwhile, Lidar (short for Light Detection and Ranging), which generates 3D depth data in 3D space, can precisely collect distance measurements to nearby objects using pulsed light. there is.

These LiDAR sensors are widely used in autonomous driving and robotics applications, for navigation workflows such as mapping, simultaneous localization and mapping (SLAM), and route planning, as well as 3D recognition such as object detection and semantic segmentation. We are making workflow possible.

Recently, 3D scanners have been released that enable precise 3D depth scanning of the surrounding 360 degrees from one location using LiDAR or lasers, while simultaneously capturing images of the surroundings.

These 3D scanners can provide point cloud data that senses the distance from a fixed location in the field to points in objects in the surrounding environment, and 360-degree panoramic images of objects in the surrounding environment that match this. there is.

Through these 3D scanners, it is possible to obtain precisely sensed depth data and image data for 3D space at one location, but there is a problem in that it is difficult to utilize as 3D map target data because it is only unprocessed raw data.

Therefore, the present invention seeks to propose a method and system for generating 3D map target data for the surrounding environment based on depth data and image data acquired from a 3D scanner. Here, the 3D map target data determines the location of the device based on the image sensed by the device at a location in the surrounding environment and simultaneously scans 3D spatial information of the surrounding environment to determine the positional relationship with surrounding objects. This refers to the constructed map data.

In addition, the present invention seeks to propose a method and system for providing applications such as augmented reality through 3D map target data generated based on scan data from a 3D scanner.

A method of generating map target data using a 3D scanner according to an embodiment of the present invention is a method of generating map target data using a 3D scanner performed by a map target program running on at least one processor of a computing device. Obtaining depth data and image data sensed at the scanning position from; generating at least one keyframe based on the acquired depth data and image data; and generating map target data including the generated keyframe.

In addition, a map target data generation system using a 3D scanner according to an embodiment of the present invention includes a 3D scanner that senses image data and depth data for the scanning area at the scanning position; A computing device that acquires image data and depth data from the 3D scanner, generates a key frame based on the acquired image data and depth data, and generates map target data to which the generated key frame is set; and a device that executes a map target data-based application based on map target data obtained from the computing device.

The map target data generated according to the embodiment is generated based on accurately and precisely sensed image data and dense depth data, so the accuracy can be greatly improved compared to the map target data generated through the existing visual slam, and the visual It has the advantage of not requiring continuous updates like Slam.

In addition, the map target data generated in this way is based on image data accurately and precisely scanned through a scanner and keyframes based on the scan data, so it can provide accurate three-dimensional spatial information.

And through accurate 3D spatial information, an application provision environment based on map target data can be built.

Figure 1 is a conceptual diagram of a map target data generation system using a 3D scanner according to an embodiment of the present invention.
Figure 2 is an internal block diagram of a computer device executing a map target data generation program according to an embodiment of the present invention.
Figure 3 is an internal block diagram of a 3D scanner according to an embodiment of the present invention.
Figure 4 is a flowchart showing a method for generating map target data using a 3D scanner according to an embodiment of the present invention.
Figure 5 is a conceptual diagram of a panoramic image provided by a 3D scanner according to an embodiment of the present invention.
Figure 6 is a conceptual diagram of a cube map image converted from a panoramic image according to an embodiment of the present invention.
Figure 7 is an example of a front image in a cubemap image according to an embodiment of the present invention.
Figure 8 shows keyframe generation based on depth data and image data according to an embodiment of the present invention.
Figure 9 shows a method of generating map target data based on scan data of a plurality of scanning positions and providing various application environments based on the generated map target data according to an embodiment of the present invention.
Figure 10 shows map target data being generated based on a plurality of scanning position scan data according to an embodiment of the present invention.
Figure 11 is a diagram for explaining the process of generating map target data based on a plurality of scanning position scan data according to an embodiment of the present invention.
Figure 12 shows an example of providing an augmented reality environment based on map target data according to an embodiment of the present invention.

Since the present invention can be modified in various ways and can have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used not in a limiting sense but for the purpose of distinguishing one component from another component. Additionally, singular expressions include plural expressions unless the context clearly dictates otherwise. Additionally, terms such as include or have mean that the features or components described in the specification exist, and do not preclude the possibility of adding one or more other features or components. Additionally, in the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are shown arbitrarily for convenience of explanation, so the present invention is not necessarily limited to what is shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. When describing with reference to the drawings, identical or corresponding components will be assigned the same reference numerals and redundant description thereof will be omitted. .

1 is a conceptual diagram of a system for providing a user interface for creating a map target according to an embodiment of the present invention.

Referring to FIG. 1, the map target data generation system 1000 using a 3D scanner according to an embodiment of the present invention includes 3 objects including at least one object based on scan data acquired from the 3D scanner 300. Dimensional map target data can be generated, and an application execution environment using various 3D spatial information can be provided based on the generated map target data.

In embodiments, applications utilizing 3D spatial information may include autonomous driving, autonomous flight, automatic mobile robots, and augmented reality applications. The following embodiment will be described based on generating map target data to provide an augmented reality application execution environment.

In an embodiment, the map target data generation system 1000 as described above includes a computing device 100, a database server 200, a map target data-based application execution device 400, a 3D scanner 300, and a network. may include.

At this time, the computing device 100, database server 200, device, and 3D scanner 300 may be connected through the following network. In detail, the network refers to a connection structure that allows information exchange between nodes such as the computing device 100, the database server 200, the device 400, and/or the 3D scanner 300. Examples of networks include the 3rd Generation Partnership Project (3GPP) network, Long Term Evolution (LTE) network, World Interoperability for Microwave Access (WIMAX) network, Internet, Local Area Network (LAN), and Wireless Local Area Network (Wireless LAN). ), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc., but are not limited thereto.

Hereinafter, the computing device 100, the database server 200, and the 3D scanner 300 that implement the system 1000 for providing a user interface for creating a map target will be described in detail with reference to the attached drawings.

- Computing device (100: Terminal)

The computing device 100 according to an embodiment of the present invention may be a computing device installed with a map target program 111 that generates map target data based on scan data of the 3D scanner 300.

The map target program 111 can display scan data acquired from the 3D scanner 300 and generate map target data for an area based on positioning data, image data, and depth data in the scan data. Map target data for multiple areas can be generated by combining map target data generated from multiple positions.

In addition, the computing device 100 may be installed with an augmented reality program to build an augmented reality environment, and the augmented reality program can be installed within a three-dimensional space implemented by the map target data based on user input for the generated map target data. By creating virtual content in the first virtual location, the augmented reality device located in the three-dimensional space can augment and display the virtual content in the first real location corresponding to the first virtual location.

This computing device 100 may include a mobile type computing device 100-1 and/or a desktop type computing device 100-2 in which the map target program 111 is installed.

Here, the mobile type computing device 100-1 may be a mobile device such as a smart phone or tablet PC on which the map target program 111 is installed.

For example, the mobile type computing device 100-1 includes smart phones, mobile phones, digital broadcasting devices, personal digital assistants (PDAs), portable multimedia players (PMPs), and tablet PCs. may be included.

In addition, the desktop type computing device 100-2 is a fixed desktop PC with the map target program 111 installed, a laptop computer, a personal computer such as an ultrabook, etc. based on wired/wireless communication. It may include a device on which a program for executing the map target user interface service is installed.

Additionally, depending on the embodiment, the computing device 100 may further include a server computing device that provides a map target user interface service environment.

Figure 2 is an internal block diagram of the computing device 100 according to an embodiment of the present invention.

Meanwhile, referring to FIG. 2, from a functional perspective, the computing device 100 may include a memory 110, a processor assembly 120, a communication processor 130, an input system 140, and a user interface system 170. You can. These components may be configured to be included within the housing of computing device 100.

In detail, the map target program 111 is stored in the memory 110 , and the map target program 111 can store one or more of various applications, data, and commands for providing a map target user interface service environment. .

That is, the memory 110 can store commands and data that can be used in a program that generates map target data.

Additionally, the memory 110 may include a program area and a data area.

Here, the program area according to the embodiment may be linked between the operating system (OS: Operating System) and functional elements that boot the computing device 100, and the data area is generated according to the use of the computing device 100. data can be stored.

Additionally, the memory 110 may include at least one non-transitory computer-readable storage medium and a temporary computer-readable storage medium.

For example, the memory 110 may be a variety of storage devices such as ROM, EPROM, flash drive, hard drive, etc., and web storage that performs the storage function of the memory 110 on the Internet. may include.

The processor assembly 120 may include at least one processor capable of executing instructions of the map target program 111 stored in the memory 110 to perform various tasks for generating a map target.

In an embodiment, the processor assembly 120 may control the overall operation of components through the map target program 111 of the memory 110 to provide a map target creation environment.

This processor assembly 120 may be a system-on-chip (SOC) suitable for the computing device 100 that includes a central processing unit (CPU) and/or a graphics processing unit (GPU), and is stored in the memory 110. An operating system (OS) and/or application programs can be executed, and each component mounted on the computing device 100 can be controlled.

Additionally, the processor assembly 120 may internally communicate with each component through a system bus and may include one or more bus structures, including a local bus.

In addition, the processor assembly 120 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and other electrical units for performing functions.

The communication processor 130 may include one or more devices for communicating with external devices. This communication processor 130 can communicate through a wireless network.

In detail, the communication processor 130 may receive scan data for creating a map target or communicate with a predetermined computing device that stores the generated map target data to transmit it, and may be used by various users such as a controller that receives user input. Can communicate with input components.

In an embodiment, the communication processor 130 may transmit and receive various data to and from another computing device 100, the database server 200, and/or the 3D scanner 300.

This communication processor 130 uses technical standards or communication methods for mobile communication (e.g., Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), 5G New Radio (NR), WIFI). Alternatively, data can be transmitted and received wirelessly with at least one of a base station, an external computing device 100, and an arbitrary server on a mobile communication network established through a communication device capable of performing short-distance communication.

The user interface system 170 may communicatively connect the computing device 100 with one or more other devices. In particular, user interface system 170 may include wired and/or wireless communication devices compatible with one or more different communication protocols.

Through this user interface system 170, the computing device 100 can be connected to various input/output devices.

For example, the user interface system 170 may be connected to an audio output device such as a headset port or a speaker to output audio.

As an example, the audio output device is connected through the user interface system 170, but embodiments in which the audio output device is installed inside the computing device 100 may also be included.

Additionally, for example, the user interface system 170 may be connected to an input device such as a keyboard 152 and/or mouse 151 to obtain user input.

As an example, the keyboard 152 and/or mouse 152 are described as being connected through the user interface system 170, but embodiments in which the keyboard 152 and/or mouse 152 are installed inside the computing device 100 may also be included.

This user interface system 170 connects devices equipped with a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. port, audio I/O (Input/Output) port, video I/O (Input/Output) port, earphone port, power amplifier, RF circuit, transceiver and other communications It may be configured to include at least one of the circuits.

Input system 140 may detect user input (e.g., a gesture, voice command, operation of a button, or other type of input) related to a map target user interface service.

In detail, the input system 140 may include a predetermined button, a touch sensor, and/or an image sensor that receives a user motion input.

Additionally, the input system 140 may be connected to an external controller through the user interface system 170 to receive user input.

The display device 170 may output various information related to map target creation as a graphic image. In an embodiment, display device 170 may be connected to computing device 100 through user interface system 170.

In an embodiment, the display device 170 may display captured images, key frames, 3D maps, object information, and/or various user interfaces.

These displays include liquid crystal display (LCD), thin film transistor-liquid crystal display (TFT LCD), organic light-emitting diode (OLED), and flexible display. , a 3D display, or an e-ink display.

The above components may be disposed within a housing of the computing device 100, and the user interface may include a touch sensor 173 on the display 171 configured to receive user touch input.

In detail, the display device 170 may include a display 171 that outputs an image and a touch sensor 173 that detects a user's touch input.

For example, the display 171 may be implemented as a touch screen by forming a mutual layer structure or being integrated with the touch sensor 173. Such a touch screen may function as a user input unit that provides an input interface between the computing device 100 and the user, and may also provide an output interface between the computing device 100 and the user.

- Database server (200: Database server)

Meanwhile, the database server 200 according to an embodiment of the present invention may perform a series of processes for creating a map target and providing an application execution environment based on the created map target.

In detail, in an embodiment, the database server 200 stores data (e.g., a map) required to run a map target-based application execution environment process in an external device such as the computing device 100 and/or the 3D scanner 300. A map target-based application execution environment can be provided by exchanging target data and augmented reality-related data) with the external device.

In more detail, in the embodiment, the database server 200 may provide an environment in which the map target program 111 can operate in the computing device 100.

Additionally, in an embodiment, the database server 200 may provide map target data for operating an application and augmented reality data based on map target data to the device 400 for executing an application based on map target data.

To this end, the database server 200 may include an application program, data, and/or commands, and may transmit and receive data based thereon with the external device.

More specifically, in the embodiment, the database server 200 reads a predetermined algorithm driving program built to perform the above functional operation from memory, and performs the corresponding functional operation according to the read predetermined algorithm system. can do.

At this time, depending on the embodiment, the above predetermined algorithm is directly included in the database server 200 or is implemented in a device and/or server separate from the database server 200 to generate the map target and based on the generated map target. Functional operations for the application execution environment can be performed.

In the following description, the predetermined algorithm is described as being included and implemented in the database server 200, but is not limited thereto.

Additionally, in the embodiment, the database server 200 may store and manage various application programs, commands, and/or data for creating a map target and implementing an execution environment for the generated map target-based application.

In an embodiment, the database server 200 may store and manage at least one 3D map target data and various user interfaces and/or algorithms required for the generated map target-based application execution environment.

Meanwhile, referring further to FIG. 1, in the embodiment, the database server 200 as described above includes at least one processor module (210) for data processing and at least one or more processor modules (210) for data exchange with external devices. A communication module (220) and at least one memory module (230) that stores various applications, data and/or commands for creating a map target and providing an execution environment for the generated map target-based application. It can be implemented with a predetermined computing device including.

Here, the memory module 220 may store one or more of an operating system (OS), various application programs, data, and commands for creating a map target and providing an execution environment for the generated map target-based application.

Additionally, the memory module 220 may include a program area and a data area.

Here, the program area according to the embodiment may be linked between the operating system (OS) that boots the server and functional elements, and the data area may store data generated according to the use of the server.

In an embodiment, the memory module 220 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and may be a web device that performs the storage function of the memory module 220 on the Internet. It could also be storage (web storage). Additionally, the memory module 220 may be a recording medium that is removable from the server.

Meanwhile, the processor module 210 can control the overall operation of each unit described above in order to create a map target and implement an application execution environment based on the created map target.

This processor module 210 may be a system-on-chip (SOC) suitable for a server that includes a central processing unit (CPU) and/or graphics processing unit (GPU), and the operating system (OS) stored in the memory module 220 ) and/or application programs, etc., and control each component mounted on the server.

Additionally, the processor module 210 may internally communicate with each component through a system bus and may include one or more bus structures, including a local bus.

In addition, the processor module 210 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and other electrical units for performing functions.

In the above description, it has been explained that the database server 200 according to an embodiment of the present invention performs the functional operations as described above. However, depending on the embodiment, at least part of the functional operations performed by the database server 200 is performed by an external device. (e.g., computing device 100 and/or 3D scanner 300, etc.) may be performed, and at least part of the functional operation performed by the external device may be further performed by the database server 200. Various embodiments such as these may be possible.

- 3D scanner (300)

The 3D scanner 300 according to an embodiment of the present invention can generate scan data that is the basis for generating map target data. Here, the scan data may include positioning data, depth data, and image data sensed at one location by the 3D scanner 300.

Figure 3 is an internal block diagram of a 3D scanner 300 according to an embodiment of the present invention. Hereinafter, elements constituting the 3D scanner 300 according to an embodiment of the present invention will be described in detail with reference to the attached drawings. However, since the components shown in FIG. 3 are not essential for the 3D scanner 300, the 3D scanner 300 may be implemented with more components or fewer components.

The 3D scanner 300 according to an embodiment may include an image sensor 310, a depth sensor 320, a processor, and an input/output unit.

In an embodiment, the sensor unit may include at least one sensor that senses a 360-degree surrounding area at one location of the 3D scanner 300.

In detail, the image sensor 310 of the sensor unit can acquire images (still images and/or moving images) of surrounding objects, and provide the shooting point to be stored in memory along with the images as a timestamp.

In an embodiment, the image sensor 310 may obtain and provide image data including images in the front, rear, top, bottom, left, and right directions at a location where the 3D scanner 300 is placed.

Referring to FIG. 5, the image data acquired by the image sensor 310 includes the front, back, top, down, left, and front of the 3D scanner 300. It may be a 360-degree panoramic image including a right side image.

This image sensor 310 can convert the image obtained by the image sensor 310 device (for example, CMOS or CCD) into an electrical signal, process the image, and store it in memory.

Additionally, the image sensor 310 may be a camera assembly including at least one camera. The camera assembly may include a general camera that photographs a visible light band, and may further include a special camera such as an infrared camera or a stereo camera.

Additionally, in the embodiment, the depth sensor 320 of the sensor unit may detect the distance to a surrounding object.

In an embodiment, the depth sensor 320 may include a LIDAR sensor (LIDAR sensor and/or a laser sensor, etc.) and/or a radar sensor (RADAR Sensor).

In an embodiment, the LIDAR sensor is a radar system that measures the position coordinates of an object by shooting a laser pulse and measuring the time it takes for it to reflect and return to surrounding objects (Time-of-Flight methodology). For example, in the case of the VLP-16 sensor, a type of LiDAR sensor, it can collect and provide its own location information according to the radius r, altitude ω, and azimuth α in spherical coordinates.

This LiDAR sensor senses the distance for each point obtained by firing a laser at a plurality of points of surrounding objects located in the front, rear, up, down, left, and right directions, and acquires a point data set including the distance for each point in real time. The 3D scanner 300 can scan the surrounding environment in 3D. At this time, timestamp data at the time of sensing for each point data may be stored together.

And the processor 330 can generate point cloud data including a pointer data set acquired from a LiDAR sensor, and model surfaces including a plurality of points based on the point cloud data to create a 3D mesh map (mesh). map) can be created.

Additionally, the processor 330 may match the image data and depth data obtained as described above based on the timestamp of each sensing point and store it as scan data.

Additionally, the sensor unit may further include a position sensor for sensing the scanning position, which is position information of the 3D scanner 300 that acquires scan data.

The position sensor may include a sensor such as GPS that senses the absolute coordinates of the scanning position.

Additionally, the position sensor may be a combination of a motion sensor for sensing relative position information between the first scanning position where the first scan data is acquired and the second scanning position where the second scan data is acquired. For example, at least one of the acceleration and rotation angle of the 3D scanner 300 may be detected. For example, the position sensor may include an accelerometer, gyroscope, magnetometer sensor, etc.

And the processor 330 of the 3D scanner 300 controls the input/output unit (I/O) 340 to operate an external computing device (in the embodiment, the computing device 100 and/or the database server 200, etc.) You can send scan data to.

The input/output unit may further include a display , and the display can output graphic images related to the image data and depth data sensed to create a map target, and may provide a user interface to proceed with the sensing process of the 3D scanner 300. It can be displayed.

Additionally, the processor 330 can control each component of the 3D scanner 300 to generate scan data.

Meanwhile, a moving part is installed in the 3D scanner 300, so that the 3D scanner 300 can move its position under control of a user or processor.

In embodiments, such mobile units may include wheel-type structural mobile devices and/or walking-type structural mobile devices, etc.

Here, the wheel-type structure moving device may include at least two driving wheels (e.g., left/right driving wheels, etc.) for moving the 3D scanner 300, and the driving wheels are used to move the 3D scanner 300. ) may be placed on at least part of the lower part. At this time, the driving wheels may be configured to rotate by at least one motor each operated by a processor, and the three-dimensional scanner 300 may change direction and move according to the driving of the motor.

In addition, the walking structure moving device includes at least two leg-type supports (for example, a first and/or second left leg support and a first and/or second right leg) for moving the 3D scanner 300. type support, etc.), and each of the leg-type supports can cause the 3D scanner 300 to change direction and walk by driving at least one motor that operates under the control of a processor.

In addition, the communication unit of the input/output unit 340 is one or more devices for communicating with an external computing device (in the embodiment, another 3D scanner 300, computing device 100, and/or database server 200, etc.) (eg, communication processor, etc.) may be included.

This communication unit sets technical standards or communication methods for mobile communication (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), High Speed Downlink Packet Access (HSDPA), and High Speed Uplink (HSUPA). Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.) and at least one of a base station, an external computing device 100, and/or a server and a wireless signal. Can send and receive.

In addition, the 3D scanner 300 may further include a storage unit, and the storage unit of the 3D scanner 300 may store one or more of an operating system (OS), various applications, data, and commands for generating scan data. You can save it.

In an embodiment, the storage unit may store and manage an information collection program based on the 3D scanner 300.

In an embodiment, this storage unit may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and web storage that performs the storage function of the storage unit 570 on the Internet. ) may be.

In addition, the processor 330 may be a system-on-chip (SOC) suitable for the 3D scanner 300 that includes a central processing unit (CPU) and/or a graphics processing unit (GPU), and an operating system stored in the storage unit. It may execute an operating system (OS) and/or a sensing program, and may include at least one processor capable of controlling each component mounted on the 3D scanner 300.

These processors 330 include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, It may be implemented using at least one of micro-controllers, microprocessors, and other electrical units for performing functions.

- How to create map target data

Figure 4 is a flowchart showing a method of generating map target data using a 3D scanner 300 according to an embodiment of the present invention.

Hereinafter, a process of generating map target data based on scan data received from the 3D scanner 300 according to the control command of the map target program 111 of the computing device 100 of the map target data generation system 1000 described above is performed. This will be described in detail with reference to FIG. 4 .

Referring to FIG. 4, the map target program 111 can acquire depth and image data sensed at one scanning position from the 3D scanner 300. (S101)

In detail, the 3D scanner 300 generates scan data including depth data and image data sensed at a fixed scanning position, and transmits the generated scan data to the computing device 100 through the input/output unit, The map target program 111 can acquire scan data. At this time, the scanning position means 6 degrees of freedom including position and orientation.

In addition, the depth data may be a point data set that includes the distance for each point by sensing the distance from the scanning position to the point of a surrounding object located on the front, rear, left, right, and left directions of the 3D scanner 300.

In detail, in the embodiment, the depth data may be point cloud data including a pointer data set, and may be a 3D mesh map modeling surfaces including a plurality of points based on the point cloud data.

And the image data may be an omnidirectional image captured omnidirectionally at a scanning position. For example, the image data may be an omnidirectional image sensed through at least one omnidirectional lens. Additionally, the image data may be a 360-degree panoramic image as shown in FIG. 5 created by flattening an omnidirectional image through panoramic image processing.

And the depth data and image data can be matched based on the timestamp, which is the respective sensing point.

The map target program 111 that receives the scan data can generate a cube map based on the image data. (S103)

A cube map may refer to a map in which an omnidirectional image is separated and arranged into face images, which are two-dimensional flat images in each of a plurality of directions.

To this end, the map target program 111 can generate a flat 360-degree panoramic image by performing panoramic image processing on the received omnidirectional image. Depending on the embodiment, a panoramic image may be received from the 3D scanner 300.

And the map target program 111 can separate the panoramic image into face images in the front, rear, top, bottom, left, and right directions, and determine the posture of each face image. For example, the face image is the front image (front), back image (back), top image (top), bottom image (down), left image (left), and right image (right) of the 3D scanner 300. It may include at least one image.

Referring to FIG. 6, in the embodiment, the cubemap divides the omnidirectional image into a front image (front), a back image (back), an upper image (top), a lower image (down), a left image (left), and a right image (right). ), the positions are the same, and the postures can be generated by classifying them into the forward, upward, downward, left, and right directions.

Additionally, the map target program 111 can generate a depth map based on depth data. (S105)

In an embodiment, the depth data acquired by the map target program 111 may be point cloud data obtained by scanning 360 degrees in all directions from the scanning position.

The map target program 111 can generate a 3D mesh map as a depth map by modeling surfaces including a plurality of points located on the same surface based on point cloud data.

Additionally, the map target program 111 can detect feature points in the face image of the cube map. (S107)

In detail, the map target program 111 can detect a plurality of feature points in each face image by executing a predetermined feature detector.

Here, a feature point literally means a characteristic point in an image, and may be a point that serves as a standard for comparing images when checking whether images match each other. In other words, feature points are used when matching images and can also be called keypoints.

At this time, the feature detector may calculate a descriptor that can specify each detected feature point for each of the feature points.

The descriptor divides the pixels around the corresponding feature point into blocks of a certain size and calculates the gradient histogram of the pixels belonging to each block, and mainly includes information such as brightness, color, direction, and/or size around the feature point. You can.

These feature detectors may be implemented based on algorithms such as FastFeatureDetector, MSER, SimpleBlobDetector, and/or GFTTDetector. However, this is only an example and is not limited to this.

Referring to FIG. 7, the descriptor (D1) of the first feature point, the descriptor (D2) of the second feature point, and the descriptor (D3) of the third feature point detected through the feature detector in the front image, which is one of the face images. indicates.

And the map target program 111 can determine the depth for each feature point of the face image based on the depth map. (S109)

The map target program 111 may calculate the distance between the feature point of the face image and the scanning position based on the depth map and detect the location of each feature point.

In the embodiment, the map target program 111 may calculate the location of the pixel included in the descriptor of the feature point based on the depth map.

In detail, the map target program 111 can calculate the depth, which is the distance from the scanning position to the image formed on the depth map by performing lane casting from the scanning position to the feature point (eg, descriptor pixel) of the face image.

In more detail, the map target program 111 determines the point on the mesh surface of the depth map when extending the line connecting the pixel included in the descriptor of the feature point at the scanning position, and calculates the distance to the determined point as the pixel It can be calculated with a depth of .

Referring to FIG. 8, when raincasting is performed on the pixels of the descriptor (D1) of the first feature point, the image is formed on the first point (P1) in the 3D mesh map, and the image of the descriptor (D2) of the second feature point is shown. When raincasting on a pixel, the image is formed on the second point (P2) on the object, and when raincasting on the pixel of the descriptor (D3) of the third feature point, the image is formed on the third point (P3) on the object in the 3D mesh map. ) shows the image forming.

In this way, the map target program 111 can determine the depth for each pixel of the feature point detected in each face image.

That is, the map target program 111 can calculate the depth of pixels included in the 2D face image and create a 3D map image.

Next, the map target program 111 may determine the face image in which the pixel depth of the feature point is determined as a key frame. (S111)

Key frames refer to specific frames within a three-dimensional space map that are set as a standard for detecting the relative position with respect to the surrounding environment through slam due to significant changes occurring among the frames.

These keyframes must contain 3D location information for each feature point in the frame, and the face image can be set as a keyframe because it includes the scanning position, feature point, and depth of the feature point.

Accordingly, the map target program 111 can set a keyframe based on the posture for each face image at the scanning position.

For example, at a common location of the scanning position, each face image of a cube map classified into front, back, top, bottom, left and right can be set as a keyframe.

Finally, the map target program 111 can generate map data in which each face image of the cube map is set as a keyframe. (S113)

The map data generated in this way is generated based on accurately and precisely sensed image data and dense depth data, so the accuracy can be greatly improved compared to the map target data generated through existing Visual Slam, and like Visual Slam, It has the advantage of not requiring continuous updates.

In this way, map data generated based on scan data sensing the surrounding area at one scanning position may be included in map target data of a wide area including the surrounding area.

- How to create and use map target data

Figure 9 shows a method of generating map target data based on scan data of a plurality of scanning positions and providing various application environments based on the generated map target data according to an embodiment of the present invention.

Referring to FIG. 9, based on scan data received from the 3D scanner 300 at a plurality of scanning positions according to a control command of the map target program 111 of the computing device 100 of the map target data generation system 1000 The process of generating map target data will be described in detail with reference to FIG. 9.

First, the map target program 111 may acquire the first depth and image data of the first scanning position sensed by the 3D scanner 300 at the first scanning position. (S201)

The map target program 111 may designate the first scanning position where scanning was performed for the first time when generating map target data as the origin. That is, the position of the first scanning position may be [0, 0, 0].

Thereafter, the map target program 111 may generate and set a first keyframe of the first scanning position based on the first depth data and the first image data. (S203)

The map target program 111 can create and set the first keyframe through steps S101 to S113 described above.

This first keyframe may include a keyframe generated from at least one of the front, top, bottom, left, and right face images.

Next, the map target program 111 can generate map data including the first keyframe. (S205)

In an embodiment, the first keyframe included in the map data includes a forward keyframe of the scanning position in a forward posture to the origin position, a rear keyframe of the scanning position in a backward posture to the origin position, and a scanning position in an upward posture to the origin position. It may include an upward keyframe of the position, a downward keyframe of the scanning position in a downward posture to the origin position, a left keyframe of the scanning position in a leftward posture to the origin position, and a right keyframe of the scanning position in a rightward posture to the origin position. It is possible.

Next, the map target program 111 can acquire the second depth data, second image data, and second pose data sensed by the 3D scanner 300 at the second scanning position moved from the first scanning position. there is. (S207)

The map target program 111 may determine the second scanning position based on pose data.

In detail, the map target program 111 may receive the position coordinates of the second scanning position as pose data and determine the second scanning position.

In another embodiment, the map target generation program may determine the relative position and relative posture of the second scanning position from the first scanning position based on movement data when moving from the first scanning position to the second scanning position using pose data. there is. At this time, the pose data may include rotation and transformation values that are relatively changed from the first scanning position to the second scanning position.

Depending on the embodiment, the 3D scanner 300 may calculate the second scanning position and transmit it to the computing device 100.

Next, the map target program 111 may generate a second keyframe of the second scanning position based on the second depth data and the second image data, and set the second keyframe based on the first scanning position. You can. (S209, S211)

The map target program 111 can generate and set a second key frame by performing steps S101 to S113 described above.

This second keyframe may include a keyframe generated from at least one of the front, top, bottom, left, and right face images.

The map target program 111 may set the front, rear, top, bottom, left, and right face images for which the depth of the feature point is determined as a keyframe for the first scanning position based on pose data.

For example, referring to FIG. 10, the map target program 111 sets the transformation (T), which has moved from the first scanning position, to the position of the second key frame (K2), and the changed rotation (R) and The posture of each key frame (K2) can be set based on the forward, upward, downward, left, and right postures distinguished from the second scanning position.

And the map target program 111 can generate map data including the second keyframe whose position and posture are determined in this way and include it in the map target data. (S213)

In this way, the process of setting key frames at multiple scanning positions can be repeated to generate map target data for multiple scanned areas.

For example, referring to FIG. 11, the map target program 111 repeats processes S207 to S213 in the third to seventh scanning positions (S3 to S7) to reach the third to seventh scanning positions (S3 to S7). Map target data for the first to seventh scanning areas (SA1 to SA7) can be generated by merging key frames in .

Finally, the map target program 111 can provide map target data to a device running a map target data-based application.

In detail, in the embodiment, the map target program 111 provides map target data and virtual content created on the map target data to an augmented reality application-based device, providing an augmented reality environment to users of augmented reality-based devices. can do. (S215)

Figure 12 shows an example of providing an augmented reality environment based on map target data according to an embodiment of the present invention.

Referring to FIG. 12, the map target data-based application execution device 400 according to an embodiment may include a wearable type computing device such as smart glasses display or head mounted display (HMD).

The smart glass type map target data-based application execution device 400 is a glass that displays virtual content (e.g., virtual object image) on the user's field of view while transmitting light so that the user can see the surrounding physical space while worn. It may include a display system including.

In detail, the map target data-based application execution device 400 of the embodiment is a transparent device that transmits light from the surrounding physical space to reach the user's eyes and at the same time reflects the virtual content displayed by the display system toward the user's eyes. It may include a glass display.

For example, the augmented reality application of the map target data-based application execution device 400 is configured to display a real object (RO) and a marker (MK) within the real object in an image taken based on map target data for the surrounding physical space 20. ), etc. can be image recognized, and can be controlled to display virtual content (VC1) in the user's field of view corresponding to the recognized marker (MK).

Additionally, the augmented reality application can provide an augmented reality environment by controlling the display of virtual content (VC2) in the user's field of view corresponding to the location of the real object recognized based on map target data.

This virtual content may include visual content, such as images or videos, that can be displayed in a portion of the user's field of view on the device. For example, virtual content may include virtual object images that overlay various parts of physical space. These virtual object images can be rendered as 2D images or 3D images.

The map target data generated in this way is based on image data accurately and precisely scanned through the 3D scanner 300 and keyframes based on the scan data, so it can provide accurate 3D spatial information.

And through accurate 3D spatial information, an application provision environment based on map target data can be built.

The embodiments according to the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory modules 220, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. A hardware device can be converted into one or more software modules to perform processing according to the invention and vice versa.

The specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For the sake of brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connections or connection members of lines between components shown in the drawings exemplify functional connections and/or physical or circuit connections, and in actual devices, various functional connections or physical connections may be replaced or added. Can be represented as connections, or circuit connections. Additionally, if there is no specific mention such as “essential,” “important,” etc., it may not be a necessary component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, those skilled in the art or those skilled in the art will understand the spirit of the present invention as described in the patent claims to be described later. It will be understood that the present invention can be modified and changed in various ways without departing from the technical scope. Therefore, the technical scope of the present invention should not be limited to what is described in the detailed description of the specification, but should be defined by the scope of the patent claims.

Claims (10)

A method of generating map target data using a 3D scanner performed by a map target program running on at least one processor of a computing device, comprising:
Obtaining depth data and image data sensed at the scanning position from a 3D scanner;
generating at least one keyframe based on the acquired depth data and image data; and
Comprising the step of generating map target data including the generated keyframe.
Map target data generation method using a 3D scanner. According to claim 1,
The step of generating at least one keyframe based on the acquired depth data and image data,
generating a face image based on the image data;
Including detecting feature points in the face image.
Map target data generation method using a 3D scanner. According to clause 2,
The step of generating a face image based on the image data is,
Including the step of flattening the omnidirectional image in the image data and separating the face image for at least one direction among front, rear, up, down, left, and right.
Map target data generation method using a 3D scanner. According to clause 3,
The step of generating a face image based on the image data is,
Comprising the step of generating a cube map that divides the face images in the front, rear, up, down, left, and right directions into a posture in each direction.
Map target data generation method using a 3D scanner. According to clause 3,
The step of generating at least one keyframe based on the acquired depth data and image data,
detecting a descriptor of a feature point of the face image;
Raincasting the pixels included in the descriptor of the feature point into a mesh map of depth data at the scanning position;
Comprising the step of determining the image formed on the mesh map as a three-dimensional position in a pixel of the descriptor of the feature point.
Map target data generation method using a 3D scanner. According to clause 5,
The step of generating at least one keyframe based on the acquired depth data and image data,
determining a depth at a feature point of the face image;
Including setting the face image with the determined depth as a keyframe based on the position and posture of the scanning position.
Map target data generation method using a 3D scanner. According to claim 1,
The step of acquiring depth data and image data sensed at the scanning position from the 3D scanner,
acquiring first scan data at a first scanning position;
Comprising acquiring second scan data at the second scanning position and pose data for the first scanning position.
Map target data generation method using a 3D scanner. According to clause 7,
The step of generating at least one keyframe based on the acquired depth data and image data,
generating a first keyframe based on the first scanning position and the first scan data;
Comprising the step of generating a second keyframe based on the second scanning position and the second scan data.
Map target data generation method using a 3D scanner. According to clause 8,
The step of generating map target data including the generated keyframe is,
Comprising the step of setting the second keyframe to the map target data based on the pose and posture for the first scanning position.
Map target data generation method using a 3D scanner. A 3D scanner that senses image data and depth data for the scanning area at the scanning position;
A computing device that acquires image data and depth data from the 3D scanner, generates a key frame based on the acquired image data and depth data, and generates map target data to which the generated key frame is set; and
Comprising a device that executes a map target data-based application based on map target data obtained from the computing device.
Map target data generation system using a 3D scanner.