KR20220087688A - Geo-spacial data estimation method of moving object based on 360 camera and digital twin and the system thereof - Google Patents

Abstract

The present invention relates to a method and system for estimating geographic data of a moving object based on a 360 camera and a digital twin, comprising the steps of receiving and preprocessing image data from a 360 camera, extracting the position of an object in the image data, and a digital twin estimating an initial position of the object by estimating an image depth through (Digital Twin) and using a 3D model disposed in the digital twin based on the initial position in the image data calculating a difference from an object, and estimating geographic data including a position and an angle of the object through optimization to minimize the difference.

Description

GEO-SPACIAL DATA ESTIMATION METHOD OF MOVING OBJECT BASED ON 360 CAMERA AND DIGITAL TWIN AND THE SYSTEM THEREOF

The present invention relates to a method and system for estimating geographic data of a moving object based on a 360 camera and a digital twin, and more particularly, to generate geographic location and angle data of moving objects such as people and vehicles based on a 360 camera. It's about technology.

Existing augmented reality (AR) technology has been focused on research centered on entertainment. Efforts to collect and analyze various large-scale data generated by cities are continuing to study urban space problems and to build smart cities. One example of such an attempt is to collect information and set up the Internet of Things (IoT) in a wide city to transform various urban phenomena into useful data. As a method of visualizing the collected information in this way, data visualization using augmented reality will be provided for various city services.

Meanwhile, despite efforts to disseminate smart services, the development of augmented reality visualization tools for city managers is still insufficient. The reason it is difficult for smart city researchers and city service designers to utilize existing IoT data is that there is no system that can collect and transmit data in real time, or it is difficult to obtain permission to access IoT data from other institution operators even if an IoT system is in place. to be.

In addition, due to the increase in demand for smart cities, IoT sensors are not provided at the time of manufacturing a city digital twin, so unnecessary time is required for development. Accordingly, it is necessary to study an IoT event generation system that will serve as virtual signal generation, data transmission and storage instead of hardware IoT sensors. In particular, in the digital twin development stage for the initial smart city service design, a simulation method using virtual IoT is required to minimize the waste of time and material costs that may occur depending on the utilization of IoT when building an actual IoT system. .

Streaming services using general cameras, such as CCTV, have limited viewing angles and cannot see all areas. Therefore, although CCTV has a function such as automatically adjusting the angle of the camera, there is still a problem in that all areas cannot be viewed at the same time. Recently, as low-cost 360 cameras have become popular, a service using a 360 camera having a viewing angle that can view all areas at the same time is required.

Furthermore, the existing image-based moving object position tracking method cannot know the geographic position, so only the position of the object relative to the camera can be known, and since only the image is used as an input value, inaccurate results are produced.

Nguyen D., Meixner G.: Comparison User Engagement of Gamified and Non-gamified Augmented Reality Assembly Training Advances in Agile and User-Centred Software Engineering. pp. 142-152. Springer International Publishing (2020)

An object of the present invention is to enable more accurate location tracking by easily acquiring object positions and distances in image data using a 360 camera and digital twin.

In a method for estimating geographic data of a moving object based on a 360 camera and a digital twin according to an embodiment of the present invention, the method comprising: receiving and preprocessing image data from a 360 camera; extracting the position of an object in the image data; , estimating the initial position of the object by estimating the image depth through a digital twin, and the image using a 3D model disposed in the digital twin based on the initial position calculating a difference with the object in the data, and estimating geographic data including the position and angle of the object through optimization to minimize the difference.

In the system for estimating geographic data of a moving object based on a 360 camera and a digital twin according to an embodiment of the present invention, a preprocessor for receiving and preprocessing image data from a 360 camera, extracting the position of an object in the image data and an initial position estimator that estimates the initial position of the object by estimating image depth through a digital twin and a 3D model disposed in the digital twin based on the initial position and a geographic data estimator for calculating a difference from the object in the image data using the estimator and estimating geographic data including the position and angle of the object through optimization for minimizing the difference.

According to an embodiment of the present invention, in an environment where IoT sensor support is not possible, a digital twin in the initial stage can be easily built using a 360 camera, and due to this, when designing a smart city service, until the construction of the IoT sensor is completed Time and physical costs incurred can be minimized.

Also, according to an embodiment of the present invention, real-time monitoring of the positions of moving objects may be possible using the IoT data visualization function.

In addition, according to an embodiment of the present invention, it provides city researchers with various city information such as traffic analysis, parking space information, and traffic passenger information to study new smart services, and several companies that are difficult to build IoT It can lay the foundation for digital twin production.

1 is a flowchart illustrating an operation of a method for estimating geographic data of a moving object according to an embodiment of the present invention.
2 is a block diagram illustrating a detailed configuration of a system for estimating geographic data of a moving object according to an embodiment of the present invention.
3 is a flowchart illustrating a process of estimating an initial position according to an embodiment of the present invention.
4 is a flowchart illustrating a process of estimating geographic data including the position and angle of an object according to an embodiment of the present invention.
5 to 7 show results of a demonstration test according to an embodiment of the present invention.
8A and 8B show examples of application according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be embodied in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, operations and/or elements mentioned. or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Embodiments of the present invention make it a gist of generating geo-based position and angle data of moving objects, such as people and vehicles, based on a 360 camera.

Using the technology according to an embodiment of the present invention, if there is only one 360 camera, real-time location information of moving objects around the camera can be tracked and monitored, and through this, IoT data essential for a digital twin such as a smart city can be more easily built. .

The present invention applies a localization technology to find out the location of an object after placing a 360 camera at the same position and angle as in reality on a digital twin expressed as a three-dimensional map based on a building and a road. For the localization technique, an image segmentation-based initial object position estimation and optimization technique for adjusting the position and angle of an object are applied.

Furthermore, the present invention demonstrates through a visualization tool that the corresponding system is effectively reflected in the digital twin by effectively generating the geographic location of moving objects by applying the corresponding technology to the actual urban space.

Hereinafter, the present invention will be described in more detail with reference to FIGS. 1 to 8 .

1 is a flowchart illustrating an operation of a method for estimating geographic data of a moving object according to an embodiment of the present invention, and FIG. 2 is a block diagram illustrating a detailed configuration of a system for estimating geographic data of a moving object according to an embodiment of the present invention. will be.

1 and 2, a method and system for estimating geographic data of a moving object according to an embodiment of the present invention generate geo-based position and angle data of moving objects such as people and vehicles based on a 360 camera.

To this end, in FIG. 2 , the system 200 for estimating geographic data of a moving object according to an embodiment of the present invention includes a preprocessor 210 , an initial location estimator 220 , and a geographic data estimator 230 . In addition, each of the steps (steps S110 to S130) of FIG. 1 is a component of the system 200 for estimating geographic data of a moving object according to the embodiment of the present invention of FIG. 2, that is, the preprocessor 210, the initial position. This may be performed by the estimator 220 and the geographic data estimator 230 .

In step S110, the pre-processing unit 210 receives and pre-processes the image data from the 360 camera.

The preprocessor 210 may receive (360 Streaming) image data in real time from a 360 camera, and preprocess it into a perspective view having a specific viewing angle.

The 360 camera has a wide field of view that covers all areas around the camera without blind spots, making it easy to monitor. The pre-processing unit 210 may pre-process the image data to support functions beyond simple streaming image data, and the present invention may create a 6-degree-of-freedom (6 DoF) virtual environment using multiple 360 image data.

The initial position estimator 220 and the geographic data estimator 230 according to an embodiment of the present invention restore an object in three dimensions using a 3D object retrieval technique to estimate a 6DoF pose. ) is performed.

In step S120 , the initial position estimator 220 extracts the position of the object in the image data, and estimates the image depth through a digital twin 240 to estimate the initial position of the object.

More specifically, the initial position estimator 220 extracts pixel coordinates for the position of an object in image data using an instance segmentation algorithm, and uses a street view and a 3D model Image depth can be estimated using camera posture and ground information in the virtual space of the digital twin built on the basis of it. Thereafter, the initial position estimator 220 may estimate an initial position that is a three-dimensional position of the object by using the pixel coordinates of the object and the depth value of the corresponding pixel.

In this case, the system 200 for estimating geographic data of a moving object according to an embodiment of the present invention may place the 3D model in the digital twin 240 .

In step S130, the geographic data estimator 230 calculates a difference from an object in the image data using a 3D model placed in the digital twin 240 based on the initial position (Initial position), The geographic data including the position and angle of the object are estimated through optimization that minimizes the difference.

The geographic data estimator 230 uses a Particle Swarm Optimization (PSO) algorithm to project an object outline of estimated data generated by projecting a 3D object of a 3D model to an initial position and an object outline in image data. ), and optimization to minimize the error can be performed.

The geographic data estimator 230 performs an optimization process by iteration of changing the position and rotation of the 3D object using the PSO algorithm, and the position of the object in the image data and a 6DoF pose including an angle may be estimated (Final position/rotation).

The digital twin 240 according to an embodiment of the present invention is used in various systems and industries such as manufacturing, medical care, transportation, aerospace, and construction, and uses standardized data to provide IoT data (IoT Data), 3D object (3D Objects) and may be formed based on a 3D map (3D Geographic Map).

3 is a flowchart illustrating a process of estimating an initial position according to an embodiment of the present invention.

In order to interact with an object and to move the object freely in a virtual space composed of three dimensions, a technology for restoring an object in an image (or image data) in three dimensions is required. Accordingly, the present invention is to find a model most similar to the object in the image to be restored among 6DoF pose estimation, which estimates the posture of an object, and 3D model, which is currently available to restore the object in three dimensions. 3D object retrieval technology is used.

For example, when image data represents a road environment and a specific car in the road environment is extracted as an object, the present invention is a 3D model that finds a model most similar to a car in a 3D model together with 6DoF pose estimation to estimate the posture of the car Dimensional object search techniques may be used.

Furthermore, in the present invention, an initial position of an object is estimated as a first step for 6DoF pose estimation. To this end, the present invention first obtains pixel coordinates of an object in image data using a Mask-RCNN based image segmentation algorithm. Then, using the camera posture and ground information in a digital twin virtual space built based on street view and 3D model, image depth information (e.g. 360 distance value between the camera and the object). Using the pixel coordinates of the object extracted as described above and the depth value of the pixel, the present invention estimates the initial position, which is the three-dimensional position of the object, and places the 3D model at the corresponding position of the digital twin.

4 is a flowchart illustrating a process of estimating geographic data including the position and angle of an object according to an embodiment of the present invention.

An error may occur in the process of image segmentation and depth data extraction in the initial position estimated through the process of FIG. 3 , and the rotation of the object is estimated in the process of FIG. 3 . Can not.

Accordingly, the present invention provides an image created by projecting a three-dimensional object disposed at an initial position estimated using a Particle Swarm Optimization (PSO) algorithm like the method used in the camera posture estimation module through the process of FIG. 4 . Optimization proceeds in the direction of minimizing the error between the object outline and the object outline of the existing image.

For example, in the process of FIG. 3 , the present invention acquires a model most similar to a car using a three-dimensional object search technique, and object outline and image data of the most similar car model obtained in the process of FIG. 3 . I can perform optimizations to minimize the error for matching between the outlines of objects in my car. At this time, the present invention performs optimization as an iterative process of changing the position and rotation of the most similar car model obtained in the process of FIG. 3 using the PSO algorithm, thereby performing the optimization of the position and angle of the car object in the image data It is possible to estimate a 6DoF pose including

5 to 7 are diagrams showing results of an empirical test according to an embodiment of the present invention.

Figure 5 (a) shows the image data (or 360 image) taken with the 360 camera used for the test, Figure 5 (b) shows the result of image data segmentation (instance segmentation), Figure 5 (c) shows the result of estimating the initial position of the object.

Referring to FIG. 5 , the present invention shows the results of an empirical test for a digital twin and Google Street View 360 images for a specific area of Berlin. As shown in Fig. 5(a), image data was used as a test image, and the result of image segmentation of the data is as shown in Fig. 5(b). Furthermore, Fig. 5(c) shows the result of restoring the 3D model of the car in the digital twin by estimating the initial position.

FIG. 6(a) shows the results of performing optimization with 2 iterations using the PSO algorithm according to an embodiment of the present invention, and FIG. 6(b) shows the results of performing optimization with 4 iterations. 6(c) shows the result of performing the optimization with 10 iterations, and FIG. 6(d) shows the result of performing the optimization with 20 iterations.

6, the results of the optimization process using the initial position of the object and the PSO algorithm according to an embodiment of the present invention are shown. In the present invention, a total of 20 iterations were performed, and as the position and rotation of the 3D object changed according to the repetition, the object of the existing image and the projected 3D object gradually matched. can check that

7 is a comparison of the 3D model restored through the estimated position and posture and the outline of the original image object. FIG. 7(a) shows the initial estimation result, and FIG. 7(b) shows the optimization process. The results after roughing are shown.

Referring to FIG. 7 , as shown in FIG. 7( b ), after the existing photo ( FIG. 7 ( a )) and the outline of the 3D object that did not match were repeated about 20 times through the optimization algorithm, It can be seen that they are almost identical.

8A and 8B show examples of application according to an embodiment of the present invention.

In more detail, Fig. 8A shows an example arrangement of a 360 streaming camera and a digital twin, and Fig. 8B shows object location recognition and display.

Referring to FIG. 8A , it shows placement of a 360 streaming camera on a digital twin at the same position and angle as in real space. (a) of FIG. 8B shows a perspective view image of video data (image) taken with a 360 camera, and (b) shows the position of an object in the image of the video data. Also, (c) shows that the geographic location of the object finally output through the initial location estimation and optimization process is placed in the digital twin in the form of a box.

Accordingly, the present invention can recognize objects around the 360 camera in real time and visualize it on the 3D map of the digital twin.

The system or apparatus described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

A method for estimating geographic data of a moving object based on a 360 camera and a digital twin, the method comprising:
receiving and pre-processing image data from a 360 camera;
estimating the initial position of the object by extracting the position of the object in the image data and estimating the image depth through a digital twin; and
Based on the initial position, a difference with the object in the image data is calculated using a 3D model disposed in the digital twin, and the position and angle of the object through optimization to minimize the difference estimating geographic data comprising
A method of estimating geographic data of a moving object comprising a. According to claim 1,
The pre-processing step is
A method for estimating geographic data of a moving object by preprocessing the image data into a perspective view having a specific viewing angle. According to claim 1,
The step of estimating the initial location and the step of estimating the geographic data include
A method for estimating geographic data of a moving object, comprising restoring the object in three dimensions using a three-dimensional object retrieval technique to perform 6DoF pose estimation. According to claim 1,
The step of estimating the initial position is
In the virtual space of the digital twin constructed based on a street view and a 3D model, the pixel coordinates for the position of the object in the image data are extracted using an instance segmentation algorithm. A method for estimating geographic data of a moving object, obtaining the image depth using a camera posture and ground information. 5. The method of claim 4,
The step of estimating the initial position is
estimating the initial position, which is a three-dimensional position of the object, using the pixel coordinates of the object and the depth value of the pixel, and arranging the 3D model in the digital twin, A method of estimating geographic data of a moving object. According to claim 1,
The step of estimating the geographic data is
Calculate the difference between the object outline of the estimated data generated by projecting the 3D object of the 3D model to the initial position by using a Particle Swarm Optimization (PSO) algorithm and the object outline in the image data A method for estimating geographic data of a moving object, performing optimization to minimize errors. 7. The method of claim 6,
The step of estimating the geographic data is
A 6DoF pose including the position and angle of the object in the image data by performing an optimization process by iteration that changes the position and rotation of the 3D object using the PSO algorithm ), a method of estimating geographic data of a moving object. A system for estimating geographic data of a moving object based on a 360 camera and a digital twin, the system comprising:
a pre-processing unit for receiving and pre-processing image data from a 360 camera;
an initial position estimator for extracting a position of an object in the image data and estimating an initial position of the object by estimating an image depth through a digital twin; and
Based on the initial position, a difference with the object in the image data is calculated using a 3D model disposed in the digital twin, and the position and angle of the object through optimization to minimize the difference A geographic data estimator for estimating geographic data comprising
Geographic data estimation system of a moving object comprising a. 9. The method of claim 8,
The preprocessor
A system for estimating geographic data of a moving object, which preprocesses the image data into a perspective view having a specific viewing angle. 9. The method of claim 8,
The initial location estimator and the geographic data estimator
A system for estimating geographic data of a moving object, wherein the object is restored in three dimensions using a 3D object retrieval technique to perform 6DoF pose estimation. 9. The method of claim 8,
The initial position estimating unit
In the virtual space of the digital twin constructed based on a street view and a 3D model, the pixel coordinates for the position of the object in the image data are extracted using an instance segmentation algorithm. A system for estimating geographic data of a moving object, which acquires the image depth using camera posture and ground information. 12. The method of claim 11,
The initial position estimating unit
estimating the initial position, which is a three-dimensional position of the object, using the pixel coordinates of the object and the depth value of the pixel, and arranging the 3D model in the digital twin, Geographic data estimation system for moving objects. 9. The method of claim 8,
The geographic data estimating unit
Calculate the difference between the object outline of the estimated data generated by projecting the 3D object of the 3D model to the initial position by using a Particle Swarm Optimization (PSO) algorithm and the object outline in the image data A system for estimating geographic data of a moving object that performs optimization to minimize errors. 14. The method of claim 13,
The geographic data estimating unit
A 6DoF pose including the position and angle of the object in the image data by performing an optimization process by iteration that changes the position and rotation of the 3D object using the PSO algorithm ), a system for estimating geographic data of a moving object.

Images