WO2023068795A1

WO2023068795A1 - Device and method for creating metaverse using image analysis

Info

Publication number: WO2023068795A1
Application number: PCT/KR2022/015931
Authority: WO
Inventors: 오우진
Original assignee: 주식회사 제이어스
Priority date: 2021-10-22
Filing date: 2022-10-19
Publication date: 2023-04-27
Also published as: KR102402170B1

Abstract

A device for creating a metaverse using image analysis according to one embodiment of the present invention comprises: a collection unit for collecting target data including photo data related to a space to create a metaverse that implements a virtual space; a first inference engine which collects the collected target data, analyzes and extracts meta information included in each target data, and includes, as the meta information, location information as well as object information and shape information obtained by extracting one or more objects included in an image of the target data based on a neural network learning-based object recognition algorithm; a library which classifies the meta information extracted from the first inference engine in association with corresponding target data as well as the target data, and save same; and a metaverse creation unit which collects meta information data extracted, classified, and saved from the library by place, and creates a metaverse which is a virtual space for a corresponding place.

Description

Apparatus and method for generating metaverse using image analysis

The present invention relates to an apparatus and method for generating a metaverse, and more particularly, to an apparatus and method for generating a metaverse using image analysis.

Recently, map services provided online have been continuously developed through image processing technology such as aerial photographs and street views in 2D (landmark, satellite).

In addition, in the case of symbolic buildings provided through car navigation systems, etc., they are converted into 3D form and processed, but are only provided on a small display due to problems of quality and high resolution, so there is a problem of lower utilization value than 2D form.

On the other hand, non-face-to-face online services using the metaverse (virtual world that transcends reality), which has recently been in the limelight, are attracting more attention as the non-face-to-face life becomes longer due to the re-proliferation of Corona 19 and the desire to expand reality into the virtual world.

Metaverse is a combination of 'Meta' meaning transcendence and 'Universe' meaning world.

As a specific example of using the metaverse, universities held entrance exam briefings, welcome parties for new students, and university festivals using the metaverse, and are recently building a library.

As another example, companies are holding new employee recruitment briefings or social contribution programs on Metaverse, and even in the political world, presidential candidates have set up campaign venues on Naver Z's metaverse platform 'Zepetto' to capture the so-called 'MZ Generation' votes. there is.

A commonly thought metaverse is a game using an avatar or a non-face-to-face meeting in a virtual space.

However, depending on how the metaverse is used, rather than limited to a specific industry, it may bring innovation to society as a whole, including education, medical care, and entertainment.

Therefore, in order to solve the above-mentioned problems, research on metaverse generation devices and methods using image analysis that can be utilized in various aspects of society has become necessary.

An object of the present invention is to collect target data related to a space to create a metaverse embodying a virtual space, analyze and extract meta information included in the target data, and generate a metaverse based on the meta information, but use an inference engine. Metaverse generation device using image analysis that can be modified and supplemented to enable accurate and sophisticated creation of metaverse space by learning about the connectivity of metaverse space and the accuracy of information included in extracted meta information by utilizing is to provide a way

Objects of the embodiments of the present invention are not limited to the above-mentioned purposes, and other objects not mentioned above will be clearly understood by those skilled in the art from the description below. .

An apparatus for generating a metaverse using image analysis according to an embodiment of the present invention includes a collection unit for collecting target data including photo data related to a space to generate a metaverse implementing a virtual space; The collected target data is collected, meta information included in each target data is analyzed and extracted, and at least one or more objects included in the image of the target data are detected based on a neural network learning-based object recognition algorithm along with location information as meta information. A first inference engine including extracted object information and shape information; a library for classifying and storing meta-information including location information, object information, and shape information extracted from the first inference engine along with the target data in association with corresponding target data; and a metaverse generating unit that collects meta information data extracted and classified from the library and stored for each place to generate a metaverse, which is a virtual space for the corresponding place.

The meta information is characterized in that it further includes text information and time information.

The library classifies the meta information analyzed and extracted from the first inference engine by element, and separates and stores the meta information in association with the target data, but can be stored separately by place using the location information or by time using the time information. In addition, it is characterized in that tagging or ID for identification is given in order to assign association with target data and classify by place or time.

The metaverse generating device using the image analysis is a second device that performs neural network learning and 3D conversion calculation to generate a 3D metaverse space based on target data in 2D form and meta information extracted from the target data. It further includes an inference engine.

The second inference engine can perform learning and calculations for correcting and supplementing the generated metaverse space, and learning about the connectivity of the metaverse space and the accuracy of information included in the extracted meta information, It is characterized by correcting and supplementing to enable accurate and sophisticated creation of the metaverse space as more target data is collected and repeatedly learned.

The metaverse generated by the metaverse generator further includes a database that is classified and stored by location using location information or classified and stored by time using time information.

In the metaverse generating method using the metaverse generating device using the image analysis according to an embodiment of the present invention, the collection unit collecting target data for generating a metaverse that is a 3D virtual space; The first inference engine collects collected target data, analyzes and extracts meta information included in each target data, and includes location information as meta information in an image of the target data based on a neural network learning-based object recognition algorithm. including object information extracted from at least one object; classifying and storing, by the library, meta information including location information, object information, and shape information extracted from the first inference engine along with the target data in association with the corresponding target data; and the step of generating the metaverse, which is a virtual space for the corresponding place, by collecting meta information data extracted, classified, and stored from the library by place.

The second inference engine may further include performing neural network training and 3D conversion calculation to generate a 3D metaverse space based on target data in 2D form and meta information extracted from the target data.

The second inference engine can perform learning and calculations for correcting and supplementing the generated metaverse space, and learning about the connectivity of the metaverse space and the accuracy of information included in the extracted meta information, The step of correcting and supplementing to enable accurate and sophisticated creation of the metaverse space as more target data is collected and repeated learning is further included.

The meta information further includes text information and time information, and in the metaverse generation method using the image analysis, the database divides and stores the metaverse generated by the metaverse generator by location and time by utilizing meta information. Include more steps.

The apparatus and method for generating a metaverse using image analysis of the present invention collects target data related to a space to generate a metaverse embodying a virtual space, analyzes and extracts meta information included in the target data, and based on the meta information Create a metaverse, but use an inference engine to learn about the connectivity of the metaverse space and the accuracy of the information included in the extracted meta information, so that the metaverse space can be corrected and supplemented to enable accurate and sophisticated creation. There are advantages.

In addition, metaverse data classified and stored by location and time is compatible with various applications upon request from various demand devices, such as VR devices, mobile devices such as smartphones, tablets, and laptops running map apps, PCs, etc. It can be implemented in, and in particular, since the corresponding place can be implemented and reproduced by time, it can provide functions such as time travel in a virtual space similar to reality implemented by metaverse data.

1 is a diagram showing the configuration concept of a metaverse generating device using image analysis according to an embodiment of the present invention.

2 is a block diagram showing the configuration of a metaverse generating device using image analysis according to an embodiment of the present invention.

Figure 3 is a flow chart for explaining the library creation process of the metaverse generation method using image analysis according to an embodiment of the present invention.

4 is a flowchart illustrating a metaverse generation process of a metaverse generation method using image analysis according to an embodiment of the present invention.

5 is a flowchart showing the entire process of a metaverse generation method using image analysis according to an embodiment of the present invention.

A sophisticated and accurate metaverse can be created when a metaverse is created by a learning-based inference engine.

After collecting photo data, data analysis is performed through neural network learning of the first reasoning engine 200 (S102).

The data analysis process of the first reasoning engine 200 is a process of extracting meta information corresponding to each element included in data, and by extracting meta information, by utilizing the information necessary to create a metaverse space, sophisticated and accurate It can help to create metaverse data, and can also assist in generating metaverse data by time.

The element may be auxiliary information such as photographing time and location (coordinate) information, but may also include object information, which is information about an object included in a photograph image, that is, an object.

By extracting such object information, it is possible to implement a sophisticated metaverse, and one or more neural network-based object recognition algorithms as described above can be utilized in order to well recognize and extract object information.

Next, after extracting elements using an object recognition algorithm, etc., elements can be classified and classified, and then stored in the library 300 after being clearly organized to be used when creating a metaverse (S104).

Next, 3D conversion and metaverse data for a specific place or building are generated through learning of the second inference engine 400 by utilizing the meta information corresponding to the element stored in the library 300 (S106).

At this time, the generated metaverse data can be generated to lead to one virtual space by neural network learning of the second inference engine 400 based on compatibility, connectivity, and sophistication of the place, and the data is divided by coordinates or sections , It can be stored (S108).

In particular, when creating a metaverse, it can be separately classified and created using the photo data and extracted elements collected by the second inference engine 400 for one place over time and stored separately in the database 600.

Furthermore, for data stored in the database 600 to protect and manage stored data, a separate blockchain server (not shown) may be prepared for integrity verification and managed through a blockchain network.

Furthermore, the authentication unit builds a blockchain network in connection with a number of blockchain servers, generates public and private keys through the established internal blockchain network, converts them into hash values, distributes and stores them, and distributes the stored public keys and users. User authentication can be performed based on the personal information of the user.

Furthermore, a plurality of customer terminals (not shown), which are demand devices capable of communicating with the metaverse generating device using image analysis of the present invention, receive individual user information along with a public key and include a hash value for user information. user certificates that can be generated, and the storage method for each user certificate can be made by a Merkle tree structure.

For example, each user certificate (transaction) is stored with a hash value in the lowest child node, and the hash value is stored in the intermediate node on the path leading to the lowest child node in the merkle root (parent node), which is the top level of the merkle tree. It is hashed and stored so that it can be shared.

When determining the authenticity of the stored user certificate through this, the user certificate copied to the individual customer terminal is compared with the user certificate in the database 600, and only the hash value hashed along the path of the Merkle tree is compared. do.

At this time, since the comparison operation is performed on the path of the Merkle tree, it is not necessary to perform the comparison operation on the blocks of all nodes, so it is possible to determine the authenticity with a relatively easy amount of calculation, and forgery of the transaction can be found easily and quickly. , transactions can be easily verified even in a customer terminal in the form of a portable terminal with a small capacity.

Furthermore, even if a plurality of photo data is extracted, an additional neural network learning algorithm may be used in the inference engine to be used for image restoration when the image required for object recognition is not clear due to limitations in camera performance or camera errors.

Specifically, by utilizing generative adversarial networks, a new image can be created or regenerated, so it can be used for restoring a damaged image.

Unlike conventional deep learning networks, generative adversarial networks are structured with multiple deep neural networks and require dozens of times more computation than existing deep neural network models to generate high-resolution images, but can provide excellent performance for image restoration. there is.

In addition, the generative adversarial neural network is an unsupervised learning-based generative model that adversarially trains two networks with a generator and discriminator. Input data is input to the generator to create fake images similar to real images. can be learned to A noise value may be input as the input data. Noise values can follow any probability distribution. For example, it may be data generated with a zero-mean Gaussian.

The discriminator can be trained to discriminate between the real image and the fake image generated by the generator. More specifically, it is possible to learn to have a high probability when a real image is input, and to have a low probability when a fake image is input. That is, the discriminator can gradually learn to discriminate between real images and fake images.

In addition, by using a generative adversarial network (GAN) algorithm to create a fake image, you can implement your own avatar instead of your real photo in the metaverse world, a virtual space. can also be created

Furthermore, among GANs, DC-GAN, which is a deep convolution generative adversarial network, can be used to implement it in the form of a deep fake avatar, and the form of a deep fake can be a form that resembles one's face. Under the premise of the agreement, deepfakes can be made within the permissible line.

DC-GAN needs to construct a Cycle-GAN consisting of two generators and two discriminators, and during the training period, two different sets of numerous images are sent as input, each with two DCGANs, i.e., these two domains as X and represented by Y.

For one of the DCGANs, the input image x is a member of domain X and the noise signal should be similar to the image of domain Y, denote the generator of the GAN of domain X different from the existing domain as G, and the image generated by G as G ( x).

Domain Y is the target domain of this GAN, and likewise for other GANs, the input image is y belonging to domain Y, and the generator (denoted by F) is also an image (F(y), which is difficult to distinguish from the display image), so X is this GAN becomes the target domain of

The discriminator of a GAN with generator G can be expressed as:

Since the goal is to distinguish the fake image G(x) from the real image in domain Y, a GAN that differs as DY can be denoted as DX.

In addition, by introducing cycle consistency loss (Lcyc), if an image is converted from one domain to another, when returning to this domain, it returns to the location where the image was last. This means that it is reasonable to map inputs and outputs, and the periodic coherence loss at this time consists of the following two parts.

One is the forward periodic coherence loss back to domain X via F(G(x)) = x (transforms the input image x from domain X to domain Y via G mapping and back to domain X via mapping). cycle-consistency loss), and the other is called backward cycle-consistency loss corresponding to G(F(y)) = y.

In this specification, a 'terminal' may be a wireless communication device with guaranteed portability and mobility, and may be, for example, any type of handheld-based wireless communication device such as a smart phone, a tablet PC, or a laptop computer. Also, the 'terminal' may be a wired communication device such as a PC capable of accessing other terminals or servers through a network. In addition, a network refers to a connection structure capable of exchanging information between nodes such as terminals and servers, such as a local area network (LAN), a wide area network (WAN), and the Internet (WWW : World Wide Web), wired and wireless data communications network, telephone network, and wired and wireless television communications network.

Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC: Visible Light Communication), LiFi, and the like, but are not limited thereto.

Claims

a collection unit that collects target data including photo data related to a space to create a metaverse that implements a virtual space;

The collected target data is collected, meta information included in each target data is analyzed and extracted, and at least one or more objects included in the image of the target data are detected based on a neural network learning-based object recognition algorithm along with location information as meta information. A first inference engine including extracted object information and shape information;

a library for classifying and storing meta-information including location information, object information, and shape information extracted from the first inference engine along with the target data in association with corresponding target data; and

a metaverse generating unit that collects the stored meta information data extracted and classified from the library by place and creates a metaverse, which is a virtual space for the place;

Metaverse generation device using image analysis containing a.
The method of claim 1,

The meta information

Metaverse generation device using image analysis that further includes text information and time information.
The method of claim 2,

The library

The meta information analyzed and extracted from the first inference engine is classified by element and stored separately in association with the target data, and stored separately by place using the location information or by time using the time information,

A metaverse generating device using image analysis, characterized in that tagging or ID for identification is given in order to associate with target data and classify by place or time.
The method of claim 1,

Based on target data in 2D form and meta information extracted from the target data, neural network learning to generate a metaverse space in 3D form and a second inference engine that performs 3D conversion operation are further included Meta using image analysis bus generator.
The method of claim 4,

The second inference engine

It is possible to perform learning and calculations to correct and supplement the created metaverse space, learn about the connectivity of the metaverse space and the accuracy of the information included in the extracted meta information, and collect a lot of target data. A metaverse generating device using image analysis, characterized in that the more it learns, the more it corrects and supplements to enable accurate and sophisticated creation of the metaverse space.
The method of claim 3,

A database that stores metaverses generated by the metaverse generator by dividing them by location using location information or by time using time information.

Metaverse generation device using image analysis further comprising a.
In the metaverse generation method using the metaverse generation device using image analysis of any one of claims 1 to 6,

collecting target data for generating a metaverse, which is a 3D virtual space;

The first inference engine collects collected target data, analyzes and extracts meta information included in each target data, and includes location information as meta information in an image of the target data based on a neural network learning-based object recognition algorithm. including object information extracted from at least one object;

classifying and storing, by the library, meta information including location information, object information, and shape information extracted from the first inference engine along with the target data in association with the corresponding target data; and

The metaverse generating unit collects the stored meta information data extracted and classified from the library by place and performs the creation of a metaverse, which is a virtual space for the place.

Metaverse generation method using image analysis containing a.
The method of claim 7,

The second inference engine performs neural network learning and 3D conversion calculation to generate a 3D metaverse space based on target data in 2D form and meta information extracted from the target data

Metaverse generation method using image analysis further comprising a.
The method of claim 8,

The second inference engine

It is possible to perform learning and calculations to correct and supplement the created metaverse space, learn about the connectivity of the metaverse space and the accuracy of the information included in the extracted meta information, and collect a lot of target data. Steps of correcting and supplementing to enable accurate and sophisticated creation of the metaverse space as more and more repeated learning is performed

Metaverse generation method using image analysis further comprising a.
The method of claim 8,

The meta information

It further includes text information and time information,

The metaverse generation method using the image analysis is

The database is a method of generating a metaverse using image analysis, further comprising storing the metaverse generated by the metaverse generator by using meta information by location and time.