KR102617222B1 - Auto topology mapping method based on omni-directional image and system thereof - Google Patents

Abstract

An automatic phase mapping processing method and system are disclosed.
The automatic phase mapping processing method includes the steps of the automatic phase mapping processing system acquiring a plurality of images - at least two of the plurality of images have a common area in which a common space is photographed, the automatic phase mapping processing system Extracting features of each of the images from each of the images through a feature extractor using a neural network, and the automatic phase mapping processing system mapping each of the images based on the features extracted from each of the images. It includes the step of judging the image.

Description

Automatic topology mapping method based on omni-directional image and system thereof}

The present invention relates to a method and system that can effectively map or connect (or match) the positional relationship between a plurality of images taken at different locations.

Techniques for phase mapping between different images are widely known. Phase mapping may be a technology for determining the relative positional relationship between different images or connecting (or matching) images.

In general, in order to connect different images, feature points are detected from each of the two images that contain a common space, and a transformation function (transformation matrix) is used to overlap the detected feature points to minimize errors. A method of converting and connecting images is used.

In addition, even if different images are not connected (matched), matching points that exist in each of the two images are used to determine the positional relationship between the two images (e.g., points in two different images that correspond to the same point in space). Technical ideas using (e.g., epipolar geometry, etc.) have been known.

However, there may be cases where multiple images exist and it is not known which of these images should be connected to each other or which images can be mapped to each other (e.g., images containing the same space). . That is, this may be the case when the location and/or direction of each image is not known.

For example, when images (e.g., 360-degree images) are taken at a plurality of different locations in an indoor space, various services such as navigation of the indoor space can be smoothly performed by specifying the positional relationship of each image. However, if the location where each image was taken is not known, it is impossible to know which images can be mapped to each other.

In this case, it must be determined whether each combination of images can be mapped to each other. However, these tasks can take a lot of resources and time. For example, if there are five different images, in order to find an image pair that can be mapped to each other, the first image can be set as a pair with the second to fifth images, respectively, to determine whether the images can be mapped to each other. . For example, the image in which the most common feature points with the first image are found may be the image mapped to the first image. This is because images that are mapped to each other exist in areas where a common space is captured, and identical feature points can be found in areas where the common space is captured.

Only by performing this task for each image pair can the phase relationship between images be determined, and then mapping can be performed between adjacent images. In this specification, mapping may mean matching two images if they can be connected (matched), but there is no need to connect (matched) images like images taken at two different locations. In this case, it can be defined to include understanding the relative positional relationship between two images.

As the number of images increases, the cost required to identify images that can be mapped and to specify the positional relationships between the identified images increases exponentially.

Therefore, the technical task to be achieved by the present invention is to provide a method and system that can quickly and effectively determine the mapping between a plurality of images.

The automatic phase mapping processing method for achieving the above technical problem includes the steps of the automatic phase mapping processing system acquiring a plurality of images - at least two of the plurality of images have a common area in which a common space is photographed; The automatic phase mapping processing system extracting features of each of the images from each of the images through a feature extractor using a neural network, the automatic phase mapping processing system extracting features from each of the images based on the features extracted from each of the images. It includes determining a mapping image for each of the images.

The step of the automatic phase mapping processing system determining a mapping image for each of the images based on the features extracted from each of the images includes constructing vectors corresponding to the features extracted from each of the images into a DB. A step of vector searching the constructed DB for a vector set that is at least some of the first vectors corresponding to first features extracted from a predetermined first image among the images, and extracting the vector set based on the vector search results. It may include determining the second image to be a mapping image of the first image.

The automatic phase mapping processing method may further include performing mapping of the first image and the second image determined to be a mapping image of the first image.

The step of performing mapping of the first image and the second image determined to be a mapping image of the first image includes the first image on the first image corresponding to each of the first features extracted from the first image. Determining feature corresponding positions and second feature corresponding positions on the second image that respectively correspond to second features extracted from the second image, the determined first feature corresponding positions and the second feature corresponding positions. It may include determining a relative positional relationship between the first image and the second image based on the positions.

The neural network is a neural network trained to divide a predetermined image so that there is an overlapping area, and then output a transformation relationship that can optimally match the corresponding points extracted from the overlapping common area of each of the divided images. It can be.

Each of the above images may be a 360-degree video taken from different locations indoors.

An automatic phase mapping processing method according to another embodiment includes the steps of an automatic phase mapping processing system acquiring a plurality of images - at least two of the plurality of images have a common area in which a common space is photographed, the automatic phase mapping processing system A phase mapping processing system extracting features from each of the images, constructing vectors corresponding to the features extracted from each of the images in a DB, and extracting a first image from a predetermined first image among the images. Performing a vector search in the constructed DB for a vector set that is at least some of the first vectors corresponding to features, determining a second image extracted based on the vector search results as a mapping image corresponding to the first image. May include steps.

The above method can be implemented by a computer program installed in a data processing device and stored in a recording medium.

An automatic phase mapping processing system according to an embodiment includes a processor and a memory storing a program implemented by the processor, wherein the program stores a plurality of images - at least two of the plurality of images are photographed in a common space. A common region exists- is acquired, features of each of the images are extracted from each of the images through a feature extractor using a neural network, and each of the images is extracted based on the features extracted from each of the images. Determine the mapping image.

The processor constructs a database of vectors corresponding to the features extracted from each of the images, and creates a vector set that is at least some of the first vectors corresponding to the first features extracted from a predetermined first image among the images. A vector search is performed on the constructed DB, and the second image extracted based on the vector search results can be determined to be a mapping image of the first image.

The processor determines first feature corresponding positions on the first image, which respectively correspond to first features extracted from the first image, and the second features corresponding to each of the second features extracted from the second image. Second feature corresponding positions on the image may be determined, and a relative positional relationship between the first image and the second image may be determined based on the determined first feature corresponding positions and the second feature corresponding positions.

An automatic phase mapping processing system according to another embodiment includes a processor and a memory storing a program implemented by the processor, wherein the program captures a plurality of images - at least two of the plurality of images have a common space. A common region exists - to obtain, extract features from each of the images, construct vectors corresponding to the features extracted from each of the images into a DB, and extract from a predetermined first image among the images. A vector set that is at least some of the first vectors corresponding to the extracted first features is searched in the constructed DB, and the second image extracted based on the vector search result is a mapping image corresponding to the first image. It can be judged as follows.

According to the present invention, it is possible to extract features in a short time using image features using a neural network, and by utilizing this, it is possible to effectively identify images that can be mapped to each other when a plurality of images exist using a vector search method. There is a possible effect.

In addition, even after identifying images that can be mapped, effective mapping can be performed by using features used in a neural network.

In order to more fully understand the drawings cited in the detailed description of the present invention, a brief description of each drawing is provided.
1 is a diagram illustrating a schematic configuration for implementing an automatic phase mapping processing method according to an embodiment of the present invention.
Figure 2 is a diagram showing the schematic configuration of an automatic phase mapping processing system according to an embodiment of the present invention.
Figure 3 is a diagram schematically illustrating the logical configuration of an automatic phase mapping processing system according to an embodiment of the present invention.
Figure 4 is a diagram for explaining the concept of using features of a neural network for an automatic phase mapping processing method according to an embodiment of the present invention.
Figure 5 is a diagram to explain the advantages of using a neural network feature according to an embodiment of the present invention.
Figure 6 is a diagram for explaining feature positions corresponding to neural network features according to an embodiment of the present invention.
Figure 7 is a flow chart to explain a method of searching for mapping images between images in the automatic phase mapping processing method according to an embodiment of the present invention.
Figure 8 is a flow chart to explain a method for mapping images in the automatic phase mapping processing method according to an embodiment of the present invention.

In order to fully understand the present invention, its operational advantages, and the objectives achieved by practicing the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Additionally, in this specification, when one component 'transmits' data to another component, the component may transmit the data directly to the other component, or through at least one other component. This means that the data can be transmitted to the other components.

Conversely, when one component 'directly transmits' data to another component, it means that the data is transmitted from the component to the other component without going through the other component.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings. The same reference numerals in each drawing indicate the same member.

1 is a diagram illustrating a schematic configuration for implementing an automatic phase mapping processing method according to an embodiment of the present invention. Additionally, Figure 2 is a diagram showing the schematic configuration of an automatic phase mapping processing system according to an embodiment of the present invention.

In order to implement an automatic phase mapping processing method according to the technical idea of the present invention, an automatic phase mapping processing system 100 is provided.

The automatic phase mapping processing system 100 may be provided with a memory 2 in which a program for implementing the technical idea of the present invention is stored, and a processor 1 for executing the program stored in the memory 2. .

An average expert in the technical field of the present invention can easily deduce that the processor 1 may be named by various names such as CPU, mobile processor, etc., depending on the implementation example of the automatic phase mapping processing system 100. .

The memory 2 stores the program and may be implemented as any type of storage device that can be accessed by the processor to run the program. Additionally, depending on the hardware implementation example, the memory 2 may be implemented not as a single storage device but as a plurality of storage devices. Additionally, the memory 2 may include not only a main memory but also a temporary memory. It may also be implemented as volatile memory or non-volatile memory, and can be defined to include all types of information storage means implemented so that the program can be stored and driven by the processor.

The automatic phase mapping processing system 100 may be implemented in various ways, such as a web server, computer, mobile phone, tablet, TV, or set-top box, depending on the embodiment, and may be implemented in any form that can perform the functions defined in this specification. It can be defined to include data processing devices.

Additionally, depending on the embodiment of the automatic phase mapping processing system 100, various peripheral devices 3 may be further provided. For example, an average expert in the field of the present invention can easily deduce that a keyboard, monitor, graphics card, communication device, etc. may be further included in the automatic phase mapping processing system 100 as peripheral devices.

The automatic phase mapping processing system 100 according to the technical idea of the present invention can identify images that can be mapped to each other, that is, mapping images, among a plurality of images. Additionally, depending on the embodiment, the automatic phase mapping processing system 100 may perform mapping between identified mapping images.

Mapping images may refer to images that have the closest phase relationship to each other. The closest phase relationship may be one in which the images are not only close in distance but must also be able to move directly to each other in space, and examples of this may be images that contain the most common space.

Additionally, performing mapping may mean matching between two images as described above, but in the present invention, the description will focus on the case of determining the phase, that is, the relative positional relationship, of the two images.

For example, as shown in FIG. 1, the automatic phase mapping processing system 100 may receive a plurality of images (eg, five). Then, the automatic phase mapping processing system 100 can determine which images, that is, mapping images, can be mapped to each other among the plurality of images, and perform mapping of the identified mapping images.

For example, in an embodiment of the present invention, images may be omnidirectional images (360 images) taken at different locations. And the mapping images may be pairs of images that share the most common space with each other.

For example, as shown in FIG. 1B, images taken at locations a, b, c, d, and e may be image 1, image 2, image 3, image 4, and image 5, respectively.

In this case, Image 1, Image 2, and Image 3 contain a significant amount of common space within the common captured image, but Image 1 and Image 2 may contain relatively more common space. Therefore, the mapping image of image 1 may be image 2.

Then, the mapping image must be searched for image 2, and at this time, image 1, for which the mapping image has already been confirmed, can be excluded. Then the mapping image of image 2 can be image 3.

In this way, the mapping image of image 3 can be image 4, and the mapping image of image 4 can be image 5.

Then, the automatic phase mapping processing system 100 can perform mapping on image 2, which is a mapping image, based on image 1. In other words, the relative position of image 2 with respect to image 1 and the relative position of image 2 with respect to image 1 can be determined. Additionally, the phase relationship between all images can be specified by sequentially determining the phase of image 3 with respect to image 2, the phase of image 4 with respect to image 3, and the phase of image 5 with respect to image 4.

Ultimately, conventionally, when there are a plurality of omnidirectional images and the exact location of each omnidirectional image is not known, considerable time and resources may be required to determine the positional relationship between the plurality of images.

For example, according to the conventional method, a predetermined feature point must be extracted for each image, and the extracted feature points must be used to determine how many common feature points exist for each pair of images. And image pairs with the most common feature points can be identified as mapping images, and mapping, that is, relative positional relationship, can be determined according to the positions of the common feature points. If registration is necessary, a transformation matrix is determined to overlap common feature points with minimal error, and the two images can be connected (registered) by transforming one image through this transformation matrix.

However, the feature points used in this conventional method require considerable time and computational effort to extract the feature points. Additionally, in order to identify the mapping image, an operation that compares feature points for each image pair must be performed, but as the number of feature points in the images increases, this operation takes a significant amount of time.

However, as described above, according to the technical idea of the present invention, it is possible to quickly and accurately automatically search for mapping images among a plurality of images and perform mapping on the searched mapping images.

To solve this problem, the automatic phase mapping processing system 100 according to the technical idea of the present invention can use neural network features.

Neural network features defined in this specification may refer to all or part of the features selected from the feature map of a certain layer of a neural network learned to achieve a certain purpose.

These features are used in a neural network (eg, a convolutional neural network) learned to achieve a specific purpose, and may be information derived from the learned neural network when the neural network is trained to achieve the specific purpose.

For example, a neural network 20 as shown in FIG. 4 may exist, and the neural network may be a convolutional neural network (CNN).

In this case, a plurality of layers 21, 22, 23, 24 may be included in the neural network 20, and an input layer 21, an output layer 24, and a plurality of hidden layers 22, 23 may be included in the neural network 20. It can exist. The output layer 24 may be a layer fully connected to the previous layer, and the automatic phase mapping processing system 100 according to the technical idea of the present invention may be used before the output layer 24 or the fully connected layer. The neural network features (f1, f2, f3) can be selected from a layer (eg, 23) containing a random feature map of .

The neural network features (f1, f2, f3) used by the automatic phase mapping processing system 100 may be all features included in the feature map of the corresponding layer, or may be some selected features among them.

The automatic phase mapping processing system 100 uses these features instead of conventional hand-crafted feature points, such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Oriented FAST and Rotated BRIEF (ORB). This can be used to identify mapping images or perform mapping between mapping images. That is, features used in a convolutional neural network can be used instead of conventional hand-crafted features.

It is desirable that the features of the image have the same characteristics regardless of scale or orientation. In the convolutional neural network, the layer before the output radar 23 includes a plurality of nonlinear convolution functions and/or These features are achieved through pooling functions, etc. Moreover, conventional handcrafted features are extracted only from characteristic locations defined by humans, such as corners in an image, and are usually extracted only from places where edges exist (for example, where edges bend, etc.).

However, there is an advantage that the neural network 20 can be learned so that neural network features can be found not in these locations but also in flat areas of the image. In addition, while handcrafted features often fail to detect feature points even though they should be detected due to image distortion or image quality, neural network features are much more robust to such image distortions, ensuring lower accuracy in feature extraction. Improvement may exist.

The neural network 20 may itself be a feature extractor. For example, when a feature is selected from the output layer 24 or the fully connected previous layer 23, the output layer 24 is designed to output the selected features (f1, f2, f3) of the previous layer 23 itself. This may be the case, and in this case, the neural network 20 itself may operate as a feature extractor.

Alternatively, the neural network 20 may have been learned to achieve a separate unique purpose (eg, clashification, object detecting, etc.). Even in this case, consistent features can always be selected from a given layer and used as neural network features. For example, in the case of FIG. 4, the combination of the remaining layers excluding the output layer 24 can operate as a feature extractor.

According to one embodiment of the present invention, the neural network 20 divides one image so that there is an overlapping area, and then the corresponding points extracted from the overlapping common area of each of the divided images are matched. It may be a neural network that has been trained to derive an optimal transformation relationship (e.g., one that minimizes errors).

For example, as shown in FIG. 4B, all or part of a given image 6 may be divided so that an overlapping common area 6-3 exists. And a predetermined number of points (eg, P11 to P14, P21 to P24) corresponding to each other can be extracted from each of the divided images 6-1 and 6-2.

Then, the points P11 to P14 extracted from the first split image 6-1 can be converted to the points P21 to P24 extracted from the second split image 6-2 with minimum error (e.g. , determining the parameters of the transformation matrix), the learned neural network can be implemented with the neural network 20.

At this time, the points (eg, P11 to P14, P21 to P24) may be arbitrarily selected points or feature points extracted in a predetermined manner from the common area of each image.

In any case, all or part of the well-trained neural network 20 can be used as a feature extractor to select and extract features from the image to achieve a predetermined purpose.

And using this feature extractor, the same feature can be extracted from the common area included in each of the different images input to the automatic phase mapping processing system 100. Therefore, the image in which the same features (corresponding features) exist the most in any one image can be determined to be the mapping image.

Meanwhile, according to the technical idea of the present invention, neural network features are expressed as vectors, so to search for a mapping image of a specific image, rather than comparing features for each image pair as in the past, by using a vector search engine capable of high-speed computation Faster determination of positional relationships may be possible.

Technologies for searching large amounts of vectors at high speed have recently been widely disclosed.

A vector search engine may be an engine built to quickly find vectors that are the closest (closest) to an input vector (or set of vectors). All vectors are indexed and stored in the DB, and the vector search engine can be designed to output a vector (or vector set) that is most similar to the input vector (or vector set).

This vector search engine can be built using known vector search techniques, such as faiss, for example. This vector search engine has the effect of enabling large-capacity, high-speed calculations when performed on a GPU basis.

A vector search engine according to the technical idea of the present invention can receive a set of features extracted from a target image (eg, image 1) and output the most similar (close range) vector or set of vectors in response. And by determining which image is the source of this vector or set of vectors, the mapping image of the target image can be determined at high speed.

For example, all features extracted from the first image may be input to the vector search engine. The vector search engine can output the vector with the shortest distance or the distance from the vector with the shortest distance to each of the features input from the vector DB. This task can be performed on an image-by-image basis.

For example, assuming that there are five images and 10 features are extracted for each image, 50 vectors can be indexed and stored in the vector DB. And information about each source image can be stored together.

Then, the vector search engine can receive 10 vectors extracted from the first image. And the vector search engine can output each of the 10 vectors and the 10 vectors with the shortest distances among the vectors extracted from the second image, or the sum of their distances. If the vectors extracted from the third image, the vectors extracted from the fourth image, and the vectors extracted from the fifth image are performed in this way, the image containing the feature sets most similar to the input vector set is searched at high speed. It can be. And the searched image may be determined to be a mapping image of the first image.

Depending on the embodiment, the vector search engine determines the shortest distance for all of the remaining vectors (40) excluding the 10 vectors extracted from the first image for each of the 10 vectors output from the first image. It can be output in vector order. For example, when a list of 10 vectors is output, the automatic phase mapping processing system 100 can analyze the vector list and output a mapping image.

The results or methods output by a vector search engine may vary. However, in any case, according to the technical idea of the present invention, features can be extracted from each of the input images, and these features can be input into a DB built to enable vector search, and the vector search engine can select the input vector or vector set. Once input is received, it can perform a function that outputs the most similar (shortest distance) vector or set of vectors. These features enable high-speed navigation of mapping images.

Depending on the embodiment, not all features of the target image, that is, the image for which the mapping image is to be found (eg, the first image), may be input, but some features may be input to the vector search engine. For example, only features corresponding to a predefined area in an image may be input to a vector search engine to determine positional relationships. Since the predefined area is usually not the center of the image but the area adjacent to the left, right, top and bottom corners, the outer area of the image is set arbitrarily, and features at locations corresponding to the set area are selectively used as input for vector search. It could be. Of course, in the vector DB, only the features corresponding to these outer areas may be input, or all features may be input.

Additionally, the location of the neural network feature in accordance with the technical idea of the present invention in the extracted image is not specified. Therefore, mapping can be performed only when the location (point) in the original image corresponding to the neural network feature is specified. Therefore, a technical idea for specifying the location on the original image corresponding to the neural network feature is required, which will be described later with reference to FIG. 6.

The automatic phase mapping processing system 100 for implementing the technical idea described above may be defined as a functional or logical configuration as shown in FIG. 3.

Figure 3 is a diagram schematically illustrating the logical configuration of an automatic phase mapping processing system according to an embodiment of the present invention.

Referring to FIG. 3, the automatic phase mapping processing system 100 according to the technical idea of the present invention includes a control module 110, an interface module 120, and a feature extractor 130. The automatic phase mapping processing system 100 may further include a mapping module 140 and/or a vector search engine 150.

The automatic phase mapping processing system 100 may refer to a logical configuration equipped with hardware resources and/or software necessary to implement the technical idea of the present invention, and necessarily refers to one physical component. It does not mean a single device. In other words, the automatic phase mapping processing system 100 may mean a logical combination of hardware and/or software provided to implement the technical idea of the present invention, and if necessary, may be installed in devices spaced apart from each other to It may be implemented as a set of logical configurations to implement the technical idea of the present invention by performing the function. Additionally, the automatic phase mapping processing system 100 may refer to a set of components implemented separately for each function or role to implement the technical idea of the present invention. For example, the control module 110, interface module 120, feature extractor 130, mapping module 140, and/or vector search engine 150 may each be located in different physical devices, or may be located in the same physical device. It may also be located on the device. In addition, depending on the implementation, software and/or hardware constituting each of the control module 110, interface module 120, feature extractor 130, mapping module 140, and/or vector search engine 150 The combination is also located in different physical devices, and components located in different physical devices may be organically combined to implement each of the modules.

Additionally, in this specification, a module may mean a functional and structural combination of hardware for carrying out the technical idea of the present invention and software for driving the hardware. For example, the module may mean a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and does not necessarily mean a physically connected code or a single type of hardware. can be easily inferred by an average expert in the technical field of the present invention.

The control module 110 includes other components included in the automatic phase mapping processing system 100 (e.g., interface module 120, feature extractor 130, mapping module 140, and/or the vector search engine 150, etc.) can be controlled.

The interface module 120 can receive a plurality of images from the outside. The plurality of images may be images taken at different locations. According to one example, the plurality of images may be 360 images captured indoors, but the image is not limited thereto.

Among the plurality of images, there may be images of a common space taken from different locations, and two images including a common space, that is, a common area, may be defined as having a mappable relationship. Among them, the image containing the most common areas can be defined as the mapping image, which can also be defined as the images with the most corresponding features.

The feature extractor 130 can extract a feature defined according to the technical idea of the present invention, that is, a neural network feature, from each of the plurality of images input through the interface module 120.

As described above, neural network features may be image features that are specified before the output layer in a certain neural network (eg, CNN). The feature extractor 130 may be the neural network 20 itself as shown in FIG. 4, and operates from the input layer 21 to a predetermined layer (e.g., 23) before the output layer 24 in the neural network. It may mean the composition of . All or part of the features included in the feature map defined by the layer 23 may be neural network features.

The neural network 20 may be trained for a separate purpose (e.g., classification, detection, etc.) other than the purpose of extracting neural network features, but as described above, it is used to match two images with minimal error. It may be a neural network designed for this purpose, or it may be learned for the purpose of extracting neural network features.

For example, in the latter case, it can be learned to output hand-crafted feature points that can well express the characteristics of the location and/or image arbitrarily set by the user, and in this case, the neural network 20 itself is a feature extractor. It could be (130).

The position arbitrarily set by the user can be set to the position set by the user (eg, the central position of the object) on a predetermined object (eg, a wall, a door, etc.). Additionally, unlike conventional hand-crafted feature points, this user-set position can be set in a flat area, that is, a flat image area without edges or corners. In this case, features can be defined even within a flat image area that is not extracted as a feature point in conventional handcrafted feature points, and when this is used, more accurate mapping image judgment and mapping can be performed.

Figure 5 is a diagram to explain the advantages of using a neural network feature according to an embodiment of the present invention.

As shown in FIG. 5, the feature extractor 130 can be trained so that any location within a predetermined object (e.g., wall, door, table) can be specified as a feature point (fp1, fp2, fp3). .

Additionally, as shown in FIG. 5, the arbitrary position may be set within a typically flat image area, such as a predetermined position for each object (eg, the center of a wall, the center of a table, the center of a door, etc.).

Of course, the feature extractor 130 may be trained to extract features corresponding to handcrafted feature points, such as conventional edges or bent corners.

For example, a user may annotate a number of images with handcrafted feature points for each object and the setting positions of a flat area set by the user, and use these as learning data to train the neural network 20. In this case, features corresponding to each feature point (fp1, p2, fp3) may be extracted, and the feature point itself may be output.

In any case, when using neural network features, as shown in FIG. 5, positions that cannot be extracted with conventional handcrafted features can be used as features, which can have an advantageous effect in defining image characteristics or mapping images. You can.

On the other hand, although the neural network feature is characteristic information of an image that is determined through multiple convolutions and/or pooling to output the desired purpose of the neural network 20, these neural network features themselves are located at a specific location in the corresponding original image. It may not represent .

Therefore, even when a neural network feature is extracted, the location on the original image corresponding to the neural network feature, that is, the feature location, needs to be specified. This is because mapping of the image can be performed only when these feature locations are specified.

As such, the technical idea for specifying the feature location of the neural network feature will be explained with reference to FIG. 6.

Figure 6 is a diagram for explaining feature positions corresponding to neural network features according to an embodiment of the present invention.

As shown in FIG. 6, a neural network feature (f) can be extracted from a predetermined layer. In this case, the neural network feature (f) corresponds to a predetermined corresponding area (Sl) in the previous predetermined layer (l), and the pixel information included in this corresponding area (Sl) is converted to a predefined convolution and pooling function. It can be mapped to the neural network feature (f).

At this time, a predetermined position (e.g., center or specific vertex, etc.) within the corresponding area (Sl) of the neural network feature (f) in the l layer is the corresponding position (PSl) in the l layer of the neural network feature (f). It can be defined as:

Then, in the same way, the corresponding area (So) on the original image corresponding to the corresponding position (PSl) in the l layer can be specified by the convolution and pooling relationship between the original image and the l layer, and the corresponding area (So) ) may be specified as a corresponding position (eg, the center) on the original image of the neural network feature (f), that is, a feature position.

If the feature location is determined for each neural network feature in this way, each feature location can become a feature point for image mapping.

Then, the mapping module 140 can perform image mapping using the corresponding feature positions between the mapping images.

Image mapping between two images may be performed using corresponding points in each of the two images, in the case of mapping that specifies the relative positional relationship between the two images. At this time, the corresponding points may be feature points of neural network features extracted from each of the two images, and the corresponding feature points can be easily searched through a vector search engine.

When corresponding points (representing the same position in space) exist in different images, technical ideas for specifying the relative positional relationship between these two images have been known.

For example, it can be easily inferred by an average expert in the technical field of the present invention that a relative positional relationship can be determined using epipolar geometry. In addition, various methods may be possible.

According to another embodiment, when the mapping between two images, that is, mapping images, matches the two images, the mapping may be performed by specifying a transformation matrix for matching the two images.

It is widely known that in order to specify such a transformation matrix, three pairs of corresponding features can be extracted and a transformation matrix defined so that the extracted three pairs can be transformed. Thus, these three feature pairs can be searched so that all features can be converted with the smallest error, and of course, algorithms such as RANSAC can be used.

As described above, the vector search engine 150 inputs vectors corresponding to the features of each image extracted by the feature extractor 130 into the DB, and extracts the features extracted from the target image (e.g., the first image). You can receive a vector set corresponding to the set as input. Then, the vexer search results can be output as described above.

Then, the control module 110 may determine an image existing in a positional relationship adjacent to the target image based on the vector search result.

Figure 7 is a flow chart to explain a method of searching for mapping images between images in the automatic phase mapping processing method according to an embodiment of the present invention.

Referring to FIG. 7, the automatic phase mapping processing system 100 according to the technical idea of the present invention can extract neural network features from each of a plurality of images (S100). Then, the features can be built into a vector DB and a vector search can be performed on the vector set (feature set) extracted from the target image (S110, S120).

Then, the mapping image of the target image can be determined based on the vector search results (S130), and the mapping image of each image can be determined by performing the same task for all images (S140).

Figure 8 is a flow chart to explain a method of mapping images in the automatic phase mapping processing method according to an embodiment of the present invention.

Referring to FIG. 8, the automatic phase mapping processing system 100 according to the technical idea of the present invention uses features corresponding to features extracted from the first image to map the first image and the second image, which are determined to be mapping images. Feature positions can be specified (S200). For this purpose, the method shown in FIG. 6 can be used.

Additionally, feature positions corresponding to features extracted from the second image can be specified (S210).

Then, the automatic phase mapping processing system 100 determines the relative positional relationship through the Epipolar Geometry algorithm based on the feature positions of each image, or uses a transformation matrix for image connection using a predetermined method (e.g., RANSAC algorithm). It can be decided through (S220).

The automatic phase mapping processing method according to an embodiment of the present invention can be implemented as a computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, hard disk, floppy disk, and optical data storage devices. Additionally, computer-readable recording media can be distributed across computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner. And functional programs, codes, and code segments for implementing the present invention can be easily deduced by programmers in the technical field to which the present invention pertains.

The present invention has been described with reference to an embodiment shown in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached registration claims.

Claims (12)

An automatic phase mapping processing system selects a plurality of images - one of the plurality of images
Obtaining that there is a common area in which two common spaces are photographed;
The automatic phase mapping processing system extracts features from each of the images.
steps;
Construct vectors corresponding to the features extracted from each of the above images into a DB
steps;
Performing a vector search in the constructed DB for a vector set that is at least some of the first vectors corresponding to first features extracted from a predetermined first image among the images - vectors included in the DB by the vector search Among them, the set of nearest vectors is specified -;
An automatic phase mapping processing method comprising determining a second image corresponding to the nearest vector set specified based on the vector search result as a mapping image of the first image.
delete delete The method of claim 1, wherein the step of mapping the first image and the second image determined to be a mapping image of the first image comprises:
First feature corresponding positions on the first image corresponding to each of the first features extracted from the first image, and first feature corresponding positions on the second image corresponding to each of the second features extracted from the second image. 2. Determining feature corresponding positions;
An automatic phase mapping processing method comprising determining a relative positional relationship between the first image and the second image based on the determined first feature corresponding positions and the second feature corresponding positions.
The method of claim 1, wherein the automatic phase mapping processing system,
An automatic system that includes a neural network trained to divide a given image so that there is an overlapping area, and then output a transformation relationship that can optimally match the corresponding points extracted from the overlapping common area of each of the divided images. Phase mapping processing method.
The method of claim 1, wherein each of the images is:
An automatic phase mapping processing method characterized by 360-degree images taken at different locations indoors.
delete A computer program installed in a data processing device and stored in a recording medium for performing the method according to any one of claims 1, 4 to 6.
processor;
Includes a memory storing a program driven by the processor,
The processor runs the program,
Acquire a plurality of images - at least two of the plurality of images have a common area in which a common space is captured - extract features from each of the images, and add features extracted from each of the images. Construct corresponding vectors into DB,
Performing a vector search in the constructed DB for a vector set that is at least some of the first vectors corresponding to first features extracted from a predetermined first image among the images - among the vectors included in the DB through the vector search The nearest vector set is specified - an automatic phase mapping processing system that determines a second image corresponding to the nearest vector set specified based on the vector search result as a mapping image of the first image.


delete delete delete