KR102428941B1 - Method, device and system for generating textured 3d scan data through image registration based on scanning of broadband lidar - Google Patents

Abstract

One embodiment of the present invention relates to a method for generating 3D scanning data textured through image matching based on scanning of a broadband LiDAR, which is performed by an apparatus. The method of the present invention comprises: acquiring first data, or a 3D image generated by scanning a target area to be scanned, from the LiDAR when the area to be scanned is scanned by the LiDAR; acquiring second data, or a 2D image generated by photographing a first object from a camera when the first object located in the target area to be scanned is photographed by the camera; setting a texture of a surface of the first object as a first material based on an image of the first object extracted from the second data when the first object is detected in the first space; and changing the surface of the first object detected in the first space to the first material to generate third data.

Description

Method, device and system for generating textured 3D scan data through wideband lidar scan-based image registration

The following embodiments relate to a technique for generating 3D scan data in which a texture is implemented through image registration based on a scan of a wideband lidar.

Scan data using a 3D scanner is increasingly widely used in the fields of geometric modeling and medical morphology. The technology using 3D scan data can also be applied to a method of checking the results of a process for manufacturing a specific product. 3D scan data may be used to verify that the manufactured product is made as designed.

Most 3D scanners can scan a target object from various angles and generate scan data. At this time, the 3D scanner can generate a continuous set of data by dividing each part of the target object from various angles.

A process of forming a 3D image of a target object based on a set of continuous data generated by the 3D scanner may be referred to as registration.

The registration process may be a process of mapping two or more data sets into one coordinate system. A coordinate system may be assigned to each of the data obtained by scanning or otherwise. Some of the data may be allocated to a fixed system, and the remaining data may be allocated to a moving system. A moving system can be transformed in virtual space to converge on a fixed system. Such transformations may include rotational transformations or translational transformations.

The plurality of scan data for one object is data obtained by scanning different parts of the object, but each of the plurality of scan data may include overlapping parts. The matching process may match the overlapping portions of each of the plurality of scan data, and generate 3D scan data of the object based on the plurality of scan data.

However, in the prior art, in generating 3D scan data of an object, there is a problem in that only the shape of the object is implemented, and even the texture of the surface of the object cannot be implemented.

Accordingly, there is an increasing demand for generating 3D scan data in which the texture of an object is implemented, and research on a related technology is required.

Korean Patent Registration No. 10-0947406 Korean Patent Registration No. 10-2056147 Korean Patent No. 10-2310662 Korean Patent No. 10-1956009

According to an embodiment, when the scan target area is scanned by the lidar, first data that is a 3D image generated by scanning the scan target area from the lidar is acquired, and the first object located in the scan target area is captured by the camera. When the image is captured by the camera, second data that is a 2D image generated by photographing the first object is obtained from the camera, and when the second data is analyzed as an image obtained by photographing the first object, in the first space identified through the first data It is checked whether the first object is detected, and when the first object is detected in the first space, based on the image of the first object extracted from the second data, the texture of the surface of the first object is converted to the first material. A method, apparatus, and method for generating 3D scan data in which a texture is implemented through wideband lidar scan-based image registration, which sets and changes the surface of the first object detected in the first space to the first material to generate third data; and Its purpose is to provide a system.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood from the description below.

According to an embodiment, in a method for generating 3D scan data with texture through image registration based on a scan of a wideband lidar performed by a device, when a scan target region is scanned by the lidar, the acquiring first data that is a 3D image generated by scanning the scan target area from the lidar; acquiring second data, which is a 2D image generated by photographing the first object from the camera, when the first object located in the scan target area is photographed by a camera; when the second data is analyzed as an image obtained by photographing the first object, checking whether the first object is detected in a first space identified through the first data; when the first object is detected in the first space, setting a texture for the surface of the first object as a first material based on the image of the first object extracted from the second data; and changing the surface of the first object detected in the first space to the first material to generate third data, 3D scan data with texture implemented through wideband lidar scan-based image registration A method of generating is provided.

In the method for generating 3D scan data in which a texture is implemented through the wideband lidar scan-based image registration, after acquiring the second data, the first object in the second data is calculating a ratio of an area occupied by a first ratio; when it is determined that the first ratio is greater than a preset reference ratio, checking a photographing target area based on the location information of the camera; obtaining a first list, which is a list of objects disposed in the first area, when the photographing target area is identified as the first area; determining whether the first object is included in the first list by comparing the shape of the objects included in the first list with the shape of the first object identified through the second data; and when it is confirmed that the first object is included in the first list, analyzing the second data as a photographed image of the first object.

The setting of the texture for the surface of the first object may include: generating a first input signal by encoding an image of the first object; inputting the first input signal to an artificial neural network, and obtaining a first output signal as an output of the first input signal; generating a classification result for the surface of the first object based on the first output signal; and setting a texture for the surface of the first object as the first material based on the classification result of the surface of the first object.

According to an embodiment, when the scan target area is scanned by the lidar, first data that is a 3D image generated by scanning the scan target area from the lidar is acquired, and the first object located in the scan target area is captured by the camera. When the image is captured by the camera, second data that is a 2D image generated by photographing the first object is obtained from the camera, and when the second data is analyzed as an image obtained by photographing the first object, in the first space identified through the first data It is checked whether the first object is detected, and when the first object is detected in the first space, based on the image of the first object extracted from the second data, the texture of the surface of the first object is converted to the first material. By setting and changing the surface of the first object detected in the first space to the first material, and generating third data, there is an effect of generating 3D scan data in which the texture of the object is implemented.

On the other hand, the effects according to the embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is a diagram schematically showing the configuration of a system according to an embodiment.
2 is a flowchart illustrating a process of generating 3D scan data in which a texture is implemented through image registration based on a scan of a wideband lidar according to an embodiment.
3 is a flowchart illustrating a process of analyzing which object is an image of which second data is captured, according to an exemplary embodiment.
4 is a flowchart illustrating a process of setting a material for a surface of an object according to an exemplary embodiment.
5 is a flowchart illustrating a process of guiding additional photographing according to an exemplary embodiment.
6 is a flowchart illustrating a process of estimating the size of an object according to an exemplary embodiment.
7 is a diagram for explaining an artificial neural network according to an embodiment.
8 is a diagram for explaining a method of learning an artificial neural network according to an embodiment.
9 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

Terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

The embodiments may be implemented in various types of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices, and the like.

An artificial intelligence (AI) system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns and makes decisions on its own. The more the AI system is used, the better the recognition rate and the more accurate understanding of user preferences, and the existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

Artificial intelligence technology consists of machine learning and element technologies using machine learning. Machine learning is an algorithm technology that categorizes/learns characteristics of input data by itself, and element technology uses machine learning algorithms such as deep learning to simulate functions such as cognition and judgment of the human brain. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question and answer, and speech recognition/synthesis. Visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, and the like. Inferential prediction is a technology for logically reasoning and predicting by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

In general, in order to apply the machine learning algorithm to real life, learning is performed in the Trial and Error method due to the characteristics of the basic methodology of machine learning. In particular, deep learning requires hundreds of thousands of iterations. It is impossible to execute this in the actual physical external environment, so instead, the actual physical external environment is implemented on a computer and learning is performed through simulation.

1 is a diagram schematically showing the configuration of a system according to an embodiment.

Referring to FIG. 1 , a system according to an embodiment may include a lidar 100 , a camera 200 , and a device 300 .

The lidar 100 may scan a wideband location through a wideband Lidar sensor to generate 3D data for a scan area. Here, the 3D data is a 3D image generated by scanning a specific area.

The lidar 100 may be configured to wirelessly communicate with the device 300 by having a communication module.

The camera 200 may photograph a specific object through camera photographing, and may generate 2D data for a photographing target. Here, the 2D data is a 2D image generated by photographing a specific object.

The camera 200 may be configured to wirelessly communicate with the device 300 by having a communication module.

According to an embodiment, the lidar 100 and the camera 200 may be implemented as one integrated device, or may be implemented as separate separate devices.

The device 300 may be a self-server owned by a person or organization providing services using the device 300, a cloud server, or a peer-to-peer (p2p) set of distributed nodes. may be The device 300 may be configured to perform all or part of an arithmetic function, a storage/referencing function, an input/output function, and a control function of a typical computer. The device 300 may include at least one artificial neural network that performs an inference function.

The device 300 may be configured to wirelessly communicate with the lidar 100 and the camera 200 , and may control overall operations of the lidar 100 and the camera 200 , respectively.

The apparatus 300 may generate 3D scan data in which a texture is implemented through image registration based on the scan of the wideband lidar. In this case, the device 300 may generate 3D scan data by matching the 3D image acquired from the lidar 100 and the 2D image acquired from the camera 200 .

The device 300 may classify the surface of an object according to a material based on artificial intelligence, and may include a pre-trained artificial neural network for this purpose.

In the present invention, artificial intelligence (AI) refers to a technology that imitates human learning ability, reasoning ability, perceptual ability, etc., and implements it with a computer, and the concepts of machine learning, symbolic logic, etc. may include Machine Learning (ML) is an algorithm technology that classifies or learns characteristics of input data by itself. Artificial intelligence technology is an algorithm of machine learning that can analyze input data, learn the results of the analysis, and make judgments or predictions based on the results of the learning. In addition, technologies that use machine learning algorithms to simulate functions such as cognition and judgment of the human brain can also be understood as a category of artificial intelligence. For example, technical fields such as verbal comprehension, visual comprehension, reasoning/prediction, knowledge expression, and motion control may be included.

Machine learning may refer to the process of training a neural network model using experience of processing data. With machine learning, computer software could mean improving its own data processing capabilities. The neural network model is constructed by modeling the correlation between data, and the correlation may be expressed by a plurality of parameters. A neural network model extracts and analyzes features from given data to derive correlations between data, and repeating this process to optimize parameters of a neural network model can be called machine learning. For example, a neural network model may learn a mapping (correlation) between an input and an output with respect to data given as an input/output pair. Alternatively, the neural network model may learn the relationship by deriving regularity between the given data even when only input data is given.

The artificial intelligence learning model or neural network model may be designed to implement a human brain structure on a computer, and may include a plurality of network nodes that simulate neurons of a human neural network and have weights. A plurality of network nodes may have a connection relationship with each other by simulating a synaptic activity of a neuron through which a neuron sends and receives a signal through a synapse. In the AI learning model, a plurality of network nodes can exchange data according to a convolutional connection relationship while being located in layers of different depths. The AI learning model may be, for example, an artificial neural network model, a convolutional neural network model, or the like. As an embodiment, the AI learning model may be machine-learned according to a method such as supervised learning, unsupervised learning, reinforcement learning, or the like. Machine learning algorithms for performing machine learning include Decision Tree, Bayesian Network, Support Vector Machine, Artificial Neural Network, Ada-boost. , Perceptron, Genetic Programming, Clustering, etc. may be used.

Among them, CNNs are a type of multilayer perceptrons designed to use minimal preprocessing. CNN consists of one or several convolutional layers and general artificial neural network layers on top of it, and additionally utilizes weights and pooling layers. Thanks to this structure, CNN can fully utilize the input data of the two-dimensional structure. Compared with other deep learning structures, CNN shows good performance in both video and audio fields. CNNs can also be trained through standard back-passing. CNNs are easier to train than other feedforward artificial neural network techniques and have the advantage of using fewer parameters.

Convolutional networks are neural networks that contain sets of nodes with bound parameters. The increased size of available training data and the availability of computational power, combined with advances in algorithms such as piecewise linear units and dropout training, have greatly improved many computer vision tasks. For huge data sets, such as those available for many tasks today, overfitting is not important, and increasing the size of the network improves test accuracy. Optimal use of computing resources becomes a limiting factor. To this end, a distributed, scalable implementation of deep neural networks may be used.

2 is a flowchart illustrating a process of generating 3D scan data in which a texture is implemented through image registration based on a scan of a wideband lidar according to an embodiment.

Referring to FIG. 2 , first, in step S201 , the device 300 may obtain first data from the lidar 100 . Here, the first data is a 3D image generated by scanning the scan target area.

Specifically, the lidar 100 is installed in a specific place, and when a scan target area is designated by a user operation, scan for the scan target area can be performed, and when the scan for the scan target area is completed, the scan target area First data, which is a 3D image generated by scanning , may be generated.

Thereafter, when the scan target area is scanned by the lidar 100 and first data is generated in the lidar 100 , the device 300 may acquire the first data from the lidar 100 .

In step S202 , the device 300 may acquire second data from the camera 200 . Here, the second data is a 2D image generated by photographing a specific object.

Specifically, the camera 200 is installed in a specific place, and when a shooting target is designated by a user operation, shooting of the shooting target can be performed. Second data that is a 2D image may be generated.

For example, when the first object is located in the scan target area, the camera 200 may photograph the first object located in the scan target area, and the second object is a 2D image generated by photographing the first object. 2 data can be generated.

Thereafter, when the first object is captured by the camera 200 and second data is generated by the camera 200 , the device 300 may acquire the second data from the camera 200 .

In step S203 , when the first data is obtained, the device 300 may check the first space through the first data. That is, since the first data is a 3D image generated by scanning the scan target area, the apparatus 300 may identify the 3D space corresponding to the scan target area through the first data.

In step S204 , the device 300 may analyze which object the second data is an image captured. A detailed description related thereto will be described later with reference to FIG. 3 .

In step S205 , the device 300 may determine whether a photographing object analyzed through the second data is recognized in the first space identified through the first data.

For example, when the second data is analyzed as an image obtained by photographing the first object, the device 300 may determine whether the first object is detected in the first space.

Specifically, the device 300 recognizes the outline of the first object from the second data, divides an area inside the outline of the first object into an object area, and sets an area outside the outline of the first object as a background area. After classifying, by deleting the background area from the second data, an image of the first object consisting only of the object area may be extracted, and based on the image of the first object and the first data, the first object in the first space It can be checked whether or not is detected.

If it is determined in step S205 that the photographing object is not detected in the first space, the process returns to step S202 , and the device 300 may acquire the second data again from the camera 200 . To this end, the device 300 may transmit a notification message requesting confirmation as to whether the shooting location is the scan target area to the camera 200, and the camera 200 is provided with a display module and displays the notification message. It can be displayed on the screen of the module.

If it is confirmed in step S205 that the first object, which is a photographed object, is detected in the first space, in step S206 , the device 300 is located on the surface of the first object based on the image of the first object extracted from the second data. texture may be set as the first material. A detailed description related thereto will be described later with reference to FIG. 4 .

For example, when the texture of the surface of the first object is analyzed as a metal material based on the image of the first object, the apparatus 300 may set the metal material as the first material.

In operation S207 , the device 300 may generate the third data by changing the surface of the first object detected in the first space to the first material.

That is, in the first data, the texture of the object in the scan target area is not implemented. After checking the texture of the object in the scan target area through the second data, the device 300 checks the first data and the second data. Through data matching, third data that is 3D scan data in which the texture of the object is implemented may be generated.

3 is a flowchart illustrating a process of analyzing which object is an image of which second data is captured, according to an exemplary embodiment.

Referring to FIG. 3 , first, in step S301 , the device 300 may acquire second data from the camera 200 .

In operation S302 , the device 300 may calculate the ratio of the area occupied by the first object in the second data as the first ratio, based on the second data.

Specifically, the device 300 recognizes the outline of the first object from the second data, divides an area inside the outline of the first object into an object area, and sets an area outside the outline of the first object as a background area. After classifying, the ratio of the area occupied by the object region in the second data may be calculated as the first ratio.

In step S303 , the device 300 may determine whether the first ratio is greater than the reference ratio. Here, the reference ratio may be set differently according to embodiments.

If it is determined in step S303 that the first ratio is smaller than the reference ratio, the process returns to step S301 , and the device 300 may acquire the second data again from the camera 200 . To this end, the device 300 may transmit a notification message to the camera 200 for requesting confirmation of focusing of object shooting, and the camera 200 is provided with a display module, so that the notification message is displayed on the screen of the display module. can be displayed

If it is determined in step S303 that the first ratio is greater than the reference ratio, in step S304 , the device 300 may identify the area to be photographed as the first zone based on the location information of the camera 200 .

Specifically, the device 300 may acquire the location information of the camera 200 through the GPS mounted on the camera 200 , and based on the location information of the camera 200 , the camera 200 is located in which zone It is possible to check the location to be filmed through the identified location.

In step S305 , when the area to be photographed is identified as the first area, the device 300 may obtain a first list that is a list of objects disposed in the first area. To this end, a list of objects classified by region is stored in the database of the device 300 , and the device 300 may obtain the first list by inquiring information stored in the database.

In step S306 , the device 300 may compare the shape of the objects included in the first list with the shape of the first object identified through the second data.

For example, when the first object, the second object, and the third object are included in the first list, the device 300 determines the shape of the first object, the second object, and the third object included in the first list. The shape of the first object identified through the second data may be compared.

The device 300 may compare the objects included in the first list with the shapes of the first objects identified through the second data, respectively, and calculate the similarity according to the shape comparison. can be considered as objects.

In step S307 , the device 300 may check whether the first object is included in the first list.

For example, the first object, the second object, and the third object are included in the first list, and the similarity between the first object included in the first list and the first object confirmed through the second data is equal to or greater than the reference value , the device 300 may confirm that the first object is included in the first list.

If it is determined in step S307 that the first object is not included in the first list, the process returns to step S301 , and the device 300 may acquire the second data again from the camera 200 . To this end, the device 300 may transmit a notification message to the camera 200 requesting confirmation that the object to be photographed and the photographing location match the scan target area, and the camera 200 is provided with a display module, A notification message can be displayed on the screen of the display module.

If it is determined in step S307 that the first object is included in the first list, in step S308 , the device 300 may analyze the second data as a photographed image of the first object.

4 is a flowchart illustrating a process of setting a material for a surface of an object according to an exemplary embodiment.

Referring to FIG. 4 , first, in step S401 , when the device 300 extracts an image of a first object consisting of only an object region from the second data, the image of the first object is encoded to generate a first input signal. can

Specifically, the device 300 may generate the first input signal by encoding the image pixels of the first object with color information. The color information may include, but is not limited to, RGB color information, brightness information, saturation information, and depth information. The device 300 may convert color information into a numerical value, and may encode the image of the first object in the form of a data sheet including this value.

In step S402 , the device 300 may input the first input signal to the pre-trained artificial neural network in the device 300 .

The artificial neural network according to an embodiment is composed of a feature extraction neural network and a classification neural network, and the feature extraction neural network sequentially stacks a convolutional layer and a pooling layer on an input signal. The convolution layer includes a convolution operation, a convolution filter, and an activation function. The calculation of the convolution filter is adjusted according to the matrix size of the target input, but a 9X9 matrix is generally used. The activation function generally uses, but is not limited to, a ReLU function, a sigmoid function, and a tanh function. The pooling layer is a layer that reduces the size of the input matrix, and uses a method of extracting representative values by tying pixels in a specific area. In general, the average value or the maximum value is often used for the calculation of the pooling layer, but is not limited thereto. The operation is performed using a square matrix, usually a 9x9 matrix. The convolutional layer and the pooling layer are repeated alternately until the corresponding input becomes small enough while maintaining the difference.

According to an embodiment, the classification neural network has a hidden layer and an output layer. Classification of artificial neural network for material classification of object surface A neural network consists of five or less hidden layers, and may include a total of 50 or less hidden layer nodes. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. There is a total of one output layer node of the classification neural network, and an output value for object surface classification can be output to the output layer node. A detailed description of the artificial neural network will be described later with reference to FIG. 7 .

In step S403 , when a first input signal is input to the artificial neural network, the device 300 may obtain a first output signal from the artificial neural network as an output for the first input signal.

In operation S404 , the device 300 may generate a classification result for the surface of the first object based on the first output signal. Here, the classification result may include information on which material the texture of the surface of the first object is classified into.

For example, as a result of checking the output value of the first output signal, when the output value is 1, the device 300 generates a classification result as that the surface of the first object corresponds to a metal material, and when the output value is 2, the second 1 It is possible to generate classification results as the surface of the object corresponds to the wood material.

In operation S405 , the apparatus 300 may set the texture of the surface of the first object as the first material based on the classification result of the surface of the first object.

That is, the apparatus 300 may determine which texture the surface of the first object has based on the classification result of the surface of the first object, and set the checked texture as the first material. Here, the texture is the texture of the material surface felt from the fine irregularities of the surface of the material or the state of the texture, and may mean a unique texture that is different depending on the material such as metal, ceramics, plastic, glass, wood, paper, cloth, etc., and the same material Since Rado may look different when it is polished to a smooth surface and when it is finished without glossiness by attaching fine irregularities to the surface to diffuse light can be distinguished.

5 is a flowchart illustrating a process of guiding additional photographing according to an exemplary embodiment.

Referring to FIG. 5 , first, in step S501 , the device 300 may determine that the first object and the second object are included in the first list after generating the third data.

Specifically, when the area to be photographed is identified as the first area, the device 300 may obtain a first list that is a list of objects disposed in the first area. In this case, the first list may be configured to include the first object and the second object.

Since it is confirmed that the first object is included in the first list in the process of generating the third data, the device 300 checks whether other objects are included in the first list in addition to the first object, It can be confirmed that not only the first object but also the second object are further included.

In step S502 , when it is confirmed that the first object and the second object are included in the first list, the device 300 may determine whether the first object and the second object are detected in the first space. In this case, since it is confirmed that the first object is detected in the first space in the process of generating the third data, the device 300 may only check whether the second object is detected in the first space.

If it is confirmed in step S502 that the first object and the second object are detected in the first space, step S601 may be performed, and a detailed description thereof will be described later with reference to FIG. 6 .

When it is confirmed in step S502 that the first object and the second object are not detected in the first space, since it is confirmed that the first object is detected in the first space in the process of generating the third data, the first object in the first space It may be confirmed that only one object is detected and that the second object is not detected. If it is confirmed that only the first object is detected and the second object is not detected in the first space, in step S503, the device 300 performs the first A first point that is a point where the object is located may be checked, and a second point that is a point where the second object is located may be checked. To this end, the database of the device 300 stores location information classified for each object, and the device 300 obtains the location information of the first object and the location information of the second object by inquiring the information stored in the database. Thereafter, the first point may be identified through the location information of the first object, and the second point may be identified through the location information of the second object.

In step S504 , the device 300 may compare the positions of the first point and the second point to determine a movement direction and a movement distance for moving from the first point to the second point.

In step S505 , the device 300 may generate guide information for guiding movement from the first point to the second point based on the moving direction and the moving distance.

For example, if it is determined that moving 500 m in the right direction is required to move from the first point to the second point, the device 300 may generate arrow-shaped guide information informing that the user needs to move 500 m in the right direction. have.

In step S506 , the device 300 may transmit guide information and an additional photographing request message for the second object to the camera 200 . To this end, the camera 200 is provided with a display module, so that an additional photographing request message can be displayed on the screen of the display module.

That is, the first object and the second object are disposed in the scan target area, and when the camera 200 captures only the first object, the device 300 determines that additional photographing of the second object is required. Accordingly, an additional photographing request message for the second object may be transmitted to the camera 200 .

6 is a flowchart illustrating a process of estimating the size of an object according to an exemplary embodiment.

Referring to FIG. 6 , first, in step S601 , the device 300 may detect a first object and a second object in a first space.

When the first object and the second object are detected in the first space, the device 300 detects a first area that is an area occupied by the first object in the first space in step S602, and in step S603, in the first space A second area that is an area occupied by the second object may be detected.

In detecting the first area, the apparatus 300 analyzes the characteristics of the shape of the first object in the first space, recognizes the boundary of the first object, and then uses the recognized boundary of the first object to perform a first area can be detected. In this case, the apparatus 300 may detect the first area in the form of a two-dimensional box in the first space of three dimensions.

In addition, in detecting the second area, the device 300 analyzes the characteristics of the shape of the second object in the first space, recognizes the boundary of the second object, and then uses the recognized boundary of the second object. The second region may be detected. In this case, the apparatus 300 may detect the second region in the form of a two-dimensional box in the first space of three dimensions.

In operation S604 , the apparatus 300 may estimate the length of the first region in the first space and set the first length as the estimated length.

For example, when the first region has a box shape, the device 300 may check the lengths of the long axis and the diagonal of the first region, and set the first length as an average value of the long axis and the diagonal of the first region.

In operation S605 , the apparatus 300 may estimate the length of the second region in the first space and set the second length as the estimated length.

For example, when the second region has a box shape, the device 300 may check the lengths of the long axis and the diagonal of the second region, and set the second length as an average value of the long axis and the diagonal of the second region.

In step S606 , the device 300 may identify a third length that is the actual length of the first object. To this end, information on the actual length divided for each object is stored in the database of the device 300, and the device 300 may check the information stored in the database to check the third length, which is the actual length of the first object. have.

In operation S607 , the device 300 may estimate a fourth length that is an actual length of the second object by using the first length, the second length, and the third length.

Specifically, the device 300 calculates a first ratio that is a ratio of the first length and the third length, applies the second length to the first ratio, and calculates the fourth length through the process of calculating the fourth length. can be estimated

For example, if the first length is 10 cm, the second length is 20 cm, and the third length is 15 cm, the device 300 may use the first length and the third length to calculate the first ratio as 1.5 And, by applying the second length to the first ratio, the fourth length can be calculated as 30 cm.

7 is a diagram for explaining an artificial neural network according to an embodiment.

The artificial neural network 700 according to an embodiment may receive as an input a first input signal generated by encoding an image of an object, and output information on which material the texture of the surface of the object is classified into. have.

Encoding according to an embodiment may be performed by storing color information for each pixel of an image in the form of a digitized data sheet, and the color information includes RGB color, brightness information, saturation information, and depth information of one pixel. can, but is not limited to.

According to an embodiment, the artificial neural network 700 is composed of a feature extraction neural network 710 and a classification neural network 720, and the feature extraction neural network 710 separates an object region and a background region from an image. In addition, the classification neural network 720 may perform a task of identifying which material the texture of the surface of the object in the image is classified into.

The method in which the feature extraction neural network 710 distinguishes the object region and the background region is that the change in each value of color information from the data sheet of the image-encoded input signal is 30% in 6 or more of 8 pixels including one pixel. A bundle of pixels detected as having an abnormal change may be used as a boundary between the object area and the background area, but is not limited thereto.

The feature extraction neural network 710 stacks the input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes a convolution operation, a convolution filter, and an activation function. The calculation of the convolution filter is adjusted according to the matrix size of the target input, but a 9X9 matrix is generally used. The activation function generally uses, but is not limited to, a ReLU function, a sigmoid function, and a tanh function. The pooling layer is a layer that reduces the size of the input matrix, and uses a method of extracting representative values by tying pixels in a specific area. In general, the average value or the maximum value is often used for the calculation of the pooling layer, but is not limited thereto. The operation is performed using a square matrix, usually a 9x9 matrix. The convolutional layer and the pooling layer are repeated alternately until the corresponding input becomes small enough while maintaining the difference.

The classification neural network 720 checks the surface of the object region separated from the background through the feature extraction neural network 710, and checks whether the surface state of a predefined material is similar to the surface state by state, so that the surface of the object region is the surface of the material. It is possible to determine which state it is classified into. In order to compare the surface state by state of the material, information stored in the database of the device 300 may be utilized.

The classification neural network 720 has a hidden layer and an output layer, and is composed of five or less hidden layers, including a total of 50 or less hidden layer nodes, and the activation function of the hidden layer is a ReLU function and a sigmoid function. and tanh functions, but is not limited thereto.

The classification neural network 720 may include only one output layer node in total.

The output of the classification neural network 720 is an output value of which state of the material the object surface is classified into, and may indicate which state of the material corresponds to. For example, when the output value is 1, it may indicate that the object surface is in a smooth state of a metal material, and when the output value is 2, it may indicate that the object surface is in a rough state of a wood material.

According to an embodiment, the artificial neural network 700 may learn by receiving the first learning signal generated by the corrected correct answer input by the user when the user discovers a problem in the output according to the artificial neural network 700 . The problem of output according to the artificial neural network 700 may mean a case in which an output value classified into a state of a different material is output with respect to the object surface.

The first learning signal according to an embodiment is created based on the error between the correct answer and the output value, and in some cases, SGD using delta, a batch method, or a method following a backpropagation algorithm may be used. The artificial neural network 700 performs learning by modifying the existing weights according to the first learning signal, and may use momentum in some cases. A cost function can be used to calculate the error, and a cross entropy function can be used as the cost function. Hereinafter, the learning contents of the artificial neural network 700 will be described with reference to FIG. 8 .

8 is a diagram for explaining a method of learning an artificial neural network according to an embodiment.

According to an embodiment, the learning apparatus may train the artificial neural network 700 . The learning device may be a separate entity different from the device 300 , but is not limited thereto.

According to an embodiment, the artificial neural network 700 includes an input layer to which training samples are input and an output layer to output training outputs, and may be learned based on a difference between the training outputs and the first labels. Here, the first labels may be defined based on a representative image registered for each state of the material. The artificial neural network 700 is connected to a group of a plurality of nodes, and is defined by weights between the connected nodes and an activation function that activates the nodes.

The learning apparatus may train the artificial neural network 700 by using a Gradient Decent (GD) technique or a Stochastic Gradient Descent (SGD) technique. The learning apparatus may use a loss function designed by the outputs and labels of the artificial neural network 700 .

The learning apparatus may calculate a training error using a predefined loss function. The loss function may be predefined with a label, an output, and a parameter as input variables, where the parameter may be set by weights in the artificial neural network 700 . For example, the loss function may be designed in a Mean Square Error (MSE) form, an entropy form, or the like, and various techniques or methods may be employed in an embodiment in which the loss function is designed.

The learning apparatus may find weights affecting the training error by using a backpropagation technique. Here, the weights are relationships between nodes in the artificial neural network 700 . The learning apparatus may use the SGD technique using labels and outputs to optimize the weights found through the backpropagation technique. For example, the learning apparatus may update the weights of the loss function defined based on the labels, outputs, and weights using the SGD technique.

According to an embodiment, the learning apparatus may obtain representative images 801 for each state of the labeled training material from the database of the apparatus 300 . The learning apparatus may obtain pre-labeled information on the representative images 801 for each state of the material, and the representative images 801 for each state of the material may be labeled according to the pre-classified state of the material.

According to an embodiment, the learning apparatus may acquire 1000 representative images 801 for each state of the labeled training material, and based on the representative images 801 for each state of the labeled training material, the first training material It is possible to generate vectors 802 for each state of . Various methods may be employed to extract the vectors 802 for each state of the first training material.

According to an embodiment, the learning apparatus may obtain the first training outputs 803 by applying the vectors 802 for each state of the first training material to the artificial neural network 700 . The learning apparatus may train the artificial neural network 700 based on the first training outputs 803 and the first labels 804 . The learning apparatus may train the artificial neural network 700 by calculating the training errors corresponding to the first training outputs 803 and optimizing the connection relationship of nodes in the artificial neural network 700 to minimize the training errors. .

9 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Device 300 according to one embodiment includes a processor 301 and a memory 302 . The processor 301 may include at least one of the devices described above with reference to FIGS. 1 to 8 , or may perform at least one method described above with reference to FIGS. 1 to 8 . A person or organization using the apparatus 300 may provide a service related to some or all of the methods described above with reference to FIGS. 1 to 8 .

The memory 302 may store information related to the above-described methods or a program in which the methods described below are implemented. Memory 302 may be volatile memory or non-volatile memory.

The processor 301 may execute a program and control the device 300 . The code of the program executed by the processor 301 may be stored in the memory 302 . The device 300 may be connected to an external device (eg, a personal computer or a network) through an input/output device (not shown), and may exchange data through wired/wireless communication.

The device 300 may be used to train an artificial neural network or use a learned artificial neural network. Memory 302 may include a learning or learned artificial neural network. The processor 301 may learn or execute an artificial neural network algorithm stored in the memory 302 . The apparatus 300 for learning an artificial neural network and the apparatus 300 for using the learned artificial neural network may be the same or may be separate.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (3)

A method for generating 3D scan data in which a texture is implemented through image registration based on a scan of a wideband lidar performed by a device, the method comprising:
when the scan target area is scanned by the lidar, acquiring first data that is a 3D image generated by scanning the scan target area from the lidar;
when the first object located in the scan target area is photographed by a camera, obtaining second data that is a 2D image generated by photographing the first object from the camera;
calculating a ratio of an area occupied by the first object in the second data as a first ratio based on the second data;
when it is determined that the first ratio is greater than a preset reference ratio, checking a photographing target area based on the location information of the camera;
obtaining a first list, which is a list of objects disposed in the first area, when the photographing target area is identified as the first area;
determining whether the first object is included in the first list by comparing the shape of the objects included in the first list with the shape of the first object identified through the second data;
when it is confirmed that the first object is included in the first list, analyzing the second data as a photographed image of the first object;
when the second data is analyzed as an image obtained by photographing the first object, checking whether the first object is detected in a first space identified through the first data;
when the first object is detected in the first space, setting a texture for the surface of the first object as a first material based on the image of the first object extracted from the second data;
generating third data by changing the surface of the first object detected in the first space to the first material;
when it is confirmed that the first object and the second object are included in the first list, checking whether the first object and the second object are detected in the first space;
If only the first object is detected and the second object is not detected in the first space, a first point that is a point where the first object is located is checked, and a second point that is a point where the second object is located is determined. checking;
comparing the positions of the first point and the second point to determine a moving direction and a moving distance for moving from the first point to the second point;
generating guide information for guiding movement from the first point to the second point based on the moving direction and the moving distance; and
Comprising the step of transmitting an additional photographing request message for the guide information and the second object to the camera,
A method of generating textured 3D scan data through wideband lidar scan-based image registration. delete The method of claim 1,
Setting a texture for the surface of the first object comprises:
generating a first input signal by encoding the image of the first object;
inputting the first input signal to an artificial neural network, and obtaining a first output signal as an output of the first input signal;
generating a classification result for the surface of the first object based on the first output signal; and
Based on the classification result of the surface of the first object, comprising the step of setting a texture for the surface of the first object as the first material,
A method of generating textured 3D scan data through wideband lidar scan-based image registration.

Images