KR20230076242A - Method and device for recognizing error of drawing - Google Patents

Abstract

A method and device for detecting the error of a drawing are disclosed. The device creates a virtual object by photographing and measuring an object of a certain space with a depth camera and lidar, overlaps the virtual object with a 3D model of a predefined drawing for the object, and identifies an error between the drawing and the virtual object based on the overlap result.

Description

Method and device for recognizing error of drawing {Method and device for recognizing error of drawing}

An embodiment of the present invention relates to a method and apparatus for determining an error between a drawing and an actual object made based on the drawing.

Digital twin technology, which implements objects such as machines, equipment, and objects in the real world into a virtual world in a computer, is used in various fields such as architecture, energy, aviation, healthcare, automobiles, and national defense. For example, by measuring the tolerance range of BIM (Building Information Modeling) made with drawings and digital twins at construction sites or industrial sites, errors occurring in the design phase, process/construction phase, and quality phase can be reduced.

However, depth cameras, which are often used when real world objects are implemented as objects in the virtual world, have errors due to lens distortion, and it is very difficult to overcome them only with software algorithms. In the field of architecture, errors in digital twins cause errors in design or process/construction stages, which are closely related to the safety of buildings, so it is necessary to implement objects in the virtual world that exactly resemble objects in the real world.

A technical problem to be achieved by an embodiment of the present invention is a method for accurately identifying the error between a real object and a drawing by generating a real object in the real world as a virtual object and comparing it with a 3D model of the drawing. and to provide the device.

An example of a drawing error detection method according to an embodiment of the present invention for achieving the above technical problem is to create a virtual object by photographing and measuring an object in a certain space with a depth camera and lidar; overlapping a 3D model of a predefined drawing of the object with the virtual object; and determining an error between the drawing and the virtual object based on the overlapping result.

An example of a drawing error detection device according to an embodiment of the present invention for achieving the above technical problem is an object generator for generating a virtual object by photographing and measuring an object in a certain space with a depth camera and lidar; an overlapping unit for overlapping the virtual object with a 3D model made based on a predefined drawing of the object; and an error detection unit that determines an error between the drawing and the virtual object based on the overlapping result.

According to an embodiment of the present invention, after generating an object in the real world as a virtual object in the virtual world, it is possible to determine whether the object exactly matches the drawing by comparing it with a 3D model of the drawing. In addition, when a virtual object is created with only a depth camera, the problem of a difference between the virtual object and the real object due to lens distortion is solved through LiDAR (Light Detection and Ranging) to accurately match or match the object. You can create matching virtual objects within tolerance.

1 is a diagram showing an example of a drawing error detection device according to an embodiment of the present invention;
2 is a diagram showing an example of a drawing error detection method according to an embodiment of the present invention;
3 is a diagram showing an example in which a virtual object and a 3D model of a drawing are overlapped and displayed according to an embodiment of the present invention;
4 is a flowchart illustrating an example of a drawing error detection method according to an embodiment of the present invention;
5 is a diagram showing an example of a method of overlapping a plurality of spatial objects with a 3D model of a drawing according to an embodiment of the present invention;
6 is a diagram showing the configuration of an example of a drawing error detection device according to an embodiment of the present invention;
7 is a diagram showing an example of a schematic configuration of a system for creating objects in a virtual space according to an embodiment of the present invention;
8 is a diagram showing an example of a photographing method of a photographing device according to an embodiment of the present invention;
9 is a flowchart illustrating an example of a method for generating an object in a virtual space according to an embodiment of the present invention;
10 is a diagram showing an example of a method for distinguishing a background and an object of a video frame and a measurement frame according to an embodiment of the present invention;
11 is a diagram showing an example of a lattice space according to an embodiment of the present invention;
12 is a diagram showing an example of displaying objects classified in a video frame in a grid space according to an embodiment of the present invention;
13 is a diagram showing an example of a method of correcting a depth value of a pixel of an object according to an embodiment of the present invention;
14 is a diagram showing an example of a method for generating a 3D virtual object according to an embodiment of the present invention;
15 is a diagram showing an example of extracting a point cloud for generating a 3D virtual object according to an embodiment of the present invention;
16 is a diagram showing an example of generating a 3D virtual object according to an embodiment of the present invention, and
17 is a diagram showing the configuration of an example of a virtual object generating device according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, a drawing error detection method and apparatus according to an embodiment of the present invention will be described in detail.

1 is a diagram showing an example of a drawing error detection device according to an embodiment of the present invention.

Referring to FIG. 1 , the drawing error detection device 100 compares a virtual object 110 made based on an object in the real world with a 3D model 120 of the drawing for the object to determine whether there is an error. In this embodiment, objects in the real world are all objects made according to drawings, such as buildings (eg, sewer pipes, columns, walls, etc.), various facilities, or various objects (wardrobe, sink, chair, shoes, etc.) includes However, in the following, for convenience of description, examples of objects are mainly presented, such as a building or an appendage of a building. The 3D model 120 means 3D data of a drawing designed in advance for object creation, and as an example, there is a 3D model made of CAD drawings.

Buildings, etc., are constructed according to pre-designed drawings. In order to check whether a building constructed according to the drawing is constructed in accordance with the drawing, it is visually checked and the length or thickness is measured, which takes a lot of time and may be difficult to accurately measure. Accordingly, in this embodiment, various objects, such as construction or completed buildings or facilities, are created as virtual objects 110 according to the drawing, and then the virtual object 110 is compared with the 3D model 120 of the drawing through a process. A method for determining whether an error 130 between the object and the real world occurs is presented.

2 is a diagram illustrating an example of a drawing error detection method according to an embodiment of the present invention.

Referring to FIG. 2 , the drawing error detection apparatus 100 creates a virtual object by photographing and measuring an object through a depth camera and lidar. In order to overlap (250) the 3D model of the virtual object and the drawing, the virtual object and the drawing must be in a file format readable by the 3D software (240).

The file format of the original data 200 of the virtual object generated by the drawing error detection device 100 may be different from the file format of the drawing. In this case, the drawing error detection device 100 converts the original data 200 of the virtual object into a first format readable by the 3D software 240 (eg, RAWDATA or “*. obj" format file) to create a virtual object file 220. Here, the 3D software 240 may be various conventional software capable of displaying 3D data such as a CAD program. As another example, the 3D software 240 may be implemented as a part of the drawing error detection device 100 .

As another embodiment, the drawing error detection apparatus 100 may convert the original data 200 of a virtual object into a plurality of different formats such as a second format in addition to the first format. That is, the drawing error detection device 100 includes a plurality of data format conversion algorithms, and converts the original data 200 of the virtual object to the same format as the drawing file 230 by applying the format conversion algorithm. there is. The data format conversion process itself is a well-known technology. The drawing error detection device may apply various conventional data format conversion algorithms to this embodiment. If the format of the original data 200 of the virtual object generated by the drawing error detection device 100 is the same as the format of the drawing file 230, the data format conversion process as in the present embodiment may be omitted.

3 is a diagram illustrating an example in which a virtual object and a 3D model of a drawing are overlapped and displayed according to an embodiment of the present invention.

Referring to FIG. 3, the drawing error detection device 100 overlaps (320) a 3D model 300 made based on the drawing and a virtual object 310 created by photographing a real object with a depth camera and lidar. Check for errors. A detailed method of generating a virtual object using a depth camera and LIDAR will be reviewed again with reference to FIGS. 7 to 16 .

For example, at least one object existing in real space is made according to a pre-designed drawing. For example, water supply and sewage pipes or wall structures of buildings are made according to construction drawings. The drawing error detection device creates a virtual object 310 by photographing an object made based on a drawing using a photographing device. The drawing error detection device 100 overlaps 320 the object 302 and the virtual object 310 existing in the 3D model 300 of the drawing to determine the portion 330 where the error exists, and the error portion ( 330) can be distinguished and displayed in a way that users can identify, such as colors or symbols.

If the first coordinate system of the virtual object 310 generated by the drawing error detection device 100 and the second coordinate system of the 3D model 300 of the drawing are the same, the virtual object 310 and the 3D model 300 are What is necessary is just to display it as it is in the 1st coordinate system (or the 2nd coordinate system). However, if the first coordinate system and the second coordinate system are different (eg, the origin or the value range of each axis), it is difficult to overlap the virtual object 310 and the 3D model 300 as they are.

In this case, the drawing error detection device 100 adjusts the size, direction, etc. of the virtual object 310 to the object corresponding to the 3D model based on feature points (eg, corners, vertices, etc.). Can be nested by transforming. Various conventional methods of extracting feature points from an image may be applied to this embodiment. In another embodiment, since a plurality of objects exist in the 3D model 300, such as a building, it may be difficult to determine which of the plurality of objects the virtual object 310 should overlap with. The drawing error detection device 100 may use an artificial intelligence model trained to recognize the same object. For example, the drawing error detection device 100 identifies the pipe object 302 in the 3D model 300 using an artificial intelligence model trained to identify the pipe object, and the identified 3D model 300 The pipe object 302 and the virtual object 310 may be overlapped by transforming the size or direction as needed. Depending on the type of virtual object, various artificial intelligence models for recognizing each object may exist. Since learning and generation of artificial intelligence models for object identification are already widely known technologies, additional explanations thereof will be omitted.

4 is a flowchart illustrating an example of a drawing error detection method according to an embodiment of the present invention.

Referring to FIG. 4 , the drawing error detection apparatus 100 creates a virtual object by photographing and measuring a real object in a real space through a depth camera and LIDAR (S400). For example, when a virtual object is created using only a depth camera, there is a problem that the virtual object is distorted due to lens distortion, etc. After correcting the depth value of the pixel of the area corresponding to the distance value of the point of the measurement frame obtained by measuring with LIDAR, a virtual object is created based on the corrected pixel depth value. Creation of virtual objects will be reviewed again below in FIG. 7 .

The drawing error detection apparatus 100 overlaps the virtual object and the 3D model of the drawing (S410) to determine the error between the drawing and the virtual object (S420). Since the virtual object is created by photographing an object made in real space based on the drawing, it is possible to determine whether the real object matches the drawing by comparing the virtual object with the 3D model of the drawing. In order to accurately determine the error with the drawing, it is important that the virtual object and the real object are created without error. To this end, the present embodiment creates a virtual object using lidar together with a depth camera as described above.

5 is a diagram illustrating an example of a method of overlapping a plurality of spatial objects with a 3D model of a drawing according to an embodiment of the present invention.

Referring to FIG. 5 , a building or the like is divided into a plurality of spaces 500, and objects of the same type or different types may exist in each space. For example, as shown in FIG. 3 , pipes such as gas pipes or water and sewage pipes may exist in various spaces of a building. According to an embodiment, a virtual object for one type of object may be created by capturing a real space, or a virtual object for a plurality of objects may be created by distinguishing a plurality of objects from a single photographed image. Hereinafter, for convenience of description, a case of recognizing an object of one type in a plurality of spaces will be mainly described.

As an example, if there is a virtual object created based on data obtained by photographing a specific space of a building, the user designates the region of the 3D model of the drawing to be compared with the virtual object to the drawing error detection device 100, or can be entered. For example, if a virtual object is created by photographing the first space, the user inputs space identification information indicating that the point at which the virtual object was created is the first space to the drawing error detection device 100 or the first space in the 3D model. You can select an area of space. The drawing error detecting apparatus 100 may determine an area of error by identifying an area corresponding to the first space in the 3D model of the drawing and then overlapping and comparing the corresponding area of the 3D model with the virtual object in the first space.

As another embodiment, the drawing error detection device 100 may create virtual objects for each of the plurality of spaces by photographing a plurality of spaces, such as a building, with a photographing device. Even in this case, as in the above example, the user may directly input to the drawing error detection device 100 which space each virtual object corresponds to among a plurality of spaces. However, if the number of virtual objects is more than tens or hundreds, it may be difficult for the user to input them one by one. As a method of solving this problem, the drawing error detection device 100 may store information about a photographing space together when storing data photographed by a photographing device. For example, the photographing device may photograph after recognizing the tag 510 (eg, barcode, QR code, RFID, etc.) located on one side of each space. Identification information for classifying each space is stored in the tag. The photographing device may store identification information and photographing data of the tag 510 or may provide the drawing error detection device 100 . As another embodiment, the photographing device may store the identification information for each space directly input from the user together with the photographing data of each space or may transmit it to the drawing error detection device 100 .

The drawing error detection apparatus 100 may generate virtual objects 520 for each space, map them to identification information of each space, and store them. The drawing error detection device 100 classifies the corresponding area of the 3D model of the drawing based on the identification information of each space, and stores the virtual object 520 mapped with the identification information of each space to the corresponding area of the 3D model. By overlapping, errors can be identified.

6 is a diagram showing the configuration of an example of a drawing error detection device according to an embodiment of the present invention.

Referring to FIG. 6 , the drawing error detection device 100 includes an object generator 600 , an overlapping unit 610 and an error detection unit 620 . As an example, the drawing error detection device 100 may be implemented as a computing device including a memory, a processor, an input/output device, and the like. In this case, each component of the present embodiment may be implemented as software, loaded into a memory, and then executed by a processor.

The object generator 600 creates a virtual object by photographing and measuring a real object in a certain space with a depth camera and lidar. A method of creating a virtual object is shown below in FIG. 7 .

The overlapping unit 610 overlaps a virtual object with a 3D model made based on a predefined drawing of the object. An example of overlap is shown in FIG. 3 . In addition, an example of a method of overlapping a plurality of virtual objects existing in a plurality of spaces with a corresponding region of the 3D model of the drawing is shown in FIG. 5 .

The error detection unit 620 determines an error between the drawing and the virtual object based on the overlapping result. Through comparison of the drawing and the virtual object, it is possible to determine whether the object created based on the drawing has errors with the design of the drawing.

7 is a diagram showing an example of a schematic configuration of a system for creating objects in a virtual space according to an embodiment of the present invention.

Referring to FIG. 7 , a photographing device 700 includes a depth camera 702 and a LIDAR 704 . The depth camera 702 is a camera that captures a certain space in which the real object 730 exists and provides a depth value of each pixel together. The depth camera 702 itself is a well-known technology, and various types of conventional depth cameras 702 may be used in this embodiment. In this embodiment, the depth camera 702 may capture a still image or a video. However, hereinafter, for convenience of description, a case in which a video is captured by a depth camera will be mainly described. Photo data including a depth value of each pixel obtained by photographing with the depth camera 702 is called an image frame. That is, a video is composed of a certain number of video frames per second.

The LIDAR 704 emits a laser into a certain space, measures a signal returned from each point in the space (ie, a reflection point), and outputs distance values for a plurality of points in the certain space. Data consisting of distance values for a plurality of points measured by the LIDAR 704 in a certain space at a certain point in time is called a measurement frame. The resolution of a plurality of points measured by the lidar 704, that is, the measurement frame, may be different depending on the lidar 704. LiDAR 704 itself is already widely known technology, and various conventional lidar 704 may be used in this embodiment.

The photographing device 700 simultaneously drives the depth camera 702 and the LIDAR 704 to photograph and measure a real object 730 in a certain space. In this embodiment, the expression 'photographing' of the photographing device 700 or 'measurement' of the photographing device may be interpreted as photographing by the depth camera 702 and measuring by the LIDAR 704 simultaneously. The number of video frames generated per second by the depth camera 702 and the number of measurement frames generated per second by the LIDAR 704 may be the same or different depending on embodiments. Also, the resolution of the video frame and the resolution of the measurement frame may be the same or different depending on the embodiment. However, since the depth camera 702 and the lidar 704 are simultaneously driven to generate an image frame and a measurement frame for a certain space, they can be synchronized by being mapped to the same time axis.

The virtual object generating device 710 uses the image frame obtained by photographing with the depth camera 702 and the measurement frame obtained by measuring with the LIDAR 704 together to create an object in the virtual space for the object 730 in the real world (that is, , digital twin). 1 to 6 have been described on the assumption that the virtual object generator 710 is implemented as a part of the drawing error detection device 100, but the virtual object generator 710 is separate from the drawing error detection device 100. It can be implemented as a device.

The virtual object generating device 710 may be connected to the photographing device 700 through a wired or wireless communication network (eg, WebRTC, etc.) to receive the image frame and the measurement frame generated by the photographing device 700 in real time. As another embodiment, the virtual object generating device 710 captures the image frame and the measurement frame generated by the photographing device 700 for a certain period of time and measures them, and then stores the storage medium (for example, Universal Serial Bus (USB)). It may be received through a wired or wireless communication network (for example, a local area network, etc.) with the photographing device 700 after shooting is completed. It can be provided to the user terminal 720 so that it can be checked.

As another embodiment, the photographing device 700 and the virtual object generating device 710 may be implemented as one device. For example, the photographing device 110 and the virtual object device 710 may be implemented as part of various devices that display augmented reality or virtual reality, such as AR (augumented reality) glasses, HMD (Head Mounded Display), or wearable devices. . As another embodiment, the photographing device 700 is implemented as a part of AR glasses, HMD, or a wearable device, and the photographing device 700 is a virtual object generating device that connects real-time photographing and measuring image frames and measurement frames to a wired or wireless communication network. 710, the AR glasses, HMD, etc. implemented with the photographing device may receive the virtual object from the virtual object generating device 710 and display the virtual object in augmented reality or virtual reality. The user can immediately check the virtual object created in real time through augmented reality or virtual reality. A detailed method of generating an object in a virtual space will be reviewed again below in FIG. 2 .

8 is a diagram illustrating an example of a photographing method of a photographing device according to an embodiment of the present invention.

Referring to FIGS. 7 and 8 , the photographing device 700 may continuously photograph one object 800 or continuously photograph a plurality of objects 800 and 810 . Alternatively, the photographing device 700 may continuously photograph objects of the same type or different types located in various spaces. That is, at least one object 800 or 810 desired by the user may exist at various points in time on the basis of the time axis in the image frame and the measurement frame acquired by the photographing device 700 . This embodiment shows an example of an image frame for convenience of description.

For example, suppose that a user travels around various parts of a building and takes pictures of a sewer pipe during construction. In this case, the user may photograph the sewer pipe in space A of the building using the photographing device 700 and move to space B to photograph the sewer pipe. When moving to space A and space B, the photographing device may continue to be maintained in a photographing state or may be turned off during movement. The same type of object, the sewage pipe, exists in the video frames and measurement frames captured in space A and space B.

As another example, a plurality of objects 800 and 810 may be captured together in an image frame and a measurement frame captured in each space. Object a and object b may exist in the video frame and measurement frame captured in space A, and object a and object c may exist in the video frame and measurement frame captured in space B.

The virtual object generating device 710 may classify the video frame and the measurement frame in units of objects. The virtual object generating apparatus 710 may assign the same identification information (or index) to the video frame and the measurement frame in which the same object exists in the video frame and the measurement frame. For example, first identification information (or first index, hereinafter referred to as identification information) is assigned to all of the plurality of image frames 820 , 822 , and 824 in which the first object 800 exists, and the second object 810 exists. Second identification information may be assigned to a plurality of image frames 830 , 832 , and 834 to be used. No identification information may be assigned to the image frames 840 and 842 in which no object exists, or third identification information may be assigned. In this embodiment, the image frames arranged along the time axis can be divided into three groups, that is, A (850), B (860), and C (870). In another embodiment, if a plurality of objects exist in the video frame and the measurement frame, identification information corresponding to each object may be assigned to one video frame and the measurement frame. That is, a plurality of pieces of identification information may be assigned to one image frame and one measurement frame.

In another embodiment, since the photographing device 700 simultaneously drives the depth camera 702 and the lidar 704 to take pictures, the video frame generated by the depth camera 702 and the measurement frame measured by the lidar are synchronized in time. It can be. Therefore, the virtual object generating device 710 determines whether the same object exists only in the video frame, determines the time period of the video frame in which the same object exists, and assigns the same identification information to the video frame during that time period. The same identification information may be assigned by considering the measurement frame generated during the time period identified for the frame as a period in which the same object exists.

The virtual object generating device 710 may use various conventional image recognition algorithms to determine whether objects existing in image frames are the same. For example, the virtual object generating device 710 may use an artificial intelligence model as an example of an image recognition algorithm. Since the method itself for determining whether objects in an image are the same is a well-known technique, a detailed description thereof will be omitted.

9 is a flowchart illustrating an example of a method of generating a virtual space object according to an embodiment of the present invention.

Referring to FIG. 9 , the virtual object generating apparatus 710 divides a first background area and a first object area from an image frame obtained by photographing a certain space with a depth camera (S900). The virtual object generating apparatus 710 may distinguish a background and an object from a plurality of image frames in which the same object is photographed. For example, as shown in FIG. 8 , at least one or more image frames (eg, group A of 820, 822, and 824) in which the same object exists may be distinguished from a background, respectively. As another embodiment, when a plurality of objects exist in an image frame, the virtual object generating apparatus 710 may distinguish the background and the plurality of objects, respectively. An example of a method of distinguishing a background and an object in an image frame will be reviewed again in FIG. 10 .

The virtual object generating device 710 distinguishes a second background area and a second object area from a measurement frame obtained by measuring a certain space with lidar (S910). The virtual object generating device 710 may distinguish a background and an object for a plurality of measurement frames in which the same object is measured. For example, as shown in FIG. 8 , a background and an object may be distinguished for a plurality of measurement frames to which the same identification information is assigned. As another embodiment, when a plurality of objects exist in the measurement frame, the virtual object generating apparatus 710 may distinguish the background and the plurality of objects, respectively. An example of a method for distinguishing a background and an object in a measurement frame is shown in FIG. 10 .

As shown in FIG. 7 , the depth camera 702 and the LIDAR 704 are spaced apart from each other by a predetermined distance within the photographing device 700, and therefore, the photographing angles of the image frame and the measurement frame are different from each other. In addition, since the resolutions of the video frame and the measurement frame may be different from each other, the positions of each pixel of the video frame and each point of the measurement frame may not be mapped on a one-to-one (1:1, scale) basis. In order to map an image frame and a measurement frame having such a difference, this embodiment uses a grid space.

Specifically, the virtual object generating apparatus 710 arranges the pixels of the first object area divided from the image frame according to the depth value in the first grid space including the grid having a predefined size (S920), and also The points of the second object area divided in the measurement frame are arranged according to the distance values in the second lattice space including the lattice of the same size (S920). Since the image frame and the measurement frame are data obtained by photographing the same space, objects existing in the first object area and the second object area are the same object. The first lattice space and the second lattice space are spaces having grids of the same size in the virtual space. An example of a grid space is shown in FIG. 11 .

The virtual object generator 710 corrects the depth value of the pixel in the first grid space based on the distance value of the point in the second grid space (S930). Since the pixels of the first object area and the points of the second object area exist in the grid space of the same size, if the position, direction, size, etc. of the first grid space and the second grid space match each other, each point of the second grid space and each pixel of the first lattice space may be mapped. A specific method of correcting a depth value of a pixel in a grid unit of a grid space using a distance value of a point will be reviewed again with reference to FIG. 13 .

The virtual object generating device 710 creates an object (ie, a virtual object) in a virtual space having surface information based on the pixel whose depth value is corrected (S940). The virtual object generating device 710 corrects the pixel depth value of an object existing in a plurality of image frames to which the same identification information is assigned to the point distance value of an object existing in a plurality of measurement frames to which the same identification information is assigned, and then generates a plurality of A virtual object can be created using the corrected pixels of the image frame of . That is, a virtual object may be created by correcting a pixel depth value of an image frame of an object photographed at various angles and positions.

The virtual object generating device 710 may generate a 3D virtual object using various types of 3D modeling algorithms. For example, the virtual object generating apparatus 710 may generate a 3D object having surface information as an artificial intelligence model using a pixel having a depth value. As another embodiment, when 3D modeling is performed using all pixels, there is a disadvantage in that the amount of computation is large, so the virtual object generator 710 is a point cloud representing corners and vertices among pixels constituting an object. A virtual object can be created by extracting and inputting the point cloud to a 3D modeling algorithm. An example of generating a virtual object using a point cloud will be reviewed again in FIG. 14 . A virtual object can be created based on the distance value of each point of the measurement frame, but generally the resolution of the measurement frame is lower than the resolution of the video frame. In this embodiment, a virtual object is created using a distance value of a pixel of an image frame having a relatively high resolution.

10 is a diagram illustrating an example of a method for distinguishing a background and an object of a video frame and a measurement frame according to an embodiment of the present invention.

Referring to FIG. 10 , there is a first artificial intelligence model 1000 that distinguishes the background of an image frame from an object, and a second artificial intelligence model 1010 that distinguishes a background from an object of a measurement frame. Each of the artificial intelligence models 1000 and 1010 is a trained model using pre-constructed learning data and may be implemented as a Convolutional Neural Network (CNN) or the like. Since the process of learning and generating an artificial intelligence model itself is already a well-known technology, a description thereof will be omitted.

The first artificial intelligence model 1000 is a model generated through machine learning to distinguish a background and an object in an image frame when an image frame is input. For example, if the first artificial intelligence model 1000 is an artificial intelligence model learned to recognize a chair, the first artificial intelligence model 1000 is a region in which a chair exists in an image frame (ie, a region of a chair in an image frame). pixels) can be distinguished.

The second artificial intelligence model 1010 is a model generated through machine learning to distinguish a background and an object in a measurement frame when a measurement frame is input. For example, if the first artificial intelligence model 1000 is an artificial intelligence model learned to recognize a chair, the second artificial intelligence model 1010 corresponds to a region where a chair exists in the measurement frame (ie, corresponds to a chair within the measurement frame). points) can be distinguished.

11 is a diagram showing an example of a lattice space according to an embodiment of the present invention.

Referring to FIG. 11 , the grid space 1100 is a space in which a region in the virtual space is divided into unit grids 1110 of a certain size. For example, the lattice space 1100 may be a space composed of unit lattices 1110 having horizontal, vertical, and height d1, d2, and d3, respectively. Depending on the embodiment, d1, d2, and d3 may all have the same size (eg, 1 mm) or different sizes.

12 is a diagram showing an example of displaying objects classified in a video frame in a grid space according to an embodiment of the present invention.

Referring to FIG. 12 , the virtual object generating apparatus 710 may display pixels 1210 of an object area divided in an image frame in a grid space 1200 using the depth values. In this embodiment, pixels are schematically illustrated for ease of understanding.

The virtual object generating apparatus 710 may determine the 3D coordinate value (or vector value of the pixel, etc.) of each pixel in the grid space by mapping the pixels of the object in the image frame to the grid space 1200 . That is, a 3D coordinate value of each pixel of the object may be generated using a predefined point in the lattice space as a reference point (0,0,0 or X,Y,Z).

In this way, the virtual object generating apparatus 710 may map points representing objects in the measurement frame to the grid space. If the point of the object in the measurement frame is displayed in the grid space, it can also be displayed in a shape similar to that of FIG. 12.

13 is a diagram illustrating an example of a method of correcting a depth value of a pixel of an object according to an embodiment of the present invention.

Referring to FIG. 13, a position corresponding to one of the first grids 1300 in the first grid space to which the objects of the image frame are mapped and the first grid 1300 in the second grid space to which the objects of the measurement frame are mapped denotes the second grating 1310, respectively. In this embodiment, it is assumed that the first lattice space in which the pixels of the object in the image frame are mapped and the second lattice space in which the points of the object in the measurement frame are mapped are first matched in position, direction, size, etc. In addition, this embodiment shows an example in which one pixel 1302 and one point 1312 are present in the first grid 1300 and the second grid 1310, respectively, for convenience of description, but one grid 1300 , 1310) may have a plurality of pixels or a plurality of points. Also, the number of pixels and points present in the first grid 1300 and the second grid 1310 may be the same or different.

The virtual object generator 710 corrects the depth value of the pixel 1302 of the first grid 1300 based on the distance value of the point 1312 of the second grid 1310. Since the distance value of the point measured by lidar is more accurate, the virtual object generator 710 corrects the depth value of the pixel based on the distance value of the point. For example, if the coordinate values of the pixels 1302 of the first grid 1300 in the grid space and the coordinate values of the points of the second grid 1310 are different from each other, the virtual object generator 710 operates the first grid The pixel 1302 of (1300) is corrected (1320) according to the coordinate values of the points of the second grid 1310.

Since the image frame and the measurement frame may have different resolutions, positions indicated by pixels of the first grid 1300 and points of the second grid 1310 may not be mapped one-to-one. Therefore, by using a plurality of points that exist in the second grid 1310 or a plurality of points that exist in the surrounding grids that exist in the upper, lower, left, right, etc. of the second grid 1310, the values existing between each point are interpolated, etc. Through this, it is possible to determine the distance value of the coordinate value of the point corresponding to the coordinate value of the pixel. In addition, coordinate values of pixels of the first grid 1300 may be corrected using distance values of points generated through interpolation.

14 is a diagram illustrating an example of a method of generating a 3D virtual object according to an embodiment of the present invention.

Referring to FIG. 14 , the virtual object generating apparatus 710 may generate a 3D virtual object 1420 including surface information by inputting a point cloud 1410 to a 3D modeling algorithm 1400 . A point cloud can be composed of points representing key points that can define an object, such as vertices and edges of an object. Various conventional methods of extracting a point cloud from an image frame captured by a depth camera may be applied to this embodiment. Since the method of extracting a point cloud itself is a well-known technique, an additional description thereof will be omitted.

The virtual object generating apparatus 710 maps the object extracted from the image frame to the lattice space, and corrects the distance value (or coordinate value) of each pixel mapped to the lattice space in the same manner as shown in FIG. 13 . Then, a point cloud to be used for generating a 3D virtual object is extracted from the corrected distance value (or coordinate value) of each pixel. An example of extracting a point cloud for the object of FIG. 12 is shown in FIG. 15 . The virtual object generating device 710 may use an artificial intelligence model such as machine learning as a 3D modeling algorithm 1400 . Various conventional algorithms for generating a 3D object based on a point cloud may be applied to this embodiment. Since the method of generating a 3D object using a point cloud itself is a well-known technology, a detailed description thereof will be omitted.

15 is a diagram illustrating an example of extracting a point cloud for generating a 3D virtual object according to an embodiment of the present invention. Referring to FIG. 15 , an example of extracting a point cloud 1510 for a chair is shown. Various conventional methods of extracting a point cloud may be applied to this embodiment. As another embodiment, a 3D virtual object may be generated by extracting a point cloud after correcting a depth value of a pixel in a plurality of image frames to which the same identification number is assigned, that is, a plurality of image frames in which the same object is photographed.

16 is a diagram illustrating an example of generating a 3D virtual object according to an embodiment of the present invention. Referring to FIG. 16 , the virtual object generator may create a virtual object 1600 including surface information using a point cloud.

17 is a diagram showing the configuration of an example of a virtual object generating device according to an embodiment of the present invention.

Referring to FIG. 17, the virtual object generating device 710 includes a first object extraction unit 1700, a second object extraction unit 1710, a first grid arrangement unit 1720, a second grid arrangement unit 1730, A correction unit 1740 and an object creation unit 1750 are included. As an example, the virtual object generator 710 may be implemented as a computing device including a memory, processor, input/output device, etc., or as a server, cloud system, etc. In this case, each component is implemented in software and loaded into the memory, and then the processor can be performed by

The first object extraction unit 1700 distinguishes a first background area and a first object area from an image frame obtained by photographing a certain space with a depth camera. The second object extraction unit 1710 distinguishes a second background area and a second object area from a measurement frame obtained by measuring the predetermined space with LIDAR. Classification of the background and the object can be performed using an artificial intelligence model, and an example thereof is shown in FIG. 10 .

The first grid arranging unit 1720 arranges pixels of the first object area according to depth values in a first grid space including a grid having a predefined size. The second grid arranging unit 1730 arranges points of the second object area according to distance values in a second grid space including a grid having a predefined size. An example of a grid space is shown in FIG. 11, and an example of mapping pixels of an object extracted from an image frame to the grid space is shown in FIG.

The correction unit 1740 corrects the depth value of the pixel in the first grid space based on the distance value of the point in the second grid space. An example of a method of correcting through grid space comparison is shown in FIG. 13 .

The object generator 1750 creates a virtual object having surface information based on pixels whose depth values are corrected. The object generator 1750 may create an object in a 3D virtual space using all pixels. However, in this case, since the amount of computation increases, the object generator 1750 may create a virtual object by creating a point cloud, examples of which are shown in FIGS. 14 to 16 .

Each embodiment of the present invention can also be implemented as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, SSD, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network to store and execute computer-readable codes in a distributed manner.

So far, the present invention has been looked at mainly with its preferred embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

Claims (11)

Creating a virtual object by photographing and measuring an object in a certain space with a depth camera and lidar;
overlapping a 3D model of a predefined drawing of the object with the virtual object; and
and determining an error between the drawing and the virtual object based on the overlapping result. According to claim 1,
The generating of the virtual object may include storing the polygon data representing the virtual object in a file format readable by software displaying a 3D model of the drawing;
The overlapping step includes reading and overlapping the 3D model and the virtual object together through the software; drawing error detection method, characterized in that it comprises. The method of claim 1, wherein determining the error comprises:
and displaying a part where the position of the 3D model and the virtual object are out of alignment with a predefined color. The method of claim 1, wherein the generating of the virtual object comprises:
In the image frame obtained by shooting with a depth camera, the depth value of the pixel of the area corresponding to the object is measured with lidar and corrected with the distance value of the point of the measurement frame, and then the virtual object is calculated based on the depth value of the corrected pixel. A drawing error identification method comprising the step of generating; According to claim 1,
The step of creating the virtual object includes storing the virtual object together with space identification information obtained by reading a tag located in the space;
The overlapping step may include overlapping the virtual object with a 3D model for a corresponding area of the drawing corresponding to the space identification information. an object creation unit that creates a virtual object by photographing and measuring an object in a certain space with a depth camera and lidar;
an overlapping unit for overlapping the virtual object with a 3D model made based on a predefined drawing of the object; and
and an error detection unit configured to determine an error between the drawing and the virtual object based on the overlapping result. According to claim 6,
The object generator stores the polygon data representing the virtual object in a file format that can be read by software displaying a 3D model of the drawing,
The overlapping unit reads the 3D model and the virtual object together through the software and displays them overlapping. The method of claim 6, wherein the error detection unit,
The drawing error detection device, characterized in that for displaying a region where the position of the 3D model and the virtual object are out of alignment with a predefined color. The method of claim 6, wherein the object creation unit,
In the image frame obtained by shooting with a depth camera, the depth value of the pixel of the area corresponding to the object is measured with lidar and corrected with the distance value of the point of the measurement frame, and then the virtual object is calculated based on the depth value of the corrected pixel. Drawing error detection device, characterized in that for generating. According to claim 6,
The object creation unit stores the virtual object together with space identification information obtained by reading a tag located in the space;
The overlapping unit overlaps the virtual object with the three-dimensional model for the corresponding area of the drawing corresponding to the space identification information. A computer-readable recording medium recording a computer program for performing the method according to any one of claims 1 to 5.

Images