KR102662978B1 - Real-world object recognition method and device - Google Patents

Abstract

The present invention relates to a method for recognizing objects in a real environment. The real-environment object recognition method includes the steps of receiving a real-environment image including a random object, extracting real-environment feature information corresponding to the random object based on the real-environment image, the extracted real-environment feature information and It includes calculating a degree of similarity between a plurality of CAD feature information extracted from a plurality of CAD files corresponding to each of a plurality of target objects, and recognizing an arbitrary object based on the calculated similarity.

Description

Real-world object recognition method and device {REAL-WORLD OBJECT RECOGNITION METHOD AND DEVICE}

The present invention relates to a method and device for recognizing objects in a real environment, and specifically, to a method and device for recognizing objects in a real environment using a CAD file.

In the field of computer vision, it is important for self-driving robots to accurately recognize objects. For example, an autonomous robot can drive in a specific area, photograph the surrounding environment using an image sensor, and recognize the type, name, and shape of objects included in the captured image. In general, these robots can recognize objects that have been previously learned by artificial neural networks, but cannot recognize objects that have not been learned.

Meanwhile, it may consume a lot of time and resources to re-train on objects that have not been learned. For example, even when learning a new object, the artificial neural network must receive a large number of images of the new object and be trained to recognize the objects included in the images. Therefore, it is difficult for these artificial neural networks to be used in industrial settings where new objects are continuously being developed.

The present invention provides a real-environment object recognition method, a computer program stored in a recording medium, and a device (system) to solve the above problems.

The present invention may be implemented in a variety of ways, including as a method, device (system), or computer program stored in a readable storage medium.

According to an embodiment of the present invention, a real-environment object recognition method performed by at least one processor includes receiving a real-environment image including an arbitrary object, and based on the real-environment image, corresponding to the arbitrary object. extracting real-world feature information, calculating a degree of similarity between the extracted real-world feature information and a plurality of CAD feature information extracted from a plurality of CAD files corresponding to each of a plurality of target objects, and calculating the calculated similarity. It includes recognizing an arbitrary object as a basis.

According to an embodiment of the present invention, the step of recognizing a random object based on the calculated similarity includes extracting the bounding box coordinates for the area corresponding to the random object from the real environment image and determining the location of the random object. Includes a recognition step.

According to an embodiment of the present invention, the step of extracting real environment feature information corresponding to a random object includes providing a real environment image to a learned third machine learning model to obtain real environment feature information corresponding to a random object. It includes the step of extracting.

According to one embodiment of the present invention, generating a graph corresponding to the CAD file using a CAD file including a target object and providing the generated graph to the learned second machine learning model to generate a CAD associated with the target object. A step of extracting feature information is further included.

According to one embodiment of the present invention, a CAD file includes a plurality of entities and reference relationships between the plurality of entities. The step of generating a graph corresponding to a CAD file includes determining a plurality of entities included in the CAD file as nodes and determining reference relationships between the plurality of entities as a plurality of edges to generate a graph.

According to an embodiment of the present invention, the real-world feature information includes a feature vector for an arbitrary object, and the plurality of CAD feature information includes a feature vector for each of a plurality of target objects. The step of calculating the similarity includes providing a feature vector for an arbitrary object and a feature vector for each of a plurality of target objects to the learned second machine learning model, and calculating the similarity between the feature vectors. .

According to one embodiment of the present invention, the third machine learning model is learned to calculate similarity between feature vectors based on a triplet loss function.

In order to execute the above-described method on a computer according to an embodiment of the present invention, a computer program stored in a computer-readable recording medium is provided.

In various embodiments of the present invention, the computing device does not use a machine learning model that can directly identify an object from an image, but uses a CAD file to identify objects included in the image, thereby detecting the presence of a new object that has not been learned. Even so, it is possible to effectively recognize or identify objects in the real environment by using only the CAD file associated with the object.

In various embodiments of the present invention, a computing device recognizes an object using a CAD file, thereby reducing a lot of time and cost when adding work on a new object to an industrial site. Accordingly, it is possible to save a lot of time and money for mass production of various types. It can be used effectively in industrial settings.

In various embodiments of the present invention, by using a STEP format CAD file rather than a 3D model file, a graph associated with the CAD file can be simply created with only a small amount of computing power, and CAD feature information associated with the target object is generated based on the generated graph. can be extracted.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as a “person skilled in the art”) in the technical field to which the present invention pertains from the description of the claims. It will be understandable.

Embodiments of the present invention will be described with reference to the accompanying drawings described below, in which like reference numerals represent like elements, but are not limited thereto.
1 is a diagram illustrating an example of a computing device performing object recognition according to an embodiment of the present invention.
Figure 2 is a functional block diagram showing the internal configuration of a computing device according to an embodiment of the present invention.
Figure 3 is a diagram illustrating an example of generating a graph associated with a CAD file according to an embodiment of the present invention.
Figure 4 is a diagram illustrating an example of calculating similarity between a plurality of machine learning models according to an embodiment of the present invention.
Figure 5 is a flowchart illustrating an example of a method for recognizing objects in a real environment according to an embodiment of the present invention.
Figure 6 is a flowchart showing an example of a method for extracting graph CAD feature information according to an embodiment of the present invention.
Figure 7 is a block diagram showing the internal configuration of a computing device according to an embodiment of the present invention.

Hereinafter, specific details for implementing the present invention will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present invention.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms, but the present embodiments only serve to ensure that the present invention is complete and that the scope of the present invention is conveyed to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

In the present invention, terms such as "comprise", "comprising", etc. may indicate the presence of features, steps, operations, elements and/or components, but may indicate that such terms include one or more other functions, It does not preclude the addition of steps, operations, elements, components and/or combinations thereof.

In the present invention, when a specific component is referred to as being “coupled,” “combined,” “connected,” or “reacting” with any other component, the specific component is directly bonded, combined, and/or connected to the other component. Alternatively, it may be connected or react, but is not limited thereto. For example, one or more intermediate components may exist between a particular component and another component. Additionally, in the present invention, “and/or” may include each of one or more listed items or a combination of at least a portion of one or more items.

In the present invention, terms such as “first” and “second” are used to distinguish specific components from other components, and the components described above are not limited by these terms. For example, the “first” component may be an element of the same or similar form as the “second” component.

FIG. 1 is a diagram illustrating an example in which the computing device 100 performs object recognition 130 according to an embodiment of the present invention. According to one embodiment, the computing device 100 may be a device for identifying images and/or objects included in a video, may include an image sensor (e.g., a camera, etc.), or may be a device associated with an image sensor. . For example, the computing device 100 may include, but is not limited to, a navigating robot.

As shown, the computing device 100 uses a real-world image 110 and a computer aided design (CAD) file 120 to recognize or identify an object included in the real-world image 110. can do. Here, the real-world image 110 is an image including one or more objects and may be an image captured by an image sensor associated with the computing device 100. Additionally, the CAD file 120 is a file containing 3D model information about the characteristics, shape, form, etc. of a specific object and may be, for example, a STEP format file. In this case, a STEP format file consists of HEADER and DATA, and DATA may be composed of multiple entities referencing each other. That is, the computing device 100 may define or determine the object included in the real environment image 110 by comparing the object included in the real environment image 110 with the object in each predefined CAD file 120. there is.

According to one embodiment, the computing device 100 receives a real-environment image 110 including an arbitrary object, and based on the real-environment image 110, real-environment features corresponding to the arbitrary object. Information can be extracted. In this case, any algorithm, machine learning model, etc. can be used to extract features such as corners, edges, and gradients related to the image. For example, the computing device 100 provides the real-world image 110 to a learned machine learning model (e.g., CNN model, etc.) to extract features of any object included in the real-world image 110. You can. For example, when the real environment image 110 includes a plurality of objects, the computing device 100 may extract real environment feature information for each of the plurality of objects.

According to one embodiment, the computing device 100 may extract CAD feature information corresponding to a target object associated with the CAD file 120 based on the CAD file 120 . That is, each CAD file may be associated with a different target object, and the computing device 100 may extract CAD feature information associated with the target object for each of the plurality of CAD files. For example, the computing device 100 may use the STEP format CAD file 120 to create a graph corresponding to the target object included in the CAD file 120, and the graph generated in this way may be CAD feature information related to the target object can be extracted by providing it to a learned machine learning model (e.g. GNN model, etc.).

Then, the computing device 100 may calculate the similarity between the real environment feature information extracted in this way and the plurality of CAD feature information extracted from the plurality of CAD files corresponding to each of the plurality of target objects. For example, real-world feature information may include a feature vector for an arbitrary object, and a plurality of CAD feature information may include a feature vector for each of a plurality of target objects. In this case, the computing device 100 may provide a feature vector for an arbitrary object and a feature vector for each of a plurality of target objects to the learned machine learning model to calculate the degree of similarity between the feature vectors. In this case, the computing device 100 may recognize or identify an arbitrary object based on the calculated similarity. For example, the computing device 100 may recognize pixels of an object in the real-world image 110 that are similar to the shape of the target object in the CAD file 120.

In FIG. 1 , the computing device 100 is described in detail as extracting CAD feature information directly from the CAD file 120, but it is not limited thereto, and the computing device 100 extracts CAD feature information from any external device, etc., from the CAD file 120. Extracted CAD feature information may be received or obtained. With this configuration, the computing device 100 does not use a machine learning model that can immediately identify objects from images, but uses CAD files to identify objects included in the images, thereby detecting the existence of new, unlearned objects. Even in this case, it is possible to effectively recognize or identify an object in the real environment by using only the CAD file associated with the object.

Figure 2 is a functional block diagram showing the internal configuration of the computing device 100 according to an embodiment of the present invention. As shown, the computing device 100 (at least one processor included in the computing device 100, etc.) includes a real-world feature extraction unit 210, a CAD feature extraction unit 220, a similarity calculation unit 230, etc. It may include, but is not limited to this. The computing device 100 communicates with external devices, etc., and can exchange data and/or information necessary for feature extraction, etc.

As described above, the real environment feature extractor 210 may receive a real environment image including an arbitrary object and extract real environment feature information corresponding to the arbitrary object based on the real environment image. Here, real-world feature information may include feature vectors for arbitrary objects, but is not limited thereto. According to one embodiment, the real environment feature extractor 210 may provide real environment images to a learned machine learning model to extract real environment feature information corresponding to an arbitrary object. For example, the machine learning model may be a convolutional neural network (CNN) model learned to extract feature vectors for specific objects based on corners, edges, etc. included in the image. That is, when the real environment image includes a plurality of objects, the real environment feature extractor 210 may extract feature vectors corresponding to each of the plurality of objects.

According to one embodiment, the CAD feature extractor 220 may use a CAD file including a target object to generate a graph corresponding to the CAD file. Additionally, the CAD feature extraction unit 220 may provide the generated graph to a learned machine learning model to extract CAD feature information associated with the target object. For example, the machine learning model may be a graph neural network (GNN) model trained to extract feature vectors corresponding to the graph using a graph. Here, the feature vector extracted from the real environment image and the feature vector extracted from the CAD file may be composed of vectors of the same dimension, but are not limited to this.

According to one embodiment, the similarity calculation unit 230 may calculate the similarity between the extracted real environment feature information and the plurality of CAD feature information extracted from the plurality of CAD files corresponding to each of the plurality of target objects. For example, the similarity calculation unit 230 may provide a feature vector for an arbitrary object and a feature vector for each of a plurality of target objects to the learned machine learning model to calculate the similarity between the feature vectors. . Here, the machine learning model may be a model learned to calculate the similarity between feature vectors based on a triplet loss function. Here, the triplet loss function can be determined as in Equation 1 below.

here, may be a feature vector extracted from a CAD file, is a feature vector extracted from a real environment image, It may be a feature vector of an object similar to is a feature vector extracted from a real environment image, It may be a feature vector of a different object. In other words, the machine learning model can be trained to reduce the difference between similar feature vectors and increase the difference between different feature vectors using the triplet loss function.

In this way, when the similarity is calculated, the computing device 100 can recognize any object included in the real environment image based on the calculated similarity. For example, the computing device 100 may recognize the location of a random object by extracting bounding box coordinates for an area corresponding to a random object from a real environment image, but is not limited to this.

In FIG. 2 , each functional configuration included in the computing device 100 is separately described and described in detail, but this is only to aid understanding of the invention, and two or more functions may be performed in one computing device. With this configuration, the computing device 100 can recognize objects using CAD files, thereby reducing a lot of time and cost when adding work on new objects to industrial sites, thereby enabling mass production of a variety of products. It can be effectively used in industrial settings.

FIG. 3 is a diagram illustrating an example of generating a graph 320 associated with a CAD file 310 according to an embodiment of the present invention. As described above, a graph 320 corresponding to the STEP format CAD file 310 including the target object may be generated, and CAD feature information associated with the target object may be extracted based on the generated graph 320. You can.

As shown, the STEP format CAD file 310 may be composed of text data containing information about the 3D model of the target object. That is, the CAD file 310 may include a HEADER area and a DATA area, and the DATA area can contain multiple entities (e.g. #101, #102, #103, #104, #105, #106, #107, etc. ) can be configured to refer to each other. In this case, the computing device (100 in FIG. 1) determines a plurality of entities included in the CAD file 310 as nodes, and determines the reference relationships between the plurality of entities as a plurality of edges. A graph 320 can be created. For example, if a reference relationship is constructed as "#107=CLOSED_SHELL(",(#101, #102, #103, #104, #105, #106));", then entity #107 and another entity A graph can be constructed so that #101, #102, #103, #104, #105, and #106 are connected to each other.

With this configuration, by using the CAD file 310 in the STEP format rather than the 3D model file, the graph 320 associated with the CAD file 310 can be easily created with only a small amount of computing power, and the generated graph 320 ) Based on this, CAD feature information associated with the target object can be extracted.

Figure 4 is a diagram illustrating an example of calculating similarity 432 by a plurality of machine learning models 410, 420, and 430 according to an embodiment of the present invention. As shown, three machine learning models 410, 420, and 430 can be used to calculate similarity. For example, the first machine learning model 410 may be a model learned to extract feature information 414 by receiving a real environment image 412, and the second machine learning model 420 is extracted from a CAD file. It may be a model trained to extract feature information 424 by receiving a graph 422. In addition, the third machine learning model 430 compares the feature information 414 and 424 to calculate the similarity 432 between the object included in the real environment image 412 and the target object associated with the graph 422. It may be a learned model.

According to one embodiment, the machine learning models 410, 420, and 430 may include artificial neural network models. An artificial neural network model is an example of a machine learning model, and in machine learning technology and cognitive science, it is a statistical learning algorithm implemented based on the structure of a biological neural network or a structure for executing the algorithm.

According to one embodiment, the artificial neural network model has nodes, which are artificial neurons that form a network through the combination of synapses, as in a biological neural network, repeatedly adjusting the weights of the synapses to produce the correct output and inference in response to a specific input. By learning to reduce the error between outputs, a machine learning model with problem-solving capabilities can be expressed. For example, the artificial neural network model may include random probability models, neural network models, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

The artificial neural network model is implemented as a multilayer perceptron (MLP), which consists of multiple layers of nodes and connections between them. The artificial neural network model according to this embodiment can be implemented using one of various artificial neural network model structures including MLP. The artificial neural network model is located between an input layer that receives input signals or data from the outside, an output layer that outputs output signals or data corresponding to the input data, and an output layer that receives signals from the input layer and extracts characteristics. It consists of n hidden layers (where n is a positive integer) that are transmitted to . Here, the output layer receives a signal from the hidden layer and outputs it to the outside. The learning method of the artificial neural network model includes a supervised learning method that learns to optimize problem solving by inputting teacher signals (correct answers), and an unsupervised learning method that does not require teacher signals. There may be, but it is not limited to this.

Figure 5 is a flowchart illustrating an example of a real-environment object recognition method 500 according to an embodiment of the present invention. The real-environment object recognition method 500 may be performed by a processor (eg, at least one processor of a computing device). As shown, the real-environment object recognition method 500 may be initiated by the processor receiving a real-environment image including an arbitrary object (S510).

The processor may extract real environment feature information corresponding to an arbitrary object based on the real environment image (S520). Here, real-world feature information may include feature vectors for arbitrary objects. For example, the processor may provide a real environment image to the learned first machine learning model to extract real environment feature information corresponding to an arbitrary object.

The processor may calculate the similarity between the extracted real-world feature information and the plurality of CAD feature information extracted from the plurality of CAD files corresponding to each of the plurality of target objects (S530). For example, the processor may provide a feature vector for an arbitrary object and a feature vector for each of a plurality of target objects to the learned third machine learning model to calculate the degree of similarity between the feature vectors. In this case, the third machine learning model can be trained to calculate the similarity between feature vectors based on a triplet loss function.

Then, the processor can recognize an arbitrary object based on the calculated similarity (S540). For example, the processor can recognize the location of a random object by extracting bounding box coordinates for an area corresponding to a random object from a real environment image. In this case, any algorithm and/or machine learning model can be used to extract the bounding box coordinates of a specific area in the image.

Figure 6 is a flowchart showing an example of a method 600 for extracting graph CAD feature information according to an embodiment of the present invention. The CAD feature information extraction method 600 may be performed by a processor (eg, at least one processor of a computing device). As shown, in the CAD feature information extraction method 600, a processor may use a CAD file including a target object to generate a graph corresponding to the CAD file (S610). For example, the processor may determine a plurality of entities included in a CAD file as nodes and determine reference relationships between the plurality of entities as a plurality of edges to generate a graph.

The processor may provide the generated graph to the learned second machine learning model to extract CAD feature information associated with the target object (S620). Here, the CAD feature information may include a feature vector for the target object associated with the CAD file. The CAD feature information generated in this way can be used together with the real environment feature information to recognize objects included in the real environment image.

FIG. 7 is a block diagram showing the internal configuration of a computing device 700 according to an embodiment of the present invention. The computing device 700 may include a memory 710, a processor 720, a communication module 730, and an input/output interface 740. As shown in FIG. 7 , the computing device 700 may be configured to communicate information and/or data over a network using a communication module 730.

Memory 710 may include any non-transitory computer-readable recording medium. According to one embodiment, the memory 710 is a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. mass storage device). As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the computing device 700 as a separate persistent storage device that is distinct from memory. Additionally, an operating system and at least one program code may be stored in the memory 710.

These software components may be loaded from a computer-readable recording medium separate from the memory 710. Recording media readable by such a separate computer may include recording media directly connectable to the computing device 700, for example, floppy drives, disks, tapes, DVD/CD-ROM drives, memory cards, etc. It may include a computer-readable recording medium. As another example, software components may be loaded into the memory 710 through the communication module 730 rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory 710 based on a computer program installed by files provided through the communication module 730 by developers or a file distribution system that distributes the installation file of the application. You can.

The processor 720 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to a user terminal (not shown) or another external system by the memory 710 or the communication module 730.

The communication module 730 may provide a configuration or function for a user terminal (not shown) and the computing device 700 to communicate with each other through a network, and the computing device 700 may be connected to an external system (e.g., a separate cloud system). etc.) may provide a configuration or function for communication. For example, control signals, commands, data, etc. provided under the control of the processor 720 of the computing device 700 pass through the communication module 730 and the network to the user terminal and/or the communication module of the external system. and/or transmitted to an external system.

Additionally, the input/output interface 740 of the computing device 700 may be connected to the computing device 700 or may be a means for interfacing with a device (not shown) for input or output that the computing device 700 may include. . In FIG. 7 , the input/output interface 740 is shown as an element configured separately from the processor 720, but the present invention is not limited thereto, and the input/output interface 740 may be included in the processor 720. Computing device 700 may include more components than those of FIG. 7 . However, there is no need to clearly show most prior art components.

The processor 720 of the computing device 700 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals and/or a plurality of external systems.

The above-described method and/or various embodiments may be implemented with digital electronic circuitry, computer hardware, firmware, software, and/or combinations thereof. Various embodiments of the present invention are executed by a data processing device, for example, one or more programmable processors and/or one or more computing devices, or as a computer program stored in a computer-readable recording medium and/or a computer-readable recording medium. It can be implemented. The above-described computer program may be written in any form of programming language, including compiled language or interpreted language, and may be distributed in any form such as a stand-alone program, module, or subroutine. A computer program may be distributed via a single computing device, multiple computing devices connected through the same network, and/or multiple computing devices distributed so as to connect through multiple different networks.

The above-described method and/or various embodiments may include one or more processors configured to execute one or more computer programs that process, store, and/or manage certain functions, functions, etc., by operating on input data or generating output data. It can be performed by . For example, the method and/or various embodiments of the present invention may be performed by special purpose logic circuits such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), and the method and/or various embodiments of the present invention may An apparatus and/or system for performing embodiments may be implemented as a special purpose logic circuit, such as an FPGA or ASIC.

The one or more processors executing the computer program may include a general purpose or special purpose microprocessor and/or one or more processors in any type of digital computing device. The processor may receive instructions and/or data from each of read-only memory and random access memory, or may receive instructions and/or data from read-only memory and random access memory. In the present invention, components of a computing device performing methods and/or embodiments may include one or more processors for executing instructions and one or more memory devices for storing instructions and/or data.

According to one embodiment, a computing device may exchange data with one or more mass storage devices for storing data. For example, a computing device can receive/receive data from a magnetic disc or an optical disc and transmit data to the magnetic or optical disc. Computer-readable storage media suitable for storing instructions and/or data associated with a computer program include semiconductor memory devices such as EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable PROM), and flash memory devices. It may include, but is not limited to, any form of non-volatile memory. For example, computer-readable storage media may include magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD-ROM, and DVD-ROM disks.

To provide interaction with a user, the computing device may include a display device (e.g., cathode ray tube (CRT), liquid crystal display (LCD), etc.) for presenting or displaying information to the user and a display device (e.g., cathode ray tube (CRT), liquid crystal display (LCD), etc.) for providing or displaying information to the user. It may include, but is not limited to, a pointing device (e.g., keyboard, mouse, trackball, etc.) capable of providing input and/or commands. That is, the computing device may further include any other types of devices for providing interaction with the user. For example, a computing device may provide any form of sensory feedback to a user for interaction with the user, including visual feedback, auditory feedback, and/or tactile feedback. In response, the user can provide input to the computing device through various gestures such as sight, voice, and movement.

In the present invention, various embodiments may be implemented in a computing system that includes a back-end component (e.g., a data server), a middleware component (e.g., an application server), and/or a front-end component. In this case, the components may be interconnected by any form or medium of digital data communication, such as a communications network. For example, a communication network may include a Local Area Network (LAN), a Wide Area Network (WAN), etc.

Computing devices based on example embodiments described herein may be implemented using hardware and/or software configured to interact with a user, including a user device, user interface (UI) device, user terminal, or client device. You can. For example, a computing device may include a portable computing device, such as a laptop computer. Additionally or alternatively, computing devices include personal digital assistants (PDAs), tablet PCs, game consoles, wearable devices, internet of things (IoT) devices, virtual reality (VR) devices, AR (augmented reality) devices, etc. may be included, but are not limited thereto. Computing devices may further include other types of devices configured to interact with a user. Additionally, the computing device may include a portable communication device (eg, a mobile phone, smart phone, wireless cellular phone, etc.) suitable for wireless communication over a network, such as a mobile communication network. The computing device may wirelessly connect to a network server using wireless communication technologies and/or protocols, such as radio frequency (RF), microwave frequency (MWF), and/or infrared ray frequency (IRF). It can be configured to communicate with.

The various embodiments herein, including specific structural and functional details, are illustrative. Accordingly, embodiments of the present invention are not limited to those described above and may be implemented in various other forms. Additionally, the terms used in the present invention are intended to describe some embodiments and are not to be construed as limiting the embodiments. For example, the singular forms of words and the like may be construed to include the plural, unless the context clearly indicates otherwise.

In the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which these concepts belong. . Additionally, commonly used terms, such as those defined in dictionaries, should be interpreted as having meanings consistent with their meaning in the context of the relevant technology.

Although the present invention has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present invention as can be understood by a person skilled in the art to which the present invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

100: computing device
110: Real environment image
120: CAD file
130: Object recognition

Claims (8)

A real-world object recognition method performed by at least one processor, comprising:
Receiving a real-world image including an arbitrary object;
extracting real environment feature information corresponding to the arbitrary object based on the real environment image;
Calculating a degree of similarity between the extracted real-world feature information and a plurality of CAD feature information extracted from a plurality of computer aided design (CAD) files in STEP format corresponding to each of a plurality of target objects; and
recognizing the arbitrary object based on the calculated similarity;
Including,
The step of calculating the similarity between the extracted real-world feature information and a plurality of CAD feature information extracted from a plurality of CAD files in STEP format corresponding to each of the plurality of target objects,
generating a plurality of graphs corresponding to each of a plurality of CAD files in STEP format corresponding to each of the plurality of target objects; and
providing the plurality of generated graphs to a learned second machine learning model to extract the plurality of CAD feature information corresponding to each of the plurality of target objects;
A method for recognizing objects in a real environment, including:
According to paragraph 1,
The step of recognizing the arbitrary object based on the calculated similarity,
recognizing the location of the random object by extracting bounding box coordinates for an area corresponding to the random object from the real environment image;
A method for recognizing objects in a real environment, including:
According to paragraph 1,
The step of extracting real environment feature information corresponding to the random object includes:
providing the real environment image to a learned first machine learning model to extract real environment feature information corresponding to the random object;
A method for recognizing objects in a real environment, including:
delete According to paragraph 1,
The CAD file includes a plurality of entities and reference relationships between the plurality of entities,
The step of generating a graph corresponding to the CAD file is:
generating the graph by determining a plurality of entities included in the CAD file as nodes and determining reference relationships between the plurality of entities as a plurality of edges;
A method for recognizing objects in a real environment, including:
According to paragraph 1,
The real environment feature information includes a feature vector for the arbitrary object,
The plurality of CAD feature information includes a feature vector for each of the plurality of target objects,
The step of calculating the similarity is,
providing a feature vector for the random object and a feature vector for each of the plurality of target objects to a learned third machine learning model, and calculating a degree of similarity between the feature vectors;
A method for recognizing objects in a real environment, including:
According to clause 6,
The third machine learning model is learned to calculate similarity between the feature vectors based on a triplet loss function.
A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 3 and 5 to 7 on a computer.

Images