KR102591078B1 - Electronic device for performing 3d object detection using an artificial intelligence model and operating method thereof - Google Patents

Abstract

The present disclosure provides an electronic device that performs 3D object detection using an artificial intelligence model and a method of operating the same. An electronic device that performs 3D object detection using an artificial intelligence model according to an embodiment includes a display, a memory storing one or more instructions, and at least one processor executing one or more instructions, and includes at least one The processor obtains a point cloud corresponding to a three-dimensional scene including a plurality of objects, converts the point cloud into a two-dimensional coordinate system, displays the converted point cloud on the display, and selects a plurality of objects from the converted point cloud. Obtain a first user input including at least one point corresponding to, encode the first user input based on the point cloud, and use an artificial intelligence model using the point cloud and the encoded first user input as input. Accordingly, one or more instructions for inferring a 3D bounding box corresponding to a plurality of objects in the point cloud may be executed.

Description

Electronic device for performing 3D object detection using artificial intelligence model and operating method thereof {ELECTRONIC DEVICE FOR PERFORMING 3D OBJECT DETECTION USING AN ARTIFICIAL INTELLIGENCE MODEL AND OPERATING METHOD THEREOF}

The present disclosure relates to an electronic device that performs three-dimensional object detection using an artificial intelligence model and an operating method thereof, and more specifically, to an artificial intelligence model that infers a three-dimensional bounding box based on a point cloud and user input. It relates to an electronic device that performs 3D object detection by using and a method of operating the same.

3D object detection technology, which has been studied for a long time, is being actively researched in the fields of autonomous driving and robotics. Point clouds are one of the representative 3D data types used to efficiently represent complex scenes. To perform 3D object detection in point clouds, various techniques such as voxelization, point-based, and graph-based methods have been studied.

However, it is difficult to secure a dataset for training a 3D object detection model in point clouds, and in particular, due to the complex data characteristics of point clouds, errors are prone to occur in the annotation processing process and are expensive. These problems are making it difficult to develop high-performance 3D object detection technology.

The purpose of the present disclosure is to provide an electronic device and a method of operating the same that perform 3D object detection using user interaction and artificial intelligence models in order to accurately and efficiently infer the 3D bounding box corresponding to the object. .

An electronic device that performs 3D object detection using an artificial intelligence model according to an embodiment includes a display, a memory that stores one or more instructions, and at least one processor that executes the one or more instructions stored in the memory. However, the at least one processor acquires a point cloud corresponding to a three-dimensional scene including a plurality of objects, converts the point cloud into a two-dimensional coordinate system, and displays the converted point cloud on the display. , obtain a first user input including at least one point corresponding to the plurality of objects among the converted point cloud, encode the first user input based on the point cloud, and encode the point The one or more instructions for inferring a three-dimensional bounding box corresponding to the plurality of objects in the point cloud using an artificial intelligence model that takes the cloud and the encoded first user input as input. It can be run.

According to one embodiment, the at least one processor displays the converted point cloud and the inferred 3D bounding box on the display, and corresponds to the inferred 3D bounding box but corresponds to the plurality of objects. Obtaining a second user input including at least one of a point that does not correspond to the inferred three-dimensional bounding box and a point that does not correspond to the plurality of objects, and receives a second user input based on the point cloud. Encoding, using an artificial intelligence model using the point cloud, and the encoded first user input and the encoded second user input as input, three-dimensional bounding corresponding to the plurality of objects in the point cloud One or more of the above instructions may further be executed to infer a box.

According to one embodiment, the artificial intelligence model includes a point encoder that extracts features based on the point cloud and the encoded first user input, and a point encoder that extracts features based on the extracted features. A centroid aggregation module that predicts a corresponding center point and groups points corresponding to the plurality of objects based on the predicted center point, the extracted features, and the encoded first user input. A spatial click propagation module that calculates information of an object of the same class as the object corresponding to the first user input based on the output of the centroid aggregation module and the output of the spatial click propagation module Thus, it may include a detection head module that outputs the three-dimensional bounding box corresponding to the plurality of objects.

According to one embodiment, the point encoder includes at least one fully connected layer and at least one downsampling layer, and the at least one processor encodes the encoded signal at the output of the at least one downsampling layer. The one or more instructions concatenating the first user input may further be executed.

According to one embodiment, the spatial click propagation module calculates similarity between information corresponding to the encoded first user input and the extracted features, and the detection head module calculates the output of the centroid aggregation module. Data linked to the similarity may be obtained, and an object of the same class as the object corresponding to the first user input may be detected based on the obtained value.

According to one embodiment, the artificial intelligence model includes a point cloud dataset, at least one positive point corresponding to an object in the point cloud dataset, and at least one corresponding to a background in the point cloud dataset. Based on negative points, it can be learned to output a 3D bounding box corresponding to the object.

According to one embodiment, in the learning process of the artificial intelligence model, the foreground score among the background points that do not correspond to the plurality of objects is based on the ground truth value and the extracted features. It may include a negative click simulation module that allocates points exceeding a threshold as the at least one negative point.

According to one embodiment, the first user input may include object location information and object class information.

According to one embodiment, the one or more instructions for encoding the first user input may be configured to encode the first user input based on the distance between the (x, y) location information of the point cloud and the object location information of the first user input. 1 May include one or more instructions that encode user input.

According to one embodiment, the artificial intelligence model may use as input a value obtained by connecting the encoded first user input to the point cloud.

According to one embodiment, incorrect 3D bounding box inference can be reduced through negative click simulation. According to one embodiment, through dense click guidance, user input is not diluted in the data processing of the artificial intelligence model, thereby maximizing the effect of user interaction. According to one embodiment, accurate inference can be made with only a small number of user inputs through spatial click propagation. According to one embodiment, a more convenient user experience and interaction tool can be provided by converting the point cloud into a two-dimensional coordinate system and providing it to the user.

1A and 1B are conceptual diagrams schematically showing the operation of an electronic device that performs 3D object detection using an artificial intelligence model according to an embodiment.
Figure 2 is a block diagram showing the configuration of an electronic device according to an embodiment.
Figure 3 is a block diagram showing an inference process using an artificial intelligence model according to an embodiment.
Figure 4 is a conceptual diagram for explaining the operation of a spatial click propagation module according to an embodiment.
Figure 5 is a block diagram showing the learning process of an artificial intelligence model according to an embodiment.
Figure 6 is a conceptual diagram for explaining the operation of a negative click simulation module according to an embodiment.
7A and 7B are diagrams exemplarily showing the operation of a 2D conversion module according to an embodiment.
8A and 8B are diagrams exemplarily showing the operation of an encoding module according to an embodiment.
9A to 9D are diagrams exemplarily showing the operation of a spatial click propagation module according to an embodiment.
Figure 10 is a flowchart showing a method of operating an electronic device according to an embodiment.
Figure 11 is a flowchart showing a method of operating an electronic device according to an embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. The embodiments will be described in clear detail so that a person skilled in the art can easily practice the present disclosure. However, the scope of rights is not limited or limited by these embodiments. The same or similar reference numerals are used for similar components in each drawing, and overlapping descriptions of the same or similar components are omitted.

The terms used in the description below have been selected as general and universal in the related technical field, but there may be different terms depending on technological developments and/or changes, customs, technicians' preferences, etc. Accordingly, the terms used in the description below should not be understood as limiting the technical idea, but should be understood as illustrative terms for describing embodiments.

In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the detailed meaning will be described in the relevant description section. Therefore, the terms used in the description below should be understood based on the meaning of the term and the overall content of the specification, not just the name of the term.

Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described herein. Additionally, terms including ordinal numbers, such as 'first' or 'second', used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When it is said that a part "includes" a certain element throughout the specification, this means that, unless specifically stated to the contrary, it does not exclude other elements but may further include other elements. Additionally, terms such as “unit” and “module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present invention. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification. In addition, the reference numerals used in each drawing are only for explaining each drawing, and the different reference numerals used in each of the different drawings are not intended to indicate different elements. Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

In this disclosure, an artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. The plurality of weights possessed by the plurality of neural network layers can be optimized by the learning results of the deep neural network model. For example, during the learning process, a plurality of weights may be updated so that loss or cost values obtained from the deep neural network model are reduced or minimized. For example, deep neural network models include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), and Bidirectional Recurrent Deep Neural Network (BRDNN). ) or Deep Q-Networks, etc., but is not limited thereto.

In the present disclosure, user input may include object location information and object class (or category) information in a two-dimensional coordinate system. For example, user input can be defined as (p _k , c _k ). Here, k can be defined as the number of points corresponding to user input. Here, p _k is object location information, for example, p _k may be defined as (p _k,x , p _k,y ). Here, c _k may be object class information, for example, defined as the class of the point among C predefined classes.

1A and 1B are conceptual diagrams schematically showing the operation of an electronic device that performs 3D object detection using an artificial intelligence model according to an embodiment.

Referring to FIG. 1A, the electronic device 100 uses the point cloud 110 and the artificial intelligence model 150 using the first user input 10 as input to a plurality of objects in the point cloud 110. A corresponding 3D bounding box (eg, 160a, 160b) can be inferred. The electronic device 100 may infer a 3D bounding box (eg, 160a, 160b) a predetermined number of iterations. Figure 1a shows an example of an initial iteration.

For example, the electronic device 100 may be a device that includes a display 130. The electronic device 100 may be a device that outputs a two-dimensional image corresponding to the point cloud 110 through the display 130. For example, the electronic device 100 includes a smart phone, a laptop computer, a personal computer (PC), a tablet PC, a digital camera, a closed-circuit television (CCTV), an e-book terminal, and a digital broadcasting device. It may include, but is not limited to, terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), navigation, or MP3 players. The electronic device 100 may be implemented as various types and types of devices including a display 130.

The electronic device 100 according to one embodiment may include a two-dimensional conversion module 120, an encoding module 140, and an artificial intelligence model 150.

The electronic device 100 may acquire a point cloud 110 corresponding to a 3D scene including a plurality of objects. For example, the electronic device 100 may include a sensor (not shown) to generate the point cloud 110 . For example, the sensor (not shown) may be comprised of at least one of an image sensor, a LiDAR sensor, an RGB-D sensor, a depth sensor, a time of flight (ToF) sensor, an ultrasonic sensor, a radar sensor, and a stereo camera. The disclosure is not limited to this.

In one embodiment, a sensor (not shown) may obtain sensing data from the outside. In one embodiment, the electronic device 100 may generate the point cloud 110 based on sensing data. In one embodiment, the electronic device 100 may receive the point cloud 110 from an external server or external electronic device.

The 2D conversion module 120 can convert the point cloud 110 into a 2D coordinate system. The electronic device 100 may display the converted point cloud 111 on the display 130. The electronic device 100 receives a first user input 10 that includes at least one point (positive click or positive point) corresponding to a plurality of objects among the converted point cloud 111. can receive. For example, the electronic device 100 may receive a first user input 10 corresponding to a point corresponding to an object among a plurality of objects through an input interface. For example, the first user input 10 may include object location information and object class information (eg, pedestrian class) in the converted point cloud 111.

The encoding module 140 may encode the first user input 10 based on the point cloud 110 . For example, the encoding module 140 may encode the first user input 10 based on the distance between the (x, y) location information of the point cloud and the object location information of the first user input 10. In one embodiment, the electronic device 100 may concatenate the encoded first user input to the point cloud 110.

The electronic device 100 may input the point cloud 110 and the encoded first user input to the (pre-trained) artificial intelligence model 150. The artificial intelligence model 150 infers (or prediction) can be made. For example, the artificial intelligence model 150 may output an annotation value corresponding to a 3D bounding box (eg, 160a, 160b). That is, the inference result may represent an annotation value of a 3D bounding box (eg, 160a, 160b). For example, the annotation value may include, but is not limited to, the center point, height, width, depth, class, and angle of the 3D bounding box.

According to one embodiment, as information about positive points corresponding to the object is provided to the artificial intelligence model 150, a 3D bounding box can be inferred with high accuracy.

For example, the 3D bounding boxes 160a_1 to 160a_5 may correspond to the pedestrian class, and the 3D bounding boxes 160b_1 and 160b_2 may correspond to the vehicle class. The 3D bounding boxes 160a_1 to 160a_3 and the 3D bounding box 160b_1 may be classified as true positive. The 3D bounding boxes 160a_4 and 160a_5 and the 3D bounding box 160b_2 may be classified as false positives. In the present disclosure, a true positive may indicate that an object that should be detected has been detected, and a false positive may indicate that an object that should not be detected has been detected. Accordingly, the 3D bounding boxes 160a_4, 160a_5, and 160b_2 are 3D bounding boxes that are incorrectly detected even though they should be detected as background parts or other objects.

Referring to FIG. 1A together with FIG. 1B, FIG. 1B shows the operation of the electronic device 100 performing object detection in an iteration after the initial iteration of FIG. 1A.

The electronic device 100 may display data corresponding to the converted point cloud 111 and the 3D bounding boxes 160a_1 to 160a_5, 160b_1, and 160b_2 inferred from the initial iteration on the display 130.

The electronic device 100 includes a point (i.e., a negative point or negative click) that corresponds to the inferred three-dimensional bounding box but does not correspond to the plurality of objects, and the inferred three-dimensional bounding box. A second user input 11 that does not correspond to the bounding box but includes at least one of the points (i.e., positive point or positive click) corresponding to the plurality of objects may be obtained. For example, the second user input 11 may include a plurality of negative points and/or a plurality of positive points. Referring to FIG. 1B, since the 3D bounding box 160a_4 was detected even though it is an object that does not correspond to the pedestrian class, the second user input 11 may be a negative point.

The encoding module 140 may encode the second user input 11 based on the point cloud 110. In one embodiment, the encoding module 140 combines the user input of the current iteration (e.g., the second user input 11) together with the user input of the previous iteration (e.g., the first user input 10). Concatenating values can be encoded.

The electronic device 100 uses the artificial intelligence model 150, which uses the point cloud 110, the encoded first user input, and the encoded second user input as input, to select a plurality of objects in the point cloud 110. The 3D bounding box corresponding to the fields can be inferred.

Referring to FIG. 1B, the artificial intelligence model 150 not only did not infer the 3D bounding box 160a_4 corresponding to the second user input 11, but also 3 with similar characteristics to the 3D bounding box 160a_4. It can be seen that the dimensional bounding box 160a_5 has not been inferred.

According to one embodiment, even though the artificial intelligence model 150 contains information about a small amount of positive points corresponding to the object or a small amount of negative points not corresponding to the object, the 3D bounding box can be inferred with high accuracy.

Figure 2 is a block diagram showing the configuration of an electronic device according to an embodiment. The configuration, operation, and function of the electronic device 200 may correspond to the configuration, operation, and function of the electronic device 100 shown in FIGS. 1A and 1B. For convenience of explanation, content that overlaps with the content described in FIGS. 1A and 1B will be omitted.

Referring to FIG. 2 , the electronic device 200 according to one embodiment may include a communication interface 210, a user interface 220, a processor 230, and a memory 240. However, the components shown in FIG. 2 are not essential components, and the electronic device 200 may omit the components or may further include additional components. For example, the electronic device 200 may be configured to include a display 130, a processor 230, and a memory 240.

The communication interface 210 may support establishing a wired or wireless communication channel between the electronic device 200 and another external electronic device (not shown) or a server (not shown) and performing communication through the established communication channel. In one embodiment, the communication interface 210 receives data from another external electronic device (not shown) or a server (not shown) through wired or wireless communication, or receives data from another external electronic device (not shown) or a server (not shown). Data can be transmitted (not shown). In one embodiment, the communication interface 210 is a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) ) may include a communication module, or a power line communication module), and at least one network (e.g., a short-range communication network (e.g., Bluetooth, WiFi direct, or IrDA (infrared data association)) using any one of the communication modules) Alternatively, the device may communicate with another external electronic device (not shown) or a server (not shown) through a long-distance communication network (e.g., a cellular network, the Internet, or a computer network (e.g., LAN or WAN)).

In one embodiment, the electronic device 200 may receive a point cloud from another external electronic device (not shown) or a server (not shown) through the communication interface 210.

The user interface 220 may include an input interface 221 and an output interface 222.

The input interface 221 is for receiving input from a user (hereinafter referred to as user input). The input interface 221 includes a key pad, a dome switch, and a touch pad (contact capacitive type, pressure resistance type, infrared detection type, surface ultrasonic conduction type, integral tension measurement type, Piezo effect type, etc.), a touch screen, a controller, a jog wheel, a jog switch, a mouse, a keyboard, a wireless pen, a voice recognition device, or a gesture recognition device, but is not limited thereto.

The output interface 222 may include a display 130 for outputting video signals. According to one embodiment, the electronic device 200 may display information related to the electronic device 200 through the display 130. For example, the electronic device 200 may convert a point cloud into a two-dimensional coordinate system and display visualized images on the display 130.

When the display 130 and the touch pad form a layered structure to form a touch screen, the display 130 can be used as an input device in addition to an output device. The display 130 may be a liquid crystal display, a thin film transistor-liquid crystal display, a light-emitting diode (LED), an organic light-emitting diode, or a flexible display. It may include at least one of a flexible display, a 3D display, and an electrophoretic display. Additionally, depending on the implementation form of the electronic device 200, it may include two or more displays.

The processor 230 may be implemented through a combination of a general-purpose processor, such as an application processor (AP), a central processing unit (CPU), or a graphic processing unit (GPU), and software. In the case of a dedicated processor, it may include a memory for implementing an embodiment of the present disclosure, or a memory processing unit for using an external memory.

The processor 230 may be comprised of a plurality of processors. In this case, it may be implemented through a combination of dedicated processors, or it may be implemented through a combination of software and multiple general-purpose processors such as AP, CPU, or GPU.

In one embodiment, the processor 230 may be equipped with a processor dedicated to artificial intelligence (AI). The artificial intelligence-specific processor may be manufactured in the form of a dedicated hardware chip for artificial intelligence, or may be manufactured as part of an existing general-purpose processor (eg, CPU or application processor) or graphics-specific processor (eg, GPU) and used in the electronic device 200. It may be mounted on . For example, an artificial intelligence-specific processor may perform data processing necessary for learning and/or inference related to the artificial intelligence model 150.

Functions related to artificial intelligence according to the present disclosure are operated through the processor 230 and memory 240. The processor 230 may be comprised of one or multiple processors. At this time, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. One or more processors control input data to be processed according to predefined operation rules or an artificial intelligence model (eg, artificial intelligence model 150) stored in memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing the artificial intelligence model 150.

The predefined operation rule or artificial intelligence model 150 is characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above. The memory 240 may store programs for processing and control of the processor 230, and may also store input/output data. The memory 240 may store the learned artificial intelligence model 150.

The memory 240 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), and RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , and may include at least one type of storage medium among optical disks. Additionally, the electronic device 200 may operate a web storage or cloud server that performs a storage function on the Internet.

In one embodiment, the memory 240 may store data processed or scheduled to be processed by the processor 230, firmware, software, and process code. In one embodiment, data and program codes corresponding to at least one of the two-dimensional conversion module 120, the encoding module 140, and the artificial intelligence model 150 may be stored in the memory 240.

Figure 3 is a block diagram showing an inference process using an artificial intelligence model according to an embodiment. The configuration, operation, and function of the artificial intelligence model 150 may correspond to the configuration, operation, and function of the artificial intelligence model 150 shown in FIGS. 1A to 2. For convenience of explanation, content that overlaps with the content described in FIGS. 1A to 2 is omitted.

Referring to FIG. 3, the artificial intelligence model 150 includes a point encoder 310, a centroid aggregation module 320, a spatial click propagation module, and a detection head. (detection head) module may be included. For example, the artificial intelligence model 150 may include multiple neural network layers. The plurality of neural network layers may be divided into a point encoder 310, a centroid aggregation module 320, a spatial click propagation module, and a detection head module according to their functions.

Point encoder 310 may obtain a point cloud and encoded user input. In one embodiment, the point encoder 310 may obtain data linked to the user input encoded in the point cloud. Point encoder 310 encodes a feature (or features) based on the point cloud and encoded user input (e.g., first user input (FIG. 1A, 10) and/or second user input (FIG. 1B, 11)) (may also be referred to as map, feature vector, implicit representation, implicit vector, etc.) can be extracted.

The point encoder 310 may include at least one fully connected layer and at least one downsampling layer. For example, the point encoder 310 may encode a total of N points of the point cloud using a fully connected layer and an activation function. For example, the point encoder 310 can sample encoded points by inputting them to a downsampling layer. In one embodiment, the downsampling layer may sample the encoded points based on the distance (farthest point sampling) or sample the highest n points based on the foreground score. However, the present disclosure is not limited to this, and encoded points may be sampled by max pooling, average pooling, strided convolution, global pooling techniques, etc.

In one embodiment, encoded user input may be concatenated for each output of the downsampling layer. In the present disclosure, a technique in which encoded user input is connected to each output of a downsampling layer may be referred to as dense click guidance. Referring to FIG. 3, it is shown that encoded user input is connected to the output of three downsampling layers, but the number of downsampling layers is not limited to this. In one embodiment, the encoded user input may be connected to the output of at least some of the downsampling layers of the point encoder 310.

According to one embodiment, the artificial intelligence model 150 can perform more efficient and accurate inference by preventing the information corresponding to the user input from being diluted while passing through the downsampling layer.

The centroid aggregation module 320 may receive features extracted from the point encoder 310. The centroid aggregation module 320 may predict the central point corresponding to a plurality of objects in the 3D scene based on the extracted features. The centroid aggregation module 320 may group points corresponding to a plurality of objects based on the predicted central point.

The spatial click propagation module 330 may receive features extracted from the point encoder 310. The spatial click propagation module 330 may receive the output of the centroid aggregation module 320. The spatial click propagation module 330 may calculate information of an object of the same class as the object corresponding to the user input, based on the extracted features (or the output of the centroid aggregation module 320) and the encoded user input. there is.

In one embodiment, the spatial click propagation module 330 may calculate the similarity between information corresponding to the encoded user input and the extracted features. The spatial click propagation module 330 may transmit information corresponding to the similarity to the detection head module 340.

According to one embodiment, information about positive points and/or negative points that were not obtained by user input by the spatial click propagation module 330 is propagated to objects of the same class, so that a three-dimensional bounding box can be efficiently inferred. there is.

The detection head module 340 may receive the output of the centroid aggregation module 320 and the output of the spatial click propagation module 330. In one embodiment, the detection head module 340 may obtain (eg, receive) data in which the output of the centroid aggregation module 320 is connected to the output of the spatial click propagation module 330. In one embodiment, the detection head module 340 may detect an object of the same class as an object corresponding to a user input (eg, a first user input or a second user input) based on the acquired data. The detection head module 340 may output a three-dimensional bounding box corresponding to a plurality of objects based on the output of the centroid aggregation module 320 and the output of the spatial click propagation module 330.

Figure 4 is a conceptual diagram for explaining the operation of a spatial click propagation module according to an embodiment. Referring to FIG. 3 along with FIG. 4 , the spatial click propagation module 330 may receive the encoded user input (E _k ) and the extracted feature (F). The spatial click propagation module 330 may generate a click prototype vector (P _k ) by dot producting the encoded user input (E _k ) and the extracted feature (F). The spatial click propagation module 330 may perform global sum pooling on the click prototype vector (P _k ). The spatial click propagation module 330 may calculate the cosine similarity (M _k ) between the extracted feature (F) and the pooled click prototype vector (P _k ). The spatial click propagation module 330 may generate an association map for each class based on cosine similarity (M _k ). The class-specific relevance map may be connected to the output of the centroid aggregation module 320.

Figure 5 is a block diagram showing the learning process of an artificial intelligence model according to an embodiment. Referring to FIG. 5, the configuration, operation, and functions of the point encoder 310, centroid aggregation module 320, spatial click propagation module 330, and detection head module 340 are similar to those of the point encoder of FIG. 3. It can correspond to the configuration, operation, and function of the centroid aggregation module 320, the spatial click propagation module 330, and the detection head module 340. Therefore, content that overlaps with the content described in FIGS. 3 and 4 will be omitted.

In the learning process of the artificial intelligence model 150, user input is not provided. Therefore, data corresponding to the positive point 520 and/or negative point 530 must be input into the artificial intelligence model 150. In one embodiment, positive points 520 may be randomly selected from the ground truth values of the point cloud dataset. In one embodiment, negative point 530 may be generated by the output of negative click simulation module 510. In one embodiment, encoding module 140 may encode positive points 520 and/or negative points 530 based on the point cloud.

The artificial intelligence model 150 is applied to a point cloud dataset, at least one positive point 520 corresponding to an object in the point cloud dataset, and a background (or object with inconsistent class information) in the point cloud dataset. Based on the corresponding at least one negative point 530, it can be learned to output a 3D bounding box corresponding to the object.

In one embodiment, the artificial intelligence model 150 may include a negative click simulation module 510. During the learning process of the artificial intelligence model 150, the negative click simulation module 510 determines the foreground score among the background points that do not correspond to the plurality of objects based on the ground truth value and the extracted features. ) points that exceed the threshold can be assigned as negative points. In one embodiment, the threshold may be predefined by user or manufacturer settings.

Figure 6 is a conceptual diagram for explaining the operation of a negative click simulation module according to an embodiment. Referring to FIG. 5 along with FIG. 6 , the negative click simulation module 510 may calculate a foreground score based on the extracted features. For example, the negative click simulation module 510 may calculate a foreground score based on the extracted features. The negative click simulation module 510 may refer to the ground truth value and determine the point at which the foreground score exceeds the threshold among the background points. Negative click simulation module 510 may assign the determined points as negative points (or may be referred to as negative clicks). The assigned negative point may be encoded by the encoding module 140. The encoded negative points can be utilized in the next iteration of training of the artificial intelligence model 150.

7A and 7B are diagrams exemplarily showing the operation of a 2D conversion module according to an embodiment. Referring to FIG. 1A along with FIGS. 7A and 7B, the 2D conversion module 120 can generate a 2D point group 720 by converting the point cloud 710 composed of a 3D scene into a 2D coordinate system. there is.

Referring to FIG. 7A, the point cloud 710 may be composed of points in x, y, and z coordinate systems. Assume that user inputs 12 and 13 are received in a 3D coordinate system using a point cloud 710. Although the user attempted to modify the user input 12 in the positive x-axis direction, the point may be moved in an unintended axis direction (eg, y-axis) and the user input 13 may be received.

Referring to FIG. 7B, it is assumed that a two-dimensional point group 720 is displayed through an output interface (eg, display). That is, assume that user input (14, 15) is received in a two-dimensional coordinate system. Since the user attempted to modify the user input 14 in the positive x-axis direction and does not consider the depth value, the point is moved in the intended axis direction so that the user input 15 can be received as intended.

According to one embodiment, the point cloud is converted into a 2D coordinate system by the 2D conversion module 120 and provided to the user, so that user input can be received quickly, easily, and accurately.

8A and 8B are diagrams exemplarily showing the operation of an encoding module according to an embodiment. Referring to Figures 1A, 1B, 3, and 5, along with Figures 8A and 8B, encoding module 140 may encode user input (i.e., positive points and/or negative points). Encoded user input may represent a distance-based heatmap.

The encoding module 140 may encode user input based on a point cloud composed of a given N number of points. For example, a point cloud is It can be expressed as For example, user input may be It can be expressed as The encoded user input can be expressed as E _k , and the method of encoding the user input can follow Equation 1.

Referring to Equation 1, d is defined as the two-dimensional Euclidean distance between p _k and (x _i , y _i ), e.g. It can be calculated as may be a hyperparameter that controls the distance threshold.

In one embodiment, encoded user input may be encoded by class. The number of classes is defined in C, e.g. It can be. User input encoded for each class can be expressed as U _c and can follow Equation 2.

Once C encoded user inputs are generated, the point cloud contains the encoded user inputs by class (i.e. ) can be connected.

Referring to FIG. 8A, the user input may be at least one point corresponding to an object 16 or 17 of the vehicle class. Referring to FIG. 8B, the encoding module 140 may generate distance-based heatmaps 18 and 19 corresponding to objects 16 and 17 of the vehicle class. Values corresponding to the distance-based heatmaps 18 and 19 may be connected to the point cloud and input into the artificial intelligence model 150.

9A to 9D are diagrams exemplarily showing the operation of a spatial click propagation module according to an embodiment. Referring to FIGS. 3 and 5 along with FIGS. 9A to 9D, information about objects corresponding to user input can be used to detect objects of the same class in the point cloud.

Referring to FIG. 9A, a value corresponding to an encoded user input (or an encoded positive point or an encoded negative point) may be input to the point encoder 310. For example, the encoded user input may correspond to a distance-based heatmap indicated by the arrows shown in FIG. 9A.

Referring to FIG. 9B, information corresponding to the encoded user input may be sampled through the downsampling layer of the point encoder 310. Referring to FIG. 9C, the spatial click propagation module 330 may receive downsampled user input. The spatial click propagation module 330 may calculate similarity between points corresponding to the downsampled user input and points of the downsampled point cloud. The spatial click propagation module 330 may generate a correlation map based on similarity.

Referring to FIG. 9D, not only the points corresponding to the user input but also the points determined by the spatial click propagation module 330 can be inferred as the 3D bounding box of the corresponding class.

Figure 10 is a flowchart showing a method of operating an electronic device according to an embodiment. Content that overlaps with the content described in FIGS. 1 to 9D will be omitted. For convenience of explanation, FIG. 10 will be described with reference to 1a and 2.

Referring to FIG. 10 , steps S1010 to S1060 may be performed by the electronic devices 100 and 200 or the processor 230 of the electronic device 200. The method of operating the electronic devices 100 and 200 according to the present disclosure is not limited to that shown in FIG. 10, and any one of the steps shown in FIG. 10 may be omitted, and may further include steps not shown in FIG. 10. You may.

In one embodiment, steps S1010 to S1060 may correspond to an initial iteration of the artificial intelligence model 150 for inferring a 3D bounding box.

In step S1010, the electronic devices 100 and 200 may acquire a point cloud corresponding to a 3D scene including a plurality of objects. For example, the point cloud may be received from an external device or server. For example, the point cloud may be acquired by a sensor of the electronic devices 100 and 200.

In step S1020, the electronic devices 100 and 200 may convert the point cloud into a two-dimensional coordinate system. For example, a point cloud can be expressed in an (x, y, z) coordinate system, and the transformed point cloud can be expressed in an (x, y) coordinate system. In one embodiment, the points of the point cloud of the electronic devices 100 and 200 may be projected and converted into a two-dimensional coordinate system.

In step S1030, the electronic devices 100 and 200 may display the converted point cloud on the display 130. For example, the electronic devices 100 and 200 may output an image corresponding to the converted point cloud to the display 130.

In step S1040, the electronic device 100 or 200 may obtain a first user input including at least one point corresponding to a plurality of objects among the converted point cloud. For example, the first user input may be at least one positive point. However, the present disclosure is not limited to this, and the first user input may include at least one negative point.

In step S1050, the electronic devices 100 and 200 may encode the first user input based on the point cloud. In one embodiment, the encoded user input may be connected to the point cloud and input into the artificial intelligence model 150.

In step S1060, the electronic devices 100 and 200 use the artificial intelligence model 150 that uses the point cloud and the encoded first user input as input to create a three-dimensional bounding box corresponding to a plurality of objects in the point cloud. can be inferred.

Figure 11 is a flowchart showing a method of operating an electronic device according to an embodiment. Content that overlaps with the content described in FIGS. 1 to 10 will be omitted. For convenience of explanation, FIG. 11 will be described with reference to 1b and 2.

Referring to FIG. 11, after step S1060 of FIG. 10, steps S1110 to S1140 may be performed by the electronic devices 100 and 200 or the processor 230 of the electronic device 200. The method of operating the electronic devices 100 and 200 according to the present disclosure is not limited to that shown in FIG. 11, and any one of the steps shown in FIG. 11 may be omitted, and may further include steps not shown in FIG. 11. You may.

In one embodiment, steps S1110 to S1140 may correspond to one of the iterations after the initial iteration of the artificial intelligence model 150 for inferring a 3D bounding box.

In step S1110, the electronic devices 100 and 200 may display the converted point cloud and the inferred 3D bounding box on the display 130. For example, the electronic devices 100 and 200 may generate a two-dimensional image using data obtained by converting the inferred three-dimensional bounding box into two dimensions along with the converted point cloud. The electronic devices 100 and 200 may display a two-dimensional image on the display 130.

In step S1120, the electronic device 100 or 200 selects a point (i.e., a negative point) that corresponds to the inferred three-dimensional bounding box but does not correspond to a plurality of objects, and a point that does not correspond to the inferred three-dimensional bounding box but does not correspond to a plurality of objects. A second user input including at least one of points (ie, positive points) corresponding to a plurality of objects may be obtained. The number of negative points and positive points may be at least one.

In step S1140, the electronic devices 100 and 200 may encode the second user input based on the point cloud. In one embodiment, the electronic devices 100 and 200 may encode the first user input and the second user input of the previous iteration together. In one embodiment, the electronic devices 100 and 200 may connect the encoded first and second user inputs to the point cloud.

In step S1140, the electronic devices 100 and 200 correspond to a plurality of objects in the point cloud using an artificial intelligence model that uses the point cloud, the encoded first user input, and the encoded second user input as input. A 3D bounding box can be inferred.

In one embodiment, unless a predefined number of iterations is reached or a user input corresponding to the end of the iteration is not received, the electronic devices 100 and 200 repeat the operations corresponding to steps S1110 to S1140. You can.

In one embodiment, the method according to the embodiments of the present disclosure may be included and provided in a computer program product. Computer program products may be distributed in the form of machine-readable recording media. Alternatively, the computer program product may be distributed through an application store or directly between multiple devices.

In the above-described embodiments, components according to the technical idea of the present disclosure have been described using terms such as first, second, and third. However, terms such as first, second, third, etc. are used to distinguish components from each other and do not limit the technical idea of the present disclosure. Terms such as first, second, third, etc. do not imply order or any form of numerical meaning.

The above-described embodiments are specific embodiments for carrying out the present disclosure. It should be understood that the present disclosure includes not only the above-described embodiments, but also embodiments that are simply designed or can be easily changed using the above-described embodiments. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of the present invention as well as the claims described below.

Claims (10)

In an electronic device that performs 3D object detection using an artificial intelligence model,
display;
A memory that stores one or more instructions; and
At least one processor executing the one or more instructions stored in the memory, wherein the at least one processor includes:
Obtain a point cloud corresponding to a three-dimensional scene including a plurality of objects,
Convert the point cloud into a two-dimensional coordinate system,
Displaying the converted point cloud on the display,
Obtaining a first user input including at least one point corresponding to the plurality of objects among the converted point cloud,
encode the first user input based on the point cloud,
Inferring a three-dimensional bounding box corresponding to the plurality of objects in the point cloud using an artificial intelligence model using the point cloud and the encoded first user input as input, the one or more An electronic device that executes instructions.
The method of claim 1, wherein the at least one processor:
Displaying the converted point cloud and the inferred 3D bounding box on the display,
A second comprising at least one of a point corresponding to the inferred 3D bounding box but not corresponding to the plurality of objects, and a point not corresponding to the inferred 3D bounding box but corresponding to the plurality of objects. obtain user input,
encode a second user input based on the point cloud,
Inferring a three-dimensional bounding box corresponding to the plurality of objects in the point cloud using an artificial intelligence model using the point cloud, the encoded first user input, and the encoded second user input as input. An electronic device that further executes the one or more instructions.
The method of claim 1, wherein the artificial intelligence model is:
a point encoder that extracts features based on the point cloud and the encoded first user input;
a centroid aggregation module that predicts a central point corresponding to the plurality of objects based on the extracted features and groups points corresponding to the plurality of objects based on the predicted central point;
A spatial click propagation module that calculates information of an object of the same class as an object corresponding to the first user input based on the extracted features and the encoded first user input; and
An electronic device comprising a detection head module that outputs the three-dimensional bounding box corresponding to the plurality of objects based on the output of the centroid aggregation module and the output of the spatial click propagation module.
According to paragraph 3,
The point encoder includes at least one fully connected layer and at least one downsampling layer,
The at least one processor,
The electronic device further executes the one or more instructions, concatenating the encoded first user input to an output of the at least one downsampling layer.
According to paragraph 3,
The spatial click propagation module,
Calculate similarity between information corresponding to the encoded first user input and the extracted features,
The detection head module is,
Obtain data with the similarity linked to the output of the centroid aggregation module,
An electronic device detecting an object of the same class as an object corresponding to the first user input based on the obtained value.
According to paragraph 3,
The artificial intelligence model includes a point cloud dataset, at least one positive point corresponding to an object in the point cloud dataset, and at least one negative point corresponding to a background in the point cloud dataset. Based on this, an electronic device is trained to output a three-dimensional bounding box corresponding to an object.
According to clause 6,
The artificial intelligence model is:
In the learning process of the artificial intelligence model, based on the ground truth value and the extracted features, the points whose foreground scores exceed a threshold among the background points that do not correspond to the plurality of objects are selected as the at least one negative. An electronic device further comprising a negative click simulation module for assigning to a point.
According to paragraph 1,
The first user input includes object location information and object class information.
According to clause 8,
The one or more instructions encoding the first user input include:
An electronic device comprising the one or more instructions, encoding the first user input based on a distance between the (x, y) location information of the point cloud and the object location information of the first user input.
According to paragraph 1,
The artificial intelligence model is an electronic device that inputs a value obtained by connecting the encoded first user input to the point cloud.