KR102433287B1 - Method for, device for, and system for generating an learning set - Google Patents

Abstract

A learning set generating method according to an embodiment of the present application includes the steps of: performing calibration on at least two cameras; acquiring a first image captured from a first camera and a second image captured from a second camera; recognizing a target object from the first image and acquiring first object information related to the target object; recognizing the target object from the second image and acquiring second object information related to the target object; acquiring a three-dimensional target bounding box related to the target object from the first object information and the second object information; acquiring sensing data corresponding to the target bounding box, wherein the sensing data is acquired through at least one of a LIDAR sensor and a radar sensor; and generating a learning set for learning a neural network model for detecting an object included in the sensing data, based on the sensing data and the target bounding box. The present invention is to train a neural network model that detects an object from sensing data obtained from a radar sensor or LIDAR sensor.

Description

A method for generating a learning set, an apparatus for generating a learning set, and a system for generating a learning set

The present application relates to a learning set generating method, a learning set generating apparatus, and a learning set generating system. Specifically, the present application relates to a learning set generating method, a learning set generating apparatus, and a learning set generating system for learning a neural network model for detecting an object from sensing data including radar data or lidar data. .

As artificial intelligence technology develops, research and development of artificial intelligence models for use in various industries are increasing. In particular, in relation to the automobile industry, interest in artificial intelligence models that recognize objects based on data measured using various sensors (eg, radar sensors, lidar sensors, etc.) and automatically control cars based on the object recognition results this is rising

One of the most difficult and important tasks to train an artificial intelligence model is the labeling task on the training set for training the artificial intelligence model. Conventionally, a method in which an expert manually assigns or annotates object information to a point cloud included in sensing data obtained from a radar sensor or a lidar sensor is adopted. However, the conventional method has a problem in that it takes considerable time and cost to prepare a training set. In addition, the radar sensor has a relatively low resolution due to the characteristics of the sensor, so it is difficult to infer the exact shape of the object based on the point cloud included in the radar data, making labeling impossible, or even if the labeling is done manually, the accuracy is low. There were restrictions.

Accordingly, there is a need for development of a learning set generating method, apparatus and system for automatically generating a learning set from sensing data.

One problem to be solved by the present invention is to provide a learning set generating method, a learning set generating apparatus, and a learning set generating system for training a neural network model for detecting an object from sensing data obtained from a radar sensor or a lidar sensor. .

The problem to be solved by the present invention is not limited to the above-described problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

A method for generating a learning set according to an embodiment of the present application includes: performing calibration on at least two or more cameras; obtaining a first image photographed from a first camera and a second image photographed from a second camera; recognizing a target object from the first image and obtaining first object information related to the target object; recognizing the target object from the second image and obtaining second object information related to the target object; obtaining a three-dimensional target bounding box related to the target object in the first object information and the second object information; obtaining sensing data corresponding to the target bounding box, wherein the sensing data is obtained through at least one of a lidar sensor and a radar sensor; and generating a training set for training a neural network model for detecting an object included in the sensing data based on the sensing data and the target bounding box.

A learning set generating apparatus according to an embodiment of the present application includes a transceiver for communicating with at least one of a first camera, a second camera, and a sensor; and a controller configured to obtain image data through the first camera and the second camera, obtain sensing data through the sensor, and generate a learning set based on the image data and the sensing data. The controller obtains a first image photographed from the first camera and a second image photographed from the second camera, recognizes a target object from the first image, and obtains first object information related to the target object, , Recognizes the target object from the second image, obtains first object information related to the target object, and based on the first object information and the second object information, a three-dimensional target bounding box related to the target object to obtain, and sensor data corresponding to the target bounding box, wherein the sensor data is acquired through at least one of a lidar sensor and a radar sensor, is obtained, and based on the sensor data and the target bounding box, the sensing It may be configured to generate a training set for training a neural network model to detect objects included in data.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to the method, apparatus, and system for generating a training set according to an embodiment of the present application, time and cost required for preparing a training set can be reduced by associating image data with a labeling operation for sensing data.

According to the method, apparatus, and system for generating a learning set according to an embodiment of the present application, a more sophisticated learning set can be generated by obtaining a bounding box related to a target object using a similarity between characteristic information of an object or a landmark of the object have.

Effects of the present invention are not limited to the above-described effects, and effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and accompanying drawings.

1 is a schematic diagram of a system for generating a learning set according to an embodiment of the present application;
2 is a diagram illustrating operations of a learning set generating system according to an embodiment of the present application.
3 is a flowchart illustrating a method of generating a training set according to an embodiment of the present application.
4 is a flowchart detailing the step of performing calibration of at least two or more cameras according to an embodiment of the present application.
5 is a flowchart detailing a step of performing calibration on internal parameters of a camera according to an embodiment of the present application.
6 is a diagram illustrating an example of a first calibration reference object according to an embodiment of the present application.
7 is a flowchart detailing a step of performing calibration on an external parameter of a camera according to an embodiment of the present application.
8 is a diagram illustrating an aspect of performing camera calibration using a second calibration reference object according to an embodiment of the present application.
9 is a flowchart detailing the step of obtaining a target bounding box according to an embodiment of the present application.
10 is a flowchart detailing the step of obtaining a target bounding box related to a target object according to an embodiment of the present application.
11 is a diagram illustrating an aspect of obtaining a three-dimensional target bounding box according to an embodiment of the present application.
12 is a flowchart detailing the step of obtaining a target bounding box related to a target object according to an embodiment of the present application.
13 is a diagram illustrating an aspect of determining a three-dimensional target bounding box according to an embodiment of the present application.
14 is a flowchart detailing the step of obtaining a target bounding box related to a target object according to another embodiment of the present application.

The above-mentioned objects, features and advantages of the present application will become more apparent from the following detailed description in conjunction with the accompanying drawings. However, since the present application may have various changes and may have various embodiments, specific embodiments will be exemplified in the drawings and described in detail below.

Throughout the specification, like reference numerals refer to like elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present application may unnecessarily obscure the gist of the present application, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are merely identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "part" for the components used in the following embodiments are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

In the following examples, the singular expression includes the plural expression unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have means that the features or components described in the specification are present, and the possibility that one or more other features or components may be added is not excluded in advance.

In the drawings, the size of the components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, and the present invention is not necessarily limited to the illustrated bar.

In cases where certain embodiments are otherwise implementable, the order of specific processes may be performed differently from the order in which they are described. For example, two processes described in succession may be performed substantially simultaneously, or may be performed in an order opposite to the order described.

In the following embodiments, when components are connected, it includes not only cases in which components are directly connected, but also cases in which components are interposed between components and connected indirectly.

For example, in the present specification, when it is said that components and the like are electrically connected, it includes not only the case where the components are directly electrically connected, but also the case where the components are interposed therebetween to be indirectly electrically connected.

A method for generating a learning set according to an embodiment of the present application includes: performing calibration on at least two or more cameras; obtaining a first image photographed from a first camera and a second image photographed from a second camera; recognizing a target object from the first image and obtaining first object information related to the target object; recognizing the target object from the second image and obtaining second object information related to the target object; obtaining a three-dimensional target bounding box related to the target object in the first object information and the second object information; obtaining sensing data corresponding to the target bounding box, wherein the sensing data is obtained through at least one of a lidar sensor and a radar sensor; and generating a training set for training a neural network model for detecting an object included in the sensing data based on the sensing data and the target bounding box.

According to an embodiment of the present application, the step of obtaining a three-dimensional target bounding box related to the target object may include reversely projecting the first bounding box included in the first object information onto 3D spatial coordinates (Revere project). to obtain a first projection volume; obtaining a second projected volume by reverse projecting a second bounding box included in the second object information onto 3D spatial coordinates; and obtaining an intersection volume between the first projection volume and the second projection volume based on the first projection volume and the second projection volume, and obtaining the target bounding box associated with the target object based on the intersection volume may include;

According to an embodiment of the present application, the acquiring the target bounding box related to the target object includes: acquiring at least one first plane constituting the first projection volume; obtaining at least one said second plane constituting said second projection volume; and acquiring a common plane of the first plane and the second plane, and acquiring the target bounding box based on the common plane.

According to an embodiment of the present application, the obtaining of the three-dimensional target bounding box related to the target object includes at least two or more three-dimensional temporary bounding boxes based on the first projection volume and the second projection volume. obtaining; obtaining first characteristic information related to the first bounding box; obtaining second characteristic information related to the second bounding box; allocating a matching score to the temporary bounding box based on a degree of similarity between the first characteristic information and the second characteristic information; and determining the target bounding box from among the temporary bounding boxes based on the matching score.

According to an embodiment of the present application, the obtaining of the three-dimensional target bounding box related to the target object includes projecting the first cell information related to the target object included in the first object information onto 3D spatial coordinates. to obtain a first projection straight line; obtaining a first projection straight line by projecting second cell information related to the target object included in the second object information onto 3D spatial coordinates; and obtaining a three-dimensional target bounding box related to the target object based on the first projection straight line and the second projection straight line.

According to an embodiment of the present application, the step of obtaining a three-dimensional target bounding box related to the target object on the basis of the first projection straight line and the second projection straight line includes the first projection straight line and the second projection straight line. obtaining at least two or more three-dimensional temporary bounding boxes including a first temporary bounding box and a second temporary bounding box based on the ; calculating a distance between the first projection straight line and the second projection straight line; allocating a first score to the first temporary bounding box and a second score to the second temporary bounding box based on the calculated distance; and determining at least one of the first temporary bounding box and the second temporary bounding box as a target bounding box by comparing the first score and the second score.

According to an embodiment of the present application, the performing the calibration on the at least two or more cameras includes performing calibration of the internal parameters of the at least two or more cameras, and the calibration of the internal parameters of the at least two or more cameras The performing may include: acquiring an image of a first calibration reference object through the camera; obtaining a first set of pixels associated with the first calibration reference object based on the image; obtaining coordinate system information of the camera; and correcting an intrinsic parameter of the camera based on the coordinate system information and the first pixel set.

According to an embodiment of the present application, performing the calibration on the at least two or more cameras includes performing calibration of external parameters of the at least two or more cameras, and calibration of the external parameters of the at least two or more cameras The performing may include: obtaining reference coordinate information for a second calibration reference object, wherein the reference coordinate information is related to a sensor-based coordinate system of at least one of a lidar sensor and a radar sensor; acquiring an image of the second calibration reference object through the camera; obtaining a second set of pixels associated with the second calibration reference object based on the image; obtaining a coordinate pair based on the reference coordinate information and the second set of pixels; and performing a correction on an external parameter of the camera based on the coordinate pair.

According to an embodiment of the present application, a computer-readable recording medium in which a program for executing the learning set generating method is recorded may be provided.

A learning set generating apparatus according to an embodiment of the present application includes a transceiver for communicating with at least one of a first camera, a second camera, and a sensor; and a controller configured to obtain image data through the first camera and the second camera, obtain sensing data through the sensor, and generate a learning set based on the image data and the sensing data. The controller obtains a first image photographed from the first camera and a second image photographed from the second camera, recognizes a target object from the first image, and obtains first object information related to the target object, , Recognizes the target object from the second image, obtains first object information related to the target object, and based on the first object information and the second object information, a three-dimensional target bounding box related to the target object to obtain, and sensor data corresponding to the target bounding box, wherein the sensor data is acquired through at least one of a lidar sensor and a radar sensor, is obtained, and based on the sensor data and the target bounding box, the sensing It may be configured to generate a training set for training a neural network model to detect objects included in data.

Hereinafter, a learning set generating method, a learning set generating apparatus, and a learning set generating system of the present application will be described with reference to FIGS. 1 to 13 .

1 is a schematic diagram of a training set generation system 10 according to an embodiment of the present application.

The training set generating system 10 according to an embodiment of the present application includes at least one camera (eg, the first camera 110 and the second camera 120 ), the sensor 200 and the server 1000 , or the training set. generating device).

At least one camera may acquire an image of the target object and transmit the image to the server 1000 through an arbitrary transceiver. For example, the first camera 110 may be disposed at a first position with respect to the target object to acquire a first image related to the target object at a first angle (or a first inclination). As another example, the second camera 120 is disposed at a second position different from the first position (or a second position adjacent to the first position) with respect to the target object at a second angle (or second inclination) to the target object. A second image related to may be acquired.

Meanwhile, in FIG. 1 , the training set generating system 10 is illustrated as including a first camera 110 and a second camera 120 . However, this is only an example for convenience of description, and the learning set generating system 10 may be configured to include any suitable number of cameras. For example, the training set creation system 10 may include a single camera. As another example, the training set generation system 10 may include at least two or more cameras.

The sensor 200 may acquire sensing data related to a target object and transmit the sensing data to the server 1000 through an arbitrary transceiver. For example, the sensor 200 may be at least one of a radar sensor and/or a lidar sensor. However, this is only an example, and the sensor 200 may be any sensor for detecting an object other than a radar sensor and/or a lidar sensor.

Meanwhile, the sensor 200 may be disposed at a third position with respect to the target object to acquire sensing data for the target object. Here, the third location may be adjacent to or different from the first location and/or the second location.

The server 1000 or the learning set generating apparatus (hereinafter referred to as the server 1000 ) may include a transceiver 1100 , a memory 1200 , and a controller 1300 .

The transceiver 1100 of the server 1000 may communicate with any external device including at least one camera 110 , 120 and a sensor 200 . For example, the server 1000 may acquire image data from the at least one camera 110 , 120 through the transceiver 1100 . Also, the server 1000 may acquire sensing data from the sensor 200 through the transceiver 1100 .

The server 1000 may access a network through the transceiver 1100 to transmit/receive various data. The transceiver 1100 may largely include a wired type and a wireless type. Since the wired type and the wireless type have their respective strengths and weaknesses, the server 1000 may be provided with both the wired type and the wireless type at the same time in some cases. Here, in the case of the wireless type, a wireless local area network (WLAN)-based communication method such as Wi-Fi may be mainly used. Alternatively, in the case of the wireless type, cellular communication, for example, LTE, 5G-based communication method may be used. However, the wireless communication protocol is not limited to the above-described example, and any suitable wireless type communication method may be used. In the case of the wired type, LAN (Local Area Network) or USB (Universal Serial Bus) communication is a representative example, and other methods are also possible.

The memory 1200 of the server 1000 may store various types of information. Various data may be temporarily or semi-permanently stored in the memory 1200 . Examples of the memory 1200 include a hard disk (HDD), a solid state drive (SSD), a flash memory, a read-only memory (ROM), a random access memory (RAM), and the like. This can be. The memory 1200 may be provided in a form embedded in the server 1000 or in a form detachable. The memory 1200 may store various data necessary for the operation of the server 1000 , including an operating program (OS) for driving the server 1000 or a program for operating each configuration of the server 1000 . have.

The controller 1300 (or processor) may control the overall operation of the server 1000 . For example, the controller 1300 performs an operation of calibrating an internal parameter or an external parameter of the cameras 110 and 120 to be described later, acquiring an image from the cameras 110 and 120 or acquiring sensing data from the sensor 200 . Operation, operation of recognizing a target object from an image and obtaining object information related to the target object, operation of obtaining a three-dimensional target bounding box based on object information, and generating a learning set based on the target bounding box and sensing data It is possible to control the overall operation of the server 1000, such as the operation. Specifically, the controller 1300 may load and execute a program for the overall operation of the server 1000 from the memory 1200 . The controller 1300 may be implemented as an application processor (AP), a central processing unit (CPU), a microcontroller unit (MCU), or a similar device according to hardware, software, or a combination thereof. In this case, in terms of hardware, it may be provided in the form of an electronic circuit that performs a control function by processing an electrical signal, and in software, it may be provided in the form of a program or code for driving a hardware circuit.

Hereinafter, an operation of the learning set generating system 10 according to embodiments of the present application will be described in detail with reference to FIGS. 2 to 13 .

2 is a diagram illustrating operations of the learning set generating system 10 according to an embodiment of the present application.

The server 1000 of the training set generating system 10 according to an embodiment of the present application includes internal parameters and/or external parameters of at least one camera (eg, the first camera 110 or the second camera 120). can be corrected.

As an example, the server 1000 acquires an image of the first calibration reference object through at least one camera, and based on the feature points included in the image of the first calibration reference object, Pixel coordinate information may be obtained. Also, the server 1000 may acquire coordinate system information of the camera. In this case, the server 1000 may perform an operation of correcting an internal parameter of at least one camera based on the coordinate system information of the camera and the pixel coordinate information related to the first calibration reference object.

As another example, the server 1000 may obtain reference coordinate information for the second calibration reference object. Here, the reference coordinate information may mean to encompass coordinate information related to a sensor-based coordinate system, including a radar sensor or a lidar sensor. In addition, the server 1000 may acquire a feature point included in the image of the second calibration reference object through at least one camera, and obtain pixel coordinate information of the feature point related to the second calibration reference object. Also, the server 1000 may obtain a coordinate pair by matching the reference coordinate information and the pixel coordinate information, and may perform an operation of correcting an external parameter of the camera based on the coordinate pair.

The operation of the learning set generating system 10 related to camera calibration will be described in detail with reference to FIGS. 4 to 8 .

The server 1000 of the training set generating system 10 according to an embodiment of the present application obtains an image from at least one camera (eg, the first camera 110 or the second camera 120) or the sensor 200 ) from which sensing data can be obtained. For example, the server 1000 may acquire an image including a target object through a camera. Also, the server 1000 may acquire sensing data obtained by measuring data related to the target object through the sensor 200 .

Meanwhile, although not shown in FIG. 2 , the server 1000 of the learning set generating system 10 according to an embodiment of the present application provides frame information (eg, number of frames information, frame time information), meta information through the sensor 200 . Any data including information and the like can be obtained. Here, the server 1000 may perform an operation of matching the sync between the sensing data and the image data acquired through the camera based on frame information, meta information, and the like.

The server 1000 of the learning set generating system 10 according to an embodiment of the present application detects (or recognizes) a target object for each frame of an image acquired through a camera, and obtains object information related to the target object can be performed. Here, the object information related to the target object includes information about a two-dimensional bounding box related to the target object and/or landmarks of the target object (eg, eyes, nose, ears, shoulders, elbows, hands, pelvis, knees, feet, etc.) ) may include arbitrary image information related to the target object including image information (eg, image coordinate information, etc.) corresponding to the .

According to an embodiment, the server 1000 detects the target object from the first image obtained from the first camera 110 and calculates a two-dimensional first bounding box related to the target object based on the target object detection result. can Also, the server 1000 may detect the target object from the second image obtained from the second camera 120 and acquire a two-dimensional second bounding box related to the target object based on the target object detection result. For example, the server 1000 may detect a target object from an image by using a deep learning technique (eg, yolo, yoloV3, ssd, Retinanet, etc.) and obtain a mask of a region corresponding to the target object. Also, the server 1000 may perform an operation of calculating a bounding box related to the target object based on the mask. As another example, the server 1000 may utilize any technique for increasing the performance of target object detection. For example, the server 1000 utilizes a Simple Online and Realtime Tracking (hereinafter referred to as SORT) algorithm in which a Kalman filter is added to an object detection technique to increase the accuracy of target object detection and perform an operation to track the location of an object. can be implemented to do so.

According to another exemplary embodiment, the server 1000 detects the target object from the first image acquired from the first camera 110 , and performs pose estimation on the detected target object to determine the landmark of the target object. related object information (eg, first cell information) may be acquired. In addition, the server 1000 detects the target object from the second image acquired from the second camera 120 , and performs pose estimation on the detected target object to obtain object information ( Yes, the second cell information) can be obtained. For example, the server 1000 may extract image information related to a landmark of the target object from within a two-dimensional bounding box related to the target object by utilizing a deep learning technique (eg, AlphaPose, etc.).

However, the above-described object detection technique and pose estimation technique are merely examples, and the server 1000 may be implemented to detect a target object from an image by using any suitable method.

The server 1000 of the learning set generating system 10 according to an embodiment of the present application is based on object information related to the target object (eg, image information corresponding to a bounding box or a landmark of the target object, etc.) in 3D An operation of obtaining a target bounding box of .

As an example, the server 1000 calculates the three-dimensional target bounding box related to the target object based on the first bounding box related to the target object of the first image and the second bounding box related to the target object of the second image. can be done In more detail, the server 1000 may reversely project the first bounding box onto the 3D spatial coordinates to obtain a first projected volume related to the target object. Also, the server 1000 may reversely project the second bounding box onto the 3D spatial coordinates to obtain a second projected volume related to the target object. In this case, the server 1000 may acquire an intersection volume based on the first projection volume and the second projection volume, and acquire a three-dimensional target bounding box related to the target object based on the intersection volume.

As another example, the server 1000 may store cell information (eg, cell coordinates, etc.) of the first image corresponding to the landmarks (eg, eyes, nose, ears, shoulders, elbows, hands, pelvis, knees, feet, etc.) of the target object. ) and cell information (eg, cell coordinates, etc.) of the second image, an operation of calculating a three-dimensional target bounding box related to the target object may be performed. In more detail, the server 1000 may acquire the first projection straight line by projecting the first cell coordinates of the first image corresponding to the landmark of the target object onto the 3D spatial coordinates. Also, the server 1000 may obtain a second projection straight line by projecting the second cell coordinates of the second image corresponding to the landmark of the target object onto the 3D spatial coordinates. At this time, the server 1000 calculates the intersection point between the first projection straight line and the second projection straight line, and the intersection points corresponding to the landmark of at least one target object (or the closest to the first projection straight line and the second projection straight line) Based on the three-dimensional space coordinates), a three-dimensional target bounding box related to the target object may be obtained.

The operation of acquiring the target bounding box will be described in more detail with reference to FIGS. 9 to 14 .

The server 1000 of the learning set generating system 10 according to an embodiment of the present application may obtain sensing data corresponding to a three-dimensional target bounding box related to a target object through the sensor 2000 . In this case, the server 1000 may perform an operation of generating a training set for training a neural network model that outputs a three-dimensional target bounding box related to a target object from the sensing data based on the target bounding box and the sensing data.

Meanwhile, although not shown in FIG. 2 , the server 1000 of the learning set generating system 10 according to an embodiment of the present application acquires the first image obtained from the first camera 110 and the second camera 120 . An operation of synchronizing a second image and/or sensing data acquired from the sensor 200 may be performed.

For example, the server 1000 reads an image broadcast on the web from the first camera 110 or the second camera 120 to obtain the first image and the second image obtained from the first camera 110 . It may be implemented to synchronize the second images obtained from the camera 120 . For example, the server 1000 may be implemented to acquire the first image and the second image by using OBS studio 27, and to synchronize the time of the first image and the second image based on a time reference of a common server.

As another example, as described above, the server 1000 may acquire arbitrary data including frame information (eg, frame number information, frame time information) and/or meta information through the sensor 200 . In this case, the server 1000 may be implemented to synchronize time between the image data and the sensed data based on frame information included in the sensed data.

However, the above is only an example, and the server 1000 may be implemented to perform an operation of synchronizing image data obtained from at least one camera and sensing data obtained from a sensor by using any suitable method. can

Hereinafter, a method for generating a learning set according to an embodiment of the present application will be described in detail with reference to FIGS. 3 to 13 .

3 is a flowchart illustrating a method of generating a training set according to an embodiment of the present application.

The method of generating a learning set according to an embodiment of the present application includes: performing calibration of at least two or more cameras (S1000), a first image photographed from the first camera and a second image photographed from the second camera obtaining (S2000), recognizing a target object from a first image and obtaining first object information related to the target object (S3000), recognizing a target object from a second image and obtaining second object information related to the target object acquiring (S4000), acquiring a three-dimensional target bounding box related to the target object based on the first object information and the second object information (S5000), acquiring sensing data corresponding to the target bounding box ( S6000), and generating a training set for training the neural network model (S7000).

In performing the calibration of at least two or more cameras ( S1000 ), the server 1000 according to an embodiment of the present application may correct the internal parameters of the first camera 110 or the second camera 120 , respectively. . For example, the server 1000 may correct the internal parameters of the cameras 110 or 120 based on the pixel coordinate information obtained from the image of the first calibration reference object and the coordinate system information of each camera.

Also, in the step of calibrating at least two or more cameras ( S1000 ), the server 1000 may correct external parameters of the first camera 110 or the second camera 120 , respectively. For example, the server 1000 corrects an external parameter of the at least one camera 110 or 120 based on the pixel coordinate information obtained from the image of the second calibration reference object and the reference coordinate information related to the coordinate system of the sensor 200 . can do. Here, the reference coordinate information may refer to information related to a coordinate system of a radar sensor or a lidar sensor as described above.

Hereinafter, the step ( S1000 ) of calibrating at least two cameras according to an embodiment of the present application will be described in more detail with reference to FIGS. 4 to 8 .

4 is a flowchart detailing the step (S1000) of performing calibration of at least two or more cameras according to an embodiment of the present application.

The step of performing the calibration of at least two or more cameras according to an embodiment of the present application (S1000) is the step of performing the calibration for the intrinsic parameter of the camera (S1100) and the external parameter of the camera (extrinsic parameter) It may include a step of performing calibration (S1200).

In the step (S1100) of performing the calibration for the intrinsic parameter of the camera, the server 1000 obtains pixel coordinate information related to the feature point of the first reference object from the image acquired through the camera, and the pixel coordinate information It is possible to correct the internal parameters of the camera based on Meanwhile, the intrinsic parameter may include at least one of a focal length, a principal point, and a skew coefficient.

5 is a flowchart detailing the step (S1100) of performing calibration on the internal parameters of the camera according to an embodiment of the present application. 6 is a diagram illustrating an example of a first calibration reference object according to an embodiment of the present application.

The step (S1100) of performing calibration on the internal parameters of the camera according to an embodiment of the present application includes: obtaining an image of the first calibration reference object through the camera (S1110), based on the first calibration Obtaining a first set of pixels related to the reference object (S1120), obtaining information on the coordinate system of the camera (S1130), and correcting the internal parameters of the camera based on the information on the coordinate system of the camera and the first set of pixels ( S1140) may be included.

In step S1110 of obtaining an image of the first calibration reference object through the camera, the server 1000 may obtain an image related to the first calibration reference object through the camera 110 or 120 . Specifically, the server 1000 may acquire an image related to the first calibration reference object obtained by photographing from various angles using the camera 110 or 120 . For example, the first calibration reference object may be a checkerboard.

Meanwhile, although not shown in FIG. 5 , the server 1000 acquires the actual width information between the first feature point (eg, P1 in FIG. 6 ) and the second feature point (eg, P2 in FIG. 6 ) of the first calibration reference object. can do.

In step S1120 of obtaining the first pixel set related to the first calibration reference object based on the image, the server 1000 from the image related to the first calibration reference object to the first feature point (eg, P1 in FIG. 6 ). The corresponding first pixel and the second pixel corresponding to the second feature point (eg, P2 of FIG. 6 ) may be detected. Also, the server 1000 may obtain a first pixel set including pixel coordinate information corresponding to the first pixel and pixel coordinate information corresponding to the second pixel.

In the step of obtaining the coordinate system information of the camera ( S1130 ), the server 1000 may obtain the coordinate system information of the camera itself.

In the step of correcting the internal parameters of the camera based on the coordinate system information and the first pixel set (S1140), the server 1000, the first pixel set obtained from the coordinate system information of the camera itself and the image related to the first calibration reference object Based on the , internal parameters of the camera (eg, focal length, focus, asymmetry coefficient, etc.) may be corrected. Specifically, the server 1000 provides pixel coordinate information corresponding to a first pixel corresponding to a first feature point (eg, P1 in FIG. 6 ) of the first calibration reference object and a second feature point (eg, FIG. 6 ) of the first calibration reference object. A width between the first pixel and the second pixel may be calculated based on pixel coordinate information corresponding to the second pixel corresponding to P2 of 6). At this time, the server 1000 based on the actual width information of the first feature point (eg, P1 in FIG. 6 ) and the second feature point (eg, P2 in FIG. 6 ) and the width information between the first pixel and the second pixel, The camera's internal parameters (eg focal length, focus, coefficient of asymmetry, etc.) can be calibrated.

On the other hand, in FIG. 6 , the content of correcting the internal parameters of the camera was mainly described based on the first feature point P1 included in the first calibration reference object and the second feature point P2 adjacent to the first feature point P1. . However, the first feature point P1 and the second feature point P2 shown in FIG. 6 are merely examples for convenience of description, and the server 1000 may set an arbitrary feature area (eg, an arbitrary feature area included in the first calibration reference object) , vertex, line, plane, etc.) may be implemented to correct the internal parameters of the camera.

In addition, although the first calibration reference object has been described with reference to a checkerboard in FIG. 6 , this is only an example, and the server 1000 corrects the internal parameters of the camera using information on objects of any type and shape. can do.

In the step (S1200) of performing the calibration of the external parameter of the camera, the server 1000 obtains pixel coordinate information related to the feature point of the second reference object from the image acquired through the camera, and the second reference It is possible to obtain reference coordinate information related to the feature point of an object and for a sensor-based coordinate system. In this case, the server 1000 may correct the external parameters of the camera based on the pixel coordinate information and the reference coordinate information. On the other hand, the external parameter (extrinsic parameter) means to encompass any parameters related to the transformation relationship between the camera coordinate system and any external coordinate system, and may include parameters related to rotation transformation or translation transformation. have.

7 is a flowchart detailing the step (S1200) of performing calibration on external parameters of the camera according to an embodiment of the present application. 8 is a diagram illustrating an aspect of performing camera calibration using a second calibration reference object according to an embodiment of the present application.

The step of performing the calibration on the external parameter of the camera according to an embodiment of the present application (S1200), the step of obtaining reference coordinate information for the second calibration reference object (S1210), the second calibration reference object through the camera Acquiring an image for (S1220), acquiring a second set of pixels related to a second calibration reference object based on the image (S1230), a coordinate pair based on the reference coordinate information and the second set of pixels It may include a step of obtaining (S1240), and a step of correcting an external parameter of the camera based on the coordinate pair (S1250).

In the step of obtaining reference coordinate information for the second calibration reference object ( S1210 ), the server 1000 may obtain reference coordinate information for the second calibration reference object. Here, the reference coordinate information may mean to encompass coordinate information related to a sensor-based coordinate system, including a radar sensor or a lidar sensor. In more detail, the sensor 200 may measure sensing data for a second calibration reference object disposed at a specific position in a predetermined sensor-based coordinate system, and transmit the sensing data to the server 1000 .

At this time, the server 1000 provides coordinate information on the sensor-based coordinate system in which the second calibration reference object is located and information on the second calibration reference object (eg, actual height (h), actual width (d) and/or actual width ( w)), it is possible to obtain reference coordinate information in the sensor-based coordinate system of an arbitrary feature point of the second calibration reference object (eg, the third feature point P3, the fourth feature point P4, etc. in FIG. 8). .

In step S1220 of obtaining an image of the second calibration reference object through the camera, the server 1000 may obtain an image related to the second calibration reference object through the camera 110 or 120 . Specifically, the server 1000 may acquire an image related to the second calibration reference object obtained by photographing from various angles using the camera 110 or 120 . For example, the second calibration reference object may be a rectangular box.

In the step of obtaining a second set of pixels related to the second calibration reference object based on the image ( S1230 ), the server 1000 , from the image related to the second calibration reference object, a third feature point (eg, P3 in FIG. 8 ) A third pixel corresponding to , and a fourth pixel corresponding to a fourth feature point (eg, P4 of FIG. 8 ) may be detected. Also, the server 1000 may obtain a second pixel set including pixel coordinate information corresponding to the third pixel and pixel coordinate information corresponding to the fourth pixel.

In the step of obtaining a coordinate pair based on the reference coordinate information and the second pixel set ( S1240 ), the server 1000 includes the reference coordinate information related to the sensor-based coordinate system and the pixel coordinate information corresponding to the third pixel and A coordinate pair may be obtained based on the second pixel set including pixel coordinate information corresponding to the fourth pixel. For example, the server 1000 may obtain a first coordinate pair in which reference coordinate information related to a sensor-based coordinate system of a third feature point (eg, P3 in FIG. 8 ) and pixel coordinate information corresponding to the third pixel are linked. can As another example, the server 1000 may obtain a second coordinate pair in which reference coordinate information related to the sensor coordinate system of the fourth feature point (eg, P4 in FIG. 8 ) and pixel coordinate information corresponding to the fourth pixel are linked. can

In step S1250 of correcting the external parameter of the camera based on the coordinate pair, the server 1000 may correct the external parameter of the camera based on the coordinate pair.

For example, the server 1000 may obtain information about the actual height (h), the actual width (d), and/or the actual width (w) of the second calibration reference object. Alternatively, the server 1000 may calculate information on the actual height (h), the actual width (d) and/or the actual width (w) of the second calibration reference object from the reference coordinate information related to the sensor-based coordinate system. Specifically, the server 1000 provides reference coordinate information of a third feature point (eg, P3 in FIG. 8 ) of the second calibration reference object and reference coordinate information of a fourth feature point (eg, P4 in FIG. 8 ) of the second calibration reference object. Information on the actual height (h), the actual width (d), and/or the actual width (w) of the second calibration reference object may be calculated based on .

In addition, the server 1000 provides the pixel coordinate information corresponding to the third pixel corresponding to the third feature point (eg, P3 of FIG. 8 ) of the second calibration reference object and the fourth feature point of the fourth calibration reference object (eg, FIG. 8 ). A distance between the third pixel and the fourth pixel may be calculated based on pixel coordinate information corresponding to the fourth pixel corresponding to P4 of 8).

In this case, the server 1000 may correct the external parameter of the camera based on the actual length information of the second calibration reference object related to the sensor-based coordinate system and the inter-pixel distance information based on the image-based pixel coordinate information. Specifically, since the coordinate pair includes image-based pixel coordinate information obtained through the camera and reference coordinate information related to the sensor-based coordinate system, the server 1000 is configured between the camera coordinate system and the sensor coordinate system outside the camera based on the coordinate pair. It is possible to correct the external parameters of the camera related to the transformation relationship of .

Meanwhile, in FIG. 8 , the correction of external parameters of the camera has been mainly described based on the third feature point P3 included in the second calibration reference object and the fourth feature point P4 adjacent to the third feature point P3 . However, the third feature point P3 and the fourth feature point P4 shown in FIG. 8 are only examples for convenience of description, and the server 1000 may set any reference area (eg, , vertex, line, plane, etc.) may be implemented to correct external parameters of the camera.

In addition, in FIG. 8 , the second calibration reference object in the form of a cuboid has been mainly described, but this is only an example, and the server 1000 may be implemented to correct external parameters of the camera using information of an object of any shape. .

In addition, although it has been described that the camera calibration operation is performed in the server 1000 in FIGS. 2 to 8 , this is only an example, and the camera calibration operation is performed in an external device (or external server) separate from the server 1000 . can be configured.

Referring back to FIG. 3 , the method for generating a learning set according to an embodiment of the present application includes acquiring a first image photographed from the first camera 110 and a second image photographed from the second camera 120 . (S2000), recognizing a target object from a first image and obtaining a two-dimensional first bounding box related to the target object (S3000), recognizing a target object from a second image and a two-dimensional second related to the target object Step of obtaining a bounding box (S4000), a step of obtaining a three-dimensional target bounding box related to the target object based on the first bounding box and the second bounding box (S5000) Obtaining sensing data corresponding to the target bounding box It may include a step (S6000), and a step (S7000) of generating a training set for training the neural network model.

In the step of acquiring the first image photographed from the first camera and the second image photographed from the second camera ( S2000 ), the server 1000 may acquire the image from at least one camera. As an example, the server 1000 may acquire a first image in which the target object is captured through the first camera 110 positioned at a first position with respect to the target object. As another example, the server 1000 may acquire a second image in which the target object is captured through the second camera 120 positioned at a second position with respect to the target object.

Meanwhile, as described above, the first image and the second image may be images that are synchronized for each frame. For example, the server 1000 may synchronize the first image acquired from the first camera 110 , the second image acquired from the second camera 120 , and/or the sensed data acquired from the sensor 200 . may be performed, and the first image and the second image of step S2000 shown in FIG. 3 may mean images after synchronization is matched for each frame.

The method for generating a learning set according to an embodiment of the present application may include recognizing a target object from a first image and acquiring first object information related to the target object ( S3000 ). In the step of recognizing the target object from the first image and obtaining the first object information related to the target object ( S3000 ), the server 1000 detects the target object for each frame of the first image, and the first object related to the target object information can be obtained. Here, the first object information is a two-dimensional bounding box related to the target object detected in the first image and/or landmarks of the target object (eg, eyes, nose, ears, shoulders, elbows, hands, pelvis, knees, It may include at least one of image information (eg, cell coordinates, etc.) corresponding to the foot, etc.).

For example, the server 1000 detects a target object from the first image by using a deep learning technique (eg, yolo, yoloV3, ssd, Retinanet, etc.) related to object detection, and the target object in the first image. It is possible to obtain a mask of an area corresponding to . Also, the server 1000 may acquire a two-dimensional first bounding box related to the target object based on the mask of the first image. Additionally, the server 1000 may utilize an arbitrary technique for increasing the performance of target object detection. For example, the server 1000 utilizes a Simple Online and Realtime Tracking (hereinafter referred to as SORT) algorithm in which a Kalman filter is added to an object detection technique to increase the accuracy of target object detection and perform an operation to track the location of an object. can be implemented to do so.

As another example, the server 1000 uses a deep learning technique (eg, AlphaPose) related to pose estimation to obtain landmarks (eg, eyes, nose, ears, shoulders, and elbows) of the target object from the first image. , hand, pelvis, knee, foot, etc.) may acquire first cell information (eg, cell coordinate information). Specifically, the server 1000 may perform pose estimation on the target object detected from the first image to obtain first cell information corresponding to the landmark of the target object as object information related to the target object.

However, the above-described object detection technique and pose estimation technique are merely examples, and the server 1000 may be implemented to detect a target object from an image by using any suitable method.

The method for generating a learning set according to an embodiment of the present application may include recognizing a target object from a second image and acquiring second object information related to the target object ( S4000 ). In the step of recognizing the target object from the second image and obtaining second object information related to the target object ( S4000 ), the server 1000 detects the target object for each frame of the second image, and the second object related to the target object information can be obtained. Here, the second object information refers to a two-dimensional bounding box related to a target object detected in the second image and/or landmarks (eg, eyes, nose, ears, shoulders, elbows, hands, pelvis, and knees) of the target object. , feet, etc.) may include at least one of image information (eg, cell coordinates, etc.).

For example, the server 1000 detects a target object from the second image by using a deep learning technique (eg, yolo, yoloV3, ssd, Retinanet, etc.) related to object detection, and the target object in the second image. It is possible to obtain a mask of an area corresponding to . Also, the server 1000 may acquire a two-dimensional second bounding box related to the target object based on the mask of the second image. Additionally, the server 1000 may utilize an arbitrary technique for increasing the performance of target object detection. For example, the server 1000 utilizes a Simple Online and Realtime Tracking (hereinafter referred to as SORT) algorithm in which a Kalman filter is added to an object detection technique to increase the accuracy of target object detection and perform an operation to track the location of an object. can be implemented to do so.

As another example, the server 1000 uses a deep learning technique (eg, AlphaPose) related to pose estimation to obtain landmarks (eg, eyes, nose, ears, shoulders, and elbows) of the target object from the second image. , hand, pelvis, knee, foot, etc.) may acquire second cell information (eg, cell coordinate information). Specifically, the server 1000 may perform pose estimation on the target object detected from the second image to obtain second cell information corresponding to the landmark of the target object as object information related to the target object.

However, the above-described object detection technique and pose estimation technique are merely examples, and the server 1000 may be implemented to detect a target object from an image by using any suitable method.

In the step of obtaining a three-dimensional target bounding box related to the target object based on the first object information and the second object information (S5000), the server 1000, the first object related to the target object obtained from the first image A three-dimensional target bounding box related to the target object may be obtained from the information and the second object information related to the target object obtained from the second image.

As an example, the server 1000 may reverse project each of the first bounding box included in the first object information and the second bounding box included in the second object information onto 3D spatial coordinates, thereby performing a 3D project related to the target object. You can obtain the bounding box of the dimension object. As another example, the server 1000 projects each of the first cell information included in the first object information and the second cell information included in the second object information onto the 3D spatial coordinates, thereby performing a three-dimensional (3D) related target object. You can obtain the target bounding box of .

Hereinafter, the step (S5000) of obtaining a three-dimensional target bounding box related to the target object will be described in more detail with reference to FIGS. 9 to 14 . Specifically, an embodiment of obtaining a target bounding box based on object information related to a two-dimensional bounding box will be described with reference to FIGS. 9 to 13, and object information related to a landmark of the target object with reference to FIG. 14 Another embodiment of obtaining the target bounding box based on the description will be made.

9 is a flowchart detailing the step (S5000) of obtaining a target bounding box according to an embodiment of the present application. The step of obtaining a target bounding box according to an embodiment of the present application (S5000) is a step of reverse-projecting a two-dimensional first bounding box included in the first object information onto three-dimensional spatial coordinates to obtain a first projection volume (S5100), a step of reversely projecting the two-dimensional second bounding box included in the second object information onto the three-dimensional spatial coordinates to obtain a second projection volume (S5200), and the first projection volume and the second projection volume It may include obtaining a target bounding box related to the target object based on the step (S5300).

In the step (S5100) of obtaining a first projection volume by reverse-projecting the two-dimensional first bounding box included in the first object information onto the three-dimensional spatial coordinates (S5100), the server 1000 and the target object included in the first image The first projection volume related to the target object may be obtained by reversely projecting the related 2D first bounding box onto the 3D spatial coordinates. Specifically, the server 1000 may obtain the first projection volume by reversely projecting the first bounding box onto the 3D spatial coordinates using the internal parameter and/or the external parameter of the first camera 110 .

In the step (S5200) of acquiring the second projection volume by projecting the second two-dimensional bounding box included in the second object information onto the three-dimensional spatial coordinates, the server 1000 relates to the target object included in the second image. A second projection volume related to the target object may be calculated by reversely projecting the 2D second bounding box onto the 3D spatial coordinates. Specifically, the server 1000 may obtain the second projection volume by reversely projecting the second bounding box onto the 3D spatial coordinates using the internal parameter and/or the external parameter of the second camera 120 .

In the step S5300 of obtaining the target bounding box related to the target object on the basis of the first projection volume and the second projection volume, the server 1000 may obtain an intersection volume between the first projection volume and the second projection volume. and may obtain a three-dimensional target bounding box related to the target object based on the intersection volume.

Hereinafter, a method of obtaining a three-dimensional target bounding box related to a target object according to an embodiment of the present application will be described in detail with reference to FIGS. 10 and 11 .

10 is a flowchart detailing the step (S5300) of obtaining a target bounding box related to a target object according to an embodiment of the present application. 11 is a diagram illustrating an aspect of obtaining a three-dimensional target bounding box according to an embodiment of the present application.

The step of obtaining the target bounding box related to the target object according to an embodiment of the present application (S5300) is the step of obtaining at least one first plane constituting the first projection volume (S5310), configuring the second projection volume obtaining at least one second plane ( S5312 ), and obtaining a common plane of the first and second planes and obtaining a target bounding box based on the common plane ( S5314 ).

In step S5310 of obtaining at least one first plane constituting the first projection volume, the server 1000 may obtain at least one first plane constituting the first projection volume. In more detail, the server 1000 may obtain a first plane including a first line (eg, m1 of FIG. 11 ) and a second line (eg, m2 of FIG. 11 ) constituting the first projection volume.

In the step of obtaining at least one second plane constituting the second projection volume ( S5312 ), the server 1000 may obtain at least one second plane constituting the second projection volume. Specifically, the server 1000 may obtain a second plane including a third line (eg, 11 in FIG. 11 ) and a fourth line (eg, 12 in FIG. 11 ) constituting the second projection volume.

In the step of acquiring a common plane of the first plane and the second plane and acquiring a target bounding box based on the common plane (S5314), the server 1000 includes at least one first plane and a second plane constituting the first projection volume. A common plane between at least one second plane constituting the two projection volumes may be obtained. Also, the server 1000 may obtain an intersection volume between the first projection volume and the second projection volume based on the acquired plurality of common planes. Also, the server 1000 may be implemented to generate a three-dimensional target bounding box related to the target object based on the intersecting volume.

Meanwhile, in FIG. 11, with respect to the intersecting volume between the first projection volume and the second projection volume, the intersecting volume including the vertex (eg, v1 in FIG. 11 ) corresponding to a position far from the first camera (or the second camera) is obtained. shown to be generated. However, this is only an example, and the server 1000 may be implemented to obtain an intersection volume between the first projection volume and the second projection volume in consideration of the angle or tilt of the camera. For example, the server 1000 may be implemented to correct a vertex (eg, v1 of FIG. 11 ) corresponding to a location far from the first camera (or the second camera) and generate an intersecting volume. Specifically, the server 1000 may be configured to generate an intersecting volume excluding a specific vertex (eg, v1 in FIG. 11 ).

On the other hand, when there are a plurality of bounding boxes obtained from the first image (or the second image) (that is, when there are a plurality of target objects in each image), the bounding box obtained from the first image and the bounding obtained from the second image There may be cases where it is not clear how the boxes match.

The server 1000 according to an embodiment of the present application obtains characteristic information related to the projection volume (or target object) corresponding to the bounding box obtained from each image, and based on the similarity between the characteristic information, three-dimensional object bounding It can be implemented to obtain a box. Specifically, the server 1000 may include first characteristic information related to the first projection volume (or target object) corresponding to the first bounding box obtained from the first image and the second bounding box corresponding to the second image obtained from the second image. The second characteristic information related to the second projection volume (or the target object) may be acquired, and the target bounding box may be acquired based on the similarity between the first characteristic information and the second characteristic information.

Hereinafter, the content of obtaining a target bounding box according to an embodiment of the present application will be described in more detail with reference to FIGS. 12 and 13 .

12 is a flowchart detailing the step (S5300) of obtaining a target bounding box related to a target object according to an embodiment of the present application. 13 is a diagram illustrating an aspect of determining a three-dimensional target bounding box according to an embodiment of the present application.

The step of obtaining the target bounding box related to the target object according to an embodiment of the present application (S5300) is the step of obtaining three-dimensional temporary bounding boxes based on the first projection volume and the second projection volume (S5320), the second A step of obtaining first characteristic information related to the first bounding box (S5322), a step of obtaining second characteristic information related to the second bounding box (S5324), a temporary basis based on the similarity between the first characteristic information and the second characteristic information It may include allocating a matching score to the bounding box (S5326), and determining a target bounding box from among temporary bounding boxes based on the matching score (S5328).

In the step of obtaining three-dimensional temporary bounding boxes based on the first projection volume and the second projection volume ( S5320 ), the server 1000 , the two-dimensional bounding box related to the first target object included in the first image. A projected volume can be obtained by reverse-projecting . In addition, the server 1000 may acquire the projection volume by reversely projecting the two-dimensional bounding box related to the first target object included in the second image into the three-dimensional space. In addition, the server 1000 may obtain the projection volume by reversely projecting the two-dimensional bounding box related to the second target object included in the first image into the three-dimensional space. In addition, the server 1000 may acquire the projection volume by reversely projecting the two-dimensional bounding box related to the second target object included in the second image into the three-dimensional space.

In this case, the server 1000 may generate a plurality of temporary bounding boxes according to a combination between the projection volume related to the first target object and the projection volume related to the second target object. For example, the server 1000 calculates the intersection volume based on the projection volume obtained from the bounding box related to the first target object of the first image and the projection volume obtained from the bounding box related to the first target object of the second image, and , a three-dimensional temporary bounding box (eg, the first temporary bounding box of FIG. 13 ) may be obtained based on the obtained intersecting volume. In addition, the server 1000 calculates the intersection volume based on the projection volume obtained from the bounding box related to the first target object of the first image and the projection volume obtained from the bounding box related to the second target object of the second image, and , a three-dimensional temporary bounding box (eg, the second temporary bounding box of FIG. 13 ) may be obtained based on the obtained intersecting volume.

However, an error may exist among the plurality of generated temporary bounding boxes. For example, the second temporary bounding box shown in FIG. 13 is a temporary bounding box obtained based on the intersection volume between the projection volume obtained from the first target object and the projection volume obtained from the second target object, obtained from different target objects. error may exist.

Accordingly, the server 1000 according to an embodiment of the present application may be implemented to determine a target bounding box from among temporary bounding boxes. Specifically, the server 1000 obtains characteristic information related to the projection volume (or target object), gives a matching score to the temporary bounding box according to the similarity between the characteristic information, and based on the matching score, from among the temporary bounding boxes. It may be implemented to determine a target bounding box. Here, the characteristic information may mean encompassing any information that can be used to determine the degree of similarity between the first bounding box and the second bounding box, including color information and distribution information related to the projected volume (or target object). Also, here, the first bounding box may mean encompassing any bounding box obtained from the first image, and the second bounding box may mean encompassing any bounding box obtained from the second image.

In the step of obtaining the first characteristic information related to the first bounding box (S5322), the server 1000, the first projection volume (or target object) corresponding to the first bounding box related to the three-dimensional temporary bounding box The first characteristic information may be acquired. For example, the server 1000 may include a first bounding box (eg, a bounding box corresponding to the first target object of the first image) related to the first temporary bounding box and first characteristic information of the first target object (eg, color information of the first target object) may be obtained. In addition, the server 1000 may provide first characteristic information of the first target object corresponding to the first bounding box (eg, the bounding box corresponding to the first target object of the first image) related to the second temporary bounding box (eg, color information of the first target object) may be obtained.

In the step of obtaining the second characteristic information related to the second bounding box (S5324), the server 1000, the second projection volume (or target object) corresponding to the second bounding box related to the three-dimensional temporary bounding box Second characteristic information may be obtained. For example, the server 1000 may include a second bounding box (eg, a bounding box corresponding to the first target object of the second image) related to the first temporary bounding box and second characteristic information of the first target object (eg, color information of the first target object) may be obtained. In addition, the server 1000 may include second characteristic information of the second target object corresponding to the second bounding box (eg, the bounding box corresponding to the second target object of the second image) related to the second temporary bounding box (eg, color information of the second target object) may be obtained.

As an example, the server 1000 may utilize a Lomo (Local Maximal Occurrence, Lomo) feature technique to obtain characteristic information related to a target object corresponding to each bounding box. The lomo feature technique is a technique capable of determining whether the object is the same object by focusing on characteristics of the object, for example, the color of the object, in order to identify the object in an image from a different angle. The server 1000 according to an embodiment of the present application may determine a similarity between a target object corresponding to the matched first bounding box and the second bounding box by using the lomo feature technique.

In the step of allocating a matching score to the temporary bounding box based on the similarity between the first characteristic information and the second characteristic information ( S5326 ), the server 1000 , the first projection volume (or target object) obtained from the first image A matching score may be given to the three-dimensional temporary bounding box based on the similarity between the first characteristic information of , and the second characteristic information of the second projection volume (or target object) obtained from the second image. For example, the server 1000 may configure the first characteristic information (eg, color information) corresponding to the first bounding box of the first image related to the first temporary bounding box and the second bounding of the second image related to the first temporary bounding box. A first score may be given to the first temporary bounding box based on the similarity between the second characteristic information (eg, color information) corresponding to the box. For another example, the server 1000 may store the first characteristic information (eg, color information) corresponding to the first bounding box of the first image related to the second temporary bounding box and the second image related to the second temporary bounding box. A second score may be given to the second temporary bounding box based on the similarity between the second characteristic information (eg, color information) corresponding to the second bounding box. In this case, the server 1000 may store the second temporary in the first temporary bounding box obtained from the first bounding box corresponding to the first target object of the first image and the second bounding box corresponding to the first target object of the second image. A relatively high score may be given compared to the second score assigned to the bounding box.

On the other hand, since the second temporary bounding box is generated from the first bounding box corresponding to the first target object of the first image and the second bounding box corresponding to the second target object of the second image, the similarity between the characteristic information is the second. 1 It may be lower than in the case of a temporary bounding box. For example, the server 1000 may give the second temporary bounding box a relatively lower score than the first score assigned to the first temporary bounding box. As another example, when the similarity between the first characteristic information and the second characteristic information is lower than a predetermined criterion, the server 1000 uniformly allocates a predetermined score so as not to determine the corresponding temporary bounding box as the target bounding box. have.

The server 1000 according to an embodiment of the present application may utilize a cosine similarity between the first characteristic information and the second characteristic information to allocate a matching score to the temporary bounding box. The cosine similarity is one of the methods of measuring the similarity between arbitrary vectors, and the higher the cosine similarity, the higher the probability that the arbitrary vectors are identical to each other. The server 1000 according to an embodiment of the present application may measure the cosine similarity between the first characteristic information and the second characteristic information, and give a matching score to the temporary bounding box based on the cosine similarity. However, this is only an example, and the server 1000 evaluates the similarity between the first characteristic information and the second characteristic information using any suitable technique, and assigns a matching score to the temporary bounding box based on the evaluated similarity. could be configured.

In step S5328 of determining the target bounding box from among the temporary bounding boxes based on the matching score, the server 1000 selects or determines the target bounding box from among the temporary bounding boxes based on the matching score of each temporary bounding box. can Specifically, the server 1000 may compare the matching scores of the temporary bounding boxes and determine the temporary bounding box assigned a high score as the target bounding box. At this time, also, the server 1000 may determine a temporary bounding box having an optimal matching score among the temporary bounding boxes as the target bounding box through the Hungarian algorithm.

Specifically, the server 1000 compares the first score given to the first temporary bounding box with the second score given to the second temporary bounding box, and selects at least one of the first temporary bounding box and the second temporary bounding box as the first It can be determined by the target bounding box associated with the target object. For example, the first temporary bounding box of FIG. 13 (that is, the first bounding box obtained from the first target object of the first image and the second bounding box obtained from the first target object of the second image The temporary bounding box) is a second temporary bounding box of FIG. 13 (that is, a first bounding box obtained from a first target object of the first image and a second bounding box obtained from a second target object of the second image. It is highly likely to receive a higher matching score than a three-dimensional bounding box). At this time, the server 1000 compares the first score given to the first temporary bounding box with the second score given to the second temporary bounding box to determine the first temporary bounding box as the target bounding box related to the first target object. can

In the above, the content of determining the target bounding box related to the first target object has been mainly described. However, this is for convenience of description, and may be analogously applied to the content of determining the target bounding box related to the second target object.

Hereinafter, another embodiment of obtaining a three-dimensional target bounding box related to a target object according to FIG. 14 will be described in detail. 14 is a flowchart detailing the step of obtaining a target bounding box related to a target object according to another embodiment of the present application.

The step of obtaining a three-dimensional target bounding box related to a target object according to another embodiment of the present application (S5000) is to obtain a first projection straight line by projecting the first cell information related to the target object included in the first object information step (S5400) of obtaining a second projection straight line by projecting second cell information related to the target object included in the second object information (S5500), and the target based on the first projection straight line and the second projection straight line It may include obtaining a three-dimensional target bounding box related to the object (S5600).

In the step of acquiring a first projection straight line by projecting the first cell information related to the target object included in the first object information ( S5400 ), the server 1000 determines the landmark (eg, the landmark of the target object included in the first object information). , eyes, nose, ears, shoulders, elbows, hands, pelvis, knees, and feet) may be projected onto the 3D spatial coordinates to obtain a first projected straight line.

In the step (S5500) of obtaining a second projection straight line by projecting the second cell information related to the target object included in the second object information, the server 1000 is configured to generate a landmark (eg, a landmark of the target object included in the second object information). , eye, nose, ear, shoulder, elbow, hand, pelvis, knee, foot) may be projected on the 3D spatial coordinates of the second cell information to obtain a second projected straight line.

In the step of obtaining a three-dimensional target bounding box related to the target object based on the first projection straight line and the second projection straight line (S5600), the server 1000 is the target object based on the first projection straight line and the second projection straight line A three-dimensional target bounding box related to can be calculated. For example, when there is an intersection point between the first projection straight line and the second projection straight line, the server 1000 may obtain an intersection point between the first projection straight line and the second projection straight line. As another example, when there is no intersection point between the first projection straight line and the second projection straight line, the server 1000 may obtain the coordinates on the 3D space coordinates closest to the first projection straight line and the second projection straight line. In addition, the server 1000 obtains a first projection straight line and a second projection straight line corresponding to each of a plurality of landmarks related to the target object, and an intersection point between the first projection straight line and the second projection straight line (or the first projection straight line and the coordinates on the nearest three-dimensional space coordinates with respect to the second projection straight line) may be obtained.

At this time, the server 1000 obtains a three-dimensional target bounding box including intersection points (or coordinates on the three-dimensional space coordinates closest to the first projection straight line and the second projection straight line) corresponding to each landmark of the target object. can do. In this case, the target bounding box may be implemented in an arbitrary shape (eg, a rectangular parallelepiped) including intersection points.

Meanwhile, as described above, it may be unclear in what combination the intersection points of landmarks obtained from the first image (or the second image) match. Specifically, there may exist a case in which a plurality of three-dimensional bounding boxes (eg, a first temporary bounding box, a second temporary bounding box, etc.) are obtained with respect to the target object. At this time, the server 1000 according to an embodiment of the present application is based on the distance between the projection straight lines corresponding to each landmark, a plurality of bounding boxes (eg, a first temporary bounding box, a 3D of the second temporary bounding box) It can be implemented to determine the target bounding box of .

Specifically, the server 1000 calculates the distance between the first projection straight line and the second projection straight line corresponding to each landmark of the target object, and based on the distance between the first projection straight line and the second projection straight line, a plurality of bounding boxes ( For example, a matching score may be given to each of the first temporary bounding box and the second temporary bounding box). As an example, the server 1000 calculates distance information including an average value and/or a median value of distances between a first projection straight line and a second projection straight line corresponding to at least one landmark related to the first temporary bounding box, A first score may be given to the first temporary bounding box based on the distance information. As another example, the server 1000 calculates distance information including the average value and/or median value of the distances between the first projection straight line and the second projection straight line corresponding to at least one landmark related to the second temporary bounding box, A second score may be given to the second temporary bounding box based on the distance information. For example, the server 1000 may assign a smaller value to the bounding box as the distance information including the average value and/or the median value of the distances between the first projection straight line and the second projection straight line includes a smaller value. In particular, when the first projection straight line and the second projection straight line intersect, the server 1000 may give 0 points to the intersection point of the corresponding bounding box.

In this case, the server 1000 may determine at least one of the first temporary bounding box and the second temporary bounding box as the target bounding box by comparing the first score with the second score. For example, by comparing the first score and the second score, a temporary bounding box having a lower score may be determined as the target bounding box. Also, the server 1000 may determine a bounding box having an optimal matching score among the first temporary bounding box and the second temporary bounding box as the target bounding box through the Hungarian algorithm.

However, the above is only an example, and as the distance information includes a small value, the server 1000 gives a higher value to the temporary bounding box, and determines the temporary bounding box having a higher score as the target bounding box. may be implemented to do so.

Referring back to FIG. 3 , the method of generating a learning set according to an embodiment of the present application may include acquiring sensing data corresponding to a target bounding box ( S6000 ). In the step of acquiring the sensing data corresponding to the target bounding box ( S6000 ), the server 1000 may acquire sensing data (eg, radar data or lidar data) related to the target bounding box. Here, the sensing data may include data in the form of an image obtained from the sensor 200 . In addition, as described above, the server 1000 may perform an operation of matching the sync between the sensed data and the image data acquired through the camera, so that the server 1000 corresponds to the target bounding box for each frame. sensing data can be acquired.

Referring back to FIG. 3 , the method of generating a training set according to an embodiment of the present application may include generating a training set for training a neural network model ( S7000 ). In the step (S7000) of generating a training set for learning the neural network model, the server 1000 learns to train the neural network model based on the sensing data and the target bounding box related to the target object acquired through at least one camera. You can create sets. For example, the server 1000 may generate a training set for training a neural network model that outputs a target bounding box related to a target object based on sensing data (eg, radar data or lidar data).

Meanwhile, although not shown in FIG. 3 , the method for generating a learning set according to an embodiment of the present application includes a first image obtained from the first camera 110 , a second image obtained from the second camera 120 , and The method may further include the step of matching a sync between the sensed data obtained from the sensor 200 and/or the sensor 200 .

For example, the server 1000 reads an image broadcast on the web from the first camera 110 or the second camera 120 to obtain the first image and the second image obtained from the first camera 110 . It may be implemented to synchronize the second images obtained from the camera 120 . For example, the server 1000 may be implemented to acquire the first image and the second image by using OBS studio 27, and to synchronize the time of the first image and the second image based on the time of the common server 1000. have.

As another example, as described above, the server 1000 may acquire arbitrary data including frame information (eg, frame number information, frame time information, etc.) and meta information through the sensor 200 . In this case, the server 1000 may be implemented to synchronize time between image data and sensing data based on frame information and/or meta information.

However, the above is only an example, and the server 1000 may be implemented to perform an operation of synchronizing image data obtained from at least one camera and sensing data obtained from a sensor by using any suitable method. can

A learning set obtained according to an embodiment of the present application detects a target object included in the sensing data based on the sensing data (eg, radar data or lidar data), and a three-dimensional target bounding box corresponding to the detected target object It can be used to train a neural network model that outputs Specifically, the training set may include a three-dimensional target bounding box corresponding to a target object based on image data obtained through the above-described training set generating method and sensing data. In this case, the neural network model may receive sensing data, and parameters of at least one node included in the neural network model may be updated to output a three-dimensional target bounding box corresponding to the target object as an output value.

The trained neural network model may be configured to receive sensing data (eg, radar data or lidar data) and output a target bounding box related to a target object included in the sensing data.

The trained neural network model according to an embodiment of the present application recognizes a target object outside the vehicle or a target object inside the vehicle based on sensing data obtained from a sensor (eg, a lidar sensor or a radar sensor) mounted on the vehicle. It can be applied to fields that recognize In addition, the neural network model that has been trained according to an embodiment of the present application may be used for fall detection of an object in a home or a nursing home. Specifically, by installing a sensor at a certain location such as a wall or ceiling, recognizing a target object (eg, a person) using a trained neural network model, and calculating the rate of change in the height of the target object, it is possible to detect a fall of the target object. have. However, this is only an example, and may be utilized in any suitable field of detecting a target object based on any type of sensing data.

The training set generation system 10 according to an embodiment of the present application generates a training set for learning an object recognition neural network model based on sensing data by automatically labeling it in association with object information detected from camera-based image data, The time and cost required for the labeling operation can be significantly saved, and it can provide an advantageous effect that the training set can be easily acquired.

In addition, the learning set generating system 10 according to an embodiment of the present application calculates a target bounding box related to a target object from camera-based image data, using similarity between characteristic information or a landmark of the target object. By elaborately creating the bounding box, a higher quality training set can be obtained.

The above-described various operations of the server 1000 may be stored in the memory 1200 of the server 1000 , and the controller 1300 of the server 1000 may be provided to perform the operations stored in the memory 1200 .

Features, structures, effects, etc. described in the above embodiments are included in at least one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified for other embodiments by those of ordinary skill in the art to which the embodiments belong. Accordingly, the contents related to such combinations and modifications should be interpreted as being included in the scope of the present invention.

In addition, although the embodiment has been mainly described in the above, this is only an example and does not limit the present invention, and those of ordinary skill in the art to which the present invention pertains to the above in a range that does not deviate from the essential characteristics of the present embodiment. It will be appreciated that various modifications and applications not illustrated are possible. That is, each component specifically shown in the embodiment can be implemented by modification. And the differences related to these modifications and applications should be construed as being included in the scope of the present invention defined in the appended claims.

10: Training Set Generation System
110: first camera
120: second camera
200: sensor
1000: server

Claims (10)

A method for generating a training set for training a neural network model for detecting an object by a training set generating apparatus for generating a training set for executing a neural network model for detecting an object, based on sensing data, the method comprising:
performing calibration on at least two or more cameras;
obtaining a first image photographed from a first camera and a second image photographed from a second camera;
recognizing a target object from the first image and obtaining first object information related to the target object;
recognizing the target object from the second image and obtaining second object information related to the target object;
obtaining a three-dimensional target bounding box related to the target object based on the first object information and the second object information;
obtaining sensing data corresponding to the target bounding box, wherein the sensing data is obtained through at least one of a lidar sensor and a radar sensor; and
Based on the sensing data and the target bounding box, generating a training set for learning a neural network model for detecting an object included in the sensing data;
The step of obtaining a three-dimensional target bounding box related to the target object,
obtaining a first projected volume by reverse projecting the first bounding box included in the first object information onto 3D spatial coordinates;
obtaining a second projected volume by reverse projecting a second bounding box included in the second object information onto 3D spatial coordinates;
obtaining an intersection volume between the first projection volume and the second projection volume based on the first projection volume and the second projection volume, and obtaining at least two or more three-dimensional temporary bounding boxes based on the intersection volume step;
obtaining first characteristic information related to the first bounding box;
obtaining second characteristic information related to the second bounding box;
allocating a matching score to the temporary bounding box based on a degree of similarity between the first characteristic information and the second characteristic information; and
Determining the target bounding box from among the temporary bounding boxes based on the matching score;
The first characteristic information relates to color information related to the first bounding box, and the second characteristic information relates to color information related to the second bounding box,
How to create a training set.
delete The method of claim 1,
The step of obtaining the target bounding box related to the target object,
obtaining at least one first plane constituting the first projection volume;
obtaining at least one second plane constituting the second projection volume; and
Including; acquiring a common plane of the first plane and the second plane, and acquiring the target bounding box based on the common plane;
How to create a training set.
delete The method of claim 1,
The step of obtaining a three-dimensional target bounding box related to the target object,
obtaining a first projection straight line by projecting first cell information related to the target object included in the first object information onto 3D spatial coordinates;
obtaining a second projection straight line by projecting second cell information related to the target object included in the second object information onto 3D spatial coordinates; and
Including; obtaining a three-dimensional target bounding box related to the target object based on the first projection straight line and the second projection straight line
How to create a training set.
6. The method of claim 5,
The step of obtaining a three-dimensional target bounding box related to the target object on the basis of the first projection straight line and the second projection straight line,
obtaining at least two or more three-dimensional temporary bounding boxes including a first temporary bounding box and a second temporary bounding box based on the first projection straight line and the second projection straight line;
calculating a distance between the first projection straight line and the second projection straight line;
allocating a first score to the first temporary bounding box and a second score to the second temporary bounding box based on the calculated distance; and
Comparing the first score and the second score, determining at least one of the first temporary bounding box and the second temporary bounding box as a target bounding box;
How to create a training set.
The method of claim 1,
The step of performing calibration for the at least two or more cameras,
Comprising the step of performing calibration of the internal parameters of at least two or more cameras,
The step of performing calibration of the internal parameters of the at least two or more cameras,
acquiring an image of a first calibration reference object through the camera;
obtaining a first set of pixels associated with the first calibration reference object based on the image;
obtaining coordinate system information of the camera; and
correcting an intrinsic parameter of the camera based on the coordinate system information and the first set of pixels;
How to create a training set.
8. The method of claim 7,
The step of performing calibration for the at least two or more cameras,
Comprising the step of performing calibration of the external parameters of at least two or more cameras,
The step of performing calibration of the external parameters of the at least two or more cameras,
obtaining reference coordinate information for a second calibration reference object, wherein the reference coordinate information is related to a sensor-based coordinate system of at least one of a lidar sensor and a radar sensor;
acquiring an image of the second calibration reference object through the camera;
obtaining a second set of pixels associated with the second calibration reference object based on the image;
obtaining a coordinate pair based on the reference coordinate information and the second set of pixels; and
Comprising; performing correction on the external parameter of the camera based on the coordinate pair;
How to create a training set.
A computer-readable recording medium in which a program for executing the method according to any one of claims 1 to 8 in a computer is recorded.
An apparatus for generating a training set for generating a training set for training a neural network model for detecting an object based on sensing data, the apparatus comprising:
a transceiver for communicating with at least one of a first camera, a second camera, and a sensor; and
A controller configured to obtain image data through the first camera and the second camera, obtain sensing data through the sensor, and generate a learning set based on the image data and the sensing data;
The controller is
Obtaining a first image photographed from the first camera and a second image photographed from the second camera, recognizing a target object from the first image, acquiring first object information related to the target object, and 2 Recognize the target object from the image, obtain second object information related to the target object, and obtain a three-dimensional target bounding box related to the target object based on the first object information and the second object information, and , sensor data corresponding to the target bounding box - the sensor data is acquired through at least one of a lidar sensor and a radar sensor - is acquired, based on the sensor data and the target bounding box, included in the sensing data configured to generate a training set for training a neural network model to detect an object,
The controller is
A first projection volume is obtained by reverse projecting the first bounding box included in the first object information on 3D spatial coordinates, and the second bounding box included in the second object information is converted into 3D spatial coordinates. to reverse project to obtain a second projection volume, and to obtain an intersection volume between the first projection volume and the second projection volume based on the first projection volume and the second projection volume; At least two or more three-dimensional temporary bounding boxes are obtained based on the volume, first characteristic information related to the first bounding box is obtained, second characteristic information related to the second bounding box is obtained, and the first Allocating a matching score to the temporary bounding box based on the similarity between the characteristic information and the second characteristic information, and determining the target bounding box from among the temporary bounding boxes based on the matching score,
The first characteristic information relates to color information related to the first bounding box, and the second characteristic information relates to color information related to the second bounding box,
Learning set generator.

Images