KR102407174B1 - Method for estimating relative information between target objects and devices using multiple radars and riders in a marine environment - Google Patents

Abstract

A method for estimating relative information between a target object and a device using a plurality of radars and lidar in a marine environment is provided. The method includes the steps of: a pre-processing unit receiving a sensor signal having a directional angle and a reflection distance of light or radio waves reflected from an object through a sensor unit; generating a group of , and generating a two-dimensional informatization matrix by deriving a minimum value from each of a plurality of groups by the preprocessor.

Description

Method for estimating relative information between target objects and devices using multiple radars and riders in a marine environment

The present invention relates to a technology for estimating relative information, and more particularly, to a method for estimating relative information between a target object and a device using a plurality of radars and lidar in a marine environment.

In the prior art, in order to move toward a target object located at another location on a device at a specific location on the sea, a sensor for detecting/detecting the target object is selected and an algorithm for processing output information is executed, and the detected/detected An algorithm that processes the control output of multiple thrusters using the relative position information of the target is being executed. The algorithm used for sensor processing is filtered in a recursive way based on the model for non-linear sensor output information. This algorithm diverges when the sensor output deviates from the expected model, and also recursively The error built into the sensor itself is accumulated through repetition, and as time goes by, a difference occurs between the estimated value and the actual value.

Registered Korea Patent Publication No. 2025113 on September 19, 2019 (Title: Image creation method using LiDAR and device therefor)

An object of the present invention is to provide a method for estimating relative information between a target object and a device using a plurality of radars and lidar in a marine environment.

In a method for estimating relative information between a target object and a device using a plurality of radars and lidars in a marine environment according to a preferred embodiment of the present invention for achieving the object as described above, the preprocessor is reflected from the object through the sensor unit. A step of receiving a sensor signal having a directional angle and a reflection distance of light or radio waves, the pre-processing unit generating a plurality of groups by dividing the sensor signal according to the directional angle and the reflection distance; and generating a two-dimensional informatization matrix by deriving a minimum value from each of the plurality of groups.

The method comprises the steps of: the pre-processing unit accumulating the two-dimensional informatization matrix for a predetermined time to generate input data having a three-dimensional structure in which a plurality of two-dimensional informatization matrices are accumulated on a time axis; detecting a target object through an area box indicating an area occupied by the target object in each of a plurality of sensor signals corresponding to each of the plurality of two-dimensional informatization matrices, and calculating a center point of the area box; Calculating the difference in heading direction angle between the navigation device and the target object from a plurality of sensor signals corresponding to the two-dimensional informatization matrix, and the learning unit from each of the plurality of sensor signals corresponding to the plurality of two-dimensional informatization matrices to the navigation device and The method further includes calculating a relative distance between marine target objects, and labeling, by the learning unit, the difference in the heading direction angle and the relative distance on the plurality of two-dimensional informatization matrices.

에 따라 상기 헤딩 방향각 차이를 산출하며, 상기 dx는 상기 목표 대상물에 대한 상기 영역상자의 중심점 중 x 좌표이고, 상기 W는 상기 센서 신호를 시각화한 영상의 수평방향의 픽셀 개수이고, 상기 F는 초점 거리인 것을 특징으로 한다. In the step of calculating the difference in the heading direction angle, the learning unit is

calculates the difference in the heading direction angle according to, dx is the x coordinate of the center point of the area box with respect to the target object, W is the number of pixels in the horizontal direction of the image visualized with the sensor signal, and F is It is characterized in that the focal length.

에 따라 상기 상대 거리를 산출하며, 상기 dx는 목표 대상물의 영역상자의 중심점 중 x 좌표이고, 상기 dy는 목표 대상물의 영역상자의 중심점 중 y 좌표이고, 상기 W는 상기 센서 신호를 시각화한 영상의 수평방향의 픽셀 개수이고, 상기 H는 상기 센서 신호를 시각화한 영상의 수직방향의 픽셀 개수이고, 상기 F는 초점 거리인 것을 특징으로 한다. the learning unit

calculates the relative distance according to, dx is the x-coordinate of the center point of the area box of the target object, dy is the y-coordinate of the center point of the area box of the target object, and W is the image obtained by visualizing the sensor signal The number of pixels in the horizontal direction, H is the number of pixels in the vertical direction of the image visualized with the sensor signal, and F is the focal length.

에 따라 상기 센서 신호의 방향 각도에 대한 분할 지점을 산출하는 단계와, 상기 전처리부가 상기 산출된 방향 각도에 대한 분할 지점에 따른 구간 별로 상기 센서 신호를 그룹화하여 방향각도 별 반사거리 값을 저장하는 단계를 포함하며, 상기 θi는 방향각도의 분할 지점이고, 상기 i는 방향각도의 분할 지점에 대한 인덱스이고, 상기 N은 방향각도 분할 수이고, 상기

는 방향각도의 화각인 것을 특징으로 한다. The step of the pre-processing unit dividing the sensor signal into a plurality of groups according to the directional angle and the reflection distance is an equation based on the number of divisions of the angle of view and the directional angle of the pre-processing unit preset directional angle

calculating a division point for the direction angle of the sensor signal according to the method, and storing the reflection distance value for each direction angle by grouping the sensor signals for each section according to the division point for the calculated direction angle by the preprocessing unit. , wherein θi is a division point of a direction angle, i is an index to a division point of a direction angle, N is a direction angle division number, and

is the angle of view of the directional angle.

에 따라 반사 거리에 대한 분할 지점을 산출하는 단계와, 상기 전처리부가 상기 산출된 반사 거리에 대한 분할 지점에 따른 구간 별로 상기 센서 신호를 그룹화하여 반사거리 값을 저장하는 단계를 더 포함하며, 상기 Lj는 반사거리의 분할 지점이며, 상기 j는 반사거리의 분할 지점에 대한 인덱스이고, 상기

는 최대 반사거리이고, 상기 K는 반사거리 분할 수 인 것을 특징으로 한다. In the method, after the step of storing the reflection distance value for each direction angle, the pre-processing unit is based on the preset maximum reflection distance and the number of reflection distance divisions.

Calculating a split point for the reflection distance according to is a dividing point of the reflection distance, and j is an index to the dividing point of the reflection distance, and

is the maximum reflection distance, and K is the reflection distance division number.

According to the present invention, it is possible to generate a dataset in the form of a learnable input for using a deep neural network model for LiDAR sensor and RADAR sensor information acquired in a marine environment.

1 is a view for explaining a marine environment to which a navigation device according to an embodiment of the present invention is applied.
2 and 3 are block diagrams for explaining the configuration of a navigation device according to an embodiment of the present invention.
4 is a diagram for explaining the configuration of a deep neural network according to an embodiment of the present invention.
5 and 6 are diagrams for explaining the configuration of a deep neural network according to another embodiment of the present invention.
7 is a flowchart illustrating a method of generating input data for a deep neural network according to an embodiment of the present invention.
8 is a flowchart illustrating a method of generating learning data for a deep neural network according to an embodiment of the present invention.
9 is a screen example for explaining a method of generating learning data for a deep neural network according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in the present specification and claims described below should not be construed as being limited to their ordinary or dictionary meanings, and the inventors should develop their own inventions in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be appropriately defined as a concept of a term for explanation. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, so various equivalents that can be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that in the accompanying drawings, the same components are denoted by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings, and the size of each component does not fully reflect the actual size.

First, an apparatus for controlling a plurality of thrusters using a deep neural network in a marine environment according to an embodiment of the present invention will be described. 1 is a view for explaining a marine environment to which a navigation device according to an embodiment of the present invention is applied. Referring to FIG. 1 , an embodiment of the present invention assumes a situation in which a ship U and at least one target object obj exist in the sea.

The target object obj includes all kinds of objects that move on the sea or are installed on the sea. For example, the target object obj may be a ship, a buoy, or a marine structure.

In the embodiment of the present invention, the vessel (U) is equipped with a navigation device (10). In order to avoid the target object obj or move to the target object obj, the navigation device 10 needs to estimate relative information between the target object and the device, for example, a difference in heading direction angle and a relative distance.

On the other hand, although the ship U equipped with the navigation device 10 according to a preferred embodiment of the present invention is described as a representative embodiment, the present invention is not limited thereto, and the navigation device 10 moves in water as well as at sea level. It can also be applied to means, for example, submarines, submersible drones, and the like.

Then, the configuration of the navigation device 10 according to the embodiment of the present invention will be described in more detail. 2 and 3 are block diagrams for explaining the configuration of a navigation device according to an embodiment of the present invention. 4 is a diagram for explaining the configuration of a deep neural network according to an embodiment of the present invention. 5 and 6 are diagrams for explaining the configuration of a deep neural network according to another embodiment of the present invention.

First, referring to FIG. 2 , the navigation device 10 according to an embodiment of the present invention includes a camera unit 11 , a sensor unit 12 , a driving unit 13 , an input unit 14 , a display unit 15 , and a storage unit. (16) and a control unit (17).

The sensor unit 12 includes, for example, a plurality of sensors that generate a sensor signal SS by emitting light or radio waves, and then detecting a signal returned by reflecting the emitted signal to the object obj. Here, the sensor signal SS includes a direction angle indicating a reflected angle and a reflection distance indicating a reflected distance. The sensor unit 12 includes at least one of a 360 ° rotation type and a solid-state lidar (LiDAR) sensor 210 , a radar (RADAR) sensor 220 , and an Imaging Sonar sensor 230 . includes

The driving unit 13 is for driving a plurality of thrusters (Thruster: Thr.). Since the plurality of thrusters (Thr.) are installed to face each other in different directions, it is possible to control the moving direction and speed of the vessel (U) by adjusting the output of each of the plurality of thrusters (Thr.). For example, it is assumed that the first thruster Thr1 is installed to face the port and the second thruster Thr2 is installed to face the starboard. In this case, if the driving unit 13 causes the first thruster Thr1 to output 90% of the maximum output and the second thruster Thr2 to output 45% of the maximum output, the vessel U will move midway between the front and port side. The driving unit 13 may control the output of each of the plurality of thristors according to the control of the control unit 17 .

The input unit 14 may receive a user's operation for controlling the navigation device 10 , generate an input signal, and transmit it to the control unit 17 . The input unit 14 includes various buttons for controlling the navigation device 10 , a telegraph, a steering gear, and the like.

The display unit 15 is for screen display, and may visually provide a menu of the navigation device 10, input data, function setting information, and other various information to the user. The display unit 15 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 15 may be implemented as a touch screen. In this case, the display unit 15 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all types of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may detect a user's touch input, generate a detection signal including input coordinates indicating the touched position, and transmit it to the controller 17 .

The storage unit 16 serves to store programs and data necessary for the operation of the navigation device 10 . The storage unit 16 may store the sensor signal generated by the sensor unit 12 . Each type of data stored in the storage unit 16 may be deleted, changed, or added according to a user's operation.

The control unit 17 may control the overall operation of the navigation device 10 and the signal flow between internal blocks of the navigation device 10 and perform a data processing function of processing data. In addition, the control unit 17 basically serves to control various functions of the navigation device (10). The control unit 17 may be exemplified by a central processing unit (CPU), a digital signal processor (DSP), or the like.

Next, referring to FIG. 3 , the control unit 17 includes a preprocessing unit 310 , a learning unit 320 , a deep neural network 400 , and a control unit 500 .

The preprocessor 310 is for generating data input to the deep neural network 400 from the sensor signal SS generated by the sensor unit 12 . That is, the data input to the deep neural network 400 may be an information matrix derived from the sensor signal SS.

The learning unit 320 is for generating learning data for the deep neural network 400 and learning the deep neural network 400 using this.

The deep neural network 400 may be a convolutional neural network such as ResNet, EfficientNet, and Xception, or a recurrent neural network such as a Recurrent Neural Network (RNN) or Long Short-Term Memory (LTSM).

According to an embodiment of the present invention, when the deep neural network 400 is a convolutional neural network, the deep neural network 400 is an input layer (IL), at least a pair of alternately repeated convolution layers (convolution layer: CL) ), a pooling layer (PL), and at least one fully-connected layer (FL) and an output layer (OL).

As shown in FIG. 4 , the deep neural network 400 according to an embodiment of the present invention sequentially includes an input layer (IL), a convolution layer (CL), a pooling layer (PL), a fully connected layer (FL) and and an output layer OL.

The convolution layer CL and the pooling layer PL include at least one feature map (FM). The feature map FM is derived as a result of receiving a value to which a weight and a threshold are applied to the operation result of the previous layer, and performing an operation on the input value. These weights are applied through a filter or kernel W that is a weight matrix of a predetermined size. In the embodiment of the present invention, the first filter W1 is used for the convolution operation of the convolutional layer CL, and the second filter W2 is used for the pooling operation of the pooling layer PL.

When input data (a matrix or vector column of a predetermined size) is input to the input layer IL, the convolution layer CL performs convolution using the first filter W1 on the input data of the input layer IL. At least one first feature map FM1 is derived by performing an operation using an operation and an activation function. Next, the pooling layer PL performs a pooling or sub-sampling operation using the second filter W2 on at least one first feature map FM1 of the convolution layer CL to obtain at least one A second feature map FM2 is derived.

The final connection layer FL includes a plurality of operation nodes f1 to fn. The plurality of operation nodes f1 to fn of the final connection layer FL calculates a plurality of operation values through operation by an activation function with respect to at least one second feature map FM2 of the pooling layer PL.

The output layer OL includes a plurality of output nodes g1 to g6 . Each of the plurality of operation nodes f1 to fn of the final connection layer FL is connected to the output nodes g1 to g6 of the output layer OL through a channel having a weight (W). In other words, a weight is applied to the plurality of operation values of the plurality of operation nodes f1 to fn and is input to each of the plurality of output nodes g1 to g6 . Accordingly, the plurality of output nodes g1 to g6 of the output layer OL calculates an output value through an activation function operation for a plurality of calculated values to which the weight of the final connection layer FL is applied.

The activation functions used in the convolutional layer (CL), final connection layer (FL) and output layer (OL) described above are Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), and ReLU. (Rectified Linear Unit), Leakly ReLU, Maxout, Minout, Softmax, etc. can be exemplified. Any one of these activation functions may be selected and applied to the convolutional layer CL, the finite connection layer FL, and the output layer OL.

On the other hand, according to another embodiment of the present invention, when the deep neural network 400 is a recurrent neural network, the deep neural network 400 is composed of a plurality of stages (St). As shown, in the embodiment of the present invention, four stages S1 to S4 are assumed.

As shown in FIG. 5 , the deep neural network 400 according to another embodiment of the present invention includes an input layer (IL), a hidden layer (HL), and an output layer (OL).

The input layer IL includes a plurality of input cells IC1 , IC2 , IC3 , IC4 corresponding to the number of stages (eg, 4 ). Each of the plurality of input cells IC1, IC2, IC3, and IC4 receives input data X1, X2, X3, and X4 for each stage.

The hidden layer HL includes a plurality of hidden cells HC1, HC2, HC3, and HC4 corresponding to the number of stages. Each of the plurality of hidden cells HC1, HC2, HC3, and HC4 calculates a state value corresponding to the current stage by performing an operation on the state value to which the weight of the previous stage is applied and the input value to which the weight of the current stage is applied. .

The output layer OL may include a plurality of output cells OC1, OC2, OC3, and OC4 corresponding to the number of stages, but may include only one output cell OC of the last stage S4. have. The output layer OL receives a state value to which a weight is applied from the hidden layer HL, and calculates an output value by performing an operation.

6, the hidden cell HCt of the t-th stage St receives the input value Xt to which the input weight Wx is applied from the input cell ICt of the t-th stage St, and The state value Ht-1 of the previous stage to which the state weight Wht-1 is applied is received from the hidden cell HCt-1 of the stage, that is, the t-1 th stage St-1, and the input weight Wx is applied by applying a threshold value b The state value Ht of the current stage is calculated by performing an operation on the input value Xt to which is applied and the state value Ht-1 of the previous stage to which the state weight value Wht-1 is applied. The hidden cell HCt of the current stage may transmit the state value Ht of the current stage to the next stage St+1 by applying the state weight Wht. Also, the hidden cell HCt of the current stage may transmit the state value Ht of the current stage to the output cell OCt of the current stage by applying the output weight value Wyt. Then, the output cell OCt of the current stage calculates the output value Yt of the current stage by performing an operation on the state value Ht of the current stage to which the output weight Wyt is applied.

In the deep neural network 400 , the operation refers to an operation to which an activation function is applied. The activation function may be exemplified by Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), Rectified Linear Unit (ReLU), Leakly ReLU, Maxout, Minout, Softmax, and the like.

In summary, as described above, the deep neural network 400 may be a convolutional neural network or a recurrent neural network. This deep neural network 400 includes a plurality of layers. In addition, the plurality of layers of the deep neural network 400 includes a plurality of operations. The calculation result of each of the plurality of layers is transmitted to the next layer by applying parameters, ie, weights, thresholds, and the like.

The deep neural network 400 is for estimating the difference in heading direction angle and the relative distance between the navigation device 10 and the target object obj in the sea.

The control unit 500 may display the difference in the heading direction angle and the relative distance between the navigation device 10 and the target object obj estimated according to the output of the first deep neural network 410 on the display unit 15 .

The operation of the control unit 17 including the pre-processing unit 310, the learning unit 320, the first deep neural network 410, the second deep neural network 420, and the controller 500 described above will be described in more detail below. will be explained

Next, a method for generating input data for a deep neural network according to an embodiment of the present invention will be described. 7 is a flowchart illustrating a method of generating input data for a deep neural network according to an embodiment of the present invention.

, 방향각도 분할 수 N, 최대 반사거리

, 반사거리 분할 수 K 및 누적 시간 수 T를 포함한다. 장치(10)가 이동하는 전면을 기준인 0도로 했을 때, 센서부(12)의 센서 신호 중 방향 각도가 -90도에서 +90도 사이의 센서 신호만을 사용한다. 이에 따라, 전처리부(310)는 방향 각도의 화각

를 180도로 설정한다. 각도 분할 수 N은 본 발명의 실시예에 따른 심층신경망(400)의 입력층(IL)의 규격에 따라 결정될 수 있다. 특히, 각도 분할 수 N은 짝수인 정수이다. 또한, 전처리부(310)는 센서의 측정 최대 반사 거리를

로 정의한다. 즉, 센서 신호의 반사 세기의 최대값이

에 대응한다. 또한, 반사거리의 분할 수 K는 정수이다. Referring to FIG. 7 , the preprocessor 310 sets and stores parameters related to the sensor signal SS in step S110 . These parameters are the angle of view of the direction

, the number of divisions in the direction angle N, the maximum reflection distance

, the number of reflection distance divisions K and the number of cumulative times T. When the front surface on which the device 10 moves is set to 0 degrees as a reference, only sensor signals having a direction angle between -90 degrees and +90 degrees among the sensor signals of the sensor unit 12 are used. Accordingly, the pre-processing unit 310 determines the angle of view of the direction angle.

is set to 180 degrees. The angle division number N may be determined according to the standard of the input layer (IL) of the deep neural network 400 according to the embodiment of the present invention. In particular, the angle division number N is an even integer. In addition, the preprocessor 310 calculates the maximum reflected distance measured by the sensor.

is defined as That is, the maximum value of the reflection intensity of the sensor signal is

respond to In addition, the division number K of the reflection distance is an integer.

In the state in which the parameter is set as described above, the preprocessor 310 may receive the sensor signal SS sensed through the sensor unit 12 in step S120 . The lidar sensor 210 and the radar sensor 220 output the directional angle and reflection distance of light or radio waves reflected from the object obj on the sea with respect to the horizontal direction on the sea surface as a sensor signal SS. The image sonar sensor 230 outputs the directional angle and the reflection distance of the sound wave reflected from the object obj in the water with respect to the horizontal direction on the sea surface as a sensor signal SS. The preprocessor 310 receives the sensor signal SS including the directional angle and the reflection distance as described above, and converts it into input data of the deep neural network 400 . A detailed description of this is as follows.

, 방향각도 분할 수 N를 기초로 센서 신호(SS)의 방향 각도에 대한 분할 지점

을 구하고, 그 방향각도의 분할 지점을 기초로 센서 신호(SS)를 구분하여 그룹화한다. 보다 상세하게 설명하면 다음과 같다. First, the preprocessor 310 classifies and groups the sensor signals SS according to the direction angle of the sensor signals SS in step S130 . In this case, the pre-processing unit 310 sets the angle of view of the preset direction angle.

, the dividing point for the directional angle of the sensor signal SS based on the directional angle division number N

, and the sensor signals SS are classified and grouped based on the dividing point of the direction angle. A more detailed description is as follows.

를 180도로 설정하였다. 그리고 각도 분할 수 N은 짝수인 정수이다. 이에 따라, 전처리부(310)는 기 설정된 방향 각도의 화각

, 방향각도 분할 수 N를 기초로 다음의 수학식 1에 따라 센서 신호(SS)의 방향 각도에 대한 분할 지점

을 구할 수 있다. As described above, when the front surface on which the device 10 moves is set to 0 degrees as a reference, only the sensor signals SS having a direction angle between -90 degrees and +90 degrees among the sensor signals of the sensor unit 12 are used. Because of the directional angle, the angle of view

was set to 180 degrees. And the angle division number N is an even integer. Accordingly, the pre-processing unit 310 sets the angle of view of the preset direction angle.

, the division point for the direction angle of the sensor signal SS according to the following Equation 1 based on the number of divisions of the direction angle N

can be obtained

는 방향각도의 화각이다. 예컨대, 수학식 1에 따라 방향각도의 분할지점을 구할 때, N이 18이면,

이 180도이므로, 방향각도를 -90도에서 +90도까지 10도씩 분할하여 취급하는 것을 의미한다. 이와 같이, 방향도의 분할 지점이 결정되면, 전처리부(310)는 센서신호(SS)를 방향각도에 따라 분할 지점 θi부터 θi+1까지의 구간 별로 그룹화하여 방향각도 별 반사거리 값을

버퍼 리스트에 저장한다.

은 가변길이의 리스트 데이터 구조가 1차원의

개로 구성된다. Here, θi is the dividing point of the direction angle, i is the index of the dividing point of the direction angle, and N is the number of dividing the direction angle. and

is the angle of view of the directional angle. For example, when calculating the dividing point of the directional angle according to Equation 1, if N is 18,

Since this is 180 degrees, it means that the direction angle is divided by 10 degrees from -90 degrees to +90 degrees and handled. In this way, when the dividing point of the direction map is determined, the preprocessor 310 groups the sensor signal SS for each section from the dividing point θi to θi+1 according to the direction angle to obtain a reflection distance value for each direction angle.

Save it to the buffer list.

is a one-dimensional list data structure of variable length.

made up of dogs.

및 반사거리 분할 수 K를 기초로 반사 거리에 대한 분할 지점 Lj를 구하고, 그 반사 거리의 분할 지점을 기초로 센서 신호(SS)를 구분하여 그룹화한다. 보다 상세하게 설명하면 다음과 같다. Next, the preprocessor 310 classifies the grouped sensor signals SS according to the direction angle in step S140 according to the reflection distance of the sensor signals SS and groups them. At this time, the pre-processing unit 310 is a preset maximum reflection distance

and a division point Lj for the reflection distance is obtained based on the reflection distance division number K, and the sensor signals SS are classified and grouped based on the division point of the reflection distance. A more detailed description is as follows.

및 반사거리 분할 수 K를 기초로 다음의 수학식 2에 따라 반사 거리에 대한 분할 지점 Lj를 산출한다. At this time, the pre-processing unit 310 is a preset maximum reflection distance

and a division point Lj for the reflection distance is calculated according to the following Equation 2 based on the reflection distance division number K.

는 최대 반사거리이고, K은 반사거리 분할 수이다. Here, Lj is the split point of the reflection distance, j is the index to the split point of the reflection distance,

is the maximum reflection distance, and K is the number of reflection distance divisions.

Next, the preprocessor 310 groups the reflection distance values stored in each list by section from the dividing point Lj to Lj+1 while increasing n with respect to Mn, so that Mnk(n=0...N-1, k= 0...K-1) is saved in the buffer list. In Mnk, a variable-length list data structure is composed of two-dimensional (K, N).

Next, the preprocessor 310 derives the minimum value among the multiple reflection distance values stored in each list of Mnk while increasing n and k for Mnk in step S150, and a two-dimensional informatization matrix of (K, N) size At to create

When T is a preset number of accumulated times and is a positive integer, the preprocessor 310 performs the informatization matrix At-(T-1) for time t-T+1 from time t-T+1 to time t in step S160. It is determined whether or not the information matrix At for time t has been accumulated and generated, and if not generated, steps S110 to S160 described above are repeated. On the other hand, when all T informatization matrices are generated from the informatization matrix At-(T-1) for time t-T+1 to the informatization matrix At for time t from time t-T+1 to time t, the preprocessor (310) merges all of the informatization matrix At for time t from the informatization matrix At-(T-1) for time t-T+1 in step S170 to form a three-dimensional informatization matrix S of (K, N, T) to create

Then, the preprocessor 310 obtains the maximum value vmax among the values of all elements of the three-dimensional informatization matrix S through the preprocessor 310 in step S180, and divides each element of S by vmax to a real value between 0 and 1. By transforming, a normalized three-dimensional informatization matrix S' can be generated. This step S190 is optional and may be omitted.

When the deep neural network 400 is a convolutional neural network such as the CNN of FIG. 4, as the input data to the deep neural network 400, the three-dimensional informatization matrix S or the normalized three-dimensional informatization matrix S' is used as it is, or the order of the dimensions is used. It can be converted and used. When the deep neural network 400 is a recurrent neural network such as the RNN of FIG. 5 , the dimension of the three-dimensional informatization matrix S or the normalized three-dimensional informatization matrix S' as input data to the deep neural network 400 is (N, T, K) ) can be converted to In this case, time T corresponds to a plurality of stages.

Next, a method for generating learning data for a deep neural network according to an embodiment of the present invention will be described. 8 is a flowchart illustrating a method of generating learning data for a deep neural network according to an embodiment of the present invention. 9 is a screen example for explaining a method of generating learning data for a deep neural network according to an embodiment of the present invention.

Referring to FIG. 8 , the learning unit 320 includes a plurality of 2 corresponding to each of a plurality of sensor signals SS output by the sensor unit 12 through measurement for a predetermined time or more through the pre-processing unit 310 in step S210 . After generating the dimensional informatization matrix At, the input data S having a three-dimensional structure is generated by accumulating it again on the time axis for a predetermined time T. Then, the learning unit 320 selectively obtains the maximum value vmax among the values of all elements of the input data S through the preprocessing unit 310, divides each element of S by vmax, and converts it into a real value between 0 and 1 and input Data can be normalized.

Next, as shown in FIG. 9 in step S220, the learning unit 320 generates an image by visualizing the sensor signal SS, and at least one target object obj from the image obtained by visualizing the sensor signal SS. ) is detected through the bounding box (B), and the center point (dx, dy) of the bounding box (B) is calculated. The area box (B) represents the area occupied by the target object obj in the sea in the image visualized with the sensor signal SS. According to an embodiment of the present invention, the learning unit 320 may automatically detect the area box B of the target object obj through an image processing algorithm from a visualized image of the sensor signal SS. According to another embodiment, an image in which the area box B of the target object obj is manually specified may be input using labeling software.

The learning unit 320 obtains a focal length F in step S230. In this case, the focal length F may be calculated according to Equation 3 below.

는 화각을 나타낸다. 라이다센서(210), 레이다센서(220) 및 영상소나센서(230)의 초점 거리 F는 최대 측정 거리값을 사용한다. Here, W is the number of pixels in the vertical direction of the image visualizing the sensor signal SS,

represents the angle of view. The focal length F of the lidar sensor 210 , the radar sensor 220 , and the image sonar sensor 230 uses the maximum measured distance value.

를 산출한다. The learning unit 320 in step S240 according to the following equation (4) in each of the plurality of sensor signals (SS) corresponding to the plurality of two-dimensional informatization matrix At heading direction angle between the navigation device 10 and the target object (obj) difference

to calculate

Here, dx is the x-coordinate of the center point (dx, dy) of the area box B of the target object obj. W is the number of pixels in the vertical direction of the sensor signal SS, and F is the focal length.

를 산출한다. The learning unit 320 is a relative distance between the navigation device 10 and the target object obj in each of the plurality of sensor signals SS corresponding to each of the plurality of two-dimensional informatization matrices At according to the following Equation 5 in step S250

to calculate

를 이용하여 산출될 수 있다. Here, dx is the x-coordinate of the center point (dx, dy) of the area box B of the target object obj, and dy is the y-coordinate of the center point (dx, dy) of the area box B of the target object obj to be. W is the number of pixels in the horizontal direction of the image in which the sensor signal SS is visualized, H is the number of pixels in the vertical direction in the image in which the sensor signal SS is visualized, and F is the focal length. As described above, the focal length F is the number of pixels W in the horizontal direction and the angle of view in the image obtained by visualizing the sensor signal SS according to Equation 3

can be calculated using

와 상대 거리

를 입력데이터 S의 복수의 정보화 행렬 At에 레이블링(Labeling)한다. 이로써, 학습 데이터가 생성된다. The learning unit 320, as described above in step S260, the heading direction angle difference between the navigation device 10 and the target object obj obtained from each of the plurality of sensor signals SS corresponding to each of the plurality of two-dimensional informatization matrices At

and relative distance

is labeled with a plurality of informatization matrices At of the input data S. In this way, learning data is generated.

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program readable by various computer means and recorded in a computer readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language wires such as those generated by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the present invention has been described above using several preferred embodiments, these examples are illustrative and not restrictive. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made in accordance with the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: navigation device 12: sensor unit
13: driving unit 14: input unit
15: display unit 16: storage unit
17: control unit 210: lidar sensor
220: radar sensor 230: image sonar sensor
310: preprocessing unit 320: learning unit
400: deep neural network 500: control unit

Claims (6)

A method for estimating relative information between a target object and a device using a plurality of radars and lidar in a marine environment, the method comprising:
receiving, by a pre-processing unit, a sensor signal having a direction angle and a reflection distance of light or radio waves reflected from an object through a sensor unit;
generating, by the pre-processing unit, a plurality of groups by dividing the sensor signal according to the direction angle and the reflection distance;
generating, by the pre-processing unit, a two-dimensional informatization matrix by deriving a minimum value of a reflection distance from each of the plurality of groups;
characterized in that it comprises
A method for estimating relative information. According to claim 1,
generating input data having a three-dimensional structure in which a plurality of two-dimensional informatization matrices are accumulated on a time axis by accumulating, by the pre-processing unit, the two-dimensional informatization matrix for a predetermined time;
The learning unit visualizes a plurality of sensor signals corresponding to each of the plurality of two-dimensional informatization matrices included in the input data to generate an image, and an area box indicating an area occupied by a target object in the visualized image of each of the plurality of sensor signals detecting a target object through and calculating a center point of the area box;
calculating, by the learning unit, a difference in heading direction angle between a navigation device and a target object from a plurality of sensor signals corresponding to the plurality of two-dimensional informatization matrices;
calculating, by the learning unit, a relative distance between a navigation device and a marine target object from each of a plurality of sensor signals corresponding to the plurality of two-dimensional informatization matrices; and
labeling, by the learning unit, the difference in the heading direction angle and the relative distance to the plurality of two-dimensional informatization matrices;
characterized in that it further comprises
A method for estimating relative information. 3. The method of claim 2,
The step of calculating the difference in the heading direction angle is
the learning department
formula

Calculate the difference in the heading direction angle according to
The dx is the x-coordinate of the center point of the area box with respect to the target object,
W is the number of pixels in the horizontal direction of the image visualized with the sensor signal,
Wherein F is the focal length
A method for estimating relative information. 3. The method of claim 2,
The step of calculating the relative distance is
the learning department
formula

Depending on the
Calculate the relative distance,
The dx is the x-coordinate of the center point of the area box of the target object,
Wherein dy is the y coordinate of the center point of the area box of the target object,
W is the number of pixels in the horizontal direction of the image visualized with the sensor signal,
Wherein H is the number of pixels in the vertical direction of the image visualizing the sensor signal,
Where F is the focal length
characterized by
A method for estimating relative information. According to claim 1,
The step of the pre-processing unit dividing the sensor signal into a plurality of groups according to the direction angle and the reflection distance
Equation based on the number of divisions of the angle of view and the direction angle of the preset direction angle by the preprocessor

calculating a dividing point for the direction angle of the sensor signal according to and
storing, by the pre-processing unit, a reflection distance value for each direction angle by grouping the sensor signals for each section according to a division point for the calculated direction angle;
includes,
The θi is the dividing point of the direction angle,
where i is an index to the dividing point of the direction angle,
where N is the number of divisions in the direction angle,
remind

is the angle of view of the directional angle, characterized in that
A method for estimating relative information. 6. The method of claim 5,
After the step of storing the reflection distance value for each direction angle,
Equation based on the preprocessor preset maximum reflection distance and number of reflection distance divisions

calculating a split point for the reflection distance according to and
storing, by the pre-processing unit, a reflection distance value by grouping the sensor signals for each section according to a division point for the calculated reflection distance;
further comprising,
The Lj is the dividing point of the reflection distance,
Where j is the index for the split point of the reflection distance,
remind

is the maximum reflection distance,
Wherein K is a reflection distance division number, characterized in that
A method for estimating relative information.

Images