KR102199627B1 - Apparatus for recognizing approaching vessel considering distance objects based on Deep Neural Networks, method therefor, and computer recordable medium storing program to perform the method - Google Patents

Abstract

The present invention provides a device for recognizing an approaching vessel considering a distance of a marine object based on a deep neural network, comprising: a camera unit photographing a surveillance image; a preprocessing unit detecting an object image by specifying an area box indicating an area occupied by an object included in the photographed surveillance image through feature point detection; a deep neural network outputting whether or not the object of the object image is a vessel; and a control unit determining whether or not the object is a vessel in accordance with the probability, calculating the degree of risk indicating the probability that a collision with the vessel occurs if the object is the vessel as a result of the determination, and warning the risk of collision if the calculated risk is greater than or equal to a threshold value.

Description

Apparatus for recognizing approaching vessel considering distance objects based on Deep Neural Networks, a device for recognizing an approaching vessel based on a deep neural network, a method for this, and a program that performs this method. , method therefor, and computer recordable medium storing program to perform the method}

The present invention relates to an approaching vessel recognition technology, and more particularly, an apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network (DNN), a method therefor, and a method for performing the method. It relates to a computer-readable recording medium on which a program is recorded.

To this day, ship accidents such as fire, flooding, and overturning occur frequently at sea. According to an investigation by the Maritime Police Agency, in 2018, maritime distress accidents occurred on 3,434 ships, of which 49 ships sank, and 89 appointments were killed or missing. Such accidents are increasing year by year, and there are problems such as the ship's supervisor sleeping or not properly inspecting the surrounding situation as the cause of the accident. An example of this is an accident in which a Russian cargo ship collided with Busan Gwangan Bridge in February last year.

Many studies are being conducted as a way to improve ship accidents that occur due to the carelessness of the ship manager in the sea where the possibility of spreading from minor accidents to major accidents is high. Among them, accidents can be prevented through research that recognizes vessels in images using deep neural networks, but real-time processing is impossible due to the large amount of computation of the deep neural network model while enlarging the image to recognize even distant vessels.

Korean Patent Laid-Open Patent No. 2019-0024400 published on March 08, 2019 (Name: object recognition device and control method thereof)

The present invention was conceived to solve the above-described problem, and the object of the present invention is to predict a ship in an image using image processing to enable real-time processing in order to prevent accidents caused by carelessness of a ship manager at sea. A device that detects the object area to be detected, recognizes the detected object image by inputting it into the deep neural network, and analyzes the recognized object area to warn the person in charge of the ship, a method for this, and a computer in which a program that performs this method is recorded It is to provide a readable recording medium.

An apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to a preferred embodiment of the present invention for achieving the above object includes a camera unit for photographing a surveillance image, and the photographed surveillance image A preprocessor for detecting an object image by specifying an area box representing an area occupied by an object included in the object through feature point detection, a deep neural network that outputs with probability whether the object of the object image is a ship, and It determines whether the object is a ship, and as a result of the determination, if the object is a ship, a risk indicating a probability of occurrence of a collision with the ship is calculated, and a control unit for warning of a collision risk if the calculated risk is greater than or equal to a threshold value.

When a feature point is detected in the surveillance image, the pre-processor detects a plurality of corner points in the surveillance image using a Harris corner algorithm, and then searches for a dense area of the detected corner points to an area occupied by the object. A region box representing is specified, and the object image is detected through the specified region box.

If the feature point is not detected in the image, the preprocessor detects a horizontal line in the surveillance image, enlarges the image above the detected horizontal line to a predetermined size, and then performs a Harris Corner algorithm on the image above the enlarged horizontal line. After detecting a plurality of corner points by using, the object image is detected through the specified area box by specifying an area box representing an area occupied by an object by finding a dense area of the detected plurality of corner points. .

The control unit calculates the width of the object image through the coordinates of the area box of the object, and determines that the risk is greater than or equal to the threshold when the ratio of the area of the surveillance image to the area of the object image is greater than or equal to a preset threshold. It is characterized.

The deep neural network includes at least one convolutional layer including an input layer into which an object image is input, at least one feature image derived by a convolution operation on the object image or feature image, and a feature image generated through a convolution operation. Including at least one pooling layer for deriving at least one feature image derived through a pooling operation, and a plurality of operation nodes for calculating a node value through an operation by an activation function by receiving the feature image or the node value of the previous layer. And an output layer including at least one fully connected layer, and a plurality of output nodes for receiving node values of the fully connected layer and calculating an output value through an operation by an activation function.

An apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to a preferred embodiment of the present invention for achieving the above object includes a camera unit for photographing a surveillance image including a plurality of frames, and , A preprocessor for detecting a plurality of object images through an area box indicating an area occupied by an object in each of the plurality of frames, and a plurality of navigation vectors arranged in chronological order corresponding to each of the plurality of object images. A deep neural network that generates and calculates a navigation prediction vector that predicts the navigation state of the object after a predetermined time by performing a plurality of calculations to which weights are applied to the plurality of navigation vectors, and after the predetermined time from the navigation prediction vector An area box indicating the navigation direction of the object and the area occupied by the object in the surveillance image is derived, the derived navigation direction is a collision possible direction, and the derived area box is at least partially with a preset warning area in the surveillance image. When overlapped, it includes a control unit that warns of the risk of collision.

The deep neural network includes a direction identification network for deriving a plurality of direction vectors representing a navigation direction of each object of the plurality of object images, and the plurality of object images on the surveillance image of each of the plurality of object images. And an adder for generating the plurality of navigation vectors by combining a region vector indicating coordinates of the region box and a time vector indicating a time when each of the plurality of object images is generated.

The deep neural network consists of a plurality of stages arranged in sequence, and the state value of the current stage is calculated by performing an operation in which the state and input weights are applied to the state value of the previous stage and the navigation vector, which is the input value of the current stage. Thereafter, a first hidden cell group including a plurality of hidden cells that transmits the calculated state value to the next stage and a state weight of the previous stage are calculated to calculate the state value of the current stage. Thereafter, a second hidden cell group including a plurality of hidden cells that transmits the calculated state value to the next stage, and an operation in which state weights are applied to the state values of the previous stage are performed to calculate the state value of the current stage. Thereafter, a navigation prediction network including a third hidden cell group including a hidden cell for calculating a navigation prediction vector as an output value by performing an operation of applying an output weight to the calculated state value of the current stage is further included.

The navigation prediction network performs at least one operation in which weights are applied in the order of the plurality of stages in correspondence with a circular input layer receiving a plurality of navigation vectors arranged according to a time order, and each of the plurality of navigation vectors. And a cyclic hidden layer including a plurality of hidden cells for calculating the navigation prediction vector, and a cyclic output layer for outputting the calculated risk.

When a feature point is detected in the surveillance image, the pre-processor detects a plurality of corner points in the surveillance image using a Harris corner algorithm, and then searches for a dense area of the detected corner points to an area occupied by the object. If a region box indicating a region box is specified, the object image is detected through the specified region box, and a feature point is not detected in the image, a horizontal line is detected in the surveillance image, and the image above the detected horizontal line is set to a predetermined size. After magnifying with, after detecting a plurality of corner points using the Harris Corner algorithm in the image above the enlarged horizontal line, the area box representing the area occupied by the object is specified by searching for a dense area of the detected plurality of corner points. And, it characterized in that the object image is detected through the specified area box.

A method for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to a preferred embodiment of the present invention for achieving the above-described object includes the steps of a camera unit photographing a surveillance image, and a preprocessor unit The step of detecting an object image by specifying an area box representing the area occupied by the object included in the surveillance image through feature point detection, the deep neural network outputting with probability whether the object of the object image is a ship, and the control unit Determining whether the object is a ship according to the probability, and if the control unit is the object as a result of the determination, calculating a risk indicating a probability of occurrence of a collision with the ship, and the control unit And warning of a collision risk if the calculated risk is greater than or equal to a threshold.

The detecting of the object image includes: when the preprocessor detects a feature point in the surveillance image, detecting a plurality of corner points in the surveillance image using a Harris corner algorithm, and the preprocessor detects the detected feature point. And specifying an area box representing an area occupied by an object by searching for a dense area of a plurality of corner points, and detecting the object image through the specified area box by the preprocessor.

The detecting of the object image includes: if the preprocessor does not detect a feature point in the image, detecting a horizontal line in the surveillance image; and expanding the image above the detected horizontal line to a predetermined size by the preprocessor; , The preprocessing unit detects a plurality of corner points using a Harris corner algorithm in the image above the enlarged horizontal line, and the preprocessor searches for a dense area of the detected corner points and indicates an area occupied by an object. And specifying an area box, and detecting the object image through the specified area box by the preprocessor.

The calculating of the risk may include: calculating, by the control unit, the width of the object image through coordinates of the area box of the object, and the ratio of the area of the surveillance image to the area of the object image by the control unit is equal to or greater than a preset threshold. If so, it includes determining that the risk is equal to or greater than a threshold.

In the deep neural network outputting with probability whether the object of the object image is a ship, the input layer of the deep neural network receives the object image, and the first convolutional layer of the deep neural network filters the object image. Deriving at least one feature image by performing a convolution operation using, and at least one feature image by performing a pooling operation using a filter on the feature image of the first convolutional layer by the first pooling layer of the deep neural network Deriving at least one feature image by performing a convolution operation using a filter on the feature image of the first pooling layer by the second convolutional layer of the deep neural network, and the second convolutional image of the deep neural network The step of deriving at least one feature image by performing a pooling operation using a filter on the feature image of the second convolutional layer by a pooling layer, and a plurality of operation nodes of the first complete connection layer of the deep neural network Calculating a node value through an operation using an activation function for the feature image of the pooling layer, and a plurality of operation nodes of the second complete connection layer of the deep neural network with respect to the node value of the first complete connection layer Calculating a node value through calculation by, and whether the object of the object image is a ship through an operation by an activation function for the node value of the second fully connected layer by a plurality of output nodes of the output layer of the deep neural network And calculating an output value that is a probability of whether or not.

According to another aspect of the present invention, there is provided a computer-readable recording medium in which a program for performing a method for recognizing an approaching vessel according to an embodiment of the present invention as described above is recorded.

According to the present invention, a situation leading to a major accident such as a ship collision that occurs due to the carelessness of the ship manager can be prevented by searching for other ships around the ship and notifying the ship manager in case of danger. In addition, it is possible to reduce the installation cost by using image processing specialized for ship recognition and ship recognition algorithm using deep learning model, rather than using a deep learning model that requires high computer power and cost.

1 is a view for explaining the configuration of an apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to an embodiment of the present invention.
2 to 4 are screen examples for explaining the operation of an apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to an embodiment of the present invention.
5 is a view for explaining the configuration of a deep neural network according to the first embodiment of the present invention.
6 to 9 are diagrams for explaining the configuration of a deep neural network according to a second embodiment of the present invention.
10 is a flowchart illustrating a method for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to the first embodiment of the present invention.
11 is a flowchart illustrating a method of detecting an object image according to an embodiment of the present invention.
12 is a diagram illustrating a method of detecting an object image corresponding to a near object according to an embodiment of the present invention.
13 is a diagram for describing a method of detecting an object image corresponding to a distant object according to an embodiment of the present invention.
14 is a flowchart illustrating a method for recognizing an approaching vessel in consideration of a maritime object distance based on a deep neural network according to a second embodiment of the present invention.

Prior to the detailed description of the present invention, terms or words used in the present specification and claims described below should not be construed as being limited to their usual or dictionary meanings, and the inventors shall use their own invention in the best way. For explanation, based on the principle that it can be appropriately defined as a concept of terms, it should be interpreted as a meaning and concept consistent with the technical idea of the present invention. Therefore, the embodiments described in the present specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all the technical spirit of the present invention, and various equivalents that can replace them at the time of application It should be understood that there may be water and variations.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this case, it should be noted that the same components in the accompanying drawings are indicated by the same reference numerals as possible. In addition, detailed descriptions of known functions and configurations that may obscure the subject matter of the present invention will be omitted. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

First, an apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to an embodiment of the present invention will be described. 1 is a view for explaining the configuration of an apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to an embodiment of the present invention. 2 to 4 are screen examples for explaining the operation of an apparatus for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus for recognizing an approaching vessel (10: hereinafter, abbreviated as'control device') according to an embodiment of the present invention includes a communication unit 11, an input unit 13, a display unit 14, and a storage unit. It includes a unit 15 and a control unit 16.

The communication unit 11 is for communicating, for example, a user device of an administrator such as a mobile phone, a smart phone, a mobile communication terminal, etc., various alarms installed on the ship, and a control server for navigation and other management of the ship. The communication unit 11 may include a radio frequency (RF) transmitter Tx for up-converting and amplifying a frequency of a transmitted signal, and an RF receiver Rx for low-noise amplifying and down-converting a received signal. In addition, the communication unit 11 may include a modem that modulates the transmitted signal and demodulates the received signal.

The camera unit 12 is for photographing an image. In particular, the camera unit 12 according to an embodiment of the present invention may photograph horizontal and vertical 360 degrees in all directions. Accordingly, the camera unit 12 can photograph all 360 degrees around the ship. To this end, the camera unit 12 may include a plurality of lenses and a plurality of image sensors. The image sensor receives light reflected from a subject and converts it into an electric signal, and may be implemented based on a Charged Coupled Device (CCD) or Complementary Metal-Oxide Semiconductor (CMOS). The camera unit 12 may further include one or more analog to digital converters, and may convert an electrical signal output from the image sensor into a digital sequence and output it to the controller 16.

The input unit 13 receives a user's key manipulation for controlling the control device 10, generates an input signal, and transmits the input signal to the control unit 16. The input unit 13 can control various types of control devices 10. When the display unit 14 is made of a touch screen, the input unit 13 may perform functions of various keys on the display unit 14, and when all functions can be performed only with the touch screen, the input unit ( 13) may be omitted.

The display unit 14 is for screen display and may visually provide a menu of the control device 10, input data, function setting information, and various other information to a user. In addition, the display unit 14 performs a function of outputting screens such as a boot screen, a standby screen, a menu screen, and the like of the control device 10. The display unit 14 may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), or the like. Meanwhile, the display unit 14 may be implemented as a touch screen. In this case, the display unit 14 includes a touch sensor. The touch sensor detects a user's touch input. The touch sensor may be composed of a touch sensing sensor such as a capacitive overlay, a pressure type, a resistive overlay, or an infrared beam, or may be composed of a pressure sensor. . In addition to the above sensors, all kinds of sensor devices capable of sensing contact or pressure of an object may be used as the touch sensor of the present invention. The touch sensor may sense a user's touch input, generate a sensing signal including input coordinates indicating the touched position, and transmit the sensing signal to the controller 16. In particular, when the display unit 14 is formed of a touch screen, some or all of the functions of the input unit 13 may be performed through the display unit 14.

The storage unit 15 serves to store programs and data necessary for the operation of the control device 10. The storage unit 15 may store various types of images and images used for object recognition according to an exemplary embodiment of the present invention for a predetermined period. Each type of data stored in the storage unit 15 may be deleted, changed, or added according to a user's manipulation.

The controller 16 may perform a data processing function of controlling the overall operation of the control device 10 and a signal flow between internal blocks of the control device 10, and processing data. In addition, the control unit 16 basically performs a role of controlling various functions of the control device 10. The control unit 16 may be a central processing unit (CPU), a digital signal processor (DSP), or the like. Next, the control unit 16 includes a preprocessor 100, a deep neural network 200, a learning unit 300, and a control unit 400.

The present invention includes the first and second embodiments, and the control unit 16 may recognize an approaching vessel in consideration of the distance of a sea object according to the first and second embodiments.

Specifically, a first embodiment of the present invention will be described with reference to FIG. 2. According to the first embodiment of the present invention, the preprocessing unit 100 of the control unit 16 photographs the surroundings of the ship through the camera unit 12 to generate a surveillance image SV of a predetermined area (SW×SH). Then, the preprocessor 100 detects an area occupied by the object obj included in the captured image through the area box B to generate the object image OV. The area box B represents the area occupied by the object obj in the surveillance image SV as a rectangle, and means a shooting type having the minimum size in which all the objects obj are included. This area box B has a central coordinate (x, y), a width (w) and a height (h).

Meanwhile, the object image OV detected by the preprocessor 100 is input to the deep neural network 200 according to the first embodiment. Then, the deep neural network 200 calculates as a probability whether the object obj of the object image OV is a ship. The control unit 400 determines whether the object obj is a ship according to the probability calculated by the deep neural network 200, and as a result of the determination, if the object obj is a ship, it indicates the probability of a collision with the ship. Calculate the risk level.

The control unit 400 calculates the width (w×h) of the object image through the coordinates of the area box B of the object obj in the surveillance image SV, and then calculates the width of the surveillance image SV (SW× The risk can be calculated through the ratio of the area (w×h) of the object image (OV) to the SH). The control unit 400 determines that there is a risk of collision if the calculated risk is greater than or equal to a preset threshold. Accordingly, the control unit 400 transmits an alarm message notifying the risk of collision to a user device, an alarm, and a control server through the communication unit 11.

Meanwhile, a second embodiment of the present invention will be described with reference to FIGS. 2, 3 and 4. According to the second embodiment, the preprocessing unit 100 of the control unit 16 photographs the surroundings of the ship through the camera unit 12 and generates a plurality of surveillance images SVt arranged in chronological order. The surveillance image SV includes a plurality of frames photographed as a video. In particular, according to an embodiment, when the frame is composed of an I-frame (Intra-coded frame), a P-frame (Predictive-coded frame) and a B-frame (Bidirectional-coded frame), the surveillance image (SVt) is I -Only Intra-coded frames may be provided. According to another embodiment, a specific frame may be extracted and provided according to a predetermined period. For example, the surveillance image SVt may provide one frame per second. In addition, the preprocessor 100 detects an area occupied by the object obj included in each of the plurality of surveillance images SVt through the area box Bt, and generates a plurality of object images OVt arranged in chronological order. Generate.

Meanwhile, a plurality of object images OVt arranged according to time order are input to the deep neural network 200 according to the second embodiment. Then, the deep neural network 200 determines whether the object included in each of the plurality of object images OVt is a ship, and if the object included in each of the plurality of object images OVt is a ship, the plurality of object images OVt A plurality of direction vectors Dt representing the navigation direction of the object obj are derived. As shown in FIG. 3, the direction in which the ship is currently sailing may be detected through the object image OVt. Accordingly, values of the detected direction vector Dt are shown in Table 1 below.

Direction (degrees) value 0(360) 0 0 0 0 0 0 0 One 45 0 0 0 0 0 0 One 0 90 0 0 0 0 0 One 0 0 135 0 0 0 0 One 0 0 0 180 0 0 0 One 0 0 0 0 225 0 0 One 0 0 0 0 0 270 0 One 0 0 0 0 0 0 315 One 0 0 0 0 0 0 0

As described above, in the embodiment of the present invention, the plurality of direction vectors Dt divide eight directions, and the value of the direction vector Dt is described as having a value by one-hot encoding. The invention is not limited thereto, and those of ordinary skill in the art will appreciate that there may be various changes and modifications in the number of directions of the direction vector Dt and the encoding method. Further, the deep neural network 200 generates a plurality of navigation vectors (Xt) by adding a region vector (Bt) and a time vector (Tt) corresponding to each of the plurality of direction vectors (Dt). Here, the region vector Bt represents an area occupied by each of the plurality of object images OVt in the surveillance image SV. This region vector Bt can be expressed through the coordinates (x, y, w, h) of the region box B. The time vector Tt represents the time when each of the plurality of object images OVt is generated. And the deep neural network 200 is the navigation state of the object (obj) after a predetermined time from a plurality of navigation vectors (Xt) each including a direction vector (Dt), a region vector (Bt) and a time vector (Tt), that is, , Calculate the navigation prediction vector (Yk) representing the navigation direction, navigation area and time.

The navigation prediction vector (Yk) includes the predicted direction vector (Dk), the region vector (Bk) and the time vector (Tk), and the control unit 400 is derived from the direction vector (Dk) of the navigation prediction vector (Yk). When the heading toward which is directed toward a possible collision area and the area occupied by the area box B derived from the area vector Bk overlaps at least partially with a preset warning area in the surveillance image SV, an alarm is issued. For example, the surveillance image (SV) of FIG. 4 is divided into 12 cells of 3 rows and 4 columns, and cells of 2 rows 3 columns and 3 rows 3 columns [(2, 3), (3, 3)] are warning areas, A direction toward the cell may be set as a possible collision direction. For example, according to the derivation, the direction vector Dk of the navigation prediction vector Yk is aimed at 3 rows and 3 columns (3, 3), which is a possible collision direction at 180 degrees, and the 3 rows and 3 columns of the surveillance image SV ( The cells of 3 and 3) and the region box B of the region vector Bk of the navigation prediction vector Yk are partially overlapped. Accordingly, the control unit 400 determines that the risk is equal to or greater than a preset threshold, and transmits an alarm message notifying the risk of collision to the user device, the alarm, and the control server through the communication unit 11.

Next, a deep neural network according to an embodiment of the present invention will be described in more detail. First, a deep neural network according to a first embodiment of the present invention will be described. 5 is a view for explaining the configuration of a deep neural network according to the first embodiment of the present invention.

Referring to FIG. 5, the deep neural network 200 according to the first embodiment includes an object identification network 210. The object identification network 210 includes a plurality of layers. That is, the object identification network 210 includes an input layer (INL), a hidden layer, and an output layer (OUL). The hidden layer includes one or more convolution layers (CVL), one or more pooling layers (POL), and one or more fully-connected layers (FCL). Referring to FIG. 2, the object identification network 210 according to the first embodiment of the present invention sequentially includes an input layer (INL), a first convolution layer (CVL1), a first pooling layer (POL1), and a second convolution layer. It includes a layer CVL2, a second pooling layer POL2, a first fully connected layer FCL1, a second fully connected layer FCL2, and an output layer OUL.

The input layer INL consists of an input matrix having a predetermined size. Elements of the input matrix of the input layer INL correspond to a plurality of pixels of the object image OV. Accordingly, a plurality of pixel values of the object image OV are input as values of each element of the input layer INL input matrix.

The first and second convolution layers CVL1 and CVL2 and the first and second pooling layers POL1 and POL2 are composed of at least one feature map (FM). The feature image FM is composed of a matrix of a predetermined size made of pixel values. The feature image FM is generated as a result of performing an operation to which a weight is applied to the value of the previous layer. These weights are applied through the filter (W). Here, the filter is a matrix of a predetermined size (for example, when N is a natural number, N×N), and the value of each element of the matrix forming the filter becomes the weight (w). Accordingly, a value calculated according to a weighted operation using the weight filter W of the previous layer is stored in each element of the matrix constituting the feature image FM.

The first convolution layer CVL1 includes at least one feature image FM. The feature image FM constituting the first convolution layer CVL1 is a convolution operation through a filter W, which is a matrix in which each element is a weighted object image OV of the input layer INL. It is created as a result of performing.

The first pooling layer POL1 includes at least one feature image FM. The feature image FM constituting the first pooling layer POL1 is pooled through a filter W, which is a matrix in which each element is a weight of the feature image FM of the first convolutional layer CVL1. It is generated as a result of executing the subsampling) operation.

The second convolution layer CVL2 includes at least one feature image FM. The feature image FM constituting the second convolution layer CVL2 performs a convolution operation through a filter W, which is a matrix in which each element is a weight, for the feature image of the first pooling layer POL1. It is created as a result of performing it.

The second pooling layer POL2 includes at least one feature image FM. The feature image FM constituting the second pooling layer POL2 is pooled through a filter W, which is a matrix in which each element is a weight of the feature image FM of the second convolutional layer CVL2. It is generated as a result of executing the subsampling) operation.

The first fully connected layer FCL1 includes a plurality of computing nodes f1, f2, ..., fn. Each value of the plurality of computational nodes (f1, f2, ..., fn) of the first fully connected layer (FCL1) is weighted through the activation function for the value of the feature image (FM) of the second pooling layer (POL2). Is calculated by performing an operation to which is applied. For example, the second pooling layer (POL2) has two feature images (FM), and the two feature images (FM) of the second pooling layer (POL2) are in the form of 2 × 8 × 8 (channel × row × column) When, these types of values are arranged in a 1×128 (channel×column) type value and input to each of the plurality of operation nodes (f1, f2, ..., fn) of the first fully connected layer (FCL1). It is entered as a value. Then, each of the plurality of operation nodes f1, f2, ..., fn calculates a node value by performing an operation to which a weight is applied through an activation function. Here, the activation function may exemplify Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), Rectified Linear Unit (ReLU), Leakly ReLU, Maxout, Minout, Softmax, etc. .

The second fully connected layer FCL2 includes a plurality of computing nodes g1, g2, ..., gn. The node values of each of the plurality of computing nodes g1, g2, ..., gn of the second fully connected layer FCL2 are the plurality of computing nodes f1, f2, ... of the first fully connected layer FCL2. , fn) is calculated by performing an operation in which a weight is applied through an activation function.

The output layer OUL includes two output nodes O1 and O2. Node values of the two output nodes O1 and O2 of the output layer OUL, that is, the output values are nodes of each of the plurality of computation nodes g1, g2, ..., gn of the second fully connected layer FCL It is calculated by performing an operation in which a weight is applied to a value through an activation function.

The output values of the two output nodes O1 and O2 of the output layer OUL become the output values of the object identification network 210. Each of the two output nodes (O1, O2) corresponds to a ship and a non-ship, and the output value of each of the two output nodes (O1, O2) is the probability that the object (obj) of the input object image (OV) is a ship, It means the probability that it is not a ship. At this time, if the output value of the first output node (O1) is 0.846 and the output value of the second output node (O2) is 0.154, it means that the probability that the object (obj) is a ship is 85%, and the probability that it is not a ship is 15%. do.

Then, a method of learning the object identification network 210 described above will be described. The learning unit 300 may acquire an image for learning in which the object obj included in the image is a ship. Then, the learning unit 300 may set the expected value according to whether the object obj included in the training image is a ship. For example, when the object (obj) included in the learning image is a ship, the learning unit 300 is an expected value such as "(O1, O2) = [1, 0]" for each of the first output node and the second output node. Can be set. Conversely, the learning unit 300 uses "(O1, O2) for each of the first output node O1 and the second output node O2 for the control training image in which the object obj included in the image is not a ship. ) = [0, 1]", you can set the expected value.

After setting the expected value, the learning unit 300 inputs the training image into the object identification network 210. Then, the object identification network 210 will calculate and output an output value through a plurality of operations in which weights of a plurality of layers are applied to the training image. Here, the output value of the object identification network 210 includes the probability that the object, which is the output value of the first output node O1, is a ship, and the probability that the object, which is the output value of the second output node O2, is not a ship. The learning unit 300 calculates the binary cross entropy loss, which is the difference between the output value of the object identification network 210 and the previously set expected value, and uses an optimization algorithm to minimize the binary cross entropy loss. 210) is optimized.

Next, a configuration of the deep neural network 200 according to the second embodiment of the present invention will be described. 6 to 9 are diagrams for explaining the configuration of a deep neural network according to a second embodiment of the present invention. First, referring to FIG. 6, a deep neural network 200 according to a second embodiment of the present invention includes a direction identification network 220, an adder 230, and a navigation prediction network 240.

In the deep neural network 200 according to the second embodiment, the coordinates of the area box B on the surveillance image SV of the plurality of object images OVt and the plurality of object images OVt from the preprocessor 100 A time vector Tt representing a time when each of the plurality of object images OVt is generated is input from the plurality of region vectors Bt and the surveillance image SV shown. Then, the direction identification network 220 calculates a direction vector Dt indicating the navigation direction of the object obj included in the plurality of object images OVt. In addition, the adder 230 generates a plurality of navigation vectors (Xt) arranged in chronological order by adding a direction vector (Dt), a region vector (Bt), and a time vector (Tt) (Xt = Dt + Bt + Tt). Then, the navigation prediction network 240 performs an operation in which weights are sequentially applied to each of the plurality of navigation vectors (Xt) arranged in chronological order, and after a predetermined time (k-t+0, k>t) The navigation prediction vector (Yk) that predicts the navigation state of the object (obj) is calculated. The navigation prediction vector (Y) is a direction prediction vector (Dk) representing the predicted navigation direction of the object (obj) after a predetermined time, and the area occupied by the object (obj) after a predetermined time in the surveillance image (SV) is an area box (B). It includes a region prediction vector Bt predicted by the coordinates of and a time vector Tt indicating after a predetermined time (k-t+0, k>t).

Then, each of the above-described direction identification network 220 and navigation prediction network 240 will be described in detail. First, the configuration of the direction identification network 220 according to an embodiment of the present invention will be described. Referring to FIG. 7, the direction identification network 220 has a different number of nodes of the object identification network 210 and the output layer (OUL) described above, and the other layers have the same structure. That is, the direction identification network 220 sequentially includes an input layer (INL), a first convolution layer (CVL1), a first pooling layer (POL1), a second convolution layer (CVL2), and a second pooling layer (POL2). , A first fully connected layer (FCL1), a second fully connected layer (FCL2), and an output layer (OUL).

The input layer INL consists of an input matrix having a predetermined size. Elements of the input matrix of the input layer INL correspond to a plurality of pixels of the object image OV. Accordingly, a plurality of pixel values of the object image OV are input as values of each element of the input layer INL input matrix.

The first convolution layer CVL1 includes at least one feature image FM. The feature image FM constituting the first convolution layer CVL1 is a convolution operation through a filter W, which is a matrix in which each element is a weighted object image OV of the input layer INL. It is created as a result of performing.

The first pooling layer POL1 includes at least one feature image FM. The feature image FM constituting the first pooling layer POL1 is pooled through a filter W, which is a matrix in which each element is a weight of the feature image FM of the first convolutional layer CVL1. It is generated as a result of executing the subsampling) operation.

The second convolution layer CVL2 includes at least one feature image FM. The feature image FM constituting the second convolution layer CVL2 performs a convolution operation through a filter W, which is a matrix in which each element is a weight, for the feature image of the first pooling layer POL1. It is created as a result of performing it.

The second pooling layer POL2 includes at least one feature image FM. The feature image FM constituting the second pooling layer POL2 is pooled through a filter W, which is a matrix in which each element is a weight of the feature image FM of the second convolutional layer CVL2. It is generated as a result of executing the subsampling) operation.

The first fully connected layer FCL1 includes a plurality of computing nodes f1, f2, ..., fn. The node value of each of the plurality of computational nodes f1, f2, ..., fn of the first fully connected layer FCL1 is determined through an activation function for the value of the feature image FM of the second pooling layer POL2. It is calculated by performing a weighted operation. For example, the second pooling layer (POL2) has two feature images (FM), and the two feature images (FM) of the second pooling layer (POL2) are in the form of 2 × 8 × 8 (channel × row × column) When, these types of values are arranged in a 1×128 (channel×column) type value and input to each of the plurality of operation nodes (f1, f2, ..., fn) of the first fully connected layer (FCL1). It is entered as a value. Then, each of the plurality of operation nodes f1, f2, ..., fn calculates a node value by performing an operation to which a weight is applied through an activation function. Here, the activation function may exemplify Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), Rectified Linear Unit (ReLU), Leakly ReLU, Maxout, Minout, Softmax, etc. .

The second fully connected layer FCL2 includes a plurality of computing nodes g1, g2, ..., gn. The node values of each of the plurality of computing nodes g1, g2, ..., gn of the second fully connected layer FCL2 are the plurality of computing nodes f1, f2, ... of the first fully connected layer FCL2. , fn) is calculated by performing an operation in which a weight is applied through an activation function.

The output layer OUL includes 9 output nodes a1 to a9. Node values of the 9 output nodes a1 to a9 of the output layer OUL, that is, the output values are nodes of each of the plurality of computation nodes g1, g2, ..., gn of the second fully connected layer FCL It is calculated by performing an operation in which a weight is applied to a value through an activation function.

The output values of the nine output nodes a1 to a9 of the output layer OUL become output values of the direction identification network 220. Of the nine output nodes a1 to a9, each of the first to eighth output nodes a1 to a8 is the navigation direction of the ship as shown in FIG. 3 (0, 45, 90, 135, 180, 225, 270, 315 degrees), and the ninth output node a9 corresponds to a non-ship. The output value of each of the first to eighth output nodes a1 to a8 is a direction (0, where the navigation direction of the object obj of the object image OV) corresponds to each of the first to eighth output nodes a1 to a8. 45, 90, 135, 180, 225, 270, 315 degrees). The output value of the ninth output node a9 corresponds to the probability that the object obj of the object image OV is not a ship. For example, if the output values of the first to ninth output nodes (a1 to a9) are (a1 a2 a3 a4 a5 a6 a7 a8 a9) = [0 0 0 1 0 0 0 0 0], the object of the object image (OV) ( It means that the probability that the navigation direction of obj) is 180 degrees is 100%, and the other probability is 0%.

Then, a learning method for the direction identification network 220 will be described. The learning unit 300 knows whether the object obj included in the image is a ship, and if the object obj is a ship, the navigation direction (0, 45, 90, 135, 180, 225, 270, 315 degrees) ) Can acquire a known learning image. Accordingly, the learning unit 300 may set an expected value for the training image as shown in Table 2 below.

Non-Ship/Ship
Navigation direction (degree) Expected value a1 a2 a3 a4 a5 a6 a7 a8 a9 Non-ship 0 0 0 0 0 0 0 0 One Navigation direction 0(360) 0 0 0 0 0 0 0 One 0 45 0 0 0 0 0 0 One 0 0 90 0 0 0 0 0 One 0 0 0 135 0 0 0 0 One 0 0 0 0 180 0 0 0 One 0 0 0 0 0 225 0 0 One 0 0 0 0 0 0 270 0 One 0 0 0 0 0 0 0 315 One 0 0 0 0 0 0 0 0

After setting the expected value, the learning unit 300 inputs the training image into the object identification network 210. Then, the object identification network 210 will calculate and output an output value through a plurality of operations in which weights of a plurality of layers are applied to the training image. Here, the output value of the object identification network 210 includes output values of each of the first to ninth output nodes a1 to a9. The output values of each of the first to ninth output nodes (a1 to a9) are the probability that the object (obj) included in the object image (OV) is a ship and, in the case of a ship, the navigation direction of the object (obj) of the object image (OV) It represents the probability of a direction (0, 45, 90, 135, 180, 225, 270, 315 degrees) corresponding to each of the first to eighth output nodes a1 to a8. The learning unit 300 calculates a cross entropy loss, which is a difference between the output value of the object identification network 210 and the expected value set as shown in Table 2 above, and uses an optimization algorithm to minimize the cross entropy loss. Optimize the weight of (210).

Next, the navigation prediction network 220 according to the second embodiment of the present invention will be described in more detail. 8 and 9, in the navigation prediction network 240, a direction vector (Dt), a region vector (Bt), and a time vector (Tt) are combined by a vector sum (Xt = Dt + Bt + Tt). A plurality of navigation vectors (Xt) arranged in order can be input. Then, the navigation prediction network 240 performs an operation in which weights are sequentially applied to each of the navigation vectors (Xt: X1 to Xn) of a plurality (eg, n, where n is a positive integer) arranged in chronological order. Then, a navigation prediction vector Yk representing the navigation state of the object obj after a predetermined time (k-n+0, k>n) is calculated. The voyage prediction network 220 may exemplify a recurrent neural network (RNN), long short-term memory models (LTSM), a gated recurrent unit (GRU), and the like.

The navigation prediction network 220 is composed of a plurality of stages (St: S1 to Sk), and includes a recurrent input layer (RIL), a recurrent hidden layer (RHL), and a recurrent output layer (ROL). ).

The circular input layer RIL consists of multiple (n) inputs that detect the current navigation state in order to measure the navigation state of the object obj after a predetermined time (k-n+0, k>n), A plurality of navigation vectors (Xt: X1 to Xn) arranged in chronological order are used as inputs. The cyclic hidden layer (RHL) includes a plurality of hidden cells (HC) corresponding to the number of stages (k), and the plurality of hidden cells (HC) is weighted for a plurality of navigation vectors (Xt: X1 to Xn). One or more of the calculations are performed in a cyclic manner to calculate the navigation prediction vector (Yk). The plurality of hidden cells HC includes first to third hidden cell groups HCG1, HCG2, and HCG3. The circulation output layer ROL outputs the navigation prediction vector Yk calculated by the hidden cell HC of the last stage Sk.

Meanwhile, referring to FIG. 13, the hidden cell HC consists of one or more operations to which weights (Weight, Wx, Wh, Wy) are applied. Here, the operation means an operation to which an activation function is applied. The activation function can be exemplified by Sigmoid, Hyperbolic tangent (tanh), Exponential Linear Unit (ELU), Rectified Linear Unit (ReLU), Leakly ReLU, Maxout, Minout, Softmax, and the like. In addition, in one hidden cell (HC), the weight is the input weight Wx applied to the input voyage vector Xt, the state weight Wh applied to the state value Ht-1 of the previous stage, and the output applied to the output value Yt. Include the weight Wy. For example, an operation to which a weight applied to the hidden cell HC is applied may be illustrated in Equation 1 below.

Here, b is a threshold or bias. In particular, the tanh function and the ReLU function can be changed to other activation functions. Referring to Equation 1 and FIG. 9, each of the plurality of hidden cells HCt is a state value Ht- of the previous stage St-1 calculated by the hidden cell HCT-1 of the previous stage St-1. 1) and the current stage (St) status value (Ht) and output value (Yt) by performing an operation in which the weight (W: Wh, Wx, Wy) is applied to the input value (Xt) of the stage (St) Can be calculated.

Each of the first to fourth navigation vectors (X1, X2, X3, X4) in which the cyclic input layer (RIL) of the navigation prediction network 220 is arranged in chronological order is a first hidden cell group of the cyclic hidden layer (RHL) ( It inputs to the hidden cells corresponding to HCG1), that is, the first to fourth hidden cells HC1, HC2, HC3, and HC4. Then, each of the first to fourth hidden cells HC1, HC2, HC3, and HC4 of the first hidden cell group HCG1 of the circulating hidden layer RHL is calculated by the hidden cell HC of the previous stage St-1. The state value (Ht) of the current stage (St) is calculated by performing an operation in which a weight is applied to the state value (Ht-1) of the previous stage (St-1) and the input value (Xt) of the own stage (St). After calculation, the calculated state value Ht is transferred to the hidden cell HCT+1 of the next stage St+1. For example, the first hidden cell HC1 applies the state weight Wh to the initial state value H0 and applies the input weight Wx to the first navigation vector X1 to determine the state of the first stage S1. Calculate the value (H1). In the case of the first hidden cell HC1, since there is no previous stage, the initial state value H0 is used. Subsequently, the second hidden cell HC2 applies the state weight Wh to the state value H1 of the first stage S1, which is the previous stage, and applies the input weight Wx to the second navigation vector X2. Thus, the state value H2 of the second stage S2 is calculated. In addition, the third hidden cell HC3 applies the state weight Wh to the state value H2 of the second stage S2, which is the previous stage, and applies the input weight Wx to the third navigation vector X3. The state value H3 of the third stage S3 is calculated. Subsequently, the fourth hidden cell HC4 applies the state weight Wh to the state value H3 of the third stage S3, which is the previous stage, and applies the input weight Wx to the fourth navigation vector X4. Thus, the state value H4 of the fourth stage S4 is calculated. In this way, the plurality of hidden cells HCt belonging to the first hidden cell group HCG1 are the state values Ht-1 of the previous stage and the state and input weights ( The operation to which Wh and Wx is applied is performed to calculate the state value Ht of the current stage, and the calculated state value Ht is transferred to the next stage St+1.

Then, the plurality of hidden cells HC5 to HCk-1 belonging to the second hidden cell group HCG2 of the cyclic hidden layer RHL are the previous stage St- which is calculated by the hidden cell HC of the previous stage St-1. The state value (Ht) of the current stage (St) is calculated by performing an operation in which the state weight (Wh) is applied to the state value (Ht-1) of 1), and then the calculated state value (Ht) is applied to the next stage. It is delivered to the hidden cell (HCt+1) of (St+1). For example, the fifth hidden cell HC5 calculates the state value H5 of the fifth stage S5 by applying the state weight Wh to the state value H4 of the fourth stage S4 that is the previous stage. Subsequently, the sixth hidden cell HC6 calculates the state value H6 of the sixth stage S6 by applying the state weight Wh to the state value H5 of the fifth stage S5 which is the previous stage. The remaining hidden cells HC6 to HCk-1 belonging to the second hidden cell group HCG2 also calculate the state value Ht in the same manner as described above, and provide the calculated state value Ht to the next stage. In addition, the k-1 hidden cell HCk-1, which is the last hidden cell of the second hidden cell group HCG2, is the state weight Hk-2 of the k-2th stage Sk-2 that is the previous stage. Wh) is applied to calculate the state value Hk-1 of the k-1th stage Sk-1 which is the current stage.

Next, the k-th hidden cell HCk of the third hidden cell group HCG3 applies a state weight Wh to the state value Hk-1 of the k-1th stage Sk-1, which is a previous stage. The state value Hk of the current stage k-th stage Sk is calculated. Then, the k-th hidden cell HCk performs an operation that applies the output weight Wy to the state value Hk of the k-th stage Sk, which is the current stage, to obtain an output value, that is, a navigation prediction vector Yk. Calculate.

In this way, in the navigation prediction network 220 according to an embodiment of the present invention, the hidden cells of the first hidden cell group HCG1 are based on the state value Ht-1 of the previous stage and the input value Xt of the current stage. The state value Ht of the current stage is calculated by performing an operation to which the state and input weights Wh and Wx are applied, and then the calculated state value Ht is transferred to the next stage. Subsequently, the hidden cell of the second hidden cell group (HCG2) calculates the state value (Ht) of the current stage by performing an operation to which the state weight (Wh) is applied to the state value (Ht-1) of the previous stage. , The calculated state value Ht is transferred to the next stage. And the last hidden cell, the hidden cell of the third hidden cell group, calculates the state value (Ht) of the current stage by performing an operation in which the state weight (Wh) is applied to the state value (Ht-1) of the previous stage. , An output value, that is, a navigation prediction vector Yk, is calculated by performing an operation that applies the output weight Wy to the calculated state value Ht of the current stage. The voyage prediction vector Yk is a value that is finally calculated by taking an input value into account and a state value taking into account the previous state to the last hidden cell. Accordingly, the navigation prediction network 220 predicts the navigation state of the object obj after a predetermined time (kn, k>n) from a plurality of navigation vectors (Xt) representing the navigation conditions that can be measured up to the present (n). The navigation prediction vector (Yk) can be calculated.

Next, a description will be given of a method for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to the first embodiment of the present invention. 10 is a flowchart illustrating a method for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to the first embodiment of the present invention.

Referring to FIG. 10, the preprocessing unit 100 of the control unit 16 continuously photographs the surroundings of the vessel through the camera unit 12 as shown in FIG. 2 in step S110 to monitor a predetermined area (SW×SH). Create an image (SV). The surveillance image SV consists of a plurality of frames Ft arranged in chronological order. This surveillance image SV may be displayed through the display unit 14.

Then, the preprocessor 100 represents an area occupied by the object obj included in the surveillance image SV for each frame Ft in the captured surveillance image SV, as shown in FIG. 2 in step S120. The object image OVt is detected by specifying the area box B through feature point detection. The area box B represents the area occupied by the object obj in the surveillance image SV as a rectangle, and means a shooting type having the minimum size in which all the objects obj are included. This area box B has a central coordinate (x, y), a width (w) and a height (h). Meanwhile, the object image OVt detected by the preprocessor 100 is input to the object identification network 210 of the deep neural network 200.

Then, the object identification network 210 calculates, as a probability, whether the object obj of the object image OVt is a ship by performing a plurality of operations to which the weight is applied in step S130. That is, the output value of each of the two output nodes O1 and O2 of the object identification network 210 indicates a probability that the object obj of the input object image OV is a ship and a probability that the object is not a ship.

Accordingly, the control unit 400 determines whether the object obj is a ship according to the probability calculated by the object identification network 210 in step S140. For example, if the output value of the first output node (O1) is 0.846 and the output value of the second output node (O2) is 0.154, it means that the probability that the object (obj) is a ship is 85%, and the probability that it is not a ship is 15%. do. In this case, the control unit 400 may determine that the object obj is a ship. On the other hand, if the output value of the first output node (O1) is 0.444 and the output value of the second output node (O2) is 0.556, it means that the probability that the object (obj) is a ship is 44%, and the probability that it is not a ship is 56%. do. In this case, the control unit 400 may determine that the object obj is not a ship.

As a result of the determination in step S140, if the object obj is a ship, the control unit 400 calculates a risk indicating the probability of a collision with the object obj determined as the ship in step S150. To this end, the control unit 400 calculates the width (w×h) of the object image through the coordinates of the area box B of the object obj in the surveillance image SV, and then calculates the width of the surveillance image SV. The risk can be calculated through the ratio of (SW×SH) to the area (w×h) of the object image (OV). That is, the control unit 400 calculates that the higher the risk is as the ratio of the area occupied by the object image OV in the surveillance image SV increases.

The control unit 400 determines whether the risk calculated in step S160 is equal to or greater than a preset threshold. As a result of the determination, if the risk is greater than or equal to the threshold, it is determined that there is a risk of collision. Accordingly, the control unit 400 transmits an alarm message notifying the risk of collision to a user device, an alarm, and a control server through the communication unit 11 in step S170. These alert messages include the image and location information of the object.

Meanwhile, according to an embodiment of the present invention, in the case of step S120 described above, the object image is detected in a different manner according to the distance to the object obj. This step S120 will be described in more detail. 11 is a flowchart illustrating a method of detecting an object image according to an embodiment of the present invention. 12 is a diagram illustrating a method of detecting an object image corresponding to a near object according to an embodiment of the present invention. 13 is a diagram for describing a method of detecting an object image corresponding to a distant object according to an embodiment of the present invention.

Referring to FIG. 11, when a surveillance image (SV) is input in step S210, the preprocessor 100 attempts to extract feature points through a predetermined algorithm in step S220. Algorithms for feature point extraction are Harris Corner, Shi & Tomasi, SIFT (Scale Invariant Feature Transform), SURF (Speeded up robust features), BRIEF (Binary robust independent elementary features), ORB (Oriented FAST and Rotated BRIEF), FAST (Features from Accelerated Segment Test), AGAST, and the like, and preferably, FAST may be used.

After attempting to extract the feature points as described above, the preprocessor 100 determines whether or not the feature points are extracted in step S230. According to this determination, when the feature point is extracted, the preprocessor 100 proceeds to step S240, and if the feature point is not extracted, the preprocessor 100 proceeds to step S250. When the feature point is extracted, the preprocessor 100 determines that the object in the surveillance image SV is a near object located closer than a predetermined reference, and performs an object image detection procedure corresponding to the near object in step S240. On the other hand, if the feature point is not extracted, the preprocessor 100 determines that the object in the surveillance image SV is a distant object located farther than a predetermined reference, and performs an object image detection procedure corresponding to the distant object in step S250.

Then, these steps S240 and S250 will be described in more detail. First, an object image detection method corresponding to a near object will be described with reference to FIG. 12. This FIG. 12 corresponds to step S240 and is for explaining step S240 in more detail. Referring to FIG. 12, when a surveillance image (SV) as shown in FIG. 12 is input in step S241, the preprocessor 100 performs a blurring process on the surveillance image (SV) input in step S242 to obtain an overall image. Blur Then, the preprocessor 100 changes the image consisting of 3 channels of RGB into one channel by applying grayscale to the surveillance image SV in step S243. Subsequently, the preprocessor 100 detects a plurality of corner points in the surveillance image SV by using a Harris corner algorithm in step S244. Subsequently, the preprocessor 100 searches for a dense area of a plurality of corner points detected in step S244 and detects an object image OV through a rectangular area box B indicating an area occupied by the object.

Next, an object image detection method corresponding to a near object will be described with reference to FIG. 13. This FIG. 13 corresponds to step S250 and is for explaining step S250 in more detail. Referring to FIG. 13, when a surveillance image (SV) as shown in FIG. 13 is input in step S251, the preprocessor 100 performs a blurring process on the surveillance image (SV) input in step S252 to provide an overall image. Blur Then, the preprocessor 100 changes the image consisting of 3 channels of RGB to one channel by applying grayscale to the surveillance image SV in step S253.

Subsequently, in step S254, the preprocessor 100 finds a line of a high-frequency component through Canny edge detection and finds a binarized line. Then, the preprocessor 100 detects a horizontal line through Hough Transform in step S255. Using the Hough transform, straight lines from long to short lengths are detected, and the longest straight line on the sea is a horizontal line, and the distance is obtained through Pythagorean theorem using the coordinates of both ends of each straight line, and the longest straight line is found. Look for

After detecting the horizontal line, the pre-processing unit 100 removes the bottom except the above horizontal line from the surveillance image SV initially input in step S256 (S251), leaving only the image above the horizontal line. Subsequently, the preprocessor 100 enlarges the image above the horizontal line to a predetermined size in step S257. Then, the preprocessor 100 detects a plurality of corner points using a Harris corner algorithm in the image above the horizontal line enlarged in step S258. Subsequently, the preprocessor 100 searches for a dense area of a plurality of corner points detected in step S259 and detects an object image OV through a rectangular area box B indicating an area occupied by the object.

Next, a description will be made of a method for recognizing an approaching vessel in consideration of a sea object distance based on a deep neural network according to a second embodiment of the present invention. 14 is a flowchart illustrating a method for recognizing an approaching vessel in consideration of a maritime object distance based on a deep neural network according to a second embodiment of the present invention.

Referring to FIG. 14, the preprocessing unit 100 of the control unit 16 continuously photographs the surrounding of the ship through the camera unit 12 as shown in FIG. 2 in step S310 to monitor a predetermined area (SW×SH). Create an image (SV). The surveillance image SV consists of a plurality of frames Ft arranged in chronological order. This surveillance image SV may be displayed through the display unit 14.

Then, as shown in FIG. 2 in step S320, the preprocessor 100 determines the area occupied by the object obj included in the surveillance image SV for each frame Ft in the captured surveillance image SV. By detecting through the box (B), a plurality of object images (OVt) are generated. The area box B represents the area occupied by the object obj in the surveillance image SV as a rectangle, and means a shooting type having the minimum size in which all the objects obj are included. This area box B has a central coordinate (x, y), a width (w) and a height (h). The method of detecting the object image OVt in step S320 is the same as in the first embodiment. That is, in the second embodiment, as described with reference to FIGS. 11 to 13 as in the first embodiment, when the feature point is extracted by attempting to extract the feature point, the preprocessor 100 makes the object in the surveillance image SV closer than the reference. The object image detection procedure corresponding to the near object is determined to be a near object located in (S240), and if the feature point is not extracted, it is determined that the object in the surveillance image (SV) is a far object located farther than the reference, and An object image detection procedure corresponding to an object is performed (S250).

The preprocessor 100 includes a plurality of area vectors Bt representing coordinates of the area box B on the surveillance image SV of the plurality of object images OVt and the plurality of object images OVt in step S330, and A time vector Tt representing a time when each of the plurality of object images OVt is generated from the surveillance image SV is input into the deep neural network 200 according to the second embodiment. Then, the direction identification network 220 of the deep neural network 200 sequentially receives each of the plurality of object images OVt, and the probability that the object obj included in the object image OVt is a ship, and if it is a ship , The probability that the navigation direction of the object obj of the object image OVt corresponds to each of a plurality of directions (0, 45, 90, 135, 180, 225, 270, 315 degrees) is calculated. Here, it is assumed that the object obj included in all of the plurality of object images OVt is a ship. Accordingly, the direction identification network 220 determines the navigation direction of the object obj of the object image OVt in step S340 in each of a plurality of directions (0, 45, 90, 135, 180, 225, 270, 315 degrees). The corresponding probability is calculated, and this probability becomes a direction vector (Dt). The direction identification network 220 outputs only the output values of the first to eighth output nodes a1 to a8 among the first to ninth output nodes a1 to a9 to obtain a direction vector Dt. Can be generated. For example, when the first to fourth object images OV1, OV2, OV3, OV4 of FIG. 4 are input to the direction identification network 220, the output values first to fourth direction vectors (D1, D2, D3, D4) Is D1=[0.01, 0.77, 0.02, 0.11, 0.09, 0.06 0.02 0.03], D2=[0.01, 0.81, 0.02, 0.11, 0.09, 0.06 0.02 0.03], D3=[0.01, 0.01, 0.79, 0.11, 0.03, 0.04 0.12 0.03], D4 = [0.01, 0.02, 0.11, 0.83, 0.09, 0.06 0.02 0.03]. That is, it can be seen that the direction of the ship changes 90 degrees, 90 degrees, 135 degrees, and 180 degrees.

Next, the adder 230 of the deep neural network 200 adds a plurality of region vectors (Bt) and a plurality of time vectors (Tt) corresponding to each of the plurality of direction vectors (Dt) in step S350, and is arranged in chronological order. Generate a plurality of navigation vectors (Xt). For example, the first to fourth navigation vectors (X1, X2, X3, X4) corresponding to the first to fourth object images OV1, OV2, OV3, and OV4 of FIG. 4 are X1=D1+B1+T1, X2. =D2+B2+T2, X3=D3+B3+T3, X4=D4+B4+T4.

Next, the navigation prediction network 220 of the deep neural network 200 calculates a navigation prediction vector by performing a plurality of calculations in which weights are applied to the plurality of navigation vectors in step S360. At this time, in the navigation prediction network 220, the hidden cells of the first hidden cell group (HCG1) are the state values (Ht-1) of the previous stage and the state and input weights (Xt), which are the input values of the current stage. After calculating the state value Ht of the current stage by performing an operation to which Wh, Wx) is applied, the calculated state value Ht is transferred to the next stage. Subsequently, the hidden cell of the second hidden cell group (HCG2) calculates the state value (Ht) of the current stage by performing an operation to which the state weight (Wh) is applied to the state value (Ht-1) of the previous stage. , The calculated state value Ht is transferred to the next stage. And the last hidden cell, the hidden cell of the third hidden cell group, calculates the state value (Ht) of the current stage by performing an operation in which the state weight (Wh) is applied to the state value (Ht-1) of the previous stage. , An output value, that is, a navigation prediction vector Yk, is calculated by performing an operation that applies the output weight Wy to the calculated state value Ht of the current stage. In this way, the navigation prediction network 220 navigates the object obj after a predetermined time (kn, k> n) based on a plurality of navigation vectors (Xt) representing the trend of the navigation state of the object (obj). The navigation prediction vector (Yk) that predicts the state can be calculated. The navigation prediction vector Yk may include a direction vector Dk, a region vector Bk, and a time vector Tk after a predetermined time (kn, k>n) (Yk=Dk+Bk+Tk). .

Accordingly, in step S370, the control unit 400 sets the area box B derived from the area vector Bk at the time indicated by the time vector Tk of the navigation prediction vector Yk in step S370. At least a part of the warning area overlaps, and the degree of risk is analyzed through whether the direction vector Dk is oriented in a possible collision direction. In addition, the control unit 400 determines whether the risk is greater than or equal to a threshold value in step S380.

At this time, the control unit 400 determines the area box B derived from the area vector Bk at the time indicated by the time vector Tk of the navigation prediction vector Yk at least with a preset warning area in the surveillance image SV. If some overlap and the direction vector Dk is directed toward a possible collision direction, it can be determined that the risk is equal to or greater than the threshold. For example, the surveillance image (SV) of FIG. 4 is divided into 12 cells of 3 rows and 4 columns, and cells of 2 rows 3 columns and 3 rows 3 columns [(2, 3), (3, 3)] are warning areas, It is assumed that the direction facing the area is a possible collision direction. As shown in Fig. 4, according to the derived navigation prediction vector Yk, the area box B of the area vector Bk at the time k indicated by the time vector Tk is 3 rows 3 of the surveillance image SV. It partially overlaps with the cells in the columns 3 and 3, and the direction vector Dk is directed toward the 3rd row and 3rd column (3, 3) of the surveillance image SV. In this case, the control unit 400 determines that the risk indicating the risk of collision is equal to or greater than a preset threshold. Then, the control unit 400 transmits an alarm message notifying the risk of collision to the user device, the alarm, and the control server through the communication unit 11 in step S390. This warning message includes the current, that is, the image of the navigation vector X4 = OV4 + B4 + T4 and location information.

As described above, according to an embodiment of the present invention, by predicting the navigation state of an object after a predetermined time using a deep neural network from a plurality of navigation vectors, it is possible to predict in advance whether a collision is possible, and if there is a possibility of collision, it can be warned .

Meanwhile, the method according to the embodiment of the present invention described above may be implemented in the form of a program that can be read through various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include a program command, a data file, a data structure, or the like alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic-optical media such as floptical disks ( magneto-optical media) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of the program instruction may include not only machine language wires such as those made by a compiler, but also high-level language wires that can be executed by a computer using an interpreter or the like. These hardware devices may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

Although the present invention has been described using several preferred embodiments, these embodiments are exemplary and not limiting. As such, those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made according to the equivalence theory without departing from the spirit of the present invention and the scope of the rights presented in the appended claims.

10: control device 11: communication unit
12: camera part 13: input part
14: display unit 15: storage unit
16: control unit 20: service server
30: shopping mall server 100: preprocessor
200: deep neural network 210: object identification network
220: direction identification network 230: adder
240: navigation network 300: learning department
400: control unit

Claims (16)

In the device for recognizing an approaching vessel considering the distance of a sea object based on a deep neural network,
A camera unit for photographing a surveillance image;
A preprocessor for detecting an object image by specifying an area box representing an area occupied by an object included in the captured surveillance image through feature point detection;
A deep neural network for outputting with probability whether the object of the object image is a ship; And
Determines whether the object is a ship according to the probability, and as a result of the determination, if the object is a ship, a risk indicating a probability of occurrence of a collision with the ship is calculated, and if the calculated risk is above a threshold, a collision risk is warned. Control department;
Including,
The pretreatment unit
If no feature point is detected in the image,
A horizontal line is detected in the surveillance image, the image above the detected horizontal line is enlarged to a predetermined size, and a plurality of corner points are detected using the Harris Corner algorithm in the image above the enlarged horizontal line. Characterized in that, by finding a dense area of a plurality of corner points, specifying an area box representing an area occupied by an object, and detecting the object image through the specified area box.
Device for recognizing approaching vessels. The method of claim 1,
The pretreatment unit
When a feature point is detected in the surveillance image,
After detecting a plurality of corner points in the surveillance image using a Harris corner algorithm, an area box representing an area occupied by an object is specified by searching for a dense area of the detected plurality of corner points, and the specified area box The object image is detected through
Device for recognizing approaching vessels. delete The method of claim 1,
The control unit
After calculating the area of the object image through the coordinates of the area box of the object,
If the ratio of the area of the surveillance image to the area of the object image is greater than or equal to a preset threshold, it is determined that the risk is greater than or equal to the threshold.
Device for recognizing approaching vessels. The method of claim 1,
The deep neural network
An input layer into which an object image is input;
At least one convolutional layer including at least one feature image derived by a convolution operation on the object image or feature image;
At least one pooling layer for deriving at least one feature image derived through a pooling operation on the feature image generated through a convolution operation;
At least one fully connected layer including a plurality of operation nodes for receiving a feature image or node value of a previous layer and calculating a node value through an operation by an activation function; And
An output layer including a plurality of output nodes for receiving a node value of the fully connected layer and calculating an output value through an operation by an activation function;
Characterized in that it comprises a
Device for recognizing approaching vessels. In the device for recognizing an approaching vessel considering the distance of a sea object based on a deep neural network,
A camera unit for photographing a surveillance image including a plurality of frames;
A preprocessor configured to sequentially detect a plurality of object images through an area box indicating an area occupied by an object in each of the plurality of frames;
Generates a plurality of navigation vectors arranged in chronological order corresponding to each of the plurality of object images, and predicts the navigation state of the object after a predetermined time by performing a plurality of calculations to which weights are applied to the plurality of navigation vectors A deep neural network that calculates a navigational prediction vector; And
From the navigation prediction vector, an area box indicating the navigation direction of the object after the predetermined time and an area occupied by the object in the surveillance image is derived, the derived navigation direction is a collision possible direction, and the derived area box is the surveillance image A control unit that warns of a risk of collision when at least a part of the warning area is overlapped with each other;
Including,
The deep neural network
A direction identification network for deriving a plurality of direction vectors indicating navigation directions of each of the plurality of object images;
The plurality of direction vectors are combined with an area vector representing the coordinates of the area box on the surveillance image of each of the plurality of object images and a time vector representing the time when each of the plurality of object images is generated. An adder that generates a navigation vector; And
It consists of a plurality of stages arranged in sequence,
Multiple concealment that calculates the state value of the current stage by calculating the state value of the current stage by applying the state and input weights to the state value of the previous stage and the voyage vector, which is the input value of the current stage, and then passes the calculated state value to the next stage. A first hidden cell group including a cell,
A second hidden cell group including a plurality of hidden cells for calculating a state value of the current stage by performing an operation in which a state weight is applied to the state value of the previous stage, and transmitting the calculated state value to the next stage;
Calculates the state value of the current stage by performing an operation in which the state weight is applied to the state value of the previous stage, and then performs an operation that applies the output weight to the calculated state value of the current stage to calculate the output value, the voyage prediction vector. Containing a third hidden cell group including a hidden cell
Navigation forecasting network;
Characterized in that it comprises a
A device for recognizing an approaching vessel. delete delete The method of claim 6,
The navigation forecasting network is
A circular input layer receiving a plurality of navigation vectors arranged according to a time order;
A cyclic concealment layer including a plurality of hidden cells for calculating the navigation prediction vector by performing one or more operations in which weights are applied in the order of the plurality of stages corresponding to each of the plurality of navigation vectors; And
A circulation output layer outputting the calculated risk level;
Characterized in that it comprises a
Device for recognizing approaching vessels. The method of claim 6,
The pretreatment unit
When a feature point is detected in the surveillance image,
After detecting a plurality of corner points in the surveillance image using a Harris corner algorithm, an area box representing an area occupied by an object is specified by searching for a dense area of the detected plurality of corner points, and the specified area box To detect the object image through,
If no feature point is detected in the image,
A horizontal line is detected in the surveillance image, the image above the detected horizontal line is enlarged to a predetermined size, and a plurality of corner points are detected using the Harris Corner algorithm in the image above the enlarged horizontal line. Characterized in that, by finding a dense area of a plurality of corner points, specifying an area box representing an area occupied by an object, and detecting the object image through the specified area box.
Device for recognizing approaching vessels. In a method for recognizing an approaching vessel considering the distance of a sea object based on a deep neural network,
Capturing a surveillance image by a camera unit;
Detecting an object image by specifying an area box indicating an area occupied by an object included in the captured surveillance image through feature point detection by a preprocessor;
Outputting, by a deep neural network, whether the object of the object image is a ship with probability; And
Determining whether the object is a ship according to the probability;
If the object is a ship as a result of the determination, the control unit calculates a risk indicating a probability of a collision with the ship; And
Warning of a collision risk when the calculated risk is greater than or equal to a threshold;
Including,
The step of detecting the object image
If the preprocessor does not detect a feature point in the image, detecting a horizontal line in the surveillance image;
Enlarging, by the pre-processing unit, an image above the detected horizontal line to a predetermined size;
Detecting, by the preprocessor, a plurality of corner points in the image above the enlarged horizontal line using a Harris corner algorithm;
Specifying, by the preprocessor, an area box indicating an area occupied by an object by searching for a dense area of the detected plurality of corner points; And
Detecting the object image through the specified area box by the preprocessor;
Characterized in that it comprises a
Method for recognizing an approaching vessel. The method of claim 11,
The step of detecting the object image
When the preprocessor detects a feature point in the surveillance image, detecting a plurality of corner points in the surveillance image using a Harris corner algorithm;
Specifying, by the preprocessor, an area box indicating an area occupied by an object by searching for a dense area of the detected plurality of corner points;
Detecting the object image through the specified area box by the preprocessor;
Characterized in that it comprises a
Method for recognizing an approaching vessel. delete The method of claim 11,
The step of calculating the risk is
Calculating, by the control unit, the width of the object image through coordinates of the area box of the object; And
Determining that the risk is greater than or equal to a threshold value when the ratio of the area of the surveillance image to the area of the object image is greater than or equal to a preset threshold;
Characterized in that it comprises a
Method for recognizing an approaching vessel. The method of claim 11,
The deep neural network outputting with probability whether the object of the object image is a ship
Receiving, by an input layer of the deep neural network, an object image;
Deriving at least one feature image by performing a convolution operation using a filter on the object image by the first convolutional layer of the deep neural network;
Deriving at least one feature image by performing, by the first pooling layer of the deep neural network, a pooling operation using a filter on the feature image of the first convolutional layer;
Deriving at least one feature image by performing a convolution operation using a filter on the feature image of the first pooling layer by the second convolutional layer of the deep neural network;
Deriving at least one feature image by performing a pooling operation using a filter on the feature image of the second convolutional layer by the second pooling layer of the deep neural network;
Calculating a node value through an operation by an activation function on the feature image of the second pooling layer by a plurality of computing nodes of the first complete connection layer of the deep neural network;
Calculating a node value through an operation by an activation function on the node value of the first complete connection layer by a plurality of operation nodes of the second complete connection layer of the deep neural network; And
Calculating an output value, which is a probability of whether the object of the object image is a ship, by calculating the node value of the second fully connected layer by an activation function by a plurality of output nodes of the output layer of the deep neural network;
Characterized in that it comprises a
Method for recognizing an approaching vessel. A computer-readable recording medium in which a program for performing a method for recognizing an approaching vessel according to any one of claims 11, 12, 14 and 15 is recorded.

Images