KR20230097958A - Apparatus for monitoring illegal fishing based on artificial neural network and method therefor - Google Patents

Abstract

A device for monitoring illegal fishing continuously receives identification information, location information, and time information of the identification device from an identification device mounted on a ship, and when a plurality of monitoring images are received from a plurality of imaging devices, a plurality of received After detecting the surveillance image corresponding to the time information and the location information among the surveillance images, a data processing unit that normalizes the detected surveillance images to generate surveillance image vectors, and an identification model, which is an artificial neural network having learned weights, is used. An identification unit for deriving one or more region boxes indicating whether the ship is in an operating state or a non-operating state while specifying an area occupied by the ship from the surveillance image vector, and deriving an output image vector including the derived region box; Among one or more area boxes included in the image vector (OV), an area box corresponding to the location information of the identification device is detected, and whether the ship is operating or not operating is confirmed through the detected area box. Includes an information processing unit that determines whether or not illegal fishing of

Description

Apparatus for monitoring illegal fishing based on artificial neural network and method therefor}

The present invention relates to a technology for monitoring illegal fishing, and more particularly, to a device and method for monitoring illegal fishing based on an artificial neural network.

Since fishery resources are traditionally recognized as non-maintenance items, they have been developed indiscriminately by the logic of preoccupation, and the fact is that the government's fisheries management tasks focused on the policy of increasing aquatic products production have focused only on fishery coordination and maintenance of fishery order rather than conservation and management of resources. am. As a result, it is not an exaggeration to say that Korea's coastal fishing industry is facing the risk of collapse of its industrial base due to resource reduction, fishing ground reduction, pollution of the fishing ground environment, excessive fishing boat power, and rapid increase in fishing expenses. The reasons for the extremely deteriorating business environment of the fishery industry are the reduction of fishing grounds due to changes in the international fishery order, the original defect of the fishery permit system that did not flexibly respond to changes in the environment of the fishery industry, and the reckless competition and excessive investment by fishermen. However, it can be said that it is largely due to illegal fishing that ignores even the minimum resource conservation measures.

In order to eradicate illegal fishing, the government not only cracks down with related organizations, but also promotes fishermen to practice law-abiding fishing. It is seldom eradicated because the aspects of social structure and social structure are working in a complex way.

In terms of the types of illegal fishing, fishing conducted in a method not permitted by law, fishing conducted without obtaining a fishing permit or license, fishing conducted by a fisherman who has obtained a fishing permit or license using a fishing boat, fishing gear, or fishing method that is different from the contents of the permit or license etc. can be distinguished. Such illegal fishing not only disturbs the fishery order, but also adversely affects the protection of resources and hinders the implementation of the government's fisheries policy.

Korean Patent Publication No. 2004-0075488 (published on August 30, 2004)

Therefore, in view of the above points, an object of the present invention is to provide a device and method for monitoring illegal fishing based on an artificial neural network.

A device for monitoring illegal fishing according to a preferred embodiment of the present invention for achieving the above object continuously receives identification information, location information and time information of the identification device from an identification device mounted on a ship, When a plurality of monitoring images are received from a plurality of imaging devices, monitoring images corresponding to the time information and location information are detected from among the plurality of monitoring images received, and then the detected monitoring images are normalized to generate a monitoring image vector. By using the data processing unit and the identification model, which is an artificial neural network having learned weights, the area occupied by the vessel is specified from the surveillance image vector, and one or more area boxes indicating whether the vessel is in operation or non-operation are derived and derived. an identification unit for deriving an output image vector including an area box of the detected region; and detecting a region box corresponding to the location information of the identification device among one or more region boxes included in the output image vector (OV), and detecting the detected region box. It includes an information processing unit that determines whether the vessel is illegally fishing by checking whether the vessel is in an operating state or a non-fishing state through

The identification model divides the surveillance image vector into a plurality of cells and then derives one or more region boxes. Confidence indicating the rate at which the area occupied by corresponds to the area where the ship exists, and the probability that the object in the area box belongs to the first class representing the operating state of the ship and the probability that it belongs to the second class representing the non-operating state of the ship Includes a classification value that includes

The information processing unit is characterized in that the information processing unit calculates ship position information in the one or more region boxes, and detects the region box having the ship position information closest to the position information of the identification device among the one or more region boxes.

The information processing unit calculates the direction angle of the area box with respect to the ship and the distance of the area box from the ship based on the imaging device, and calculates the direction angle and the distance based on the location information of the imaging device. It is characterized in that the location information of the ship in the area box is derived.

,

에 따라 상기 방향각을 산출하고, 수학식

에 따라 상기 거리를 산출한다. The information processing unit Equation

,

Calculate the orientation angle according to the equation

Calculate the distance according to

는 수평 방향의 방향각이고, 상기

는 수직 방향의 방향각이고, 상기 d는 거리이고, 상기 X는 출력영상벡터의 중심좌표 중 수평 방향의 좌표이고, 상기 Y는 출력영상벡터의 중심좌표 중 수직 방향의 좌표이고, 상기 x는 영역상자의 중심좌표 중 수평 방향의 좌표이고, 상기 y는 영역상자의 중심좌표 중 수직 방향의 좌표이고, 상기

는 영상장치의 카메라모듈의 초점 거리인 것을 특징으로 한다. here, above

is the orientation angle in the horizontal direction, and

is a direction angle in the vertical direction, d is a distance, X is a coordinate in the horizontal direction among the coordinates of the center of an output image vector, Y is a coordinate in the vertical direction among the coordinates of the center of an output image vector, and x is an area is the coordinate in the horizontal direction among the coordinates of the center of the box, y is the coordinate in the vertical direction among the coordinates of the center of the area box,

Is the focal length of the camera module of the imaging device.

If the probability of belonging to the first class is greater than a predetermined threshold and the probability of belonging to the second class is higher, the information processing unit determines that the ship in the area box is in an operating state, and the probability of belonging to the second class exceeds a predetermined threshold. If the probability of belonging to the first class is higher than the above, it is characterized in that it is determined that the vessel of the area box is in a non-operational state.

The data processing unit extracts operation information including a permitted area indicating an area in which operation is permitted for the vessel indicated by the identification information and a permitted time indicating a permitted time for operation in the permitted area, and When the location information and the time information do not conform to the permitted area and permitted time of the operation information, a monitoring image corresponding to the time information and the location information is detected.

에 따라 상기 감시영상의 명도를 조절하고, 수학식

에 따라 상기 감시영상벡터를 생성한다. The data processing unit Equation

Adjust the brightness of the surveillance image according to the equation

The surveillance image vector is generated according to the above.

는 명도가 조절된 감시영상의 픽셀값이고, 상기

는 감시영상의 픽셀값이고, 상기 m은 감시영상의 평균 픽셀값이고, 상기 q는 감시영상의 픽셀값의 표준 편차이고, 상기

는 명암대비 지수이고, 상기

는 명도가 조절된 감시영상의 평균 픽셀값이고, 상기

는 감시영상벡터의 픽셀값인 것을 특징으로 한다. Here, the u and the v are pixel coordinates, and the

Is a pixel value of the surveillance image whose brightness is adjusted,

is the pixel value of the surveillance image, m is the average pixel value of the surveillance image, q is the standard deviation of the pixel values of the surveillance image,

is the contrast index, and

Is the average pixel value of the surveillance image whose brightness is adjusted,

is a pixel value of a surveillance image vector.

The apparatus prepares a surveillance video vector for learning and a measured video vector corresponding to the surveillance video vector for learning, inputs the surveillance video vector for learning to an identification model whose learning has not been completed, and then the identification model determines the surveillance video vector for learning. When an output image vector including an output image vector for learning is generated by performing a plurality of operations applying weights for which learning between a plurality of layers is not completed, the difference between the output image vector generated through the loss function and the measured image vector is represented. A learning unit that calculates a total loss and performs optimization of correcting weights of the loss model so that the total loss is minimized is further included.

In order to achieve the above object, in a method for monitoring illegal fishing according to a preferred embodiment of the present invention, the data processing unit continuously transmits identification information, location information, and time information of the identification device from an identification device mounted on a ship. receiving, when the data processing unit receives a plurality of surveillance images from a plurality of imaging devices, detecting a surveillance image corresponding to the time information and the location information among the plurality of surveillance images received; and processing the data. Generating a surveillance image vector by normalizing the surveillance image detected by the sub; and identifying the region occupied by the ship from the surveillance image vector by using an identification model, an artificial neural network having learned weights in the identification section, to determine whether the ship is in operating state or not. Deriving one or more region boxes indicating whether or not the operating state is in operation, and deriving an output image vector including the derived region boxes; Detecting a region box corresponding to the location information of and determining whether the ship is operating illegally by checking whether the ship is operating or not operating through the detected region box by the information processing unit. do.

In the step of deriving the output image vector, after the identification model divides the monitoring image vector into a plurality of cells, one or more area boxes are derived, and the area box has center coordinates and width and height based on the center coordinates. Coordinates of the region box, reliability representing the rate at which the region occupied by the region box coincides with the region in which the ship exists, and the probability that an object in the region box belongs to the first class representing the operating state of the ship and the ship's non-operation and a classification value including a probability of belonging to the second class representing the state.

The step of detecting the area box corresponding to the location information includes the step of calculating, by the information processing unit, the location information of the ship of the one or more area boxes, and the information processing unit calculating the location information of the identification device among the one or more area boxes and and detecting a region box having location information of the nearest ship.

The calculating of the location information of the ship may include calculating, by the information processing unit, the direction angle of the area box with respect to the ship and the distance between the area box and the ship based on the imaging device, and the information processing unit calculating the image and deriving location information of a ship in the region box according to the direction angle and the distance based on the location information of the device.

에 따라 상기 방향각을 산출하는 단계와, 상기 정보처리부가 수학식

에 따라 상기 거리를 산출하는 단계를 포함한다. Calculating the direction angle of the area box with respect to the ship and the distance of the area box from the ship is performed by the information processing unit using the equation

Calculating the direction angle according to, and the information processing unit Equation

and calculating the distance according to

는 수평 방향의 방향각이고, 상기

는 수직 방향의 방향각이고, 상기 d는 거리이고, 상기 X는 출력영상벡터의 중심좌표 중 수평 방향의 좌표이고, 상기 Y는 출력영상벡터의 중심좌표 중 수직 방향의 좌표이고, 상기 x는 영역상자의 중심좌표 중 수평 방향의 좌표이고, 상기 y는 영역상자의 중심좌표 중 수직 방향의 좌표이고, 상기

는 영상장치의 카메라모듈의 초점 거리인 것을 특징으로 한다. here, above

is the orientation angle in the horizontal direction, and

is a direction angle in the vertical direction, d is a distance, X is a coordinate in the horizontal direction among the coordinates of the center of an output image vector, Y is a coordinate in the vertical direction among the coordinates of the center of an output image vector, and x is an area is the coordinate in the horizontal direction among the coordinates of the center of the box, y is the coordinate in the vertical direction among the coordinates of the center of the area box,

Is the focal length of the camera module of the imaging device.

In the step of determining whether the vessel is illegally operated, if the information processing unit has a probability of belonging to the first class greater than a predetermined threshold and a probability of belonging to the second class is higher, it is determined that the vessel of the area box is in an operating state, , if the probability of belonging to the second class is greater than a predetermined threshold and the probability of belonging to the first class is higher, determining that the ship of the area box is in a non-operational state.

The step of detecting the monitoring image may include operation information including a permitted area indicating an area in which operation is permitted for the vessel indicated by the identification information and a permitted time indicating a permitted time for operation in the permitted area by the data processing unit. extracting, and detecting a surveillance image corresponding to the time information and the location information when the location information and the time information of the identification device do not conform to the permitted area and permitted time of the operation information by the data processing unit Include steps.

에 따라 상기 감시영상의 명도를 조절하고, 수학식

에 따라 상기 감시영상벡터를 생성한다. The step of generating the monitoring image vector is performed by the data processing unit by Equation

Adjust the brightness of the surveillance image according to the equation

The surveillance image vector is generated according to the above.

는 명도가 조절된 감시영상의 픽셀값이고, 상기

는 감시영상의 픽셀값이고, 상기 m은 감시영상의 평균 픽셀값이고, 상기 q는 감시영상의 픽셀값의 표준 편차이고, 상기

는 명암대비 지수이고, 상기

는 명도가 조절된 감시영상의 평균 픽셀값이고, 상기

는 감시영상벡터의 픽셀값인 것을 특징으로 한다. Here, the u and the v are pixel coordinates, and the

Is a pixel value of the surveillance image whose brightness is adjusted,

is the pixel value of the surveillance image, m is the average pixel value of the surveillance image, q is the standard deviation of the pixel values of the surveillance image,

is the contrast index, and

Is the average pixel value of the surveillance image whose brightness is adjusted,

is a pixel value of a surveillance image vector.

The method includes, before the step of continuously receiving the identification information, location information, and time information of the identification device, a learning unit preparing a monitoring image vector for learning and an actual video vector corresponding to the monitoring image vector for learning, and the learning unit preparing the monitoring image vector for learning. The step of inputting a monitoring image vector for learning to an identification model for which learning has not been completed; and the identification model performing a plurality of calculations by applying weights for which learning has not been completed between a plurality of layers to the monitoring image vector for learning, and then output image vector for learning. Generating an output image vector including ; calculating, by the learning unit, a total loss representing a difference between an output image vector generated through a loss function and the measured image vector; The step of performing optimization to modify the weights of the loss model is further included.

The present invention can identify a vessel using an identification model, which is an artificial neural network, and monitor illegal fishing of the identified vessel.

1 is a diagram for explaining the configuration of a system for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention.
2 is a diagram for explaining the configuration of an apparatus for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention.
3 is a diagram for explaining an identification model for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention.
4 is a diagram for explaining an actual image vector for training an identification model and an output image vector calculated by the identification model according to an embodiment of the present invention.
5 is a flowchart for explaining a method of learning an identification model (DM) for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention.
6 is a flowchart illustrating a method for monitoring illegal fishing according to an embodiment of the present invention.
7 is a diagram illustrating a computing device according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in this specification and claims described below should not be construed as being limited to a common or dictionary meaning, and the inventors should use their own invention in the best way. It should be interpreted as a meaning and concept corresponding to the technical idea of the present invention based on the principle that it can be properly defined as a concept of a term for explanation. Therefore, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical ideas of the present invention, so various equivalents that can replace them at the time of the present application. It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. At this time, it should be noted that the same components in the accompanying drawings are indicated by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

First, the configuration of a system for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention will be described. 1 is a diagram for explaining the configuration of a system for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention. Referring to FIG. 1, a system for monitoring illegal fishing based on artificial neural networks (10, hereinafter referred to as 'illegal fishing monitoring system') according to an embodiment of the present invention includes a monitoring device 10 and an identification device 20 and an imaging device 30 .

The identification device 20 is mounted on a ship, and the identification device 20 based on an artificial satellite signal (eg, a Global Positioning System (GPS) signal) or location information indicating the current location of the ship and time indicating the time at which the location information was received It is for detecting time information and transmitting the detected location information and time information together with identification information for identifying the identification device 20 to the monitoring device 10 . To this end, the identification device 20 includes a satellite signal processing unit (not shown) that continuously detects location information and time information based on artificial satellite signals (eg, GPS signals) when the ship starts operating, and the detected location information and time information. It includes a communication unit (not shown) that transmits to the monitoring device 10. The identification device 20 continuously detects location information and time information based on artificial satellite signals (eg, GPS signals) through the satellite signal processing unit when the ship starts operating, and together with the location information and time information detected through the communication unit. Identification information is sent to the monitoring device 10.

The imaging device 30 is for capturing a monitoring image and transmitting the captured monitoring image to the monitoring device 10 . Such an imaging device 30 may be a closed-circuit television (CCTV). The imaging device 30 includes a camera module (not shown) for capturing surveillance images and a communication module (not shown) for transmitting the captured surveillance images by wire or wirelessly. A plurality of imaging devices 30 are installed across the periphery of the harbor. The imaging device 30 may be installed on the ground around a port or installed on a buoy in the sea. The maritime area around the harbor is divided into a plurality of surveillance areas, and the plurality of surveillance areas are assigned to each of the plurality of imaging devices 30 . Accordingly, each of the plurality of imaging devices 30 generates a surveillance image by continuously photographing the assigned surveillance area through a camera module (not shown), and monitors the generated surveillance image through a communication module (not shown). Send to (10).

The monitoring device 10 stores in advance ship information, which is information for identifying a ship, and operation information of the ship corresponding to the identification information. Here, the operation information includes a permitted area indicating an area in which operation of the vessel is permitted and a permitted time indicating a time period during which operation is permitted in the permitted area.

The monitoring device 10 may continuously receive identification information, location information, and time information from the identification device. Then, the monitoring device 10 identifies the vessel using the identification information received from the identification device 20, extracts the operation information of the identified vessel, and then matches the received location information and time information with the extracted operation information. check whether it does Accordingly, the monitoring device 10 can check whether the vessel is located in an unlicensed area at an unauthorized time. In addition, the monitoring device 10 may generate an identification model (DM), which is a machine learning model or deep learning model based on an artificial neural network according to an embodiment of the present invention, based on the monitoring image. In addition, when the identification model (DM) is generated, the monitoring device 10 may check whether the ship is in an operating state or a non-operating state using the generated identification model (DM). Accordingly, the monitoring device 10 can determine that the vessel is engaged in illegal fishing when the vessel is located in an unlicensed area at an unlicensed time and the vessel is in an operating state. there is.

Next, the configuration of an apparatus for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention will be described. 2 is a diagram for explaining the configuration of an apparatus for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention. 3 is a diagram for explaining an identification model for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention. 4 is a diagram for explaining an actual image vector for training an identification model and an output image vector calculated by the identification model according to an embodiment of the present invention.

Referring to FIG. 2 , the monitoring device 10 includes a learning unit 100, a data processing unit 200, an identification unit 300, and an information processing unit 400.

The learning unit 100 is for generating an identification model (DM), which is an artificial neural network-based learning model, through machine learning or deep learning. As shown in FIGS. 3 and 4, the learning unit 100 specifies the area occupied by the ship when the surveillance image vector IV is input to the identification model DM, and the object within the area box B, that is, , Learn to derive an output image vector (OV) including one or more bounding boxes (B) indicating whether the ship is operating or non-operating. A method of generating the identification model (DM) by the learning unit 100 will be described in more detail below.

In the embodiment of the present invention, the identification model (DM) will be described as being software. However, the identification model (DM) may be implemented in hardware or a complex mixture of software and hardware. For example, the identification model (DM) may be implemented through one or more computing devices. For example, such a computing device may include a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and the like. Also, the identification model (DM) may be implemented through a hardware accelerator such as, for example, a tensor processing unit (TPU).

The identification model DM includes a plurality of layers, and each of the plurality of layers includes a plurality of calculation modules for performing a plurality of calculations. Each of the plurality of calculation results of any one layer is transferred to the next layer after a weight is applied. This means that a weight is applied to the calculation result of the current layer and input to the calculation of the next layer. In other words, the identification model (DM) performs a plurality of calculations to which weights of a plurality of layers are applied. The plurality of layers include a Convolution Layer (CVL) that performs a convolution operation, a Pooling Layer (PLL) that performs a Down Sampling operation or an Up Sampling operation, It includes a Fully Connected Layer (FCL) that performs operations based on an activation function. Each of the convolution, downsampling, and upsampling operations uses a kernel composed of a predetermined matrix, and values of elements of the matrix constituting the kernel become weights w. Here, the activation function may include sigmoid, hyperbolic tangent (tanh), exponential linear unit (ELU), rectified linear unit (ReLU), leaky ReLU, Maxout, Minout, Softmax, and the like. . The identification model (DM) may exemplify models to which algorithms such as You Only Look Once (YOLO), YOLOv2, YOLO9000, YOLOv3, YOLOv4, and YOLOv5 are applied.

The identification model (DM) detects one or more region boxes (B) through a plurality of calculations to which weights of a plurality of layers are applied. Area box B includes the coordinates, reliability and classification values of area box B. The coordinates of the area box B have center coordinates (x, y) in any one of a plurality of cells (eg, (1, 1) to (4, 4) in FIG. 4) of the output image vector (OV), The width and height (w, h) are defined based on the center coordinates (x, y, w, h). Confidence (0~1: 0%~100%) is a ground-truth box (GT: ground-truth box) in which the area occupied by the area box (B) contains 100% of the area occupied by the object (obj), that is, the ship. ) indicates the degree of agreement. In other words, confidence indicates the degree to which the area occupied by the area box B includes the area occupied by the ship. The classification value includes a probability that an object in the area box B belongs to a first class representing the operating state of the ship and a probability of belonging to a second class representing the non-operating state of the ship. Then, the identification model (DM) selects a region box (B) whose reliability is higher than a predetermined value among one or more detected region boxes (B), displays the selected region box (B) on the input image vector (IV), and outputs the result. An image vector (OV) is derived.

The data processing unit 200 continuously receives a plurality of monitoring images obtained by taking a plurality of surveillance areas allocated from a plurality of imaging devices 30, and identifies the identification device 20 from the identification device 20 mounted on the ship. Information, location information and time information can be continuously received. In addition, the data processing unit 200 may extract operation information of a ship indicated by the identification information of the identification device 20 . Here, the operation information includes a permitted area indicating an area in which operation of the vessel is permitted and a permitted time indicating a time period during which operation is permitted in the permitted area. Further, the data processing unit 200 may determine whether the location information and time information of the identification device 20 conform to the permitted area and permitted time of the operation information. As a result of this determination, if the location information and time information of the identification device 20 do not conform to the permitted area and permitted time of the operation information, the data processing unit 200 monitors the monitoring area corresponding to the location information transmitted by the identification device 20. The assigned imaging device 30 is selected, and among the monitoring images captured by the selected imaging device 30, a monitoring image captured at a time corresponding to the time information transmitted by the identification device 20 is detected.

The data processing unit 200 normalizes the detected surveillance image to generate a surveillance image vector. At this time, the data processing unit 200 converts pixels of the surveillance image vector so that the ship is highlighted in the surveillance image to generate the surveillance image vector. To this end, the data processing unit 200 first adjusts the brightness of the monitoring image. At this time, the data processor 200 may adjust the brightness of the surveillance image by modifying each pixel value of the surveillance image according to Equation 1 below.

는 명도가 조절된 감시영상의 픽셀값이다. 또한,

는 감시영상의 픽셀값, m은 감시영상의 평균 픽셀값, q는 감시영상의 픽셀값의 표준 편차이다. where u and v are pixel coordinates,

is the pixel value of the surveillance image whose brightness is adjusted. also,

is the pixel value of the surveillance image, m is the average pixel value of the surveillance image, and q is the standard deviation of the pixel values of the surveillance image.

Then, the data processing unit 200 increases the contrast of the surveillance image whose brightness is adjusted to generate a surveillance image vector. At this time, the data processing unit 200 increases the contrast as shown in Equation 2 below based on the average pixel value of the monitoring image whose brightness is adjusted.

는 명도가 조절된 감시영상의 픽셀값이다.

는 명암대비 지수이며, 미리 설정되는 값이다.

는 명도가 조절된 감시영상의 평균 픽셀값을 나타내며,

는 명암대비(contrast)가 증가된 감시영상, 즉, 감시영상벡터의 픽셀값이다. where u and v are pixel coordinates,

is the pixel value of the surveillance image whose brightness is adjusted.

is a contrast index and is a preset value.

represents the average pixel value of the surveillance image whose brightness is adjusted,

is a pixel value of a surveillance image with increased contrast, that is, a surveillance image vector.

The identification unit 300 is for deriving an output image vector (OV) from a monitoring image vector (IV) by using the identification model (DM). To this end, when an image vector is input from the data processing unit 200, the identification unit 300 inputs the input image vector to the identification model DM. Then, the identification model (DM) derives one or more bounding boxes (B) through a plurality of calculations to which weights between a plurality of layers are applied. Here, each of the one or more derived region boxes (B) includes coordinates of the region box (B), confidence (0 to 1), and a match rate. When the output image vector (OV) is divided into a plurality of cells, the coordinates of the region box (B) are the center coordinates (x, y) of each cell and the width (based on the center coordinates (x, y)) w) and height (h). Confidence represents the probability that the area occupied by the area box B includes all the areas occupied by the ship. The classification value includes a probability that an object in the area box B belongs to a first class representing the operating state of the ship and a probability of belonging to a second class representing the non-operating state of the ship.

Subsequently, the identification model (DM) selects a region box (B) having a reliability of a predetermined value, for example, 1 (100%) or more, among one or more derived region boxes (B), in the output image vector (OV). Visualize. That is, as shown in FIG. 4, the identification model (DM) displays the video image B according to the coordinates of the video image B, displays the classification value, and derives the output image vector OV.

The information processing unit 400 detects an area box B specifying a ship on which the identification device 20 is installed among one or more area boxes B of the output image vector OV, and determines the number of the detected area boxes B. This is to derive whether the ship is in an operating state or a non-operating state through the classification value. Accordingly, when the corresponding vessel is confirmed to be operating in an unlicensed area or an unlicensed time, the information processing unit 400 may determine that the vessel is illegally operating.

Next, a method for learning an identification model (DM) for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention will be described. 5 is a flowchart for explaining a method of learning an identification model (DM) for monitoring illegal fishing based on an artificial neural network according to an embodiment of the present invention.

Referring to FIGS. 3 to 5 , the learning unit 100 prepares learning data in step S110. The learning data includes a monitoring image vector IV for learning and a measured image vector GV corresponding to the monitoring image vector IV for learning. Here, the measured image vector (GV) is a label of the surveillance image vector (IV) for learning. The monitoring image vector (IV) for learning is a monitoring image captured by the imaging device 30 converted into a monitoring image vector (IV). As described above, the method of converting the monitoring image for learning into the monitoring image vector (IV) is an image in which brightness is adjusted and contrast is increased through Equations 1 and 2. That is, the surveillance image vector IV for learning is generated in the same way as the method in which the data processing unit 200 converts the surveillance image into the surveillance image vector IV. The measured image vector (GV) is an image obtained by adding a ground-truth box (GT) to the monitoring image vector (IV) for learning. The measurement box (GT) represents the area actually occupied by the ship in the monitoring image vector (IV) for learning, and includes coordinates (x, y, w, h), reliability, and classification values of the measurement box (GT). The classification value of the measurement box (GT) is a known value because it is learning data, and if the object in the measurement box (GT) belongs to the first class (CL1) representing the case of the ship's operating state, [CL1=1, CL2= 0], and the object in the measurement box GT belongs to the second class CL2 indicating the case in which the ship is not operating, it may be expressed as [CL1 = 0, CL2 = 1].

The coordinates of the measurement box GT have center coordinates (x, y) in any one cell. For example, as shown in FIG. 4 , the center coordinates (x, y) of the measurement box GT exist in cells (2, 2). In addition, the coordinates of the measurement box GT further have a width w and a height h based on the center coordinates (x, y). And the reliability of the measurement box (GT) is 1 (100%).

Next, the learning unit 100 inputs the surveillance image vector (IV) for training to the identification model (DM) having weights for which learning has not been completed in step S120.

Then, in step S130, the identification model (DM) derives an output image vector (OV) for learning by performing a plurality of calculations to which a weight for which learning between a plurality of layers has not been completed is applied to the monitoring image vector (IV) for learning. At this time, the identification model (DM) derives one or more region boxes (B), and among the one or more derived region boxes (B), the region box having a reliability of a predetermined value or more in a cell in which the center coordinates of the measurement box (GT) exist ( Select B) as the learning area box (B). Then, the identification model (DM) generates an output image vector (OV) for learning to which the learning region box (B) is applied.

Then, in step S140, the learning unit 100 determines the difference between the output image vector for learning (OV) and the measured image vector (GV), more specifically, the learning area box (B) and the measured image vector of the output image vector (OV) for learning. Calculate the loss representing the difference between (GV) and the actual measurement box (GT). At this time, the learning unit 100 may calculate the loss through a loss function such as Equation 3 below.

는 영역상자의 변수에 대한 값을 더 반영하기 위한 것으로, 영역상자의 좌표(x, y, w, h)에 대한 손실과 다른 손실들과의 균형을 위한 하이퍼파라미터이다.

는 영역상자의 변수에 대한 값을 더 반영하고, 물체가 없는 영역에 대한 값을 덜 반영하기 위한 것이다. 즉,

는 객체가 있는 영역상자와 객체가 없는 영역상자 간의 균형을 위한 하이퍼파라미터이다. 여기서,

=5이고,

=0.5가 될 수 있다.

는 셀 i에 객체, 예컨대, 선박이 있는 경우 1이고, 없는 경우 0을 나타낸다.

는 셀 i에 있는 영역상자 j에 객체, 예컨대, 선박이 있으면 1이고, 없으면 0을 나타낸다.

는 셀 i에 있는 영역상자 j에 객체, 예컨대, 선박이 없으면 1이고, 있으면 0을 나타낸다. S represents the number of cells, and C represents reliability. B represents the number of area boxes (B) in one cell. pi(c) represents the probability that the object obj in the ith cell is of the corresponding class c, that is, the classification value. In an embodiment of the present invention, there are two classes, a first class (CL1: c=1) and a second class (CL2: c=2). For example, pi(1) is the probability that the object in the area box B belongs to the first class representing the operating state of the ship, and pi(2) is the probability of belonging to the second class representing the non-operating state of the ship indicates Here, i is a parameter indicating a cell in which an object exists, and j is a parameter indicating a predicted region box. In addition, x and y represent the coordinates of the center of the region box, and w and h represent the width and height of the region box, respectively.

is to further reflect the value of the variable of the region box, and is a hyperparameter for balancing the loss with respect to the coordinates (x, y, w, h) of the region box and other losses.

is to reflect more the value of the variable of the area box and less reflect the value of the area without the object. in other words,

is a hyperparameter for the balance between a region box with an object and a region box without an object. here,

=5, and

= 0.5.

represents 1 when there is an object, eg, a ship, in cell i, and 0 when there is no cell i.

represents 1 if there is an object, for example, a ship, in area box j in cell i, and 0 otherwise.

represents 1 if there is no object, for example, a ship, in the area box j in cell i, and 0 if there is.

Looking at the loss function of Equation 3, the first term of Equation 3 is equal to Equation 4 below.

)와 선박이 차지하는 영역을 나타내는 실측상자(GT)의 좌표(

)와의 차이를 나타내는 좌표 손실(coordinate loss)을 산출하기 위한 것이다. Equation 4 is the coordinates of the learning area box (B) (

) and the coordinates of the measurement box (GT) representing the area occupied by the ship (

) to calculate the coordinate loss representing the difference.

In addition, the second and third terms of Equation 3 are the same as Equation 5 below.

는 학습용 영역상자(B)의 신뢰도이고,

는 실측상자(GT)의 신뢰도이다. Equation 5 is for calculating the confidence loss representing the difference between the area occupied by the learning area box B and the area occupied by the measurement box GT. in other words,

Is the reliability of the learning area box (B),

Is the reliability of the measurement box (GT).

Finally, the last term of Equation 3 is equal to Equation 6 below.

는 학습용 영역상자(B)의 분류값이고,

는 실측상자(GT)의 분류값이다. Equation 6 is for calculating a classification loss representing a difference between an object in the region box B and an object in the measurement box GT.

Is the classification value of the learning area box (B),

Is the classification value of the measurement box GT.

As described above, when the total loss including the coordinate loss, the reliability loss, and the classification loss is calculated through the loss function, the learning unit 100 determines that the total loss including the coordinate loss, the reliability loss, and the classification loss is the minimum in step S150. Optimization is performed to modify the weights of the identification model (DM) so that

Meanwhile, the learning unit 100 repeatedly performs steps S120 to S150 described above using a plurality of different learning data until a predetermined end condition is satisfied. According to this repetition, the weight of the identification model (DM) is repeatedly updated. In addition, this repetition may be performed until the total loss is less than or equal to a predetermined target value. Therefore, according to one embodiment, the learning unit 100 determines whether the total loss calculated earlier (S140) is less than or equal to a preset target value in step S160, and if the total loss is less than or equal to the preset target value, in step S170, the identification model ( DM) to complete the learning.

Meanwhile, according to an alternative embodiment of step S160, the learning unit 100 completes learning of the identification model (DM) in step S170 when the coordinate loss and the reliability loss are less than or equal to the preset target value and the classification loss converges. do.

Next, as described above, a method for monitoring illegal fishing using the identification model (DM) having weights learned after learning has been completed will be described. 6 is a flowchart illustrating a method for monitoring illegal fishing according to an embodiment of the present invention.

Referring to FIG. 6 , a plurality of imaging devices 30 are installed around the harbor. The maritime area around the harbor is divided into a plurality of surveillance areas, and the plurality of surveillance areas are assigned to each of the plurality of imaging devices 30 . Accordingly, each of the plurality of imaging devices 30 continuously captures the assigned monitoring area to generate a monitoring image, and transmits the generated monitoring image to the monitoring device 10 .

On the other hand, the identification device 20 mounted on the ship continuously detects position information and time information based on artificial satellite signals (eg, GPS signals) when the ship starts operating, and identifies the detected location information and time information as the identification device ( 20) is transmitted to the monitoring device 10 together with the identification information for identifying it.

Accordingly, the data processing unit 200 continuously receives a plurality of surveillance images obtained by photographing a plurality of surveillance areas allocated from a plurality of imaging devices 30 in step S210, and identifies them from the identification device 20 mounted on the ship. The identification information, location information and time information of the device 20 are continuously received.

The data processing unit 200 extracts the operation information of the ship indicated by the identification information of the identification device 20 in step S220. Here, the operation information includes a permitted area indicating an area in which operation of the vessel is permitted and a permitted time indicating a time period during which operation is permitted in the permitted area.

Subsequently, the data processing unit 200 determines whether the location information and time information of the identification device 20 coincide with the permitted area and permitted time of the operation information in step S230. As a result of the determination in step S230, if the location information and time information of the identification device 20 do not match the permitted area and permitted time of the operation information, the data processing unit 200 proceeds to step S240.

Accordingly, the data processing unit 200 selects the imaging device 30 to which the monitoring area corresponding to the location information transmitted by the identification device 20 is allocated in step S240, and the monitoring image captured by the selected imaging device 30. Among them, a monitoring image taken at a time corresponding to the time information transmitted by the identification device 20 is detected.

Subsequently, the data processing unit 200 normalizes the detected surveillance image in step S250 to generate a surveillance image vector. That is, the data processing unit 200 adjusts brightness and contrast of the detected surveillance image according to Equations 1 and 2 to generate a surveillance image vector. That is, the data processing unit 200 converts the pixels of the surveillance image vector to generate the surveillance image vector so that the ship is highlighted in the surveillance image. To this end, the data processor 200 first adjusts the brightness of the surveillance image by adjusting each pixel value of the surveillance image according to Equation 1. Then, the data processing unit 200 generates a surveillance image vector (IV) by increasing the contrast of the surveillance image whose brightness is adjusted based on the average pixel value of the surveillance image whose brightness is adjusted according to Equation 2. do.

Next, the identification unit 300 derives an output image vector (OV) from the monitoring image vector (IV) using the identification model (DM) in step S260. That is, the identification unit 300 identifies the ship from the surveillance image vector IV by using the identification model, which is an artificial neural network having learned weights, and derives one or more area boxes B for identifying whether or not it is in an operational state. , an output image vector (OV) including the derived area box (B) is derived. At this time, when the identification unit 300 inputs the surveillance image vector IV to the identification model DM, the identification model DM applies a plurality of weights to which a plurality of inter-layer learning has been completed for the surveillance image vector IV. After dividing the surveillance image vector (IV) into a plurality of cells by performing the operation of, one or more area boxes (B) are derived. Here, each region box (B) has center coordinates (x, y) and coordinates (x, y, w, h) of the region box representing the width and height (w, h) based on the center coordinates (x, y) , Confidence indicating the rate at which the area occupied by the area box (B) coincides with the area occupied by the ship, which is an object, and the probability that the object in the area box (B) belongs to the first class representing the operating state of the ship and the ship's A classification value including a probability of belonging to the second class representing the non-operational state is included. Next, the identification model (DM) expresses the region box (B) in the surveillance image vector (IV) to generate an output image vector (OV). When learning is completed, since the reliability of the derived region box (B) is greater than or equal to the reference value (eg, 0.95 (95%), 1 (100%), etc.), the output image vector (OV) contains the coordinates of the region box (B) ( Only x, y, w, h) and classification values (eg, CL1 = 0.9001, CL2 = 0.0999) can be expressed.

When the output image vector OV is derived, the information processing unit 400 performs a region box corresponding to the location information of the identification device 20 among one or more region boxes B included in the output image vector OV in step S270 ( B) is detected. That is, the information processing unit 400 detects the location information of one or more area boxes (B), and among the one or more area boxes (B), the area box (B) having the location information closest to the location information of the identification device 20 detect Here, the location information of the area box (B) means the location information of the object (ship) of the area box (B).

을 산출하고, 다음의 수학식 8에 따라 영상장치(30)로부터 영역상자(B) 내의 객체(선박)와의 거리 d를 산출할 수 있다. To this end, the information processing unit 400 calculates the bearing angle and distance to the object in the area box B, that is, the ship, based on the imaging device 30 . In particular, the information processing unit 400 calculates an azimuth angle for an object (ship) within the area box B based on the imaging device 30 according to Equation 7 below.

, and the distance d from the imaging device 30 to the object (ship) in the area box B can be calculated according to Equation 8 below.

는 수평 방향의 방향각이고,

는 수직 방향의 방향각을 나타낸다. 수학식 8에서, d는 영상장치(30)로부터 영역상자(B) 내의 객체와의 거리를 나타낸다. 수학식 7 및 수학식 8에서, X는 출력영상벡터(OV)의 중심좌표(X, Y) 중 수평 방향의 좌표이고, Y는 출력영상벡터(OV)의 중심좌표(X, Y) 중 수직 방향의 좌표이다. 그리고 x는 영역상자(B)의 중심좌표(x, y) 중 수평 방향의 좌표이고, y는 영역상자(B)의 중심좌표(x, y) 중 수직 방향의 좌표이다. 또한,

는 영상장치(30)의 카메라모듈의 초점 거리(focal length)를 나타낸다. In Equation 7,

is the orientation angle in the horizontal direction,

represents the orientation angle in the vertical direction. In Equation 8, d represents the distance from the imaging device 30 to the object within the area box B. In Equations 7 and 8, X is the coordinate in the horizontal direction among the center coordinates (X, Y) of the output image vector (OV), and Y is the vertical coordinate among the center coordinates (X, Y) of the output image vector (OV). is the coordinate of the direction. Further, x is a coordinate in the horizontal direction among the coordinates (x, y) of the center of the region box B, and y is a coordinate in the vertical direction among the coordinates (x, y) of the center of the region box B. also,

represents the focal length of the camera module of the imaging device 30.

As described above, when the direction angle and distance to the object (ship) in the area box B are calculated based on the imaging device 30, the information processing unit 400 calculates the direction angle and distance and the imaging device 30 ) can be used to calculate the location information of the object (ship) of the area box (B). Accordingly, the information processing unit 400 may detect the area box B having the location information of the identification device 20 and the location information of the object in the area box B that is closest to the location information.

Next, the information processing unit 400 determines whether or not illegal fishing is performed according to the classification value of the area box B detected in step S280. That is, the information processing unit 400 checks the classification value of the area box B to determine whether the corresponding ship is in an operating state or a non-operating state. The classification value calculated by the identification model (DM) includes a probability that the object is in the first class representing the operating state of the ship and a probability of being in the second class representing the non-operational state of the ship. Accordingly, the information processing unit 400 determines that the vessel in the area box B is in operation if the probability of being in the first class is greater than a predetermined threshold and higher than the probability of being in the second class. According to the judgment based on these classification values, when the corresponding vessel is in the operating state, the information processing unit 400 determines the area or time that does not conform to the permitted area and permitted time based on the previously described operation information (S220, S230). Since it is being operated, it can be judged to be illegal fishing.

On the other hand, the information processing unit 400 determines that the vessel in the area box B is in a non-operational state if the probability of being in the second class is greater than a predetermined threshold and higher than the probability of being in the first class. Accordingly, the information processing unit 400 determines that the vessel in the area box B is not illegally operated because the operation itself was not performed.

7 is a diagram illustrating a computing device according to an embodiment of the present invention. The computing device TN100 of FIG. 7 may be a device described herein, for example, the monitoring device 10 .

In the embodiment of FIG. 7 , the computing device TN100 may include at least one processor TN110, a transceiver TN120, and a memory TN130. In addition, the computing device TN100 may further include a storage device TN140, an input interface device TN150, and an output interface device TN160. Elements included in the computing device TN100 may communicate with each other by being connected by a bus TN170.

The processor TN110 may execute program commands stored in at least one of the memory TN130 and the storage device TN140. The processor TN110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Processor TN110 may be configured to implement procedures, functions, methods, and the like described in relation to embodiments of the present invention. The processor TN110 may control each component of the computing device TN100.

Each of the memory TN130 and the storage device TN140 may store various information related to the operation of the processor TN110. Each of the memory TN130 and the storage device TN140 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory TN130 may include at least one of read only memory (ROM) and random access memory (RAM).

The transmitting/receiving device TN120 may transmit or receive a wired signal or a wireless signal. The transmitting/receiving device TN120 may perform communication by being connected to a network.

On the other hand, the method according to the embodiment of the present invention described above can be implemented in the form of a program readable through various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program commands may include high-level language wires that can be executed by a computer using an interpreter, as well as machine language wires such as those produced by a compiler. These hardware devices may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

The present invention has been described above using several preferred examples, but these examples are illustrative and not limiting. As such, those skilled in the art to which the present invention belongs will understand that various changes and modifications can be made according to the doctrine of equivalents without departing from the spirit of the present invention and the scope of rights set forth in the appended claims.

10: monitoring device
20: identification device
30: imaging device
100: learning unit
200: data processing unit
300: identification unit
400: information processing unit

Claims (10)

In the device for monitoring illegal fishing,
When the identification information, location information, and time information of the identification device are continuously received from the identification device mounted on the ship, and a plurality of monitoring images are received from a plurality of imaging devices, the time information and the monitoring images are received from a plurality of monitoring images. After detecting the surveillance image corresponding to the location information, the data processing unit normalizes the detected surveillance image to generate a surveillance image vector;
Using the identification model, which is an artificial neural network with learned weights, the area occupied by the ship is specified from the surveillance image vector, and one or more area boxes indicating whether the vessel is in operation or non-operation are derived, and the derived area box is an identification unit for deriving an output image vector including; and
Among one or more area boxes included in the output image vector (OV), an area box corresponding to the location information of the identification device is detected, and whether the ship is operating or not operating is confirmed through the detected area box. Information processing unit that determines whether the ship is illegally fishing;
characterized in that it includes
A device for monitoring illegal fishing. According to claim 1,
The identification model is
After dividing the surveillance image vector into a plurality of cells,
Derive one or more domain boxes;
The area box is
Coordinates of the center coordinates and the area box representing the width and height based on the center coordinates;
Confidence indicating the rate at which the area occupied by the area box coincides with the area where the vessel exists; and
Characterized in that it includes a classification value including a probability that the object in the area box belongs to a first class representing the operating state of the ship and a probability belonging to a second class representing the non-operating state of the ship
A device for monitoring illegal fishing. According to claim 2,
The information processing unit
Calculate location information of ships in the one or more area boxes;
Characterized in that detecting an area box having location information of a ship closest to the location information of the identification device among the one or more area boxes
A device for monitoring illegal fishing. According to claim 3,
The information processing unit
Calculate the direction angle of the area box with respect to the ship and the distance of the area box from the ship based on the imaging device;
Characterized in that the location information of the ship in the area box is derived according to the direction angle and the distance based on the location information of the imaging device
A device for monitoring illegal fishing. According to claim 4,
The information processing unit
math formula

Calculate the direction angle according to,
math formula

Calculate the distance according to,
remind

is the orientation angle in the horizontal direction,
remind

is the orientation angle in the vertical direction,
Where d is the distance,
The X is a coordinate in the horizontal direction among the coordinates of the center of the output image vector,
Y is the coordinate in the vertical direction among the coordinates of the center of the output image vector,
The x is a coordinate in the horizontal direction among the coordinates of the center of the region box,
y is the coordinate in the vertical direction among the coordinates of the center of the region box,
remind

Is the focal length of the camera module of the imaging device
A device for monitoring illegal fishing. According to claim 3,
The information processing unit
If the probability of belonging to the first class is greater than a predetermined threshold and the probability of belonging to the second class is higher, it is determined that the vessel in the area box is in operation;
When the probability of belonging to the second class is greater than a predetermined threshold and the probability of belonging to the first class is higher, it is determined that the vessel in the area box is in a non-operational state.
A device for monitoring illegal fishing. According to claim 1,
The data processing unit extracts operation information including a permitted area indicating an area in which operation is permitted for the vessel indicated by the identification information and a permitted time indicating a permitted time for operation in the permitted area,
If the location information and the time information of the identification device do not conform to the permitted area and permitted time of the operation information,
Characterized in that for detecting a surveillance image corresponding to the time information and the location information
A device for monitoring illegal fishing. According to claim 1,
The data processing unit
math formula

Adjusting the brightness of the surveillance image according to,
math formula

Generating the surveillance image vector according to;
The u and the v are pixel coordinates,
remind

is the pixel value of the surveillance image whose brightness is adjusted,
remind

is the pixel value of the surveillance image,
The m is the average pixel value of the surveillance image,
q is the standard deviation of pixel values of the surveillance image;
remind

is the contrast index,
remind

is the average pixel value of the surveillance image whose brightness is adjusted,
remind

Is the pixel value of the surveillance image vector
A device for monitoring illegal fishing. According to claim 1,
Preparing a monitoring image vector for learning and an actual measurement image vector corresponding to the monitoring image vector for learning;
After inputting the surveillance image vector for learning to an identification model in which learning is not completed,
When the identification model generates an output image vector including an output image vector for learning by performing a plurality of operations to which a weight for which learning between a plurality of layers has not been completed is applied to the surveillance image vector for learning,
Calculating a total loss representing a difference between an output image vector generated through a loss function and the measured image vector;
Optimization is performed to modify the weights of the loss model so that the total loss is minimized.
learning department;
characterized in that it further comprises
A device for monitoring illegal fishing. In a method for monitoring illegal fishing,
Continuously receiving identification information, location information, and time information of the identification device from the identification device mounted on the ship by the data processing unit;
detecting, when the data processing unit receives a plurality of surveillance images from a plurality of imaging devices, a surveillance image corresponding to the time information and the location information among the plurality of surveillance images received;
generating a surveillance image vector by normalizing the detected surveillance image by the data processing unit;
The identification unit derives one or more area boxes indicating whether the vessel is in operation or non-operation status while specifying the area occupied by the vessel from the surveillance image vector using the identification model, which is an artificial neural network having learned weights, and the derived area deriving an output image vector including a box;
detecting, by an information processing unit, a region box corresponding to the location information of the identification device among one or more region boxes included in an output image vector (OV); and
determining whether the vessel is illegally operated by checking whether the vessel is in an operating state or a non-operating state through the detected region box by the information processing unit;
characterized in that it includes
Methods to monitor illegal fishing.

Images