KR20210090572A - Device and method for monitoring a berthing - Google Patents

Abstract

The present invention relates to a device for monitoring berthing to monitor ships and a port and a method thereof. According to one embodiment of the present invention, a berthing monitoring method performed by a computing means comprises the following steps of: acquiring an image of a port including the sea and a plurality of ships by using a camera installed in the port to capture the image; using an artificial neural network learned by using a learning set, in which class values indicating the sea, a ship, and a feature, respectively, are labeled to the input image and pixels corresponding to objects including the sea, the ship, and the feature included in the input image, to generate segmentation images for the objects from the port image; tracking a plurality of ships on the basis of first pixels to which a class value indicating a ship of the segmented image is assigned to acquire tracking information including location information of representative points representing each of the plurality of ships; determining a berthing target ship to be guided among the ships tracked on the basis of the tracking information; and acquiring berthing guide information including a bow distance, which is a distance between the bow of the target ship and a quay wall, and a stern distance, which is a distance between the stern of the target ship and the quay wall, on the basis of the segmentation image.

Description

DEVICE AND METHOD FOR MONITORING A BERTHING

The present invention relates to an eyepiece monitoring apparatus and method, and more particularly, to an apparatus and method for performing image-based eyepiece monitoring.

In recent years, many accidents have occurred in the operation of ships and berthing and berthing in ports. Most of the accidents are caused by the inability to accurately check the situation around the ship or in the port through video when berthing.

In the past, various types of sensors such as ECDIS and radar are used to support berthing, but in the case of ECDIS, there are limitations due to GPS inaccuracy, AIS update cycle, and AIS unregistered moving objects. There are limitations due to presence and noise.

Therefore, there is a need to develop a technology for actually monitoring the situation around a ship or in a port through an image when berthing.

One problem to be solved by the present specification is to provide a berthing monitoring apparatus and method for monitoring the perimeter of a ship and a port.

Another object to be solved in the present specification is to provide a berthing monitoring apparatus and method for determining a vessel to be monitored, and calculating and outputting information about the monitoring target.

Another object to be solved by the present specification is to provide an eyepiece monitoring apparatus and method using image segmentation.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to one aspect of the present specification, in the berthing monitoring method performed by the computing means, the steps of acquiring an image of the harbor including the sea and a plurality of ships using a camera installed in the harbor to take an image, an input image and Using an artificial neural network learned using a running set in which class values indicating the sea, ship, and feature, respectively, are labeled in pixels corresponding to objects including the sea, ship, and feature included in the input image. generating a segmentation image for the objects from a harbor image; a representative representing each of the plurality of vessels by tracking the plurality of vessels based on first pixels assigned with a class value indicating the vessel of the segmentation image Acquiring tracking information including location information of points, determining a target vessel to be guided berthing among the tracked vessels based on the tracking information, and between the bow and the quay wall of the target vessel based on the segmentation image An berthing monitoring method comprising the step of obtaining berthing guide information including a bow distance that is a distance and a stern distance that is a distance between the stern and the quay wall of the target vessel may be provided.

In addition, according to another aspect of the present specification, a camera installed in a harbor to capture an image, obtains an image of a harbor including the sea and a vessel captured by the camera, and an input image and a sea, a vessel included in the input image And Segmentation of the objects from the harbor image using an artificial neural network learned using a running set in which pixel corresponding to objects including a feature is labeled with class values indicating the sea, a ship, and a feature, respectively Generates an image, tracks the plurality of ships based on first pixels assigned a class value indicating a ship of the segmentation image, tracking information including location information of representative points representing each of the plurality of ships Obtain, based on the tracking information, determine a target vessel to be guided berthing among the vessels being tracked, and based on the segmentation image, the bow distance, which is the distance between the bow of the target vessel and the quay wall, and the stern and quay wall of the target vessel An eyepiece monitoring device including a control module for obtaining eyepiece guide information including a distance between the stern distances, and a communication module for transmitting the eyepiece guide information of the target vessel to a remotely located terminal may be provided.

The solutions of the problems of the present specification are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to an embodiment of the present specification, by calculating berthing guide information for monitoring the periphery of the vessel and the port, monitoring of the vessel and the port may be performed.

According to an embodiment of the present specification, it is possible to efficiently perform berthing monitoring by determining a vessel to be monitored, and calculating and outputting information on the determined monitoring target.

According to an embodiment of the present specification, a vessel may be recognized using image segmentation, and a monitoring target may be determined based thereon.

The effects of the present invention are not limited to the above-mentioned effects, and the effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings.

1 is a diagram of image-based monitoring according to an embodiment.
2 is a view of an eyepiece monitoring device according to an embodiment.
3 and 4 are diagrams related to an embodiment of an eyepiece monitoring device according to an embodiment.
5 is a diagram illustrating a viewing angle and a depth of field according to an exemplary embodiment.
6 and 7 are views of an installation position of a sensor module according to an embodiment.
8 is a diagram related to image analysis according to an exemplary embodiment.
9 to 11 are diagrams of an object recognition step according to an exemplary embodiment.
12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
14 and 15 are diagrams for estimating position/movement information of an object according to an embodiment.
16 is a flowchart for determining a target vessel according to an embodiment.
17 is a view related to an example of tracking a ship according to an embodiment.
18 is a diagram illustrating an example of determining a target vessel based on location information according to an embodiment.
19 is a diagram illustrating an example of determining a target vessel based on movement information according to an embodiment.
20 is a diagram illustrating an example of determining a target vessel based on navigation information according to an embodiment.
21 is a diagram illustrating an example of determining a target vessel based on user input information according to an embodiment.
22 is a view related to obtaining information on the berthing guide of the target vessel according to an embodiment.
23 is a flowchart of monitoring information output according to an embodiment.
24 is a diagram illustrating an example of outputting monitoring information according to an embodiment.
25 to 27 are diagrams illustrating another example of outputting monitoring information according to an embodiment.
28 is a diagram related to image-based monitoring based on a plurality of images according to an embodiment.
29 and 30 are diagrams of another example of outputting monitoring information according to an embodiment.
31 and 32 are diagrams of another example of outputting monitoring information according to an embodiment.
33 and 34 are diagrams for view transformation according to an embodiment.
35 is a flowchart of eyepiece monitoring according to an embodiment.

The embodiments described in this specification are for clearly explaining the spirit of the present invention to those of ordinary skill in the art to which the present invention belongs, so the present invention is not limited by the embodiments described in this specification, and the present invention It should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in the present specification are selected as widely used general terms as possible in consideration of the functions in the present invention, but they may vary depending on the intention, custom, or emergence of new technology of those of ordinary skill in the art to which the present invention belongs. can However, if a specific term is defined and used with an arbitrary meaning, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the terms and the contents of the entire specification, rather than the names of simple terms.

The drawings attached to this specification are for easy explanation of the present invention, and the shapes shown in the drawings may be exaggerated as necessary to help understand the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

As used herein, the term 'port image' may be understood as an image related to a port, for example, a port image includes an image captured by a camera installed in a port, an image including at least a part of the port, and the like. can do.

According to an aspect of the present specification, there is provided a berthing monitoring method performed by a computing means, the method comprising: acquiring an image of a harbor including the sea and a plurality of ships using a camera installed in a harbor to take an image; An artificial neural network learned using a learning set in which the input image and the pixels corresponding to the objects including the sea, the ship, and the feature included in the input image are labeled with class values indicating the sea, the ship, and the feature, respectively. generating segmentation images for the objects from the harbor image using; acquiring tracking information including location information of representative points representing each of the plurality of ships by tracking the plurality of ships based on first pixels to which a class value indicating a ship of the segmented image is assigned; determining a target vessel to be guided berthing among the vessels being tracked on the basis of the tracking information; and obtaining berthing guide information including a bow distance that is a distance between a bow of the target ship and a quay wall and a stern distance that is a distance between the stern and a quay wall of the target ship based on the segmentation image; An eyepiece monitoring method comprising a may be provided.

Here, the representative point may be one point on the segmentation area of each of the plurality of ships of the segmentation image.

Here, the representative point may be a point located at the center of the segmentation area of each of the plurality of ships.

Here, the acquiring of the tracking information may include acquiring a distance between the representative point and the quay wall, and the target vessel may be determined based on the distance between the representative point and the quay wall.

Here, the target vessel may be determined as a vessel in which a distance between the representative point and the quay wall among the plurality of vessels is less than a preset value.

Here, the target vessel may be determined as a vessel in which the representative point is located on a preset area among the plurality of vessels.

Here, the acquiring of the tracking information may include acquiring movement information of the representative point based on the location information of the representative point, and the target vessel may be determined based on the movement information of the representative point.

Here, the target vessel may be determined based on a magnitude of an approach speed of the representative point among the plurality of vessels to the quay wall.

Here, the acquiring of the tracking information may include acquiring at least one of VTS information and AIS information of the plurality of vessels, and the target vessel may be determined based on at least one of the VTS information and AIS information. .

Here, the target vessel may be determined based on a destination according to at least one of the VTS information and the AIS information among the plurality of vessels.

Here, the acquiring of the tracking information may include acquiring user input information on the plurality of ships from a user terminal, and the target ship may be determined based on user input information on the plurality of ships.

Here, the target vessel may be determined as a vessel selected according to the user input information among the plurality of vessels.

Here, the target vessel may be determined as one vessel that satisfies the condition later when there are a plurality of vessels that satisfy the condition used for determining the target vessel.

Here, the artificial neural network is an input image and pixels corresponding to objects including ships and features excluding the sea, tugboat, and tugboat included in the input image, respectively. It can be learned using a learning set that labels the indicated class values.

Here, the step of obtaining the berthing guide information is the step of extracting a pair of points corresponding to both ends of the bottom surface of the target ship in contact with the sea level, and the bow distance and the stern distance based on the pair of points It may include the step of obtaining.

Here, the step of obtaining the berthing guide information is a bow speed, which is a speed at which the bow of the target ship approaches the quay wall, based on the bow distance and the stern distance, and a speed at which the stern of the target ship approaches the quay wall. and obtaining a stern speed.

Here, the step of obtaining the eyepiece guide information may include obtaining a distance between the target ship and the other ship based on the segmentation image.

Here, the step of obtaining the berthing guide information may include obtaining a relative speed between the target ship and the other ship based on a distance between the target ship and the other ship.

Here, the step of obtaining the berthing guide information may include obtaining information related to the risk of collision with another vessel or the quay wall of the vessel.

Here, the eyepiece monitoring method comprises: outputting the eyepiece guide information together with the port image; may further include.

Here, the harbor image may include a panoramic image in which a plurality of harbor images are registered.

In addition, a recording medium in which a program for executing the above-described method is recorded may be provided.

In addition, according to an aspect of the present specification, a camera is installed in the harbor to take an image; Acquires a port image including the sea and ships captured by the camera, and applies the sea, ship, and features to pixels corresponding to the input image and objects including the sea, ships, and features included in the input image, respectively. A first pixel to which segmentation images for the objects are generated from the harbor image using an artificial neural network trained using a running set that label class values indicating, and a class value indicating a vessel of the segmentation image is assigned By tracking the plurality of ships based on the acquisition tracking information including the location information of representative points representing each of the plurality of ships, based on the tracking information, determining a target ship to be guided among the ships being tracked based on the tracking information And, based on the segmentation image, a control module for obtaining berthing guide information including a bow distance that is a distance between a bow of the target ship and a pier and a stern distance that is a distance between the stern and a quay wall of the target ship; and a communication module for transmitting the berthing guide information of the target vessel to a terminal located remotely. An eyepiece monitoring device comprising a may be provided.

Hereinafter, an eyepiece monitoring apparatus and method according to an embodiment will be described.

In this specification, monitoring refers to grasping or recognizing the surrounding situation, detecting a detection target such as a certain area or a specific object using various sensors and providing the detection result to the user, as well as through calculation based on the detection result. It should be construed broadly to include, for example, providing additional information.

Image-based monitoring may mean identifying or recognizing a surrounding situation based on an image. For example, monitoring may mean acquiring information for calculating berthing guide information during berthing or berthing of a ship or recognizing other ships or obstacles therefrom by acquiring images around the ship when the ship is operating.

Berthing guide information refers to the surrounding environment necessary for berthing, such as recognizing other ships or obstacles, understanding the port situation, whether the berth is accessible, the distance and speed from the quay wall, the distance and speed from other ships, and the presence of obstacles in the navigation route can mean information about In this specification, although mainly described for monitoring when berthing is performed in ships and ports, it is not limited thereto and may be applied to the case of driving of a vehicle, operation of an aircraft, and the like.

1 is a diagram of image-based monitoring according to an embodiment. Referring to FIG. 1 , image-based monitoring may include an image acquisition step S10 and an image analysis step S20 .

The image acquisition step S10 may mean a step in which the device 10 acquires an image. Here, the type of image may be various, such as an RGB image, an IR image, a depth image, a lidar image, and a radar image, and there is no limitation. In addition, not only a two-dimensional image but also a three-dimensional image is possible.

The image analysis step S20 may refer to a step of obtaining an analysis result based on the image. For example, the image analysis step (S20) may include calculating the eyepiece guide information through the image. Alternatively, the image analysis step ( S20 ) may refer to a step of analyzing characteristics of an object included in the image. Alternatively, the image analysis step S20 may include determining a situation indicated by the image. or,

Details of the image acquisition step ( S10 ) and the image analysis step ( S20 ) will be described later. Hereinafter, information acquired through the image acquisition step ( S10 ) or the image analysis step ( S20 ) is referred to as monitoring information.

2 is a view of an eyepiece monitoring device according to an embodiment. Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100 , a control module 200 , and a communication module 300 .

The sensor module 100 may sense information about a ship or a ship's vicinity and a port. The sensor module 100 may include an automatic identification system (AIS), an image generating unit, a position measuring unit, an attitude measuring unit, a casing, and the like.

The image generating unit may generate an image. The image generating unit may include a camera, lidar, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

The position measuring unit may measure the position of a component included in the sensor module, such as a sensor module or an image generating unit. As an example, the location measurement unit may be a Global Positioning System (GPS). In particular, Real-Time Kinematic GPS (RTK-GPS) may be used to improve the accuracy of location measurement.

The position measuring unit may acquire position information at predetermined time intervals. Here, the time interval may vary depending on the installation position of the sensor module. For example, when the sensor module is installed in a moving object such as a ship, the position measurement unit may acquire position information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the position measurement unit may acquire position information at long time intervals. The time interval for obtaining the location information of the location measuring unit may be changed.

The posture measuring unit may measure the posture of components included in the sensor module, such as a sensor module or an image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measuring unit may acquire posture information at predetermined time intervals. Here, the time interval may vary depending on the installation position of the sensor module. For example, when the sensor module is installed in a moving object such as a ship, the posture measuring unit may acquire posture information at short time intervals. On the other hand, when the sensor module is installed in a fixed body such as a port, the posture measuring unit may acquire posture information at long time intervals. The time interval for acquiring the posture information of the posture measuring unit may be changed.

The casing may protect a sensor module such as an image generating unit, a position measuring unit, and a posture measuring unit.

At least one of an image generating unit, a position measuring unit, and a posture measuring unit may be present inside the casing. The casing may prevent equipment such as an image generating unit existing therein from being corroded by salt water. Alternatively, the casing may protect it by preventing or mitigating the impact applied to the equipment existing therein.

A cavity may be formed in the interior of the casing to contain an image generating unit or the like therein. For example, the casing may have a rectangular parallelepiped shape with an empty interior, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like can be disposed.

When the image generating unit is disposed inside the casing, an opening may be formed in one area of the casing or a transparent material such as glass may be formed in one area of the casing to secure a view of the image generating unit. The image generating unit may image the periphery of the vessel and the harbor through the opening or the transparent area.

The casing may be provided with a strong material to protect the image generating unit and the like from external impact. Alternatively, the casing may be provided with a material such as an alloy for seawater in order to prevent corrosion due to salt.

The casing may include equipment for removing foreign substances from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove the foreign material. As another example, foreign substances may be removed by applying water or washer liquid through a liquid spray included in the casing, or the foreign substances may be physically removed using a wiper after application.

The debris removal equipment can be operated manually, but can also be operated automatically. For example, the foreign material removal equipment may operate at a predetermined time interval. Alternatively, the foreign material removal equipment may be operated using a sensor that detects whether a foreign material is attached to the image generating unit. Alternatively, after determining whether a foreign material is captured in the image by using the image captured by the image generating unit, the foreign material removal equipment may be operated when it is determined that the foreign material is present. Here, whether a foreign material is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as including two or more identical cameras.

The control module 200 may perform image analysis. In addition, the operation of receiving various data through the sensor module 100, the operation of outputting various outputs through the output device, the operation of storing various data in the memory or acquiring various data from the memory, etc. It can be done by control. Hereinafter, various operations or steps disclosed in the embodiments of the present specification may be interpreted as being performed by the control module 200 or being performed by the control of the control module 200 unless otherwise specified.

Examples of the control module 200 include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an on-demand system. There may be a semiconductor (Application Specific Integrated Circuit, ASIC), a Radio-Frequency Integrated Circuit (RFIC), and a combination thereof.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or unidirectional communication. For example, the device 10 may transmit information to an external output device through the communication module 300 to output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.

The sensor module 100 , the control module 200 , and the communication module 300 may include a control unit. The control unit may process and operate various types of information within the module, and may control other components constituting the module. The control unit may be provided in the form of an electronic circuit that physically processes an electrical signal. A module may physically include only a single control unit, or alternatively may include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be mounted on a physically separated server and a terminal and provided as processors that cooperate through communication. Examples of the control unit include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an application specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100 , the control module 200 , and the communication module 300 may include a communication unit. The modules may transmit and receive information through a communication unit. For example, the sensor module 100 may transmit information obtained from the outside through the communication unit, and the control module 200 may receive information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100 , the control module 200 , and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing the processing of the programs, or processing result data, and the like. For example, the memory may store data required for learning and/or inference, an artificial neural network in progress or learned, and the like. Memory includes non-volatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or any other tangible non-volatile recording medium. etc. can be implemented.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100 , and each sensor module 100 may include two cameras again.

3 and 4 are diagrams related to an embodiment of an eyepiece monitoring device according to an embodiment.

Referring to FIG. 3 , the monitoring device 10 may include a sensor module 100 and a control module 200 . The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 , and perform image analysis such as position/movement information estimation and image registration, which will be described later, through the control unit 220 . Also, the control module 200 may transmit an analysis result such as location/movement information and a matched image to the cloud server through the communication unit 210 . The cloud server may transmit the analysis result received from the control module 200 to a user terminal such as a smart phone, a tablet, a PC, or receive an instruction from the user terminal.

Referring to FIG. 4 , the monitoring device 10 may include a sensor module 100 . The sensor module 100 may generate an image through the camera 130 and transmit the image to the cloud server through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as position/movement information estimation and image matching, which will be described later. In addition, the cloud server may transmit the image analysis result to a user terminal such as a smartphone, tablet, or PC or receive an instruction from the user terminal.

The apparatus 10 shown in FIGS. 2 to 4 is merely an example, and the configuration of the apparatus 10 is not limited thereto.

As an example, the device 10 may include an output module 400 . The output module 400 may output a result of an operation performed by the control module 200 . For example, the output module 400 may output an analysis result. The output module 400 may be, for example, a display, a speaker, a signal output circuit, and the like, but is not limited thereto. In this case, information may be output through the output module 400 in addition to transmitting information to an external output device such as a user terminal and the external output device outputting information.

As another example, the device 10 may not include a sensor module. In this case, the control module 200 may receive information from an external sensor device and perform an image-based monitoring operation, such as performing image analysis. For example, the control module 200 may perform image analysis by receiving information from an AIS, a camera, a lidar, a radar, etc. already installed in a ship or a port.

In addition, the steps performed by each of the components of FIGS. 2 to 4 are not necessarily performed by the corresponding components and may be performed by other components. For example, although it has been described that the control unit 120 of the sensor module 100 performs viewpoint transformation in FIG. 3 above, the control unit 220 of the control module 200 or the cloud server may perform viewpoint transformation. .

Hereinafter, the monitoring apparatus 10 and the method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100 . For example, an image may be acquired through an image generating unit included in the sensor module 100 . Alternatively, as described above, an image may be acquired from an external sensor device. Images for vessel and port monitoring will generally include sea, vessel, buoy, obstacle, terrain, port, sky, building, and the like. Hereinafter, the analysis and monitoring of an image acquired through a visible light camera will be mainly described, but the present invention is not limited thereto.

A field of view and a depth of field may vary according to an image generating unit. 5 is a diagram illustrating a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 5 , a field of view (FOV) may mean to what extent an image is included in an image horizontally or vertically, and is generally expressed as an angle (degree, degree). A larger viewing angle may mean generating an image including areas having a larger width left and right or generating an image including areas having a larger width up and down. The depth of field may mean a distance range recognized as being in focus of the image, and a deep depth of field may mean that the distance range recognized as being in focus of the image is wide. Referring to FIG. 5 , according to the depth of field (DOF), an image may include an area A1 recognized as being in focus and an area A2 other than that. Hereinafter, an area included in the image is referred to as an imaging area (A1 + A2) and an area recognized as being in focus is referred to as an effective area (A1). Image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on , or based on a part of the imaging area, an area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and a shallow depth of field is a wide-angle camera. Examples of cameras with a small field of view and a deep depth of field include high magnification cameras and zoom cameras.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, a crane, a ship, etc. in a port, and there is also no limitation in the number. However, the installation location or number of the sensor module 100 may vary according to characteristics such as the type and performance of the sensor module 100 . For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more from the water surface for efficient monitoring, or a plurality of sensor modules 100 may be installed to have different imaging areas. In addition, the position and posture of the sensor module 100 may be manually or automatically adjusted during or after installation.

6 and 7 are views of an installation position of the sensor module 100 according to an embodiment. Referring to FIGS. 6 and 7 , the sensor module 100 may be installed at a fixed location, such as a port or on land, or may be installed on a moving object, such as a ship. Here, when the sensor module 100 is installed in a vessel, it may be installed in a vessel to be monitored as shown in FIG. 7 (hereinafter referred to as a “target vessel”), and as shown in FIG. It may be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition, the sensor module 100 may be installed on a drone or the like to monitor a target vessel.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or at a separate location.

As described above, image analysis for image-based monitoring may include acquiring object characteristics. Examples of the object may include a ship, port, buoy, sea, terrain, sky, building, person, animal, fire, smoke, and the like. Examples of object characteristics may include a type of object, a location of an object, a distance to the object, absolute and relative speed and speed of the object, and the like. another example,

Image analysis for image-based monitoring may include recognizing/judging a surrounding situation. For example, the image analysis may be to determine that a fire situation has occurred from an image of a fire in the port, or determine that an intruder has entered the port from an image captured by a person who entered the port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each of the modules 100 and 200 .

8 is a diagram related to image analysis according to an exemplary embodiment. Referring to FIG. 8 , image analysis may include object recognition ( S210 ) and location/movement information estimation ( S220 ).

The image analysis may include an object recognition step ( S210 ). The object recognition step S210 may include recognizing an object included in an image. For example, object recognition may be determining whether an object such as a ship, tugboat, sea, or harbor is included in the image. Furthermore, object recognition may be determining at which position in an image the object exists.

9 to 11 are diagrams of an object recognition step according to an exemplary embodiment.

9 is an image captured by a camera, and an object may be recognized as shown in FIG. 10 or 11 through the object recognition step.

Specifically, FIG. 10 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation. In this case, the object recognition step may mean a segmentation step. Through segmentation, a characteristic corresponding to a pixel on an image may be assigned or calculated from an image. It could be said that the pixel has been assigned a property or labeled. 9 and 10 , a segmentation image as shown in FIG. 10 may be obtained by performing segmentation based on an image captured by the camera of FIG. 9 . In FIG. 10 , the first pixel area P1 is the area on the image of the pixel corresponding to the ship, the second pixel area P2 is the sea, the third pixel area P3 is the quay wall of the harbor, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area P5 is the area on the image of the pixel corresponding to the sky.

Although FIG. 10 shows that information on the type of object corresponding to each pixel on the image is calculated by performing segmentation, information obtainable through segmentation is not limited thereto. For example, characteristics such as position, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, the different characteristics may be expressed independently or may be expressed by reflecting them at the same time.

Table 1 is a table related to labeling in which information on the type and distance of an object is simultaneously reflected, according to an embodiment. Referring to Table 1, a class may be set in consideration of information on the type and distance of an object, and an identification value may be assigned to each class. For example, the second identification value may be assigned in consideration of both topography, which is information about the type of object, and short distance, which is information about distance. Table 1 is an example of a case in which type information and distance information are considered together. In addition, other information such as direction information, obstacle movement direction, speed, and route mark may also be considered. In addition, not all identification values need to include a plurality of pieces of information, nor do they need to include the same type of information. For example, a specific identification value includes only information about the type (for example, identification value 1 does not include information about distance) and another identification value includes information about type and distance, etc. It can be expressed in various ways.

identification value class 0 sky and others One Sea 2 terrain + close range 3 Terrain + Medium 4 Terrain + Ranged 5 Fixed obstacle + close range 6 Fixed obstacle + medium range 7 Fixed Obstacle + Ranged 8 Dynamic obstacles + close range 9 Dynamic Obstacles + Medium Range 10 Dynamic Obstacle + Ranged

11 is a diagram showing a position of an object in an image with a bounding box, which is also referred to as detection. In this case, the object recognition step may mean a detection step. Compared with segmentation, detection can be viewed as detecting the position of an object in the form of a box, rather than calculating a characteristic for each pixel of an image. Referring to FIGS. 9 and 11 , a detection image as shown in FIG. 11 may be obtained by performing detection based on an image captured by the camera of FIG. 9 . 11 , it can be seen that a vessel is detected on the image and the position of the vessel is expressed as a rectangular bounding box BB. Although only one object is detected in FIG. 11 , two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks, and the final result may be calculated by combining the results.

An artificial neural network is a kind of algorithm modeled after the structure of a human neural network, and may include one or more nodes or one or more layers including neurons, and each node may be connected through a synapse. Data input to the artificial neural network (input data) may be output (output data) through a node through a synapse, and information may be obtained through this.

Types of artificial neural networks include a convolutional neural network (CNN) that extracts features using a filter, and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back as an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), relational network (RN), etc. can be applied and have limitations. it is not

Before using the artificial neural network, it is necessary to train it. Alternatively, training may be performed using an artificial neural network. Hereinafter, the step of learning the artificial neural network is expressed as a learning step and the step of using the artificial neural network as an inference step.

The artificial neural network may be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

12 and 13 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.

12 is an embodiment of the learning step of the artificial neural network, in which an untrained artificial neural network receives learning data or training data and outputs output data, and compares the output data with the labeling data. The artificial neural network can be trained through the backpropagation of the error. The training data, output data, and labeling data may be images. The labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or a program.

13 is an example of an inference step of an artificial neural network, and a learned artificial neural network may receive input data and output output data. Information that can be inferred in the inference step may vary according to information of the learning data in the learning step. Also, the accuracy of the output data may vary according to the learning degree of the artificial neural network.

The image analysis may include a position/movement information estimation step ( S220 ). The location/movement information estimation step ( S220 ) may include estimating information about the location and/or movement of at least some of the objects recognized in the object recognition step ( S210 ). Here, the position information may include an absolute position such as coordinates of an object, a relative position from a specific reference, a distance (distance from an arbitrary point, a distance range, etc.), a direction, etc., and the movement information includes an absolute velocity, a relative velocity, Information about the movement of the object, such as speed, may be included.

The position/movement information of the object may be used when the vessel is berthing or berthing. For example, it is possible to assist or guide the safe berthing or berthing of a vessel by using the distance to the berth or quay wall, the approach speed based on them, the distance from other vessels, and the relative speed when berthing or disembarking a vessel.

The position/movement information of the object may be used when the vessel is operating. For example, it can assist or guide the safe operation of a ship, such as detecting other ships or obstacles around the ship, warning of collisions using the distance to them, and their moving speed, or creating/recommending routes. . Alternatively, autonomous navigation may be performed based on this information.

The position/movement information of the object may be calculated based on the image. For example, location/movement information of a ship may be calculated based on an image including a ship, sea, and land as objects. Hereinafter, an object for estimating location/movement information is referred to as a target object. For example, in the above example, a ship may be a target object. Also, there may be a plurality of target objects. For example, when estimating a position or speed of each of a plurality of ships included in an image, the plurality of ships may be a target object.

The location/movement information of the object may be expressed in a plurality of categories having a certain range. For example, the distance information may be expressed in a short distance, a medium distance, and a long distance, and the direction information may be expressed in a left direction, a front direction, and a right direction. It may be possible to combine these to express the left near field, the right far field, and the like. The movement information may be expressed as a high speed, a low speed, or the like.

The position/movement information of the object may be expressed as an actual distance value, a direction value, a velocity value, and the like. For example, distance information may be expressed in units of meters (m), direction information may be expressed in units of degrees, and movement information may be expressed in units of cm/s.

The location/movement information of the object may be estimated based on an area or a point. For example, the distance between the ship and the quay wall may be estimated by calculating the distance between one point of the ship and one point of the quay wall, or may be estimated by calculating the shortest distance between one point of the ship and the quay wall. As another example, the distance between the vessels may be estimated by calculating a distance between a point of the first vessel and a point of the second vessel. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

The position/movement information of the object may be estimated based on image pixels. As described above, when estimating location/movement information based on a point, a point on an image may correspond to a pixel. Accordingly, the position/movement information of the object may be calculated based on the distance between image pixels.

Distance information between points may be calculated based on an interval between pixels. For example, a predetermined distance may be allocated to each pixel interval, and the distance between points may be calculated in proportion to the interval between pixels. As another example, the distance between pixels may be calculated based on the coordinate values of the pixels on the image, and the distance between points may be calculated based on this.

Movement information between points may be calculated based on a change in distance information between points. In this case, movement information may be calculated based on a plurality of images or image frames. For example, movement information between points may be calculated based on a distance between points in a previous frame, a distance between points in a current frame, and a time interval between frames.

14 and 15 are diagrams for estimating position/movement information of an object according to an embodiment.

Referring to FIG. 14 , the position/movement information estimation step includes position/movement information f1, f2 with the berthing wall OBJ2 or other vessels OBJ3, OBJ4 when the vessel OBJ1 is berthing or berthing. It may include estimating information f3, f4. As shown in FIG. 14 , the position/movement information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 may be calculated for two points of the vessel OBJ1 . In this case, the two points may correspond to a point where the ship OBJ1 is in contact with the sea. Further, the distance between the vessel OBJ1 and the quay wall OBJ2 may be the shortest distance between the two points and the quay wall OBJ2. The position/movement information f3 and f4 between the vessel OBJ1 and the other vessels OBJ3 and OBJ4 may be position/movement information between points corresponding to the bow or stern of the vessels OBJ1, OBJ3, OBJ4. . In this way, when the location/movement information is used to assist or guide the berthing or berthing of the vessel, it may be referred to as berthing guide information or berthing guide information.

Referring to FIG. 15 , the step of estimating position/movement information may include estimating position/movement information (f5, f6) with another vessel (OBJ6) or an obstacle (OBJ7) such as a buoy during operation of the vessel (OBJ5). have.

Port operation or management may be performed based on the data calculated in the location/movement information estimation step. For example, when a ship collides with a fender, the time to replace the insect repellent may be predicted by calculating the amount of impact from movement information such as the speed of the ship.

In the above, an embodiment of image analysis in which position/movement information is estimated after performing the object recognition step has been described. Alternatively, object recognition and location/movement information estimation may be performed in one step. For example, it is possible to recognize an object by performing segmentation or detection and at the same time estimate position/movement information of the object.

In the case of monitoring all the ships on the image, the amount of data calculation for acquiring the berthing guide information of the ship increases, so that the processing speed of the apparatus 10 may be reduced. In addition, by determining the target vessel to be monitored, obtaining the berthing guide information of the target vessel only (for example, bow distance, stern distance, etc.), and outputting the obtained berthing guide information, the user is You can be provided with the necessary information for berthing. To this end, the control module 200 may perform image analysis to determine a target vessel when a plurality of vessels are included in at least one image acquired using the image generating unit.

16 is a flowchart for determining a target vessel according to an embodiment.

Referring to FIG. 16 , the image-based monitoring may include a ship tracking step ( S213 ), a tracking information acquisition step ( S216 ), and a target ship determination step ( S219 ).

According to one embodiment, the control module 200 may track the vessel on the image (S213).

Tracking refers to the act of tracking temporally and spatially changing feature points (object, tracking target) on an image. For example, the tracking may include tracking the position, shape, movement, trajectory, information, and the like of a temporally and spatially changing feature point on an image. Also, the tracking may include continuing to preserve the image of the feature points on the image.

According to an embodiment, the control module 200 may perform tracking of the vessel based on the segmented image. Specifically, the control module 200 may track the vessel based on a region in which the class value of the pixel in the segmented image corresponds to the vessel. For example, the control module 200 may track a class value of segmented images (continuous) of a continuously captured image based on a pixel corresponding to a ship. Here, the control module 200 may track a representative point (pixel) of a vessel within a vessel area formed by a pixel having a class value corresponding to the vessel between successive frames.

17 is a view related to an example of tracking a ship according to an embodiment.

Referring to FIG. 17 , the control module 200 may track the vessels of the image by tracking representative points 141 , 142 , and 143 representing the vessel in the continuously photographed image. For example, the control module 200 may track one point on an area corresponding to each vessel of the segmentation image.

The representative points 141 , 142 , and 143 of the vessel may be any characteristic point, but since the segmented image has a single class value, the control module 200 exemplarily controls the point located in the center of the region corresponding to the vessel. (141) may be determined as a representative point.

Of course, the control module 200 determines the representative points 141, 142, and 143 of the vessel as points of other positions such as the highest point, the lowest point 142, the leftmost point, and the rightmost point 143 of the vessel area. In addition, the control module 200 may also track the outline of an area corresponding to the ship or the area itself instead of determining one pixel as a representative point of the ship and tracking it.

Various techniques can be used for tracking, for example, Kalman filter, Extended Kalman filter, Unscented Kalman filter, Particle filter, Information filter ), a histogram filter, and the like, and tracking at least one of them may be applied.

Of course, this is not necessarily the case, and the control module 200 can track the vessel based on images such as captured images, such as tracking the vessel using the background image and the difference image. Ships can be tracked.

In addition, the tracking step ( S213 ) may include tracking a vessel other than the tugboat among the vessels included in the image by the control module 200 . For example, the control module 200 may track a vessel other than a tugboat among vessels based on the segmentation image. Specifically, the control module 200 applies the input image and pixels corresponding to the objects including the sea, tugboat, and ship and features excluding the sea, tugboat, and tugboat included in the input image, respectively, to the sea, tugboat, ship and feature except for the tugboat. Image segmentation may be performed using an artificial neural network trained using a running set that labels class values indicating water, and ships except for tugs may be tracked. Here, the generated segmentation image may include pixels labeled to correspond to ships other than the tugboat, and the control module 200 may track pixels corresponding to the ships except the tugboat.

According to an embodiment, the control module 200 may obtain tracking information of a vessel to be tracked on the image (S216).

The tracking information may mean information about a vessel being tracked, and the tracking information may be used to determine a target vessel. For example, but not limited to, the tracking information may include location information of a ship, movement information of a ship, VTS information of a ship, AIS information of a ship, user input information, and the like.

According to an embodiment, obtaining the tracking information ( S216 ) may include obtaining, by the control module 200 , the location information of the vessel from the image. For example, the control module 200 may acquire location information of a representative point of a vessel tracked in the continuously photographed images. Here, the position information of the representative point of the ship may be information about a distance between the representative point and a quay wall, a distance between the representative point and another object, coordinates of the representative point on an image, and the like.

According to an embodiment, obtaining the tracking information ( S216 ) may include obtaining, by the control module 200 , movement information of the vessel from the image. For example, the control module 200 may acquire movement information of a representative point of a vessel tracked in the continuously photographed images. Here, the control module 200 may obtain movement information of the representative point based on the position information of the representative point. Here, the movement information of the representative point of the vessel may be information on the approach speed of the representative point to the quay wall, the approach speed of the representative point to another object, the speed change of the representative point, and the like.

According to an embodiment, obtaining the tracking information ( S216 ) may include obtaining, by the control module 200 , navigation information such as VTS information and AIS information of the tracked vessel. For example, the control module 200 may obtain at least one of VTS information and AIS information. Here, the control module 200 may acquire information about the destination of the ship according to the navigation information, the scheduled berthing time of the ship, and the like.

According to an embodiment, obtaining the tracking information ( S216 ) may include obtaining the user input information about the vessel being tracked by the control module 200 . For example, the control module 200 may obtain user input information used to determine the target vessel from the user terminal.

However, the acquisition of the tracking information need not be limited to being performed by the above-described method, and may be performed in various ways.

According to an embodiment, the control module 200 may determine a target vessel based on the obtained tracking information (S219).

According to an embodiment, the control module 200 may determine the target vessel based on the acquired position information of the vessel. For example, the control module 200 may determine the target vessel based on the distance between the vessel and the quay wall.

18 is a diagram illustrating an example of determining a target vessel based on location information according to an embodiment.

Referring to FIG. 18 , when there are a plurality of ships on an image, the control module 200 may determine a target ship to be monitored based on location information.

According to an embodiment, the control module 200 may determine the target vessel based on the distance from the quay wall of the vessel being tracked. Here, the distance from the quay wall of the tracked vessel may be a distance between a representative point of the tracked vessel and the quay wall.

For example, the control module 200 may determine, as the target vessel, a vessel having a distance from a quay wall less than a threshold distance among the vessels being tracked. Referring to FIG. 18 , the control module 200 may determine a vessel having a distance from a quay wall _{smaller than a preset critical distance d 0} among tracked vessels as a target vessel. The control module 200 may determine that the vessel 151 is not the target vessel when the vessel 151 determined as the target vessel moves and the distance from the quay wall is _greater than d 0 . Also, the control module 200 may determine the vessel 152 as the target vessel when the vessel 152 moves and the distance to the quay wall _{is located closer than d 0 .}

The control module 200 may variously set a threshold distance used for determining a target vessel. For example, the control module 200 may set the threshold distance based on the recognized size of the vessel. Specifically, the control module 200 may set the critical distance to be larger as the size of the vessel increases. In addition, the control module 200 may determine, as the target vessel, a vessel having a distance from a specific object (structure, etc.) other than a quay wall among vessels on the image smaller than a critical distance.

According to an embodiment, the control module 200 may determine the target vessel based on a location on the image of the tracked vessel. Here, the location on the image of the tracked ship may be the location on the image of the representative point of the tracked ship. For example, the control module 200 may determine, as the target vessel, a vessel located on a predetermined specific area of the image among vessels on the image. Referring to FIG. 18 , the control module 200 may determine, as a target vessel, a vessel 151 located in a predetermined square-shaped area 153 on the image among the vessels being tracked. The control module 200 may determine the vessel 152 as the target vessel when the vessel 152 moves and is located within the region 153 . Also, when the vessel 151 determined as the target vessel moves and is located outside the area 153 , the control module 200 may exclude the vessel 151 from the target vessel.

The control module 200 may set the area 153 in various ways. For example, the control module 200 may variously determine the size, shape, number, etc. of the regions 153 . Specifically, the control module 200 may determine the area 153 as an area around the path where the ship mainly enters from the berth where the image is captured, the area where the ship mainly docks, and the like. Also, conversely, the control module 200 may determine, as the target vessel, a vessel located in an area other than a predetermined specific region 153 among vessels on the image.

According to an embodiment, the control module 200 may determine the target vessel based on the acquired movement information of the vessel. For example, the control module 200 may determine the target vessel based on the speed at which the vessel approaches the quay wall.

19 is a diagram illustrating an example of determining a target vessel based on movement information according to an embodiment.

Referring to FIG. 19 , when there are a plurality of ships on an image, the control module 200 may determine a target ship based on movement information.

According to an embodiment, the control module 200 may determine the target vessel based on the speed of the tracked vessel. For example, the control module 200 may determine, as the target vessel, a vessel in which the magnitude of the velocity (absolute value of the velocity) among vessels on the image is greater than a threshold velocity. Referring to FIG. 16 , the control module 200 may determine, as a target vessel, a vessel 161 approaching at a speed _{v o} , which is a value greater than a threshold speed value, among the tracked vessels. The control module 200 may determine that the vessel 161 is not the target vessel when the vessel 161 determined as the target vessel is standing still in the berth after berthing is completed. Also, the control module 200 may determine the vessel 162 as the target vessel when the anchored vessel 162 moves and moves at a speed greater than the threshold speed (regardless of direction).

The control module 200 may variously set the size of the critical speed used for determining the target vessel. For example, the control module 200 may determine the threshold speed used for determining the target vessel based on the size of the vessel. Specifically, the control module 200 may determine that the critical speed increases as the size of the vessel increases.

In addition, the control module 200 may determine the target vessel based on conditions related to movement information, such as the direction, magnitude, and speed change (acceleration) of the speed among the tracked vessels. For example, the control module 200 may determine a vessel having a magnitude of acceleration (absolute value of acceleration) greater than a preset value among vessels on the image as the target vessel.

According to an embodiment, the control module 200 may determine a target vessel based on the acquired navigation information of the vessel. Here, the navigation information may include information related to the navigation of the vessel, and for example, the navigation information may be AIS information, VTS information, CITS information, or the like. The navigation information may include information such as a departure point, an arrival point, a mooring location, and a navigation route of the ship in relation to the navigation of the ship.

For example, the control module 200 may determine the target ship based on the destination according to the navigation information of the tracked ship.

20 is a diagram illustrating an example of determining a target vessel based on navigation information according to an embodiment.

Referring to FIG. 20 , when there are a plurality of ships on an image, the control module 200 may determine a target ship based on navigation information.

According to an embodiment, the control module 200 may determine the target vessel based on the destination according to the voyage information of the tracked vessel. For example, the control module 200 may determine, as a target ship, a ship whose destination according to the navigation information among ships on the image is a berth in which a camera that captures an image is installed. Referring to FIG. 17 , a target vessel may be determined to be a vessel 171 , which is a berth in which a camera that captures a corresponding image of a destination according to navigation information among the tracked vessels is installed. The control module 200 may determine that the vessel 171 is not the target vessel when the vessel 171 determined as the target vessel moves away or departs according to the navigation information. In addition, the control module 200 determines the vessel 172 as the target vessel when the vessel 172 is moored elsewhere and then docks with a berth in which a camera that captures a corresponding image is installed according to new navigation information. .

In addition, the control module 200 may determine the target vessel based on other conditions related to navigation information. For example, the control module 200 may determine, as the target vessel, a vessel recognized on the image at a time scheduled for berthing at the berth according to the navigation information.

According to an embodiment, the control module 200 may determine the target vessel based on the acquired user input information. The user input information may include information input and received from the user's device. For example, the user input information may include information input by the user related to selection of a vessel to be monitored, selection of a berth, and the like.

21 is a diagram illustrating an example of determining a target vessel based on user input information according to an embodiment.

Referring to FIG. 21 , when there are a plurality of ships on an image, the control module 200 may determine a target ship based on user input information. For example, the control module 200 may determine a vessel selected as a monitoring target according to user input information among the tracked vessels as the target vessel. Referring to FIG. 18 , the control module 200 may determine a vessel 181 selected from the indicators 193 displayed on the screen according to user input information among the tracked vessels as the target vessel. After the vessel 181 determined as the target vessel is berthed, the control module 200 may determine that the vessel 181 that is not selected according to the user input information is not the target vessel. Also, when the unselected vessel 182 is selected by the user according to user input information, the control module 200 may determine the vessel 182 as the target vessel.

Here, the user input information used to determine the target vessel may be provided in various ways. For example, the user input information may be information for determining a target vessel by selecting a vessel from a list of vessels entering and exiting a port. Also, the user input information may be information for determining a target vessel for a vessel at a position selected on an output port image (eg, an image output from a user terminal).

In addition, the control module 200 may determine an image of a berth to which monitoring information is output among images captured from a plurality of berths. For example, the control module 200 may output an image of a berth selected according to user input information among images captured from a plurality of berths and determine a vessel included in the image as a target vessel.

In addition, the step of determining the target vessel based on the obtained tracking information ( S216 ) may include determining, by the control module 200 , one target vessel on the image.

According to an embodiment, the control module 200 may determine, as the target vessel, one of a plurality of vessels meeting a condition determined as the target vessel on the image. For example, when there are a plurality of vessels that meet the condition determined as the target vessel on the image, the control module 200 may determine one vessel determined as the target vessel later as the target vessel. For example, referring to FIG. 18 , when the vessel 151 and the vessel 152 are sequentially located in the region 153 , the control module 200 initially determines that the vessel 151 is a target vessel and then determines that the vessel 152 is a target vessel. ) is located within the area 153 , then the target vessel may be determined as the vessel 152 . Conversely, the control module 200 may determine one vessel determined as the target vessel as the target vessel earlier than when there are a plurality of vessels meeting the condition determined as the target vessel on the image.

However, the determination of the target ship is not limited to being performed by the above-described method, and may be performed in various ways.

22 is a view related to obtaining information on the berthing guide of the target vessel according to an embodiment.

Referring to FIG. 22 , the control module 200 may acquire berthing guide information of the target vessel 191 among the vessels 191 and 192 being tracked. Here, the berthing guide information may include a bow distance, a stern distance, a bow speed, a stern speed, a distance between a target vessel and another vessel, a relative speed between the target vessel and another vessel, and the like. The bow distance may mean the distance the bow of the ship is away from the quay wall, and the stern distance may mean the distance the stern of the ship is separated from the quay wall. The bow speed may mean the speed at which the bow of the ship approaches the quay wall, and the stern speed may mean the speed at which the stern of the ship approaches the quay wall.

According to an embodiment, the control module 200 may obtain a distance between the target vessel 191 and the quay wall.

For example, the control module 200 may acquire the bow distance 193 that is the distance between the bow of the target ship 191 and the quay wall. Specifically, the control module 200 may determine one point corresponding to the bow of the target ship, and obtain a distance between the one point corresponding to the bow and the quay wall. As another example, the control module 200 may acquire the stern distance 194 that is the distance between the stern and the quay wall of the target vessel 191 . Specifically, the control module 200 may determine one point corresponding to the stern of the target ship, and obtain a distance between one point corresponding to the stern and the quay wall.

According to an embodiment, the control module 200 may extract a pair of points corresponding to both ends of the bottom surface of the vessel in contact with the sea level. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may acquire the distance between the quay wall and one of a pair of points corresponding to both ends of the bottom surface of the vessel in contact with the sea level in order to acquire the bow distance. In addition, the control module 200 may acquire the distance between the quay wall and the other one of a pair of points corresponding to both ends of the bottom surface of the vessel in contact with the sea level in order to acquire the stern distance.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target vessel 191 to obtain any one point or two or more points of the target vessel 191 instead of obtaining the bow and stern distances of the vessel. It is also possible to obtain the distance between the ship and the quay wall by extracting the points, or the distance between the ship and the quay wall may be obtained based on the outline of the area or the area itself.

According to an embodiment, the control module 200 may acquire the speed at which the target vessel 191 approaches the quay wall.

For example, the control module 200 may acquire the bow speed, which is the speed at which the bow of the target vessel 191 approaches the quay wall. Specifically, the control module 200 may determine one point corresponding to the bow of the target vessel 191 and acquire the speed at which one point corresponding to the bow approaches the quay wall. Here, the control module 200 may acquire the speed at which the bow of the target vessel 191 approaches the quay wall based on the bow distance 193 of the target vessel 191 . The control module 200 compares the bow distance 193 of the target vessel 191 in the current frame with the bow distance of the target vessel 191 in the subsequent frame, and the speed at which the bow of the target vessel 191 approaches the quay wall. can be obtained.

For another example, the control module 200 may acquire the stern velocity, which is the velocity at which the stern of the target vessel 191 approaches the quay wall. Specifically, the control module 200 may determine one point corresponding to the stern of the target ship 191 and acquire a speed at which one point corresponding to the stern approaches the quay wall. Here, the control module 200 may acquire the speed at which the stern of the target ship 191 approaches the quay wall based on the stern distance 194 of the target ship 191 . The control module 200 compares the stern distance 194 of the target vessel 191 in the current frame with the stern distance of the target vessel 191 in the subsequent frame, and the speed at which the stern of the target vessel 191 approaches the quay wall. can be obtained.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target vessel 191 to obtain the bow velocity and stern velocity of the vessel, instead of obtaining any one point or two or more points of the target vessel 191 . It is also possible to obtain the speed at which the vessel approaches the quay wall by extracting points, and it is ok to obtain the speed at which the vessel approaches the quay wall based on the outline of the area or the area itself.

According to an embodiment, the control module 200 may obtain a distance between the target vessel 191 and the other vessel 192 .

For example, the control module 200 may obtain the distance between the bow of the target vessel 191 and the other vessel 192 . Specifically, the control module 200 may determine one point corresponding to the bow of the target vessel 191 , and obtain a distance between one point corresponding to the bow and the other vessel 192 .

As another example, the control module 200 may obtain the distance between the stern of the target vessel 191 and the other vessel 192 . Specifically, the control module 200 may determine one point corresponding to the stern of the target ship 191 , and obtain a distance between one point corresponding to the stern and the quay wall.

According to an embodiment, the control module 200 may extract a pair of points corresponding to the front end and the stern end in the region corresponding to the vessel. Here, the control module 200 may determine the pair of points as one point corresponding to the bow and one point corresponding to the stern. For example, the control module 200 may obtain the distance between one of the pair of points corresponding to the fore end and the stern end in the area corresponding to the vessel and the other vessel in order to obtain the distance between the vessel and the other vessel. can Here, the control module 200 may acquire the distance between the other ships based on a point closer to the other ship among a pair of points corresponding to the front end and the aft end.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target vessel 191 to obtain a distance between the bow and stern of the vessel and another vessel, instead of obtaining any one point of the target vessel 191 . Alternatively, it is also possible to obtain the distance between the vessel and the other vessel by extracting two or more points, or the distance between the vessel and the other vessel may be obtained based on the outline of the region or the region itself.

According to an embodiment, the control module 200 may acquire the relative speed between the target vessel 191 and the other vessel 192 .

For example, the control module 200 may acquire the speed at which the bow of the target vessel 191 approaches the other vessel 192 . Specifically, the control module 200 may determine one point corresponding to the bow of the target vessel 191 , and acquire a speed at which one point corresponding to the bow approaches the other vessel 192 . Here, the control module 200 may acquire the speed at which the bow of the target vessel 191 approaches the other vessel 192 based on the distance between the bow of the target vessel 191 and the other vessel 192 . The control module 200 compares the distance between the bow of the target vessel 191 and the other vessel 192 in the current frame with the distance between the bow of the target vessel 191 and the other vessel 192 in the subsequent frame. The speed at which the bow of the target vessel 191 approaches the other vessel 192 may be acquired.

As another example, the control module 200 may acquire the speed at which the stern of the target vessel 191 approaches the other vessel 192 . Specifically, the control module 200 may determine one point corresponding to the stern of the target vessel 191 , and obtain a speed at which one point corresponding to the stern approaches the other vessel 192 . Here, the control module 200 may acquire a speed at which the stern of the target vessel 191 approaches the other vessel 192 based on the distance between the stern of the target vessel 191 and the other vessel 192 . The control module 200 compares the distance between the stern of the target vessel 191 and the other vessel 192 in the current frame with the distance between the stern of the target vessel 191 and the other vessel 192 in the subsequent frame. A speed at which the stern of the target vessel 191 approaches the other vessel 192 may be acquired.

Of course, the control module 200 extracts points corresponding to the bow and stern of the target vessel 191 to obtain the speed at which the bow and stern of the vessel approach the other vessel 192 , instead of obtaining an arbitrary position of the target vessel 191 . It is also possible to obtain the relative speed between the target vessel 191 and the other vessel 192 by extracting one point or two or more points of the target vessel 191 and the target vessel 191 based on the outline of the region or the region itself. It is okay to obtain the relative speed between the different ships (192).

23 is a flowchart of monitoring information output according to an embodiment.

Referring to FIG. 23, eyepiece monitoring may further include outputting monitoring information (S30). The information output in the monitoring information output step is not limited as long as it is information related to image-based monitoring, such as images of the vicinity of a ship, the ocean, or a port, and the type and distance/speed of objects included in the image.

The device 10 may visually output monitoring information. For example, the device 10 may output monitoring information through an output module such as a display. As another example, the device 10 may transmit monitoring information to the user's terminal through the communication module 300 and transmit a control signal to output the monitoring information through the display of the user terminal.

The monitoring information output step S30 may include displaying the image obtained in the image obtaining step using an image generating unit. In addition to this, it may include displaying various images related to image-based monitoring, such as an image that has undergone a pre-processing step, an image after segmentation or detection, and an image after viewpoint conversion.

In addition, the monitoring information output step ( S30 ) may include displaying the position/movement information estimated in the image analysis step by the control module 200 through an output module or a user terminal.

24 is a diagram illustrating an example of outputting monitoring information according to an embodiment.

Referring to FIG. 24 , the image and the eyepiece guide information may be displayed together. 24 , the displayed eyepiece guide information may include a bow distance, a bow speed, a stern distance, and a stern speed of the target vessel.

The monitoring information output step ( S30 ) may include providing information to the user in another way, such as the device 10 outputs sound or vibration in addition to a visual display. For example, the control module 200 emits a warning sound, such as when there is a risk that the target vessel collides with a quay wall, another vessel, obstacle, etc. It can output through an output module or a user terminal.

Image-based monitoring may include surveillance. Here, the surveillance may mean to provide the user with information about the occurrence of an emergency situation, such as a fire, and security-related information such as monitoring an intruder or monitoring an access of an unregistered vessel to a port.

Device 10 may monitor for intruders based on whether the image includes a person and when the image was taken. For example, the control module 200 may determine that an intruder is present when a person is included in the port image captured at the time when the operation is not in progress in the port.

The device 10 may monitor the vessel based on whether the image includes the vessel. For example, the control module 200 may monitor the vessel in a manner that provides information about the detection of a vessel not registered with the AIS to the user.

Also, the apparatus 10 may perform the surveillance by detecting a person or a ship based on an image through segmentation or detection.

25 to 27 are diagrams of another example of outputting monitoring information according to an embodiment.

25 and 26 , the image and the eyepiece guide information may be displayed together, and when there are a plurality of target vessels, a plurality of eyepiece guide information may be output.

25, the position at which the eyepiece guide information is output may correspond to the position of the target vessel on the image. For example, the berthing guide information of the left target vessel may be displayed on the left side, and the berthing guide information of the right target vessel may be displayed on the right side.

In addition, as shown in FIG. 26 , the eyepiece guide information may be output together with the identifier of the target vessel. For example, an arbitrary identifier of 'vessel 1' to the left target vessel on the image and 'ship 2' to the right target vessel on the image is assigned, respectively, and the assigned identifier and berthing guide information of each target vessel may be output together. . Also, the assigned identifier and eyepiece guide information may be displayed around the target vessel on the image.

Also, referring to FIG. 27 , the image and the berthing guide information may be displayed together, and when only one target vessel is determined when there are a plurality of vessels, only the berthing guide information of the target vessel may be output. For example, when the target vessel is determined as the left vessel in which berthing is in progress, only the berthing guide information of the left vessel determined as the target vessel may be output on the screen.

In the above, we looked at image-based monitoring based on a single image. In addition, image-based monitoring may be performed based on a plurality of images. When image analysis is performed based on a plurality of images, the total monitoring area of the image-based monitoring apparatus 10 may increase or monitoring accuracy may be improved.

28 is a diagram of image-based monitoring based on a plurality of images according to an exemplary embodiment.

Referring to FIG. 28 , the image acquisition step may include a first image acquisition step S11 and a second image acquisition step S12 , and the image analysis step S20 is obtained in the first image acquisition step S11 . Image analysis may be performed based on the first image and the second image acquired in the second image acquiring step ( S12 ).

After generating one image based on a plurality of images, image analysis may be performed. For example, the control module 200 may generate a panoramic image in which a plurality of images are matched, and perform image analysis of the panoramic image. Specifically, the control module 200 may generate a registered image or a fusion image by registering or fusing the first image and the second image, and may perform image analysis based on the generated registered image or the fusion image.

Alternatively, a final analysis result may be calculated based on a result of performing image analysis based on each of the plurality of images. For example, after image analysis is performed from the first image to obtain first monitoring information, and image analysis is performed to obtain second monitoring information from the second image, the final monitoring information is based on the first monitoring information and the second monitoring information. Monitoring information can be obtained.

As an example of a method of acquiring the final monitoring information from the plurality of monitoring information, there may be a method of calculating the final monitoring information by considering the plurality of monitoring information for each weight.

Alternatively, the final monitoring information may be calculated based on whether a plurality of pieces of monitoring information do not match each other or whether the difference is equal to or greater than a threshold equal to a specific value (hereinafter referred to as “error occurrence”). For example, based on the occurrence of an error, the final monitoring information is calculated by considering a plurality of monitoring information by weight, or the final monitoring information is calculated by giving priority to specific monitoring information among a plurality of monitoring information, or the specific monitoring information is used for other monitoring There may be a method of compensating with information or ignoring the corresponding monitoring information, but is not limited thereto.

The plurality of images may be images of the same type. For example, when two sensor modules include the same image generating unit, or two identical sensor modules including one image generating unit are disposed to perform image-based monitoring, the first image and the second image are of the same type can be

Monitoring areas of the plurality of images may be different from each other. For example, the first image may monitor a short distance from the image generating unit, and the second image may monitor a long distance. Alternatively, the first image may monitor the left side from the image generating unit, and the second image may monitor the right side.

Also, the device 10 may output monitoring information obtained by analyzing a plurality of images together with an appropriate image.

The device 10 may output monitoring information obtained by analyzing a plurality of images together with an image including a target vessel among a plurality of images. For example, the control module 200 performs image analysis of a plurality of images acquired by a plurality of cameras that capture images of at least partly different regions, and then selects the target vessel among the plurality of images through an output module or a user terminal. The first image taken may be output together with the first eyepiece guide information calculated from the first image.

29 and 30 are diagrams of another example of outputting monitoring information according to an embodiment.

29 and 30 , when a target vessel is determined after analyzing a plurality of images, the piercing guide information of the target vessel may be displayed together with an image including the target vessel among the plurality of images. For example, when the target vessel is determined as 'vessel 1', berthing guide information of 'vessel 1' may be displayed together with an image of a berth including 'vessel 1'. In addition, when the target vessel is determined to be 'Ship 3', the berthing guide information of 'Ship 3' may be displayed together with the image of the berth including 'Ship 3'.

The device 10 may output the monitoring information obtained by analyzing the plurality of images together with the destination image of the vessel among the plurality of images. For example, the control module 200 performs image analysis of a plurality of images acquired by a plurality of cameras that capture an image of a region where at least a part overlaps, and then, when the target vessel is taken together in the plurality of images, the plurality of images A control signal may be generated so that an image obtained using a camera installed at a destination according to the navigation information of the target vessel and the berthing guide information calculated from the image are output together.

31 and 32 are diagrams of another example of outputting monitoring information according to an embodiment.

31 and 32 show that the eyepiece guide information is displayed together with a first image and a second image that are images of respective berths adjacent to each other acquired by a plurality of cameras that capture images of an area at least partially overlapping.

31 and 32 , the vessel 252 is taken together on two images at the same time. When the destination of the vessel 252 is not the berth of the first image but the berth of the second image, it is preferable to output the berthing guide information of the vessel 252 passing toward another place with the first image rather than the second image may not 31, it is preferable that the first image is output along with the berthing guide information of the ship 251 where the berth of the first image is the destination according to the navigation information. In addition, as shown in FIG. 32, it is preferable that the berthing guide information of the ship 252 where the berth of the second image is the destination according to the voyage information is output together with the second image.

Of course, the monitoring information obtained by analyzing a plurality of images may be output together with an arbitrary image among the plurality of images, and is not limited to the above description.

The image-based monitoring may further include a viewpoint conversion step.

In general, an image generated by an image generating unit such as a camera may be displayed as a perspective view. Converting this to a top view (planar view), a side view, another perspective view, etc. may be referred to as view transformation. Of course, a top view or a side view image may be converted into another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, it may not be necessary to perform view point conversion.

33 and 34 are diagrams for view transformation according to an embodiment.

Referring to FIG. 33 , another perspective view image may be acquired through viewpoint transformation of the perspective view image. Here, viewpoint conversion may be performed so that the quay wall OBJ8 is positioned along a horizontal direction (left and right direction on the image) on the image. Referring to FIG. 34 , a top view image may be obtained through viewpoint transformation of a perspective viewpoint image. Here, the top view image may be a view looking down at the sea level in a direction perpendicular to the sea level. Also, as in FIG. 33 , viewpoint conversion may be performed so that the quay wall OBJ9 is positioned along the horizontal direction on the image.

After obtaining the image, after converting the viewpoint, image analysis can be performed to obtain information necessary for the berthing guide of the vessel. According to an example, the acquisition of the eyepiece guide information may be performed based on the segmented image in which the viewpoint is converted. Specifically, the viewpoint may be converted from the segmentation image of the perspective viewpoint into the top-view segmentation image, and the berthing guide information of the vessel may be obtained based on the region in which the class value of the pixel in the top-view segmented image corresponds to the vessel. Of course, the viewpoint-converted segmentation image may be a segmentation image in which the viewpoint is converted to various viewpoints, such as a side viewpoint, not a top view. Alternatively, the location/movement information of the vessel may be acquired based on the area itself.

Of course, this is not always the case, and it may be performed based on an image, such as a photographed image in which the viewpoint is converted.

In addition, monitoring information such as displaying an image to the user may be output after the viewpoint is changed after acquiring the image. In this case, information on the surrounding situation can be more easily provided to the user through viewpoint conversion.

The viewpoint transformation of the image may be performed in various ways.

As an example of the viewpoint transformation, inverse projection transformation (IPM) may be performed. A two-dimensional image is generated when light reflected from a subject in a three-dimensional space is incident on an image sensor through the lens of the camera, and the relationship between two dimensions and three dimensions depends on the image sensor and lens, for example, Equation 1 and can be expressed together.

Here, the matrix on the left side indicates two-dimensional image coordinates, the first matrix on the right side indicates intrinsic parameters, the second matrix indicates external parameters, and the third matrix indicates 3-dimensional coordinates. Specifically, fx and fy denote focal lengths, cx and cy denote principal points, and r and t denote rotation and translation transformation parameters, respectively.

By projecting a two-dimensional image onto an arbitrary plane in three dimensions through inverse projection transformation, the viewpoint can be changed. For example, a perspective view image may be converted into a top view image through inverse projection transformation, or may be converted into another perspective view image.

Internal parameters may be required for viewpoint transformation. As an example of a method for obtaining an internal parameter, the Zhang method may be used. The Zhang method is a type of polynomial model, and is a method of acquiring internal parameters by photographing a grid with a known size at various angles and distances.

Information on the position and/or posture of the image generating unit/sensor module capturing the image may be required for viewpoint conversion. Such information may be obtained from the position measuring unit and the posture measuring unit.

Alternatively, information on the position and/or posture may be acquired based on the position of the fixture included in the image. For example, at a first time point, the image generating unit may generate a first image including a target fixture that is disposed at a first position and/or a first posture and is a fixed object such as a terrain or a building. Thereafter, at a second time point, the image generating unit may generate a second image including the target fixture. Comparing the position of the target fixture on the first image and the position of the target fixture on the second image to calculate a second position and/or a second posture that is the position and/or posture of the image generating unit at the second time point can do.

In addition, the accuracy of image analysis may vary depending on the selection of the reference plane when changing the viewpoint. For example, when a perspective view image is converted into a top view image, the accuracy of image analysis based on the top view image may vary according to the height of the reference plane. In order to accurately calculate the distance between objects on the sea level, it may be preferable that the reference plane be the sea level when changing the viewpoint. Since the height of the sea level may change with time, it may be desirable to improve the accuracy of image analysis to perform viewpoint conversion in consideration of the height of the sea level.

The above-described viewpoint transformation method is merely an example, and viewpoint transformation may be performed in a different way. The viewpoint transformation information includes viewpoint transformation such as information on the matrix, parameters, coordinates, position and/or posture of Equation 1 above. contains the necessary information for

Image-based monitoring may include a pre-processing step. Preprocessing refers to all kinds of processing performed on images, including image normalization, image equalization, histogram equalization, image resizing, and upscaling to change the resolution/size of the image. and downscaling, cropping, noise removal, and the like. Here, the noise may include fog, rain, water droplets, sea clutter, fine dust, direct sunlight, salt, and combinations thereof. may include

Referring to normalization as an example of preprocessing, normalization may mean obtaining an average of RGB values of all pixels of an RGB image and subtracting it from the RGB image.

Referring to defogging as another example of preprocessing, defogging may mean converting an image of a foggy area to look like an image of a clear area through preprocessing. 13 is a view related to fog removal according to an embodiment. Referring to FIG. 13 , an image of a foggy area as shown in FIG. 13 (a) may be converted into an image in which fog is removed as shown in FIG. 13(b) through fog removal.

Looking at water drop removal as another example of pre-processing, water drop removal may mean converting water droplets on the front of the camera so that the water droplets appear to have been removed through pre-processing in the captured image.

According to one embodiment, after the image acquisition step (S10), the image analysis step (S20) may be performed through a pre-processing step. For example, image analysis may be performed after performing pre-processing on an image obtained by using the image generating unit. Image preprocessing may facilitate image analysis or improve accuracy.

Image preprocessing may be performed through an artificial neural network. For example, you can input an image of a foggy area into an artificial neural network to convert a clear area to look like a photographed image, etc. You can input an image containing noise into an artificial neural network to obtain an image with noise removed. can Examples of the artificial neural network include, but are not limited to, a GAN.

Alternatively, image preprocessing may be performed using an image mask. For example, by applying an image mask to an image of a foggy area, you can transform a clear area to look like a photographed image. Here, examples of the image mask include a deconvolution filter, a sharpen filter, and the like, and an image mask may be generated through an artificial neural network such as a GAN, but is not limited thereto.

In the above, the case of performing image analysis after image preprocessing has been discussed. Alternatively, image analysis including preprocessing may be performed. For example, when the image analysis step includes segmentation or detection, it may be implemented so that a result of performing segmentation or detection of an image including noise is equivalent to a result of performing segmentation or detection of an image that does not include noise.

35 is a flowchart of image-based monitoring according to an embodiment.

Referring to Figure 35, the eyepiece monitoring is image acquisition step (S1010), segmentation step (S1210), vessel tracking step (S1213), tracking information acquisition step (S1216), target vessel determination step (S1219), berthing guide information acquisition step (S1220) and monitoring information output step (S1030) may be included. Each step may be implemented as described above.

The device 10 may acquire an image including an object through an image acquisition step ( S1010 ). The device 10 may obtain a segmentation image by performing a segmentation step S1210 based on the image. The device 10 may track at least one vessel recognized based on the segmentation image by performing the vessel tracking step ( S1213 ). The device 10 may obtain tracking information by performing a tracking information acquisition step (S1216). Here, the tracking information may include at least one of location information of the vessel, movement information of the vessel, navigation information of the vessel, and user input information. The device 10 may determine the target vessel among the vessels to be tracked based on the obtained tracking information by performing the target vessel determining step ( S1219 ). The device 10 may perform the step of obtaining the eyepiece guide information ( S1220 ) to obtain the determined eyepiece guide information of the target vessel based on the segmentation image. Here, the eyepiece guide information may include at least one of a bow distance and a stern distance of the target vessel. The device 10 may perform the monitoring information output step (S1030) to output the eyepiece guide information together with the image.

The embodiment of FIG. 35 is merely an example, and image-based monitoring may be performed in a different way.

For example, some steps may not be performed in the embodiment of FIG. 35 . Referring to FIG. 35 , the step of outputting monitoring information ( S1030 ) may not be performed.

As another example, another step may be added in the embodiment of FIG. 35 , such as adding an image matching step or adding an image viewpoint conversion step.

As another example, some steps may be replaced with other steps in the embodiment of FIG. 35 , such as a segmentation step being replaced with a detection step.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In the above, the configuration and features of the present invention have been described with reference to the embodiments, but the present invention is not limited thereto, and it will be appreciated by those skilled in the art that various changes or modifications can be made within the spirit and scope of the present invention. It is evident that such changes or modifications are intended to fall within the scope of the appended claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication unit
220: control unit
300: communication module

Claims (1)

A method of eyepiece monitoring performed by computing means, comprising:
Acquiring an image of the harbor including the sea and a plurality of ships using a camera installed in the harbor to capture the image;
An artificial neural network learned using a learning set in which the input image and the pixels corresponding to the objects including the sea, the ship, and the feature included in the input image are labeled with class values indicating the sea, the ship, and the feature, respectively. generating segmentation images for the objects from the harbor image using;
acquiring tracking information including location information of representative points representing each of the plurality of ships by tracking the plurality of ships based on first pixels to which a class value indicating a ship of the segmented image is assigned;
determining a target vessel to be guided berthing among the vessels being tracked on the basis of the tracking information; and
obtaining berthing guide information including a bow distance that is a distance between a bow of the target ship and a pier and a stern distance that is a distance between the stern and a quay of the target ship based on the segmentation image; containing
How to monitor the berth.

Images