KR20220055555A - Method and device for monitoring harbor and ship - Google Patents

Abstract

The present invention relates to a method and device for monitoring a harbor and ships to measure a sea level. According to one embodiment of the present invention, the method comprises the following steps of: acquiring an image taken by a camera; acquiring LiDAR data including a plurality of LiDAR points acquired by a LiDAR sensor having a viewing angle at least partially overlapping with a viewing angle of the camera; detecting a first area corresponding to a sea in the image and a second area corresponding to a ship in the image; selecting first LiDAR points related to LiDAR beams reflected from the sea in consideration of positions of pixels included in the first area from the plurality of LiDAR points; calculating a first estimated sea level by using the first LiDAR points; selecting second LiDAR points related to the LiDAR beams reflected from the ship from the plurality of LiDAR points in consideration of positions of pixels included in the second area; determining third LiDAR points related to LiDAR beams reflected from the lower area of the ship in contact with the sea from the second LiDAR points on the basis of height values of the second LiDAR points; calculating a second estimated sea level by using the third LiDAR points; and determining the sea level in consideration of both the first estimated sea level and the second estimated sea level.

Description

Port and vessel monitoring method and apparatus

The present invention relates to a method and apparatus for monitoring a port and a vessel, and more particularly, to a method and apparatus for monitoring a port and a vessel for measuring sea level height.

Many accidents occur in the operation of ships and berthing and berthing in ports, and it is known that the main cause of the accidents is human negligence. Here, the negligence of navigation is mainly caused by the fact that the situation around the ship or in the port cannot be accurately monitored through the naked eye. Currently, various types of obstacle sensors are used to supplement this problem, but there are still limitations.

Accordingly, in recent years, a technology for monitoring the situation around a ship or in a port through an image has been developed, but monitoring through an inaccurate image such as an image with distortion in the image processing process may similarly cause an accident. The sea level can be an important factor to consider in the image processing process such as viewpoint conversion and registration of images.

Therefore, there is a need to develop a technology for actually measuring the sea level for accurate monitoring.

An object of the present application is to provide a method and apparatus for monitoring a port and a vessel for measuring the sea level by using lidar data related to a lidar beam reflected from the sea and a vessel.

One problem to be solved by the present application is to provide a method and apparatus for monitoring a port and a vessel for measuring sea level height by efficiently convergence data of a camera and lidar.

The problem to be solved of the present application is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to one embodiment, a method for monitoring a port comprising: acquiring an image captured by a camera; lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera; obtaining, detecting a first area corresponding to the sea in the image and a second area corresponding to a ship in the image, and determining the pixel position of a pixel included in the first area among the plurality of lidar points. Selecting first LIDAR points related to the LIDAR beam reflected from the sea in consideration, calculating a first estimated sea level height using the first LIDAR points, and pixels of pixels included in the second area selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of a location, and the second lidar point based on a height value of the second lidar point determining third LiDAR points associated with a lidar beam reflected from a lower region of the vessel in contact with the sea from A port monitoring method may be provided, including determining the sea level height in consideration of both the first estimated sea level height and the second estimated sea level height.

According to another embodiment, there is provided a method for monitoring a port, comprising: acquiring an image captured by a camera; Acquiring lidar data; detecting a first region corresponding to the sea in the image and a second region corresponding to a vessel in the image; selecting first lidar points associated with a lidar beam reflected from the sea in consideration of a pixel position; determining a first reliability of the first lidar points based on characteristics of the lidar points; selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in an area; based on the vessel information obtained from the image determining the second reliability of second lidar points, and calculating the sea level height using at least one of the first lidar points and the second lidar points in consideration of the first reliability and the second reliability A port monitoring method comprising the step of estimating may be provided.

According to another embodiment, there is provided a method for monitoring a port, comprising: acquiring lidar data including a plurality of lidar points obtained by a lidar sensor; related to a lidar beam reflected from the sea among the plurality of lidar points selecting first lidar points, selecting second lidar points associated with a lidar beam reflected from the vessel from among the plurality of lidar points, the first lidar points and the second lidar Determining the reliability of each of the first LiDAR points and the second LiDAR points for estimating the sea level height based on at least one of a number of points, a deviation of a height value, and a distance value; A port monitoring method comprising the step of estimating a sea level height using at least one of the first LiDAR points and the second LiDAR points in consideration of the reliability of the IDA points and the second LiDAR points can be provided.

According to another embodiment, there is provided a method for monitoring a port, comprising: acquiring an image captured by a camera; Acquiring lidar data, detecting a vessel region corresponding to the vessel in the image using an artificial neural network, wherein the artificial neural network includes a plurality of training images and object information labeled in pixels of the plurality of training images wherein at least some of the plurality of training images include a vessel and a sea, the object information reflects an object type, and pixels of the vessel indicate the vessel of the object type Selecting first LiDAR points related to the LiDAR beam reflected from the vessel from among the plurality of LiDAR points in consideration of pixel positions of pixels labeled with object information and included in the vessel area, the first determining second lidar points related to a lidar beam reflected from a lower region of the vessel in contact with the sea from the first lidar points based on a height value of the lidar point, and the second lidar point A port monitoring method including the step of estimating the sea level by using these may be provided.

According to another embodiment there is provided a port monitoring method, comprising: acquiring an image captured by a camera; a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera; Acquiring IDA data, detecting a ship area corresponding to a ship in the image, taking into account pixel positions of pixels included in the ship area, a LiDAR beam reflected from the ship among the plurality of LiDAR points selecting first LiDAR points related to, obtaining vessel information related to the vessel from the image, wherein the vessel information includes the detection of the vessel, the size of the vessel area, the distance to the vessel, and the vessel A port monitoring method comprising at least one of detection of related occlusion and estimating sea level height using the first lidar point when a preset condition is satisfied by the ship information may be provided. there is.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to an embodiment of the present application, the sea level can always be measured regardless of circumstances by using lidar data related to the lidar beam reflected from the sea and ship.

According to another embodiment of the present application, it is possible to accurately measure the sea level height by efficiently fusing the data of the camera and the lidar.

The effect of the present application is not limited to the above-described effects, and the effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings.

1 is a view of a port monitoring apparatus according to an embodiment.
2 and 3 are diagrams related to an embodiment of a monitoring device according to an embodiment.
4 is a view of a viewing angle and a depth of field according to an exemplary embodiment.
5 and 6 are views of an installation position of a sensor module according to an embodiment.
7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.
10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
12 is a view for explaining eyepiece guide information according to an embodiment.
13 is a view for explaining information on the berthing guide between the ship and the quay wall according to an embodiment.
14 is a view for explaining a method of obtaining eyepiece guide information according to an embodiment.
15 is a diagram of viewpoint transformation according to an embodiment.
16 is a flowchart of a method for calculating a sea level height according to an exemplary embodiment.
17 is an example of acquisition of sensor data according to an embodiment.
18 is a flowchart of a method for calculating a sea level height using a matched camera image and lidar data according to an embodiment.
19 is an example of registration of a camera image and lidar data according to an embodiment.
20 is a flowchart of a method for calculating sea level height using a lidar point related to a lidar beam reflected from the sea according to an exemplary embodiment.
21 is an example of lidar points associated with a lidar beam reflected from the sea according to an embodiment.
22 is a flowchart of a method for calculating sea level height using a lidar point related to a lidar beam reflected from a ship according to an exemplary embodiment.
23 is an example of lidar points associated with a lidar beam reflected from a vessel according to an embodiment.
24 is a flowchart of a method for calculating a sea level height using an estimated sea level height according to an exemplary embodiment.
25 is a flowchart of a method for calculating the sea level height using the reliability of a lidar point according to an embodiment.
26 is a flowchart of a method for calculating a sea level height using a result of analyzing a camera image according to an exemplary embodiment.
27 is a flowchart of a vessel monitoring method in consideration of the calculated sea level height according to an embodiment.
28 is a view related to viewpoint transformation in different reference planes/reference heights.
29 is a view related to image correction for calculating the eyepiece guide information on the sea level / sea level plane according to an embodiment.

The embodiments described herein are for clearly explaining the spirit of the present invention to those of ordinary skill in the art to which the present invention pertains, so the present invention is not limited by the embodiments described herein, and the present invention It should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as widely used general terms as possible in consideration of the functions in the present invention, but they may vary depending on the intention, custom, or emergence of new technology of those of ordinary skill in the art to which the present invention belongs. can However, if a specific term is defined and used in an arbitrary sense, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the terms and the contents of the entire specification, rather than the names of simple terms.

The drawings attached to this specification are for easily explaining the present invention, and the shapes shown in the drawings may be exaggerated as necessary to help understand the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

In the present specification, the sea level height should be broadly interpreted to include not only the absolute sea level height, but also the relative sea level height and the relative sea level height compared with the average sea level height of a specific sea area. For example, sea level height may vary with changes in sea level height, such as the distance between sea level and a quay wall, the distance between sea level and a monitoring device (e.g., an image generating unit), and the length of objects exposed above sea level. may include

Monitoring in the present specification refers to understanding or recognizing a surrounding situation, detecting a detection target such as a certain area or a specific object using various sensors and providing the detection result to the user through calculations based on the detection result, etc. It should be construed broadly to include, for example, providing additional information.

Image-based monitoring may mean identifying or recognizing a surrounding situation based on an image. For example, monitoring may mean acquiring information for calculating berthing guide information during berthing or berthing of a ship or recognizing other ships or obstacles therefrom by acquiring images around the ship when the ship is operating.

Berthing guide information is information about the environment, such as recognizing other ships or obstacles, understanding the port situation, whether the berth is accessible, the distance and speed from the quay wall, the distance and speed from other ships, and the presence of obstacles in the navigation route. can mean In this specification, although mainly described for monitoring when berthing is performed in a ship and a port, it is not limited thereto and may be applied to the case of driving of a vehicle, operation of an aircraft, and the like.

According to an aspect of the present invention, there is provided a method for monitoring a port comprising: acquiring an image captured by a camera; Acquiring lidar data; detecting a first region corresponding to the sea in the image and a second region corresponding to a vessel in the image; Selecting first LIDAR points related to the LIDAR beam reflected from the sea in consideration of the pixel position, calculating a first estimated sea level using the first LIDAR points, included in the second area selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels; determining third lidar points associated with a lidar beam reflected from a lower region of the vessel facing the sea from lidar points; calculating a second estimated sea level height using the third lidar points; and determining the sea level height in consideration of both the first estimated sea level height and the second estimated sea level height.

In an embodiment, the first estimated sea level height may be an average of height values of the first lidar points.

In an embodiment, the determining of the third lidar points includes generating lidar lines including lidar points of substantially the same height from the second lidar points, and among the lidar lines. The method may include selecting a lidar line having the lowest height value as the third lidar points.

In an embodiment, the lidar line may have the number and/or length of lidar points in a preset range.

In an embodiment, the sea level height may be a weighted sum of the first estimated sea level height and the second estimated sea level height.

In an embodiment, the first weight given to the first estimated sea level height is determined based on the distance value of the first LiDAR points, and the second weight given to the second estimated sea level height is the second It may be determined based on the distance value of the points.

In one embodiment, the port monitoring method further comprises calculating a first wave height from the first lidar points and calculating a second wave height from the second lidar points, wherein the first estimation A first weight given to the sea level height is determined based on a change in the first wave height per unit time, and a second weight assigned to the second estimated sea level height is determined based on a change in the second wave height per unit time. can

In an embodiment, the first region and the second region are detected using an artificial neural network, wherein the artificial neural network is a training set comprising a plurality of training images and object information labeled in pixels of the plurality of training images. can be learned using

In an embodiment, the first lidar points and/or the second lidar points are determined in consideration of characteristics of lidar points, and the characteristics of lidar points are the number of lidar points and deviation of height values. , may include at least one of a distance value.

In an embodiment, the second lidar points are determined based on vessel information obtained from the image, and the vessel information includes detection of the second region, a size of the second region, a distance to the vessel, and the and detecting at least one of occlusion associated with the vessel.

According to another aspect of the present invention there is provided a port monitoring method comprising: acquiring an image captured by a camera; acquiring lidar data, detecting a first region corresponding to the sea in the image and a second region corresponding to a vessel in the image, a pixel included in the first region among the plurality of lidar points selecting first lidar points related to a lidar beam reflected from the sea in consideration of the pixel position of selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in area 2; based on vessel information obtained from the image determining the second reliability of the second LiDAR points and using at least one of the first LiDAR points and the second LiDAR points in consideration of the first reliability and the second reliability to the sea level height A port monitoring method comprising the step of estimating may be provided.

In an embodiment, the characteristic of the lidar points may include at least one of the number of lidar points, a deviation of a height value, and a distance value.

In an embodiment, the vessel information may include at least one of detection of the second region, a size of the second region, a distance to the vessel, and detection of occlusion related to the vessel.

According to another aspect of the present invention, there is provided a port monitoring method, comprising: acquiring lidar data including a plurality of lidar points obtained by a lidar sensor; a lidar beam reflected from the sea among the plurality of lidar points; selecting first associated lidar points; selecting second lidar points associated with a lidar beam reflected from the vessel from among the plurality of lidar points; the first lidar points and the second lidar points; Determining the reliability of each of the first LiDAR points and the second LiDAR points for estimating the sea level height based on at least one of a number of IDA points, a deviation of a height value, and a distance value, and the first Port monitoring method comprising the step of estimating sea level height using at least one of the first LiDAR points and the second LiDAR points in consideration of the reliability of the LiDAR points and the second LiDAR points can be provided.

According to another aspect of the present invention there is provided a port monitoring method comprising: acquiring an image captured by a camera; Acquiring lidar data to: detecting a vessel region corresponding to the vessel in the image using an artificial neural network, wherein the artificial neural network includes a plurality of training images and object information labeled in pixels of the plurality of training images is learned using a training set comprising, at least some of the plurality of training images include a ship and a sea, the object information reflects an object type, and the pixels of the ship indicate a ship of the object type. Selecting first LiDAR points related to the LiDAR beam reflected from the vessel from among the plurality of LiDAR points in consideration of pixel positions of pixels labeled with the object information and included in the vessel area; determining second lidar points related to a lidar beam reflected from a lower region of the vessel in contact with the sea from the first lidar points based on a height value of one lidar point, and the second lidar A port monitoring method including estimating the sea level height using the points may be provided.

In one embodiment, the port monitoring method may further include verifying the sea level height based on at least one of the number of the first LiDAR points, a deviation of a height value, and a distance value.

According to another aspect of the present invention there is provided a port monitoring method comprising: acquiring an image captured by a camera; obtaining LiDAR data, detecting a ship area corresponding to the ship in the image, and taking into consideration pixel positions of pixels included in the ship area, the radar reflected from the ship among the plurality of LiDAR points. selecting first LiDAR points associated with the IDA beam, obtaining vessel information related to the vessel from the image, the vessel information comprising the detection of the vessel, the size of the vessel area, the distance to the vessel and the A port monitoring method comprising at least one of detection of occlusion related to a vessel and estimating a sea level height using the first lidar point when a preset condition is satisfied by the vessel information is provided can be

In one embodiment, the method for monitoring the port includes detecting a sea area corresponding to the sea from the image, and the LiDAR reflected from the sea in consideration of pixel positions of pixels included in the sea area among the plurality of LiDAR points. The method may further include selecting second LiDAR points related to the beam, and when the preset condition is not satisfied by the ship information, estimating the sea level height using the second LiDAR point. .

According to another aspect of the present invention, a computer-readable recording medium in which a program for performing the above-described methods is recorded may be provided.

Hereinafter, a vessel and port monitoring apparatus and method according to an embodiment will be described.

1 is a view of a port monitoring apparatus according to an embodiment.

Referring to FIG. 1 , the monitoring device 10 may include a sensor module 100 , a control module 200 , and a communication module 300 .

The sensor module 100 may sense information about a ship or a ship's vicinity and a port. The sensor module 100 may include an automatic identification system (AIS), an image generation unit, a LIDAR sensor, a position measurement unit, an attitude measurement unit, a casing, and the like.

The image generating unit may generate an image. The image generating unit may include a camera, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

The lidar sensor is a sensor for detecting a distance and a position of an object using a laser. For example, the distance between the lidar sensor and the object and the position of the object with respect to the lidar sensor may be represented by a three-dimensional coordinate system. For example, the distance between the lidar sensor and the object and the position of the object with respect to the lidar sensor may be expressed in a rectangular coordinate system, a spherical coordinate system, a cylindrical coordinate system, or the like. The lidar sensor may have a plurality of channels in a vertical or horizontal direction, and may be, for example, a lidar sensor having 32 or 64 channels.

The lidar sensor may use a laser reflected from the object to determine the distance R from the object. For example, the lidar sensor may use a time of flight (TOF), which is a time difference between an emitted laser and a detected laser, to determine the distance to the object. To this end, the lidar sensor may include a laser output unit for outputting a laser and a receiving unit for detecting the reflected laser. The lidar sensor determines the time the laser is output from the laser output unit, checks the time the receiver detects the laser reflected from the object, and determines the distance to the object based on the difference between the emitted time and the sensed time can do. Of course, the LiDAR sensor uses other methods such as triangulation based on the detected position of the detected laser to determine the distance (R) from the object, such as using a phase shift of the detected laser. R) may be determined.

The lidar sensor may determine the position of the object by using the angle of the irradiated laser. For example, when the irradiation angle of one laser irradiated from the lidar sensor toward the scan region of the lidar sensor is known, if a laser reflected from an object existing on the scan region is detected by the receiver, lidar The sensor may determine the position of the object based on the irradiation angle of the irradiated laser.

The lidar sensor may have a scan area including the object in order to detect the position of an arbitrary object in the vicinity. Here, the scan area represents a detectable area as one screen, and may mean a set of points, lines, and planes that form one screen during one frame. In addition, the scan area may mean the irradiation area of the laser irradiated from the lidar sensor, and the irradiation area may mean a set of points, lines, and surfaces where the laser irradiated during one frame meets a sphere at the same distance (R). can In addition, the field of view (FOV) means a detectable area, and may be defined as an angular range of the scan area when the lidar sensor is viewed as the origin.

The position measuring unit may measure the position of components included in the sensor module 100 , such as the sensor module 100 or the image generating unit. As an example, the location measurement unit may be a Global Positioning System (GPS). In particular, Real-Time Kinematic GPS (RTK-GPS) may be used to improve the accuracy of location measurement.

The position measuring unit may acquire position information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module 100 . For example, when the sensor module 100 is installed in a moving object such as a ship, the position measuring unit may acquire position information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a port, the position measurement unit may acquire position information at long time intervals. The time interval for obtaining the location information of the location measuring unit may be changed.

The posture measuring unit may measure the posture of components included in the sensor module 100 such as the sensor module 100 or the image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measuring unit may acquire posture information at predetermined time intervals. Here, the time interval may vary depending on the installation location of the sensor module 100 . For example, when the sensor module 100 is installed in a moving object such as a ship, the posture measuring unit may acquire posture information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a port, the posture measuring unit may acquire posture information at long time intervals. The time interval for acquiring the posture information of the posture measuring unit may be changed.

The casing may protect a sensor module such as an image generating unit, a position measuring unit, and a posture measuring unit.

At least one of an image generating unit, a position measuring unit, and a posture measuring unit may be present inside the casing. The casing may prevent equipment such as an image generating unit existing therein from being corroded by salt water. Alternatively, the casing may protect it by preventing or mitigating the impact applied to the equipment existing therein.

A cavity may be formed in the interior of the casing to contain an image generating unit or the like therein. For example, the casing may have a rectangular parallelepiped shape with an empty interior, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like can be disposed.

When the image generating unit is disposed inside the casing, an opening may be formed in one area of the casing or a transparent material such as glass may be formed in one area of the casing to secure a view of the image generating unit. The image generating unit may image the perimeter of the vessel and the harbor through the opening or transparent area.

The casing may be provided with a strong material to protect the image generating unit or the like from external impact. Alternatively, the casing may be provided with a material such as an alloy for seawater in order to prevent corrosion due to salt.

The casing may include equipment for removing foreign substances from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove foreign substances. As another example, foreign substances may be removed by applying water or washer liquid through a liquid spray included in the casing, or the foreign substances may be physically removed using a wiper after application.

The debris removal equipment can be operated manually, but can also be operated automatically. For example, the foreign material removal equipment may operate at a predetermined time interval. Alternatively, the foreign material removal device may be operated using a sensor that detects whether a foreign material has adhered to the image generating unit. Alternatively, after determining whether a foreign material is captured in the image by using the image captured by the image generating unit, the foreign material removal device may be operated when it is determined that the foreign material is present. Here, whether a foreign material is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as including two or more identical cameras.

The control module 200 may perform image analysis. In addition, the operation of receiving various data through the sensor module 100, the operation of outputting various outputs through the output device, the operation of storing various data in the memory or acquiring various data from the memory, etc. It can be done by control. Hereinafter, various operations or steps disclosed in the embodiments of the present specification may be interpreted as being performed by the control module 200 or being performed by the control of the control module 200 unless otherwise specified.

Examples of the control module 200 include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, an on-demand There may be a semiconductor (Application Specific Integrated Circuit, ASIC), a Radio-Frequency Integrated Circuit (RFIC), and a combination thereof.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or unidirectional communication. For example, the device 10 may transmit information to an external output device through the communication module 300 to output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.

The sensor module 100 , the control module 200 , and the communication module 300 may include a control unit. The control unit may process and calculate various types of information within the module, and may control other components constituting the module. The control unit may be provided in the form of an electronic circuit that physically processes an electrical signal. A module may include only a single control unit physically, or may include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be mounted on a physically separated server and a terminal and provided to processors that cooperate through communication. Examples of the control unit include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an application specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100 , the control module 200 , and the communication module 300 may include a communication unit. The modules may transmit/receive information through a communication unit. For example, the sensor module 100 may transmit information obtained from the outside through the communication unit, and the control module 200 may receive information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100 , the control module 200 , and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing processing of the programs, or data as a result of such processing. For example, the memory may store data required for learning and/or inference, an artificial neural network in progress or learned, and the like. Memory includes non-volatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or other tangible, non-volatile recording media. etc. can be implemented.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100 , and each sensor module 100 may include two cameras again.

2 and 3 are diagrams related to an embodiment of a monitoring device according to an embodiment.

Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100 and a control module 200 . The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 , and perform image analysis such as estimating position/movement information and image matching, which will be described later, through the control unit 220 . Also, the control module 200 may transmit an analysis result such as location/movement information and a matched image to the cloud server through the communication unit 210 . The cloud server may transmit the analysis result received from the control module 200 to a user terminal, such as a smartphone, tablet, or PC, or receive an instruction from the user terminal.

Referring to FIG. 3 , the monitoring device 10 may include a sensor module 100 . The sensor module 100 may generate an image through the camera 130 and transmit the image to the cloud server through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may convert the viewpoint of the image by performing viewpoint transformation, which will be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as position/movement information estimation and image registration, which will be described later. In addition, the cloud server may transmit the image analysis result to a user terminal such as a smartphone, tablet, or PC or receive an instruction from the user terminal.

The apparatus 10 illustrated in FIGS. 1 to 3 is merely an example, and the configuration of the apparatus 10 is not limited thereto.

As an example, the device 10 may include an output module 400 . The output module 400 may output a result of an operation performed by the control module 200 , and the like. For example, the output module 400 may output an analysis result. The output module 400 may be, for example, a display, a speaker, a signal output circuit, and the like, but is not limited thereto. In this case, information may be output through the output module 400 in addition to transmitting information to an external output device such as a user terminal and the external output device outputting information.

As another example, the device 10 may not include a sensor module. In this case, the control module 200 may receive information from an external sensor device and perform an image-based monitoring operation, such as performing image analysis. For example, the control module 200 may perform image analysis by receiving information from an AIS, a camera, a lidar, a radar, etc. already installed in a ship or a port.

In addition, the steps performed by each configuration of FIGS. 1 to 3 are not necessarily performed by the corresponding configuration and may be performed by other configurations. For example, in FIG. 2 above, although it has been described that the control unit 120 of the sensor module 100 performs viewpoint transformation, the control unit 220 of the control module 200 or the cloud server may perform viewpoint transformation. .

Hereinafter, the monitoring apparatus 10 and the method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100 . For example, an image may be acquired through an image generating unit included in the sensor module 100 . Alternatively, as described above, an image may be acquired from an external sensor device. Images for vessel and port monitoring will generally include sea, vessel, buoy, obstacle, terrain, port, sky, building, and the like. Hereinafter, the analysis and monitoring of an image acquired through a visible light camera will be mainly described, but the present invention is not limited thereto.

A field of view (FOV) and a depth of field may vary according to an image generating unit. 4 is a view of a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 4 , a field of view (FOV) may mean to what extent an image is included in an image horizontally or vertically, and is generally expressed as an angle (degree, degree). A larger viewing angle may mean generating an image including areas having a larger width left and right or generating an image including areas having a larger width up and down. The depth of field may mean a range of distances recognized as being in focus of the image, and the meaning of a deep depth of field may mean that the range of distances recognized as being in focus of the image is wide. Referring to FIG. 4 , according to the depth of field (DOF), the image may include an area A1 recognized as being in focus and an area A2 other than that. Hereinafter, an area included in the image is referred to as an imaging area (A1 + A2), and an area recognized as being in focus is referred to as an effective area (A1). Image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on , or based on a part of the imaging area, an area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and a shallow depth of field is a wide-angle camera. Examples of cameras with a small field of view and a deep depth of field include high magnification cameras and zoom cameras.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, a crane, a ship, etc. in a port, and there is no limitation in the number. However, the installation location or number of the sensor module 100 may vary according to characteristics such as the type and performance of the sensor module 100 . For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more above the water surface for efficient monitoring, or a plurality of sensor modules may be installed to have different imaging areas. In addition, the position and posture of the sensor module 100 may be manually or automatically adjusted during or after installation.

The viewing angle of the lidar may at least partially overlap with the viewing angle of the image generating unit (eg, a camera). For example, the field of view (FOV) of the lidar may be smaller than the field of view of the camera. As will be described later, in this case, lidar data outside the viewing angle of lidar for which lidar data is not acquired may be estimated using an image acquired from a camera.

5 and 6 are views of an installation position of a sensor module according to an embodiment. 5 and 6 , the sensor module 100 may be installed at a fixed location, such as a port or on land, or installed on a moving object, such as a ship. Here, when the sensor module 100 is installed in a vessel, it may be installed in a vessel to be monitored as shown in FIG. 6 (hereinafter referred to as a “target vessel”), and berthing or berthing of the target vessel as shown in FIG. 5 It may be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition, the sensor module 100 may be installed on a drone or the like to monitor a target vessel.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or at a location separate from it.

As described above, image analysis for image-based monitoring may include acquiring object characteristics. Examples of the object may include a ship, port, buoy, sea, terrain, sky, building, person, animal, fire, smoke, and the like. Examples of object characteristics may include the type of object, the position of the object, the distance to the object, absolute and relative speed and speed of the object, and the like.

Image analysis for image-based monitoring may include recognizing/judging a surrounding situation. For example, the image analysis may be to determine that a fire situation has occurred from an image of a fire in the port, or to determine that an intruder has entered the port from an image captured by a person entering the port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each of the modules 100 and 200 .

The device 10 may recognize an object. For example, the device 10 may recognize an object included in an image. For example, the device 10 may determine whether an object such as a ship, a tugboat, the sea, or a port is included in the image. Here, the object recognition may be determining at which position in the image the object exists.

7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.

7 is an image captured by a camera, and an object may be recognized as in FIG. 8 or 9 through the object recognition step.

Specifically, FIG. 8 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation. In this case, the object recognition step may mean a segmentation step. Through segmentation, a characteristic corresponding to a pixel on the image may be assigned or calculated from the image. It could be said that a characteristic has been assigned or labeled to a pixel. 7 and 8 , a segmentation image as shown in FIG. 8 may be obtained by performing segmentation based on an image captured by the camera of FIG. 7 . In FIG. 8 , the first pixel area P1 is the area on the image of the pixel corresponding to the ship, the second pixel area P2 is the sea, the third pixel area P3 is the quay wall of the harbor, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area P5 is an area on the image of a pixel corresponding to the sky.

Although FIG. 8 illustrates that information on the type of object corresponding to each pixel on an image is calculated by performing segmentation, information obtainable through segmentation is not limited thereto. For example, characteristics such as position, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, different characteristics may be expressed independently or may be expressed by reflecting them at the same time.

Table 1 is a table related to labeling in which information on the type and distance of an object is simultaneously reflected, according to an embodiment. Referring to Table 1, a class may be set in consideration of information on the type and distance of an object, and an identification value may be assigned to each class. For example, the second identification value may be assigned in consideration of both topography, which is information about the type of an object, and short distance, which is information about distance. Table 1 is an example in which type information and distance information are considered together. In addition, other information such as direction information, obstacle movement direction, speed, and route mark may also be considered. Also, not all identification values need to include a plurality of pieces of information, nor do they need to include the same type of information. For example, a specific identification value contains only information about the type (eg, identification value 1 does not include information about the distance) and another identification value contains information about the type and distance, etc. It can be expressed in various ways. For another example, other classes such as tugs, ropes, sides and decks of ships may be added or modified to other classes.

identification value class 0 sky and others One Sea 2 terrain + close range 3 Terrain + Medium 4 Terrain + Ranged 5 Fixed obstacle + close range 6 Fixed obstacle + medium range 7 Fixed Obstacle + Ranged 8 Dynamic obstacles + close range 9 Dynamic Obstacle + Medium Range 10 Dynamic Obstacle + Ranged

9 is a diagram showing a position of an object in an image with a bounding box, which is also referred to as detection. In this case, the object recognition step may mean a detection step. Compared with segmentation, detection can be viewed as detecting the position of an object in the form of a box, rather than calculating a characteristic for each pixel of an image. 7 and 9 , a detection image as shown in FIG. 9 may be obtained by performing detection based on an image captured by the camera of FIG. 7 . In FIG. 9 , it can be seen that a vessel is detected on the image and the position of the vessel is expressed as a rectangular bounding box BB. Although it is illustrated in FIG. 9 that only one object is detected, two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks, and the final result may be calculated by combining the results.

An artificial neural network is a kind of algorithm modeled after the structure of a human neural network, and may include one or more nodes or one or more layers including neurons, and each node may be connected through a synapse. Data input to the artificial neural network (input data) may be output (output data) through a node through a synapse, and information may be obtained through this.

Types of artificial neural networks include a convolutional neural network (CNN) that extracts features using a filter and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back as an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), relational network (RN), etc. can be applied and have limitations. it is not

Before using the artificial neural network, it is necessary to train it. Alternatively, training may be performed using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and the step of using the artificial neural network will be expressed as an inference step.

The artificial neural network may be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.

10 is an embodiment of the learning step of the artificial neural network, in which the untrained artificial neural network receives learning data or training data and outputs the output data, and compares the output data with the labeling data. The artificial neural network can be trained through the backpropagation of the error. The training data, output data, and labeling data may be images. The labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or a program.

11 is an example of an inference step of an artificial neural network, and a learned artificial neural network may receive input data and output output data. According to the information of the learning data in the learning stage, information that can be inferred in the inference stage may vary. Also, the accuracy of output data may vary according to the learning degree of the artificial neural network.

The recognition of the object is not limited to the above description and may be implemented in other ways. For example, although it has been described that the identification value is used for object recognition for convenience of description, the identification value is only used as one type of index. For example, a vector type index may be used for object recognition, and the training data, output data, and labeling data of the artificial neural network may be vector type data.

According to one embodiment, device 10 may provide berthing guide information for a vessel. The berthing guide information may be used in the berthing of a ship and may refer to information for assisting or guiding the berthing of a user such as a pilot or a captain. The eyepiece guide information may include information about distance/velocity. Examples of the distance/velocity information include, but are not limited to, an absolute position such as coordinates, a relative position from a specific reference, a distance from an arbitrary point, a distance range, a direction, an absolute velocity, a relative velocity, a velocity, and the like.

The distance/velocity information may be estimated based on a predetermined area or point. For example, the distance between the ship and the quay wall may be estimated by calculating the distance between one point of the ship and one point of the quay wall, or may be estimated by calculating the shortest distance between one point of the ship and the quay wall. As another example, the distance between the vessels may be estimated by calculating a distance between a point of the first vessel and a point of the second vessel. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

The distance/speed information may be expressed as a predetermined distance value, a direction value, a speed value, and the like. For example, distance information may be expressed as 1 m, 2 m, etc., direction information may be expressed as 10°, 20°, etc., and speed information may be expressed as 5 cm/s, 10 cm/s, or the like.

The distance/velocity information may be expressed as an index corresponding to a plurality of categories having a predetermined range. For example, distance information may be expressed in short-range, medium-distance, and long-distance, direction information may be expressed in a left direction, a front direction, and a right direction, and speed information may be expressed in low speed, medium speed, high speed, etc. there is. It may be possible to combine these to express the left near field, the right far field, and the like.

12 is a view for explaining eyepiece guide information according to an embodiment. Referring to Figure 12 (a), the berthing guide information is the berthing guide information between the vessel (OBJ1) and the quay wall (OBJ2) (f1, f2) and the vessel (OBJ1) and the other vessel (OBJ3, OBJ4) between the berthing guide It may include information f3 and f4. Specifically, the berthing guide information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 may include information about the distance/speed with the quay wall OBJ2 of the vessel OBJ1, and the vessel OBJ1 The berthing guide information (f3, f4) between the and other vessels (OBJ3, OBJ4) may include information about the distance/speed of the vessel (OBJ1) and the other vessels (OBJ3, OBJ4). Referring to FIG. 12 (b), the berthing guide information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 is an area corresponding to an area in contact with the sea level of the vessel OBJ1, a boundary region and a quay wall OBJ2 ) may be information between The boundary area may include not only a line or point where the ship and the sea level come into contact with each other, but also a predetermined area near the line or point. In addition, the berthing guide information between the vessel (OBJ1) and the other vessels (OBJ3, OBJ4) may be information between a predetermined area corresponding to the bow/stern of the vessel (OBJ1, OBJ3, OBJ4), where the predetermined area may mean not only the bow/stern point of the ships OBJ1, OBJ3, OBJ4, but also a certain area in the vicinity thereof.

The berthing guide information f1 and f2 between the vessel OBJ1 and the quay wall OBJ2 may include, but is not limited to, two eyepiece guide information as shown in FIG. 12, but one eyepiece guide information or three or more It may include berthing guide information. In addition, the berthing guide information f3, f4 between the vessel OBJ1 and the other vessels OBJ3, OBJ4 may include one berthing guide information between the two vessels as shown in FIG. 12, but is limited thereto No, it may include two or more berthing guide information between two ships. The above-described information may be referred to as eye guide information when used in the eye of the vessel.

13 is a view for explaining information on the berthing guide between the ship and the quay wall according to an embodiment.

The eyepiece guide information may be information on sea level/sea level. For example, the berthing guide information f5 may include information about the distance/speed between the vessel OBJ1 and the quay wall OBJ2 at sea level.

The eyepiece guide information may be information at the ground level (or the height of the quay wall). For example, the eyepiece guide information f6 may include information about the distance/velocity between the vessel OBJ1 and the quay wall OBJ2 at the height of the quay wall OBJ2 .

The eyepiece guide information may be information at a predetermined height. Here, the predetermined height may be between the height of the sea level and the height of the quay wall, as in the eyepiece guide information f7 shown in FIG. 13, but is not limited thereto.

The berthing guide information between the vessel and the quay wall may be information f8 between the fenders (fender, OBJ6) installed on the vessel (OBJ1) and the quay wall (OBJ2). Since the ship (OBJ1) collides with the insect repellent (OBJ6) when berthing or comes into contact with the insect repellent (OBJ6) when anchored, it is important to obtain information (f8) on the distance/speed between the ship (OBJ1) and the insect repellent (OBJ6). can be advantageous In FIG. 13, although the insect repellent (OBJ6) is shown to be installed at a position higher than sea level, at least a portion of the insect repellent (OBJ6) may be installed to be submerged in the sea (OBJ5), in this case the berthing guide information is at sea level. It may include information about the distance/speed between the vessel and the insect repellent.

The berthing guide information between the vessel and the other vessel may include information on the distance/speed between the vessel and the other vessel at sea level. The berthing guide information between the vessel and the other vessel may include information on the distance/speed between the vessel and the other vessel at a predetermined height. Here, the predetermined height may be determined in consideration of the shape of the hull. For example, the predetermined height may be the height of a region in which the vessel protrudes in the direction of another vessel. In this case, it may be advantageous to the berthing guide by more accurately grasping the possibility of collision between ships.

According to an embodiment, the device 10 may calculate the eyepiece guide information based on the lidar data. For example, the device 10 may calculate the distance/velocity to the object by using the three-dimensional coordinates of the lidar points included in the obtained lidar data. Examples of objects may include ships, seas and land, ports, quays, buoys, terrain, sky, buildings, people, animals, and the like.

According to an embodiment, the eyepiece guide information may be calculated based on the image. Here, the image may be an image generated by an image generating unit such as a camera or an image processed through image segmentation therefrom. For example, information on distance/velocity of a ship is calculated based on an image including a ship, sea, and land as objects, or information on distance/speed of a ship based on a segmentation image generated from the image through segmentation can be calculated. Examples of the object may include a port, a quay wall, a buoy, a terrain, a sky, a building, a person, an animal, etc. in addition to a ship, sea, and land.

Hereinafter, an object for estimating eyepiece guide information is referred to as a target object. For example, in the above example, a ship may be a target object. Also, there may be a plurality of target objects. For example, when estimating a distance or speed of each of a plurality of ships included in an image, the plurality of ships may be a target object.

According to an embodiment, the eyepiece guide information may be calculated based on image pixels. As described above, when the eyepiece guide information is calculated based on the point, the point on the image may correspond to a pixel. Accordingly, the eyepiece guide information may be calculated based on the spacing between image pixels.

Information on the distance between points may be calculated based on the distance between pixels. For example, a predetermined distance may be allocated to each pixel interval, and the distance between points may be calculated in proportion to the interval between pixels. As another example, the distance between pixels may be calculated based on the coordinate values of the pixels on the image, and the distance between points may be calculated based on this.

The information on the speed between points may be calculated based on a change in the information on the distance between the points. In this case, movement information may be calculated based on a plurality of images or image frames. For example, information on the speed between points may be calculated based on the distance between points in the previous frame, the distance between points in the current frame, and the time interval between frames.

14 is a view for explaining a method of obtaining eyepiece guide information according to an embodiment. The step of acquiring the eyepiece guide information is a step of acquiring an image generated by an image generating unit such as a camera (FIG. 14 (a)), and performing image segmentation on the image to perform image segmentation (or an image obtained by visualizing the segmentation image) generating ((b) of FIG. 14), finding a point for calculating eyepiece guide information based on the segmentation image ((c) of FIG. 14) and calculating eyepiece guide information corresponding to the point may include In FIG. 14 , a method for calculating eyepiece guide information from a segmentation image has been described, but this is only an embodiment, and without a step of generating a segmentation image, a point is found based on the image generated by the image generating unit and the eyepiece guide information may be calculated. There will be.

Acquiring the eyepiece guide information may include converting a viewpoint of the image. For example, the step of acquiring the eyepiece guide information may include, after the step of acquiring the image generated by the image generating unit, performing viewpoint transformation on the image, and performing segmentation on the viewpoint transformed image to obtain a segmented image. Or, after the step of generating the segmented image, by performing a viewpoint transformation on the segmented image, it is possible to calculate eyepiece guide information for the viewpoint-converted segmented image. Hereinafter, viewpoint transformation will be described.

In general, an image generated by an image generating unit such as a camera may be displayed as a perspective view. Converting this into a top view (planar view), a side view, another perspective view, etc. may be referred to as view transformation. Of course, a top view or a side view image may be converted into another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, it may not be necessary to perform view point conversion.

15 is a diagram of viewpoint transformation according to an embodiment. Referring to (a) of FIG. 15 , another perspective view image may be acquired through viewpoint transformation of the perspective view image. Here, viewpoint conversion may be performed so that the quay wall OBJ2 is positioned along a horizontal direction (left and right direction on the image) on the image. Referring to (b) of FIG. 15 , a top view image may be obtained through viewpoint transformation of a perspective viewpoint image. Here, the top view image may be a view looking down at the sea level in a direction perpendicular to the sea level. Also, as in (a) of FIG. 15 , viewpoint conversion may be performed so that the quay wall OBJ2 is positioned along the horizontal direction on the image.

It is possible to improve the ease, convenience, and accuracy when calculating the eyepiece guide information through viewpoint conversion. For example, in the case of pixel-based distance calculation, if a top-view image is used, a distance corresponding to an interval between pixels may be the same for the entire image or at least a partial region.

As an example of the viewpoint transformation, inverse projection transformation (IPM) may be performed. A two-dimensional image is generated when light reflected from a subject in a three-dimensional space is incident on an image sensor through the lens of the camera, and the relationship between two dimensions and three dimensions depends on the image sensor and lens, for example, Equation 1 and can be expressed together.

Here, the matrix on the left side means two-dimensional image coordinates, the first matrix on the right side means internal parameters, the second matrix means external parameters, and the third matrix means three-dimensional coordinates. Specifically, fx and fy denote focal lengths, cx and cy denote principal points, and r and t denote rotation and translation transformation parameters, respectively.

By projecting a two-dimensional image on an arbitrary plane in three dimensions through inverse projection transformation, the viewpoint can be changed. For example, a perspective view image may be converted into a top view image through inverse projection transformation, or may be converted into another perspective view image.

Internal parameters may be required for viewpoint transformation. As an example of a method for obtaining an internal parameter, the Zhang method may be used. The Zhang method is a type of polynomial model, and is a method of acquiring internal parameters by photographing a grid with a known size at various angles and distances.

Information on the position and/or posture of the image generating unit/sensor module capturing the image may be required for viewpoint transformation. Such information may be obtained from the position measuring unit and the posture measuring unit.

Alternatively, information on the position and/or posture may be acquired based on the position of the fixture included in the image. For example, at a first time point, the image generating unit may generate a first image including a target fixture that is disposed at a first position and/or a first posture and is a fixed object such as a terrain or a building. Thereafter, at a second time point, the image generating unit may generate a second image including the target fixture. Comparing the position of the target fixture on the first image and the position of the target fixture on the second image to calculate a second position and/or a second posture that is the position and/or posture of the image generating unit at the second time point can do.

Acquisition of information on a position and/or posture for viewpoint transformation may be performed at predetermined time intervals. Here, the time interval may depend on an installation position of the image generating unit/sensor module. For example, when the image generating unit/sensor module is installed in a moving object such as a ship, there may be a need to obtain information on the position and/or posture at short time intervals. On the other hand, when the image generating unit/sensor module is installed in a fixed body such as a port, information on the position and/or posture may be acquired at relatively long time intervals or may be initially acquired only once. When moving and fixing are repeated like a crane, it may be implemented in a manner that acquires information about a position and/or posture only after moving. In addition, the time interval for obtaining information on such a position and/or posture may be changed.

The device 10 may transform the image based on the reference plane. For example, the device 10 may view-convert an image from a plane in which a quay wall is located and parallel to the sea level to a reference plane. Here, the reference plane may depend on the calculated sea level height. Of course, the device 10 is not limited to the above description, and the view of the image may be converted into a reference plane on another plane such as sea level, a part of the ship other than the plane where the quay wall is located (for example, the reference plane of the deck height).

In addition, the above-described viewpoint transformation method is merely an example, and viewpoint transformation may be performed in a different way. The viewpoint transformation information includes information about the matrix, parameters, coordinates, position and/or posture of Equation 1 described above, such as viewpoint transformation. contains the necessary information for

Device 10 may calculate sea level height for vessel and/or port monitoring. For example, the device 10 may calculate the sea level height from the acquired sensor data, and use the calculated sea level height for monitoring of a ship or a port. For example, the device 10 may use the sea level height as height information up to the projection plane during viewpoint transformation or inverse projection transformation of the camera image. As another example, the device 10 may use the sea level height as a variable when converting an image for matching a camera image. As another example, the device 10 may use the sea level height as a data conversion variable when the lidar data and the camera image are fused.

16 is a flowchart of a method for calculating a sea level height according to an exemplary embodiment.

Referring to FIG. 16 , the method for calculating the sea level height according to an embodiment may include obtaining sensor data ( S1000 ) and calculating the sea level height ( S2000 ).

The device 10 may acquire sensor data (S1000).

According to an embodiment, the device 10 may obtain an image from a camera. For example, the device 10 may acquire an image from a camera installed on a berth toward the sea. For example, the device 10 may acquire an image of the sea, and if there is a ship, it may also acquire an image of the ship.

According to an embodiment, the device 10 may obtain lidar data from a lidar sensor. For example, the device 10 may acquire lidar data from a lidar sensor installed on a berth toward the sea. For example, the device 10 may acquire lidar data for the sea, and when the vessel enters the berth, it may also acquire lidar data for the vessel.

According to an embodiment, the lidar sensor may acquire lidar data for an area corresponding to an area captured by a camera acquiring an image. For example, the lidar sensor may have a field of view that at least partially overlaps with a field of view of a camera that acquires the image.

17 is an example of acquisition of sensor data according to an embodiment. Referring to FIG. 17 , the device 10 according to an embodiment may acquire sensor data for the same area from the lidar and the camera installed around the berth. For example, the lidar and the camera may be installed on the pier toward or looking at the berth to acquire data on an area where the berth is located. Here, the viewing angle of the lidar may at least partially overlap with the viewing angle of the camera.

Referring to FIG. 17( a ), the device 10 may acquire an image of the sea, and if there is a ship, it may also acquire an image of the ship.

Referring to FIG. 17(b) , the device 10 may acquire lidar data for the sea, and when the vessel enters the berth, it may also acquire lidar data for the vessel. The lidar data may include a plurality of lidar points captured by lidar sensors. For example, the lidar data may include a plurality of lidar points for each vertical or horizontal channel.

The device 10 may know the distance from the surface of the ship or the sea using lidar points corresponding to the vessel or the sea from the acquired lidar data, and may calculate the sea level height using this.

The device 10 may calculate the sea level height (S2000).

The device 10 may calculate the sea level height by using the acquired sensor data. For example, the device 10 may calculate the sea level height by using at least one of a camera installed on a berth toward the sea and sensor data obtained from a lidar sensor.

As an example, the device 10 may obtain a distance to the sea surface using the three-dimensional coordinates of the LiDAR points included in the obtained LiDAR data, and calculate the sea level height using this. The device 10 may calculate the height of the sea level in consideration of the distance from the sea surface, the installation position of the lidar sensor, the installation angle, and the like. As another example, the device 10 may obtain a distance between the sea surface and an area of the ship in contact with the sea surface by using the three-dimensional coordinates of the LiDAR points included in the obtained LiDAR data, and calculate the sea level height using this. The device 10 may calculate the height of the sea level in consideration of the distance between the sea surface and the area of the ship in contact with the installation position of the lidar sensor, the installation angle, and the like.

However, there is a problem in that it is difficult to distinguish which of the lidar points of the low-resolution lidar data is the sea region or the ship region because low-performance lidar sensors are often installed in the berth for cost reasons. To solve this, according to an embodiment, the device 10 may match the acquired camera image and lidar data, and calculate the sea level height using this.

18 is a flowchart of a method for calculating a sea level height using a matched camera image and lidar data according to an embodiment. Referring to FIG. 18 , the method for calculating the sea level height using the matched camera image and lidar data according to an embodiment uses the matching camera image and lidar data (S2010) and the matched camera image and lidar data. to calculate the sea level height (S2020) may be included.

The device 10 may match the camera image and lidar data (S2010).

According to an embodiment, the device 10 may match the camera image and the lidar data using the information for matching. For example, the device 10 may match the camera image and the lidar data by matching the coordinate system on the camera image and the coordinate system on the lidar data. That is, the coordinate system of the camera image and the lidar data may be converted to each other.

The device 10 may match the camera image and lidar data in consideration of the installation position of the camera, the installation angle of the camera, the installation position of the lidar sensor, the installation angle of the lidar sensor, and the like. Here, the device 10 may realign the camera image and lidar data by reflecting the calculated sea level height. As an example, when the lidar data and the camera image are matched, the device 10 may use the calculated sea level height as a data conversion variable to realign the camera image and the lidar data.

19 is an example of registration of a camera image and lidar data according to an embodiment. Referring to FIG. 19( a ), it can be seen that the lidar data acquired from the lidar sensor scanning the same area as the image acquired from the camera installed on the berth are matched.

According to one embodiment, the device 10 may match the segmented image with the lidar data. For example, the device 10 may generate a segmentation image from an image obtained from a camera using an artificial neural network, and may match the generated segmentation image with lidar data. Referring to FIG. 19(b), it can be seen that the lidar data acquired from the lidar sensor scanning the same area as the image acquired from the camera installed on the berth are matched.

The matching of the camera image and the lidar data is not limited to the above description, and may be implemented in other ways, such as matching the image detected using an artificial neural network with the lidar data.

It will be described again by returning to FIG. 18 .

The device 10 may calculate the sea level height using the matched camera image and lidar data (S2020).

According to an embodiment, the device 10 may calculate the sea level height by using a selected lidar point using an image matched with lidar data among a plurality of lidar points of lidar data. For example, the device 10 may calculate the sea level height using a lidar point matched to an area corresponding to the sea in the image. As another example, device 10 may calculate sea level height using lidar points matched to regions of the vessel in the image.

According to an embodiment, the device 10 may calculate the sea level height by using a lidar point related to a lidar beam reflected from the sea.

20 is a flowchart of a method for calculating sea level height using a lidar point related to a lidar beam reflected from the sea according to an exemplary embodiment. Referring to FIG. 20 , the method for calculating the sea level height using a lidar point related to a lidar beam reflected from the sea according to an embodiment includes the steps of selecting lidar points related to the lidar beam reflected from the sea (S2110) and the sea It may include calculating the sea level height from the lidar points related to the lidar beam reflected from (S2120).

The device 10 may select lidar points related to the lidar beam reflected from the sea (S2110).

According to an embodiment, the device 10 may select lidar points associated with a lidar beam reflected from the sea using the image matched with the lidar data.

To this end, the device 10 may detect an area corresponding to the sea in the image. For example, the device 10 may detect a region corresponding to the sea in the image using an artificial neural network. As an example, the device 10 may generate a segmented image from the image using an artificial neural network, and detect an area in which a pixel labeled with object information representing the sea of an object type is located as an area corresponding to the sea. The apparatus 10 is not limited to the above description, and may sense an area corresponding to the sea in the image in another method, such as determining an area corresponding to the sea through image detection.

The device 10 may use an area corresponding to the sea in the sensed image to select lidar points associated with a lidar beam reflected from the sea. For example, the device 10 may select LIDAR points associated with a LIDAR beam reflected from the sea in consideration of pixel positions of pixels included in an area corresponding to the sea in the image.

21 is an example of lidar points associated with a lidar beam reflected from the sea according to an embodiment. Referring to FIG. 21 , the device 10 may select lidar points 401 related to a lidar beam reflected from the sea from among lidar points matched with the image. For example, the device 10 may select LiDAR points matched to pixels included in an area corresponding to the sea in the image, among the plurality of LiDAR points, as the lidar points 401 associated with the lidar beam reflected from the sea. ) can be selected. Without being limited to FIG. 21 , the lidar points 401 associated with the lidar beam reflected from the sea may be selected in other ways where a segmented image or a detected image is used.

The selection of the lidar points 401 related to the lidar beam reflected from the sea is not limited to the above description and may be implemented in other ways. According to one embodiment, the device 10 may use the lidar data to select lidar points associated with the lidar beam 401 reflected from the sea. For example, the device 10 may select the lidar points 401 related to the lidar beam reflected from the sea in consideration of the distribution and number of lidar data.

It will be described again by returning to FIG. 20 .

The device 10 may calculate the sea level height from the lidar points related to the lidar beam reflected from the sea (S2120).

The device 10 may calculate the sea level height by using the three-dimensional coordinate values of the lidar points related to the lidar beam reflected from the sea. According to an embodiment, the device 10 may calculate the sea level height using height values (eg, Z values among 3D coordinate values) of lidar points related to the lidar beam reflected from the sea. For example, the device 10 may calculate an average value of height values of lidar points related to a lidar beam reflected from the sea, and calculate the average value of the calculated height values as sea level height.

According to an embodiment, the device 10 may calculate the wave height of the sea level by using the three-dimensional coordinate values of the lidar points related to the lidar beam reflected from the sea.

For example, the device 10 may calculate the wave height using height values (eg, Z values among 3D coordinate values) of lidar points related to a lidar beam reflected from the sea. As an example, the apparatus 10 may calculate the maximum and minimum height values of the lidar points related to the lidar beam reflected from the sea, and calculate the difference between the maximum value and the minimum value as a wave height.

Of course, the device 10 is not limited to the above description, and the sea level height is calculated in other ways, such as calculating the sea level height by reflecting the height value of the lidar points related to the lidar beam reflected from the sea at a preset ratio. is also free Here, the preset ratio may be determined in consideration of the location and number of each lidar point.

22 is a flowchart of a method for calculating sea level height using a lidar point related to a lidar beam reflected from a ship according to an exemplary embodiment. Referring to FIG. 22 , the method of calculating the sea level height using a lidar point related to a lidar beam reflected from a ship according to an embodiment includes selecting lidar points related to a lidar beam reflected from a ship ( S2210 ), a ship Determining lidar points related to the lidar beam reflected from the lower region of the vessel in contact with the sea among lidar points related to the reflected lidar beam from (S2220) and the lidar points reflected from the lower region of the vessel in contact with the sea (S2220) It may include calculating the sea level height from the LiDAR points related to the IDA beam ( S2230 ).

The device 10 may select lidar points related to the lidar beam reflected from the vessel (S2210).

According to one embodiment, the device 10 may use the image matched with the lidar data to select lidar points associated with the lidar beam reflected from the vessel.

To this end, the device 10 may detect an area corresponding to the vessel in the image. For example, the device 10 may detect a region corresponding to the vessel in the image using an artificial neural network. As an example, the device 10 may generate a segmented image from the image using an artificial neural network, and detect an area in which a pixel labeled with object information indicating a ship of an object type is located as an area corresponding to the ship. The apparatus 10 is not limited to the above description, and may detect an area corresponding to the vessel in the image in another method, such as determining an area corresponding to the vessel through image detection.

The device 10 may use an area corresponding to the vessel in the sensed image to select lidar points associated with the lidar beam reflected from the vessel. For example, the device 10 may select lidar points associated with a lidar beam reflected from the vessel in consideration of pixel positions of pixels included in an area corresponding to the vessel in the image.

23 is an example of lidar points associated with a lidar beam reflected from a vessel according to an embodiment. Referring to FIG. 23 , the device 10 may select lidar points 411 related to the lidar beam reflected from the vessel from among lidar points matched with the image. For example, the apparatus 10 may select lidar points 411 associated with a lidar beam reflected from the vessel that are matched to pixels included in an area corresponding to the vessel in the image among the plurality of lidar points. ) can be selected. Without being limited to FIG. 23 , the lidar points 411 associated with the lidar beam reflected from the vessel may be selected in another way where a segmented image or a detected image is used.

The selection of the lidar points 411 related to the lidar beam reflected from the vessel is not limited to the above description and may be implemented in other ways. According to an embodiment, the device 10 may use the lidar data to select lidar points 411 associated with the lidar beam reflected from the vessel. For example, the device 10 may select the lidar points 411 related to the lidar beam reflected from the ship in consideration of the distribution and number of lidar data.

The apparatus 10 may determine lidar points related to the lidar beam reflected from the lower region of the vessel in contact with the sea among lidar points related to the lidar beam reflected from the vessel ( S2220 ).

Referring to FIG. 23 , the device 10 relates to the lidar beam reflected from the lower region of the vessel in contact with the sea using the three-dimensional coordinate values of the lidar points 411 related to the lidar beam reflected from the vessel. Lidar points 412 may be determined. According to one embodiment, the device 10 is a vessel in contact with the sea based on a height value (eg, a Z value among three-dimensional coordinate values) of the lidar points 411 related to the lidar beam reflected from the vessel. It is possible to determine lidar points 412 associated with the lidar beam reflected from the lower region. For example, the device 10 generates lidar lines including lidar points having substantially the same height value, and generates lidar lines including lidar points having the lowest height value among the generated lidar lines. It can be determined by the lidar points related to the lidar beam reflected from the lower region of the vessel facing the sea. Here, the lidar lines may have the number or length of lidar points within a preset range.

For another example, the device 10 includes a lidar sensor having a plurality of channels having different vertical angles, and for each channel, a vessel abutting the lidar points including the lidar point of the lowest height value to the sea. It can be determined by the lidar points 412 related to the lidar beam reflected from the lower region of .

The determination of the lidar points 412 related to the lidar beam reflected from the lower region of the vessel facing the sea is not limited to the above description, and the height value of a preset range is not limited to the lidar point of the lowest height value. It may be implemented in a different way, such as determined by the lidar point having.

It will be described again by returning to FIG. 22 .

The device 10 may calculate the sea level height from the lidar points related to the lidar beam reflected from the lower region of the vessel in contact with the sea ( S2230 ).

The apparatus 10 may calculate the sea level height by using the three-dimensional coordinate values of the lidar points related to the lidar beam reflected from the lower region of the vessel in contact with the sea. According to an embodiment, the device 10 calculates the sea level height by using the height values (eg, Z values among 3D coordinate values) of lidar points related to the lidar beam reflected from the lower region of the ship in contact with the sea. can be calculated For example, the device 10 may calculate the height value of the lidar points related to the lidar beam reflected from the lower region of the vessel in contact with the sea as the sea level height.

According to an embodiment, the device 10 may calculate the wave height of the sea level by using the three-dimensional coordinate values of the lidar points related to the lidar beam reflected from the lower region of the vessel.

For example, the device 10 may calculate the wave height using height values (eg, Z values among 3D coordinate values) of lidar points related to the lidar beam reflected from the lower region of the ship. As an example, the apparatus 10 may estimate LiDAR points related to the LIDAR beam reflected from the lower region of the ship as a Gaussian distribution, and calculate a range of height values from the estimated distribution as a wave height.

According to an embodiment, the device 10 may verify the calculated sea level height. For example, the device 10 considers the characteristics (eg, number, deviation of height value, distance value, etc.) of lidar points related to the lidar beam reflected from the vessel in consideration of the lidar beam reflected from the vessel. The validity of the sea level height calculated from the lidar points can be checked. As an example, the device 10 may detect the radar reflected from the ship when the number of LiDAR points related to the reflected LiDAR beam from the vessel is too small, the deviation of the height value is too large, or the distance value is too close or far. It can be determined that the validity of the sea level height calculated from the LiDAR points related to the IDA beam is low.

Of course, the device 10 is not limited to the above description, and the height values of the lidar points related to the lidar beam reflected from the lower region of the ship in contact with the sea are adjusted to a preset condition to calculate the sea level height, etc. It is okay to calculate the sea level height in this way.

The accuracy of sea level height calculated from lidar points may vary depending on the situation. For example, the device 10 is generally easy to acquire lidar data from a wavy sea, but on a day when the weather is very good and the sea is very calm, the lidar data obtained from the sea is very small and noise is present. There is a possibility that lidar data may be obtained. That is, the accuracy of the sea level height calculated from the lidar points related to the lidar beam reflected from the sea may be lowered.

For another example, the device 10 is generally easy to obtain lidar data from a vessel that is at an appropriate distance from the lidar sensor. However, if the lidar sensor and the vessel are too close, the lower area of the vessel facing the sea is the berth. may be obscured by , and if the lidar sensor and the vessel are too far away, there is a possibility that lidar data with a large distance measurement error may be obtained. That is, the accuracy of the sea level height calculated from the lidar points related to the lidar beam reflected from the ship may be lowered.

Accordingly, the apparatus 10 may perform a sea level height calculation method to increase the accuracy of the calculated sea level height in various situations.

According to an embodiment, the device 10 may calculate the sea level height using the estimated sea level height.

24 is a flowchart of a method for calculating a sea level height using an estimated sea level height according to an exemplary embodiment. Referring to FIG. 24 , the method of calculating the sea level using the estimated sea level according to an embodiment includes calculating a first estimated sea level from LiDAR points related to a LiDAR beam reflected from the sea ( S2310 ), from a ship Calculating a second estimated sea level height from the lidar points related to the reflected LiDAR beam (S2320) and calculating the sea level height using at least one of the first estimated sea level height and the second estimated sea level height (S2330) may include

The apparatus 10 may calculate a first estimated sea level height from the lidar points related to the lidar beam reflected from the sea ( S2310 ). The device 10 may calculate the sea level height calculated from the lidar points related to the lidar beam reflected from the sea as the first estimated sea level height. In this step, since the above-described content may be applied as it is, a more detailed description will be omitted.

The device 10 may calculate a second estimated sea level height from the lidar points related to the lidar beam reflected from the vessel ( S2320 ). The device 10 may calculate the sea level height calculated from the lidar points related to the lidar beam reflected from the vessel as the second estimated sea level height. In this step, since the above-described content may be applied as it is, a more detailed description will be omitted.

The apparatus 10 may calculate the sea level height by using at least one of the first estimated sea level height and the second estimated sea level height ( S2330 ).

According to an embodiment, the device 10 may calculate the sea level height based on a weighted sum of the first estimated sea level height and the second estimated sea level height. For example, the device 10 may calculate the sea level height by a weighted sum of the first estimated sea level height to which the first weight is assigned and the second estimated sea level height to which the second weight is assigned.

According to an embodiment, the apparatus 10 may determine a weight given to the estimated sea level height based on the distance from the berth.

For example, the apparatus 10 weights the first estimated sea level height estimated from the lidar points related to the lidar beam reflected from the sea, the closer the distance to the berth of the lidar points, the higher the weights can be assigned. This is because the accuracy of the lidar point associated with the lidar beam reflected from the sea close to the berth is higher than the lidar point associated with the lidar beam reflected from the sea far from the berth.

For example, the apparatus 10 weights the second estimated sea level height estimated from lidar points related to the lidar beam reflected from the vessel, the greater the distance from the berth of the lidar points increases. weights can be assigned. Here, the apparatus 10 may give a higher weight as the distance from the berth increases up to a specific distance, and may give a lower weight as the distance from the berth increases over a specific distance. This is, when the lidar sensor and the vessel are too close, the lower area of the vessel in contact with the sea may be obscured by the berth, so there may be noise. This is because there is a possibility that data may be obtained.

Of course, the apparatus 10 is not limited to the above description, and may be weighted in other ways, such as by giving a high weight to the sea level height estimated from lidar points within a specific distance range.

According to an embodiment, the apparatus 10 may determine a weight given to the estimated sea level height based on the calculated wave height. For example, the device 10 may determine a weight given to the estimated sea level height based on a change in wave height per unit time. This is because, in general, the change in wave height per unit time is small.

For example, the apparatus 10 weights the first estimated sea level height estimated from the lidar points associated with the lidar beam reflected from the sea, wherein the lidar points associated with the lidar beam reflected from the sea are weighted. A higher weight can be given as the value of the wave height calculated from .

As another example, the apparatus 10 weights the second estimated sea level height estimated from lidar points associated with the lidar beam reflected from the vessel, in weighting the lidar point associated with the lidar beam reflected from the vessel. A higher weight can be given as the value of the wave height calculated from the waves decreases with time.

Of course, the device 10 is not limited to the above description, and weights may be assigned in other ways, such as adding weights by combining the above-described methods.

According to an embodiment, the device 10 may calculate the sea level height using the reliability of the lidar point.

25 is a flowchart of a method for calculating the sea level height using the reliability of a lidar point according to an embodiment. Referring to FIG. 25 , the method for calculating the sea level height using the reliability of the lidar point according to an embodiment includes calculating the first reliability of the lidar points related to the lidar beam reflected from the sea ( S2410 ), and the reflection from the ship It may include calculating the second reliability of the LiDAR points related to the obtained LiDAR beam (S2420) and calculating the sea level height in consideration of at least one of the first reliability and the second reliability (S2430). Reliability may refer to accuracy of whether a lidar point related to a lidar beam reflected from a specific object is actually generated by being reflected from the specific object.

The device 10 may calculate the first reliability of the lidar points related to the lidar beam reflected from the sea (S2410). For example, the device 10 may use the lidar data to calculate a first reliability of lidar points associated with a lidar beam reflected from the sea.

According to an embodiment, the device 10 may calculate a first reliability of the lidar points associated with the lidar beam reflected from the sea based on the characteristics of the lidar points. The characteristics of the lidar points may include, for example, the number of lidar points, a deviation of a height value, a distance value, coordinates, and the like. For example, the device 10 may calculate the first confidence low if the number of lidar points associated with the lidar beam reflected from the sea is too small. As another example, the apparatus 10 may calculate the first reliability to be small if the deviation of the height values of the lidar points related to the lidar beam reflected from the sea is too large. As another example, the apparatus 10 may calculate the first reliability to be small if the deviation of distance values of lidar points related to the lidar beam reflected from the sea is too large. The apparatus 10 is not limited to the description described above, and the first reliability may be calculated in another method, such as calculating the first reliability of the lidar points related to the lidar beam reflected from the sea using the camera image.

The device 10 may calculate the second reliability of the lidar points related to the lidar beam reflected from the vessel (S2420). For example, the device 10 may use the lidar data to calculate a second confidence level of lidar points associated with a lidar beam reflected from the vessel.

According to an embodiment, the apparatus 10 may calculate a second reliability of the lidar points associated with the lidar beam reflected from the vessel based on the characteristics of the lidar points. The characteristics of the lidar points may include, for example, the number of lidar points, a deviation of a height value, a distance value, coordinates, and the like. For example, the device 10 may calculate the second reliability to be small if the number of lidar points associated with the lidar beam reflected from the vessel is too small. As another example, the apparatus 10 may calculate the second reliability to be small if the deviation of the height values of the lidar points related to the lidar beam reflected from the ship is too large. As another example, the apparatus 10 may calculate the second reliability to be small if the deviation of distance values of lidar points related to the lidar beam reflected from the ship is too large.

According to an embodiment, the device 10 may calculate a second reliability of the lidar points associated with the lidar beam reflected from the vessel using the camera image. For example, the device 10 may calculate the second reliability of the lidar points related to the lidar beam reflected from the vessel by using the vessel information obtained from the camera image. Vessel information means information about a vessel, and for example, whether a vessel in the image is detected, information about an area corresponding to the vessel in the image (eg, size, shape, etc.), the distance to the vessel obtained from the image, It may include the speed of the vessel obtained from the image, occlusion related to the vessel, and the like.

As an example, the device 10 may greatly calculate a second reliability of lidar points associated with a lidar beam reflected from the vessel when the vessel is detected in the image. As another example, when the size of the region corresponding to the vessel in the image falls within a specific size range, the apparatus 10 may greatly calculate the second reliability of lidar points related to the lidar beam reflected from the vessel. As another example, when the distance to the vessel obtained from the image falls within a specific distance range, the apparatus 10 may greatly calculate the second reliability of the lidar points related to the lidar beam reflected from the vessel. As another example, when there is no occlusion associated with the vessel obtained from the image, the apparatus 10 may greatly calculate the second reliability of the lidar points associated with the lidar beam reflected from the vessel.

The apparatus 10 is not limited to the above description, and the second reliability may be calculated in other ways, such as calculating the second reliability by combining the above-described methods.

The device 10 may calculate the sea level height in consideration of at least one of the first reliability and the second reliability ( S2430 ).

According to an embodiment, the device 10 may calculate the sea level height by comparing the first reliability and the second reliability with each other. For example, the device 10 may calculate the sea level height from lidar points associated with a lidar beam reflected from the vessel when the first confidence level is less than the second confidence level. As another example, if the first confidence level is greater than the second confidence level, the device 10 may calculate the sea level height from lidar points associated with the lidar beam reflected from the sea.

According to an embodiment, the device 10 may calculate the sea level height by comparing the threshold reliability with at least one of the first reliability and the second reliability. For example, if the first confidence level is equal to or greater than the threshold confidence level, the device 10 may calculate the sea level height from lidar points associated with a lidar beam reflected from the sea. For another example, if the second reliability is greater than or equal to the threshold reliability, the apparatus 10 may calculate the sea level height from the lidar points associated with the lidar beam reflected from the vessel.

The apparatus 10 is not limited to the above description, and calculates the sea level height by reflecting the sea level heights calculated from the lidar points related to the lidar beam reflected from the sea and the ship at the ratio of the first reliability and the second reliability. It is okay to calculate the sea level in other ways, such as

According to an embodiment, the device 10 may calculate the sea level height by using a result of analyzing the camera image. For example, the device 10 may recognize a situation through a camera image, and calculate the sea level height using lidar data suitable for the recognized situation.

26 is a flowchart of a method for calculating a sea level height using a result of analyzing a camera image according to an exemplary embodiment. Referring to FIG. 26 , the method for calculating the sea level height using the result of analyzing the camera image according to an embodiment includes the steps of analyzing the camera image ( S2510 ) and calculating the sea level height from the lidar data using the image analysis result ( S2510 ) (S2520) may be included.

The device 10 may analyze the camera image (S2510). For example, the device 10 may analyze the camera image and determine the existence of a vessel.

According to an embodiment, the device 10 may obtain vessel information related to the vessel from the camera image. Vessel information means information about a vessel, and for example, whether a vessel in the image is detected, information about an area corresponding to the vessel in the image (eg, size, shape, etc.), the distance to the vessel obtained from the image, It may include the speed of the vessel obtained from the image, occlusion related to the vessel, and the like. For example, the device 10 may obtain vessel information in an image using an artificial neural network. For example, the device 10 generates a segmented image from an image using an artificial neural network, and detects an area in which a pixel labeled with object information indicating a ship of an object type is located as an area corresponding to the ship, and from this, ship information can be obtained. Here, the vessel information includes whether a vessel is detected, information about an area corresponding to the vessel in the image (eg, size, shape, etc.), the distance to the vessel obtained from the image, the speed of the vessel obtained from the image, and the vessel-related It may include occlusion and the like. Since the above-described content may be applied to the acquisition of ship information, more detailed content will be omitted.

The device 10 is not limited to the above description, and may acquire vessel information from the image in another method, such as determining an area corresponding to the vessel through image detection.

The device 10 may calculate the sea level height from the lidar data using the image analysis result (S2520). For example, device 10 may calculate sea level height from different lidar points depending on the presence or absence of a vessel. As an example, the device 10 calculates the sea level height from the lidar point related to the lidar beam reflected from the vessel when there is a vessel according to the image analysis result, and from the sea when the vessel does not exist according to the image analysis result The sea level height can be calculated from the lidar point relative to the reflected lidar beam.

According to an embodiment, the device 10 may calculate the sea level height from the lidar data by using the ship information obtained from the camera image. For example, the device 10 may calculate the sea level height from a specific lidar point based on vessel information obtained from a camera image.

According to an embodiment, the device 10 may calculate the sea level height from the lidar data based on whether a preset condition is satisfied by the ship information obtained from the camera image. For example, the device 10 may calculate the sea level height from a specific lidar point according to whether a preset condition is satisfied.

For example, when a predetermined condition is satisfied by the vessel information, the device 10 may calculate the sea level height from the lidar point related to the lidar beam reflected from the vessel. Here, the preset condition includes a condition in which a vessel is detected, a condition in which the size of an area corresponding to the vessel belongs to a specific range, a condition in which the distance to the vessel belongs to a specific range, a condition in which occlusion related to the vessel is not detected, etc. can do. That is, the device 10 calculates the sea level height from the lidar point related to the lidar beam reflected from the vessel when the vessel is detected from the camera image or when the detected vessel meets a specific condition that guarantees a certain level of accuracy. can do.

As another example, when a condition preset by the vessel information is not satisfied, the device 10 may calculate the sea level height from a lidar point related to a lidar beam reflected from the sea. Here, the preset conditions include a condition in which a vessel is not detected, a condition in which the size of an area corresponding to the vessel does not belong to a specific range, a condition in which the distance to the vessel does not belong to a specific range, a condition in which occlusion related to a vessel is detected, etc. may include That is, the device 10 determines the sea level height from the lidar point associated with the lidar beam reflected from the sea if the vessel is not detected from the camera image, or the detected vessel does not meet certain conditions that guarantee a certain level of accuracy. can be calculated.

In addition, the device 10 may calculate the sea level height from the lidar data in consideration of the vessel information obtained from the subsequent image. For example, the device 10 calculates the sea level height from the lidar point associated with the lidar beam reflected from the vessel in which the preset condition is satisfied, and the preset condition is not satisfied by the vessel information obtained from the subsequent image. In this case, the calculated sea level may be maintained as the sea level. For another example, the device 10 calculates the sea level height from the lidar point related to the lidar beam reflected from the vessel when the preset condition is satisfied, and the preset condition is also satisfied by the vessel information obtained from the subsequent image. In this case, the calculated sea level height may be updated in consideration of the sea level height calculated from subsequent LiDAR data. Here, the subsequent image and the subsequent LiDAR data may correspond to each other, for example, acquired at the same time point.

The method of calculating the sea level height from the lidar data using the image analysis result is not limited to the above description, and may be implemented in other ways, such as adding the shape of a vessel detected to a preset condition.

According to an embodiment, the device 10 may monitor a ship or a port in consideration of the calculated sea level height.

27 is a flowchart of a vessel monitoring method in consideration of the calculated sea level height according to an embodiment. Referring to FIG. 27 , the method for monitoring a vessel in consideration of the calculated sea level according to an embodiment may include calculating the sea level (S2000) and monitoring the vessel in consideration of the calculated sea level (S3000). .

The device 10 may calculate the sea level height (S2000). In this step, since the above-described content may be applied as it is, a more detailed description will be omitted.

The device 10 may monitor the vessel in consideration of the calculated sea level (S3000).

According to an embodiment, the device 10 may provide the eyepiece guide information calculated in consideration of the sea level height information that reflects the height of the sea level.

28 is a view of viewpoint transformation at different reference planes/reference heights, and even if the same image is the same, images converted according to the viewpoint transformation criterion may be different. For example, the images on the left in (b) and (c) of Figs. 28 are the same, but the images converted according to the criteria at the time of viewpoint conversion are the same as the images on the right side of Figs. 28 (b) and (c) respectively. The relative position of may appear to be different. In this case, since the berthing guide information of the target vessel is also different, setting of a reference plane/reference height may be important in order to calculate accurate berthing guide information, and this reference plane/reference height may depend on the sea level. Therefore, if the viewpoint is converted in consideration of the sea level information, the accuracy of the berthing guide information may be increased.

28 , the reference plane may be a plane parallel to the sea level, but is not limited thereto.

The device 10 may perform viewpoint transformation by updating viewpoint transformation information according to sea level height information whenever the piercing guide information is calculated in consideration of sea level height. Alternatively, the viewpoint conversion information may be updated by reflecting the sea level height according to a predetermined time interval. Here, it is not necessary to update all parameters of the view transformation information, but only some parameters among them may be updated.

29 is a view related to image correction for calculating the eyepiece guide information on the sea level height / sea level plane according to an embodiment, the viewpoint conversion of the segmentation image with respect to the reference plane / reference height, and the converted segmentation image sea level height information It is to calculate the eyepiece guide information after correction in consideration. As the difference between the sea level height and the reference height increases, the difference between the images before and after correction may be large.

However, monitoring of a ship or port considering sea level is not limited to the above description, and for example, a viewpoint conversion step is performed in consideration of sea level information, berthing guide information is calculated from this, and sea level height information is considered again. It may be implemented in other ways, such as correcting the eyepiece guide information.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication department
220: control unit
300: communication module

Claims (19)

A port monitoring method comprising:
acquiring an image captured by a camera;
acquiring lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera;
detecting a first area corresponding to the sea in the image and a second area corresponding to the vessel in the image;
selecting first LiDAR points related to a LiDAR beam reflected from the sea in consideration of a pixel position of a pixel included in the first area from among the plurality of LiDAR points;
calculating a first estimated sea level height using the first lidar points;
selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the second area;
determining third lidar points related to the lidar beam reflected from the lower region of the vessel in contact with the sea from the second lidar points based on the height value of the second lidar point;
calculating a second estimated sea level height using the third lidar points; and
determining the sea level height in consideration of both the first estimated sea level height and the second estimated sea level height;
How to monitor ports.
According to claim 1,
The first estimated sea level height is an average of the height values of the first LiDAR points.
How to monitor ports.
According to claim 1,
The step of determining the third LiDAR points includes:
generating lidar lines including lidar points having substantially the same height from the second lidar points; and
selecting a lidar line having a lowest height value among the lidar lines as the third lidar points;
How to monitor ports.
4. The method of claim 3,
The lidar line has a number and/or length of lidar points in a preset range.
How to monitor ports.
According to claim 1,
The sea level height is a weighted sum of the first estimated sea level height and the second estimated sea level height
How to monitor ports.
6. The method of claim 5,
A first weight given to the first estimated sea level is determined based on a distance value of the first LiDAR points,
A second weight given to the second estimated sea level is determined based on a distance value of the second LiDAR points,
How to monitor ports.
6. The method of claim 5,
calculating a first wave height from the first lidar points; and
Calculating a second wave height from the second lidar points; further comprising,
A first weight given to the first estimated sea level height is determined based on a change in the first wave height per unit time,
A second weight given to the second estimated sea level is determined based on a change in the second wave height per unit time.
How to monitor ports.
According to claim 1,
The first region and the second region are detected using an artificial neural network,
The artificial neural network is trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images.
How to monitor ports.
According to claim 1,
The first lidar points and/or the second lidar points are determined in consideration of characteristics of lidar points,
The characteristics of the lidar points include at least one of the number of lidar points, the deviation of the height value, and the distance value
How to monitor ports.
According to claim 1,
The second lidar points are determined based on vessel information obtained from the image,
The vessel information includes at least one of detection of the second region, a size of the second region, a distance to the vessel, and detection of occlusion associated with the vessel.
How to monitor ports.
A port monitoring method comprising:
acquiring an image captured by a camera;
acquiring lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera;
detecting a first area corresponding to the sea in the image and a second area corresponding to the vessel in the image;
selecting first LiDAR points related to a LiDAR beam reflected from the sea in consideration of a pixel position of a pixel included in the first area from among the plurality of LiDAR points;
determining a first reliability of the first lidar points based on a characteristic of the lidar points;
selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the second area;
determining a second reliability of the second lidar points based on the ship information obtained from the image; and
estimating the sea level height using at least one of the first LiDAR points and the second LiDAR points in consideration of the first reliability and the second reliability;
How to monitor ports.
12. The method of claim 11,
The characteristics of the lidar points include at least one of the number of lidar points, the deviation of the height value, and the distance value
How to monitor ports.
12. The method of claim 11,
The vessel information includes at least one of detection of the second region, a size of the second region, a distance to the vessel, and detection of occlusion associated with the vessel.
How to monitor ports.
A port monitoring method comprising:
acquiring lidar data including a plurality of lidar points obtained by a lidar sensor;
selecting first lidar points related to a lidar beam reflected from the sea from among the plurality of lidar points;
selecting second lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points;
The first LiDAR points and the second LiDAR points for estimating a sea level height based on at least one of the number of the first LiDAR points and the second LiDAR points, a deviation of a height value, and a distance value determining each reliability level; and
estimating a sea level height using at least one of the first LiDAR points and the second LiDAR points in consideration of the reliability of the first LiDAR points and the second LiDAR points; containing
How to monitor ports.
A port monitoring method comprising:
acquiring an image captured by a camera;
acquiring lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera;
detecting a vessel region corresponding to the vessel in the image using an artificial neural network;
The artificial neural network is trained using a training set including a plurality of training images and object information labeled in pixels of the plurality of training images,
At least some of the plurality of training images include ships and seas,
The object information reflects the object type,
the pixels of the vessel are labeled with the object information of the object type indicating the vessel;
selecting first lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the vessel area;
determining second lidar points related to a lidar beam reflected from a lower region of the vessel in contact with the sea from the first lidar points based on a height value of the first lidar point; and
Including; estimating the sea level height using the second lidar points
How to monitor ports.
16. The method of claim 15,
Further comprising the step of verifying the sea level height based on at least one of the number of the first LiDAR points, the deviation of the height value, and the distance value
How to monitor ports.
A port monitoring method comprising:
acquiring an image captured by a camera;
acquiring lidar data comprising a plurality of lidar points obtained by a lidar sensor having a viewing angle that at least partially overlaps a viewing angle of the camera;
detecting a vessel area corresponding to the vessel in the image;
selecting first lidar points related to the lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the vessel area;
obtaining vessel information related to the vessel from the image;
the vessel information includes at least one of detection of the vessel, a size of the vessel area, a distance to the vessel, and detection of occlusion associated with the vessel; and
When a preset condition is satisfied by the ship information, estimating the sea level height using the first lidar point; including
How to monitor ports.
18. The method of claim 17,
detecting a sea area corresponding to the sea in the image;
selecting second lidar points related to a lidar beam reflected from the sea in consideration of pixel positions of pixels included in the sea area from among the plurality of lidar points; and
When the preset condition is not satisfied by the ship information, estimating the sea level height using the second lidar point; further comprising
How to monitor ports.
A computer-readable recording medium in which a program for performing the method of any one of claims 1 to 18 is recorded.

Images