KR102530847B1 - Method and device for monitoring harbor and ship - Google Patents

Abstract

The present invention relates to a method and apparatus for monitoring ports and ships, and according to one embodiment, obtaining an image captured by a camera, a plurality of sensors obtained by a lidar sensor having a field of view that at least partially overlaps the field of view of the camera Acquiring lidar data including lidar points of , detecting a first area corresponding to the sea in the image and a second area corresponding to a ship in the image, and the first area corresponding to the sea in the image, the first area corresponding to the sea among the plurality of lidar points Selecting first lidar points related to a lidar beam reflected from the sea in consideration of pixel positions of pixels included in area 1, calculating a first estimated sea level height using the first lidar points, Selecting second lidar points related to the lidar beam reflected from the ship from among the plurality of lidar points in consideration of pixel positions of pixels included in the second area, height of the second lidar point Determining third lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from the second lidar points based on the value, using the third lidar points to determine the second lidar points A port monitoring method may include calculating the estimated sea level height and determining the sea level height by considering both the first estimated sea level height and the second estimated sea level height.

Description

Port and ship monitoring method and device {METHOD AND DEVICE FOR MONITORING HARBOR AND SHIP}

The present invention relates to a method and device for monitoring ports and ships, and more particularly, to a method and device for monitoring ports and ships for measuring sea level height.

Many accidents occur in the operation of ships, berthing and unloading in ports, and the main cause of these accidents is known to be carelessness of human operation. Here, negligence in operation is mainly caused by the fact that the situation around the ship or in the port cannot be accurately monitored through the naked eye. Currently, various types of obstacle sensors are used to supplement this, but there are still limitations.

Accordingly, recently, a technology for monitoring a situation around a ship or in a port through an image has been developed, but monitoring through an inaccurate image, such as an image in which distortion occurs in the image processing process, may cause an accident as well. The sea level height of the sea can be an important factor to consider in image processing processes such as conversion of viewpoints and registration of images, but there is a problem in that the technology for obtaining the sea level height is not well developed.

Therefore, it is necessary to develop a technology for substantially measuring the sea level height for accurate monitoring.

One problem to be solved by the present application is to provide a port and ship monitoring method and apparatus for measuring sea level height using lidar data related to lidar beams reflected from the sea and ships.

One problem to be solved by the present application is to provide a port and vessel monitoring method and apparatus for measuring sea level height by efficiently converging camera and LIDAR data.

The problem to be solved in the present application is not limited to the above-described problem, and problems not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. .

Acquiring an image captured by a camera as a harbor monitoring method according to an embodiment, lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Obtaining a first area corresponding to the sea in the image and a second area corresponding to a ship in the image Detecting a pixel position of a pixel included in the first area among the plurality of lidar points Selecting first LiDAR points related to the lidar beam reflected from the sea, calculating a first estimated sea level height using the first LiDAR points, pixels of pixels included in the second area Selecting second lidar points related to a lidar beam reflected from the ship from among the plurality of lidar points in consideration of a position, the second lidar point based on a height value of the second lidar point Determining third lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from a field, calculating a second estimated sea level height using the third lidar points, and calculating a second estimated sea level height using the third lidar points A port monitoring method may include determining the sea level height by considering both the first estimated sea level height and the second estimated sea level height.

According to another embodiment, a harbor monitoring method comprising: obtaining an image captured by a camera, a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Acquiring lidar data, detecting a first area corresponding to the sea in the image and a second area corresponding to a ship in the image, and determining the number of pixels included in the first area among the plurality of lidar points. Selecting first lidar points related to a lidar beam reflected from the sea in consideration of a pixel location, determining a first reliability of the first lidar points based on characteristics of the lidar points, the second Selecting second lidar points related to a lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the area; Determining a second reliability level of second lidar points, and determining the sea level height using at least one of the first lidar points and the second lidar points in consideration of the first reliability level and the second reliability level. A port monitoring method comprising the step of estimating may be provided.

According to another embodiment, as a port monitoring method, obtaining lidar data including a plurality of lidar points obtained by a lidar sensor, related to lidar beams reflected from the sea among the plurality of lidar points Selecting first lidar points, selecting second lidar points related to the lidar beam reflected from the ship from among the plurality of lidar points, the first lidar points and the second lidar points Determining reliability of each of the first lidar points and the second lidar points for estimating the sea level height based on at least one of the number of points, a deviation of the height value, and a distance value; and A port monitoring method comprising estimating a sea level height using at least one of the first lidar points and the second lidar points in consideration of the reliability of the first lidar points and the second lidar points can be provided.

According to another embodiment, a harbor monitoring method comprising: obtaining an image captured by a camera, a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Acquiring lidar data, detecting a ship area corresponding to a ship in the image using an artificial neural network, wherein the artificial neural network generates a plurality of training images and object information labeled on pixels of the plurality of training images. at least some of the plurality of training images include ships and seas, the object information reflects an object type, and the pixels of the ship indicate the object type indicating the ship. Selecting first lidar points related to a lidar beam reflected from the ship from among the plurality of lidar points labeled with object information and considering pixel positions of pixels included in the vessel area; Determining second lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from the first lidar points based on height values of lidar points, and the second lidar point A harbor monitoring method comprising the step of estimating the sea level height using the can be provided.

According to another embodiment, a harbor monitoring method comprising: obtaining an image taken by a camera; a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera; Acquiring data, detecting a ship area corresponding to a ship in the image, and a lidar beam reflected from the ship among the plurality of lidar points in consideration of pixel positions of pixels included in the ship area. Selecting first lidar points related to, obtaining ship information related to the ship from the image, the ship information is the detection of the ship, the size of the ship area, the distance to the ship and the ship A port monitoring method may be provided, including at least one of detection of related occlusion and estimating the sea level height using the first lidar point when a predetermined condition is satisfied by the vessel information. there is.

The solutions to the problems of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings. You will be able to.

According to an embodiment of the present application, the sea level height can always be measured regardless of circumstances using lidar data related to lidar beams reflected from the sea and ships.

According to another embodiment of the present application, it is possible to accurately measure the sea level height by efficiently integrating camera and lidar data.

The effects of the present application are not limited to the above-mentioned effects, and effects not mentioned will be clearly understood by those skilled in the art from this specification and the accompanying drawings.

1 is a diagram of a port monitoring device according to an embodiment.
2 and 3 are diagrams of an embodiment of a monitoring device according to an embodiment.
4 is a view related to a viewing angle and a depth of field according to an exemplary embodiment.
5 and 6 are diagrams of an installation position of a sensor module according to an embodiment.
7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.
10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.
12 is a diagram for explaining eyepiece guide information according to an embodiment.
13 is a diagram for explaining berthing guide information between a ship and a quay according to an embodiment.
14 is a diagram for explaining a method of acquiring eyepiece guide information according to an embodiment.
15 is a diagram related to viewpoint conversion according to an exemplary embodiment.
16 is a flowchart of a sea level calculation method according to an embodiment.
17 is an example of acquisition of sensor data according to an embodiment.
18 is a flowchart of a method for calculating sea level height using registered camera images and LIDAR data according to an embodiment.
19 is an example of matching camera images and lidar data according to an embodiment.
20 is a flowchart of a sea level height calculation method using lidar points related to lidar beams reflected from the sea according to an embodiment.
21 is an example of lidar points related to a lidar beam reflected from the sea according to an embodiment.
22 is a flowchart of a sea level height calculation method using lidar points related to lidar beams reflected from ships according to an embodiment.
23 is an example of lidar points related to a lidar beam reflected from a ship according to an embodiment.
24 is a flowchart of a method for calculating sea level height using estimated sea level height according to an embodiment.
25 is a flowchart of a method for calculating sea level height using reliability of lidar points according to an embodiment.
26 is a flowchart of a method for calculating the height of sea level using a result of analyzing a camera image according to an embodiment.
27 is a flowchart of a ship monitoring method considering the calculated sea level height according to an embodiment.
28 is a view related to conversion of viewpoints at different reference planes/reference heights.
29 is a diagram related to image correction for calculating berthing guide information on sea level/sea level according to an embodiment.

The embodiments described in this specification are intended to clearly explain the spirit of the present invention to those skilled in the art to which the present invention belongs, and the present invention is not limited by the embodiments described in this specification, and the present invention The scope of should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as general terms that are currently widely used as much as possible in consideration of the functions in the present invention, but they may vary depending on the intention, custom, or the emergence of new technologies of those skilled in the art in the technical field to which the present invention belongs. can However, in the case where a specific term is defined and used in an arbitrary meaning, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not the simple name of the term.

The drawings accompanying this specification are intended to easily explain the present invention, and the shapes shown in the drawings may be exaggerated as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

If it is determined that a detailed description of a known configuration or function related to the present invention in this specification may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

In this specification, sea level height should be broadly interpreted to include not only absolute sea level height, but also relative sea level height, relative sea level height compared to the average sea level height of a specific sea area, and the like. For example, the sea level height may include a variety of information that may vary depending on the change in sea level height, such as the distance between the sea level and the quay wall, the distance between the sea level and the monitoring device (e.g., an image generating unit), and the length of an object exposed above the sea level. may also include

In this specification, monitoring refers to grasping or recognizing a surrounding situation, and not only detecting a sensing object such as a certain area or a specific object using various sensors and providing the sensing result to the user, but also through calculation based on the sensing result. It should be interpreted broadly, including providing additional information.

Image-based monitoring may mean grasping or recognizing a surrounding situation based on an image. For example, monitoring may mean obtaining information for recognizing other vessels or obstacles by acquiring images around the vessel during operation of the vessel, or calculating berthing guide information when the vessel is berthing or unloading.

Berthing guide information is information about the environment, such as recognition of other ships or obstacles, identification of port conditions, access to berths, distance and speed from the quay, distance and speed from other ships, and identification of obstacles on the navigation route. can mean In the present specification, monitoring is mainly described when berthing is performed in a ship and a port, but is not limited thereto and may be applied to driving of a vehicle, operation of an aircraft, and the like.

According to one aspect of the present invention, as a port monitoring method, obtaining an image captured by a camera, including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with the viewing angle of the camera Acquiring lidar data, detecting a first area corresponding to the sea in the image and a second area corresponding to a ship in the image, and determining the number of pixels included in the first area among the plurality of lidar points. Selecting first lidar points related to a lidar beam reflected from the sea in consideration of a pixel position, calculating a first estimated sea level height using the first lidar points, and included in the second area Selecting second lidar points related to a lidar beam reflected from the ship from among the plurality of lidar points in consideration of pixel positions of pixels, based on a height value of the second lidar point; Determining third lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from lidar points, calculating a second estimated sea level height using the third lidar points and determining the sea level height by considering both the first estimated sea level height and the second estimated sea level height.

In one embodiment, the first estimated sea level height may be an average of height values of the first lidar points.

In one embodiment, the determining of the third lidar points includes generating lidar lines including lidar points having substantially the same height value from the second lidar points, and among the lidar lines A step of selecting a lidar line having the lowest height value as the third lidar points may be included.

In one embodiment, the lidar line may have the number and/or length of lidar points within a preset range.

In one embodiment, the sea level height may be a weighted sum of the first estimated sea level height and the second estimated sea level height.

In one embodiment, a first weight assigned to the first estimated sea level height is determined based on distance values of the first LiDAR points, and a second weight assigned to the second estimated sea level height is determined based on the distance values of the second LiDAR points. It can be determined based on the distance values of the points.

In one embodiment, the harbor monitoring method further comprises calculating a first wave height from the first lidar points and calculating a second wave height from the second lidar points, wherein the first estimation A first weight assigned to the sea level height is determined based on a change in the first wave height per unit time, and a second weight assigned to the second estimated sea level height is determined based on a change in the second wave height per unit time. can

In one embodiment, the first area and the second area are detected using an artificial neural network, and the artificial neural network includes a plurality of training images and a training set including object information labeled on pixels of the plurality of training images. can be learned using

In one embodiment, the first lidar points and / or the second lidar points are determined in consideration of characteristics of lidar points, and the characteristics of the lidar points are the number of lidar points and the deviation of the height value. , may include at least one of distance values.

In one embodiment, the second lidar points are determined based on vessel information obtained from the image, and the vessel information includes the detection of the second area, the size of the second area, the distance to the vessel and the It may include at least one of detecting occlusion associated with the vessel.

According to another aspect of the present invention, a harbor monitoring method includes obtaining an image captured by a camera, including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Acquiring lidar data to detect a first area corresponding to the sea in the image and a second area corresponding to a ship in the image, and pixels included in the first area among the plurality of lidar points. Selecting first lidar points related to a lidar beam reflected from the sea in consideration of a pixel position of the lidar point, determining a first reliability of the first lidar points based on characteristics of the lidar points, Selecting second lidar points related to a lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in area 2, based on vessel information obtained from the image The step of determining the second reliability of the second lidar points and the sea level height using at least one of the first lidar points and the second lidar points in consideration of the first and second reliability scores. A port monitoring method comprising the step of estimating may be provided.

In one embodiment, the characteristics of the lidar points may include at least one of the number of lidar points, a height deviation, and a distance value.

In one embodiment, the vessel information may include at least one of detection of the second area, size of the second area, distance to the vessel, and detection of occlusion related to the vessel.

According to another aspect of the present invention, a port monitoring method includes obtaining lidar data including a plurality of lidar points obtained by a lidar sensor, a lidar beam reflected from the sea among the plurality of lidar points, and Selecting related first lidar points, selecting second lidar points related to a lidar beam reflected from the ship from among the plurality of lidar points, and selecting the first lidar points and the second lidar points. Determining reliability of each of the first and second lidar points for estimating the sea level height based on at least one of the number of points, a deviation of the height value, and a distance value; and A port monitoring method comprising estimating a sea level height using at least one of the first lidar points and the second lidar points in consideration of the reliability of lidar points and the second lidar points. this can be provided.

According to another aspect of the present invention, a harbor monitoring method includes obtaining an image captured by a camera, including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Acquiring lidar data that corresponds to the ship in the image, detecting a ship area corresponding to the ship in the image using an artificial neural network, wherein the artificial neural network uses a plurality of training images and object information labeled on pixels of the plurality of training images. At least some of the plurality of training images include a ship and the sea, the object information reflects an object type, and the pixels of the ship are of the object type indicating the ship. Selecting first lidar points related to a lidar beam reflected from the ship from among the plurality of lidar points labeled with the object information and considering pixel positions of pixels included in the vessel area; Determining second lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from the first lidar points based on a height value of one lidar point, and the second lidar point A port monitoring method may be provided that includes estimating the sea level height using the points.

In one embodiment, the harbor monitoring method may further include verifying the sea level height based on at least one of the number of the first lidar points, a height deviation, and a distance value.

According to another aspect of the present invention, a harbor monitoring method includes obtaining an image captured by a camera, including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera Acquiring lidar data that corresponds to the ship in the image, detecting a ship area corresponding to the ship in the image, and considering the pixel positions of pixels included in the ship area, the radar reflected from the ship among the plurality of lidar points. Selecting first lidar points related to the beam, acquiring ship information related to the ship from the image, the ship information includes detection of the ship, size of the ship area, distance to the ship, and Provided is a port monitoring method comprising at least one of detection of occlusion related to a ship and estimating a sea level height using the first lidar point when a predetermined condition is satisfied by the ship information. It can be.

In one embodiment, the harbor monitoring method includes detecting a sea area corresponding to the sea in the image, a lidar reflected from the sea in consideration of a pixel position of a pixel included in the sea area among the plurality of lidar points. The method may further include selecting second lidar points related to the beam, and estimating the sea level height using the second lidar points when the predetermined condition is not satisfied by the vessel information. .

According to another aspect of the present invention, a computer-readable recording medium on which a program for performing the above-described methods is recorded may be provided.

Hereinafter, an apparatus and method for monitoring ships and ports according to an embodiment will be described.

1 is a diagram of a port monitoring device according to an embodiment.

Referring to FIG. 1 , the monitoring device 10 may include a sensor module 100 , a control module 200 and a communication module 300 .

The sensor module 100 may sense information about a ship or its surroundings and a port. The sensor module 100 may include an automatic identification system (AIS), an image generating unit, a LIDAR sensor, a position measurement unit, a posture measurement unit, and a casing.

The image generating unit may generate an image. The image generating unit may include a camera, radar, ultrasonic detector, and the like. Examples of cameras include, but are not limited to, monocular cameras, binocular cameras, visible light cameras, IR cameras, and depth cameras.

A lidar sensor is a sensor for detecting a distance and position of an object using a laser. For example, the distance between the lidar sensor and the target object and the position of the target object based on the lidar sensor may be expressed in a 3D coordinate system. For example, the distance between the lidar sensor and the object and the position of the object based on the lidar sensor may be expressed in a Cartesian coordinate system, a spherical coordinate system, a cylindrical coordinate system, and the like. The lidar sensor may have a plurality of channels in a vertical or horizontal direction, and may be, for example, a lidar sensor having 32 or 64 channels.

The lidar sensor may use laser reflected from the target object to determine the distance R from the target object. For example, the lidar sensor may use time of flight (TOF), which is a time difference between an emitted laser and a detected laser, to determine a distance to an object. To this end, the lidar sensor may include a laser output unit for outputting a laser beam and a receiver unit for detecting the reflected laser beam. The lidar sensor checks the time when the laser is output from the laser output unit and the time when the receiver detects the laser reflected from the object, and determines the distance to the object based on the difference between the emitted time and the detected time. can do. Of course, the lidar sensor uses other methods such as triangulation based on the detected position of the detected laser to determine the distance R from the target object, or using phase shift of the detected laser. R) may be determined.

The lidar sensor may determine the position of the target object using the angle of the irradiated laser. For example, when the lidar sensor can know the irradiation angle of one laser irradiated toward the scan area of the lidar sensor, if the laser reflected from the object existing on the scan area is detected by the receiver, the lidar sensor The sensor may determine the location of the object based on an irradiation angle of the irradiated laser.

A lidar sensor may have a scan area including an object in order to detect a position of an object in the vicinity. Here, the scan area represents a detectable area as one screen, and may mean a set of points, lines, and planes forming one screen for one frame. In addition, the scan area may mean the irradiation area of the laser irradiated from the lidar sensor, and the irradiation area may mean a set of points, lines, and planes where the laser irradiated during one frame meets a sphere at the same distance (R). can In addition, a field of view (FOV) means a detectable area, and may be defined as an angular range of a scan area when viewing the LIDAR sensor as an origin.

The position measurement unit may measure the position of components included in the sensor module 100, such as the sensor module 100 or the image generating unit. For example, the location measurement unit may be a Global Positioning System (GPS). In particular, RTK-GPS (Real-Time Kinematic GPS) may be used to improve the accuracy of position measurement.

The location measurement unit may obtain location information at predetermined time intervals. Here, the time interval may vary according to the installation location of the sensor module 100. For example, when the sensor module 100 is installed on a moving object such as a ship, the position measurement unit may acquire position information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a port, the position measurement unit may obtain position information at long time intervals. A time interval for acquiring location information of the location measurement unit may be changed.

The posture measurement unit may measure the posture of components included in the sensor module 100, such as the sensor module 100 or the image generating unit. For example, the posture measurement unit may be an inertial measurement unit (IMU).

The posture measurement unit may obtain posture information at every predetermined time interval. Here, the time interval may vary according to the installation location of the sensor module 100. For example, when the sensor module 100 is installed on a moving object such as a ship, the attitude measurement unit may acquire attitude information at short time intervals. On the other hand, when the sensor module 100 is installed in a fixed body such as a harbor, the attitude measurement unit may obtain attitude information at every long time interval. A time interval for acquiring attitude information of the attitude measuring unit may be changed.

The casing may protect sensor modules such as an image generating unit, a position measurement unit, and a posture measurement unit.

At least one of an image generating unit, a position measurement unit, and a posture measurement unit may be present inside the casing. The casing can prevent equipment, such as an image generating unit, from being corroded by salt water. Alternatively, the casing may protect the equipment existing therein by preventing or mitigating impact applied thereto.

A cavity may be formed inside the casing to contain an image generating unit or the like therein. For example, the casing may have a hollow rectangular parallelepiped shape, but is not limited thereto and may be provided in various shapes in which an image generating unit or the like may be disposed.

When the image generating unit is disposed inside the casing, an opening may be formed in one region of the casing or a transparent material such as glass may be formed in one region of the casing to secure the field of view of the image generating unit. The image generation unit may capture an image of the ship's periphery and the harbor through the opening or the transparent area.

The casing may be made of a strong material to protect the image generating unit from external impact. Alternatively, the casing may be made of a material such as an alloy for seawater to prevent corrosion due to salt.

The casing may include equipment for removing foreign substances from the image generating unit. For example, foreign substances adhering to the surface of the image generating unit may be physically removed through a wiper included in the casing. Here, the wiper may be provided in a linear or plate shape having the same or similar curvature as the surface so as to be in close contact with the surface to remove foreign substances. As another example, foreign substances may be removed by applying water or washer fluid through a liquid spray included in the casing, or foreign substances may be physically removed using a wiper after application.

The foreign matter removal equipment may be operated manually, but may also be operated automatically. For example, the foreign material removal equipment may be operated at predetermined time intervals. Alternatively, the foreign material removal equipment may be operated using a sensor that detects whether or not a foreign material is attached to the image generating unit. Alternatively, after determining whether a foreign material is captured in the image using the image captured by the image generating unit, the foreign material removing device may be operated when it is determined that the foreign material exists. Here, whether a foreign substance is captured in the image may be determined through an artificial neural network.

One sensor module 100 may include a plurality of identical equipment, such as two or more identical cameras.

The control module 200 may perform image analysis. In addition, an operation of receiving various data through the sensor module 100, an operation of outputting various outputs through an output device, an operation of storing various data in a memory or acquiring various data from a memory, and the like are of the control module 200. It can be done by control. Hereinafter, various operations or steps disclosed as embodiments of this specification may be interpreted as being performed by the control module 200 or under the control of the control module 200 unless otherwise stated.

Examples of the control module 200 include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), a state machine, and an on-demand Semiconductors (Application Specific Integrated Circuits, ASICs), radio-frequency integrated circuits (Radio-Frequency Integrated Circuits, RFICs), and combinations thereof may be used.

The communication module 300 may transmit information from the device 10 to the outside or receive information from the outside. The communication module 300 may perform wired or wireless communication. The communication module 300 may perform bi-directional or unidirectional communication. For example, the device 10 may transfer information to an external output device through the communication module 300 and output a control result performed by the control module 200 through the external output device. In addition, the communication module 300 may receive vessel-related VTS information or CITS (Costal Intelligent Transport System) information from a Vessel Traffic Service (VTS) that controls the vessel.

The sensor module 100, the control module 200, and the communication module 300 may include a control unit. The control unit may process and calculate various types of information within the module, and may control other components constituting the module. The controller may be physically provided in the form of an electronic circuit that processes electrical signals. A module may physically include only a single control unit, but may also include a plurality of control units. For example, the control unit may be one or a plurality of processors mounted on one computing means. As another example, the control unit may be provided as processors that are mounted on physically separated servers and terminals and collaborate through communication. Examples of the control unit include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a state machine, and an application specific semiconductor (Application Specific Integrated Circuit (ASIC), Radio-Frequency Integrated Circuit (RFIC), and combinations thereof.

The sensor module 100, the control module 200, and the communication module 300 may include a communication unit. The modules may transmit and receive information through a communication unit. For example, the sensor module 100 may transmit information acquired from the outside through the communication unit, and the control module 200 may receive the information transmitted by the sensor module 100 through the communication unit. The communication unit may perform wired or wireless communication. The communication unit may perform bi-directional or unidirectional communication.

The sensor module 100, the control module 200, and the communication module 300 may include a memory. The memory may store various processing programs, parameters for performing program processing, or such processing result data. For example, the memory may store data necessary for learning and/or reasoning, learning is in progress or a learned artificial neural network, and the like. Memory is a non-volatile semiconductor memory, hard disk, flash memory, RAM (Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or any other tangible non-volatile recording medium. etc. can be implemented.

The monitoring device 10 may include a plurality of identical modules, such as including two or more sensor modules 100 . For example, one device 10 may include two sensor modules 100, and each sensor module 100 may further include two cameras.

2 and 3 are diagrams of an embodiment of a monitoring device according to an embodiment.

Referring to FIG. 2 , the monitoring device 10 may include a sensor module 100 and a control module 200. The sensor module 100 may generate an image through the camera 130 and transmit the image to the control module 200 through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may change the viewpoint of the image by performing a viewpoint conversion to be described later. The control module 200 may receive an image from the sensor module 100 through the communication unit 210 and perform image analysis such as location/movement information estimation and image matching, which will be described later, through the control unit 220. In addition, the control module 200 may transmit analysis results such as location/movement information and matched images to the cloud server through the communication unit 210 . The cloud server may transmit the analysis result received from the control module 200 to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

Referring to FIG. 3 , the monitoring device 10 may include a sensor module 100 . The sensor module 100 may generate an image through the camera 130 and transmit the image to a cloud server through the communication unit 110 . In addition, the controller 120 of the sensor module 100 may change the viewpoint of the image by performing a viewpoint conversion to be described later. The cloud server may receive an image from the sensor module 100 and perform image analysis such as location/movement information estimation and image matching, which will be described later. In addition, the cloud server may transmit an image analysis result to a user terminal such as a smartphone, tablet, or PC, or may receive an instruction from the user terminal.

The device 10 shown in FIGS. 1 to 3 is only an example, and the configuration of the device 10 is not limited thereto.

As an example, the device 10 may include an output module 400 . The output module 400 may output a result of an operation performed by the control module 200 and the like. For example, the output module 400 may output analysis results. The output module 400 may be, for example, a display, a speaker, a signal output circuit, etc., but is not limited thereto. In this case, information may be output through the output module 400 in addition to outputting the information by transferring the information to an external output device such as a user terminal.

As another example, device 10 may not include a sensor module. In this case, the control module 200 may perform an image-based monitoring operation such as performing image analysis by receiving information from an external sensor device. For example, the control module 200 may perform image analysis by receiving information from an AIS, camera, lidar, radar, or the like already installed in a ship or port.

In addition, the steps performed by each component of FIGS. 1 to 3 are not necessarily performed by the corresponding component, but may be performed by other components. For example, in FIG. 2 above, it is described that the controller 120 of the sensor module 100 performs the viewpoint conversion, but the controller 220 of the control module 200 or the cloud server may perform the viewpoint conversion. .

Hereinafter, the monitoring device 10 and method will be described in more detail.

Image acquisition for image-based monitoring may be performed through the sensor module 100 . For example, an image may be obtained through an image generation unit included in the sensor module 100 . Alternatively, as described above, an image may be obtained from an external sensor device. Images for vessel and harbor monitoring will generally include sea, vessel, buoy, obstacle, terrain, harbor, sky, building, and the like. Hereinafter, a method of analyzing and monitoring an image acquired through a visible ray camera will be mainly described, but is not limited thereto.

Depending on the image generating unit, a field of view (FOV) and a depth of field may vary. 4 is a view related to a viewing angle and a depth of field according to an exemplary embodiment. Referring to FIG. 4 , the field of view (FOV) may mean how much range is included in an image horizontally or vertically, and is generally expressed as an angle (degree). The meaning of having a larger viewing angle may mean generating an image including an area with a wider width horizontally or vertically. The depth of field may mean a distance range recognized as being in focus, and the depth of field may mean that the distance range recognized as being in focus is wide. Referring to FIG. 4 , according to the depth of field (DOF), the image may include an area A1 recognized as being in focus and an area A2 other than that. Hereinafter, the area included in the image is referred to as the imaging area (A1 + A2), and the area recognized as being in focus is referred to as the effective area (A1), and image analysis and monitoring may be performed based on the effective area, but the entire imaging area Since it may be performed based on or a part of the imaging area, the area used to perform image analysis and monitoring is referred to as a monitoring area.

An example of a camera with a large field of view and shallow depth of field is a wide-angle camera. Examples of a camera with a small field of view and a deep depth of field include a high-magnification camera and a zoom camera.

The sensor module 100 may be installed without limitation in its position or posture, such as a lighting tower, crane, ship, etc. in a port, and there is no limitation in the number thereof. However, depending on the type and performance of the sensor module 100, the installation location or number may vary. For example, when the sensor module 100 is a camera, the sensor module 100 may be installed at an altitude of 15 m or more from the surface of the water for efficient monitoring, or a plurality may be installed to have different imaging areas. In addition, the position and attitude of the sensor module 100 may be manually or automatically adjusted during or after installation.

A field of view of the lidar may at least partially overlap a field of view of an image generating unit (eg, a camera). For example, the field of view (FOV) of the lidar may be smaller than the field of view of the camera. As will be described later, in this case, lidar data outside the field of view of lidar where lidar data is not acquired may be estimated using an image obtained from a camera.

5 and 6 are diagrams of an installation position of a sensor module according to an embodiment. Referring to FIGS. 5 and 6 , the sensor module 100 may be installed in a fixed location such as a port or land, or installed in a moving object such as a ship. Here, when the sensor module 100 is installed on a ship, it can be installed on a ship to be monitored (hereinafter referred to as “target ship”) as shown in FIG. 6, and as shown in FIG. It can also be installed on a third vessel that is not subject to monitoring, such as an auxiliary tugboat. In addition to this, the sensor module 100 may be installed in a drone or the like to monitor a target ship.

Other components of the monitoring device 10 may be installed together with the sensor module 100 or in a separate location.

As described above, image analysis for image-based monitoring may include obtaining object characteristics. Examples of objects may include ships, harbors, buoys, seas, terrain, sky, buildings, people, animals, fire, and smoke. Examples of object properties may include the type of object, the position of the object, the distance to the object, the absolute and relative speed and speed of the object, and the like.

Image analysis for image-based monitoring may include recognizing/determining the surrounding situation. For example, image analysis may determine that a fire situation has occurred from an image of a fire in a port, or that an intruder has entered from a captured image of a person entering a port at an unscheduled time. For another example, image analysis may include detecting a fire from an image in which smoke is present.

Image analysis for image-based monitoring may be performed through the control module 200 or the controllers 120 and 220 included in each module 100 and 200 .

The device 10 can recognize objects. For example, the device 10 may recognize an object included in an image. For example, the device 10 may determine whether an object such as a ship, a tugboat, the sea, or a harbor is included in the image. Here, object recognition may be determining which position of an image an object exists.

7 to 9 are diagrams of an object recognition step according to an exemplary embodiment.

7 is an image taken by a camera, and an object can be recognized as shown in FIG. 8 or 9 through an object recognizing step.

Specifically, FIG. 8 shows which object the corresponding pixel corresponds to for each pixel of the image, which is also referred to as segmentation. In this case, the object recognition step may mean the segmentation step. Through segmentation, characteristics corresponding to pixels on the image may be assigned or calculated from the image. One might say that a pixel is assigned or labeled with a property. Referring to FIGS. 7 and 8 , a segmented image as shown in FIG. 8 may be obtained by performing segmentation based on an image captured by the camera of FIG. 7 . In FIG. 8 , the first pixel area P1 is an area on the image of a pixel corresponding to a ship, the second pixel area P2 is the sea, the third pixel area P3 is the quay wall of the harbor, and the fourth pixel area ( P4) is the terrain, and the fifth pixel area P5 is an area on the image of a pixel corresponding to the sky.

Although FIG. 8 shows that information about the type of object corresponding to each pixel on an image is calculated by performing segmentation, information that can be obtained through segmentation is not limited thereto. For example, characteristics such as location, coordinates, distance, and direction of an object may also be acquired through segmentation. In this case, different characteristics may be expressed independently or may be reflected and expressed at the same time.

Table 1 is a table related to labeling that simultaneously reflects information on the type and distance of an object according to an embodiment. Referring to Table 1, classes may be set in consideration of object type and distance information, and an identification value may be assigned to each class. For example, identification value 2 may be assigned by considering both topography, which is information about the type of object, and short distance, which is information about distance. Table 1 is an example of a case in which information on type and information on distance are considered together. In addition, other information such as direction information, moving direction of obstacles, speed, and navigational aids may also be considered. In addition, all identification values do not have to include a plurality of pieces of information, nor do they have to include the same type of information. For example, a specific identification value includes only type information (eg, identification value 1 does not include information about distance), and other identification values include information about type and distance. It can be expressed in various ways. Other classes, such as tugboat, rope, ship's side and deck, for example, may be added or modified to other classes.

identification value class 0 sky and more One ocean 2 terrain + close range 3 Terrain + Medium 4 Terrain + Far 5 fixed obstacles + short range 6 fixed obstacles + medium distance 7 fixed obstacles + far 8 Dynamic Obstacles + Short Range 9 Dynamic Obstacles + Medium Range 10 Dynamic Obstacles + Far

FIG. 9 shows where an object exists in an image with a bounding box, which is also referred to as detection. In this case, the object recognition step may mean the detection step. Compared to segmentation, detection can be seen as detecting a position in which an object is included in a box shape rather than calculating characteristics for each pixel of an image. Referring to FIGS. 7 and 9 , a detection image as shown in FIG. 9 may be obtained by performing detection based on an image captured by the camera of FIG. 7 . In FIG. 9, it can be seen that the ship is detected on the image and the location of the ship is expressed as a rectangular bounding box (BB). Although only one object is illustrated in FIG. 9 as being detected, two or more objects may be detected from one image.

Segmentation and detection may be performed using an artificial neural network. Segmentation/detection may be performed through one artificial neural network, or each artificial neural network may perform segmentation/detection using a plurality of artificial neural networks and combine the results to calculate a final result.

An artificial neural network is a type of algorithm modeled after the structure of a human neural network, and may include one or more layers including one or more nodes or neurons, and each node may be connected through a synapse. Data (input data) input to the artificial neural network may be output (output data) through nodes through synapses, through which information may be obtained.

Types of artificial neural networks include a convolution neural network (CNN) that extracts features using a filter and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back to an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), and relational networks (RN) can be applied, and there are limitations. It is not.

Before using an artificial neural network, a training step is required. Alternatively, it may be trained using an artificial neural network. Hereinafter, the step of learning the artificial neural network will be expressed as a learning step, and the step of using it as an inference step.

The artificial neural network may be trained through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning.

10 and 11 are diagrams of a learning step and an inference step of an artificial neural network according to an embodiment.

10 is an embodiment of a learning step of an artificial neural network, in which an untrained artificial neural network receives learning data or training data, outputs output data, and compares the output data with labeling data. The artificial neural network can be trained through backpropagation of the error. Training data, output data, and labeling data may be images. Labeling data may include ground truth. Alternatively, the labeling data may be data generated by a user or program.

11 is an example of an artificial neural network inference step, in which the trained artificial neural network receives input data and outputs output data. Information that can be inferred in the inference step may vary depending on the information of the learning data in the learning step. In addition, the accuracy of output data may vary according to the degree of learning of the artificial neural network.

Recognition of an object is not limited to the above description and may be implemented in other ways. For example, although it has been described that an identification value is used to recognize an object for convenience of description, the identification value is only used as one type of index. For example, an index in the form of a vector rather than a value may be used to recognize an object, and training data, output data, and labeling data of an artificial neural network may be data in the form of a vector.

According to one embodiment, the device 10 may provide berthing guide information for a ship. The berthing guide information may be used for berthing of a ship and may refer to information for assisting or guiding a user such as a pilot or a captain in berthing. Eyepiece guide information may include information on distance/velocity. Examples of information on distance/velocity include, but are not limited to, an absolute position such as coordinates, a relative position from a specific reference point, a distance from an arbitrary point, a distance range, a direction, an absolute speed, a relative speed, and a speed.

Information on distance/velocity may be estimated based on a predetermined area or point. For example, the distance between the ship and the quay may be estimated by calculating the distance between one point of the ship and one point of the quay, or may be estimated by calculating the shortest distance between one point of the ship and the quay. As another example, the distance between ships may be estimated by calculating the distance between one point of the first ship and one point of the second ship. One point of the ship may correspond to a point of the ship in contact with the sea or may correspond to the bow or stern of the ship, but is not limited thereto.

Information on distance/speed may be expressed as a predetermined distance value, direction value, and speed value. For example, distance information may be expressed as 1m, 2m, etc., direction information may be expressed as 10°, 20°, etc., and speed information may be expressed as 5cm/s, 10cm/s, etc.

Information on distance/speed may be expressed as an index corresponding to a plurality of categories having a certain range. For example, distance information may be expressed as short distance, medium distance, and long distance, direction information may be expressed as left direction, front direction, and right direction, and speed information may be expressed as low speed, medium speed, and high speed. there is. It will also be possible to combine these and express them as left near distance, right far distance, and the like.

12 is a diagram for explaining eyepiece guide information according to an embodiment. Referring to (a) of FIG. 12, the berthing guide information is berthing guide information between the ship (OBJ1) and the quay (OBJ2) (f1, f2) and the berthing guide between the ship (OBJ1) and other ships (OBJ3, OBJ4) Information f3 and f4 may be included. Specifically, the berthing guide information (f1, f2) between the ship (OBJ1) and the quay wall (OBJ2) may include information on the distance/velocity of the ship (OBJ1) to the quay wall (OBJ2), and the ship (OBJ1) The berthing guide information f3 and f4 between the ship OBJ3 and the other ships OBJ3 and OBJ4 may include information about the distance/velocity of the ship OBJ1 to the other ships OBJ3 and OBJ4. Referring to (b) of FIG. 12, berthing guide information (f1, f2) between the ship OBJ1 and the quay wall OBJ2 is the boundary area corresponding to the area in contact with the sea surface of the ship OBJ1 and the quay wall OBJ2. ) may be information between. The boundary area may include not only a line or a point where the ship and the sea surface meet, but also a certain area near the line or point. In addition, the berthing guide information between the ship (OBJ1) and the other ships (OBJ3, OBJ4) may be information between a predetermined area corresponding to the bow / stern of the vessel (OBJ1, OBJ3, OBJ4), where the predetermined area may mean not only the bow/stern points of the ships OBJ1, OBJ3, and OBJ4, but also a certain area near them.

The berthing guide information f1 and f2 between the ship OBJ1 and the quay wall OBJ2 may include two berthing guide information as shown in FIG. 12, but is not limited thereto, and one berthing guide information or three or more It may include berthing guide information. In addition, the berthing guide information f3 and f4 between the ship OBJ1 and the other ships OBJ3 and OBJ4 may include one berthing guide information between the two ships as shown in FIG. 12, but is not limited thereto Instead, two or more berthing guide information may be included between the two ships. The above-described information may be referred to as unloading guide information when used for unloading of a ship.

13 is a diagram for explaining berthing guide information between a ship and a quay according to an embodiment.

The berthing guide information may be sea level height/sea level information. For example, the berthing guide information f5 may include information about the distance/speed between the ship OBJ1 and the quay wall OBJ2 at sea level.

The eyepiece guide information may be information at ground level (or quay wall level). For example, the berthing guide information f6 may include information about the distance/velocity between the ship OBJ1 and the quay wall OBJ2 at the height of the quay wall OBJ2.

The eyepiece guide information may be information at a predetermined height. Here, the predetermined height may be between the height of the sea level and the height of the quay, as shown in the berthing guide information f7 shown in FIG. 13, but is not limited thereto.

The berthing guide information between the ship and the quay may be information f8 between the fender (OBJ6) installed on the ship (OBJ1) and the quay wall (OBJ2). Since the ship (OBJ1) collides with the insect repellent (OBJ6) when berthing or comes into contact with the insect repellent (OBJ6) when anchored, it is important to acquire information (f8) on the distance/speed between the ship (OBJ1) and the insect repellent (OBJ6). can be advantageous In FIG. 13, although the insect repellent OBJ6 is shown as being installed at a position higher than the sea level, at least a portion of the insect repellent OBJ6 may be installed so as to be submerged in the sea OBJ5. In this case, the berthing guide information is at sea level. It may include information on the distance/velocity between the vessel and the insect repellent.

The berthing guide information between the ship and the other ship may include information about the distance/velocity between the ship and the other ship at sea level. The berthing guide information between the ship and the other ship may include information about the distance/velocity between the ship and the other ship at a predetermined height. Here, the predetermined height may be determined in consideration of the shape of the hull. For example, the predetermined height may be the height of a region where a ship protrudes in a direction of another ship. In this case, it may be advantageous to the berthing guide by more accurately grasping the possibility of collision between ships.

According to one embodiment, the device 10 may calculate eyepiece guide information based on lidar data. For example, the device 10 may calculate the distance/velocity to the object using 3D coordinates of lidar points included in the obtained lidar data. Examples of objects may include ships, sea and land, ports, quay walls, buoys, terrain, sky, buildings, people, and animals.

According to one embodiment, eyepiece guide information may be calculated based on images. Here, the image may be an image generated by an image generating unit such as a camera or an image processed through image segmentation therefrom. For example, information on the distance/speed of a ship is calculated based on an image including a ship, sea, and land as objects, or information on the distance/speed of a ship based on a segmentation image generated through segmentation from the image. can be calculated. Examples of objects may include ships, seas, and lands as well as harbors, quay walls, buoys, terrain, sky, buildings, people, and animals.

Hereinafter, an object for estimating eyepiece guide information is referred to as a target object. For example, in the above example, a ship may be a target object. Also, the number of target objects may be plural. For example, when estimating the distance or speed of each of a plurality of ships included in the image, the plurality of ships may be the target object.

According to one embodiment, eyepiece guide information may be calculated based on image pixels. As described above, when eyepiece guide information is calculated based on a point, a point on an image may correspond to a pixel. Thus, eyepiece guide information can be calculated based on the spacing between image pixels.

Information on the distance between points may be calculated based on the spacing between pixels. For example, a predetermined distance may be assigned to each pixel interval and a distance between points may be calculated in proportion to the interval between pixels. As another example, distances between pixels may be calculated based on coordinate values of pixels on an image, and distances between points may be calculated based on the distances between pixels.

Information about the speed between points may be calculated based on changes in information about the distance between points. In this case, movement information may be calculated based on a plurality of images or video frames. For example, information on speed between points may be calculated based on a distance between points in a previous frame, a distance between points in a current frame, and a time interval between frames.

14 is a diagram for explaining a method of acquiring eyepiece guide information according to an embodiment. Obtaining eyepiece guide information is a step of acquiring an image generated by an image generating unit such as a camera (FIG. 14(a)), performing image segmentation on the image to obtain a segmented image (or an image obtained by visualizing the segmented image) Generating (Fig. 14 (b)), finding a point for calculating eyepiece guide information based on the segmentation image (Fig. 14 (c)), and calculating eyepiece guide information corresponding to the point can include Although a method for calculating eyepiece guide information from a segmentation image has been described in FIG. 14, this is only an embodiment, and it is possible to find a point based on an image generated by an image generating unit and calculate eyepiece guide information without generating a segmentation image. There will be.

Obtaining eyepiece guide information may include converting a viewpoint of an image. For example, the obtaining of the eyepiece guide information includes obtaining an image generated by an image generating unit, performing viewpoint transformation on the image, and segmenting the viewpoint-converted image to obtain a segmented image. Eyepiece guide information may be calculated for the segmentation image converted from the viewpoint through the step of generating the segmentation image or performing viewpoint conversion on the segmentation image after the step of generating the segmentation image. Hereinafter, viewpoint conversion will be described.

In general, an image generated by an image generating unit such as a camera may appear as a perspective view. Converting this to a top view (planar viewpoint), a side view, or another perspective viewpoint may be referred to as viewpoint conversion. Of course, a top view or side view image may be converted to another view, and the image generating unit may generate a top view image or a side view image, etc. In this case, the view conversion may not need to be performed.

15 is a diagram related to viewpoint conversion according to an exemplary embodiment. Referring to (a) of FIG. 15 , another near perspective image may be acquired through conversion of a perspective view image. Here, the viewpoint conversion may be performed so that the pier OBJ2 is positioned along the horizontal direction (left-right direction on the image) on the image. Referring to (b) of FIG. 15 , a top view image may be acquired through conversion of a perspective view image of a perspective view image. Here, the top view image may be a view looking down at the sea level from a direction perpendicular to the sea level. Also, as in (a) of FIG. 15 , viewpoint conversion may be performed so that the quay wall OBJ2 is located along the horizontal direction on the image.

Convenience, convenience, and accuracy can be improved when calculating eyepiece guide information through viewpoint conversion. For example, in the case of pixel-based distance calculation, if a top-view image is used, a distance corresponding to an interval between pixels may be the same for the entire image or at least a partial area.

As an example of viewpoint transformation, inverse projective mapping (IPM) may be performed. A 2D image is generated when light reflected from a subject in a 3D space is incident on an image sensor through a lens of a camera, and the relationship between 2D and 3D depends on the image sensor and lens. For example, Equation 1 and can be expressed as

Here, the matrix on the left side denotes 2D image coordinates, the first matrix on the right side denotes intrinsic parameters, the second matrix denotes extrinsic parameters, and the third matrix denotes 3D coordinates. Specifically, fx and fy denote focal lengths, cx and cy denote principal points, and r and t denote rotation and translation parameters, respectively.

A viewpoint can be changed by projecting a 2D image onto an arbitrary 3D plane through back-projection transformation. For example, a perspective view image may be converted into a top view image through back projection transformation, or may be converted into another perspective view image.

Internal parameters may be required for viewpoint conversion. As an example of a method for obtaining internal parameters, the Zhang method may be used. The Zhang method is a kind of polynomial model, which acquires internal parameters by taking pictures of a grid having a known grid size from various angles and distances.

Information on the position and/or attitude of the image generating unit/sensor module that captured the image may be required for viewpoint conversion. This information may be obtained from the position measurement unit and the attitude measurement unit.

Alternatively, information on the position and/or posture may be obtained based on the position of the fixed body included in the image. For example, the image generating unit may generate a first image including a target fixture, which is a fixed object such as a terrain or a building, disposed in a first position and/or a first posture at a first viewpoint. Then, at a second viewpoint, the image generation unit may generate a second image including the target fixture. The position of the target fixture on the first image and the position of the target fixture on the second image are compared to calculate the second position and/or the second posture, which is the position and/or posture of the image generating unit at the second viewpoint. can do.

Acquisition of information on a position and/or posture for viewpoint conversion may be performed at predetermined time intervals. Here, the time interval may depend on the installation location of the image generating unit/sensor module. For example, when the image generating unit/sensor module is installed on a mobile body such as a ship, there may be a need to acquire position and/or attitude information at short time intervals. On the other hand, when the image generating unit/sensor module is installed in a fixed body such as a port, information on the position and/or attitude may be obtained at a relatively long time interval or initially obtained only once. When movement and fixation are repeated, such as in a crane, it may be implemented in a manner in which information on position and/or attitude is obtained only after movement. In addition, the time interval for acquiring information on the location and/or posture may be changed.

The device 10 may convert the viewpoint of the image based on the reference plane. For example, the device 10 may transform the viewpoint of an image using a plane parallel to the sea level and a reference plane where the quay wall is located. Here, the reference plane may depend on the calculated sea level height. Of course, the device 10 is not limited to the above description, and it is also possible to convert the viewpoint of an image to a reference plane other than the sea level or a part of the ship other than the plane where the quay is located (for example, a reference plane at deck height).

In addition, the above-described viewpoint conversion method is only an example, and viewpoint conversion may be performed in a different way, and the viewpoint conversion information is information about the matrix of Equation 1, parameters, coordinates, position, and/or attitude, etc. contains the information necessary for

Device 10 may calculate sea level height for ship and/or harbor monitoring. For example, the device 10 may calculate the sea level height from the obtained sensor data, and use the calculated sea level height for monitoring a ship or a port. For example, the device 10 may use the sea level height as height information up to the projection plane when converting a viewpoint of a camera image or converting an inverse projection. For another example, the device 10 may use the sea level height as a variable when converting a camera image to an image for matching. As another example, the device 10 may use the sea level height as a variable of data conversion when converging lidar data and camera images.

16 is a flowchart of a sea level calculation method according to an embodiment.

Referring to FIG. 16 , the sea level height calculation method according to an embodiment may include acquiring sensor data ( S1000 ) and calculating the sea level height ( S2000 ).

The device 10 may acquire sensor data (S1000).

According to one embodiment, device 10 may obtain an image from a camera. For example, the device 10 may obtain an image from a camera installed on a berth facing the sea. For example, the device 10 may acquire an image of the sea and, if there is a ship, may also acquire an image of the ship.

According to one embodiment, the device 10 may obtain lidar data from a lidar sensor. For example, the device 10 may obtain lidar data from a lidar sensor installed on a berth facing the sea. For example, the device 10 may acquire lidar data for the sea, and when a ship enters a berth, may also acquire lidar data for the ship.

According to an embodiment, the lidar sensor may obtain lidar data for an area corresponding to an area captured by a camera acquiring an image. For example, a lidar sensor may have a field of view that at least partially overlaps the field of view of a camera acquiring an image.

17 is an example of acquisition of sensor data according to an embodiment. Referring to FIG. 17 , the device 10 according to an embodiment may obtain sensor data for the same area from lidar and cameras installed around a berth. For example, lidar and a camera may be installed on a pier toward or looking at a berth to acquire data about an area where the berth is located. Here, the viewing angle of the lidar may overlap at least partially with the viewing angle of the camera.

Referring to FIG. 17(a), the device 10 may acquire an image of the sea and, if there is a ship, may also acquire an image of the ship.

Referring to FIG. 17 (b), the device 10 may acquire lidar data for the sea, and may also acquire lidar data for the ship when the ship enters the berth. The lidar data may include a plurality of lidar points captured by lidar sensors. For example, lidar data may include a plurality of lidar points for each vertical or horizontal channel.

The apparatus 10 may determine the distance from the ship or sea surface using lidar points corresponding to the ship or sea in the obtained lidar data, and may calculate the sea level height using this.

The device 10 may calculate the sea level height (S2000).

The device 10 may calculate the sea level height using the acquired sensor data. For example, the device 10 may calculate the sea level height using at least one of sensor data obtained from a camera installed on a berth facing the sea and a LIDAR sensor.

For example, the device 10 may obtain a distance from the sea surface using 3D coordinates of lidar points included in the obtained lidar data, and calculate the height of the sea level using the obtained distance. The device 10 may calculate the height of the sea level in consideration of the distance from the sea surface, the installation position of the lidar sensor, and the installation angle. As another example, the device 10 may use the 3D coordinates of the lidar points included in the obtained lidar data to obtain a distance to the area of the ship in contact with the sea surface, and calculate the sea level height using this. The device 10 may calculate the height of the sea level in consideration of the distance between the ship's area and the installation location and installation angle of the lidar sensor.

However, due to cost issues, low-performance lidar sensors are often installed on berths, and there is a problem in that it is difficult to distinguish which lidar points among lidar points of low-resolution lidar data are the sea area or the ship area. In order to solve this problem, according to an embodiment, the device 10 may match the obtained camera image and LIDAR data, and calculate the sea level height using this.

18 is a flowchart of a method for calculating sea level height using registered camera images and LIDAR data according to an embodiment. Referring to FIG. 18 , the sea level height calculation method using the matched camera image and lidar data according to an embodiment includes matching the camera image and lidar data (S2010) and using the matched camera image and lidar data. and calculating the sea level height (S2020).

The device 10 may match the camera image and lidar data (S2010).

According to an embodiment, the device 10 may match camera images and LIDAR data using information for matching. For example, the device 10 may match the camera image and lidar data by matching the coordinate system of the camera image with the coordinate system of lidar data. That is, the coordinate systems of the camera image and LIDAR data may be mutually convertible.

The device 10 may match the camera image and lidar data by considering the camera installation position, the camera installation angle, the lidar sensor installation position, the lidar sensor installation angle, and the like. Here, the device 10 may re-synthesize the camera image and LIDAR data by reflecting the calculated sea level. For example, when matching lidar data and camera images, the device 10 may re-match the camera image and lidar data by using the calculated sea level height as a variable of data conversion.

19 is an example of matching camera images and lidar data according to an embodiment. Referring to FIG. 19 (a), it can be seen that lidar data obtained from a lidar sensor scanning the same area as an image obtained from a camera installed on a berth is matched.

According to one embodiment, device 10 may match the segmented image with lidar data. For example, the device 10 may generate a segmentation image from an image obtained from a camera using an artificial neural network and match the generated segmentation image with LIDAR data. Referring to FIG. 19 (b), it can be seen that the lidar data obtained from the lidar sensor scanning the same area as the image obtained from the camera installed on the berth is matched.

Matching of camera images and lidar data is not limited to the above description, and may be implemented in other ways, such as matching images detected using an artificial neural network and lidar data.

Returning to FIG. 18 again, description will be given.

The device 10 may calculate the sea level height using the matched camera image and LIDAR data (S2020).

According to one embodiment, the device 10 may calculate the sea level height using a lidar point selected from among a plurality of lidar points of lidar data using an image matched with lidar data. For example, the device 10 may calculate the sea level height using a LIDAR point matched to an area corresponding to the sea in the image. In another example, device 10 may calculate sea level height using lidar points matched to the area of the vessel in the image.

According to one embodiment, the device 10 may calculate the sea level height using a lidar point associated with a lidar beam reflected from the sea.

20 is a flowchart of a method for calculating the sea level height using lidar points related to lidar beams reflected from the sea according to an embodiment. Referring to FIG. 20, the sea level calculation method using lidar points related to lidar beams reflected from the sea according to an embodiment includes selecting lidar points related to lidar beams reflected from the sea (S2110) and sea level. It may include calculating the sea level height from lidar points related to the lidar beam reflected from ( S2120 ).

The device 10 may select lidar points related to the lidar beam reflected from the sea (S2110).

According to one embodiment, device 10 may select lidar points associated with a lidar beam reflected from the sea using the lidar data and the matched image.

To this end, the device 10 may detect an area corresponding to the sea in the image. For example, the device 10 may detect an area corresponding to the sea in an image using an artificial neural network. For example, the device 10 may generate a segmentation image from an image using an artificial neural network, and detect an area where a pixel labeled with object information indicating an object type of the sea is located as an area corresponding to the sea. The device 10 is not limited to the above description, and may detect an area corresponding to the sea in an image in another manner, such as determining an area corresponding to the sea through image detection.

The device 10 may select lidar points related to the lidar beam reflected from the sea by using an area corresponding to the sea in the sensed image. For example, the device 10 may select lidar points related to a lidar beam reflected from the sea in consideration of pixel positions of pixels included in an area corresponding to the sea in the image.

21 is an example of lidar points related to a lidar beam reflected from the sea according to an embodiment. Referring to FIG. 21 , the apparatus 10 may select lidar points 401 related to a lidar beam reflected from the sea among lidar points matched with an image. For example, the apparatus 10 may include lidar points matched to pixels included in an area corresponding to the sea in an image among a plurality of lidar points, and lidar points 401 related to a lidar beam reflected from the sea. ) can be selected. It is not limited to FIG. 21, and lidar points 401 related to the lidar beam reflected from the sea may be selected in another method in which a segmented image or a detected image is used.

The selection of the lidar points 401 related to the lidar beam reflected from the sea is not limited to the above description and may be implemented in other ways. According to one embodiment, the device 10 may use lidar data to select lidar points associated with the lidar beam 401 reflected from the sea. For example, the apparatus 10 may select lidar points 401 related to lidar beams reflected from the sea in consideration of the distribution, number, and the like of lidar data.

Returning to FIG. 20 again, description will be made.

The apparatus 10 may calculate the sea level height from lidar points related to the lidar beam reflected from the sea (S2120).

The device 10 may calculate the sea level height using 3D coordinate values of lidar points related to lidar beams reflected from the sea. According to an embodiment, the device 10 may calculate the sea level height using height values (eg, Z values among 3D coordinate values) of lidar points related to lidar beams reflected from the sea. For example, the apparatus 10 may calculate an average value of height values of lidar points associated with a lidar beam reflected from the sea, and calculate the average value of the calculated height values as the height of the sea level.

According to an embodiment, the device 10 may calculate the wave height of the sea surface using 3D coordinate values of lidar points related to the lidar beam reflected from the sea.

For example, the apparatus 10 may calculate a wave height using height values (eg, Z values among 3-dimensional coordinate values) of lidar points related to lidar beams reflected from the sea. For example, the device 10 may calculate a maximum value and a minimum value of height values of lidar points related to a lidar beam reflected from the sea, and calculate a difference between the maximum value and the minimum value as a wave height.

Of course, the device 10 is not limited to the above description, and calculates the height of the sea level in other ways, such as calculating the height of the sea level by reflecting the height value of the lidar points related to the lidar beam reflected from the sea at a predetermined ratio. also free Here, the predetermined ratio may be determined in consideration of the location, number, and the like of each lidar point.

22 is a flowchart of a sea level height calculation method using lidar points related to lidar beams reflected from ships according to an embodiment. Referring to FIG. 22, the sea level height calculation method using lidar points related to the lidar beam reflected from the ship according to an embodiment includes selecting lidar points related to the lidar beam reflected from the ship (S2210), the vessel Determining lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea among lidar points related to the lidar beam reflected from (S2220) and the lidar points reflected from the lower area of the ship in contact with the sea A step of calculating the sea level height from the LIDAR points related to the LIDAR beam (S2230) may be included.

The device 10 may select lidar points related to the lidar beam reflected from the ship (S2210).

According to one embodiment, the device 10 may select lidar points associated with a lidar beam reflected from a ship using the lidar data and the matched image.

To this end, the device 10 may detect a region corresponding to a ship in the image. For example, the device 10 may detect a region corresponding to a ship in an image using an artificial neural network. For example, the device 10 may generate a segmentation image from an image using an artificial neural network, and detect an area where a pixel labeled with object information indicating a ship as an object type is located as an area corresponding to the ship. The apparatus 10 is not limited to the description described above, and may detect a region corresponding to a ship in an image in another manner, such as determining a region corresponding to a ship through image detection.

The device 10 may select lidar points related to the lidar beam reflected from the vessel using a region corresponding to the vessel in the sensed image. For example, the apparatus 10 may select lidar points associated with a lidar beam reflected from a vessel in consideration of pixel positions of pixels included in an area corresponding to a vessel in the image.

23 is an example of lidar points related to a lidar beam reflected from a ship according to an embodiment. Referring to FIG. 23 , the apparatus 10 may select lidar points 411 related to the lidar beam reflected from the ship among lidar points matched with the image. For example, the apparatus 10 includes lidar points matched to pixels included in an area corresponding to a ship in an image among a plurality of lidar points, and lidar points 411 associated with a lidar beam reflected from a ship. ) can be selected. It is not limited to FIG. 23 , and lidar points 411 related to lidar beams reflected from a ship may be selected in another method in which a segmented image or a detected image is used.

The selection of the lidar points 411 related to the lidar beam reflected from the ship is not limited to the above description and may be implemented in other ways. According to one embodiment, the device 10 may select lidar points 411 associated with a lidar beam reflected from a ship using lidar data. For example, the apparatus 10 may select lidar points 411 related to lidar beams reflected from a ship in consideration of the distribution, number, and the like of lidar data.

The apparatus 10 may determine lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea among lidar points related to the lidar beam reflected from the ship (S2220).

Referring to FIG. 23, the apparatus 10 is related to the lidar beam reflected from the lower area of the ship in contact with the sea using the 3-dimensional coordinate values of the lidar points 411 associated with the lidar beam reflected from the ship. LiDAR points 412 may be determined. According to an embodiment, the apparatus 10 determines the height of a ship in contact with the sea based on the height value (eg, Z value of 3-dimensional coordinate values) of the lidar points 411 related to the lidar beam reflected from the ship. LiDAR points 412 associated with the lidar beam reflected from the lower region may be determined. For example, the apparatus 10 generates lidar lines including lidar points having substantially the same height value, and selects a lidar line including lidar points having the lowest height value among the generated lidar lines. It can be determined by the lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea. Here, the lidar lines may have the number or length of lidar points within a preset range.

In another example, the apparatus 10 includes lidar sensors having a plurality of channels having different vertical angles, and includes lidar points including a lidar point with the lowest height value for each channel, a ship in contact with the sea. It can be determined as lidar points 412 related to the lidar beam reflected from the lower area of .

Determination of the lidar points 412 related to the lidar beam reflected from the lower area of the ship in contact with the sea is not limited to the above description, and a height value of a predetermined range rather than the lidar point of the lowest height value It may be implemented in a different way, such as determined by a LiDAR point having

Return to FIG. 22 and explain.

The apparatus 10 may calculate the sea level height from lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea (S2230).

The device 10 may calculate the sea level height using the 3D coordinate values of lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea. According to an embodiment, the device 10 calculates the sea level height by using height values (eg, Z values among 3-dimensional coordinate values) of lidar points related to lidar beams reflected from the lower area of the ship in contact with the sea. can be calculated For example, the device 10 may calculate the height values of lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea as the height of the sea level.

According to an embodiment, the apparatus 10 may calculate a sea level wave height using 3D coordinate values of lidar points related to lidar beams reflected from the lower area of the ship.

For example, the device 10 may calculate the wave height using height values (eg, Z values among 3D coordinate values) of lidar points related to lidar beams reflected from the lower area of the ship. For example, the device 10 may estimate lidar points related to the lidar beam reflected from the lower area of the ship as a Gaussian distribution, and calculate a range of height values in the estimated distribution as a wave height.

According to one embodiment, device 10 may verify the calculated sea level height. For example, the device 10 is related to the lidar beam reflected from the ship in consideration of characteristics (eg, number, height deviation, distance value, etc.) of lidar points related to the lidar beam reflected from the ship. The validity of the sea level height calculated from lidar points can be checked. For example, the device 10 may be used when the number of lidar points related to the lidar beam reflected from the ship is too small, the deviation of the height value is too large, or the distance value is too close or too far. It can be determined that the effectiveness of the sea level height calculated from LiDAR points related to the IDA beam is low.

Of course, the device 10 is not limited to the above description, and other factors such as calculating the height of the sea level by adjusting the height values of the lidar points related to the lidar beam reflected from the lower area of the ship in contact with the sea to a predetermined condition It is also possible to calculate the sea level height using this method.

The accuracy of sea level height calculated from lidar points may vary depending on circumstances. For example, the device 10 is generally easy to obtain LiDAR data from a wavy sea, but on a day when the weather is very good and the sea is very calm, the LiDAR data obtained from the sea is very small and noise is present There is a possibility that lidar data to be obtained. That is, the accuracy of the sea level height calculated from lidar points related to the lidar beam reflected from the sea may be lower.

For another example, the device 10 is generally easy to obtain lidar data from a ship at a reasonable distance from the lidar sensor, but if the lidar sensor and the ship are too close, the lower area of the ship in contact with the sea is the berth. It can be covered by, and if the lidar sensor and the ship are too far away, there is a possibility that lidar data with a large distance measurement error can be obtained. That is, the accuracy of the sea level height calculated from lidar points related to the lidar beam reflected from the ship may be lower.

Accordingly, the device 10 may perform a sea level calculation method for increasing the accuracy of the calculated sea level height in various situations.

According to one embodiment, the device 10 may calculate the sea level height using the estimated sea level height.

24 is a flowchart of a method for calculating sea level height using estimated sea level height according to an embodiment. Referring to FIG. 24, the method for calculating the sea level height using the estimated sea level height according to an embodiment includes calculating a first estimated sea level height from lidar points related to a lidar beam reflected from the sea (S2310), Calculating a second estimated sea level height from lidar points related to the reflected lidar beam (S2320) and calculating a sea level height using at least one of the first estimated sea level height and the second estimated sea level height (S2330) can include

The apparatus 10 may calculate a first estimated sea level height from lidar points related to lidar beams reflected from the sea (S2310). The apparatus 10 may calculate the sea level height calculated from lidar points related to the lidar beam reflected from the sea as the first estimated sea level height. In this step, since the above-described information can be applied as it is, more detailed information will be omitted.

The apparatus 10 may calculate a second estimated sea level height from lidar points related to the lidar beam reflected from the ship (S2320). The device 10 may calculate the sea level height calculated from lidar points associated with the lidar beam reflected from the ship as the second estimated sea level height. In this step, since the above-described information can be applied as it is, more detailed information will be omitted.

The device 10 may calculate the sea level height using at least one of the first estimated sea level height and the second estimated sea level height (S2330).

According to an embodiment, the device 10 may calculate the sea level height based on a weighted sum of the first estimated sea level height and the second estimated sea level height. For example, the device 10 may calculate the sea level height as a weighted sum of a first estimated sea level height to which a first weight is assigned and a second estimated sea level height to which a second weight is assigned.

According to one embodiment, the device 10 may determine a weight to be given to the estimated sea level height based on the distance from the berth.

For example, in assigning weights to the first estimated sea level height estimated from lidar points related to lidar beams reflected from the sea, the apparatus 10 gives a higher value as the distance of the lidar points to the berth is closer. weight can be assigned. This is because the accuracy of the lidar point associated with the lidar beam reflected from the sea at a distance close to the berth is higher than the accuracy of the lidar point associated with the lidar beam reflected from the sea at a distance from the berth.

For example, in assigning weights to the second estimated sea level height estimated from lidar points related to the lidar beam reflected from the ship, the apparatus 10 increases the weight as the distance of the lidar points from the berth increases. weight can be assigned. Here, the device 10 may assign a higher weight as the distance from the berth increases up to a specific distance, and then assign a lower weight as the distance from the berth increases beyond the specific distance. This is because if the lidar sensor and the ship are too close, the lower area of the ship in contact with the sea may be covered by the berth and there may be noise, and if the lidar sensor and the ship are too far away, the distance measurement error is large. This is because there is a possibility that data can be acquired.

Of course, the device 10 is not limited to the above description, and it is okay to apply weights in other ways, such as giving high weights to sea level heights estimated from LIDAR points within a specific distance range.

According to an embodiment, the device 10 may determine a weight to be assigned to the estimated sea level height based on the calculated wave height. For example, the device 10 may determine a weight applied to the estimated sea level height based on a change in wave height per unit time. This is because there is generally little change in wave height during unit time.

For example, in assigning a weight to the first estimated sea level height estimated from the lidar points associated with the lidar beam reflected from the sea, the apparatus 10 determines the lidar points associated with the lidar beam reflected from the sea. A higher weight can be assigned as the value of the wave height calculated from is less changed over time.

In another example, the apparatus 10 weights the second estimated sea level height estimated from the lidar points associated with the lidar beam reflected from the ship, the lidar point associated with the lidar beam reflected from the ship. The higher the value of the wave height calculated from the field changes over time, the higher the weight can be assigned.

Of course, the device 10 is not limited to the above description, and weighting may be applied in other ways, such as weighting by combining the above methods.

According to one embodiment, the device 10 may calculate the sea level height using the reliability of the LIDAR point.

25 is a flowchart of a method for calculating sea level height using reliability of lidar points according to an embodiment. Referring to FIG. 25, a method for calculating sea level height using reliability of lidar points according to an embodiment includes calculating a first reliability of lidar points related to a lidar beam reflected from the sea (S2410), reflected from a ship. It may include calculating a second reliability of lidar points related to the detected lidar beam (S2420) and calculating a sea level height considering at least one of the first reliability and the second reliability (S2430). Reliability may mean accuracy of whether a lidar point related to a lidar beam reflected from a specific object is actually generated by being reflected from the specific object.

The apparatus 10 may calculate first reliability of lidar points related to the lidar beam reflected from the sea (S2410). For example, the device 10 may use the lidar data to calculate a first reliability of lidar points associated with a lidar beam reflected from the sea.

According to one embodiment, device 10 may calculate a first reliability of lidar points associated with a lidar beam reflected from the sea based on characteristics of the lidar points. The characteristics of lidar points may include, for example, the number of lidar points, deviation of height values, distance values, coordinates, and the like. For example, device 10 may calculate a small first reliability if the number of lidar points associated with a lidar beam reflected from the sea is too small. For another example, the apparatus 10 may calculate a small first reliability when a deviation of height values of lidar points related to a lidar beam reflected from the sea is too large. For another example, the apparatus 10 may calculate a small first reliability when a deviation of distance values of lidar points related to a lidar beam reflected from the sea is too large. The device 10 is not limited to the above description, and may calculate the first reliability in other ways, such as calculating the first reliability of lidar points related to the lidar beam reflected from the sea using the camera image.

The apparatus 10 may calculate the second reliability of lidar points related to the lidar beam reflected from the ship (S2420). For example, the device 10 may calculate the second reliability of lidar points related to lidar beams reflected from the ship using lidar data.

According to an embodiment, the apparatus 10 may calculate a second reliability of lidar points associated with a lidar beam reflected from a ship based on characteristics of the lidar points. The characteristics of lidar points may include, for example, the number of lidar points, deviation of height values, distance values, coordinates, and the like. For example, the apparatus 10 may calculate a small second reliability if the number of lidar points associated with a lidar beam reflected from a ship is too small. For another example, the apparatus 10 may calculate a small second reliability when the deviation of the height values of lidar points related to the lidar beam reflected from the ship is too large. For another example, the apparatus 10 may calculate a small second reliability when a deviation of distance values of lidar points related to a lidar beam reflected from a ship is too large.

According to one embodiment, the device 10 may calculate the second reliability of lidar points associated with the lidar beam reflected from the ship using the camera image. For example, the device 10 may calculate the second reliability of lidar points related to the lidar beam reflected from the vessel by using vessel information obtained from the camera image. Vessel information refers to information about a vessel, for example, whether or not a vessel is detected in an image, information about a region corresponding to a vessel in an image (eg, size, shape, etc.), distance to a vessel obtained from an image, It may include the speed of the ship obtained from the image, occlusion related to the ship, and the like.

For example, when a ship is detected in the image, the device 10 may calculate a large second reliability of lidar points related to a lidar beam reflected from the ship. As another example, the apparatus 10 may calculate a large second reliability of lidar points related to a lidar beam reflected from a vessel when a size of a region corresponding to a vessel in the image falls within a specific size range. As another example, the apparatus 10 may calculate a large second reliability of lidar points related to the lidar beam reflected from the vessel when the distance to the vessel obtained from the image falls within a specific distance range. As another example, the apparatus 10 may calculate a large second reliability of lidar points associated with a lidar beam reflected from a vessel when there is no occlusion associated with the vessel obtained from the image.

The device 10 is not limited to the above description, and may calculate the second reliability in other ways, such as calculating the second reliability by combining the above methods.

The device 10 may calculate the sea level height in consideration of at least one of the first reliability level and the second reliability level (S2430).

According to an embodiment, the device 10 may calculate the sea level height by comparing the first reliability level and the second reliability level with each other. For example, when the first reliability is less than the second reliability, the apparatus 10 may calculate the sea level height from lidar points related to the lidar beam reflected from the ship. For another example, when the first reliability is greater than the second reliability, the device 10 may calculate the sea level height from lidar points related to the lidar beam reflected from the sea.

According to an embodiment, the device 10 may calculate the sea level height by comparing at least one of the first reliability level and the second reliability level with a critical level of reliability. For example, when the first reliability is greater than or equal to a threshold reliability, the device 10 may calculate the sea level height from lidar points associated with a lidar beam reflected from the sea. For another example, when the second reliability is greater than or equal to the threshold reliability, the apparatus 10 may calculate the sea level height from lidar points related to lidar beams reflected from the ship.

The apparatus 10 is not limited to the above description, and calculates the sea level height by reflecting sea level heights calculated from lidar points related to lidar beams reflected from the sea and ships as a ratio of the first reliability and the second reliability. It is also possible to calculate sea level height in other ways, such as

According to one embodiment, the device 10 may calculate the sea level height using a result of analyzing the camera image. For example, the device 10 may recognize a situation through a camera image and calculate the sea level height using LIDAR data suitable for the recognized situation.

26 is a flowchart of a method for calculating the height of sea level using a result of analyzing a camera image according to an embodiment. Referring to FIG. 26, the method for calculating the sea level height using the result of analyzing the camera image according to an embodiment includes analyzing the camera image (S2510) and calculating the sea level height from lidar data using the image analyzed result (S2510). (S2520) may be included.

The device 10 may analyze the camera image (S2510). For example, the device 10 may analyze a camera image and determine whether a ship exists.

According to one embodiment, the device 10 may obtain vessel information related to the vessel from the camera image. Vessel information refers to information about a vessel, for example, whether or not a vessel is detected in an image, information about a region corresponding to a vessel in an image (eg, size, shape, etc.), distance to a vessel obtained from an image, It may include the speed of the ship obtained from the image, occlusion related to the ship, and the like. For example, the device 10 may obtain ship information from an image using an artificial neural network. For example, the device 10 generates a segmentation image from an image using an artificial neural network, detects an area where a pixel labeled with object information indicating a ship as an area corresponding to the ship is located, and from this, the ship information. can be obtained. Here, the vessel information includes whether or not a vessel is detected, information on a region corresponding to the vessel in the image (eg, size, shape, etc.), distance to the vessel obtained from the image, speed of the vessel obtained from the image, and information related to the vessel. It may include occlusion and the like. Since the above-described information can be applied to the acquisition of vessel information as it is, more detailed information will be omitted.

The apparatus 10 is not limited to the above description, and may obtain vessel information from an image in another manner, such as determining a region corresponding to a vessel through image detection.

The device 10 may calculate the sea level height from lidar data using the image analysis result (S2520). For example, device 10 may calculate sea level height from different lidar points depending on whether or not a vessel is present. For example, the device 10 calculates the sea level height from a lidar point related to a lidar beam reflected from the ship when a ship exists according to the image analysis result, and from the sea when the ship does not exist according to the image analysis result. From the lidar points associated with the reflected lidar beam, the sea level height can be calculated.

According to an embodiment, the device 10 may calculate the sea level height from LIDAR data using vessel information obtained from a camera image. For example, the device 10 may calculate the sea level height from a specific lidar point based on vessel information obtained from a camera image.

According to an embodiment, the device 10 may calculate the sea level height from LIDAR data based on whether a predetermined condition is satisfied by ship information obtained from a camera image. For example, the device 10 may calculate the sea level height from a specific LIDAR point according to whether a predetermined condition is satisfied.

For example, the device 10 may calculate the sea level height from a lidar point related to a lidar beam reflected from a vessel when a predetermined condition is satisfied by the vessel information. Here, the predetermined conditions include a condition in which a ship is detected, a condition in which the size of an area corresponding to a ship falls within a specific range, a condition in which a distance to a ship falls within a specific range, a condition in which occlusion related to a ship is not detected, and the like. can do. That is, the device 10 calculates the sea level height from a lidar point associated with a lidar beam reflected from a ship when a vessel is detected from the camera image or when the detected vessel meets a specific condition ensuring a certain level of accuracy. can do.

As another example, the device 10 may calculate the sea level height from a lidar point related to a lidar beam reflected from the sea when a predetermined condition is not satisfied by the vessel information. Here, the preset conditions include a condition in which a ship is not detected, a condition in which the size of an area corresponding to a ship does not belong to a specific range, a condition in which a distance to a ship does not belong to a specific range, a condition in which occlusion related to a ship is detected, etc. can include That is, the device 10 determines the height of the sea level from the lidar point associated with the lidar beam reflected from the sea if no vessel is detected from the camera image or the detected vessel does not meet a specific condition to ensure a certain level of accuracy. can be calculated.

In addition, the device 10 may calculate the sea level height from lidar data in consideration of vessel information obtained from subsequent images. For example, the device 10 calculates the sea level height from a lidar point associated with a lidar beam reflected from a ship when a preset condition is satisfied, and the preset condition is not satisfied by ship information obtained from a subsequent image. In this case, the calculated sea level height may be maintained as the sea level level. In another example, the device 10 calculates the sea level height from a lidar point associated with a lidar beam reflected from a ship when a preset condition is satisfied, and the preset condition is also satisfied by ship information obtained from a subsequent image. In this case, the calculated sea level height may be updated in consideration of the sea level height calculated from subsequent LIDAR data. Here, the subsequent image and the subsequent LIDAR data may correspond to each other, for example, acquired at the same time point.

The method of calculating the sea level height from lidar data using image analysis results is not limited to the above description, and may be implemented in other ways, such as adding the detected shape of a ship to a predetermined condition.

According to one embodiment, the device 10 may monitor a ship or a port in consideration of the calculated sea level.

27 is a flowchart of a ship monitoring method considering the calculated sea level height according to an embodiment. Referring to FIG. 27 , the ship monitoring method considering the calculated sea level height according to an embodiment may include calculating the sea level height (S2000) and monitoring the ship considering the calculated sea level height (S3000). .

The device 10 may calculate the sea level height (S2000). In this step, since the above-described information can be applied as it is, more detailed information will be omitted.

The device 10 may monitor the ship in consideration of the calculated sea level height (S3000).

According to one embodiment, the device 10 may provide berthing guide information calculated in consideration of sea level height information reflecting the height of sea level.

28 is a view related to conversion of viewpoints at different reference planes/reference heights. Even in the same image, images converted according to criteria of viewpoint conversion may be different. For example, the left images of FIG. 28 (b) and (c) are the same, but the images converted according to the criterion at the time of converting the viewpoint are ships for the quay as shown in the right images of FIG. 28 (b) and (c), respectively. The relative position of may appear different. In this case, since berthing guide information of the target ship is also different, it may be important to set the reference plane/reference height in order to calculate accurate berthing guide information, and this reference plane/reference height may depend on the sea level height. Therefore, the accuracy of the berthing guide information can be increased if the viewpoint conversion is performed in consideration of the sea level height information.

As shown in FIG. 28, the reference plane may be a plane parallel to the sea level, but is not limited thereto.

The device 10 may perform viewpoint conversion by updating viewpoint conversion information according to the sea level height information whenever berthing guide information considering the sea level height is calculated. Alternatively, the viewpoint conversion information may be updated by reflecting the sea level at a predetermined time interval. Here, it is not necessary to update all parameters of the view conversion information, but only some of them may be updated.

29 is a view related to image correction for calculation of berthing guide information on sea level/sea level according to an embodiment, wherein a segmentation image is converted to a viewpoint with respect to a reference plane/reference height, and sea level height information is converted into the converted segmentation image. It is to calculate the berthing guide information after taking into account and correcting it. The greater the difference between the sea level height and the reference height, the greater the difference between images before and after correction.

However, ship or harbor monitoring considering sea level is not limited to the above description. For example, after performing a viewpoint conversion step in consideration of sea level information, berthing guide information is calculated therefrom, and then considering sea level height information again, It may be implemented in other ways, such as correcting eyepiece guide information.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

10: monitoring device
100: sensor module
110: communication department
120: control unit
130: camera
200: control module
210: communication department
220: control unit
300: communication module

Claims (19)

As a port monitoring method,
acquiring an image taken by a camera;
obtaining lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera;
detecting a first area corresponding to a sea in the image and a second area corresponding to a ship in the image;
selecting first lidar points related to lidar beams reflected from the sea in consideration of pixel positions of pixels included in the first area from among the plurality of lidar points;
calculating a first estimated sea level height using the first lidar points;
selecting second lidar points related to the lidar beam reflected from the ship from among the plurality of lidar points in consideration of pixel positions of pixels included in the second area;
Determining third lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from the second lidar points based on height values of the second lidar points;
calculating a second estimated sea level height using the third lidar points; and
Determining a sea level height by considering both the first estimated sea level height and the second estimated sea level height;
Port monitoring methods.
According to claim 1,
The first estimated sea level height is an average of height values of the first LiDAR points.
Port monitoring methods.
According to claim 1,
Determining the third lidar points
generating lidar lines including lidar points having substantially the same height value from the second lidar points; and
Selecting a lidar line having the lowest height value among the lidar lines as the third lidar points
Port monitoring methods.
According to claim 3,
The lidar line has a number and / or length of lidar points in a predetermined range
Port monitoring methods.
According to claim 1,
The sea level height is a weighted sum of the first estimated sea level height and the second estimated sea level height
Port monitoring methods.
According to claim 5,
A first weight assigned to the first estimated sea level height is determined based on a distance value of the first lidar points,
A second weight assigned to the second estimated sea level height is determined based on a distance value of the second LiDAR points,
Port monitoring methods.
According to claim 5,
Calculating a first wave height from the first lidar points; and
Further comprising: calculating a second wave height from the second lidar points;
A first weight assigned to the first estimated sea level height is determined based on a change in the first wave height per unit time,
A second weight assigned to the second estimated sea level height is determined based on a change in the second wave height per unit time.
Port monitoring methods.
According to claim 1,
The first area and the second area are detected using an artificial neural network,
The artificial neural network is learned using a learning set including a plurality of training images and object information labeled on pixels of the plurality of training images.
Port monitoring methods.
According to claim 1,
The first lidar points and / or the second lidar points are determined in consideration of characteristics of lidar points,
The characteristics of the lidar points include at least one of the number of lidar points, a deviation of the height value, and a distance value.
Port monitoring methods.
According to claim 1,
The second lidar points are determined based on vessel information obtained from the image,
The vessel information includes at least one of detection of the second area, size of the second area, distance to the vessel, and detection of occlusion related to the vessel.
Port monitoring methods.
As a port monitoring method,
acquiring an image taken by a camera;
obtaining lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera;
detecting a first area corresponding to a sea in the image and a second area corresponding to a ship in the image;
selecting first lidar points related to lidar beams reflected from the sea in consideration of pixel positions of pixels included in the first area from among the plurality of lidar points;
determining first reliability of the first lidar points based on characteristics of the lidar points;
selecting second lidar points related to the lidar beam reflected from the ship from among the plurality of lidar points in consideration of pixel positions of pixels included in the second area;
determining second reliability of the second lidar points based on vessel information obtained from the image; and
Estimating a sea level height using at least one of the first lidar points and the second lidar points in consideration of the first reliability level and the second reliability level; including
Port monitoring methods.
According to claim 11,
The characteristics of the lidar points include at least one of the number of lidar points, a deviation of the height value, and a distance value.
Port monitoring methods.
According to claim 11,
The vessel information includes at least one of detection of the second area, size of the second area, distance to the vessel, and detection of occlusion related to the vessel.
Port monitoring methods.
As a port monitoring method,
Acquiring lidar data including a plurality of lidar points obtained by lidar sensors;
selecting first lidar points related to lidar beams reflected from the sea from among the plurality of lidar points;
selecting second lidar points related to the lidar beam reflected from the ship from among the plurality of lidar points;
The first lidar points and the second lidar points for estimating the sea level height based on at least one of the number of the first lidar points and the second lidar points, a deviation of a height value, and a distance value determining the reliability of each; and
estimating a sea level height using at least one of the first lidar points and the second lidar points in consideration of the reliability of the first lidar points and the second lidar points; containing
Port monitoring methods.
As a port monitoring method,
acquiring an image taken by a camera;
obtaining lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera;
Detecting a ship area corresponding to the ship in the image using an artificial neural network;
The artificial neural network is learned using a learning set including a plurality of training images and object information labeled on pixels of the plurality of training images,
At least some of the plurality of training images include ships and seas;
The object information reflects the object type,
pixels of the vessel are labeled with the object information of the object type indicating the vessel;
selecting first lidar points related to a lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the vessel area;
Determining second lidar points related to a lidar beam reflected from a lower area of the ship in contact with the sea from the first lidar points based on height values of the first lidar points; and
Estimating the sea level height using the second lidar points; including
Port monitoring methods.
According to claim 15,
Further comprising verifying the sea level height based on at least one of the number of first lidar points, a deviation of a height value, and a distance value.
Port monitoring methods.
As a port monitoring method,
acquiring an image taken by a camera;
obtaining lidar data including a plurality of lidar points obtained by a lidar sensor having a viewing angle at least partially overlapping with a viewing angle of the camera;
detecting a ship area corresponding to a ship in the image;
selecting first lidar points related to a lidar beam reflected from the vessel from among the plurality of lidar points in consideration of pixel positions of pixels included in the vessel area;
Obtaining vessel information related to the vessel from the image;
the vessel information includes at least one of detection of the vessel, size of the area of the vessel, distance to the vessel, and detection of occlusion related to the vessel; and
Including, when a predetermined condition is satisfied by the vessel information, estimating the sea level height using the first lidar point,
detecting a sea area corresponding to the sea in the image;
selecting second lidar points related to lidar beams reflected from the sea in consideration of pixel positions of pixels included in the sea area from among the plurality of lidar points; and
Estimating the sea level height using the second lidar point when the predetermined condition is not satisfied by the vessel information; further comprising
Port monitoring methods.
delete A computer-readable recording medium in which a program for performing the method of any one of claims 1 to 17 is recorded.

Images