KR102156936B1 - Image processing-based water surface observation system and method for marine environment monitoring - Google Patents

Abstract

Disclosed is an image analysis-based water surface observation system for marine environment monitoring. The image analysis-based water surface observation system comprises: a surface image observation device controlling an image acquisition timing and a sampling interval according to a purpose of ocean observation, analyzing the acquired water surface image of waves, surface flow velocity, surface water temperature, shoreline, suspended sand, and white waves and deriving observation results including observation images, and transmitting the observation results to a server; a data collection and visualization server collecting and visualizing observation results obtained from the surface image observation device; and a user terminal having a program for real-time confirmation of the observation results and photographs of the waves, the surface flow velocity, the surface water temperature, the shoreline, the suspended sand, and the white waves. Surface images are acquired by using the surface image observation device in fields of marine environment monitoring of sea, river, and coastal areas and coastal sedimentation/sediment monitoring. A tool is used to analyze changes in the waves, the surface flow velocity, the surface water temperature, shoreline observation, and slope change. In the field of marine environment monitoring, wave observation, shoreline observation, floating sand observation, and white cap observation analysis became possible.

Description

Image processing-based water surface observation system and method for marine environment monitoring}

The present invention relates to an image analysis-based surface observation system and method for monitoring a marine environment, and more particularly, to set an image acquisition time and sampling interval, acquire a surface image, and analyze the acquired surface image according to an observation purpose. ), surface flow velocity (m/s), surface water temperature (℃), shoreline, floating sand, white cap, etc., and observation results including observation images are transmitted to the server, and observation results are stored and observed on the server. It relates to an image analysis-based water surface observation system and method for monitoring the marine environment, which can be checked with a PC or smartphone in real time in connection with information display and server.

Ocean environment observation can be largely divided into direct observation and indirect observation, and direct observation and indirect observation can be divided into contact and non-contact with the object to be observed. Direct observation methods include: i) installing equipment underwater, ii) using buoys, and iii) using ships and diving. Indirect observation methods include: iv) using an artificial satellite, and iv) using aerial photographs and video images.

Direct observation has the advantage of high accuracy of observation, so it is currently most often used, but it is expensive to install and maintain observation equipment, and has a high possibility of equipment loss. Indirect observation, on the other hand, compensates for the disadvantages of direct observation, but has a disadvantage in that the accuracy of observation is poor, but the accuracy is greatly improved due to the development of analysis techniques (signal processing, image processing, etc.) and is currently expanding.

Investigation using a submersible or buoy is still not easy due to various reasons such as cost/maintenance.

Survey by buoy is a method of collecting ocean data continuously over time at a point in the sea. This time-series data is very useful for scientists who require ocean information that changes from time to time. useful.

Buoys are largely divided into a mooring buoy that can continuously observe meteorological and ocean data at a peak, and a drift buoy that moves along an ocean current and transmits observation data.

Existing mooring buoys are fixed and installed with one anchor, and are connected by one mooring line, so the risk of loss is high and accurate measurement is difficult in oceans with strong currents or in oceans with large tidal differences.

In addition, since the sensor or the like cannot be automatically adjusted according to the depth of water to be measured, when observation is finished at the point where it is installed, there is an inconvenience of having to go and adjust again.

As prior art 1 related to this, Patent Publication No. 10-2017-0065653 discloses a "marine environment monitoring system".

The system 100 for monitoring the marine environment performs synchronous detection of an object in the marine environment, transmits a plurality of sensor signals respectively related to the position of the object in the marine environment through the communication network 107, and transmits a synchronization signal. A plurality of wireless detection and ranging devices (101, 103, 105), configured to receive a plurality of wireless detection and ranging devices (101, 103, 105) configured to synchronize an operation according to a synchronization signal. Generates a synchronization signal for synchronizing the operation of two wireless detection and ranging devices, and a plurality of wireless detection and ranging devices (101, 103, 105), and a plurality of wireless detection of the synchronization signal via the communication network (107) And a synchronization source (109) configured to provide to the ranging device (101, 103, 105), and a plurality of sensor signals from the plurality of wireless detection and ranging devices (101, 103, 105), the marine environment And a processing device 111, configured to determine a position of an object in the at based on the plurality of sensor signals.

As a related prior art 2, a "marine environment monitoring system" is registered in patent registration number 10-11094600000.

The marine environment monitoring system uses Winry that can automatically or manually lift the underwater sensor unit to enable regular monitoring of the marine environment even in the sea with strong flow rates (East Sea) or the sea with large tidal differences (West Sea). An underwater sensor unit capable of measuring specific physical properties; A body having an inner space in which the underwater sensor unit can be lifted; An upper frame part located above the body and on which various sensors, antennas, etc. are mounted; And it provides a marine environment monitoring system including a winch that is installed on the body or the upper frame portion that can automatically or manually lift the underwater sensor unit. In addition, self-power generation is possible by having a wind power generator or a solar power generator, and various wireless communication support is possible.

1 is a cross-sectional view of a conventional marine environment monitoring system.

The marine environment monitoring system includes a body 100, an underwater sensor unit 200, and an upper frame unit 300.

The underwater sensor unit 200 may include various underwater sensors 202 for monitoring the marine environment.

For example, one or more turbidity sensors can be installed to measure the turbidity of each layer of the mud layer, and sensors for measuring various salinity, pH, water temperature, and flow rate can be installed.

The general buoy or marine environment monitoring system is configured to operate more than 1m away from the sea floor for the safety of equipment (underwater sensor unit), but since the sediment usually moves within 1m from the sea floor, the existing marine environment monitoring The system is very difficult to observe the sediments on the sea floor.

One or more turbidity sensors 207 and 208 may be mounted at the bottom of the underwater sensor unit 200 through separate connection lines 207a and 208a to facilitate observation.

A CTD (Conductivity, Temperature, Depth) sensor or simply a depth sensor may be installed.

The depth sensor is a sensor that measures the water pressure and calculates the depth according to the hydrostatic pressure equation.

Existing ocean observations are performed by installing separate equipment for each item.

The use of separate marine observation equipment is expensive.

In the case of observation with each independent ocean observation equipment, time synchronization is difficult, and in order to determine the cause of a specific phenomenon in the ocean, each observation item must be observed at the same time. Observation items can be set according to the purpose of observation.

The observation items of the image analysis unit are developed in the form of tools for each item, and can be selected according to need. For example, when applied to the field of coastal sedimentation and sedimentation monitoring, only necessary observations such as waves, surface flow rates, suspended solids, and shorelines can be performed.

However, the existing ocean observation system uses a tool that analyzes waves, surface flow velocity, shoreline observation, and key changes in order to apply to the field of monitoring sea, river, and coastal coastal settlement and sedimentation by utilizing image observation devices for ocean observation. In the field of marine environment monitoring, no tools were provided to analyze wave observations, shoreline observations, suspended solids observations, and white caps.

Patent Publication No. 10-2017-0065653 (Publication Date June 13, 2017), "Marine Environment Monitoring System", Maritime Radar Systems Limited G.M.S. Global Maritime Services Limited Patent registration number 10-11094600000 (registration date January 18, 2012), "Marine Environment Monitoring System", Ocean Tech Co., Ltd., Korea Ocean Research Institute

An object of the present invention for solving the above problems is a surface image observation device equipped with a camera for image acquisition and an image analysis program, a server for data collection and visualization, an image analysis program on a smartphone, and a marine observation purpose Set the image acquisition time and sampling interval according to the conditions, acquire the surface image, and analyze the acquired surface image to observe the wave, surface flow velocity (m/s), surface water temperature, shoreline, suspended sand, white cap, etc. It transmits the observation image and observation result to the server, stores the observation result and displays the observation information, and can be checked with a PC or smartphone in real time in connection with the server, and the wave of the sea, river, shore, and surface velocity (m /s), surface water temperature, shoreline, change of sand, floating sand, white cap, etc., provides an image analysis-based surface observation system for marine environment monitoring.

Another object of the present invention is to provide an image analysis-based water surface observation method for monitoring a marine environment.

In order to achieve the object of the present invention, an image analysis-based surface observation system for marine environment monitoring controls an image acquisition timing and sampling interval according to an observation purpose for marine environment monitoring, and transmits the acquired surface image to a wave, Analyze surface flow velocity, surface water temperature, shoreline, suspended sand, and white cap, derive observation results including surface flow velocity (m/s) and observation images, and transmit observation results including the observation images to the server A sleep image observation device; A data collection and visualization server for collecting and visualizing observation results obtained from the surface image observation device; And a user terminal having a program that checks in real time observation results of waves, surface flow rates, surface water temperatures, shorelines, floating sands, and white waves,
In the blue observation, a stereo image of the water surface acquired by two cameras is analyzed and reconstructed into three-dimensional phase data composed of x, y, and z to observe a wave having a wave height, a period, and a wave direction of the wave wave,
In the shoreline observation, an average image and a dispersion image are produced by overlapping images acquired from two cameras, and the boundary between the land surface and sea level is extracted for each cross-shore direction of the coast,
The average image is an image obtained by superimposing a plurality of images to visualize the average of the pixel intensity of each pixel, and the distributed image is an image obtained by superimposing a plurality of images to visualize the standard deviation of the pixel intensity of each pixel,
The shoreline observation provides a shoreline observation result by extracting each from the average image and/or the variance image,
The white wave observation is used by converting the acquired image of the sea water surface into a gray scale image, and the white cap is a phenomenon in which the blue head is broken at the sea level, and in this process, white appears due to the mixture of sea water and gas. , When the white wave occurs, the white wave in the image has a relatively higher pixel intensity than the surrounding seawater surface.
In the process of observing white waves, Step A: Acquisition of time series images, Step B: Change the acquired sea surface image to a gray scale image, Steps C to D: Analyze the observed image for more than 10 minutes to determine the criteria for separating sea water and white waves. Determining the threshold value, Step E to Step F: From each image, based on the threshold value, a pixel with a white wave is extracted and the white wave is observed.

The temporal resolution of the surface image observation device is 1 ~ 15 Hz, and the spatial resolution can be adjusted to 0.1 ~ 2.0m.

The user terminal uses a PC or smartphone.

The surface image observation device is used as a camera sea level image observation IoT device for monitoring the marine environment,

A camera unit that acquires a surface image for monitoring the marine environment of the sea, river, and shore; A controller configured to set an image acquisition timing and a sampling frequency, transmit the observation result to a server, and perform overall control of the system; An image analysis unit connected to the control unit and analyzing the acquired water surface image according to observation items of blue, surface flow velocity, surface water temperature, shoreline, floating sand, and white wave, and providing the observation result; A storage unit connected to the control unit and configured to store observation results; And a communication unit connected to the control unit and having a 3G modem or an LTE 4G/5G modem to transmit the observation result to the server.

The surface image observation device is connected to the control unit, and further includes a display unit for displaying measurement time information, the observation result, and observation photograph.

For the wave observation, three-dimensional surface data is generated from the surface image (stereo image) captured by two cameras, and then the wave height, period, and direction of the short-period and long-period waves are observed through separate post-processing (blue statistics). do.

The blue observation is performed by analyzing a stereo image of the water surface acquired by two cameras and reconstructing it into three-dimensional phase data composed of x, y, and z to observe the blue (wave height, period, wave direction of the blue),

The blue observation is

Step A: Acquire a stereo image time series,

Step B: Extract keypoints of feature points using SURF (Speeded Up Robust Features) algorithm from each image,

Step C: Match the keypoints of the feature points extracted from each image,

Step D: Eliminate matching errors (Mismatching keypoints, keypoints with large errors, etc.) through RANSAC-based epipolar filter for keypoints of mutually matched feature points,

Step E: Rotation matrix between images, calculation of translation vector,

Step F: The image is reconstructed into 3D information (x, y, z) and the result is visualized to observe blue.

In the shoreline observation, an average image and a distributed image are produced by overlapping images acquired from a camera, and the boundary between the land surface and sea level is extracted for each cross-shore direction of the coast,

The average image is an image obtained by superimposing a plurality of images to visualize the average of the pixel intensity of each pixel, and the distributed image is an image obtained by superimposing a plurality of images to visualize the standard deviation of the pixel intensity of each pixel,

The shoreline observation provides a shoreline observation result by extracting each from the average image and/or the variance image,

The coastline observation process

Step A: Time series image acquisition,

Step B: Create an average image by overlapping two images (an image that visualizes the average of pixel intensity by overlapping images) and a distributed image (an image that visualizes the standard deviation of pixel intensity by overlapping images).

Step C: Geometric correction based on RTK (Real Time Kinematic, real-time moving positioning) survey data,

Step D: Remove noise in the analysis area by applying a threshold that is a reference when extracting the shoreline (primary noise removal),

Step E: Extracting the section where the coastline can exist (secondary noise removal),

Step F: Extraction of shoreline for each baseline configured at regular intervals, and

Steps G~H: Connect the points extracted from each baseline and visualize the shoreline result to extract the boundary line between the land surface and sea level.

The white wave observation is used by converting the acquired image of the sea water surface into a gray scale image, and the white cap is a phenomenon in which the blue head is broken at the sea level, and in this process, white appears due to the mixture of sea water and gas. , When the white wave occurs, the white wave in the image has a relatively higher pixel intensity than the surrounding seawater surface.

The white wave observation process

* Step A: Time series image acquisition,

* Step B: Change the acquired seawater surface image to a gray scale image,

* Step C~D: Determine the boundary value that is the basis for separating seawater and white waves by analyzing images observed for 10 minutes or longer.

* Steps E to F: The white wave is observed by extracting the pixels in which the white wave has occurred based on the boundary value in each image.

The suspended solid observation is performed by converting the pixel intensity of RGB according to the concentration of the suspended solid to the suspended solid concentration in the surface image of the seawater surface to observe the suspended solid,

The process of observation of suspended solids

* Step A: Time series image acquisition,

* Step B: Separate each band (R-band, G-band, B-band) of the RGB band image and perform geometric correction based on RTK (Real Time Kinematic, real-time moving positioning) survey data,

* Step C: Selecting the band to be applied in the target sea area, the selection criterion is the band that responds most sensitively to suspended sand, and the applied band varies depending on the water quality and ocean color of the target sea area.

* Step D: Calculate the pixel intensity of the applied band as the reference pixel intensity ratio,

* Step E: derive a correlation equation between the point-observed suspended solid concentration and pixel intensity ratio in the target sea area,

* Step F: The observed suspended solid concentration is expanded into a two-dimensional distribution by applying the pixel intensity ratio correlation equation, and the result is visualized to observe the suspended solid.

In the observation of the surface flow velocity, the surface flow velocity is observed by observing the surface image of the sea level, tracking the movement path of the feature points of the water bubbles in the surface layer of the surface image, and measuring the moving distance per unit time.

The surface water temperature observation uses an infrared image (near infrared ray or thermal imaging camera image) to observe the surface water temperature at sea level.

For the observation of the surface image, an RGB image, a grayscale image, a stereo image, and an infrared image are selected and used as necessary.

In order to achieve another object of the present invention, an image analysis-based surface observation method for monitoring a marine environment includes an image of a surface image observation device suitable for an observation purpose in order to obtain a surface image for monitoring the marine environment of the sea, river, and shore. Acquisition timing and sampling interval are set, and blue observation, surface flow velocity, surface water temperature, shoreline observation, suspended solids observation, and white cap for water surface images of sea/river/shore through at least one camera of the surface image observation device. ) 1 step of controlling the acquisition timing and acquisition method of the coastal image of the observation; A second step of acquiring a surface image of a camera of the surface image observation device; A third step of analyzing the image of the surface image acquired through the camera of the surface image observation apparatus for blue observation, surface flow velocity, surface water temperature, shoreline observation, floating sand observation, and white cap observation; The fourth step of collecting data from the surface image observing device 100 and transmitting the observation results including observation images about a wave observation, a shoreline observation, a floating sand observation, and a white cap to monitor the marine environment ; The data collection and visualization server stores wave observation, shoreline observation, floating sand observation, and white cap observation results, and displays the information in real time and visually visualizes them; And a sixth step of interlocking with the data collection and visualization server to check in real time the results of observation of blue waves, observations of shorelines, observations of suspended solids, and observations of white caps through a wired/wireless communication network.

The method further includes the step of observing the surface image and tracking the movement paths of the feature points of the water bubbles in the surface layer of the surface image, measuring the moving distance per unit time, and observing the surface flow velocity.

The method further includes the step of observing the surface water temperature at the sea level using an infrared image (near infrared ray or a thermal imaging camera image) when observing the surface water temperature.

The user terminal uses a PC or a smartphone,

For the observation of the water surface, an RGB image, a grayscale image, and an infrared image are selected and used as necessary.

The image analysis-based water surface observation system for marine environment monitoring of the present invention includes a water surface image observation device equipped with an ocean observation program equipped with a camera, and data collection and analysis of the image processing-based surface observation system, and image analysis on a server, PC, and smartphone. In order to have a program and apply it to the field of monitoring of sedimentation and sedimentation along the sea, river, and shore, a tool that analyzes changes in waves, surface flow rates, shoreline observations, and slopes is used. It is effective in providing water temperature, shoreline observation, floating sand observation, and white cap observation analysis.

In the field of marine environment monitoring of sea, river and coastal areas and coastal sedimentation and sedimentation monitoring, an image processing-based water surface observation device is used to analyze waves, surface flow velocity, surface water temperature, shoreline observation, and key changes, and marine environment monitoring In the field, wave observation, surface flow velocity, surface water temperature, shoreline observation, suspended solids observation, and white cap observation analysis became possible.

In addition, an image processing-based water surface observation device will be developed for surface flow velocity and surface water temperature observation.

1 is a cross-sectional view of a conventional marine environment monitoring system.
2 is a block diagram of an image analysis-based surface observation system for monitoring a marine environment according to the present invention.
3 is a photograph of a control board and a control program of the image observation apparatus.
4 is a photograph of an image acquisition unit 1 and an image acquisition unit 2 during ocean exploration.
5 is a diagram illustrating a structure of a wave.
6 is a photograph of a wave observation, a shoreline observation, a floating sand observation, and a white cap observation in the marine environment monitoring field according to an embodiment of the present invention.
7 is a diagram showing a wave observation process.
8 shows an average image and a variance image.
9 is a diagram showing a coastline observation process.
10 shows pixel intensity characteristics when a white wave is generated.
11 is a diagram showing a process of observing a white wave.
12 is a diagram showing a process of observing suspended solids.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram of an image analysis-based surface observation system for monitoring a marine environment according to the present invention.

3 is a photograph of a control board and a control program of an image observation apparatus, and FIG. 4 is a photograph of an image acquisition unit 1 and an image acquisition unit 2 during ocean exploration.

The image analysis-based surface observation system for marine environment monitoring of the present invention systemizes the process of acquiring a surface image, observation through image analysis, and real-time confirmation of the observation result, and more specifically, when the image is acquired according to the purpose of marine observation. And sampling interval setting, sea/river/coastal surface image capture, and analysis of acquired water surface images by observation items such as blue observation, surface flow velocity, surface water temperature, shoreline observation, floating sand observation, and white cap observation. And it transmits measurement time information, observation images (multiple images) and analysis results to the server, stores the analysis results on the server and displays observation information, and provides a system that can be checked with a PC or smartphone in real time by interlocking with the server.

The image analysis-based surface image observation system for marine environment monitoring includes a "image processing-based surface observation device 100" equipped with an ocean observation program equipped with a camera, and a data collection and visualization server and a PC of the image analysis-based surface observation system, Equipped with an image analysis program on a smartphone, it provides observation results and observation photos such as wave, surface flow rate (m/s), surface water temperature (℃), shoreline, floating sand, and white cap.

In addition, the image processing-based water surface observing apparatus 100 will be developed for observing surface flow velocity and surface water temperature.

The process of image acquisition → image analysis → real-time confirmation was made into a system, and the image processing-based water surface observation device developed a program for each category of blue, surface flow rate, surface water temperature, shoreline, floating sand, and white cap. I did.

The image processing-based surface observation device has been developed for wave, shoreline, floating sand, and white wave observation for marine environment monitoring. Surface flow velocity and surface water temperature are currently being developed, and observation items will be continuously developed and added.

The image analysis-based surface observation system for monitoring the marine environment of the present invention

In order to monitor the marine environment, the image acquisition timing and sampling interval are controlled according to the observation purpose, and the acquired water surface image is converted to wave, surface flow rate (m/s), surface water temperature (℃), coastline, suspended sand, and white wave ( white cap), derives observation results including observation images, and transmits measurement time information and observation results including the observation images to a server;

A data collection and visualization server 200 for collecting and visualizing observation results obtained from the surface image observation apparatus 100; And

A program that checks the observation results in real time, including observation images and observation photographs (images) of wave, surface flow velocity (m/s), surface water temperature (℃), shoreline, floating sand, and white cap. It includes a user terminal 300 provided.

The temporal resolution of the surface image observation device is 1 to 15 Hz, and the spatial resolution is adjustable to 0.1 to 2.0 m.

For example, if you take a 15Hz temporal resolution of a surface image observation device for 10 minutes, an image of 10 minutes x 60 seconds x 15 frames = 9000 frames is captured.

The user terminal 300 uses a PC or smartphone, and has an image analysis program for viewing the results of analysis of blue, surface flow rate, surface water temperature, coastline, suspended sand, white cap and observation photos in the marine environment monitoring field. do.

The image processing-based surface image observation apparatus 100 is used as a camera image observation IoT device for monitoring the marine environment,

A camera unit 110 that acquires a surface image for monitoring the marine environment of the sea, river, and shore; A control unit 111 for setting an image acquisition timing and sampling frequency, transmitting the observation result to a server, and performing overall control of the system; It is connected to the control unit 111 and analyzes the acquired water surface image according to each observation item of blue, surface flow velocity, surface water temperature, shoreline, floating sand, and white wave, and includes, for example, observation images and observation photos of 10 minutes. An image analysis unit 112 for providing the result of the observed observation; A storage unit 113 connected to the control unit 111 and storing observation results; And a communication unit 120 connected to the control unit 111 and having a 3G modem or an LTE 4G/5G modem to transmit the observation result to the server.

The surface image observation apparatus 100 is connected to the control unit 111 and further includes a timer for counting time.

The surface image observation apparatus 100 is connected to the control unit 111 and further includes a display unit for displaying measurement time information, the observation result, and observation photograph.

The image analysis SW of the image analysis unit 112 uses image processing in any one of gray scale, RGB, and YCbCr conversion of water surface images of the sea, river, and shore.

For example, the image analysis SW converts an RGB image into a gray scale image, uses a gray scale image to buffer and stores the camera input image, provides image processing and image analysis functions, and provides a region of interest ( A histogram of the image of the region of interest (ROI) using a specific threshold by an experiment in which the ROI (region of interest) is converted into grayscale and the outline of the region of interest (ROI) is extracted Calculate the [pixel value of each pixel of the x-axis image, the number of corresponding pixel values on the y-axis (frequency)], and then binarize the ROI image to 0,1 based on the threshold value using the Ostu algorithm. , Pre-processing of the image of the ROI is performed through histogram equalization, and the x-direction differential and Calculate the derivative in the y-direction, multiply the weight of the sobel mask by the pixel value of the image, and detect the edge of the image of the ROI (pixels located at the boundary of the object area, background area) by convolution adding these and a specific threshold value. Is applied to detect the contours (eg, coastline) of the objects of the edge image, and extract shape features.

In the case of using the threshold method, when the pixel values f(x,y) are separated into an object area and a background area of the input image based on a specific threshold value, and the pixel value f(x,y) is greater than a specific threshold value It is determined as a pixel belonging to the object region. Conversely, if the pixel value f(x,y) is less than a specific threshold, it is determined as a pixel belonging to the background region.

For example, in order to convert an RGB image to a grayscale image, a method of converting an RGB color image to a grayscale image in OpenCV will be described.

One of the most widely used methods uses Luma Coding, the formula is as follows:

Y'= 0.2126 ＼times R'+ 0.7152 ＼times G'+ 0.0722 ＼times B'

Here, R', G', B'are Gamma-compressed values of the video system for the Red, Green, and Blue channels. Y'calculated from these is a nonlinear Luma component, which is a grayscaled value. The OpenCV Source Code, which converts RGB color to image Grayscale image based on this formula, is explained as follows.

[example code]

#include <opencv2/opencv.hpp>

#include <stdlib.h>

#include <stdio.h>

using namespace std;

using namespace cv;

/* function main */

int main( int argc, char** argv)

{

// Load an image

cv::Mat image = cv::imread( {YOUR_IMAGE_PATH}, 1 );

cv::Mat image_gray;

// Copy image

image.copyTo( image_gray );

int nRows = image.rows;

int nCols = image.cols;

// Convert to gray

float fGray = 0.0f;

float chBlue, chGreen, chRed;

for( int j = 0; j <nRows; j++) {

for( int i = 0; i <nCols; i++) {

chBlue = (float)( image.at<cv::Vec3b>(j,i)[0] );

chGreen = (float)( image.at<cv::Vec3b>(j,i)[1]);

chRed = (float)( image.at<cv::Vec3b>(j,i)[2] );

fGray = 0.2126f * chRed + 0.7152f * chGreen + 0.0722f * chBlue;

if( fGray <0.0) fGray = 0.0f;

if( fGray> 255.0) fGray = 255.0f;

image_gray.at<cv::Vec3b>(j,i)[0] = (int)fGray;

image_gray.at<cv::Vec3b>(j,i)[1] = (int)fGray;

image_gray.at<cv::Vec3b>(j,i)[2] = (int)fGray;

}

}

// Creates window

cv::namedWindow( "Image Original", CV_WINDOW_AUTOSIZE );

cv::namedWindow("Image Grayed", CV_WINDOW_AUTOSIZE);

// Show stuff

cv::imshow( "Image Original", image );

cv::imshow("Image Grayed", image_gray);

// Wait until user press some key

cv::waitKey();

return 0;

}

In the case of an RGB image, the image analysis SW uses a conventional RGB image processing algorithm according to a corresponding codec.

In the case of a YCbCr image, the image analysis SW uses a conventional YCbCr image processing algorithm according to a corresponding codec.

Wave is a swell created by attenuation as the storm on the surface of the water and the storm caused by the wind travel to other sea areas.

Shoreline observation is performed by superimposing two RGB images to obtain the average and variance of each pixel to generate an average image and a variance image (an image that visualizes the standard deviation of pixel intensity by overlapping images), and applies a specific threshold to the land surface. The boundary line where the sea level intersects is extracted.

Suspended sand observation measures the concentration of suspended sand according to the RGB intensity of the surface image of the sea/river/shore.

Suspended sediment (SS) is a floating sediment that floats from the bottom by the flow or wave of water in a river or shore and is moved underwater. It is a floating sediment that is transferred in a floating state in the water. Suspended sediment discharge is one of the important factors that determine the characteristics of streams. Information on particle size, distribution, or concentration per volume of suspended solids becomes basic information for characterization studies on the behavior of erosion and sedimentary environments in rivers or coastal areas, and big data obtained by continuously accumulating detailed information of suspended solids. ) Is used as important information for predicting topographic changes in rivers or coastal areas.

In the observation of the surface flow velocity, the lifetime image is observed, the movement path of the feature points such as water bubbles in the surface layer of the surface image is tracked, the moving distance per unit time is measured, and the surface velocity is observed. For example, in the lifetime image, the surface flow velocity is calculated by measuring the moving distance per unit time through which water bubbles flow.

The surface water temperature is measured using a near-infrared camera (thermal imaging camera).

In the observation of the surface flow velocity, the surface flow velocity is observed by observing the lifetime image and tracking the movement path of specific points such as water bubbles in the surface layer and measuring the moving distance per unit time.

The surface water temperature observation uses an infrared image (near infrared or thermal image) to observe the surface water temperature at sea level.

For the observation of the surface image, an RGB image, a grayscale image, a stereo image, an infrared image, etc. are selected and used as necessary.

In addition, as another embodiment 2, the image observing apparatus 100 uses a wireless IP camera having a 3G modem or an LTE 4G/5G modem that transmits an image of the camera through a mobile communication network, and a wireless IP camera -> NVR server and Storage -> It may be transmitted to the monitoring client of the user terminal 300.

In this case, when the server processes the sleep image by image processing to analyze the observation result, the observation result may be provided to the user terminal using the sleep image stored in the storage of the NVR server.

The marine environment monitoring data collection and analysis server 100 collects, stores, and analyzes the water surface image data of the sea/river/shore for monitoring the marine environment through a camera image photographing device. ), surface flow rate (km/h), surface water temperature, shoreline, suspended sand, and white cap observation results are provided.

5 is a diagram for explaining the structure of a wave.

Referring to the structure of wave, blue is defined as the highest part of the water surface as the floor, the lowest part as the valley, and the difference in height between the floor and the valley as wave height. The distance between them is defined as the wavelength and the time between them as the period. Elements that represent the characteristics of wave include wave height, period, wave direction, wavelength, wave pressure, and wave speed. Waves are typically deep water waves, where the motion of water particles by the waves hardly reaches the sea floor, and shallow water waves that sufficiently reach the sea floor due to shallow water depth. ). In addition, it is divided into long-period waves and short-period waves based on the period of the wave, and various classification criteria are applied as necessary.

Coastline (coastline or shoreline) refers to the boundary line where the land surface and sea level cross, and is called a water line or a fixed line. The sea level constantly rises and descends in tides and waves, so the location of the shoreline is not strictly constant. The coastline of Gojo is called the Gojo coastline, and the coastline of Lowjo is called the low tide coastline. However, in most cases, the coastline simply refers to the boundary between the average sea level and land.

Suspended sediment (suspended sediment or suspended load) is a floating sediment that is moved underwater by floating from the bottom by the flow or wave of water in a river or shore.

The floating sand is due to the turbulent flow of water that floats on the water in the flowing water. Most of the floating yarns are clay, silt, and fine yarn, and the floating yarns are distributed and transported evenly regardless of the water depth. Turbulence occurs due to disturbing motion caused by friction between the running water and the bed, and from this, it becomes active rapidly with an increase in flow rate and running water. This reaches its peak during the flood period, when suspended solids are also intensively transported.

The white cap refers to the head of the wave broken white by the influence of wind, etc., and plays an important role in gas exchange and heat exchange between the atmosphere and the ocean.

* Advantages of image processing-based sleep observation device

Each independent image observation device is capable of only point observation in the ocean, and image observation is capable of area observation.

The temporal resolution of the image processing-based sleep observation device is about 1 to 15 Hz, and the spatial resolution is about 0.1 m to 2.0 m, and provides high-resolution observation results in temporal and spatial terms.

The image processing-based sleep observing device can visually check the observation result, so it is convenient to check errors.

Since each observation item is time-synchronized (by analyzing the same data and extracting the observation result for each observation item), it is very effective in determining the cause of a specific phenomenon in the ocean.

Direct observation equipment is only available for point observation in the ocean, and image observation is possible for area observation.

Direct observation equipment is expensive to install and maintain observation equipment, and the possibility of equipment loss is very high.

Image processing-based sleep observation system and method

In order to acquire a surface image for monitoring the marine environment of the sea, river, and shore, the image acquisition timing and sampling interval of the surface image observation device are set according to the observation purpose, and through at least one camera of the surface image observation device 100 1 to control the acquisition timing and acquisition method of coastal images for sea/river/shore surface imaging, blue observation, surface flow velocity, surface water temperature, shoreline observation, floating sand observation, and white cap observation for the acquired surface image step; A second step of acquiring a sleep image from a camera of the sleep image observation apparatus 100; A third step of analyzing the image for wave observation, surface flow velocity, surface water temperature, shoreline observation, floating sand observation, and white cap observation of the water surface image acquired through the camera of the surface image observation device; Data collection and visualization server from the surface image observing device 100 for observation results including observation images (multiple observation pictures) about blue observation, shoreline observation, floating sand observation, and white cap to monitor the marine environment Step 4 of transmitting to 200;

The data collection and visualization server 200 stores wave observation, shoreline observation, floating sand observation, and white cap observation results, and displays information in real time and visually visualizes them; And data collection and visualization server 200 is interlocked with the user terminal (PC, smart phone) 300 through a wired/wireless communication network to monitor blue, shoreline, floating, and white cap visualized observation results in real time. It includes six steps to check.

The method further includes the step of observing the surface image and tracking the movement paths of the feature points of the water bubbles in the surface layer of the surface image, measuring the moving distance per unit time, and observing the surface flow velocity.

The method further includes the step of observing the surface water temperature at the sea level using an infrared image (near infrared ray or a thermal imaging camera image) when observing the surface water temperature.

The user terminal uses a PC or a smartphone,

For the observation of the water surface, an RGB image, a grayscale image, and an infrared image are selected and used as necessary.

6 is a photograph of a wave observation, a shoreline observation, a floating sand observation, and a white cap observation in the marine environment monitoring field according to an embodiment of the present invention.

In an embodiment, the temporal resolution of the image processing-based sleep observing apparatus is about 1 to 15 Hz, and the spatial resolution is about 0.1 to 2 m, and provides a high-resolution image temporally and spatially.

Analysis of wave, surface velocity, shoreline observation, and key changes in sea/river/shore water surface images using image processing-based water surface observation devices in the fields of marine environment monitoring of sea, river and coastal areas and coastal sedimentation monitoring. With the use of tools to monitor the marine environment and monitoring of coastal sedimentation and sedimentation, wave observation, shoreline observation, floating sand observation, and white cap observation analysis became possible.

Blue observation uses only 2 cameras. In contrast, observation of the shoreline, observation of suspended solids, and observation of white waves uses a single camera.

Blue observation analyzes the stereo image of the water surface from the water surface image (stereo image) acquired by two cameras, and observes the blue (wave height, period, wave direction) by reconstructing 3D phase data consisting of x, y, and z. do.

7 shows a wave observation process.

* Step A: Stereo image time series acquisition,

* Step B: Keypoint extraction of feature points using SURF (Speeded Up Robust Features) algorithm from each image, and SURF algorithm is Herbert Bay to improve the computation speed of SIFT (Scale Invariant Feature Transform) algorithm applied to feature point extraction. In 2008, the Hessian Matrix is applied based on the Multi-Scale Space Theory, and an integral image is used to reduce the amount of computation in this process. The integral image was proposed in 2001 by Paul Viola.

* Step C: Match the keypoints of the feature points extracted from each image,

* Step D: After removing the matching error (Mismatching keypoint, keypoint with a large error, etc.) through a RANSAC-based epipolar filter, the keypoints of the mutually matched feature points are removed, and then a rotation matrix (translation vector) is created, and the epipolar geometry is different It is the geometry of two scenes (stereo images, etc.) acquired from the orientation, and is a rotation matrix (translation vector) that contains information on internal parameters of the camera as well as the relative rotation and movement components of the two images. RANSAC (Random Sampling Consensus) is a method proposed in 1981 by Fischler and Bolles, and can be applied to data containing significant errors of a significant percentange, and is suitable for image analysis involving errors. The RANSAC method randomly extracts a model consisting of the smallest number from the corresponding points obtained after matching the feature points, and estimates the coefficients of the model using the least squares method. In addition, another model is created using the data not used above and the coefficients of the model are estimated. For all corresponding points, an error value with the base matrix obtained from each model is estimated, and then the number of error values smaller than a predetermined threshold is calculated. This is a method of repeating the above process up to the maximum number of iterations to select the model with the largest number of the models including an error value smaller than the threshold value as the final model.

* Step E: Reconstruct the image into 3D information (x, y, z) by applying the basic matrix between the images

* Step F: Visualization of results.

In the shoreline observation, an average image and a dispersion image are produced by overlapping images acquired from one or two cameras, and the boundary between the land surface and sea level is extracted for each cross-shore direction cross-section of the coast.

The average image is an image obtained by superimposing a plurality of images to visualize the average of the pixel intensity of each pixel.

The distributed image is an image obtained by superimposing a plurality of images to visualize the standard deviation of the pixel intensity of each pixel.

This shoreline observation provides shoreline observation results by extracting each from the average image and/or the variance image. In the coastline observation process, the shoreline observation is made from the variance image or the shoreline observation is made from the average image.

The coastline observation process is representative of the shoreline observation process in the variance image, and the same process is applied to the shoreline observation in the average image. The analysis process is the same for the coastline in the distributed image and the coastline in the average image, only the definition of the coastline is different.

6 shows a coastline observation process, and FIG. 8 shows an average image and a variance image.

The coastline observation process

* Step A: Time series image acquisition,

* Step B: Create an average image (an image that visualizes the average pixel intensity by overlapping images) and a distributed image (an image that visualizes the standard deviation of pixel intensity by overlapping images),

* Step C: Geometric correction based on RTK (Real Time Kinematic, real-time moving positioning) survey data,

* Step D: Remove noise in the analysis area by applying a threshold that is a reference when extracting the shoreline (primary noise removal),

* Step E: Extracting the possible section of the coastline (removing the secondary noise),

* Step F: Extraction of shoreline by baseline configured at regular intervals,

* Level G ~ Level H: Visualize coastline results by connecting points extracted from each baseline.

The white wave observation is used by converting the acquired sea surface image into a gray scale image. White cap is a phenomenon in which the head of the wave breaks at the sea level, and in this process, the white color appears due to the mixture of sea water and gas. In the image, the white wave has a much higher pixel intensity than the surrounding seawater surface.

10 shows pixel intensity characteristics when a white wave is generated, and FIG. 11 shows a process of observing a white wave.

* Step A: Time series image acquisition,

* Step B: Change the acquired seawater surface image to a gray scale image,

* Step C~D: Determine the boundary value that is the basis for separating seawater and white waves by analyzing images observed for 10 minutes or longer.

* Steps E~F: Extracting pixels with white waves based on the boundary value in each image.

The suspended solid observation is performed by converting the pixel intensity of RGB according to the concentration of the suspended solid into the suspended solid concentration in the water surface image of the seawater surface to observe the suspended solid. 12 shows the process of observing suspended solids.

* Step A: Time series image acquisition,

* Step B: Separate each band (R-band, G-band, B-band) of the RGB band image and perform geometric correction based on RTK (Real Time Kinematic, real-time moving positioning) survey data,

* Step C: Selecting the band to be applied in the target sea area, the selection criterion is the band that responds most sensitively to suspended sand, and the applied band varies depending on the water quality and ocean color of the target sea area.

* Step D: Calculate the pixel intensity of the applied band as the reference pixel intensity ratio,

* Step E: derive a correlation equation between the point-observed suspended solid concentration and pixel intensity ratio in the target sea area,

* Step F: The observed suspended solid concentration is expanded to a two-dimensional distribution by applying the pixel intensity ratio correlation equation and the result is visualized to observe the suspended solid.

In the observation of the surface flow velocity, the surface surface image of the sea level is observed, the movement paths of the feature points of the water bubbles in the surface layer of the surface image are tracked to measure the moving distance per unit time, and the surface flow velocity is observed.

The surface water temperature observation uses an infrared image (near infrared ray or thermal imaging camera image) to observe the surface water temperature at sea level.

For the observation of the surface image, an RGB image, a grayscale image, a stereo image, an infrared image, and the like are selected and used as necessary.

The embodiments according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include storage, magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Magneto-optical media, and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, etc. may be included. Examples of program instructions may include those produced by a compiler, and high-level language codes that can be executed by a computer using an interpreter as well as machine language codes. The hardware device may be configured to operate as one or more software modules to perform the operation of the present invention.

As described above, the method of the present invention is implemented as a program and can be read using software of a computer, such as a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.). ) Can be stored.

Although described with reference to a preferred embodiment of the present invention, various modifications or variations of the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims by those of ordinary skill in the relevant technical field You will understand that it can be done.

100: surface image observation device
110: camera unit 111: control unit
112: image analysis unit 113: storage unit
120: communication department
200: Data collection and visualization server
300: User terminal (PC, smartphone)

Claims (16)

In order to monitor the marine environment, the timing of image acquisition and sampling interval are controlled according to the observation purpose, and the acquired water surface image is analyzed for blue, surface flow rate, surface water temperature, shoreline, floating sand, and white waves, and observation results including observation images are derived. And a surface image observation device that transmits the observation result to a server;
A data collection and visualization server for collecting and visualizing observation results obtained from the surface image observation device; And
It includes a user terminal equipped with a program that checks the observation results of waves, surface flow rate, surface water temperature, shoreline, suspended sand, and white waves in real time,
In the blue observation, a stereo image of the water surface acquired by two cameras is analyzed and reconstructed into three-dimensional phase data composed of x, y, and z to observe a wave having a wave height, a period, and a wave direction of the wave wave,
In the shoreline observation, an average image and a dispersion image are produced by overlapping images acquired from two cameras, and the boundary between the land surface and sea level is extracted for each cross-shore direction of the coast,
The average image is an image obtained by superimposing a plurality of images to visualize the average of the pixel intensity of each pixel, and the distributed image is an image obtained by superimposing a plurality of images to visualize the standard deviation of the pixel intensity of each pixel,
The shoreline observation provides a shoreline observation result by extracting each from the average image and/or the variance image,
The white wave observation is used by converting the acquired image of the sea water surface into a gray scale image, and the white cap is a phenomenon in which the blue head is broken at the sea level, and in this process, white appears due to the mixture of sea water and gas. , When the white wave occurs, the white wave in the image has a relatively higher pixel intensity than the surrounding seawater surface.
In the process of observing white waves, Step A: Acquisition of time series images, Step B: Change the acquired sea surface image to a gray scale image, Steps C to D: Analyze the observed image for more than 10 minutes to determine the criteria for separating sea water and white waves. Determination of the threshold value, Step E~F: An image analysis-based surface observation system for monitoring the marine environment by extracting the pixels where the white wave has occurred based on the boundary value from each image and observing the white wave. The method of claim 1,
The temporal resolution of the surface image observation device is 1 to 15 Hz, and the spatial resolution is adjustable to 0.1 to 2.0 m, an image analysis-based surface observation system for marine environment monitoring. The method of claim 1,
The user terminal using a PC or smartphone, image analysis-based surface observation system for marine environment monitoring. The method of claim 1,
The surface image observation device is used as a camera image observation IoT device for monitoring the marine environment,
A camera unit that acquires a surface image for monitoring the marine environment of the sea, river, and shore;
A controller configured to set an image acquisition timing and a sampling frequency, transmit the observation result to a server, and perform overall control of the system;
An image analysis unit connected to the control unit and analyzing the acquired water surface image according to observation items of blue, surface flow velocity, surface water temperature, shoreline, floating sand, and white wave, and providing the observation result;
A storage unit connected to the control unit and configured to store observation results; And
A communication unit connected to the control unit and having a 3G modem or an LTE 4G/5G modem to transmit observation results to a server;
Image analysis-based surface observation system for monitoring the marine environment comprising a. The method of claim 4,
The sleep image observation device
An image analysis-based surface observation system for marine environment monitoring, connected to the control unit, and further comprising a display unit for displaying measurement time information, the observation result, and the observation picture. The method of claim 1,
The blue observation is
Step A: Acquire a stereo image time series,
Step B: Extract keypoints of feature points using SURF (Speeded Up Robust Features) algorithm from each image,
Step C: Match the keypoints of the feature points extracted from each image,
Step D: Remove matching errors (Mismatching keypoints, keypoints with large errors, etc.) through RANSAC-based epipolar geometry filtering for keypoints of mutually matched feature points, and then create a rotation matrix (translation vector).
Step E: Reconstruct the image into 3D information (x, y, z) by applying the basic matrix between the images,
Step F: Visualization of results,
Containing, image analysis-based surface observation system for marine environment monitoring. The method of claim 1,
The coastline observation process
Step A: Time series image acquisition,
Step B: Create an average image (an image that visualizes the average pixel intensity by overlapping images) and a distributed image (an image that visualizes the standard deviation of pixel intensity by overlapping images),
Step C: Geometric correction based on RTK (Real Time Kinematic, real-time moving positioning) survey data,
Step D: Remove noise in the analysis area by applying a threshold that is a reference when extracting the shoreline (primary noise removal),
Step E: Extracting the section where the coastline can exist (secondary noise removal),
Step F: Extraction of shoreline for each baseline configured at regular intervals, and
Steps G~H: An image analysis-based surface observation system for marine environment monitoring by visualizing the shoreline result by connecting points extracted from each baseline to extract the boundary line between the land surface and sea level. delete The method of claim 1,
The suspended solid observation is performed by converting the pixel intensity of RGB according to the concentration of the suspended solid to the suspended solid concentration in the surface image of the seawater surface to observe the suspended solid,
The process of observation of suspended solids
* Step A: Time series image acquisition,
* Step B: Separate each band (R-band, G-band, B-band) of the RGB band image and perform geometric correction based on RTK (Real Time Kinematic, real-time moving positioning) survey data,
* Step C: Selecting the band to be applied in the target sea area, the selection criterion is the band that responds most sensitively to suspended sand, and the applied band varies depending on the water quality and ocean color of the target sea area.
* Step D: Calculate the pixel intensity of the applied band as the reference pixel intensity ratio,
* Step E: derive a correlation equation between the point-observed suspended solid concentration and pixel intensity ratio in the target sea area,
* Step F: Image analysis-based surface observation system for monitoring the marine environment, expanding the observed suspended solid concentration into a two-dimensional distribution by applying the pixel intensity ratio correlation equation and visualizing the result to observe the suspended solid. The method of claim 1,
The surface flow velocity observation is an image analysis-based surface observation system for monitoring the marine environment, by observing the surface image and tracking the movement path of the feature points of the water bubbles in the surface layer of the surface image and measuring the moving distance per unit time to observe the surface flow velocity. The method of claim 1,
The surface water temperature observation is an image analysis-based surface observation system for monitoring the marine environment by observing the surface water temperature of the sea surface using an infrared image (near infrared or thermal imaging camera image). The method of claim 1,
The surface image observation is an image analysis-based surface observation system for marine environment monitoring by selecting and using an RGB image, a grayscale image, and an infrared image as necessary. delete delete delete delete

Images